Dear community,
The ElasticSearch bots, tests and tools in IntelMQ need some maintenance which I am unable to provide. As ES is a very common tool I am sure that there is know-how available in the community and we are able to continue the support for it.
The oldest know issue is a broken unittest: https://github.com/certtools/intelmq/issues/1480
But there are also incompatibilties with current ElasticSearch version, e.g. I had problems with the elasticmapper tool using ES 7.6.1 (maybe easy to fix). Using 7.5.0 failed on the indices tests https://github.com/certtools/intelmq/issues/1479
Further, the only supported elasticsearch python library version is currently 'elasticsearch>=5.0.0,<6.0.0' while the latest release is 7.6.0.
Please consider contributing
best regards Sebastian
Maybe of interest?
"In response to the COVID-19 crisis, we're offering *free* on-demand training courses over the next few weeks to support our #Elasticsearch community."
https://twitter.com/elastic/status/1242175551206494209?s=09
- Thomas
CERT-Bund Incident Response & Malware Analysis Team
On 20.03.20 16:20, Sebastian Wagner wrote:
Dear community,
The ElasticSearch bots, tests and tools in IntelMQ need some maintenance which I am unable to provide. As ES is a very common tool I am sure that there is know-how available in the community and we are able to continue the support for it.
The oldest know issue is a broken unittest: https://github.com/certtools/intelmq/issues/1480
But there are also incompatibilties with current ElasticSearch version, e.g. I had problems with the elasticmapper tool using ES 7.6.1 (maybe easy to fix). Using 7.5.0 failed on the indices tests https://github.com/certtools/intelmq/issues/1479
Further, the only supported elasticsearch python library version is currently 'elasticsearch>=5.0.0,<6.0.0' while the latest release is 7.6.0.
Please consider contributing
best regards Sebastian
Intelmq-dev mailing list Intelmq-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
Hi everyone,
we could simplify ES setup for IntelMQ based on our setup:
Last bot in the pipeline puts the event in another redis queue where it is picked up by Logstash (free part of Elastic stack) where optional mutations can be applied etc. and stored in the db. This setup does not need any special elasticsearch bot on the IntelMQ side. Then the es bot could be deprecated.
I find this solution better than having ES output bot, because we would be pretty much reinventing the wheel, Logstash works well and can work for other data sources as well (which is also our case where we store other data in the cluster ES besides IntelMQ events).
I can put together simple guide to be placed in the docs if there is interest.
Filip CSIRT.CZ
On 3/24/20 5:09 PM, Thomas Hungenberg wrote:
Maybe of interest?
"In response to the COVID-19 crisis, we're offering *free* on-demand training courses over the next few weeks to support our #Elasticsearch community."
https://twitter.com/elastic/status/1242175551206494209?s=09
- ThomasCERT-Bund Incident Response & Malware Analysis Team
On 20.03.20 16:20, Sebastian Wagner wrote:
Dear community,
The ElasticSearch bots, tests and tools in IntelMQ need some maintenance which I am unable to provide. As ES is a very common tool I am sure that there is know-how available in the community and we are able to continue the support for it.
The oldest know issue is a broken unittest: https://github.com/certtools/intelmq/issues/1480
But there are also incompatibilties with current ElasticSearch version, e.g. I had problems with the elasticmapper tool using ES 7.6.1 (maybe easy to fix). Using 7.5.0 failed on the indices tests https://github.com/certtools/intelmq/issues/1479
Further, the only supported elasticsearch python library version is currently 'elasticsearch>=5.0.0,<6.0.0' while the latest release is 7.6.0.
Please consider contributing
best regards Sebastian
Intelmq-dev mailing list Intelmq-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
Intelmq-dev mailing list Intelmq-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
On 24.03.2020, at 18:34, Filip Pokorny filip.pokorny@csirt.cz wrote:
Hi everyone,
we could simplify ES setup for IntelMQ based on our setup:
Last bot in the pipeline puts the event in another redis queue where it is picked up by Logstash (free part of Elastic stack) where optional mutations can be applied etc. and stored in the db. This setup does not need any special elasticsearch bot on the IntelMQ side. Then the es bot could be deprecated.
I find this solution better than having ES output bot, because we would be pretty much reinventing the wheel, Logstash works well and can work for other data sources as well (which is also our case where we store other data in the cluster ES besides IntelMQ events).
I agree with this approach. It keeps the duties where they can be handled better. Logstash is better at that for getting things into ES. But yes, it needs an example and/or tutorial.
I can put together simple guide to be placed in the docs if there is interest.
That would be great!
Best, Aaron.
Hi all,
Good news, Stone Zach is able to make required changes for ElasticSearch 7, see https://github.com/certtools/intelmq/issues/1479#issuecomment-605454294 and https://github.com/certtools/intelmq/pull/1513 - Thanks stone!
However the code for ES 7 is incompatible to the code for ES 5 and Stone can only support the ES 7 code. That would mean that - if nothing changes - the support for ES 5 would be probably dropped in the next feature release (2.2.0).
In parallel to these efforts, I welcome the contribution by Filip to document how IntelMQ can be used to feed ES via Redis + Logstash. Thanks!
Sebastian
On 3/20/20 4:20 PM, Sebastian Wagner wrote:
Dear community,
The ElasticSearch bots, tests and tools in IntelMQ need some maintenance which I am unable to provide. As ES is a very common tool I am sure that there is know-how available in the community and we are able to continue the support for it.
The oldest know issue is a broken unittest: https://github.com/certtools/intelmq/issues/1480
But there are also incompatibilties with current ElasticSearch version, e.g. I had problems with the elasticmapper tool using ES 7.6.1 (maybe easy to fix). Using 7.5.0 failed on the indices tests https://github.com/certtools/intelmq/issues/1479
Further, the only supported elasticsearch python library version is currently 'elasticsearch>=5.0.0,<6.0.0' while the latest release is 7.6.0.
Please consider contributing
best regards Sebastian
Intelmq-dev mailing list Intelmq-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
Elastic, like many other datastores, is not very frequently updated. Organizations stay on older version for longer duration. Can we keep both versions? We can focus maintenance efforts only on the latest version. As for shipping to ES via LogStash, an alternative could be to use filebeat with file-output bot. I have seen setups where out of ELK, L(ogstatsh) is not used.
On Mon, Mar 30, 2020 at 9:44 AM Sebastian Wagner wagner@cert.at wrote:
Hi all,
Good news, Stone Zach is able to make required changes for ElasticSearch 7, see https://github.com/certtools/intelmq/issues/1479#issuecomment-605454294 and https://github.com/certtools/intelmq/pull/1513 - Thanks stone!
However the code for ES 7 is incompatible to the code for ES 5 and Stone can only support the ES 7 code. That would mean that - if nothing changes - the support for ES 5 would be probably dropped in the next feature release (2.2.0).
In parallel to these efforts, I welcome the contribution by Filip to document how IntelMQ can be used to feed ES via Redis + Logstash. Thanks!
Sebastian On 3/20/20 4:20 PM, Sebastian Wagner wrote:
Dear community,
The ElasticSearch bots, tests and tools in IntelMQ need some maintenance which I am unable to provide. As ES is a very common tool I am sure that there is know-how available in the community and we are able to continue the support for it.
The oldest know issue is a broken unittest:https://github.com/certtools/intelmq/issues/1480
But there are also incompatibilties with current ElasticSearch version, e.g. I had problems with the elasticmapper tool using ES 7.6.1 (maybe easy to fix). Using 7.5.0 failed on the indices testshttps://github.com/certtools/intelmq/issues/1479
Further, the only supported elasticsearch python library version is currently 'elasticsearch>=5.0.0,<6.0.0' while the latest release is 7.6.0.
Please consider contributing
best regards Sebastian
Intelmq-dev mailing listIntelmq-dev@lists.cert.athttps://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
-- // Sebastian Wagner wagner@cert.at wagner@cert.at - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
Intelmq-dev mailing list Intelmq-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
Hi Navtej,
On 4/5/20 7:47 PM, Navtej Singh wrote:
Elastic, like many other datastores, is not very frequently updated. Organizations stay on older version for longer duration. Can we keep both versions?
Sure. However, that requires some maintenance which I am unable to provide alone. I would expect that the major element of the maintenance is testing. For example for the upcoming changes/additions in the ES output bot, we would need your help in testing the backwards compatibility with older ES versions.
As for shipping to ES via LogStash, an alternative could be to use filebeat with file-output bot. I have seen setups where out of ELK, L(ogstatsh) is not used.
Thanks for the idea. Would anyone on the list be willing to contribute documentation for this? - The same applies to the Logstash-idea.
best wishes Sebastian