[Intelmq-dev] EleasticSearch Know-How needed

[Filip Pokorny]

Hi everyone,

we could simplify ES setup for IntelMQ based on our setup:

Last bot in the pipeline puts the event in another redis queue where it
is picked up by Logstash (free part of Elastic stack) where optional
mutations can be applied etc. and stored in the db. This setup does not
need any special elasticsearch bot on the IntelMQ side. Then the es bot
could be deprecated.

I find this solution better than having ES output bot, because we would
be pretty much reinventing the wheel, Logstash works well and can work
for other data sources as well (which is also our case where we store
other data in the cluster ES besides IntelMQ events).

I can put together simple guide to be placed in the docs if there is
interest.

Filip
CSIRT.CZ

On 3/24/20 5:09 PM, Thomas Hungenberg wrote:
> Maybe of interest?
> 
> "In response to the COVID-19 crisis, we're offering *free* on-demand training courses
>  over the next few weeks to support our #Elasticsearch community."
> 
> https://twitter.com/elastic/status/1242175551206494209?s=09
> 
> 
>      - Thomas
> 
> CERT-Bund Incident Response & Malware Analysis Team
> 
> 
> On 20.03.20 16:20, Sebastian Wagner wrote:
>> Dear community,
>>
>> The ElasticSearch bots, tests and tools in IntelMQ need some maintenance
>> which I am unable to provide. As ES is a very common tool I am sure that
>> there is know-how available in the community and we are able to continue
>> the support for it.
>>
>> The oldest know issue is a broken unittest:
>> https://github.com/certtools/intelmq/issues/1480
>>
>> But there are also incompatibilties with current ElasticSearch version,
>> e.g. I had problems with the elasticmapper tool using ES 7.6.1 (maybe
>> easy to fix).
>> Using 7.5.0 failed on the indices tests
>> https://github.com/certtools/intelmq/issues/1479
>>
>> Further, the only supported elasticsearch python library version is
>> currently 'elasticsearch>=5.0.0,<6.0.0' while the latest release is 7.6.0.
>>
>> Please consider contributing
>>
>> best regards
>> Sebastian
>>
>>
>> _______________________________________________
>> Intelmq-dev mailing list
>> Intelmq-dev at lists.cert.at
>> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
>>
> 
> _______________________________________________
> Intelmq-dev mailing list
> Intelmq-dev at lists.cert.at
> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
>