On 24.03.2020, at 18:34, Filip Pokorny filip.pokorny@csirt.cz wrote:
Hi everyone,
we could simplify ES setup for IntelMQ based on our setup:
Last bot in the pipeline puts the event in another redis queue where it is picked up by Logstash (free part of Elastic stack) where optional mutations can be applied etc. and stored in the db. This setup does not need any special elasticsearch bot on the IntelMQ side. Then the es bot could be deprecated.
I find this solution better than having ES output bot, because we would be pretty much reinventing the wheel, Logstash works well and can work for other data sources as well (which is also our case where we store other data in the cluster ES besides IntelMQ events).
I agree with this approach. It keeps the duties where they can be handled better. Logstash is better at that for getting things into ES. But yes, it needs an example and/or tutorial.
I can put together simple guide to be placed in the docs if there is interest.
That would be great!
Best, Aaron.