Dear Mika,
On 8/8/22 1:48 PM, Mika Silander wrote:
A few issues still remain to be solved before going to production and one of them is how to collect statistics on events. It looks to me that lib/bot.py contains code for collecting statistics and pushing cached data to a Redis instance, but I'd like to know if there's documentation describing the statistics collection in general and recommendations on its use? Any best practices or similar docs?
No, unfortunately not. More work was intended, but as always, priorities shift. Therefore these internal bot statistics are in "beta" state and not enabled by default (at least they were, back then...). There is only one script cronjob_intelmq_statistics.py (https://github.com/certtools/intelmq/tree/develop/contrib/check_mk) which queries these values and feeds them to a Check MK instance (but without doing any alerting)
I see there is the EventDB implementation but that in turn seems to be limited to statistics data being pushed out from intelmq via an SQL output bot.
yes, that's something else.
- Internal statistics on bots: Optional feature in IntelMQ Core (implemented in lib/bot.py by Edvard Rejthar) which saves counters of successful and failed process loops. Similar to the "X messages since last logging" messages, but more frequently updated. - EventDB statistics: "EventDB" is called an optional Postgres (or similar) database, fed by an SQL Output. It holds the event data and statistics means stats over the events. See https://intelmq.readthedocs.io/en/latest/user/eventdb.html
In our case we'd also need to keep a count on events being fed into intelmq. We'll probably also need some intermediate collection points within our bot net to keep track of the number of anomalous etc events. Therefore, leveraging somehow the constructs within lib/bot.py seems a better choice for us.
Yes, that's implemented in __stats: https://github.com/certtools/intelmq/blob/21ffafcf3d72af8aa7cfcdf8444622b00a...
Contributions to make this data useful, are welcome :)
best regards Sebastian
As always, pointers are welcome and please correct me if I've understood something wrong above.
Br, Mika _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/