[IntelMQ-dev] Documentation on bot statistics collection

[Mika Silander]

Hi Sebastian,

 Thank you for shedding light on the current status on intelmq's statistics mngt. Knowing now that info on the subject is scarce saves me and others a lot of time.

Br, Mika

----- Original Message -----
From: "Sebix" <sebix at sebix.at>
To: "Mika Silander" <mika.silander at csc.fi>, "intelmq-dev" <intelmq-dev at lists.cert.at>
Sent: Tuesday, 9 August, 2022 12:32:07
Subject: Re: [IntelMQ-dev] Documentation on bot statistics collection

Dear Mika,

On 8/8/22 1:48 PM, Mika Silander wrote:
>  A few issues still remain to be solved before going to production and one of them is how to collect statistics on events. It looks to me that lib/bot.py contains code for collecting statistics and pushing cached data to a Redis instance, but I'd like to know if there's documentation describing the statistics collection in general and recommendations on its use? Any best practices or similar docs?
No, unfortunately not. More work was intended, but as always, priorities
shift. Therefore these internal bot statistics are in "beta" state and
not enabled by default (at least they were, back then...).
There is only one script cronjob_intelmq_statistics.py
(https://github.com/certtools/intelmq/tree/develop/contrib/check_mk)
which queries these values and feeds them to a Check MK instance (but
without doing any alerting)
>  I see there is the EventDB implementation but that in turn seems to be limited to statistics data being pushed out from intelmq via an SQL output bot.

yes, that's something else.

- Internal statistics on bots: Optional feature in IntelMQ Core
(implemented in lib/bot.py by Edvard Rejthar) which saves counters of
successful and failed process loops. Similar to the "X messages since
last logging" messages, but more frequently updated.
- EventDB statistics: "EventDB" is called an optional Postgres (or
similar) database, fed by an SQL Output. It holds the event data and
statistics means stats over the events. See
https://intelmq.readthedocs.io/en/latest/user/eventdb.html

> In our case we'd also need to keep a count on events being fed into intelmq. We'll probably also need some intermediate collection points within our bot net to keep track of the number of anomalous etc events. Therefore, leveraging somehow the constructs within lib/bot.py seems a better choice for us.

Yes, that's implemented in __stats:
https://github.com/certtools/intelmq/blob/21ffafcf3d72af8aa7cfcdf8444622b00a41bea3/intelmq/lib/bot.py#L453

Contributions to make this data useful, are welcome :)

best regards
Sebastian

>  As always, pointers are welcome and please correct me if I've understood something wrong above.
>
> Br, Mika
> _______________________________________________
> IntelMQ-dev mailing list
> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
> https://intelmq.readthedocs.io/

-- 
Institute for Common Good Technology
gemeinnütziger Kulturverein - nonprofit cultural society
https://sebix.at/
ZVR 1510673578