Hi Kamil & all,

From my experience it looks like collecting statistics varies wildly based on each team's particular needs. In our case, the built-in statistics functionality does not provide all the info we need so we wrote a simple expert bot to collect statistics on the events processed (IPs, constituency, classification.type, abuse C address etc etc). It outputs the statistics to rsyslog. In principle, we can put another instance of the bot to a latter stage in the bot chain to send the same information about events that have passed all sanity checks done by expert bots in between. This way the same bot answers the questions a) how many events have we processed, b) how many of these events have been successfully passed on to our clients (=constituency). Finally, the ticketing system gives us statistics about how many of those events have been properly resolved, but that's out of scope in terms of generating statistics inside intelmq.

Don't take me wrong but in this case I would vote for removing the initial statistics functionality from intelmq and keep statistics collection completely separate. This would keep the intelmq code base also (marginally) smaller.

I also remember a recommendation on this forum that intelmq instances should be monitored with the help of the log files created. I see monitoring and statistics collection as two orthogonal functionalities and this is the other reason why I would like to keep statistics collection independent, i.e. neither as part of the core modules of intelmq, nor as an indirect means of monitoring.

Well, this is just my view on the issue (the famous 5 cents), and, I am unaware of other use cases and requirements that could tilt the balance towards the current way of implementing statistics.

Br, Mika

----- Original Message ----- From: "Kamil Mankowski via IntelMQ-dev" intelmq-dev@lists.cert.at Cc: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 27 November, 2023 13:23:56 Subject: Re: [IntelMQ-dev] STATS IN INTELMQ?

Hi all,

I think there is currently no ongoing work for any improved statistics directly in the IntelMQ, but if you have development capacity to extend the current state, I think it could be useful.

However, I can say what is currently available, and how I use/plan to use it:

1) for final events, we use - as mentioned by Aaron - a database. We are going to use Timescale DB, as described in: https://docs.intelmq.org/develop/admin/database/postgresql/#using-eventdb-wi... Currently we have some set of scripts generating stats, and we plan to move fully to TimescaleDB+Grafana.

2) for monitoring the ongoing work, there are basic stats exposed in the database 3 in Redis (see changelog: https://docs.intelmq.org/develop/changelog/?h=statistics#configurations). I think this feature isn't well documented. It's not perfect, but I use it to keep an eye on the botnet & failures, using a custom scripts to integrate it with CheckMK monitoring and alert on troubles.

Best regards

// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 11/27/23 11:51, L. Aaron Kaplan wrote:

Hi,

Most users of intelmq use an "eventsDB" as an output . From there, it is usually quite doable to do stats on top of events.

I did an initial version for CERT.at back then which is still here: https://github.com/certtools/stats-portal You can build on top of this if it suits you.

Hope it helps, Aaron.

...

On 27.11.2023, at 11:37, Homma, L.J. (Luitzen) via IntelMQ-dev intelmq-dev@lists.cert.at wrote:

Dear IntelMQ Developers & Users,

We are curious if there are any plans on the roadmap to incorporate statistical features into IntelMQ. About 1.5 years ago, we participated in an online session where it was mentioned that there were some early plans to integrate stats into IntelMQ. As far as we could find, there have not been any steps in this direction. Are we correct?

Currently, we are working in our experimental environment to develop stats based on a Prometheus bot, using Prometheus as a time-series database, and utilizing Grafana for dashboarding and visualization. Are there more members of the community working on this? Our goal is to gain better insights into the input, filtering, and output of our IntelMQ pipeline(s). We hope to hear from others about their thoughts on this.

Met vriendelijke groet,

Luitzen Homma Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is gezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten.

This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages. _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

---

IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

_______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

Hi Kamil,

Thanks for the clarification. I've also noticed there are some hurdles and problems when relying entirely on logs for monitoring. Just out of curiosity, what are the operational monitoring needs you would tackle with statistics collection in intelmq (and can't be covered using normal logs)? I think we have managed so far with using the logs as the only source of information for monitoring, and it would be interesting to compare and see if we've possibly forgotten some monitoring target/issue.

Br, Mika

----- Original Message ----- From: "Kamil Mankowski via IntelMQ-dev" intelmq-dev@lists.cert.at To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 27 November, 2023 14:59:34 Subject: Re: [IntelMQ-dev] STATS IN INTELMQ?

Hi Mika,

Thanks for the comment. I totally agree that the statistics needed are definitely different for different teams.

I would vote for removing the initial statistics functionality from

intelmq and keep statistics collection completely separate.

I can't agree with that, because there are two different types of statistics - as you said, the monitoring and the business statistics. I think I used the word "statistics", but what I meant with my second point is purely operational monitoring, and I'd like to keep the built-in functionality operation-oriented. Monitoring through logs is also possible, although it has some drawbacks.

However, there have been some ideas to introduce hook capabilities. If we want to change the monitoring (e.g. integrate with Open Telemetry) and/or allow more statistics directly from IntelMQ, I'd go in the direction of optional support via hooks, without extending the base source code.

Best regards

// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 11/27/23 13:41, Mika Silander wrote:

Hi Kamil & all,

From my experience it looks like collecting statistics varies wildly based on each team's particular needs. In our case, the built-in statistics functionality does not provide all the info we need so we wrote a simple expert bot to collect statistics on the events processed (IPs, constituency, classification.type, abuse C address etc etc). It outputs the statistics to rsyslog. In principle, we can put another instance of the bot to a latter stage in the bot chain to send the same information about events that have passed all sanity checks done by expert bots in between. This way the same bot answers the questions a) how many events have we processed, b) how many of these events have been successfully passed on to our clients (=constituency). Finally, the ticketing system gives us statistics about how many of those events have been properly resolved, but that's out of scope in terms of generating statistics inside intelmq.

Don't take me wrong but in this case I would vote for removing the initial statistics functionality from intelmq and keep statistics collection completely separate. This would keep the intelmq code base also (marginally) smaller.

I also remember a recommendation on this forum that intelmq instances should be monitored with the help of the log files created. I see monitoring and statistics collection as two orthogonal functionalities and this is the other reason why I would like to keep statistics collection independent, i.e. neither as part of the core modules of intelmq, nor as an indirect means of monitoring.

Well, this is just my view on the issue (the famous 5 cents), and, I am unaware of other use cases and requirements that could tilt the balance towards the current way of implementing statistics.

Br, Mika

----- Original Message ----- From: "Kamil Mankowski via IntelMQ-dev" intelmq-dev@lists.cert.at Cc: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 27 November, 2023 13:23:56 Subject: Re: [IntelMQ-dev] STATS IN INTELMQ?

Hi all,

I think there is currently no ongoing work for any improved statistics directly in the IntelMQ, but if you have development capacity to extend the current state, I think it could be useful.

However, I can say what is currently available, and how I use/plan to use it:

1. for final events, we use - as mentioned by Aaron - a database. We are

going to use Timescale DB, as described in: https://docs.intelmq.org/develop/admin/database/postgresql/#using-eventdb-wi... Currently we have some set of scripts generating stats, and we plan to move fully to TimescaleDB+Grafana.

2. for monitoring the ongoing work, there are basic stats exposed in the

database 3 in Redis (see changelog: https://docs.intelmq.org/develop/changelog/?h=statistics#configurations). I think this feature isn't well documented. It's not perfect, but I use it to keep an eye on the botnet & failures, using a custom scripts to integrate it with CheckMK monitoring and alert on troubles.

Best regards

// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 11/27/23 11:51, L. Aaron Kaplan wrote:

...

Hi,

Most users of intelmq use an "eventsDB" as an output . From there, it is usually quite doable to do stats on top of events.

I did an initial version for CERT.at back then which is still here: https://github.com/certtools/stats-portal You can build on top of this if it suits you.

Hope it helps, Aaron.

...

On 27.11.2023, at 11:37, Homma, L.J. (Luitzen) via IntelMQ-dev intelmq-dev@lists.cert.at wrote:

Dear IntelMQ Developers & Users,

We are curious if there are any plans on the roadmap to incorporate statistical features into IntelMQ. About 1.5 years ago, we participated in an online session where it was mentioned that there were some early plans to integrate stats into IntelMQ. As far as we could find, there have not been any steps in this direction. Are we correct?

Currently, we are working in our experimental environment to develop stats based on a Prometheus bot, using Prometheus as a time-series database, and utilizing Grafana for dashboarding and visualization. Are there more members of the community working on this? Our goal is to gain better insights into the input, filtering, and output of our IntelMQ pipeline(s). We hope to hear from others about their thoughts on this.

Met vriendelijke groet,

Luitzen Homma Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is gezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten.

This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages. _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

---

IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

---

IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/ _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

_______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

Dear Kamil,

On 11/27/23 14:48, Kamil Mankowski via IntelMQ-dev wrote:

I collect the number of events processed, overall and separately for each destination path, the number of errors, bot runs, and the size of the input queue. The last one is not visible in the logs, the number of processed with some (larger) delay, and the separation for target paths only in debug mode (as far as I can remember; although this is rarely useful).

Do you still use these check_mk scripts that I wrote some years ago?

https://github.com/certtools/intelmq/tree/develop/contrib/check_mk

best regards Sebastian

IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578

On 11/27/23 14:56, Kamil Mankowski wrote:

Nope, I've written a new one in Python & with more capabilities (e.g. each bot is a separated service in the CheckMK).

Is that Open Source too?

Sebastian

Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578

Hi Kamil,

Thanks for elaborating on your monitoring setup and needs. Here's a description of ours:

- number of events: our event logger expert, we can duplicate it on the paths that need to be monitored - our bot net is small so it is not an issue, but for a bigger one the number of extra bots due to this could be big

- number of errors: from the logs for those bots that do report them to their log files - the weakness for us here is one can't rely on uniform/consistent reporting of errors from one bot to another (depends on implementation of each, right?) - and, we do check all log files for exceptions, but an exception is usually an indication there could be a severe implementation problem in the bot itself, - and, we monitor presence of event dump files

- number of bot runs / run frequency can be pulled from log files assuming the log level is sufficiently talkative

- we don't monitor the size of the input queues since the number does not indicate a problem nor tell if the processing of events progresses. One should check if individual events "move forward" in the bot chain to answer that. At most I'd monitor the number does not grow past some allowed maximum and the same information can probably be pulled from redis (or the message queue used) using their own tools or some wrapper scripts(?)

- whether bots are running: we use "intelmqctl status" and just analyse every bot's status line. Is this too an innocent approach?

- whether collectors are producing new data: we can detect this indirectly by analysing the logs of the parsers, i.e. do events get forwarded sufficiently often? - this requires of course we have parsers that write something useful to the logs when they receive a message from a collector and forward events.

All in all, I think we aggregate almost the same monitoring data although we have to ensure sufficiently detailed information is written to the logs ... so we also monitor the log level remains as INFO or chattier :-). It would be handy to be able to set the log level per bot to reduce the logging where applicable ... have to check if this can be done.

Br, Mika

----- Original Message ----- From: "Kamil Mankowski via IntelMQ-dev" intelmq-dev@lists.cert.at To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 27 November, 2023 15:48:39 Subject: Re: [IntelMQ-dev] STATS IN INTELMQ?

I collect the number of events processed, overall and separately for each destination path, the number of errors, bot runs, and the size of the input queue. The last one is not visible in the logs, the number of processed with some (larger) delay, and the separation for target paths only in debug mode (as far as I can remember; although this is rarely useful).

I also check whether the bots are running, whether the collectors are producing new data, and the error rate, as well as the execution of periodic jobs (db updates etc). It's not perfect, but it works pretty well :)

Best regards

// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 11/27/23 14:31, Mika Silander wrote:

Hi Kamil,

Thanks for the clarification. I've also noticed there are some hurdles and problems when relying entirely on logs for monitoring. Just out of curiosity, what are the operational monitoring needs you would tackle with statistics collection in intelmq (and can't be covered using normal logs)? I think we have managed so far with using the logs as the only source of information for monitoring, and it would be interesting to compare and see if we've possibly forgotten some monitoring target/issue.

Br, Mika

----- Original Message ----- From: "Kamil Mankowski via IntelMQ-dev" intelmq-dev@lists.cert.at To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 27 November, 2023 14:59:34 Subject: Re: [IntelMQ-dev] STATS IN INTELMQ?

Hi Mika,

Thanks for the comment. I totally agree that the statistics needed are definitely different for different teams.

...

I would vote for removing the initial statistics functionality from

intelmq and keep statistics collection completely separate.

I can't agree with that, because there are two different types of statistics - as you said, the monitoring and the business statistics. I think I used the word "statistics", but what I meant with my second point is purely operational monitoring, and I'd like to keep the built-in functionality operation-oriented. Monitoring through logs is also possible, although it has some drawbacks.

However, there have been some ideas to introduce hook capabilities. If we want to change the monitoring (e.g. integrate with Open Telemetry) and/or allow more statistics directly from IntelMQ, I'd go in the direction of optional support via hooks, without extending the base source code.

Best regards

// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 11/27/23 13:41, Mika Silander wrote:

...

Hi Kamil & all,

From my experience it looks like collecting statistics varies wildly based on each team's particular needs. In our case, the built-in statistics functionality does not provide all the info we need so we wrote a simple expert bot to collect statistics on the events processed (IPs, constituency, classification.type, abuse C address etc etc). It outputs the statistics to rsyslog. In principle, we can put another instance of the bot to a latter stage in the bot chain to send the same information about events that have passed all sanity checks done by expert bots in between. This way the same bot answers the questions a) how many events have we processed, b) how many of these events have been successfully passed on to our clients (=constituency). Finally, the ticketing system gives us statistics about how many of those events have been properly resolved, but that's out of scope in terms of generating statistics inside intelmq.

Don't take me wrong but in this case I would vote for removing the initial statistics functionality from intelmq and keep statistics collection completely separate. This would keep the intelmq code base also (marginally) smaller.

I also remember a recommendation on this forum that intelmq instances should be monitored with the help of the log files created. I see monitoring and statistics collection as two orthogonal functionalities and this is the other reason why I would like to keep statistics collection independent, i.e. neither as part of the core modules of intelmq, nor as an indirect means of monitoring.

Well, this is just my view on the issue (the famous 5 cents), and, I am unaware of other use cases and requirements that could tilt the balance towards the current way of implementing statistics.

Br, Mika

----- Original Message ----- From: "Kamil Mankowski via IntelMQ-dev" intelmq-dev@lists.cert.at Cc: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 27 November, 2023 13:23:56 Subject: Re: [IntelMQ-dev] STATS IN INTELMQ?

Hi all,

I think there is currently no ongoing work for any improved statistics directly in the IntelMQ, but if you have development capacity to extend the current state, I think it could be useful.

However, I can say what is currently available, and how I use/plan to use it:

1. for final events, we use - as mentioned by Aaron - a database. We are

going to use Timescale DB, as described in: https://docs.intelmq.org/develop/admin/database/postgresql/#using-eventdb-wi... Currently we have some set of scripts generating stats, and we plan to move fully to TimescaleDB+Grafana.

2. for monitoring the ongoing work, there are basic stats exposed in the

database 3 in Redis (see changelog: https://docs.intelmq.org/develop/changelog/?h=statistics#configurations). I think this feature isn't well documented. It's not perfect, but I use it to keep an eye on the botnet & failures, using a custom scripts to integrate it with CheckMK monitoring and alert on troubles.

Best regards

// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 11/27/23 11:51, L. Aaron Kaplan wrote:

...

Hi,

Most users of intelmq use an "eventsDB" as an output . From there, it is usually quite doable to do stats on top of events.

I did an initial version for CERT.at back then which is still here: https://github.com/certtools/stats-portal You can build on top of this if it suits you.

Hope it helps, Aaron.

...

On 27.11.2023, at 11:37, Homma, L.J. (Luitzen) via IntelMQ-dev intelmq-dev@lists.cert.at wrote:

Dear IntelMQ Developers & Users,

We are curious if there are any plans on the roadmap to incorporate statistical features into IntelMQ. About 1.5 years ago, we participated in an online session where it was mentioned that there were some early plans to integrate stats into IntelMQ. As far as we could find, there have not been any steps in this direction. Are we correct?

Currently, we are working in our experimental environment to develop stats based on a Prometheus bot, using Prometheus as a time-series database, and utilizing Grafana for dashboarding and visualization. Are there more members of the community working on this? Our goal is to gain better insights into the input, filtering, and output of our IntelMQ pipeline(s). We hope to hear from others about their thoughts on this.

Met vriendelijke groet,

Luitzen Homma Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is gezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten.

This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages. _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

---

IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

---

IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/ _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

---

IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/ _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

_______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

I think this discussion shows that it would be two good HOWTO documents (statistics and a monitoring HOWTO) in the updated documentation branch ;-)

What do you think?

On 27.11.2023, at 16:23, Kamil Mankowski via IntelMQ-dev intelmq-dev@lists.cert.at wrote:

Signed PGP part Hi Mika,

Thanks for the explanation of your monitoring. I think we monitor mostly similar things, but there is a small misunderstanding - all things from the second paragraph of my message do not come from the statistics in redis db, but mostly from intelmqctl :)

Here some additional explanation I think may be useful:

re: number of events & botnet size: this is exactly my case, with more than 100 bots in use (and growing).

re: size of input queues: in my case, it helps us with e.g. detecting stuck output bots (we have a few different) and occasionally bottlenecks in the botnet. Of course, it alerts when the given maximum is reached.

re: collectors producing data: this is also a little different in my case, I have e.g. multiple collectors talking to only one parser.

re: logging level per bot: this is easily possible, just set the 'logging_level' in bot's config.

It looks like the operational monitoring can also vary between deployments, depending on local needs. There is no one-for-all approach ;)

Best regards

// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 11/27/23 15:42, Mika Silander wrote:

...

Hi Kamil, Thanks for elaborating on your monitoring setup and needs. Here's a description of ours:

• number of events: our event logger expert, we can duplicate it on the paths that need to be monitored
  • our bot net is small so it is not an issue, but for a bigger one the number of extra bots due to this could be big
• number of errors: from the logs for those bots that do report them to their log files
  • the weakness for us here is one can't rely on uniform/consistent reporting of errors from one bot to another (depends on implementation of each, right?)
  • and, we do check all log files for exceptions, but an exception is usually an indication there could be a severe implementation problem in the bot itself,
  • and, we monitor presence of event dump files
• number of bot runs / run frequency can be pulled from log files assuming the log level is sufficiently talkative
• we don't monitor the size of the input queues since the number does not indicate a problem nor tell if the processing of events progresses. One should check if individual events "move forward" in the bot chain to answer that. At most I'd monitor the number does not grow past some allowed maximum and the same information can probably be pulled from redis (or the message queue used) using their own tools or some wrapper scripts(?)
• whether bots are running: we use "intelmqctl status" and just analyse every bot's status line. Is this too an innocent approach?
• whether collectors are producing new data: we can detect this indirectly by analysing the logs of the parsers, i.e. do events get forwarded sufficiently often?
  • this requires of course we have parsers that write something useful to the logs when they receive a message from a collector and forward events.

All in all, I think we aggregate almost the same monitoring data although we have to ensure sufficiently detailed information is written to the logs ... so we also monitor the log level remains as INFO or chattier :-). It would be handy to be able to set the log level per bot to reduce the logging where applicable ... have to check if this can be done. Br, Mika ----- Original Message ----- From: "Kamil Mankowski via IntelMQ-dev" intelmq-dev@lists.cert.at To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 27 November, 2023 15:48:39 Subject: Re: [IntelMQ-dev] STATS IN INTELMQ? I collect the number of events processed, overall and separately for each destination path, the number of errors, bot runs, and the size of the input queue. The last one is not visible in the logs, the number of processed with some (larger) delay, and the separation for target paths only in debug mode (as far as I can remember; although this is rarely useful). I also check whether the bots are running, whether the collectors are producing new data, and the error rate, as well as the execution of periodic jobs (db updates etc). It's not perfect, but it works pretty well :) Best regards // Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien On 11/27/23 14:31, Mika Silander wrote:

...

Hi Kamil,

Thanks for the clarification. I've also noticed there are some hurdles and problems when relying entirely on logs for monitoring. Just out of curiosity, what are the operational monitoring needs you would tackle with statistics collection in intelmq (and can't be covered using normal logs)? I think we have managed so far with using the logs as the only source of information for monitoring, and it would be interesting to compare and see if we've possibly forgotten some monitoring target/issue.

Br, Mika

----- Original Message ----- From: "Kamil Mankowski via IntelMQ-dev" intelmq-dev@lists.cert.at To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 27 November, 2023 14:59:34 Subject: Re: [IntelMQ-dev] STATS IN INTELMQ?

Hi Mika,

Thanks for the comment. I totally agree that the statistics needed are definitely different for different teams.

...

I would vote for removing the initial statistics functionality from

intelmq and keep statistics collection completely separate.

I can't agree with that, because there are two different types of statistics - as you said, the monitoring and the business statistics. I think I used the word "statistics", but what I meant with my second point is purely operational monitoring, and I'd like to keep the built-in functionality operation-oriented. Monitoring through logs is also possible, although it has some drawbacks.

However, there have been some ideas to introduce hook capabilities. If we want to change the monitoring (e.g. integrate with Open Telemetry) and/or allow more statistics directly from IntelMQ, I'd go in the direction of optional support via hooks, without extending the base source code.

Best regards

// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 11/27/23 13:41, Mika Silander wrote:

...

Hi Kamil & all,

From my experience it looks like collecting statistics varies wildly based on each team's particular needs. In our case, the built-in statistics functionality does not provide all the info we need so we wrote a simple expert bot to collect statistics on the events processed (IPs, constituency, classification.type, abuse C address etc etc). It outputs the statistics to rsyslog. In principle, we can put another instance of the bot to a latter stage in the bot chain to send the same information about events that have passed all sanity checks done by expert bots in between. This way the same bot answers the questions a) how many events have we processed, b) how many of these events have been successfully passed on to our clients (=constituency). Finally, the ticketing system gives us statistics about how many of those events have been properly resolved, but that's out of scope in terms of generating statistics inside intelmq.

Don't take me wrong but in this case I would vote for removing the initial statistics functionality from intelmq and keep statistics collection completely separate. This would keep the intelmq code base also (marginally) smaller.

I also remember a recommendation on this forum that intelmq instances should be monitored with the help of the log files created. I see monitoring and statistics collection as two orthogonal functionalities and this is the other reason why I would like to keep statistics collection independent, i.e. neither as part of the core modules of intelmq, nor as an indirect means of monitoring.

Well, this is just my view on the issue (the famous 5 cents), and, I am unaware of other use cases and requirements that could tilt the balance towards the current way of implementing statistics.

Br, Mika

----- Original Message ----- From: "Kamil Mankowski via IntelMQ-dev" intelmq-dev@lists.cert.at Cc: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 27 November, 2023 13:23:56 Subject: Re: [IntelMQ-dev] STATS IN INTELMQ?

Hi all,

I think there is currently no ongoing work for any improved statistics directly in the IntelMQ, but if you have development capacity to extend the current state, I think it could be useful.

However, I can say what is currently available, and how I use/plan to use it:

1. for final events, we use - as mentioned by Aaron - a database. We are

going to use Timescale DB, as described in: https://docs.intelmq.org/develop/admin/database/postgresql/#using-eventdb-wi... Currently we have some set of scripts generating stats, and we plan to move fully to TimescaleDB+Grafana.

2. for monitoring the ongoing work, there are basic stats exposed in the

database 3 in Redis (see changelog: https://docs.intelmq.org/develop/changelog/?h=statistics#configurations). I think this feature isn't well documented. It's not perfect, but I use it to keep an eye on the botnet & failures, using a custom scripts to integrate it with CheckMK monitoring and alert on troubles.

Best regards

// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 11/27/23 11:51, L. Aaron Kaplan wrote:

...

Hi,

Most users of intelmq use an "eventsDB" as an output . From there, it is usually quite doable to do stats on top of events.

I did an initial version for CERT.at back then which is still here: https://github.com/certtools/stats-portal You can build on top of this if it suits you.

Hope it helps, Aaron.

...

On 27.11.2023, at 11:37, Homma, L.J. (Luitzen) via IntelMQ-dev intelmq-dev@lists.cert.at wrote:

Dear IntelMQ Developers & Users, We are curious if there are any plans on the roadmap to incorporate statistical features into IntelMQ. About 1.5 years ago, we participated in an online session where it was mentioned that there were some early plans to integrate stats into IntelMQ. As far as we could find, there have not been any steps in this direction. Are we correct? Currently, we are working in our experimental environment to develop stats based on a Prometheus bot, using Prometheus as a time-series database, and utilizing Grafana for dashboarding and visualization. Are there more members of the community working on this? Our goal is to gain better insights into the input, filtering, and output of our IntelMQ pipeline(s). We hope to hear from others about their thoughts on this. Met vriendelijke groet, Luitzen Homma Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is gezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten.

This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages. _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

---

IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

---

IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/ _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

---

IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/ _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

---

IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/

Good idea.

But please note that timescaleDB is - optional (plain postgresql with the proper indices usually does the job just as well) - actually timescaledb is just a few extensions on top of plain postgresql.

So, I don't really see a lock-in here.

But a notice would be great. Maybe as part of any PR which includes more timescaledb support...

My 2 cents of experience with working with timescaledb.

PS: back then at CERT.at we used timescaledb initially for https://github.com/certtools/stats-portal already and it is just great for its purpose: to do very fast SELECT ... GROUP BY .. WHERE timestamp in INTERVAL(...) queries. That's where it shines (and coincidentally that's often what you need for data-cubes/statistics on large DBs). But you can achieve similar functionality also with plain postgresql or other databases. I see that as a functionality which may be replaced by other similar systems - if needed (and that depends only on the size of your eventDB).

On 28.11.2023, at 10:15, Bernhard Reiter bernhard@intevation.de wrote:

Signed PGP part Hello,

Am Montag 27 November 2023 12:23:56 schrieb Kamil Mankowski via IntelMQ-dev:

...

However, I can say what is currently available, and how I use/plan to use it:

1. for final events, we use - as mentioned by Aaron - a database. We are

going to use Timescale DB,

note that the more recent features of timescale db are non-free (aka proprietary) software.

https://docs.timescale.com/about/latest/timescaledb-editions/ "Many of the most recent features of TimescaleDB are only available in TimescaleDB Community Edition."

"You cannot sell TimescaleDB Community Edition as a service"

"You can modify the TimescaleDB Community Edition source code and run it for production use."

Maybe adding a proper warning against the lock-in in the document would match IntelMQ's idea to stay open.

Regards Bernhard

-- https://intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter

On 28.11.2023, at 10:42, Homma, L.J. (Luitzen) l.j.homma@minezk.nl wrote:

Thank you for all replies :) Thank you, Aron, for pointing to the stats portal. I agree with your note. Based on all replies, it's interesting to hear how others are trying to solve the stats/monitoring questions. It's also a good thing to address that there is a difference between stats and monitoring. However, I think there is also quite some overlap because you probably want to set up monitoring/alerting based on your stats. Shiny management reports are indeed a different thing :)

Actually, that's a really good point I had not thought about. If your stats on incidents which go through intelmq go through the roof at some point (per classification.identifier or classificiat.type, etc.) then you might want to alert. Good point!

Monitoring and dashboarding itself probably should not be part of the IntelMQ core.

Totally agree. If we keep the focus of intelmq on the hard part of data collection, filtering, enriching and ingestion into some DB, some ELK or Splunk or whatever, then we already solve a hard problem. If we diversify too much, we risk doing everything but nothing really well.

However, I think it would be very useful to share monitoring/alerting options in the documentation or share code (bots) to feed your stats/reporting/dashboarding solution. Everyone is free to make its own configuration setup that fits its needs but could use a default bot to feed his reporting engine.

True, like HOWTOs or examples?

Our goal for stats in more detail:

• To get technical insights into what is coming into the pipeline. So, event counts per bot, but also more granulated numbers for each report/feed.
• Get insights on how many events are deduped.
• We have a lot of filter bots in our environment. We want stats on the event filter ins/outs.
• We have multiple output bots where we also want stats.
• In the end, you want a kind of CRC paper trail check to ensure your line is in control and not "losing" any data on the way in the journey through the data pipeline(s) :)

@Aron, we will use your repo for inspiration and will probably develop our stats needs further in 2024. If it's relevant to the community, we'll see if we can share it.

That would be lovely. Note that mine is already a bit outdated, but - yes, as you said, it might give you inspirations.

Best, Aaron.

Regards,

Luitzen Homma

-----Oorspronkelijk bericht----- Van: IntelMQ-dev intelmq-dev-bounces@lists.cert.at Namens L. Aaron Kaplan Verzonden: dinsdag 28 november 2023 10:31 Aan: Bernhard Reiter bernhard@intevation.de CC: intelmq-dev@lists.cert.at Onderwerp: Re: [IntelMQ-dev] timescale db is non-free (was: Stats in Intelmq?)

Good idea.

But please note that timescaleDB is

• optional (plain postgresql with the proper indices usually does the job just as well)
• actually timescaledb is just a few extensions on top of plain postgresql.

So, I don't really see a lock-in here.

But a notice would be great. Maybe as part of any PR which includes more timescaledb support...

My 2 cents of experience with working with timescaledb.

PS: back then at CERT.at we used timescaledb initially for https://github.com/certtools/stats-portal already and it is just great for its purpose: to do very fast SELECT ... GROUP BY .. WHERE timestamp in INTERVAL(...) queries. That's where it shines (and coincidentally that's often what you need for data-cubes/statistics on large DBs). But you can achieve similar functionality also with plain postgresql or other databases. I see that as a functionality which may be replaced by other similar systems - if needed (and that depends only on the size of your eventDB).

...

On 28.11.2023, at 10:15, Bernhard Reiter bernhard@intevation.de wrote:

Signed PGP part Hello,

Am Montag 27 November 2023 12:23:56 schrieb Kamil Mankowski via IntelMQ-dev:

...

However, I can say what is currently available, and how I use/plan to use it:

1. for final events, we use - as mentioned by Aaron - a database. We

are going to use Timescale DB,

note that the more recent features of timescale db are non-free (aka proprietary) software.

https://docs.timescale.com/about/latest/timescaledb-editions/ "Many of the most recent features of TimescaleDB are only available in TimescaleDB Community Edition."

"You cannot sell TimescaleDB Community Edition as a service"

"You can modify the TimescaleDB Community Edition source code and run it for production use."

Maybe adding a proper warning against the lock-in in the document would match IntelMQ's idea to stay open.

Regards Bernhard

-- https://intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter

Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is gezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten.

This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages.