Hi all,
I think there is currently no ongoing work for any improved statistics directly in the IntelMQ, but if you have development capacity to extend the current state, I think it could be useful.
However, I can say what is currently available, and how I use/plan to use it:
1) for final events, we use - as mentioned by Aaron - a database. We are going to use Timescale DB, as described in: https://docs.intelmq.org/develop/admin/database/postgresql/#using-eventdb-wi... Currently we have some set of scripts generating stats, and we plan to move fully to TimescaleDB+Grafana.
2) for monitoring the ongoing work, there are basic stats exposed in the database 3 in Redis (see changelog: https://docs.intelmq.org/develop/changelog/?h=statistics#configurations). I think this feature isn't well documented. It's not perfect, but I use it to keep an eye on the botnet & failures, using a custom scripts to integrate it with CheckMK monitoring and alert on troubles.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 11/27/23 11:51, L. Aaron Kaplan wrote:
Hi,
Most users of intelmq use an "eventsDB" as an output . From there, it is usually quite doable to do stats on top of events.
I did an initial version for CERT.at back then which is still here: https://github.com/certtools/stats-portal You can build on top of this if it suits you.
Hope it helps, Aaron.
On 27.11.2023, at 11:37, Homma, L.J. (Luitzen) via IntelMQ-dev intelmq-dev@lists.cert.at wrote:
Dear IntelMQ Developers & Users,
We are curious if there are any plans on the roadmap to incorporate statistical features into IntelMQ. About 1.5 years ago, we participated in an online session where it was mentioned that there were some early plans to integrate stats into IntelMQ. As far as we could find, there have not been any steps in this direction. Are we correct?
Currently, we are working in our experimental environment to develop stats based on a Prometheus bot, using Prometheus as a time-series database, and utilizing Grafana for dashboarding and visualization. Are there more members of the community working on this? Our goal is to gain better insights into the input, filtering, and output of our IntelMQ pipeline(s). We hope to hear from others about their thoughts on this.
Met vriendelijke groet,
Luitzen Homma Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is gezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten.
This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages. _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/