IntelMQ-dev

intelmq-dev@lists.cert.at

241 discussions

STATS IN INTELMQ?
by Homma, L.J. (Luitzen) 28 Nov '23

28 Nov '23

Dear IntelMQ Developers & Users, We are curious if there are any plans on the roadmap to incorporate statistical features into IntelMQ. About 1.5 years ago, we participated in an online session where it was mentioned that there were some early plans to integrate stats into IntelMQ. As far as we could find, there have not been any steps in this direction. Are we correct? Currently, we are working in our experimental environment to develop stats based on a Prometheus bot, using Prometheus as a time-series database, and utilizing Grafana for dashboarding and visualization. Are there more members of the community working on this? Our goal is to gain better insights into the input, filtering, and output of our IntelMQ pipeline(s). We hope to hear from others about their thoughts on this. Met vriendelijke groet, Luitzen Homma Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is gezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten. This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages.

6 18

Introducing proposals to IntelMQ Data Format (IEP008 and 9)
by Kamil Mankowski 02 Nov '23

02 Nov '23

Dear IntelMQ Developers & Users, I'd like to propose two changes to the IntelMQ data format to make it more suitable for current use cases. 1. IEP008 - add constituency identifier (this has already been shared, but only on the developer list): https://github.com/certtools/ieps/pull/9 2. IEP009 - add product and vulnerability information: https://github.com/certtools/ieps/pull/10 I'd like to hear your feedback on these suggestions. Feel free to comment here or on the Github pull requests. My plan is to implement them, unless rejected, by December. -- Best regards, Kamil Mankowski CERT.at GmbH www.cert.at

1 0

Expected behaviour of bot reloading after log rotation
by Mika Silander 24 Oct '23

24 Oct '23

Hi all, Apologies for a lengthy message, but I had a set of monitoring scripts that worked like a charm on a physical machine but when moved to a virtual machine with a newer Ubuntu, I see odd failures in the form of monitoring script processes left hanging on void file handles. I suspect the problem is somehow related to log rotation in connection to bot reloading but I can't figure out how. A few questions at the end. Our setup for reference: Ubuntu 22.04 LTS python 3.10.12 intelmq 3.2.2a1 intelmq-api 3.1.0-rc1 intelmq-manager 3.2.0 We have a simple bot chain: Mail Attachment Collector -> Shadowserver Parser -> N expert bots -> 1 output bot. /etc/logrotate.d/intelmq contains: /var/log/intelmq/*.log { su intelmq intelmq daily maxsize 10M rotate 60 notifempty compress delaycompress create 644 intelmq intelmq sharedscripts postrotate sudo -u intelmq /usr/bin/intelmqctl --quiet reload endscript } The cron daemon runs run-parts for targets under /etc/cron.daily/ at 06:25, including /etc/cron.daily/logrotate. Thus, "intelmqctl --quiet reload" gets also executed after all the intelmq log files have been rotated. What I see is that no bot reloads soon after run-parts has finished. The first bot to perform a reload is the Mail Attachment Collector immediately after midnight. The others remain as is until the Mail Attachment Collector reads the day's first report and forwards it to the Shadowserver Parser bot. The parser bot then performs a reload and forwards the events to the first expert in the chain. Once the first event from the parser trickles through the entire bot chain, every bot in turn first performs a reload and then processes the event. In practise, the logrotate triggered bot reload can thus be delayed almost a full day - our first mail reports tend to arrive during the very early morning hours. My question is, is the above reload behaviour as expected? I found it strange that the Mail Attachment Collector reloads at midnight even though it has processed new mail reports after the "intelmqctl reload" call done by logrotate. I would have expected the Mail Attachment Collector to reload itself when the first report arrives for processing after 06:25, i.e. the first report after the call to intelqmctl reload. And, what controls the timing of the Mail Attachment Collector's reload? All its reloads I see in older log files indicate that the reload is consistently run after midnight irrespective of incoming reports and the timing of the intelmqctl reload command. Thanks again and best regards from a puzzled Mika

3 6

IEP008 - Constituency field in IntelMQ Data Format
by Kamil Mankowski 09 Oct '23

09 Oct '23

Dear IntelMQ Community, I'd like to introduce an IEP with small extension to the current IntelMQ Data Format. As supporting multiple constituency groups is becoming more popular, we find it useful to include this information directly in the data format. IEP: https://github.com/certtools/ieps/pull/9/files This should not introduce any troubles to single-constituency setups, as well as do not define, how to mark the constituencies. The idea is just to open a possibility for that. I'd appreciate a feedback from the Community! -- Best regards, Kamil Mankowski CERT.at GmbH www.cert.at

1 0

intelmq-manager problem at login
by Mika Silander 19 Sep '23

19 Sep '23

Hi again, Hints would be welcome: I've got intelmq-manager installed, the front page opens ok but when I try to login, the developer view of the browser reports the following error after I specify the account and the password: DevTools failed to load source map: Could not load content for https://FQDN_HERE/intelmq_manager/plugins/bootstrap/bootstrap.min.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE Checking what's in /usr/share/intelmq_manager/html/plugins/bootstrap/ reveals there is no file there called bootstrap.min.css.map. The files that do exist in that directory are: bootstrap.css bootstrap.js bootstrap.min.css bootstrap.min.js My setup running on a Ubuntu 22.04 LTS / Python 3.10.12 is now: ii intelmq 3.2.2-1 all Solution for IT security teams for collecting ii intelmq-api 3.1.0~rc2-1 all FastAPI-based API for the intelmq project ii intelmq-manager 3.2.0-1 all Graphical interface to manage configurations As a side note, the intelmq-api package was upgraded from intelmq-api 3.1.0-rc1 in case this could be the culprit. Br and have a nice weekend, Mika

3 14

Trouble building deb package for intelmq-3.2.2
by Mika Silander 04 Sep '23

04 Sep '23

Hi all, Before digging any deeper I'll ask the list for advice. It's possible someone's bumped into the same error. I've upgraded from Ubuntu 20 LTS with intelmq-api 3.1.0~rc-1 intelmq-manager 3.2.0-1 intelmq 3.1.0-1 python3.8 to Ubuntu 22 LTS. Now, when trying to build intelmq-3.2.2 + a few of our own bots on this upgraded host with dpkg-buildpackage -us -us -b the build fails with: test-bot: Bot stopped. test-bot: Processed 1 messages since last logging. test-bot: Bot stopped. test-bot: Processed 1 messages since last logging. test-bot: Bot stopped. E: pybuild pybuild:369: test: plugin distutils failed with: exit code=1: python3.10 setup.py test dh_auto_test: error: pybuild --test -i python{version} -p 3.10 returned exit code 13 make: *** [debian/rules:24: build] Error 255 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2 The host has python3-distutils 3.10.8-1~22.04 installed. Any hints on what lies behind this error? Br & thanks, Mika

3 3

IntelMQ 3.2.1 release with two important bug fixes
by Sebix 28 Aug '23

28 Aug '23

Dear IntelMQ community, Together with CERT.at we just released IntelMQ version 3.2.1 containing two important bug fixes. 1. A bug in the bot's core library: > Fixes an issue which prevented bots from stopping gracefully after reloading. > As logrotate reloads all bots regularly, this bug affects most IntelMQ installations. 2. A bug in the Reverse DNS Expert bot: > Until IntelMQ version 3.2.0, the bot incorrectly cached and re-used results for /24 networks instead of single IP addresses. > If the bot retrieved the PTR for `192.0.43.7`, it was cached for `192.0.43.0/24` and used for all IP addresses in this range, for example for `192.0.43.8`. > IntelMQ version 3.2.1 fixes this issue. > The bugfix will correctly increase the cache sizes and decrease the performance, as less (incorrect) data is re-used. Upgrading: https://intelmq.readthedocs.io/en/develop/user/upgrade.html Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html The PyPI package and git tag/GitHub release are published. The new deb-/rpm-packages will hit the repositories in the next few minutes and the updated Docker image follows soon. best regards Sebastian -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://sebix.at/ ZVR 1510673578

1 0

Shift of focus, new fields in IDF needed for scanning results?
by Otmar Lendl 21 Aug '23

21 Aug '23

Folks, I've been looking at some of the data that we recently processed with our IntelMQ setup (leading to https://cert.at/de/aktuelles/2023/8/verwundbare-webserver-status-in-osterre…), and I found that we need a few changes to IntelMQ to better process these kind of feeds. What happened? Initially, our focus with IntelMQ was on botnet drones which were detected via sinkholing. That's why we have the source.*, destination.*, and malware.* parameters in the data model. A C2 communication was the main inspiration. These days, a majority of the data feeds we process are generated by scanning the Internet. That can be Shodan, that can be Shadowserver, that can be from our own local scans. This is fine and good, it gives us valuable telemetry on what's going on in our constituency. But the fields needed to store the data in a consistent way are missing in the current iteration of the IDF. I don't want to pack that into extra.* or any non-standard field, I think we should come to an agreement what those elements are and how they should be handled. Here are some of the data fields I'm missing: What did the scanning find? * vendor * product * software version Are there any problems with it? * Vulnerability identifier (CVE or other, potentially multi-valued, which is another can of worms) * Severity information (e.g. associated CVSS score) * CWE IDs (https://cwe.mitre.org/index.html) Generic tagging * I see e.g. "iot", "ics" or similar ones popping up. As a minimum I think we need "vendor", "product" and "vulnerability". What do you all think? otmar ----------- References: https://datapedia.shodan.io/ There is e.g.: os string Operating system platform string product string Name of the software that powers the service. vendor string https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-http-r… info on the vendor and CVEs is stored in the tag field. -- // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

4 4

IntelMQ 3.2 announcement
by L. Aaron Kaplan 20 Jul '23

20 Jul '23

IntelMQ 3.2 release announcement ================================== [TLP:WHITE, please pass this on, if appropriate] Dear everyone, We are excited to announce that today we can release 3.2.0 of the open source IntelMQ (https://github.com/certtools/intelmq) incident handling automation framework. Special thanks go to Kamil (CERT.at), Sebix, Intevation, Filip Pokorny (Gethvi) and Jason from Shadowserver (you all know who you are). Also I would like to thank the 93 contributors [1] who have been helping in many ways over the last year(s) with IntelMQ. Core members of IntelMQ are committed to continuing to develop, support and enhance IntelMQ in the long run. Especially since IntelMQ is a quite well adopted extract transform load (ETL) tool for fetching incident report feeds, processing them and sending them to the right place. It is widely used in Europe and in the whole world, in CERTs, SOCs and MSSPs. And best of it, it is open source and freely available. Again, we can not thank all the contributors enough. So, what's in release 3.2? =========================== Highlights: * The main change is that IntelMQ's code can now be used as a python module / library. That means: you can easily embed the hard work that IntelMQ parser writers did into your application. It's as simple as instantiating an IntelMQ bot in python and giving it data. Thanks go to Sebix. See also https://github.com/certtools/ieps/tree/main/007 * the IntelMQ API moved from hug to fastapi [4]. The API interface stayed the same. Thanks a lot to Kamil. * We have an updated intelmq developer version docker image. Apart from that, a few other highlights: * Shadowserver parser improvements. * New shadowserver data feeds are supported * Changes time_format parameter to use new TimeFormat class * new CIF3 output bot * intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar) As always, you can find all the nitty gritty details in the CHANGELOG [2] file and in the NEWS file [3] What's next? ============== * BSI funded a complete re-write of webinput-csv (https://github.com/Intevation/intelmq-webinput-csv), watch for an announcement soon. * Improved and re-done documentation via mkdocs (Thanks to Filip) * The upcoming next version 3.3 will have an update for the dynamic config loading of the shadowserver parser [4]. This will always give you the most current shadowserver feeds and parsers without having to wait for an official intelmq release. If you depend a lot on shadowserver feeds, it's ok to wait for the next release, which will be following very shortly. (And a big hooray to shadowserver for their help, we know that you need this feature quickly) We plan to release 3.3 in approx 1 week. Should I update? ================== Yes! **Note well**: we still have some issues with proper Debian packages for fastapi (a dependency of the new IntelMQ API server). This means re-packaging fastapi with the help of Debian developers or using the old hug API for now (no difference from a user perspective though). The deb-package for the new fastapi-based intelmq-api is currently only available on Debian 11. We are also sorting out an issue with the package build of intelmq on Debian 12: https://github.com/certtools/intelmq/issues/2384#issuecomment-1637799252 and issues on Ubuntu related to fastapi-Versions. Packages of the previous compatible hug-based intelmq-api remain the package repositories of affected distributions. Once these issues are sorted out, the new packages will become available for installation and upgrade in the respective repositories. For installations via pip or development setups, nothing needs to be kept in mind. How to update? =============== * Upgrading: https://intelmq.readthedocs.io/en/3.2.0/user/upgrade.html * Fresh installations: https://intelmq.readthedocs.io/en/3.2.0/user/installation.html (Note that the docker installation might lag behind a bit) Need help? =========== You can contact us on the mailing lists: * intelmq-dev for developers [5] * intelmq-users for users [6] See also: https://intelmq.readthedocs.io/en/3.2.0/user/support.html Thank you and thanks to all the contributors! Aaron Kaplan, Sebix, Kami, Filip (for the whole IntelMQ team) [1] https://github.com/certtools/intelmq/graphs/contributors [2] https://github.com/certtools/intelmq/blob/3.2.0/CHANGELOG.md [3] https://github.com/certtools/intelmq/blob/3.2.0/NEWS.md [4] https://github.com/certtools/intelmq/pull/2372 [5] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev [6] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users

1 0

Odd intelmqdump testing errors
by Mika Silander 26 May '23

26 May '23

Hi all, Running the entire unit test suite of intelmq I come across a few test failures which make me ponder whether my development environment is out of sync or outdated. Tests on intelmqdump fail in the following manner: -- FAIL: test_list_and_select_dump_from_custom_location (intelmq.tests.bin.test_intelmqdump.TestIntelMQDump) [0] ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/my_development/intelmq/intelmq/tests/bin/test_intelmqdump.py", line 165, in test_list_and_select_dump_from_custom_location self.assertIn("0: test-1 empty file", output[1]) AssertionError: '0: test-1 empty file' not found in ' 0: test-1 empty file' -- So, the assertion string looks "almost" what is expected. Can this be an old(er) python library issue or something? Currently I develop on Ubuntu 20.04.6 LTS, using python 3.8.10 and the intelmqdump sources tested are from the intelmq-github/develop branch, commit tag 0780ee5c1281da8a6defcdd8f8c4722ccdbee4e8. Br, Mika

3 4

← Newer
1
2
3
4
5
6
7
...
25
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev