IntelMQ-dev

intelmq-dev@lists.cert.at

1 participants
244 discussions

Shift of focus, new fields in IDF needed for scanning results?
by Otmar Lendl 21 Aug '23

21 Aug '23

Folks, I've been looking at some of the data that we recently processed with our IntelMQ setup (leading to https://cert.at/de/aktuelles/2023/8/verwundbare-webserver-status-in-osterre…), and I found that we need a few changes to IntelMQ to better process these kind of feeds. What happened? Initially, our focus with IntelMQ was on botnet drones which were detected via sinkholing. That's why we have the source.*, destination.*, and malware.* parameters in the data model. A C2 communication was the main inspiration. These days, a majority of the data feeds we process are generated by scanning the Internet. That can be Shodan, that can be Shadowserver, that can be from our own local scans. This is fine and good, it gives us valuable telemetry on what's going on in our constituency. But the fields needed to store the data in a consistent way are missing in the current iteration of the IDF. I don't want to pack that into extra.* or any non-standard field, I think we should come to an agreement what those elements are and how they should be handled. Here are some of the data fields I'm missing: What did the scanning find? * vendor * product * software version Are there any problems with it? * Vulnerability identifier (CVE or other, potentially multi-valued, which is another can of worms) * Severity information (e.g. associated CVSS score) * CWE IDs (https://cwe.mitre.org/index.html) Generic tagging * I see e.g. "iot", "ics" or similar ones popping up. As a minimum I think we need "vendor", "product" and "vulnerability". What do you all think? otmar ----------- References: https://datapedia.shodan.io/ There is e.g.: os string Operating system platform string product string Name of the software that powers the service. vendor string https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-http-r… info on the vendor and CVEs is stored in the tag field. -- // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

4 4

IntelMQ 3.2 announcement
by L. Aaron Kaplan 20 Jul '23

20 Jul '23

IntelMQ 3.2 release announcement ================================== [TLP:WHITE, please pass this on, if appropriate] Dear everyone, We are excited to announce that today we can release 3.2.0 of the open source IntelMQ (https://github.com/certtools/intelmq) incident handling automation framework. Special thanks go to Kamil (CERT.at), Sebix, Intevation, Filip Pokorny (Gethvi) and Jason from Shadowserver (you all know who you are). Also I would like to thank the 93 contributors [1] who have been helping in many ways over the last year(s) with IntelMQ. Core members of IntelMQ are committed to continuing to develop, support and enhance IntelMQ in the long run. Especially since IntelMQ is a quite well adopted extract transform load (ETL) tool for fetching incident report feeds, processing them and sending them to the right place. It is widely used in Europe and in the whole world, in CERTs, SOCs and MSSPs. And best of it, it is open source and freely available. Again, we can not thank all the contributors enough. So, what's in release 3.2? =========================== Highlights: * The main change is that IntelMQ's code can now be used as a python module / library. That means: you can easily embed the hard work that IntelMQ parser writers did into your application. It's as simple as instantiating an IntelMQ bot in python and giving it data. Thanks go to Sebix. See also https://github.com/certtools/ieps/tree/main/007 * the IntelMQ API moved from hug to fastapi [4]. The API interface stayed the same. Thanks a lot to Kamil. * We have an updated intelmq developer version docker image. Apart from that, a few other highlights: * Shadowserver parser improvements. * New shadowserver data feeds are supported * Changes time_format parameter to use new TimeFormat class * new CIF3 output bot * intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar) As always, you can find all the nitty gritty details in the CHANGELOG [2] file and in the NEWS file [3] What's next? ============== * BSI funded a complete re-write of webinput-csv (https://github.com/Intevation/intelmq-webinput-csv), watch for an announcement soon. * Improved and re-done documentation via mkdocs (Thanks to Filip) * The upcoming next version 3.3 will have an update for the dynamic config loading of the shadowserver parser [4]. This will always give you the most current shadowserver feeds and parsers without having to wait for an official intelmq release. If you depend a lot on shadowserver feeds, it's ok to wait for the next release, which will be following very shortly. (And a big hooray to shadowserver for their help, we know that you need this feature quickly) We plan to release 3.3 in approx 1 week. Should I update? ================== Yes! **Note well**: we still have some issues with proper Debian packages for fastapi (a dependency of the new IntelMQ API server). This means re-packaging fastapi with the help of Debian developers or using the old hug API for now (no difference from a user perspective though). The deb-package for the new fastapi-based intelmq-api is currently only available on Debian 11. We are also sorting out an issue with the package build of intelmq on Debian 12: https://github.com/certtools/intelmq/issues/2384#issuecomment-1637799252 and issues on Ubuntu related to fastapi-Versions. Packages of the previous compatible hug-based intelmq-api remain the package repositories of affected distributions. Once these issues are sorted out, the new packages will become available for installation and upgrade in the respective repositories. For installations via pip or development setups, nothing needs to be kept in mind. How to update? =============== * Upgrading: https://intelmq.readthedocs.io/en/3.2.0/user/upgrade.html * Fresh installations: https://intelmq.readthedocs.io/en/3.2.0/user/installation.html (Note that the docker installation might lag behind a bit) Need help? =========== You can contact us on the mailing lists: * intelmq-dev for developers [5] * intelmq-users for users [6] See also: https://intelmq.readthedocs.io/en/3.2.0/user/support.html Thank you and thanks to all the contributors! Aaron Kaplan, Sebix, Kami, Filip (for the whole IntelMQ team) [1] https://github.com/certtools/intelmq/graphs/contributors [2] https://github.com/certtools/intelmq/blob/3.2.0/CHANGELOG.md [3] https://github.com/certtools/intelmq/blob/3.2.0/NEWS.md [4] https://github.com/certtools/intelmq/pull/2372 [5] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev [6] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users

1 0

Odd intelmqdump testing errors
by Mika Silander 26 May '23

26 May '23

Hi all, Running the entire unit test suite of intelmq I come across a few test failures which make me ponder whether my development environment is out of sync or outdated. Tests on intelmqdump fail in the following manner: -- FAIL: test_list_and_select_dump_from_custom_location (intelmq.tests.bin.test_intelmqdump.TestIntelMQDump) [0] ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/my_development/intelmq/intelmq/tests/bin/test_intelmqdump.py", line 165, in test_list_and_select_dump_from_custom_location self.assertIn("0: test-1 empty file", output[1]) AssertionError: '0: test-1 empty file' not found in ' 0: test-1 empty file' -- So, the assertion string looks "almost" what is expected. Can this be an old(er) python library issue or something? Currently I develop on Ubuntu 20.04.6 LTS, using python 3.8.10 and the intelmqdump sources tested are from the intelmq-github/develop branch, commit tag 0780ee5c1281da8a6defcdd8f8c4722ccdbee4e8. Br, Mika

3 4

Speed dumping events in bots
by Mika Silander 17 May '23

17 May '23

Hi again, Afaik, if handling an event fails in a bot, the default behaviour of a bot is to sleep 15 seconds and then retry processing. If the retry fails, the bot dumps the event and picks the next event from the inqueue. We have a bot where it would be desirable to change this default behaviour so that the dump is done immediately on the first failure. Is there a way to configure a single bot to behave differently from other bots as described? Or will a change in configuration affect the entire bot net? If there's no easy way configuring, I guess technically I could implement exceptions to be thrown in those situations where quick dumping is desired and then call the _dump_message(?) method. However, I would prefer to modify as few bots of the official distribution as possible. Comments, pointers to docs, sources or the like are most welcome. Br, Mika

3 5

Structural change in one feed's events output by the Shadowserver Parser bot?
by Mika Silander 05 May '23

05 May '23

Hi all, Testing with intelmq 3.1.0 and specifically its Shadowserver Parser bot. I've become used to being able to read in an event output by this bot as plain JSON with the json libraries bundled in python3, but now I came across one event sample where this fails. The feed is Accessible Traversal Utilities for NAT. The troublemaker event contained the field extra.amplification, and it was output by the bot as a dict structure like 'extra.amplification': Decimal("5.2") whereas I expected the field value to be a string "5.2" or number 5.2. Please let me know what I've missed again. A json library update? Br and have a nice weekend, Mika

1 1

RFC on IEP007: Running IntelMQ as Python Library
by Sebix 04 May '23

04 May '23

Dear community, I invite you to discuss a new (IEP - IntelMQ Enhancement Proposal) IEP007: Running IntelMQ as Python Library Have you ever wondered if you can write a Python script, call a bot's process method, pass it some data and get back the enriched/modified data? (pseudo code) bot_instance = Bot(parameters) bot_instance.process_message(input message) -> output messages Strictly speaking, it *is* actually possible with the current version, but it requires some bizarre hacks like re-defining Bot's methods and overwriting internal values. Staying on the wishlist for quite a while, we intend to implement the feature now. I started the IEP007 draft and need some input from you to maximize the benefit for all the IntelMQ community (developers): https://github.com/certtools/ieps/pull/7/files Or in a readable display: https://github.com/certtools/ieps/tree/iep-007/007 What features and interfacing options would you expect when starting the bot as a library? Do you think the `Bot.process` method should be rewritten entirely now, removing the calls receive_message/send_message and converting the method into a generator (an API-breaking change)? And if yes, should this be done in one step, or separated from this bot-as-library feature, reducing the complexity of development steps? Looking forward for your ideas Sebastian -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://sebix.at/ ZVR 1510673578

4 8

IntelMQ 3.1 release
by L. Aaron Kaplan 10 Feb '23

10 Feb '23

[TLP:WHITE, please pass this on, if appropriate] Dear everyone, it took a bit of time but I am excited to announce that today we finally pushed out release 3.1 of the open source IntelMQ (https://github.com/certtools/intelmq) incident handling automation framework. Special thanks go to Kamil, Sebix, Filip Pokorny (Gethvi) and Jason from Shadowserver (you all know who you are). Also I would like to thank the 91 (!) contributors [1] who have been helping in many ways over the last year(s) with IntelMQ. This truly is became a community effort and I am very happy that we can push this project further and maintain IntelMQ in the long run as a community project. IntelMQ is by now quite well respected as a "glue" / extract transform load (ETL) tool for fetching incident report feeds, processing them and sending them to the right place. It is widely used in Europe, in CERTs, SOCs and MSSPs. And best of it, it is open source and freely available. Again, we can not thank all the contributors enough. So, what's in release 3.1? =========================== (TL;DR: - find all things here: https://github.com/certtools/intelmq/blob/develop/NEWS.md) Highlights: * Excellent support for new Shadowserver feeds (thanks to Jason and Shadowserver). This is for sure the biggest part here. Shadowserver explicitly mentions IntelMQ for processing their feeds and will maintain the IntelMQ parsers for their feeds. * Big improvements for the dataplane.org parsers * Big shodan parser improvements Apart from that, a few other highlights: * More CI/CD tests * Tons of code clean-ups * New bots such as the url expert * Enhancements of existing bots: * SQL Output support MSSQL now, * GitHub collector, * Alienvault parser improvements, * Shodan parser improvements, * maxmind geoIP expert improvements, etc etc. * Tons of behind the scenes bugfixes! * Core changes (which you won't notice much) in the core part (pipeline class) etc etc.. We also dropped some support for old, outdated versions (python 3.6 or older Debian and old RedHat systems). This slims down the code base a bit again. As always, you can find all the nitty gritty details in the CHANGELOG [2] file and in the NEWS file [3] What's next? ============== * API: We want to move the IntelMQ-API [4] to fastapi (thanks to Kamil). Due to minor issues with packaging, this is not happening today, but this is planned for next week. So, stay tuned. IntelMQ-api and intelmq will be both numbered with the same version 3.1. * More user-feedback in the next few weeks. * Update the docker images for intelmq 3.1 * Upcoming task-specific tutorials on intelmq.org (for example how to send data to Splunk, how to automate all shadowserver feeds, etc.). Courtesy of David Ruefenacht. Should I update? ================== Yes! (but first on a DEV instance!) If you run a production instance, please update on a dev instance first and cross check. Because every IntelMQ setup is different. We try to cover a lot in our testing[5] , but we can't give a guarantee that everything is completely covered, especially if there are local changes. So, yes, please update, but tell us [6] in case something breaks in your DEV environment. (But yes! you really should update - since you might really want to have all those Shadowserver and dataplane.org updates) How to update? =============== * Upgrading: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html * Fresh installations: https://intelmq.readthedocs.io/en/maintenance/user/installation.html# (Note that the docker installation might lag behind a bit) Thank you and thanks to all the contributors! Aaron Kaplan. (for the whole IntelMQ team) [1] https://github.com/certtools/intelmq/graphs/contributors [2] https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md [3] https://github.com/certtools/intelmq/blob/develop/NEWS.md [4] https://github.com/certtools/intelmq-api [5] https://intelmq.readthedocs.io/en/maintenance/dev/guide.html#testing [6] https://github.com/certtools/intelmq/issues

1 0

Unexpected log entry ordering in bots' logs
by Mika Silander 31 Oct '22

31 Oct '22

Hi all, I've been implementing a set of monitoring scripts for our intelmq instance. The scripts analyse the log files the bots output. One basic assumption of these scripts is that log entries are ordered according to time: the log entry on line N in the log file is older than the one on line N+1 and so forth. One of our scripts alerts if this assumption does not hold. This morning it reported the following unexpected ordering in the log of mail attachment bot (logrotated once, the file was /var/log/intelmq/mail-attachment-collector.log.1) : ... 2022-10-30 03:58:49,387 - mail-attachment-collector - INFO - Idling for 60.0s (1m) now. 2022-10-30 03:59:49,537 - mail-attachment-collector - INFO - Idling for 60.0s (1m) now. 2022-10-30 03:00:49,681 - mail-attachment-collector - INFO - Idling for 60.0s (1m) now. 2022-10-30 03:01:49,829 - mail-attachment-collector - INFO - Idling for 60.0s (1m) now. ... Any ideas as to why this happens? I already suspected logrotate and the bot net reload functionality, but logrotate is run for the bot net at 06:25 so this seems to be unrelated. I also guess this does not have to do with the mail bot's implementation but one could rather see this happen with other bots as well. A file handle flushing issue? Br, Mika

3 3

Feed handling bug in shadowserver parser bot?
by Mika Silander 05 Sep '22

05 Sep '22

Hi, We've been running intelmq in a production-like setup for some time now and occasionally we see the Shadowserver Parser bot behave in a way that looks odd. We push reports to it via the mail attachment collector bot. At times, the parser bot ends up treating all reports as being of the type Vulnerable ISAKMP. We don't know what triggers this behaviour but so far we've seen this when the parser parses Sandbox URL and Open TFTP reports, both filling the log (debug level) with notifications of "missing keys" or "optional keys not found" because the parser assumes the feed to be Vulnerable ISAKMP. And before this occurs, the parser has been running for a good while. We've picked up the the problematic Sandbox URL and Open TFTP reports above and made unit test cases of these - all tests are correctly parsed, the feed name is deduced correctly and the tests are successful, so the parser handles these correctly if these are the first reports parsed. Therefore, our assumption is that this problem occurs only when the parser has been running for a longer time and some state information about the feed type does not get cleared between parsing incoming reports. Anyone else experiencing similar problems? This is a tricky one to debug so I decided to ask on the list first. Br, Mika P.S: Our setup is intelmq 3.0.2 on an Ubuntu 20.04 LTS

3 7

Documentation on bot statistics collection
by Mika Silander 09 Aug '22

09 Aug '22

Hi, A few issues still remain to be solved before going to production and one of them is how to collect statistics on events. It looks to me that lib/bot.py contains code for collecting statistics and pushing cached data to a Redis instance, but I'd like to know if there's documentation describing the statistics collection in general and recommendations on its use? Any best practices or similar docs? I see there is the EventDB implementation but that in turn seems to be limited to statistics data being pushed out from intelmq via an SQL output bot. In our case we'd also need to keep a count on events being fed into intelmq. We'll probably also need some intermediate collection points within our bot net to keep track of the number of anomalous etc events. Therefore, leveraging somehow the constructs within lib/bot.py seems a better choice for us. As always, pointers are welcome and please correct me if I've understood something wrong above. Br, Mika

2 2

← Newer
1
2
3
4
5
6
7
8
...
25
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev