- IntelMQ-dev - CERT.at

RFC new compromised_iot report
by elsif 24 Sep '24

24 Sep '24

Hello, We have a new report that will begin tomorrow. https://www.shadowserver.org/what-we-do/network-reporting/compromised-iot-r… Please let me know if you have any recommended changes for the following mapping for the report. Regards, Jason "compromised_iot" : { "constant_fields" : { "classification.identifier" : "compromised-iot", "classification.taxonomy" : "intrusions", "classification.type" : "system-compromise" }, "feed_name" : "Compromised-IoT-Device", "file_name" : "compromised_iot", "optional_fields" : [ [ "malware.name", "family", "validate_to_none" ], [ "extra.", "severity", "validate_to_none" ], [ "protocol.transport", "protocol" ], [ "source.reverse_dns", "hostname" ], [ "extra.", "tag" ], [ "source.asn", "asn", "invalidate_zero" ], [ "source.geolocation.cc", "geo" ], [ "source.geolocation.region", "region" ], [ "source.geolocation.city", "city" ], [ "extra.source.naics", "naics", "invalidate_zero" ], [ "extra.", "hostname_source", "validate_to_none" ], [ "extra.source.sector", "sector", "validate_to_none" ], [ "extra.", "device_vendor", "validate_to_none" ], [ "extra.", "device_type", "validate_to_none" ], [ "extra.", "device_model", "validate_to_none" ], [ "extra.", "device_version", "validate_to_none" ], [ "extra.", "source_type", "validate_to_none" ], [ "event_description.text", "category", "validate_to_none" ], [ "status", "status" ], [ "extra.", "detail", "validate_to_none" ], [ "extra.", "public_source", "validate_to_none" ], [ "source.account", "account", "validate_to_none" ], [ "extra.", "server_host_key", "validate_to_none" ], [ "extra.", "malpubkey_sha256", "validate_to_none" ] ], "required_fields" : [ [ "time.source", "timestamp", "add_UTC_to_timestamp" ], [ "source.ip", "ip", "validate_ip" ], [ "source.port", "port", "convert_int" ] ], "url" : "https://www.shadowserver.org/what-we-do/network-reporting/compromised-iot-r…" },

2 1

IntelMQ 3.3.1 bugfix release
by Sebix 03 Sep '24

03 Sep '24

Dear IntelMQ community, users, developers, and Incident Response teams! We are excited to announce the release of IntelMQ version 3.3.1, which includes important bug fixes. The updated version is available on PyPI, in the git repository, and the deb/rpm repositories. Please see the list of all changes below. Documentation: https://doc.intelmq.org/ Source code: https://github.com/certtools/intelmq Thanks to all contributors to this release, in alphabetical order: * DigitalTrustCenter * Edvard Rejthar (CSIRT.CZ) * elsif2 (Shadowserver Foundation) * Kamil Mankowski (CERT.at) * Mikk Margus Möll (CERT.ee) * Sebastian Wagner (Institute for Common Good Technology, Intevation & BSI) The full list of changes: Core ==== - `intelmq.lib.utils.drop_privileges`: When IntelMQ is called as `root` and dropping the privileges to user `intelmq`, also set the non-primary groups associated with the `intelmq` user. Makes the behaviour of running intelmqctl as `root` closer to the behaviour of `sudo -u intelmq ...` (PR#2507 by Mikk Margus Möll). - `intelmq.lib.utils.unzip`: Ignore directories themselves when extracting data to prevent the extraction of empty data for a directory entries (PR#2512 by Kamil Mankowski). Bots ==== Collectors ---------- - `intelmq.bots.collectors.shadowserver.collector_reports_api.py`: - Added support for the types parameter to be either a string or a list (PR#2495 by elsif2). - Refactored to utilize the type field returned by the API to match the requested types instead of a sub-string match on the filename. - Fixed timezone issue for collecting reports (PR#2506 by elsif2). - Fixed behaviour if parameter `reports` value is empty string, behave the same way as not set, not like no report (PR#2523 by Sebastian Wagner). - `intelmq.bots.collectors.shodan.collector_stream` (PR#2492 by Mikk Margus Möll): - Add `alert` parameter to Shodan stream collector to allow fetching streams by configured alert ID - `intelmq.bots.collectors.mail._lib`: Remove deprecated parameter `attach_unzip` from default parameters (PR#2511 by Sebastian Wagner). Parsers ------- - `intelmq.bots.parsers.shadowserver._config`: - Fetch schema before first run (PR#2482 by elsif2, fixes #2480). - `intelmq.bots.parsers.dataplane.parser`: Use ` | ` as field delimiter, fix parsing of AS names including `|` (PR#2488 by DigitalTrustCenter). - all parsers: add `copy_collector_provided_fields` parameter allowing copying additional fields from the report, e.g. `extra.file_name`. (PR#2513 by Kamil Mankowski). Experts ------- - `intelmq.bots.experts.sieve.expert`: - For `:contains`, `=~` and `!~`, convert the value to string before matching avoiding an exception. If the value is a dict, convert the value to JSON (PR#2500 by Sebastian Wagner). - Add support for variables in Sieve scripts (PR#2514 by Mikk Margus Möll, fixes #2486). - `intelmq.bots.experts.filter.expert`: - Treat value `false` for parameter `filter_regex` as false (PR#2499 by Sebastian Wagner). Outputs ------- - `intelmq.bots.outputs.misp.output_feed`: Handle failures if saved current event wasn't saved or is incorrect (PR by Kamil Mankowski). - `intelmq.bots.outputs.smtp_batch.output`: Documentation on multiple recipients added (PR#2501 by Edvard Rejthar). Documentation ============= - Bots: Clarify some section of Mail collectors and the Generic CSV Parser (PR#2510 by Sebastian Wagner). -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578

1 0

Re: [IntelMQ-dev] Small unpatching issue during build of intelmq v3.3.0
by Mika Silander 04 Apr '24

04 Apr '24

-- Hi Sebastian, all, Thanks for your reply. I spent a good while digging through my Ubuntu's python3 packages and it seems some of the system python's packages were corrupt for God knows what reason (well, probably because of me). Fixing the mess was a pain but I can now confirm that, quilt unpatching succeeds as well as the "dpkg-buildpackage -us -uc -b -d" so the build runs as expected. Br, Mika ----- Original Message ----- From: "Sebix" <sebix(a)sebix.at> To: "mika silander" <mika.silander(a)csc.fi>, "intelmq-dev" <intelmq-dev(a)lists.cert.at> Sent: Thursday, 4 April, 2024 12:12:02 Subject: Re: [IntelMQ-dev] Small unpatching issue during build of intelmq v3.3.0 Hi Mika, On 4/4/24 10:33, Mika Silander via IntelMQ-dev wrote: > Gave a try on intelmq v3.3.0 (cloned from github and on the develop branch) today and > got this when running "dpkg-buildpackage -us -uc -b -d" on a Ubuntu 22.04 LTS: > > dpkg-buildpackage: warning: debian/changelog(l5): ignoring invalid week day 'Fr' > [...] > LINE: -- Aaron Kaplan <aaron(a)lo-res.org> Fr, 01 Mar 2024 14:19:00 +0100 Side note: This warning will be fixed with https://github.com/certtools/intelmq/pull/2490 > Patch fix-intelmq-paths.patch does not remove cleanly (refresh it or enforce with -f) > dh_quilt_unpatch: error: quilt --quiltrc /dev/null pop -a || test $? = 2 returned exit code 1 > make: *** [debian/rules:20: clean] Error 1 > dpkg-buildpackage: error: fakeroot debian/rules clean subprocess returned exit status 2 > > What buttons, levers, cranks and the like should be turned to get passed the patching > issue? I'd prefer not to have to enforce the thing as suggested. I don't have a solution, just a few observations and thoughts: The patch applies in our package builds on Ubuntu 22.04 (see https://build.opensuse.org/build/home:sebix:intelmq/xUbuntu_22.04/x86_64/in…) and in the GitHub CI tests using buildah (https://github.com/certtools/intelmq/actions/runs/8551898452/job/2343187443…) The latter uses dpkg-buildpackage -us -uc -b (without -d). But -d, --no-check-builddeps shouldn't make a difference for the patching. Neither the patch, nor the affected file were changed in the release (last changes of both were in 2021). best regards Sebastian -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578

1 0

Small unpatching issue during build of intelmq v3.3.0
by mika.silander＠csc.fi 04 Apr '24

04 Apr '24

-- Hi, Gave a try on intelmq v3.3.0 (cloned from github and on the develop branch) today and got this when running "dpkg-buildpackage -us -uc -b -d" on a Ubuntu 22.04 LTS: dpkg-buildpackage: warning: debian/changelog(l5): ignoring invalid week day 'Fr' LINE: -- Aaron Kaplan <aaron(a)lo-res.org> Fr, 01 Mar 2024 14:19:00 +0100 dpkg-buildpackage: info: source package intelmq dpkg-buildpackage: info: source version 3.3.0-1 dpkg-buildpackage: info: source distribution stable dpkg-buildpackage: info: source changed by Aaron Kaplan <aaron(a)lo-res.org> dpkg-buildpackage: info: host architecture amd64 dpkg-source --before-build . dpkg-source: warning: intelmq_v2/debian/changelog(l5): ignoring invalid week day 'Fr' LINE: -- Aaron Kaplan <aaron(a)lo-res.org> Fr, 01 Mar 2024 14:19:00 +0100 fakeroot debian/rules clean dh clean --with python3,sphinxdoc --without python2 --buildsystem=pybuild --with quilt --with systemd dh: warning: Compatibility levels before 10 are deprecated (level 9 in use) dh_auto_clean -O--buildsystem=pybuild dh_auto_clean: warning: Compatibility levels before 10 are deprecated (level 9 in use) I: pybuild base:239: python3.10 setup.py clean running clean removing '/home/mika/intelmqwork/intelmq_v2/.pybuild/cpython3_3.10_intelmq/build' (and everything under it) 'build/bdist.linux-x86_64' does not exist -- can't clean it 'build/scripts-3.10' does not exist -- can't clean it dh_quilt_unpatch -O--buildsystem=pybuild Patch fix-intelmq-paths.patch does not remove cleanly (refresh it or enforce with -f) dh_quilt_unpatch: error: quilt --quiltrc /dev/null pop -a || test $? = 2 returned exit code 1 make: *** [debian/rules:20: clean] Error 1 dpkg-buildpackage: error: fakeroot debian/rules clean subprocess returned exit status 2 What buttons, levers, cranks and the like should be turned to get passed the patching issue? I'd prefer not to have to enforce the thing as suggested. Br, Mika

2 1

test, please ignore
by L. Aaron Kaplan 03 Apr '24

03 Apr '24

test test

1 0

Proposed classification for new loop-dos report
by elsif 20 Mar '24

20 Mar '24

We are in the process of creating a new daily report based on the previous loop-dos special report: https://www.shadowserver.org/what-we-do/network-reporting/loop-dos-special-… I would like to propose the following constant_fields: classification.taxonomy = vulnerable classification.type = vulnerable-system protocol.application = application Where the application would be tftp or dns for example.

4 5

intelmq 3.3.0 release
by L. Aaron Kaplan 01 Mar '24

01 Mar '24

Dear intelmq users, developers and IR automation people, we are very happy to announce (finally announce!) IntelMQ 3.3.0 See: https://github.com/certtools/intelmq Important changes =================== The most relevant changes are: 1. Documentation ------------------ great new documentation (mkdocs) at https://docs.intelmq.org/ (Thanks a lot to Filip/ @gethvi! Amazing work) Documentation has been updated and restructured into User, Administrator and Developer Guide. The documentation is easier to navigate, slicker and just way more readable now. We know that the user-experience with a framework such as IntelMQ is very important. Good documentation helps a lot here. Please do check it out and give us feedback! 2. dynamic shadowserver reports and -parser ------------------------------------------- Lots of thanks to Shadowserver / @elsif2 for his continuous and dedicated contributions to make IntelMQ much better with shadowserver feeds. Motivation for this: Shadowserver adds new scans on a nearly weekly basis. IntelMQ's release cycle and the need for a stable release could not keep up with this high intensity of shadowserver parser changes. We therefore (thanks to @eslif2) move the shadowserver reports collector and parser to a new, dynamic system. It can: • fetch the shadowserver schema from shadowserver (https://interchange.shadowserver.org/intelmq/v1/schema) • dynamically collect new reports (see also https://docs.intelmq.org/latest/user/bots/?h=shadow#shadowserver-reports-api) • parse the new reports You can find all about the new shadowserver reports in IntelMQ here: https://docs.intelmq.org/latest/user/bots/#shadowserver-reports-api and https://docs.intelmq.org/latest/user/bots/#shadowserver 3. More ------------ And of course, lots of changes in the background and the core. Big thanks to Kamil and Sebix! You can find the full details here: https://github.com/certtools/intelmq/blob/develop/NEWS.md (short version) and here: https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md 4. Future of IntelMQ and release cycles ---------------------------------------- We know that the last one, two years, IntelMQ releases were a bit sporadic. We plan to change this and make it more deterministic. We therefore commit to 2 releases per year + two release candidates (intermediate, BETA) releases: Release 1: in quarter 1: March. Release 2: release-candidate in Q2 (end of June). Release 3: Sept Release 4: release-candidate release in Q4 (Dec). Also, in order to secure the future sustainability of IntelMQ, we will join the "ossbase" Open Source Security Software alliance. The idea here is to join a bigger foundation (similarly structured to the Apache Foundation) which will be created together with CIRCL.lu. Thanks Alexandre for continuing to push this idea. Being a part of ossbase will allow us to support IntelMQ in the future years while keeping the project independent, alive, kicking and a great community project. And that's what relevant. Stay tuned for updates here. 5. Thank you -------------- And finally and again: a big THANK YOU for all the committers, contributors, bug reporters, etc.!!! Without this community, we would not exist. You are mentioned here: https://github.com/certtools/intelmq/blob/develop/AUTHORS https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md https://github.com/certtools/intelmq/issues

1 0

Shadowserver parser extract_cve_from_tag
by Thomas Hungenberg 08 Feb '24

08 Feb '24

Hi, like for other Shadowserver reports, we are sending email notifications to affected network owners in Germany based on the Vulnerable-HTTP report. Unlike other reports (like Open-Portmapper), the Vulnerable-HTTP report contains information on different kinds of software/systems. So we are sending the notifications with a different template for each software/system. For some software/systems (like Fortinet or VMWare), the report contains information on different vulnerabilities (CVEs) and each host included in the report can be affected by one or more of them as listed in "tag", e.g. "cve-2019-5544;cve-2020-3992;cve-2021-21974;vmware" "cve-2020-3992;cve-2021-21974;vmware" "cve-2019-5544;vmware" "cve-2021-21974;vmware" "cve-2021-21972;cve-2023-34048;vcenter-dcerpc-exposed;vmware" "cve-2021-21972;cve-2023-20892;cve-2023-34048;vmware" So we need to tell the recipients of our notifications to which of the vulnerabilities their host is exactly vulnerable to. We do this by adding a column "cve" in addition to "ip", "port", etc. to the CSV data included with our notifications. However, including the raw "tag" attribute from the Shadowserver report would confuse the recipients as "tag" often includes additional information not relevant in the context of the specific report like "ssl-poodle", e.g. "cve-2024-23897;jenkins;ssl;ssl-poodle" "cve-2023-46805;cve-2024-21887;ivanti-connect-secure;ssl;ssl-freak;vpn" So we added a function "extract_cve_from_tag" to our local Shadowserver parser _config.py (included with IntelMQ 3.2.1) which returns a sorted comma separated list of CVEs included in "tag", e.g. extract_cve_from_tag("cve-2023-46805;cve-2024-21887;ivanti-connect-secure;ssl;ssl-freak;vpn") -> "cve-2023-46805,cve-2024-21887" The result is stored in "extra.cve" by the parser scan_http_vulnerable = { 'optional_fields': [ ('protocol.transport', 'protocol'), ('source.reverse_dns', 'hostname'), ('extra.', 'tag', validate_to_none), ('extra.cve', 'tag', extract_cve_from_tag), and this way can later easily be included in the CSV data generated for the notifications. I think this might be useful for others as well and would be glad if it could be added to the parser code and schema for the next release so we would not need to patch it locally after each update of the parser/schema. I have just created a pull request for this: https://github.com/certtools/intelmq/pull/2457 @elsif: If the PR is accepted, would you like to change the schema for scan_http_vulnerable and scan6_http_vulnerable like this? [ "extra.", "tag", "validate_to_none" ], + [ + "extra.cve", + "tag", + "extract_cve_from_tag" + ], Kind regards Thomas

3 6

classification attributes in IntelMQ Shadowserver parser schema
by Thomas Hungenberg 08 Feb '24

08 Feb '24

Hi, Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed. This is very important as our setup (and most probably others as well) heavily depends on known classification identifiers like "open-rdp" and classification types from the initial parsing of events up to notification_rules and formats/templates for mailgen. So with a change of a classification attribute lots of scripts and configs would need to be changed as well. Looking at the current schema, I see the classification identifiers are still correct for some feeds for both IPv4 and IPv6 like here: "scan_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver", "scan6_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver", However, for other feeds the classification identifier has been kept correctly for IPv4 like here: "scan_rdp" : { "constant_fields" : { "classification.identifier" : "open-rdp", "compromised_website" : { "constant_fields" : { "classification.identifier" : "compromised-website", but for IPv6 it has changed to the name of the feed: "scan6_rdp" : { "constant_fields" : { "classification.identifier" : "scan6-rdp", <- should be "open-rdp" "compromised_website6" : { "constant_fields" : { "classification.identifier" : "compromised-website6", <- should be "compromised-website" The classification.identifier should describe the incident (like "open-rdp") and not the source (like "scan6-rdp"). May I ask you to check and adjust all classification identifiers and types in the schema so they are consistent with the ones generated by the current _config.py? Thanks a lot for all your work on the new schema based parser! Kind regards Thomas

6 18

Shadowserver parser: Bad mapping for malware events
by Thomas Hungenberg 31 Jan '24

31 Jan '24

Hi all, the parsers for malware events provided by different sources usually store the malware name in malware.name and classification.identifier is left blank (or set to the feed's name). When using the malware name mapping, a harmonized malware name is subsequently written to classification.identifier. So finally you have the original name in malware.name and the harmonized name in classification.identifier. Formerly (in the version we initially provided), the Shadowserver parser also stored the malware name in malware.name, see e.g. <https://github.com/certtools/intelmq/blob/c61ff2fd4232d6937f3815377b75f682a…> line 387 However, for some time the Shadowserver parser now writes the malware name ("infection") to classification.identifier and "family" to malware.name instead. This is bad for several reasons: - it is not consistent with parsers for other malware feeds - it breaks deduplicators matching on malware.name - the malware name mapping overwrites classification.identifier with the value of "family" (which often is empty) Here is a patch (for the version included with IntelMQ 3.2.1) to fix this problem and make malware events parsed by the Shadowserver parser consistent with other parsers again: =============================================== diff --git a/_config.py.orig b/_config.py index bea3d0c..431bcb9 100644 --- a/_config.py.orig +++ b/_config.py @@ -867,10 +867,9 @@ event_sinkhole = { ('source.port', 'src_port', convert_int), ], 'optional_fields': [ - ('classification.identifier', 'infection', validate_to_none), - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('source.asn', 'src_asn', invalidate_zero), ('source.geolocation.cc', 'src_geo'), @@ -899,6 +898,7 @@ event_sinkhole = { 'constant_fields': { 'classification.taxonomy': 'malicious-code', 'classification.type': 'infected-system', + 'classification.identifier': 'sinkhole-events', }, } @@ -944,10 +944,9 @@ event_sinkhole_http = { ('source.port', 'src_port', convert_int), ], 'optional_fields': [ - ('classification.identifier', 'tag'), - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('source.asn', 'src_asn', invalidate_zero), ('source.geolocation.cc', 'src_geo'), @@ -982,6 +981,7 @@ event_sinkhole_http = { 'constant_fields': { 'classification.taxonomy': 'malicious-code', 'classification.type': 'infected-system', + 'classification.identifier': 'sinkhole-http-events', 'protocol.application': 'http', }, } @@ -992,9 +992,9 @@ event_sinkhole_http_referer = { ('time.source', 'timestamp', add_UTC_to_timestamp), ], 'optional_fields': [ - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('extra.', 'http_referer_ip', validate_ip), ('extra.', 'http_referer_port', convert_int), =============================================== Kind regards Thomas

4 11

Re: [IntelMQ-dev] [IntelMQ-users] IntelMQ & Kubernetes
by Bernhard Reiter 18 Jan '24

18 Jan '24

Hello Jessica, Am Mittwoch 17 Januar 2024 09:47:25 schrieb Jessica Schumacher via IntelMQ-users: > Now we want to take the “next” step to Kubernetes. > Does anyone have experience with running IntelMQ on Kubernetes? what Kubernetes implementation or components are you planning to use? E.g. k3s or something else? What is the problem you want to solve with this step? (I am seriously interested, because my experience with Kubernetes is very limited, so I want to understand where it actually helps. So far I've seen it has an advantage for large, multi-machine scaled application settings which does not entirely fit my mental model of IntelMQ. I expect it to be possible like you can put any application in such a setting, but it is worth the effort?) Regards, Bernhard -- https://intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter

1 0

STATS IN INTELMQ?
by Homma, L.J. (Luitzen) 28 Nov '23

28 Nov '23

Dear IntelMQ Developers & Users, We are curious if there are any plans on the roadmap to incorporate statistical features into IntelMQ. About 1.5 years ago, we participated in an online session where it was mentioned that there were some early plans to integrate stats into IntelMQ. As far as we could find, there have not been any steps in this direction. Are we correct? Currently, we are working in our experimental environment to develop stats based on a Prometheus bot, using Prometheus as a time-series database, and utilizing Grafana for dashboarding and visualization. Are there more members of the community working on this? Our goal is to gain better insights into the input, filtering, and output of our IntelMQ pipeline(s). We hope to hear from others about their thoughts on this. Met vriendelijke groet, Luitzen Homma Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is gezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten. This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages.

6 18

Introducing proposals to IntelMQ Data Format (IEP008 and 9)
by Kamil Mankowski 02 Nov '23

02 Nov '23

Dear IntelMQ Developers & Users, I'd like to propose two changes to the IntelMQ data format to make it more suitable for current use cases. 1. IEP008 - add constituency identifier (this has already been shared, but only on the developer list): https://github.com/certtools/ieps/pull/9 2. IEP009 - add product and vulnerability information: https://github.com/certtools/ieps/pull/10 I'd like to hear your feedback on these suggestions. Feel free to comment here or on the Github pull requests. My plan is to implement them, unless rejected, by December. -- Best regards, Kamil Mankowski CERT.at GmbH www.cert.at

1 0

Expected behaviour of bot reloading after log rotation
by Mika Silander 24 Oct '23

24 Oct '23

Hi all, Apologies for a lengthy message, but I had a set of monitoring scripts that worked like a charm on a physical machine but when moved to a virtual machine with a newer Ubuntu, I see odd failures in the form of monitoring script processes left hanging on void file handles. I suspect the problem is somehow related to log rotation in connection to bot reloading but I can't figure out how. A few questions at the end. Our setup for reference: Ubuntu 22.04 LTS python 3.10.12 intelmq 3.2.2a1 intelmq-api 3.1.0-rc1 intelmq-manager 3.2.0 We have a simple bot chain: Mail Attachment Collector -> Shadowserver Parser -> N expert bots -> 1 output bot. /etc/logrotate.d/intelmq contains: /var/log/intelmq/*.log { su intelmq intelmq daily maxsize 10M rotate 60 notifempty compress delaycompress create 644 intelmq intelmq sharedscripts postrotate sudo -u intelmq /usr/bin/intelmqctl --quiet reload endscript } The cron daemon runs run-parts for targets under /etc/cron.daily/ at 06:25, including /etc/cron.daily/logrotate. Thus, "intelmqctl --quiet reload" gets also executed after all the intelmq log files have been rotated. What I see is that no bot reloads soon after run-parts has finished. The first bot to perform a reload is the Mail Attachment Collector immediately after midnight. The others remain as is until the Mail Attachment Collector reads the day's first report and forwards it to the Shadowserver Parser bot. The parser bot then performs a reload and forwards the events to the first expert in the chain. Once the first event from the parser trickles through the entire bot chain, every bot in turn first performs a reload and then processes the event. In practise, the logrotate triggered bot reload can thus be delayed almost a full day - our first mail reports tend to arrive during the very early morning hours. My question is, is the above reload behaviour as expected? I found it strange that the Mail Attachment Collector reloads at midnight even though it has processed new mail reports after the "intelmqctl reload" call done by logrotate. I would have expected the Mail Attachment Collector to reload itself when the first report arrives for processing after 06:25, i.e. the first report after the call to intelqmctl reload. And, what controls the timing of the Mail Attachment Collector's reload? All its reloads I see in older log files indicate that the reload is consistently run after midnight irrespective of incoming reports and the timing of the intelmqctl reload command. Thanks again and best regards from a puzzled Mika

3 6

IEP008 - Constituency field in IntelMQ Data Format
by Kamil Mankowski 09 Oct '23

09 Oct '23

Dear IntelMQ Community, I'd like to introduce an IEP with small extension to the current IntelMQ Data Format. As supporting multiple constituency groups is becoming more popular, we find it useful to include this information directly in the data format. IEP: https://github.com/certtools/ieps/pull/9/files This should not introduce any troubles to single-constituency setups, as well as do not define, how to mark the constituencies. The idea is just to open a possibility for that. I'd appreciate a feedback from the Community! -- Best regards, Kamil Mankowski CERT.at GmbH www.cert.at

1 0

intelmq-manager problem at login
by Mika Silander 19 Sep '23

19 Sep '23

Hi again, Hints would be welcome: I've got intelmq-manager installed, the front page opens ok but when I try to login, the developer view of the browser reports the following error after I specify the account and the password: DevTools failed to load source map: Could not load content for https://FQDN_HERE/intelmq_manager/plugins/bootstrap/bootstrap.min.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE Checking what's in /usr/share/intelmq_manager/html/plugins/bootstrap/ reveals there is no file there called bootstrap.min.css.map. The files that do exist in that directory are: bootstrap.css bootstrap.js bootstrap.min.css bootstrap.min.js My setup running on a Ubuntu 22.04 LTS / Python 3.10.12 is now: ii intelmq 3.2.2-1 all Solution for IT security teams for collecting ii intelmq-api 3.1.0~rc2-1 all FastAPI-based API for the intelmq project ii intelmq-manager 3.2.0-1 all Graphical interface to manage configurations As a side note, the intelmq-api package was upgraded from intelmq-api 3.1.0-rc1 in case this could be the culprit. Br and have a nice weekend, Mika

3 14

Trouble building deb package for intelmq-3.2.2
by Mika Silander 04 Sep '23

04 Sep '23

Hi all, Before digging any deeper I'll ask the list for advice. It's possible someone's bumped into the same error. I've upgraded from Ubuntu 20 LTS with intelmq-api 3.1.0~rc-1 intelmq-manager 3.2.0-1 intelmq 3.1.0-1 python3.8 to Ubuntu 22 LTS. Now, when trying to build intelmq-3.2.2 + a few of our own bots on this upgraded host with dpkg-buildpackage -us -us -b the build fails with: test-bot: Bot stopped. test-bot: Processed 1 messages since last logging. test-bot: Bot stopped. test-bot: Processed 1 messages since last logging. test-bot: Bot stopped. E: pybuild pybuild:369: test: plugin distutils failed with: exit code=1: python3.10 setup.py test dh_auto_test: error: pybuild --test -i python{version} -p 3.10 returned exit code 13 make: *** [debian/rules:24: build] Error 255 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2 The host has python3-distutils 3.10.8-1~22.04 installed. Any hints on what lies behind this error? Br & thanks, Mika

3 3

IntelMQ 3.2.1 release with two important bug fixes
by Sebix 28 Aug '23

28 Aug '23

Dear IntelMQ community, Together with CERT.at we just released IntelMQ version 3.2.1 containing two important bug fixes. 1. A bug in the bot's core library: > Fixes an issue which prevented bots from stopping gracefully after reloading. > As logrotate reloads all bots regularly, this bug affects most IntelMQ installations. 2. A bug in the Reverse DNS Expert bot: > Until IntelMQ version 3.2.0, the bot incorrectly cached and re-used results for /24 networks instead of single IP addresses. > If the bot retrieved the PTR for `192.0.43.7`, it was cached for `192.0.43.0/24` and used for all IP addresses in this range, for example for `192.0.43.8`. > IntelMQ version 3.2.1 fixes this issue. > The bugfix will correctly increase the cache sizes and decrease the performance, as less (incorrect) data is re-used. Upgrading: https://intelmq.readthedocs.io/en/develop/user/upgrade.html Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html The PyPI package and git tag/GitHub release are published. The new deb-/rpm-packages will hit the repositories in the next few minutes and the updated Docker image follows soon. best regards Sebastian -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://sebix.at/ ZVR 1510673578

1 0

Shift of focus, new fields in IDF needed for scanning results?
by Otmar Lendl 21 Aug '23

21 Aug '23

Folks, I've been looking at some of the data that we recently processed with our IntelMQ setup (leading to https://cert.at/de/aktuelles/2023/8/verwundbare-webserver-status-in-osterre…), and I found that we need a few changes to IntelMQ to better process these kind of feeds. What happened? Initially, our focus with IntelMQ was on botnet drones which were detected via sinkholing. That's why we have the source.*, destination.*, and malware.* parameters in the data model. A C2 communication was the main inspiration. These days, a majority of the data feeds we process are generated by scanning the Internet. That can be Shodan, that can be Shadowserver, that can be from our own local scans. This is fine and good, it gives us valuable telemetry on what's going on in our constituency. But the fields needed to store the data in a consistent way are missing in the current iteration of the IDF. I don't want to pack that into extra.* or any non-standard field, I think we should come to an agreement what those elements are and how they should be handled. Here are some of the data fields I'm missing: What did the scanning find? * vendor * product * software version Are there any problems with it? * Vulnerability identifier (CVE or other, potentially multi-valued, which is another can of worms) * Severity information (e.g. associated CVSS score) * CWE IDs (https://cwe.mitre.org/index.html) Generic tagging * I see e.g. "iot", "ics" or similar ones popping up. As a minimum I think we need "vendor", "product" and "vulnerability". What do you all think? otmar ----------- References: https://datapedia.shodan.io/ There is e.g.: os string Operating system platform string product string Name of the software that powers the service. vendor string https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-http-r… info on the vendor and CVEs is stored in the tag field. -- // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

4 4

IntelMQ 3.2 announcement
by L. Aaron Kaplan 20 Jul '23

20 Jul '23

IntelMQ 3.2 release announcement ================================== [TLP:WHITE, please pass this on, if appropriate] Dear everyone, We are excited to announce that today we can release 3.2.0 of the open source IntelMQ (https://github.com/certtools/intelmq) incident handling automation framework. Special thanks go to Kamil (CERT.at), Sebix, Intevation, Filip Pokorny (Gethvi) and Jason from Shadowserver (you all know who you are). Also I would like to thank the 93 contributors [1] who have been helping in many ways over the last year(s) with IntelMQ. Core members of IntelMQ are committed to continuing to develop, support and enhance IntelMQ in the long run. Especially since IntelMQ is a quite well adopted extract transform load (ETL) tool for fetching incident report feeds, processing them and sending them to the right place. It is widely used in Europe and in the whole world, in CERTs, SOCs and MSSPs. And best of it, it is open source and freely available. Again, we can not thank all the contributors enough. So, what's in release 3.2? =========================== Highlights: * The main change is that IntelMQ's code can now be used as a python module / library. That means: you can easily embed the hard work that IntelMQ parser writers did into your application. It's as simple as instantiating an IntelMQ bot in python and giving it data. Thanks go to Sebix. See also https://github.com/certtools/ieps/tree/main/007 * the IntelMQ API moved from hug to fastapi [4]. The API interface stayed the same. Thanks a lot to Kamil. * We have an updated intelmq developer version docker image. Apart from that, a few other highlights: * Shadowserver parser improvements. * New shadowserver data feeds are supported * Changes time_format parameter to use new TimeFormat class * new CIF3 output bot * intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar) As always, you can find all the nitty gritty details in the CHANGELOG [2] file and in the NEWS file [3] What's next? ============== * BSI funded a complete re-write of webinput-csv (https://github.com/Intevation/intelmq-webinput-csv), watch for an announcement soon. * Improved and re-done documentation via mkdocs (Thanks to Filip) * The upcoming next version 3.3 will have an update for the dynamic config loading of the shadowserver parser [4]. This will always give you the most current shadowserver feeds and parsers without having to wait for an official intelmq release. If you depend a lot on shadowserver feeds, it's ok to wait for the next release, which will be following very shortly. (And a big hooray to shadowserver for their help, we know that you need this feature quickly) We plan to release 3.3 in approx 1 week. Should I update? ================== Yes! **Note well**: we still have some issues with proper Debian packages for fastapi (a dependency of the new IntelMQ API server). This means re-packaging fastapi with the help of Debian developers or using the old hug API for now (no difference from a user perspective though). The deb-package for the new fastapi-based intelmq-api is currently only available on Debian 11. We are also sorting out an issue with the package build of intelmq on Debian 12: https://github.com/certtools/intelmq/issues/2384#issuecomment-1637799252 and issues on Ubuntu related to fastapi-Versions. Packages of the previous compatible hug-based intelmq-api remain the package repositories of affected distributions. Once these issues are sorted out, the new packages will become available for installation and upgrade in the respective repositories. For installations via pip or development setups, nothing needs to be kept in mind. How to update? =============== * Upgrading: https://intelmq.readthedocs.io/en/3.2.0/user/upgrade.html * Fresh installations: https://intelmq.readthedocs.io/en/3.2.0/user/installation.html (Note that the docker installation might lag behind a bit) Need help? =========== You can contact us on the mailing lists: * intelmq-dev for developers [5] * intelmq-users for users [6] See also: https://intelmq.readthedocs.io/en/3.2.0/user/support.html Thank you and thanks to all the contributors! Aaron Kaplan, Sebix, Kami, Filip (for the whole IntelMQ team) [1] https://github.com/certtools/intelmq/graphs/contributors [2] https://github.com/certtools/intelmq/blob/3.2.0/CHANGELOG.md [3] https://github.com/certtools/intelmq/blob/3.2.0/NEWS.md [4] https://github.com/certtools/intelmq/pull/2372 [5] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev [6] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users

1 0

Odd intelmqdump testing errors
by Mika Silander 26 May '23

26 May '23

Hi all, Running the entire unit test suite of intelmq I come across a few test failures which make me ponder whether my development environment is out of sync or outdated. Tests on intelmqdump fail in the following manner: -- FAIL: test_list_and_select_dump_from_custom_location (intelmq.tests.bin.test_intelmqdump.TestIntelMQDump) [0] ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/my_development/intelmq/intelmq/tests/bin/test_intelmqdump.py", line 165, in test_list_and_select_dump_from_custom_location self.assertIn("0: test-1 empty file", output[1]) AssertionError: '0: test-1 empty file' not found in ' 0: test-1 empty file' -- So, the assertion string looks "almost" what is expected. Can this be an old(er) python library issue or something? Currently I develop on Ubuntu 20.04.6 LTS, using python 3.8.10 and the intelmqdump sources tested are from the intelmq-github/develop branch, commit tag 0780ee5c1281da8a6defcdd8f8c4722ccdbee4e8. Br, Mika

3 4

Speed dumping events in bots
by Mika Silander 17 May '23

17 May '23

Hi again, Afaik, if handling an event fails in a bot, the default behaviour of a bot is to sleep 15 seconds and then retry processing. If the retry fails, the bot dumps the event and picks the next event from the inqueue. We have a bot where it would be desirable to change this default behaviour so that the dump is done immediately on the first failure. Is there a way to configure a single bot to behave differently from other bots as described? Or will a change in configuration affect the entire bot net? If there's no easy way configuring, I guess technically I could implement exceptions to be thrown in those situations where quick dumping is desired and then call the _dump_message(?) method. However, I would prefer to modify as few bots of the official distribution as possible. Comments, pointers to docs, sources or the like are most welcome. Br, Mika

3 5

Structural change in one feed's events output by the Shadowserver Parser bot?
by Mika Silander 05 May '23

05 May '23

Hi all, Testing with intelmq 3.1.0 and specifically its Shadowserver Parser bot. I've become used to being able to read in an event output by this bot as plain JSON with the json libraries bundled in python3, but now I came across one event sample where this fails. The feed is Accessible Traversal Utilities for NAT. The troublemaker event contained the field extra.amplification, and it was output by the bot as a dict structure like 'extra.amplification': Decimal("5.2") whereas I expected the field value to be a string "5.2" or number 5.2. Please let me know what I've missed again. A json library update? Br and have a nice weekend, Mika

1 1

RFC on IEP007: Running IntelMQ as Python Library
by Sebix 04 May '23

04 May '23

Dear community, I invite you to discuss a new (IEP - IntelMQ Enhancement Proposal) IEP007: Running IntelMQ as Python Library Have you ever wondered if you can write a Python script, call a bot's process method, pass it some data and get back the enriched/modified data? (pseudo code) bot_instance = Bot(parameters) bot_instance.process_message(input message) -> output messages Strictly speaking, it *is* actually possible with the current version, but it requires some bizarre hacks like re-defining Bot's methods and overwriting internal values. Staying on the wishlist for quite a while, we intend to implement the feature now. I started the IEP007 draft and need some input from you to maximize the benefit for all the IntelMQ community (developers): https://github.com/certtools/ieps/pull/7/files Or in a readable display: https://github.com/certtools/ieps/tree/iep-007/007 What features and interfacing options would you expect when starting the bot as a library? Do you think the `Bot.process` method should be rewritten entirely now, removing the calls receive_message/send_message and converting the method into a generator (an API-breaking change)? And if yes, should this be done in one step, or separated from this bot-as-library feature, reducing the complexity of development steps? Looking forward for your ideas Sebastian -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://sebix.at/ ZVR 1510673578

4 8

IntelMQ 3.1 release
by L. Aaron Kaplan 10 Feb '23

10 Feb '23

[TLP:WHITE, please pass this on, if appropriate] Dear everyone, it took a bit of time but I am excited to announce that today we finally pushed out release 3.1 of the open source IntelMQ (https://github.com/certtools/intelmq) incident handling automation framework. Special thanks go to Kamil, Sebix, Filip Pokorny (Gethvi) and Jason from Shadowserver (you all know who you are). Also I would like to thank the 91 (!) contributors [1] who have been helping in many ways over the last year(s) with IntelMQ. This truly is became a community effort and I am very happy that we can push this project further and maintain IntelMQ in the long run as a community project. IntelMQ is by now quite well respected as a "glue" / extract transform load (ETL) tool for fetching incident report feeds, processing them and sending them to the right place. It is widely used in Europe, in CERTs, SOCs and MSSPs. And best of it, it is open source and freely available. Again, we can not thank all the contributors enough. So, what's in release 3.1? =========================== (TL;DR: - find all things here: https://github.com/certtools/intelmq/blob/develop/NEWS.md) Highlights: * Excellent support for new Shadowserver feeds (thanks to Jason and Shadowserver). This is for sure the biggest part here. Shadowserver explicitly mentions IntelMQ for processing their feeds and will maintain the IntelMQ parsers for their feeds. * Big improvements for the dataplane.org parsers * Big shodan parser improvements Apart from that, a few other highlights: * More CI/CD tests * Tons of code clean-ups * New bots such as the url expert * Enhancements of existing bots: * SQL Output support MSSQL now, * GitHub collector, * Alienvault parser improvements, * Shodan parser improvements, * maxmind geoIP expert improvements, etc etc. * Tons of behind the scenes bugfixes! * Core changes (which you won't notice much) in the core part (pipeline class) etc etc.. We also dropped some support for old, outdated versions (python 3.6 or older Debian and old RedHat systems). This slims down the code base a bit again. As always, you can find all the nitty gritty details in the CHANGELOG [2] file and in the NEWS file [3] What's next? ============== * API: We want to move the IntelMQ-API [4] to fastapi (thanks to Kamil). Due to minor issues with packaging, this is not happening today, but this is planned for next week. So, stay tuned. IntelMQ-api and intelmq will be both numbered with the same version 3.1. * More user-feedback in the next few weeks. * Update the docker images for intelmq 3.1 * Upcoming task-specific tutorials on intelmq.org (for example how to send data to Splunk, how to automate all shadowserver feeds, etc.). Courtesy of David Ruefenacht. Should I update? ================== Yes! (but first on a DEV instance!) If you run a production instance, please update on a dev instance first and cross check. Because every IntelMQ setup is different. We try to cover a lot in our testing[5] , but we can't give a guarantee that everything is completely covered, especially if there are local changes. So, yes, please update, but tell us [6] in case something breaks in your DEV environment. (But yes! you really should update - since you might really want to have all those Shadowserver and dataplane.org updates) How to update? =============== * Upgrading: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html * Fresh installations: https://intelmq.readthedocs.io/en/maintenance/user/installation.html# (Note that the docker installation might lag behind a bit) Thank you and thanks to all the contributors! Aaron Kaplan. (for the whole IntelMQ team) [1] https://github.com/certtools/intelmq/graphs/contributors [2] https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md [3] https://github.com/certtools/intelmq/blob/develop/NEWS.md [4] https://github.com/certtools/intelmq-api [5] https://intelmq.readthedocs.io/en/maintenance/dev/guide.html#testing [6] https://github.com/certtools/intelmq/issues

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev