IntelMQ-dev

intelmq-dev@lists.cert.at

3 participants
238 discussions

Start a nNew thread

test, please ignore
by L. Aaron Kaplan 03 Apr '24

03 Apr '24

test test

1 0

Proposed classification for new loop-dos report
by elsif 20 Mar '24

20 Mar '24

We are in the process of creating a new daily report based on the previous loop-dos special report: https://www.shadowserver.org/what-we-do/network-reporting/loop-dos-special-… I would like to propose the following constant_fields: classification.taxonomy = vulnerable classification.type = vulnerable-system protocol.application = application Where the application would be tftp or dns for example.

4 5

intelmq 3.3.0 release
by L. Aaron Kaplan 01 Mar '24

01 Mar '24

Dear intelmq users, developers and IR automation people, we are very happy to announce (finally announce!) IntelMQ 3.3.0 See: https://github.com/certtools/intelmq Important changes =================== The most relevant changes are: 1. Documentation ------------------ great new documentation (mkdocs) at https://docs.intelmq.org/ (Thanks a lot to Filip/ @gethvi! Amazing work) Documentation has been updated and restructured into User, Administrator and Developer Guide. The documentation is easier to navigate, slicker and just way more readable now. We know that the user-experience with a framework such as IntelMQ is very important. Good documentation helps a lot here. Please do check it out and give us feedback! 2. dynamic shadowserver reports and -parser ------------------------------------------- Lots of thanks to Shadowserver / @elsif2 for his continuous and dedicated contributions to make IntelMQ much better with shadowserver feeds. Motivation for this: Shadowserver adds new scans on a nearly weekly basis. IntelMQ's release cycle and the need for a stable release could not keep up with this high intensity of shadowserver parser changes. We therefore (thanks to @eslif2) move the shadowserver reports collector and parser to a new, dynamic system. It can: • fetch the shadowserver schema from shadowserver (https://interchange.shadowserver.org/intelmq/v1/schema) • dynamically collect new reports (see also https://docs.intelmq.org/latest/user/bots/?h=shadow#shadowserver-reports-api) • parse the new reports You can find all about the new shadowserver reports in IntelMQ here: https://docs.intelmq.org/latest/user/bots/#shadowserver-reports-api and https://docs.intelmq.org/latest/user/bots/#shadowserver 3. More ------------ And of course, lots of changes in the background and the core. Big thanks to Kamil and Sebix! You can find the full details here: https://github.com/certtools/intelmq/blob/develop/NEWS.md (short version) and here: https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md 4. Future of IntelMQ and release cycles ---------------------------------------- We know that the last one, two years, IntelMQ releases were a bit sporadic. We plan to change this and make it more deterministic. We therefore commit to 2 releases per year + two release candidates (intermediate, BETA) releases: Release 1: in quarter 1: March. Release 2: release-candidate in Q2 (end of June). Release 3: Sept Release 4: release-candidate release in Q4 (Dec). Also, in order to secure the future sustainability of IntelMQ, we will join the "ossbase" Open Source Security Software alliance. The idea here is to join a bigger foundation (similarly structured to the Apache Foundation) which will be created together with CIRCL.lu. Thanks Alexandre for continuing to push this idea. Being a part of ossbase will allow us to support IntelMQ in the future years while keeping the project independent, alive, kicking and a great community project. And that's what relevant. Stay tuned for updates here. 5. Thank you -------------- And finally and again: a big THANK YOU for all the committers, contributors, bug reporters, etc.!!! Without this community, we would not exist. You are mentioned here: https://github.com/certtools/intelmq/blob/develop/AUTHORS https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md https://github.com/certtools/intelmq/issues

1 0

Shadowserver parser extract_cve_from_tag
by Thomas Hungenberg 08 Feb '24

08 Feb '24

Hi, like for other Shadowserver reports, we are sending email notifications to affected network owners in Germany based on the Vulnerable-HTTP report. Unlike other reports (like Open-Portmapper), the Vulnerable-HTTP report contains information on different kinds of software/systems. So we are sending the notifications with a different template for each software/system. For some software/systems (like Fortinet or VMWare), the report contains information on different vulnerabilities (CVEs) and each host included in the report can be affected by one or more of them as listed in "tag", e.g. "cve-2019-5544;cve-2020-3992;cve-2021-21974;vmware" "cve-2020-3992;cve-2021-21974;vmware" "cve-2019-5544;vmware" "cve-2021-21974;vmware" "cve-2021-21972;cve-2023-34048;vcenter-dcerpc-exposed;vmware" "cve-2021-21972;cve-2023-20892;cve-2023-34048;vmware" So we need to tell the recipients of our notifications to which of the vulnerabilities their host is exactly vulnerable to. We do this by adding a column "cve" in addition to "ip", "port", etc. to the CSV data included with our notifications. However, including the raw "tag" attribute from the Shadowserver report would confuse the recipients as "tag" often includes additional information not relevant in the context of the specific report like "ssl-poodle", e.g. "cve-2024-23897;jenkins;ssl;ssl-poodle" "cve-2023-46805;cve-2024-21887;ivanti-connect-secure;ssl;ssl-freak;vpn" So we added a function "extract_cve_from_tag" to our local Shadowserver parser _config.py (included with IntelMQ 3.2.1) which returns a sorted comma separated list of CVEs included in "tag", e.g. extract_cve_from_tag("cve-2023-46805;cve-2024-21887;ivanti-connect-secure;ssl;ssl-freak;vpn") -> "cve-2023-46805,cve-2024-21887" The result is stored in "extra.cve" by the parser scan_http_vulnerable = { 'optional_fields': [ ('protocol.transport', 'protocol'), ('source.reverse_dns', 'hostname'), ('extra.', 'tag', validate_to_none), ('extra.cve', 'tag', extract_cve_from_tag), and this way can later easily be included in the CSV data generated for the notifications. I think this might be useful for others as well and would be glad if it could be added to the parser code and schema for the next release so we would not need to patch it locally after each update of the parser/schema. I have just created a pull request for this: https://github.com/certtools/intelmq/pull/2457 @elsif: If the PR is accepted, would you like to change the schema for scan_http_vulnerable and scan6_http_vulnerable like this? [ "extra.", "tag", "validate_to_none" ], + [ + "extra.cve", + "tag", + "extract_cve_from_tag" + ], Kind regards Thomas

3 6

classification attributes in IntelMQ Shadowserver parser schema
by Thomas Hungenberg 08 Feb '24

08 Feb '24

Hi, Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed. This is very important as our setup (and most probably others as well) heavily depends on known classification identifiers like "open-rdp" and classification types from the initial parsing of events up to notification_rules and formats/templates for mailgen. So with a change of a classification attribute lots of scripts and configs would need to be changed as well. Looking at the current schema, I see the classification identifiers are still correct for some feeds for both IPv4 and IPv6 like here: "scan_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver", "scan6_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver", However, for other feeds the classification identifier has been kept correctly for IPv4 like here: "scan_rdp" : { "constant_fields" : { "classification.identifier" : "open-rdp", "compromised_website" : { "constant_fields" : { "classification.identifier" : "compromised-website", but for IPv6 it has changed to the name of the feed: "scan6_rdp" : { "constant_fields" : { "classification.identifier" : "scan6-rdp", <- should be "open-rdp" "compromised_website6" : { "constant_fields" : { "classification.identifier" : "compromised-website6", <- should be "compromised-website" The classification.identifier should describe the incident (like "open-rdp") and not the source (like "scan6-rdp"). May I ask you to check and adjust all classification identifiers and types in the schema so they are consistent with the ones generated by the current _config.py? Thanks a lot for all your work on the new schema based parser! Kind regards Thomas

6 18

Shadowserver parser: Bad mapping for malware events
by Thomas Hungenberg 31 Jan '24

31 Jan '24

Hi all, the parsers for malware events provided by different sources usually store the malware name in malware.name and classification.identifier is left blank (or set to the feed's name). When using the malware name mapping, a harmonized malware name is subsequently written to classification.identifier. So finally you have the original name in malware.name and the harmonized name in classification.identifier. Formerly (in the version we initially provided), the Shadowserver parser also stored the malware name in malware.name, see e.g. <https://github.com/certtools/intelmq/blob/c61ff2fd4232d6937f3815377b75f682a…> line 387 However, for some time the Shadowserver parser now writes the malware name ("infection") to classification.identifier and "family" to malware.name instead. This is bad for several reasons: - it is not consistent with parsers for other malware feeds - it breaks deduplicators matching on malware.name - the malware name mapping overwrites classification.identifier with the value of "family" (which often is empty) Here is a patch (for the version included with IntelMQ 3.2.1) to fix this problem and make malware events parsed by the Shadowserver parser consistent with other parsers again: =============================================== diff --git a/_config.py.orig b/_config.py index bea3d0c..431bcb9 100644 --- a/_config.py.orig +++ b/_config.py @@ -867,10 +867,9 @@ event_sinkhole = { ('source.port', 'src_port', convert_int), ], 'optional_fields': [ - ('classification.identifier', 'infection', validate_to_none), - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('source.asn', 'src_asn', invalidate_zero), ('source.geolocation.cc', 'src_geo'), @@ -899,6 +898,7 @@ event_sinkhole = { 'constant_fields': { 'classification.taxonomy': 'malicious-code', 'classification.type': 'infected-system', + 'classification.identifier': 'sinkhole-events', }, } @@ -944,10 +944,9 @@ event_sinkhole_http = { ('source.port', 'src_port', convert_int), ], 'optional_fields': [ - ('classification.identifier', 'tag'), - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('source.asn', 'src_asn', invalidate_zero), ('source.geolocation.cc', 'src_geo'), @@ -982,6 +981,7 @@ event_sinkhole_http = { 'constant_fields': { 'classification.taxonomy': 'malicious-code', 'classification.type': 'infected-system', + 'classification.identifier': 'sinkhole-http-events', 'protocol.application': 'http', }, } @@ -992,9 +992,9 @@ event_sinkhole_http_referer = { ('time.source', 'timestamp', add_UTC_to_timestamp), ], 'optional_fields': [ - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('extra.', 'http_referer_ip', validate_ip), ('extra.', 'http_referer_port', convert_int), =============================================== Kind regards Thomas

4 11

Re: [IntelMQ-dev] [IntelMQ-users] IntelMQ & Kubernetes
by Bernhard Reiter 18 Jan '24

18 Jan '24

Hello Jessica, Am Mittwoch 17 Januar 2024 09:47:25 schrieb Jessica Schumacher via IntelMQ-users: > Now we want to take the “next” step to Kubernetes. > Does anyone have experience with running IntelMQ on Kubernetes? what Kubernetes implementation or components are you planning to use? E.g. k3s or something else? What is the problem you want to solve with this step? (I am seriously interested, because my experience with Kubernetes is very limited, so I want to understand where it actually helps. So far I've seen it has an advantage for large, multi-machine scaled application settings which does not entirely fit my mental model of IntelMQ. I expect it to be possible like you can put any application in such a setting, but it is worth the effort?) Regards, Bernhard -- https://intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter

1 0

STATS IN INTELMQ?
by Homma, L.J. (Luitzen) 28 Nov '23

28 Nov '23

Dear IntelMQ Developers & Users, We are curious if there are any plans on the roadmap to incorporate statistical features into IntelMQ. About 1.5 years ago, we participated in an online session where it was mentioned that there were some early plans to integrate stats into IntelMQ. As far as we could find, there have not been any steps in this direction. Are we correct? Currently, we are working in our experimental environment to develop stats based on a Prometheus bot, using Prometheus as a time-series database, and utilizing Grafana for dashboarding and visualization. Are there more members of the community working on this? Our goal is to gain better insights into the input, filtering, and output of our IntelMQ pipeline(s). We hope to hear from others about their thoughts on this. Met vriendelijke groet, Luitzen Homma Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is gezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten. This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages.

6 18

Introducing proposals to IntelMQ Data Format (IEP008 and 9)
by Kamil Mankowski 02 Nov '23

02 Nov '23

Dear IntelMQ Developers & Users, I'd like to propose two changes to the IntelMQ data format to make it more suitable for current use cases. 1. IEP008 - add constituency identifier (this has already been shared, but only on the developer list): https://github.com/certtools/ieps/pull/9 2. IEP009 - add product and vulnerability information: https://github.com/certtools/ieps/pull/10 I'd like to hear your feedback on these suggestions. Feel free to comment here or on the Github pull requests. My plan is to implement them, unless rejected, by December. -- Best regards, Kamil Mankowski CERT.at GmbH www.cert.at

1 0

Expected behaviour of bot reloading after log rotation
by Mika Silander 24 Oct '23

24 Oct '23

Hi all, Apologies for a lengthy message, but I had a set of monitoring scripts that worked like a charm on a physical machine but when moved to a virtual machine with a newer Ubuntu, I see odd failures in the form of monitoring script processes left hanging on void file handles. I suspect the problem is somehow related to log rotation in connection to bot reloading but I can't figure out how. A few questions at the end. Our setup for reference: Ubuntu 22.04 LTS python 3.10.12 intelmq 3.2.2a1 intelmq-api 3.1.0-rc1 intelmq-manager 3.2.0 We have a simple bot chain: Mail Attachment Collector -> Shadowserver Parser -> N expert bots -> 1 output bot. /etc/logrotate.d/intelmq contains: /var/log/intelmq/*.log { su intelmq intelmq daily maxsize 10M rotate 60 notifempty compress delaycompress create 644 intelmq intelmq sharedscripts postrotate sudo -u intelmq /usr/bin/intelmqctl --quiet reload endscript } The cron daemon runs run-parts for targets under /etc/cron.daily/ at 06:25, including /etc/cron.daily/logrotate. Thus, "intelmqctl --quiet reload" gets also executed after all the intelmq log files have been rotated. What I see is that no bot reloads soon after run-parts has finished. The first bot to perform a reload is the Mail Attachment Collector immediately after midnight. The others remain as is until the Mail Attachment Collector reads the day's first report and forwards it to the Shadowserver Parser bot. The parser bot then performs a reload and forwards the events to the first expert in the chain. Once the first event from the parser trickles through the entire bot chain, every bot in turn first performs a reload and then processes the event. In practise, the logrotate triggered bot reload can thus be delayed almost a full day - our first mail reports tend to arrive during the very early morning hours. My question is, is the above reload behaviour as expected? I found it strange that the Mail Attachment Collector reloads at midnight even though it has processed new mail reports after the "intelmqctl reload" call done by logrotate. I would have expected the Mail Attachment Collector to reload itself when the first report arrives for processing after 06:25, i.e. the first report after the call to intelqmctl reload. And, what controls the timing of the Mail Attachment Collector's reload? All its reloads I see in older log files indicate that the reload is consistently run after midnight irrespective of incoming reports and the timing of the intelmqctl reload command. Thanks again and best regards from a puzzled Mika

3 6

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev