IntelMQ-dev January 2024

intelmq-dev@lists.cert.at

5 participants
3 discussions

classification attributes in IntelMQ Shadowserver parser schema
by Thomas Hungenberg 08 Feb '24

08 Feb '24

Hi, Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed. This is very important as our setup (and most probably others as well) heavily depends on known classification identifiers like "open-rdp" and classification types from the initial parsing of events up to notification_rules and formats/templates for mailgen. So with a change of a classification attribute lots of scripts and configs would need to be changed as well. Looking at the current schema, I see the classification identifiers are still correct for some feeds for both IPv4 and IPv6 like here: "scan_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver", "scan6_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver", However, for other feeds the classification identifier has been kept correctly for IPv4 like here: "scan_rdp" : { "constant_fields" : { "classification.identifier" : "open-rdp", "compromised_website" : { "constant_fields" : { "classification.identifier" : "compromised-website", but for IPv6 it has changed to the name of the feed: "scan6_rdp" : { "constant_fields" : { "classification.identifier" : "scan6-rdp", <- should be "open-rdp" "compromised_website6" : { "constant_fields" : { "classification.identifier" : "compromised-website6", <- should be "compromised-website" The classification.identifier should describe the incident (like "open-rdp") and not the source (like "scan6-rdp"). May I ask you to check and adjust all classification identifiers and types in the schema so they are consistent with the ones generated by the current _config.py? Thanks a lot for all your work on the new schema based parser! Kind regards Thomas

6 18

Shadowserver parser: Bad mapping for malware events
by Thomas Hungenberg 31 Jan '24

31 Jan '24

Hi all, the parsers for malware events provided by different sources usually store the malware name in malware.name and classification.identifier is left blank (or set to the feed's name). When using the malware name mapping, a harmonized malware name is subsequently written to classification.identifier. So finally you have the original name in malware.name and the harmonized name in classification.identifier. Formerly (in the version we initially provided), the Shadowserver parser also stored the malware name in malware.name, see e.g. <https://github.com/certtools/intelmq/blob/c61ff2fd4232d6937f3815377b75f682a…> line 387 However, for some time the Shadowserver parser now writes the malware name ("infection") to classification.identifier and "family" to malware.name instead. This is bad for several reasons: - it is not consistent with parsers for other malware feeds - it breaks deduplicators matching on malware.name - the malware name mapping overwrites classification.identifier with the value of "family" (which often is empty) Here is a patch (for the version included with IntelMQ 3.2.1) to fix this problem and make malware events parsed by the Shadowserver parser consistent with other parsers again: =============================================== diff --git a/_config.py.orig b/_config.py index bea3d0c..431bcb9 100644 --- a/_config.py.orig +++ b/_config.py @@ -867,10 +867,9 @@ event_sinkhole = { ('source.port', 'src_port', convert_int), ], 'optional_fields': [ - ('classification.identifier', 'infection', validate_to_none), - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('source.asn', 'src_asn', invalidate_zero), ('source.geolocation.cc', 'src_geo'), @@ -899,6 +898,7 @@ event_sinkhole = { 'constant_fields': { 'classification.taxonomy': 'malicious-code', 'classification.type': 'infected-system', + 'classification.identifier': 'sinkhole-events', }, } @@ -944,10 +944,9 @@ event_sinkhole_http = { ('source.port', 'src_port', convert_int), ], 'optional_fields': [ - ('classification.identifier', 'tag'), - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('source.asn', 'src_asn', invalidate_zero), ('source.geolocation.cc', 'src_geo'), @@ -982,6 +981,7 @@ event_sinkhole_http = { 'constant_fields': { 'classification.taxonomy': 'malicious-code', 'classification.type': 'infected-system', + 'classification.identifier': 'sinkhole-http-events', 'protocol.application': 'http', }, } @@ -992,9 +992,9 @@ event_sinkhole_http_referer = { ('time.source', 'timestamp', add_UTC_to_timestamp), ], 'optional_fields': [ - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('extra.', 'http_referer_ip', validate_ip), ('extra.', 'http_referer_port', convert_int), =============================================== Kind regards Thomas

4 11

Re: [IntelMQ-dev] [IntelMQ-users] IntelMQ & Kubernetes
by Bernhard Reiter 18 Jan '24

18 Jan '24

Hello Jessica, Am Mittwoch 17 Januar 2024 09:47:25 schrieb Jessica Schumacher via IntelMQ-users: > Now we want to take the “next” step to Kubernetes. > Does anyone have experience with running IntelMQ on Kubernetes? what Kubernetes implementation or components are you planning to use? E.g. k3s or something else? What is the problem you want to solve with this step? (I am seriously interested, because my experience with Kubernetes is very limited, so I want to understand where it actually helps. So far I've seen it has an advantage for large, multi-machine scaled application settings which does not entirely fit my mental model of IntelMQ. I expect it to be possible like you can put any application in such a setting, but it is worth the effort?) Regards, Bernhard -- https://intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev January 2024