IntelMQ-users January 2024

intelmq-users@lists.cert.at

4 participants
3 discussions

Shadowserver parser: Bad mapping for malware events
by Thomas Hungenberg 24 Jan '24

24 Jan '24

Hi all, the parsers for malware events provided by different sources usually store the malware name in malware.name and classification.identifier is left blank (or set to the feed's name). When using the malware name mapping, a harmonized malware name is subsequently written to classification.identifier. So finally you have the original name in malware.name and the harmonized name in classification.identifier. Formerly (in the version we initially provided), the Shadowserver parser also stored the malware name in malware.name, see e.g. <https://github.com/certtools/intelmq/blob/c61ff2fd4232d6937f3815377b75f682a…> line 387 However, for some time the Shadowserver parser now writes the malware name ("infection") to classification.identifier and "family" to malware.name instead. This is bad for several reasons: - it is not consistent with parsers for other malware feeds - it breaks deduplicators matching on malware.name - the malware name mapping overwrites classification.identifier with the value of "family" (which often is empty) Here is a patch (for the version included with IntelMQ 3.2.1) to fix this problem and make malware events parsed by the Shadowserver parser consistent with other parsers again: =============================================== diff --git a/_config.py.orig b/_config.py index bea3d0c..431bcb9 100644 --- a/_config.py.orig +++ b/_config.py @@ -867,10 +867,9 @@ event_sinkhole = { ('source.port', 'src_port', convert_int), ], 'optional_fields': [ - ('classification.identifier', 'infection', validate_to_none), - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('source.asn', 'src_asn', invalidate_zero), ('source.geolocation.cc', 'src_geo'), @@ -899,6 +898,7 @@ event_sinkhole = { 'constant_fields': { 'classification.taxonomy': 'malicious-code', 'classification.type': 'infected-system', + 'classification.identifier': 'sinkhole-events', }, } @@ -944,10 +944,9 @@ event_sinkhole_http = { ('source.port', 'src_port', convert_int), ], 'optional_fields': [ - ('classification.identifier', 'tag'), - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('source.asn', 'src_asn', invalidate_zero), ('source.geolocation.cc', 'src_geo'), @@ -982,6 +981,7 @@ event_sinkhole_http = { 'constant_fields': { 'classification.taxonomy': 'malicious-code', 'classification.type': 'infected-system', + 'classification.identifier': 'sinkhole-http-events', 'protocol.application': 'http', }, } @@ -992,9 +992,9 @@ event_sinkhole_http_referer = { ('time.source', 'timestamp', add_UTC_to_timestamp), ], 'optional_fields': [ - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('extra.', 'http_referer_ip', validate_ip), ('extra.', 'http_referer_port', convert_int), =============================================== Kind regards Thomas

1 0

Shadowserver parser flooding logs
by Thomas Hungenberg 24 Jan '24

24 Jan '24

Hello, I noticed that our logs for bots based on the Shadowserver parser grew VERY large. Reason for this is that with recent versions of the parser, a WARNING is logged for every _optional_ key not found in the feed data. As the optional key "sic" is no longer included in any feed, at least one WARNING is logged for each and every event processed by Shadowserver parsers. :-/ This looks like: 2024-01-16 06:41:55,551 - shadowserver-parser-xyz - WARNING - Optional key 'sic' not found in feed 'XYZ'. Possible change in data format or misconfiguration. A quick fix is to disable logging of the WARNINGs in .../intelmq/bots/parsers/shadowserver/parser.py. diff --git a/parser.py.orig b/parser.py index 70ba3b4..2ad29ac 100644 --- a/parser.py.orig +++ b/parser.py @@ -140,8 +140,8 @@ class ShadowserverParserBot(ParserBot): intelmqkey, shadowkey = item[:2] if shadowkey not in fields: if shadowkey not in row: # key does not exist in data (not even in the header) - self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data' - ' format or misconfiguration.'.format(shadowkey, self.feedname)) + # self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data' + # ' format or misconfiguration.'.format(shadowkey, self.feedname)) continue else: # key is used twice fields.append(shadowkey) Or replace "self.logger.warning" by "self.logger.debug" if not running the bot in debug mode. Kind regards Thomas

2 3

IntelMQ & Kubernetes
by Jessica Schumacher 17 Jan '24

17 Jan '24

Hello all, we’ve been using IntelMQ for a long time at Switch and moved IntelMQ on Docker a few years ago. Now we want to take the “next” step to Kubernetes. Does anyone have experience with running IntelMQ on Kubernetes? At the moment we are just at the brainstorming phase and thinking about the design. Especially how we should split it up into different pots and redundancy. Any feedback about experiences or inputs are highly appreciated. Kind regards, Jessica — Switch Jessica Schumacher, Switch CERT Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland phone +41 44 268 15 15, direct +41 44 268 16 33 jessica.schumacher(a)switch.ch, http://www.switch.ch

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users January 2024