[IntelMQ-users] Shadowserver parser flooding logs

[Thomas Hungenberg]

Hello,

I noticed that our logs for bots based on the Shadowserver parser grew VERY large.

Reason for this is that with recent versions of the parser, a WARNING is logged
for every _optional_ key not found in the feed data.
As the optional key "sic" is no longer included in any feed, at least one WARNING
is logged for each and every event processed by Shadowserver parsers. :-/
This looks like:

2024-01-16 06:41:55,551 - shadowserver-parser-xyz - WARNING - Optional key 'sic' not found in feed 'XYZ'.
Possible change in data format or misconfiguration.

A quick fix is to disable logging of the WARNINGs in .../intelmq/bots/parsers/shadowserver/parser.py.

diff --git a/parser.py.orig b/parser.py
index 70ba3b4..2ad29ac 100644
--- a/parser.py.orig
+++ b/parser.py
@@ -140,8 +140,8 @@ class ShadowserverParserBot(ParserBot):
              intelmqkey, shadowkey = item[:2]
              if shadowkey not in fields:
                  if shadowkey not in row:  # key does not exist in data (not even in the header)
-                    self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data'
-                                        ' format or misconfiguration.'.format(shadowkey, self.feedname))
+                    # self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data'
+                    #                     ' format or misconfiguration.'.format(shadowkey, self.feedname))
                      continue
                  else:  # key is used twice
                      fields.append(shadowkey)


Or replace "self.logger.warning" by "self.logger.debug" if not running the bot in debug mode.


Kind regards
Thomas