Hello,
I noticed that our logs for bots based on the Shadowserver parser grew VERY large.
Reason for this is that with recent versions of the parser, a WARNING is logged for every _optional_ key not found in the feed data. As the optional key "sic" is no longer included in any feed, at least one WARNING is logged for each and every event processed by Shadowserver parsers. :-/ This looks like:
2024-01-16 06:41:55,551 - shadowserver-parser-xyz - WARNING - Optional key 'sic' not found in feed 'XYZ'. Possible change in data format or misconfiguration.
A quick fix is to disable logging of the WARNINGs in .../intelmq/bots/parsers/shadowserver/parser.py.
diff --git a/parser.py.orig b/parser.py index 70ba3b4..2ad29ac 100644 --- a/parser.py.orig +++ b/parser.py @@ -140,8 +140,8 @@ class ShadowserverParserBot(ParserBot): intelmqkey, shadowkey = item[:2] if shadowkey not in fields: if shadowkey not in row: # key does not exist in data (not even in the header) - self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data' - ' format or misconfiguration.'.format(shadowkey, self.feedname)) + # self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data' + # ' format or misconfiguration.'.format(shadowkey, self.feedname)) continue else: # key is used twice fields.append(shadowkey)
Or replace "self.logger.warning" by "self.logger.debug" if not running the bot in debug mode.
Kind regards Thomas
Hi Thomas,
nice to see you use the newest parser. This is an important observation, but I'd suggest that it has done a right job gaining your attention. Could you verify, that your system is updating ShadowServer dynamic schema?
This field has been removed in November: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl... In our production environment, we do not have any warnings.
You can ensure updating schema by configuring `auto_update` in bots, or a cron job (this is how we do it). See: https://docs.intelmq.org/latest/user/bots/#shadowserver
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 1/24/24 09:21, Thomas Hungenberg via IntelMQ-users wrote:
Hello,
I noticed that our logs for bots based on the Shadowserver parser grew VERY large.
Reason for this is that with recent versions of the parser, a WARNING is logged for every _optional_ key not found in the feed data. As the optional key "sic" is no longer included in any feed, at least one WARNING is logged for each and every event processed by Shadowserver parsers. :-/ This looks like:
2024-01-16 06:41:55,551 - shadowserver-parser-xyz - WARNING - Optional key 'sic' not found in feed 'XYZ'. Possible change in data format or misconfiguration.
A quick fix is to disable logging of the WARNINGs in .../intelmq/bots/parsers/shadowserver/parser.py.
diff --git a/parser.py.orig b/parser.py index 70ba3b4..2ad29ac 100644 --- a/parser.py.orig +++ b/parser.py @@ -140,8 +140,8 @@ class ShadowserverParserBot(ParserBot): intelmqkey, shadowkey = item[:2] if shadowkey not in fields: if shadowkey not in row: # key does not exist in data (not even in the header) - self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data' - ' format or misconfiguration.'.format(shadowkey, self.feedname)) + # self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data' + # ' format or misconfiguration.'.format(shadowkey, self.feedname)) continue else: # key is used twice fields.append(shadowkey)
Or replace "self.logger.warning" by "self.logger.debug" if not running the bot in debug mode.
Kind regards Thomas
Hi Kamil,
we are not using the newest parser with the dynamic schema but the static parser included with IntelMQ 3.2.1.
Kind regards Thomas
On 24.01.24 09:34, Kamil Mankowski via IntelMQ-users wrote:
Hi Thomas,
nice to see you use the newest parser. This is an important observation, but I'd suggest that it has done a right job gaining your attention. Could you verify, that your system is updating ShadowServer dynamic schema?
This field has been removed in November: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl... In our production environment, we do not have any warnings.
You can ensure updating schema by configuring `auto_update` in bots, or a cron job (this is how we do it). See: https://docs.intelmq.org/latest/user/bots/#shadowserver
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 1/24/24 09:21, Thomas Hungenberg via IntelMQ-users wrote:
Hello,
I noticed that our logs for bots based on the Shadowserver parser grew VERY large.
Reason for this is that with recent versions of the parser, a WARNING is logged for every _optional_ key not found in the feed data. As the optional key "sic" is no longer included in any feed, at least one WARNING is logged for each and every event processed by Shadowserver parsers. :-/ This looks like:
2024-01-16 06:41:55,551 - shadowserver-parser-xyz - WARNING - Optional key 'sic' not found in feed 'XYZ'. Possible change in data format or misconfiguration.
A quick fix is to disable logging of the WARNINGs in .../intelmq/bots/parsers/shadowserver/parser.py.
diff --git a/parser.py.orig b/parser.py index 70ba3b4..2ad29ac 100644 --- a/parser.py.orig +++ b/parser.py @@ -140,8 +140,8 @@ class ShadowserverParserBot(ParserBot): intelmqkey, shadowkey = item[:2] if shadowkey not in fields: if shadowkey not in row: # key does not exist in data (not even in the header) - self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data' - ' format or misconfiguration.'.format(shadowkey, self.feedname)) + # self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data' + # ' format or misconfiguration.'.format(shadowkey, self.feedname)) continue else: # key is used twice fields.append(shadowkey)
Or replace "self.logger.warning" by "self.logger.debug" if not running the bot in debug mode.
Kind regards Thomas
Ah, I've missed that the same log is also in the previous version. Thanks for the correction, this is indeed the best temporary fix until new release is done.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 1/24/24 09:47, Thomas Hungenberg via IntelMQ-users wrote:
Hi Kamil,
we are not using the newest parser with the dynamic schema but the static parser included with IntelMQ 3.2.1.
Kind regards Thomas
On 24.01.24 09:34, Kamil Mankowski via IntelMQ-users wrote:
Hi Thomas,
nice to see you use the newest parser. This is an important observation, but I'd suggest that it has done a right job gaining your attention. Could you verify, that your system is updating ShadowServer dynamic schema?
This field has been removed in November: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl... In our production environment, we do not have any warnings.
You can ensure updating schema by configuring `auto_update` in bots, or a cron job (this is how we do it). See: https://docs.intelmq.org/latest/user/bots/#shadowserver
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 1/24/24 09:21, Thomas Hungenberg via IntelMQ-users wrote:
Hello,
I noticed that our logs for bots based on the Shadowserver parser grew VERY large.
Reason for this is that with recent versions of the parser, a WARNING is logged for every _optional_ key not found in the feed data. As the optional key "sic" is no longer included in any feed, at least one WARNING is logged for each and every event processed by Shadowserver parsers. :-/ This looks like:
2024-01-16 06:41:55,551 - shadowserver-parser-xyz - WARNING - Optional key 'sic' not found in feed 'XYZ'. Possible change in data format or misconfiguration.
A quick fix is to disable logging of the WARNINGs in .../intelmq/bots/parsers/shadowserver/parser.py.
diff --git a/parser.py.orig b/parser.py index 70ba3b4..2ad29ac 100644 --- a/parser.py.orig +++ b/parser.py @@ -140,8 +140,8 @@ class ShadowserverParserBot(ParserBot): intelmqkey, shadowkey = item[:2] if shadowkey not in fields: if shadowkey not in row: # key does not exist in data (not even in the header) - self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data' - ' format or misconfiguration.'.format(shadowkey, self.feedname)) + # self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data' + # ' format or misconfiguration.'.format(shadowkey, self.feedname)) continue else: # key is used twice fields.append(shadowkey)
Or replace "self.logger.warning" by "self.logger.debug" if not running the bot in debug mode.
Kind regards Thomas