[IntelMQ-users] Shadowserver parser flooding logs

[Thomas Hungenberg]

Hi Kamil,

we are not using the newest parser with the dynamic schema but the static parser included with IntelMQ 3.2.1.

Kind regards
Thomas

On 24.01.24 09:34, Kamil Mankowski via IntelMQ-users wrote:
> Hi Thomas,
> 
> nice to see you use the newest parser. This is an important observation, but I'd suggest that it has done a right job gaining your attention. Could 
> you verify, that your system is updating ShadowServer dynamic schema?
> 
> This field has been removed in November: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/completed-changes.md In our production 
> environment, we do not have any warnings.
> 
> You can ensure updating schema by configuring `auto_update` in bots, or a cron job (this is how we do it). See: 
> https://docs.intelmq.org/latest/user/bots/#shadowserver
> 
> Best regards
> 
> // Kamil Mańkowski <mankowski at cert.at> - T: +43 676 898 298 7204
> // CERT Austria - https://www.cert.at/
> // CERT.at GmbH, FB-Nr. 561772k, HG Wien
> 
> On 1/24/24 09:21, Thomas Hungenberg via IntelMQ-users wrote:
>> Hello,
>>
>> I noticed that our logs for bots based on the Shadowserver parser grew VERY large.
>>
>> Reason for this is that with recent versions of the parser, a WARNING is logged
>> for every _optional_ key not found in the feed data.
>> As the optional key "sic" is no longer included in any feed, at least one WARNING
>> is logged for each and every event processed by Shadowserver parsers. :-/
>> This looks like:
>>
>> 2024-01-16 06:41:55,551 - shadowserver-parser-xyz - WARNING - Optional key 'sic' not found in feed 'XYZ'.
>> Possible change in data format or misconfiguration.
>>
>> A quick fix is to disable logging of the WARNINGs in .../intelmq/bots/parsers/shadowserver/parser.py.
>>
>> diff --git a/parser.py.orig b/parser.py
>> index 70ba3b4..2ad29ac 100644
>> --- a/parser.py.orig
>> +++ b/parser.py
>> @@ -140,8 +140,8 @@ class ShadowserverParserBot(ParserBot):
>>               intelmqkey, shadowkey = item[:2]
>>               if shadowkey not in fields:
>>                   if shadowkey not in row:  # key does not exist in data (not even in the header)
>> -                    self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data'
>> -                                        ' format or misconfiguration.'.format(shadowkey, self.feedname))
>> +                    # self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data'
>> +                    #                     ' format or misconfiguration.'.format(shadowkey, self.feedname))
>>                       continue
>>                   else:  # key is used twice
>>                       fields.append(shadowkey)
>>
>>
>> Or replace "self.logger.warning" by "self.logger.debug" if not running the bot in debug mode.
>>
>>
>> Kind regards
>> Thomas
>>
> 

-- 
- Thomas

CERT-Bund Incident Response & Malware Analysis Team