[IntelMQ-users] Shadowserver parser flooding logs

[Kamil Mankowski]

Ah, I've missed that the same log is also in the previous version. 
Thanks for the correction, this is indeed the best temporary fix until 
new release is done.

Best regards

// Kamil Mańkowski <mankowski at cert.at> - T: +43 676 898 298 7204
// CERT Austria - https://www.cert.at/
// CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 1/24/24 09:47, Thomas Hungenberg via IntelMQ-users wrote:
> Hi Kamil,
> 
> we are not using the newest parser with the dynamic schema but the 
> static parser included with IntelMQ 3.2.1.
> 
> Kind regards
> Thomas
> 
> On 24.01.24 09:34, Kamil Mankowski via IntelMQ-users wrote:
>> Hi Thomas,
>>
>> nice to see you use the newest parser. This is an important 
>> observation, but I'd suggest that it has done a right job gaining your 
>> attention. Could you verify, that your system is updating ShadowServer 
>> dynamic schema?
>>
>> This field has been removed in November: 
>> https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/completed-changes.md In our production environment, we do not have any warnings.
>>
>> You can ensure updating schema by configuring `auto_update` in bots, 
>> or a cron job (this is how we do it). See: 
>> https://docs.intelmq.org/latest/user/bots/#shadowserver
>>
>> Best regards
>>
>> // Kamil Mańkowski <mankowski at cert.at> - T: +43 676 898 298 7204
>> // CERT Austria - https://www.cert.at/
>> // CERT.at GmbH, FB-Nr. 561772k, HG Wien
>>
>> On 1/24/24 09:21, Thomas Hungenberg via IntelMQ-users wrote:
>>> Hello,
>>>
>>> I noticed that our logs for bots based on the Shadowserver parser 
>>> grew VERY large.
>>>
>>> Reason for this is that with recent versions of the parser, a WARNING 
>>> is logged
>>> for every _optional_ key not found in the feed data.
>>> As the optional key "sic" is no longer included in any feed, at least 
>>> one WARNING
>>> is logged for each and every event processed by Shadowserver parsers. 
>>> :-/
>>> This looks like:
>>>
>>> 2024-01-16 06:41:55,551 - shadowserver-parser-xyz - WARNING - 
>>> Optional key 'sic' not found in feed 'XYZ'.
>>> Possible change in data format or misconfiguration.
>>>
>>> A quick fix is to disable logging of the WARNINGs in 
>>> .../intelmq/bots/parsers/shadowserver/parser.py.
>>>
>>> diff --git a/parser.py.orig b/parser.py
>>> index 70ba3b4..2ad29ac 100644
>>> --- a/parser.py.orig
>>> +++ b/parser.py
>>> @@ -140,8 +140,8 @@ class ShadowserverParserBot(ParserBot):
>>>               intelmqkey, shadowkey = item[:2]
>>>               if shadowkey not in fields:
>>>                   if shadowkey not in row:  # key does not exist in 
>>> data (not even in the header)
>>> -                    self.logger.warning('Optional key {!r} not found 
>>> in feed {!r}. Possible change in data'
>>> -                                        ' format or 
>>> misconfiguration.'.format(shadowkey, self.feedname))
>>> +                    # self.logger.warning('Optional key {!r} not 
>>> found in feed {!r}. Possible change in data'
>>> +                    #                     ' format or 
>>> misconfiguration.'.format(shadowkey, self.feedname))
>>>                       continue
>>>                   else:  # key is used twice
>>>                       fields.append(shadowkey)
>>>
>>>
>>> Or replace "self.logger.warning" by "self.logger.debug" if not 
>>> running the bot in debug mode.
>>>
>>>
>>> Kind regards
>>> Thomas
>>>
>>
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20240124/34c71a28/attachment.sig>