[IntelMQ-users] Shadowserver parser flooding logs

[Kamil Mankowski]

Hi Thomas,

nice to see you use the newest parser. This is an important observation, 
but I'd suggest that it has done a right job gaining your attention. 
Could you verify, that your system is updating ShadowServer dynamic schema?

This field has been removed in November: 
https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/completed-changes.md 
In our production environment, we do not have any warnings.

You can ensure updating schema by configuring `auto_update` in bots, or 
a cron job (this is how we do it). See: 
https://docs.intelmq.org/latest/user/bots/#shadowserver

Best regards

// Kamil Mańkowski <mankowski at cert.at> - T: +43 676 898 298 7204
// CERT Austria - https://www.cert.at/
// CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 1/24/24 09:21, Thomas Hungenberg via IntelMQ-users wrote:
> Hello,
> 
> I noticed that our logs for bots based on the Shadowserver parser grew 
> VERY large.
> 
> Reason for this is that with recent versions of the parser, a WARNING is 
> logged
> for every _optional_ key not found in the feed data.
> As the optional key "sic" is no longer included in any feed, at least 
> one WARNING
> is logged for each and every event processed by Shadowserver parsers. :-/
> This looks like:
> 
> 2024-01-16 06:41:55,551 - shadowserver-parser-xyz - WARNING - Optional 
> key 'sic' not found in feed 'XYZ'.
> Possible change in data format or misconfiguration.
> 
> A quick fix is to disable logging of the WARNINGs in 
> .../intelmq/bots/parsers/shadowserver/parser.py.
> 
> diff --git a/parser.py.orig b/parser.py
> index 70ba3b4..2ad29ac 100644
> --- a/parser.py.orig
> +++ b/parser.py
> @@ -140,8 +140,8 @@ class ShadowserverParserBot(ParserBot):
>               intelmqkey, shadowkey = item[:2]
>               if shadowkey not in fields:
>                   if shadowkey not in row:  # key does not exist in data 
> (not even in the header)
> -                    self.logger.warning('Optional key {!r} not found in 
> feed {!r}. Possible change in data'
> -                                        ' format or 
> misconfiguration.'.format(shadowkey, self.feedname))
> +                    # self.logger.warning('Optional key {!r} not found 
> in feed {!r}. Possible change in data'
> +                    #                     ' format or 
> misconfiguration.'.format(shadowkey, self.feedname))
>                       continue
>                   else:  # key is used twice
>                       fields.append(shadowkey)
> 
> 
> Or replace "self.logger.warning" by "self.logger.debug" if not running 
> the bot in debug mode.
> 
> 
> Kind regards
> Thomas
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20240124/673af661/attachment.sig>