[IntelMQ-dev] Alerting on repeated connection failures in a bot

[Sebix]

Dear Mika,

On 3/23/22 1:27 PM, Mika Silander wrote:
>  Calling out to the list again before trying to reinvent a wheel. We have a collector that tries to connect to the source of security events, but the source service is (temporarily) inaccessible. IntelMQ restarts the bot, an error message and the connection failure exception gets output to the log, and after a while, the bot gets restarted. For us it is acceptable there's a few failures in a configurable time frame, but if the situation prevails, we'd like to be alerted and if possible, prevent the bot from restarting.
>
>  There are bot configuration parameters to stop a bot in case the processing of events fails repeatedly (error_procedure=stop, error_max_retries=suitable_number as in https://intelmq.readthedocs.io/en/maintenance/user/configuration-management.html#id14), but after some experimenting it appears they are not applicable here since we never reach the stage of processing events.

If the fetch fails, an exception in thrown in process(). That enters the
error handling in start() as described in the documentation. But
collectors are a bit different to parsers, experts and outputs, as they
don't have an incoming message as trigger, but operation on their own,
e.g. rate limiting.

Reading the code, I think your bot should stop, as error_on_message is
active in this case, which increases the error counter. When the counter
is hit, it should stop if error_procedure == "stop".

>  Any ideas how to address this issue? We'd prefer not to touch the bot's source code.

IntelMQ by itself does not notify you of any failures. I have used the
logcheck rules to monitor IntelMQ's logs:
https://github.com/certtools/intelmq/tree/develop/contrib/logcheck While
that's not a perfect solution, it is super easy. For anything more
advanced, an integration into a monitoring would be feasible.

kind regards
Sebastian

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20220325/957468bb/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20220325/957468bb/attachment.sig>