Hi again,
Afaik, if handling an event fails in a bot, the default behaviour of a bot is to sleep 15 seconds and then retry processing. If the retry fails, the bot dumps the event and picks the next event from the inqueue. We have a bot where it would be desirable to change this default behaviour so that the dump is done immediately on the first failure. Is there a way to configure a single bot to behave differently from other bots as described? Or will a change in configuration affect the entire bot net?
If there's no easy way configuring, I guess technically I could implement exceptions to be thrown in those situations where quick dumping is desired and then call the _dump_message(?) method. However, I would prefer to modify as few bots of the official distribution as possible.
Comments, pointers to docs, sources or the like are most welcome.
Br, Mika
Hi,
I'm not aware of any way to just dump the message after the first issue. You could implement this feature - this would be just change in the intelmq.lib.bot. I'd suggest implementing a config option how the library should behave, and then keep the current flow as default.
However, how do you need retries after dumping message? If not, then the 'error_max_retries' should be set to 1 for the bot you require. But it would disable the retries for given bot.
Best regards, Kamil Mankowski CERT.at GmbH www.cert.at
On 5/15/23 11:40, Mika Silander wrote:
Hi again,
Afaik, if handling an event fails in a bot, the default behaviour of a bot is to sleep 15 seconds and then retry processing. If the retry fails, the bot dumps the event and picks the next event from the inqueue. We have a bot where it would be desirable to change this default behaviour so that the dump is done immediately on the first failure. Is there a way to configure a single bot to behave differently from other bots as described? Or will a change in configuration affect the entire bot net?
If there's no easy way configuring, I guess technically I could implement exceptions to be thrown in those situations where quick dumping is desired and then call the _dump_message(?) method. However, I would prefer to modify as few bots of the official distribution as possible.
Comments, pointers to docs, sources or the like are most welcome.
Br, Mika _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Hi Mika,
well I see a way to implement this so that existing bots don't get modified or impacted:
* change it in the core so that *only* if a specific config flag is present ("dont_retry_and_dump_quickly": True or similar), then the bot will behave as you write * handle exceptions and if the flag is set, otherwise continue as normal.
--> effect for existing bots is zero.
Best, a.
On 15.05.2023, at 11:40, Mika Silander mika.silander@csc.fi wrote:
Hi again,
Afaik, if handling an event fails in a bot, the default behaviour of a bot is to sleep 15 seconds and then retry processing. If the retry fails, the bot dumps the event and picks the next event from the inqueue. We have a bot where it would be desirable to change this default behaviour so that the dump is done immediately on the first failure. Is there a way to configure a single bot to behave differently from other bots as described? Or will a change in configuration affect the entire bot net?
If there's no easy way configuring, I guess technically I could implement exceptions to be thrown in those situations where quick dumping is desired and then call the _dump_message(?) method. However, I would prefer to modify as few bots of the official distribution as possible.
Comments, pointers to docs, sources or the like are most welcome.
Br, Mika _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
And Kamil already answered it better ... :)
sorry for the noise.
Best, a.
On 16.05.2023, at 15:27, Kamil Mankowski via IntelMQ-dev intelmq-dev@lists.cert.at wrote:
Signed PGP part Hi,
I'm not aware of any way to just dump the message after the first issue. You could implement this feature - this would be just change in the intelmq.lib.bot. I'd suggest implementing a config option how the library should behave, and then keep the current flow as default.
However, how do you need retries after dumping message? If not, then the 'error_max_retries' should be set to 1 for the bot you require. But it would disable the retries for given bot.
Best regards, Kamil Mankowski CERT.at GmbH www.cert.at
On 5/15/23 11:40, Mika Silander wrote:
Hi again, Afaik, if handling an event fails in a bot, the default behaviour of a bot is to sleep 15 seconds and then retry processing. If the retry fails, the bot dumps the event and picks the next event from the inqueue. We have a bot where it would be desirable to change this default behaviour so that the dump is done immediately on the first failure. Is there a way to configure a single bot to behave differently from other bots as described? Or will a change in configuration affect the entire bot net? If there's no easy way configuring, I guess technically I could implement exceptions to be thrown in those situations where quick dumping is desired and then call the _dump_message(?) method. However, I would prefer to modify as few bots of the official distribution as possible. Comments, pointers to docs, sources or the like are most welcome. Br, Mika _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
<OpenPGP_0x3E911155943C0414.asc>
Hi Kamil, Aaron, all,
Thanks for your comments. Of all suggestions, I'll try setting the error_max_retries first. At the moment I don't know the future proportion of events needing immediate dumping vs. those that merit a retry. However, if it turns out dumping is the most common case, setting the error_max_retries will suffice.
The background motivation here is that those cases dumped immediately are in our setup a sign of missing bot configuration information or a flawed bot configuration. Once the configuration is fixed, we may push the dumped events again through the bot using intelmqdump, and the bot should be able to process the events without problems. I hope Kamil this answers your question below.
Br, Mika
----- Original Message ----- From: "Kamil Mankowski via IntelMQ-dev" intelmq-dev@lists.cert.at To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Tuesday, 16 May, 2023 16:27:51 Subject: Re: [IntelMQ-dev] Speed dumping events in bots
Hi,
I'm not aware of any way to just dump the message after the first issue. You could implement this feature - this would be just change in the intelmq.lib.bot. I'd suggest implementing a config option how the library should behave, and then keep the current flow as default.
However, how do you need retries after dumping message? If not, then the 'error_max_retries' should be set to 1 for the bot you require. But it would disable the retries for given bot.
Best regards, Kamil Mankowski CERT.at GmbH www.cert.at
On 5/15/23 11:40, Mika Silander wrote:
Hi again,
Afaik, if handling an event fails in a bot, the default behaviour of a bot is to sleep 15 seconds and then retry processing. If the retry fails, the bot dumps the event and picks the next event from the inqueue. We have a bot where it would be desirable to change this default behaviour so that the dump is done immediately on the first failure. Is there a way to configure a single bot to behave differently from other bots as described? Or will a change in configuration affect the entire bot net?
If there's no easy way configuring, I guess technically I could implement exceptions to be thrown in those situations where quick dumping is desired and then call the _dump_message(?) method. However, I would prefer to modify as few bots of the official distribution as possible.
Comments, pointers to docs, sources or the like are most welcome.
Br, Mika _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
_______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Great, thanks for the use case explanation
Best regards, Kamil Mankowski CERT.at GmbH www.cert.at
On 5/17/23 08:25, Mika Silander wrote:
Hi Kamil, Aaron, all,
Thanks for your comments. Of all suggestions, I'll try setting the error_max_retries first. At the moment I don't know the future proportion of events needing immediate dumping vs. those that merit a retry. However, if it turns out dumping is the most common case, setting the error_max_retries will suffice.
The background motivation here is that those cases dumped immediately are in our setup a sign of missing bot configuration information or a flawed bot configuration. Once the configuration is fixed, we may push the dumped events again through the bot using intelmqdump, and the bot should be able to process the events without problems. I hope Kamil this answers your question below.
Br, Mika
----- Original Message ----- From: "Kamil Mankowski via IntelMQ-dev" intelmq-dev@lists.cert.at To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Tuesday, 16 May, 2023 16:27:51 Subject: Re: [IntelMQ-dev] Speed dumping events in bots
Hi,
I'm not aware of any way to just dump the message after the first issue. You could implement this feature - this would be just change in the intelmq.lib.bot. I'd suggest implementing a config option how the library should behave, and then keep the current flow as default.
However, how do you need retries after dumping message? If not, then the 'error_max_retries' should be set to 1 for the bot you require. But it would disable the retries for given bot.
Best regards, Kamil Mankowski CERT.at GmbH www.cert.at
On 5/15/23 11:40, Mika Silander wrote:
Hi again,
Afaik, if handling an event fails in a bot, the default behaviour of a bot is to sleep 15 seconds and then retry processing. If the retry fails, the bot dumps the event and picks the next event from the inqueue. We have a bot where it would be desirable to change this default behaviour so that the dump is done immediately on the first failure. Is there a way to configure a single bot to behave differently from other bots as described? Or will a change in configuration affect the entire bot net?
If there's no easy way configuring, I guess technically I could implement exceptions to be thrown in those situations where quick dumping is desired and then call the _dump_message(?) method. However, I would prefer to modify as few bots of the official distribution as possible.
Comments, pointers to docs, sources or the like are most welcome.
Br, Mika _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/