[IntelMQ-dev] Event rate limiting

[L. Aaron Kaplan]

Hi Mika,

in principle what you proposed (sleep) is safe - just be aware of potential overflowing (redis) queues.
Basically if you are out of RAM -> redis will not like it.
Same for disk space in case you configured redis to write on the disk.

Please take a look at https://github.com/certtools/intelmq/blob/develop/intelmq/bots/experts/wait/expert.py
It's already there :) just not that well documented I guess. But it's there.

Best,
Aaron.



> On 28.01.2021, at 07:32, Mika Silander <mika.silander at csc.fi> wrote:
> 
> Hi,
> 
> Maybe this is answered somewhere already but I wasn't able to find any hints and I've just recently started developing bots so here's one for the list:
> 
> I'd need to limit the rate at which events are processed to avoid swamping a back end system with requests. Big bursts of events should remain in the inqueue of such a rate limiting bot, and then be forwarded at a configurable rate e.g. X events/minute. Afaik, by default, bots attempt to process all the events in their inqueue without delay. How should one go about to create such a bot? Is it safe to put the bot process (=operating system process) to sleep for N seconds from within the bot's process method and then proceed, can we do busy-looping (not nice!), or, some other solution?
> 
> Br, Mika
> 
> 
> _______________________________________________
> IntelMQ-dev mailing list
> IntelMQ-dev at lists.cert.at
> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210128/1526d2ab/attachment.sig>