[IntelMQ-dev] Event rate limiting

[Mika Silander]

Hi Aaron,

 Thanks for a prompt reply. I feared I had missed something and it seems I did ... I've skimmed through the available bots several times but the wait bot had escaped the radar :-(.
 Speaking of documentation: as a new developer having good documentation is crucial. Do you accept contributions and if yes, how? Git pushes? I'm not thinking of any major undertakings but rather occasional contributions if I come across something I feel could be improved, needs clarifying etc etc.

Cheers, Mika

----- Original Message -----
From: "L. Aaron Kaplan" <aaron at lo-res.org>
To: "Mika Silander" <mika.silander at csc.fi>
Cc: "intelmq-dev" <intelmq-dev at lists.cert.at>
Sent: Thursday, 28 January, 2021 09:33:41
Subject: Re: [IntelMQ-dev] Event rate limiting

Hi Mika,

in principle what you proposed (sleep) is safe - just be aware of potential overflowing (redis) queues.
Basically if you are out of RAM -> redis will not like it.
Same for disk space in case you configured redis to write on the disk.

Please take a look at https://github.com/certtools/intelmq/blob/develop/intelmq/bots/experts/wait/expert.py
It's already there :) just not that well documented I guess. But it's there.

Best,
Aaron.



> On 28.01.2021, at 07:32, Mika Silander <mika.silander at csc.fi> wrote:
> 
> Hi,
> 
> Maybe this is answered somewhere already but I wasn't able to find any hints and I've just recently started developing bots so here's one for the list:
> 
> I'd need to limit the rate at which events are processed to avoid swamping a back end system with requests. Big bursts of events should remain in the inqueue of such a rate limiting bot, and then be forwarded at a configurable rate e.g. X events/minute. Afaik, by default, bots attempt to process all the events in their inqueue without delay. How should one go about to create such a bot? Is it safe to put the bot process (=operating system process) to sleep for N seconds from within the bot's process method and then proceed, can we do busy-looping (not nice!), or, some other solution?
> 
> Br, Mika
> 
> 
> _______________________________________________
> IntelMQ-dev mailing list
> IntelMQ-dev at lists.cert.at
> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev