[IntelMQ-dev] Event rate limiting

[L. Aaron Kaplan]

Hi Mika,


> On 28.01.2021, at 09:40, Mika Silander <mika.silander at csc.fi> wrote:
> 
> Hi Aaron,
> 
> Thanks for a prompt reply. I feared I had missed something and it seems I did ... I've skimmed through the available bots several times but the wait bot had escaped the radar :-(.
Np worries :) That's what this list is for.

> Speaking of documentation: as a new developer having good documentation is crucial. Do you accept contributions and if yes, how? Git pushes?

Sure! IntelMQ was always and will be a community project - and open source.
This was the thing me and Tomas Lima back then in the days when the project started agreed upon (based on the lessons of trying out a previous system).

So, yes, please ! By all means send GitHub pull requests for the documentation.
The project lives by and for the community. I know CERT.at is doing a lot of the heavy weight lifting right now, but essentially it is a community project. And that's super important for the long term survivability and adaption to user's and developer's needs.

> I'm not thinking of any major undertakings but rather occasional contributions if I come across something I feel could be improved, needs clarifying etc etc.
> 

Even if it is typo or grammar corrections or wording ;-) It's welcome! And thank you very much.

Cheers,
Aaron.

> Cheers, Mika
> 
> ----- Original Message -----
> From: "L. Aaron Kaplan" <aaron at lo-res.org>
> To: "Mika Silander" <mika.silander at csc.fi>
> Cc: "intelmq-dev" <intelmq-dev at lists.cert.at>
> Sent: Thursday, 28 January, 2021 09:33:41
> Subject: Re: [IntelMQ-dev] Event rate limiting
> 
> Hi Mika,
> 
> in principle what you proposed (sleep) is safe - just be aware of potential overflowing (redis) queues.
> Basically if you are out of RAM -> redis will not like it.
> Same for disk space in case you configured redis to write on the disk.
> 
> Please take a look at https://github.com/certtools/intelmq/blob/develop/intelmq/bots/experts/wait/expert.py
> It's already there :) just not that well documented I guess. But it's there.
> 
> Best,
> Aaron.
> 
> 
> 
>> On 28.01.2021, at 07:32, Mika Silander <mika.silander at csc.fi> wrote:
>> 
>> Hi,
>> 
>> Maybe this is answered somewhere already but I wasn't able to find any hints and I've just recently started developing bots so here's one for the list:
>> 
>> I'd need to limit the rate at which events are processed to avoid swamping a back end system with requests. Big bursts of events should remain in the inqueue of such a rate limiting bot, and then be forwarded at a configurable rate e.g. X events/minute. Afaik, by default, bots attempt to process all the events in their inqueue without delay. How should one go about to create such a bot? Is it safe to put the bot process (=operating system process) to sleep for N seconds from within the bot's process method and then proceed, can we do busy-looping (not nice!), or, some other solution?
>> 
>> Br, Mika
>> 
>> 
>> _______________________________________________
>> IntelMQ-dev mailing list
>> IntelMQ-dev at lists.cert.at
>> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210128/d5ad01c7/attachment.sig>