[Intelmq-dev] Run modes + Systemd + Crontab

[Sebastian Wagner]

Addendum:

Some of you are using supervisord to manage intelmq bots. As discussed
at the IHAP meeting in December, this can be implemented equivalently.
Although, as far as I found out, it does not support scheduled jobs by
itself, so cron has to be used for this (as opposed to systemd which has
timers).

Are there any other process managers which are in use for intelmq (or
planned to be used)?

The pull request "Implement oneshots, scheduled via cron"
https://github.com/certtools/intelmq/pull/846 implements one part of
this proposal (it's not a PoC).

Sebastian


On 01/30/2017 12:12 PM, Tomás Lima wrote:
> Folks,
>
> We have been thinking about things to improve IntelMQ in order to have
> it really suitable for production... and now we want to raise the
> discussion about the following ideas or other related ideas that you
> might think of:
>
> - Run modes: stream (always running, e.g. shadowserver-parser bot) or
> scheduled (run once every day at 10:00AM, e.g.
> shadowserver-chargen-collector bot)
> - Systemd: take care of bots which are running permanently. Use
> systemd to manage bots in stream run mode that cannot stop and also,
> do a better job with managing which bots processes which are really
> running, not using the current implementation with pidfiles.
> - Crontab: manage bots that need to run once, e.g. at midnight every
> day. Use crontab to manage bots in scheduled run mode to just run once
> on a pre-defined scheduled time.
>
> Why these ideas came up?
> Mainly because of two things:
> 1. Run modes are needed due the different purposes of a bot which is
> collecting a real-time feed and a bot that is collecting shadowserver
> reports once in a day.
> 2. Systemd and Crontab are two well-know tools that we all know about
> it and IntelMQ seems to need them since we have seen some problems
> with managing bots processes (running or not) and report automatically
> failures by email, just to give some examples.
>
>
> Until now, we don't have any of these features implemented, we only
> have developed some "rubbish" code in order to do some
> Proof-of-concept but nothing serious. So, please guys, share your
> thoughts about these ideas or others related in order to define, as a
> community, a strategy for the next month.
>
> Thank you.
>
> Cheers!

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 50564167201 
// CERT Austria - http://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20170130/f7dadb43/attachment.sig>