[Intelmq-dev] Proposal (Request For Comments) - IntelMQ with Run Modes & Process Management

[Sascha Wilde]

Bernhard Reiter <bernhard at intevation.de> writes:
> Am Freitag 21 April 2017 12:26:12 schrieb Sebastian Wagner:
Just my 2¢ on some specific point coming up:

[...]
> If we want intelmq simple, my strong recommendation is:
> a) implement (and thus support) only one process management solution.
>    So if a proposal including systemd is considered the leading solution 
>    after the discussion, we should implement it and remove other process
>    management approaches.

Even if systemd turns out to be a good choice, I'd vote against making
it a hard dependency.  The reasons are very much the same Bernhard
stated him self in an earlier post: it would make IntelMQ a Linux only
product, which would be a shame given the overall open and portable
design of it.

[...]
>> Flow control is definitely an issue and a big topic we should discuss in
>> depth. We (certat) do not need flow control currently but maybe you do?
>
> What I mean by flow control is that we take the relations between the bots 
> into account and implement strategies and tactics based on intelmq specific 
> information. In Navtej's friday post you can see how he makes use of this 
> information and proposes improved solutions to steer the flow within
> intelmq. To me it feels like an intelmq process manager can do this much 
> better, because it already know how the pipes are wired together.
> 
> Sooner or later I guess intelmq will need this kind of "flow control" to be 
> able to fulfill its promise of providing a fast and fully automatable system.
> So it may become interesting to you at certat as well. :)

This might or might not be true, currently the problems we are observing
are quite fundamental and don't need overly clever solutions.  I'd like
to point to the proposal Bernard Herzog made in issue 709 last year:
https://github.com/certtools/intelmq/issues/709
it outlines a rather simple solution to much of the resource problems,
and demonstrates how to build solutions that don't depend on an higher
level service, with in depth knowledge of the bots interactions.

[...]
>> Does "reload" more than "restart"? AFAIU, they are performing the same
>> checks. The only difference is, that restart stops/starts the running
>> continuous bots, and reload sends sighup to those.
>
> If it does not do more, get rid of it. (I thought it aims for doing more, but 
> then bots would need to be prepared to flush some of their datastructure 
> while running. It is much simpler for bot writers to just write for stop and 
> start.)

Ack.

sascha
-- 
Sascha Wilde                                  OpenPGP key: BB2185144BB86568
http://www.intevation.de/                  http://www.intevation.de/~wilde/
Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998
Geschäftsführer:   Frank Koormann,  Bernhard Reiter,  Dr. Jan-Oliver Wagner