[Intelmq-dev] Proposal (Request For Comments) - IntelMQ with Run Modes & Process Management

[Bernhard Reiter]

Dear Tomás, dear Intelmqers,

Am Donnerstag, 23. März 2017 20:00:05 schrieb Tomás Lima:
> Proposal:
> https://github.com/SYNchroACK/intelmq/blob/proposal/docs/proposal.md

thanks to you, Sebastian and Aaron for working out a proposal
and starting a discussion.

> Reasons why we are starting this discussion is because two main things (I
> guess):
>
> 1) There is a need to configure bots to only execute in a specific time,
> therefore, it seems that there is a requirement to configure a bot in
> different run modes, in this proposal: scheduled and continuous. (see
> proposal for more details).

I agree that there are periodic tasks to be done.
Examples include:
 * periodic renewal of support data, e.g. importing abuse data
   from ripe
 * fetching feeds after a specific time of the day

Proposing several run modes like you did (startup, scheduled, one-shot, 
continuous) is one way of solving this need. 

There are others that should be considered as well: An alternative would be: 
Each bot gets helping functions that allows it to be woken up on some events 
and event types include a sleep until an alarm goes off. The time of being 
alarmed can be set by the bot itself. This way only one run-mode is necessary 
as a bot can see if it still wants to configure something (like "on_boot") , 
run or be woken up later at a certain time.

> 2) IntelMQ is now being used by multiple teams and it requires stability
> during execution time, etc... it seems that there is a need for
> integrations with tools like systemd.

Again I agree with the requirement that intelmq shall run stably and that it 
increasingly need to. The reason is that usage is supposed to be more in 
production and a setup will be less likely to get care from administrators 
which happen to be intelmq-developer at the same time.

However again I'm unsure if systemd is a good solution.
If a solution is implemented I think it should be the only way to run intelmq, 
to reduce complexity and maintenance costs in the end. Using systemd for 
running bots thus means adding a hard dependency on systemd and thus 
restricting intelmq to be able to run on GNU/Linux-systems that come with 
systemd. It would exclude some operating systems like the 
netbsd/freebsd/openbsd group,  or even more exotic ones like Illumos. The 
restriction to GNU/Linux systems with systemd is something I could live with.

My second concern regarding systemd is more important: systemd is a general 
process manager, for intelmq we may need a process manager that implements 
tactics specific to abuse handling and intelmq's design. For example about 
the question: What shall be done if we get congestion in intelmq? Should 
starting of scheduled bots runs be queued up or dropped? Which ones are more 
important than others? My gut feeling is that we will need some sort of flow 
control and that systemd will be unable to manage that without additional 
stunts.

> Please, if you have time, read the proposal and write to the mailing-list
> your thoughts spitted by "What you like" and "What you don't like".

(Just doing so, sorry that it took a few days, I was time pressed to get the 
webfrontend to an IntelMQ setup we call intelmq-cb-mailgen released in a beta 
version and then had a few days off. So here we go:)

In the == intelmqctl principles:

 1)
The first two seems to collide:
"in the background" and "interactive" mode does not go together.
Overall it is okay to be able to start processes in the foreground to me, may 
it be for diagnostic purposes or interaction. I guess the first should should 
be rephrased to say "by default bots are started in the background"
or "unless a different option is given".

I'm less sure that everything shall be a "bot", though. (BTW: The 
phrasing "bot" still feels odd, I'm not sure what it really means in 
comparison to other daemons or processes.)

 2)
What does "provide the best log messages" mean?
Each diagnostic message added to the code will be in best faith anyway
and using the highest log-level in all circumstances seems undesireable.

 3)
Principle of checking configuration and trying to repair it:
If intelmqctl can repair/clean a configuration, why should it stop and ask for 
a rerun? It seems to have enough information to continue, so it could 
continue right away.

== more thoughts/questions

My suggestion is to come up with language that indicates if a bot is
"enabled" in a sense that it shall run according to its spec or "disabled"
and that "is running" or "is not running" means if it actually has the program 
pointer or not. Right now we have three states:
 * shall be autostarted on reboot or not (called "enabled")
 * shall be running always or woken up at certain times (called "started")
 * has active control about a program pointer (in one thread or process), not
   called anything yet.

The --oneshot flag seems to be there for diagnostic purposes, but shouldn't it 
be a run-mode then? Up to this point it seems we are having four run 
modes. :)
(This somehow does not match what I see under intelmqctl start 2.3+2.4 
where --oneshot is given to all starts. It would mean that schedules bots 
would only process one message??)

The "reload" action is unclear to me, why making things more complicated?
From my perspective doing more than "restart" would only be helpful if there 
are resources to be saved (startup costs) or data to be keep from being lost.
However intelmq shall never lose data anyway because of its message bus design
and I cannot imagine resources that are worth adding the complexity of a 
reload.

So after reading the proposal for the first time I believe that I have a hard 
time evaluating it because it does not explain clear enough what kind of 
problems it is going to solve. Such an explanation should include examples to 
be able to judge if we a solution would be a good.

(In addition potential alternatives are not discussed, though this is less 
important than giving the problems that shall be solved.)

Personally I am lacking in depth experience with systemd myself so I cannot 
judge its abilities. I have a tendency to like designs that I can understand 
and implement myself, systemd is mainly a black box for me, so I do not know 
what kind of advantages or disadvantages it bears for intelmq and they are 
not explained so far.)

My intention is to help you and us all to build a great system, 
this is why I'm trying to be clear about what I do not understand.
You have asked for my opinion, here it is. I hope it is helpful.

Best Regards,
Bernhard

-- 
www.intevation.de/~bernhard   +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20170421/569a0987/attachment.sig>