Folks,
We have been thinking about things to improve IntelMQ in order to have it really suitable for production... and now we want to raise the discussion about the following ideas or other related ideas that you might think of:
- Run modes: stream (always running, e.g. shadowserver-parser bot) or scheduled (run once every day at 10:00AM, e.g. shadowserver-chargen-collector bot) - Systemd: take care of bots which are running permanently. Use systemd to manage bots in stream run mode that cannot stop and also, do a better job with managing which bots processes which are really running, not using the current implementation with pidfiles. - Crontab: manage bots that need to run once, e.g. at midnight every day. Use crontab to manage bots in scheduled run mode to just run once on a pre-defined scheduled time.
Why these ideas came up? Mainly because of two things: 1. Run modes are needed due the different purposes of a bot which is collecting a real-time feed and a bot that is collecting shadowserver reports once in a day. 2. Systemd and Crontab are two well-know tools that we all know about it and IntelMQ seems to need them since we have seen some problems with managing bots processes (running or not) and report automatically failures by email, just to give some examples.
Until now, we don't have any of these features implemented, we only have developed some "rubbish" code in order to do some Proof-of-concept but nothing serious. So, please guys, share your thoughts about these ideas or others related in order to define, as a community, a strategy for the next month.
Thank you.
Cheers!
Addendum:
Some of you are using supervisord to manage intelmq bots. As discussed at the IHAP meeting in December, this can be implemented equivalently. Although, as far as I found out, it does not support scheduled jobs by itself, so cron has to be used for this (as opposed to systemd which has timers).
Are there any other process managers which are in use for intelmq (or planned to be used)?
The pull request "Implement oneshots, scheduled via cron" https://github.com/certtools/intelmq/pull/846 implements one part of this proposal (it's not a PoC).
Sebastian
On 01/30/2017 12:12 PM, Tomás Lima wrote:
Folks,
We have been thinking about things to improve IntelMQ in order to have it really suitable for production... and now we want to raise the discussion about the following ideas or other related ideas that you might think of:
- Run modes: stream (always running, e.g. shadowserver-parser bot) or
scheduled (run once every day at 10:00AM, e.g. shadowserver-chargen-collector bot)
- Systemd: take care of bots which are running permanently. Use
systemd to manage bots in stream run mode that cannot stop and also, do a better job with managing which bots processes which are really running, not using the current implementation with pidfiles.
- Crontab: manage bots that need to run once, e.g. at midnight every
day. Use crontab to manage bots in scheduled run mode to just run once on a pre-defined scheduled time.
Why these ideas came up? Mainly because of two things:
- Run modes are needed due the different purposes of a bot which is
collecting a real-time feed and a bot that is collecting shadowserver reports once in a day. 2. Systemd and Crontab are two well-know tools that we all know about it and IntelMQ seems to need them since we have seen some problems with managing bots processes (running or not) and report automatically failures by email, just to give some examples.
Until now, we don't have any of these features implemented, we only have developed some "rubbish" code in order to do some Proof-of-concept but nothing serious. So, please guys, share your thoughts about these ideas or others related in order to define, as a community, a strategy for the next month.
Thank you.
Cheers!
Hi Tomás,
Am Montag 30 Januar 2017 12:12:21 schrieb Tomás Lima:
- Systemd and Crontab are two well-know tools that we all know about it
and IntelMQ seems to need them
right now crontab seems already be used (at least recommended) if I remember correctly to read in some data needed for some bots.
So there is the overall question: # What are the target systems of intelmq? Where should it run?
Is it GNU/Linux systems (which would be easiest), GNU/Linux like systems including OpenBSD, or systems like Mac OS X and MS Windows?
** systemd is a tool very specific to operating systems with the kernel Linux that also use systemd already. Making it a requirement would exclude a number of other operating systems (or even GNU/Linux distribution).
# I agree that we should have a way to "schedule" components that run continuously or just at certain times. The question is: Do we consider this a task to be configured within intelmq or as a requirement for its environment?
The answer to these two questions would highly influence my design decisions.
Regards, Bernhard
Hi,
On 01/31/2017 01:25 PM, Bernhard Reiter wrote:
right now crontab seems already be used (at least recommended) if I remember correctly to read in some data needed for some bots.
That's optional tough.
# What are the target systems of intelmq? Where should it run?
All operating systems we (the community) are interested in.
Is it GNU/Linux systems (which would be easiest), GNU/Linux like systems including OpenBSD, or systems like Mac OS X and MS Windows?
I know that Debian and Ubuntu are used and thus intelmq is good tested on these systems. Other *nix OSs can be supported too if someone is willing to provide support for it. But OS X and Win are definitely out of scope for us currently. However, if someone wants to port intelmq to these platforms, feel free to do so.
** systemd is a tool very specific to operating systems with the kernel Linux that also use systemd already. Making it a requirement would exclude a number of other operating systems (or even GNU/Linux distribution).
As I said, that's not specific to systemd, together with @ondj I'd like to support supervisord too. You can also add SysV support if you want to, however I don't see any reason to do so currently.
Sebastian
Am Dienstag 31 Januar 2017 13:40:48 schrieb Sebastian Wagner:
On 01/31/2017 01:25 PM, Bernhard Reiter wrote:
right now crontab seems already be used (at least recommended) if I remember correctly to read in some data needed for some bots.
That's optional tough.
The question for me always is: What fraction of typical intelmq installation would need those function? I'd say a significant one. Thus it should be a core functionality of intelmq. (Of course some installations may chose to not use these intelmq capabilities.)
# What are the target systems of intelmq? Where should it run?
All operating systems we (the community) are interested in.
Is it GNU/Linux systems (which would be easiest), GNU/Linux like systems including OpenBSD, or systems like Mac OS X and MS Windows?
I know that Debian and Ubuntu are used and thus intelmq is good tested on these systems. Other *nix OSs can be supported too if someone is willing to provide support for it. But OS X and Win are definitely out of scope for us currently. However, if someone wants to port intelmq to these platforms, feel free to do so.
We should state on our pages for development vision/goals that the current IntelMQ Initiative only aims for GNU/Linux systems, especially Debian based. More GNU/Linux like system can be supported if maintainers step up.
For our technical design this means that we can use methods that are available on a typical GNU/Linux system or can be easily deployed on thus a system.
** systemd is a tool very specific to operating systems with the kernel Linux that also use systemd already. Making it a requirement would exclude a number of other operating systems (or even GNU/Linux distribution).
As I said, that's not specific to systemd, together with @ondj I'd like to support supervisord too. You can also add SysV support if you want to, however I don't see any reason to do so currently.
I agree that we should build a mechanism for process management and for scheduling time based that deals with dependencies and additionally congestion control.
From my perspective it would be good design to chose one prefered method.
So far I have the feeling that the configuration should be internal to intelmq, so it provides an "interface" to the bots. And the configuration can be placed close to the bot's code, e.g. the default directly in the code itself.
As for how the scheduling it done, I guess that we'll probably need one scheduling daemon for intelmq that will supervise the other processes. Before selecting components that could help us with this (e.g. systemd, at, cron, http://supervisord.org/, runit) we should collect a list of requirements (wiki?, issue?)
Best Regards, Bernhard
Am Freitag 03 Februar 2017 09:00:21 schrieb Bernhard Reiter:
As for how the scheduling it done, I guess that we'll probably need one scheduling daemon for intelmq that will supervise the other processes.
Another reasons for this is separation of access control. One example for this is configuration.
Right now intelmqctl runs as the unix user who has access to the configuration files itself. If we want a different unix user (that has restricted access to intelmq resources) to change something in particular like one value, we have to implement some access control. The typical way of implementing this priviledge separation is setuid/setguid or sudo/pkexec. Both have drawback if we want only a few selected points to be accessed. Setuid priviledges a full process, sudo restricts its configuration to a set of parameters for the process.
A control daemon could just offer whatever is allowed. (Postfix uses the daemon concept nicely for priviledge separation).
Just my 2 ¢, Bernhard
On 02/13/2017 01:59 PM, Bernhard Reiter wrote
Right now intelmqctl runs as the unix user who has access to the configuration files itself. If we want a different unix user (that has restricted access to intelmq resources) to change something in particular like one value, we have to implement some access control. The typical way of implementing this priviledge separation is setuid/setguid or sudo/pkexec. Both have drawback if we want only a few selected points to be accessed. Setuid priviledges a full process, sudo restricts its configuration to a set of parameters for the process.
Systemd also has the possibility to restrict the daemon's privileges, it's called "portable services". https://lwn.net/Articles/706025/ So instead of dropping privileges inside intelmq, that could be done by the process manager too.
Sebastian
Am Mittwoch, 12. April 2017 12:32:52 schrieb Sebastian Wagner:
So instead of dropping privileges inside intelmq, that could be done by the process manager too.
Yes, there are probably several ways how to create a process with lower privileges and letting it keep some higher privileged resources (like access to serve tcp port 80).
The main point of my argument is different, though and becomes an argument for solving the problem that we may need less priviledges users to be able to change some selected intelmq configuration values. Access control to these setting would need to be intelmq specific. So it probably is done well within intelmq's own code. If you follow this argument it becomes and advantage for implementing a intelmq master process (like the one on postfix) to coordinate the others.
Best Regards, Bernhard