[Intelmq-dev] Packaging Strategy for Bots with dependencies [SEC=UNCLASSIFIED]

[Clark, Andrew]

UNCLASSIFIED
Hi Bernhard, all,

Sorry to come into this a bit late ... my comments are below:

> Am Mittwoch, 20. April 2016 11:55:59 schrieb L. Aaron Kaplan:
> > > Aren't we running into the same issue as with the BOTS file? If
> >
> > The BOTS file is simply a template of which bots exist in general.
> > It is mainly used for the intelmq-manager to display which ones 
> > could get selected. And which parameters they have.
>
> In order to get this moving, I propose to tackle it in small steps.
>
> What about implementing a solution for the contents of BOTS first?
> We keep the logic of etc/defaults.conf and etc/runtime.conf for this step. 
> So no hiearchical configuration (for now) and then we take it from there.

If I understand correctly the IntelMQ 'runtime' configuration is currently spread across three configuration files (runtime.conf, pipeline.conf and startup.conf), and there are other configuration files as well (e.g. defaults.conf, harmonization.conf and system.conf). I think it's a great idea to consider how to improve on the current monolithic BOTS template configuration, but also wonder if some of the configuration that is currently in separate files could be merged. For example, could startup and runtime be merged? Maybe even pipeline too?
 
> Thinking about how to solve the BOTS file first:
> a) (proposed by Dusting)  bots.d/ directory solution, 
>     means adding and removing filenames.
> b) adding an initialisation function to each module which 
>     returns the necessary information
> c) write a small packaging helper that adds/removes sections
>     from BOTS
>
> Some thoughts (being still new to the code base):
> b) seems attractive to me, because it keeps information 
>     close to where it is maintainted. Its main drawback probably is
>     that is the largest change. Are there other drawbacks?
> c) has the drawback of requiring running code during (de)installation.
>    And it does not add much over a).

I have a STRONG preference for an approach similar to 'a' above. In my opinion we should be aiming to decouple the configuration from the bot code itself as much as possible, and encouraging more generic bots (i.e., less bots that are more generic (and more configurable), and more configuration files. In general each bot should potentially accept many configurations. This approach should serve to limit the explosion in the number of bots (I think the parser bots are a good example of where we could reduce the number of bots focusing on 'what' they are parsing, rather than their source).

Along the lines of option 'a' above I think the 'modules-available' and 'modules-enabled' approach used by the Apache web server could work for us too. All available configurations (templates, etc.) could be stored in a 'etc/bots-available' location, and the copied/edited into the 'etc/bots-enabled' location to enable them.

Andrew

-- 

Andrew Clark | Senior Technical Advisor | CERT Australia Attorney-General's Department, Australian Government
Phone: +61 2 6141 2538
Online: www.cert.gov.au

For all CERT Australia operational matters, please call our
                hotline: 1300 172 499, or +61 26141 2999 or
                email: info at cert.gov.au 


---------------------------------------------------- 
If you have received this transmission in error please
notify us immediately by return e-mail and delete all
copies. If this e-mail or any attachments have been sent
to you in error, that error does not constitute waiver
of any confidentiality, privilege or copyright in respect
of information in the e-mail or attachments.