[IntelMQ-dev] Intelmq + intelmq-manager from package distribution + own bots & confs

[Mika Silander]

Hi Sebastian,

 Thanks for spending time on this and outlining the potential options for integration.

 Our bots are very much tailored to our particular needs so I don't see much value in bringing them upstream, unless you're interested in inventive bugs of course :-). There might be one bot that - after some modifications - could be perhaps of more general use. It maps IP addresses to abuse contact addresses and covers the case where we have knowledge of organizations' abuse contacts within single ASs, i.e. we can deduce a more specific abuse contact than what would be possible based on ASN only. You have to have knowledge of what IP networks belong to what organizations in a db, conf file or similar.

 Maintaining our own fork: this is attractive but requires a lot more in terms of learning, I still consider myself novice with intelmq. When the install in this case is based on pip3, doesn't it also open the path to a virtual environment install of intelmq? I don't see a great risk in automatic system python3 package updates breaking an intelmq installation, but wouldn't a venv + pip3 install at least in theory protect even more against such problems?

 The manual option is interesting because it seems requiring less familiarization effort from me but is there a risk that it also needs to be more often modified/adapted between intelmq (major) version upgrades? I'm now thinking for the long run, hoping that our team could use intelmq for years to come.

Best regards, Mika

----- Original Message -----
From: "Sebastian Wagner" <wagner at cert.at>
To: "Mika Silander" <mika.silander at csc.fi>, "intelmq-dev" <intelmq-dev at lists.cert.at>
Sent: Tuesday, 9 March, 2021 15:59:34
Subject: Re: [IntelMQ-dev] Intelmq + intelmq-manager from package distribution + own bots & confs

Hi Mika,

I'll try to response to both e-mails here. If I missed an aspect of the
question(s), please forgive me :)

I see some options:

* Bring your code upstream (if'd like to get advice on that, let us
know). Install the upstream version.
or
* Maintain your fork: Clone the certtools/intelmq Repo, using either
master or maintenance branch as base. Add your non-publishable code
there. Merge from upstream when a new version is ready. Install with
`pip3 install -e path`.
or
* Do it manually, similar to how Intevation does it for the
certbund-contact:
https://github.com/Intevation/intelmq-certbund-contact/ (look at the
postinst file in debian/):
  * Install the bots in the right place. That can be done with creating
your own setup.py, which maps the executables to the modules, and
installs the modules with the same directory structure as the upstream
IntelMQ.
  * Add your bots to BOTS, so that they are visible in the IntelMQ Manager

You can have a look at https://github.com/jhemp/intelmq-tools but I
don't know it's status and about the compatibility with 2.3.0 and 3.0.0

In IntelMQ 3.0 / the develop branch it's a bit easier, as you don't need
to hassle with BOTS (once #1751 is merged). You still need to create the
Python modules. As always, contributions are welcome :)

kind regards
Sebastian

#1751: https://github.com/certtools/intelmq/pull/1751

On 3/9/21 9:18 AM, Mika Silander wrote:
> Hi Bernhard,
>
>  Thanks for the advice. I also thought 2.3.0 may be a safer choice for me at the time being but I wanted to hear yours and Sebastian's opinions on this first. Still, my small problem remains: how to add my own bots and libraries to the production setup installed from the package management (in this case Ubuntu 20.04). I suppose there are again several bad ways of achieving this and a few good ones. Conf files are easy but having my own bots and libraries in use alongside the ones coming from package management e.g. subdirectories /usr/bin/ , /usr/lib/python3/dist-packages etc is a bit more complicated. I'd like to find an easy solution for this. The elegant (but tedious) option
> would be to include my bots and libraries into the process generating the intelmq packages but at the moment it feels like overkill. Anyway, happy to hear suggestions and ideas related to this.
>
> Best regards, Mika
On 3/8/21 2:13 PM, Mika Silander wrote:
> Hi Sebastian,
>
>  If it is not too much trouble, I'd like to hear the answer for both 2.3.0 and 3.0.0 alpha.
> A lot of effort has gone into my development, time flies, and I'd need to find an installation
> path to a reasonably stable production system for us. If you have to choose between the
> above two versions, provide the answer for stabler one, please.
>
> Best regards, Mika 
>
>
> ----- Original Message -----
> From: "Sebastian Wagner" <wagner at cert.at>
> To: "Mika Silander" <mika.silander at csc.fi>, "intelmq-dev" <intelmq-dev at lists.cert.at>
> Sent: Monday, 8 March, 2021 15:07:13
> Subject: Re: [IntelMQ-dev] Intelmq + intelmq-manager from package distribution + own bots & confs
>
> Hi,
>
> On 3/8/21 1:59 PM, Mika Silander wrote:
>>  Sorry to bother again with something which is probably evident. Now that 3.0.0 is out I'd like
> As it has an impact on the answer: Do you mean 2.3.0 or 3.0.0 alpha
> (develop branch)?
>
> Sebastian
>
-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg