Hi,
Sorry to bother again with something which is probably evident. Now that 3.0.0 is out I'd like to retry setting up intelmq & intelmq-manager from the package distribution (Ubuntu 20.04) and then add our own bots and configurations to the setup.
Is this the correct order of operations? Am I missing something?
1. install intelmq and intelmq-manager from package management (with LSB paths option) 2. copy our bots to proper subdirectories below /usr/lib/python3/dist-packages/intelmq/bots 3. copy our library modules below /usr/lib/python3/dist-packages/intelmq/lib 4. run intelmqsetup 5. adapt runtime.conf, defaults.conf, BOTS and pipeline.conf under /etc/intelmq to our needs 6. start our bot net with intelmqctl
Thanks again, Mika
Hi,
On 3/8/21 1:59 PM, Mika Silander wrote:
Sorry to bother again with something which is probably evident. Now that 3.0.0 is out I'd like
As it has an impact on the answer: Do you mean 2.3.0 or 3.0.0 alpha (develop branch)?
Sebastian
Hi Sebastian,
If it is not too much trouble, I'd like to hear the answer for both 2.3.0 and 3.0.0 alpha. A lot of effort has gone into my development, time flies, and I'd need to find an installation path to a reasonably stable production system for us. If you have to choose between the above two versions, provide the answer for stabler one, please.
Best regards, Mika
----- Original Message ----- From: "Sebastian Wagner" wagner@cert.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 8 March, 2021 15:07:13 Subject: Re: [IntelMQ-dev] Intelmq + intelmq-manager from package distribution + own bots & confs
Hi,
On 3/8/21 1:59 PM, Mika Silander wrote:
Sorry to bother again with something which is probably evident. Now that 3.0.0 is out I'd like
As it has an impact on the answer: Do you mean 2.3.0 or 3.0.0 alpha (develop branch)?
Sebastian
Hi Mika,
Am Montag 08 März 2021 14:13:09 schrieb Mika Silander:
I'd need to find an installation path to a reasonably stable production system for us.
version 2.3.0 is stable, so you'd probably should go for this one. (The big change vom 2.2 to 2.3 is that the Manager uses a new backend, called API, this may lead to a few minor defects that will usually be resolved quickly.)
So far I haven't installed the new 2.3.0 version, but usually someone just follows the installation requirements and yes, I'd go with native packages for a production setup (unless there are good reasons not to).
Regards, Bernhard
On 09.03.2021, at 08:56, Bernhard Reiter bernhard@intevation.de wrote:
Signed PGP part Hi Mika,
Am Montag 08 März 2021 14:13:09 schrieb Mika Silander:
I'd need to find an installation path to a reasonably stable production system for us.
version 2.3.0 is stable, so you'd probably should go for this one. (The big change vom 2.2 to 2.3 is that the Manager uses a new backend, called API, this may lead to a few minor defects that will usually be resolved quickly.)
So far I haven't installed the new 2.3.0 version, but usually someone just follows the installation requirements and yes, I'd go with native packages for a production setup (unless there are good reasons not to).
Agreed with Bernhard.
The nice thing with the native packages is that you get the updates / patches automatically and that most always worked so far for me.
So, yeah... that's definitely the stable route.
Version 3.0 might see a major change in the internal data format. We still need to figure out how to make this transition seamless for the existing bots.
You can see a list of proposed changes for 3.0 here:
https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectur...
Best, Aaron Kaplan.
Hi Bernhard,
Thanks for the advice. I also thought 2.3.0 may be a safer choice for me at the time being but I wanted to hear yours and Sebastian's opinions on this first. Still, my small problem remains: how to add my own bots and libraries to the production setup installed from the package management (in this case Ubuntu 20.04). I suppose there are again several bad ways of achieving this and a few good ones. Conf files are easy but having my own bots and libraries in use alongside the ones coming from package management e.g. subdirectories /usr/bin/ , /usr/lib/python3/dist-packages etc is a bit more complicated. I'd like to find an easy solution for this. The elegant (but tedious) option would be to include my bots and libraries into the process generating the intelmq packages but at the moment it feels like overkill. Anyway, happy to hear suggestions and ideas related to this.
Best regards, Mika
----- Original Message ----- From: "Bernhard Reiter" bernhard@intevation.de To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Tuesday, 9 March, 2021 09:56:00 Subject: Re: [IntelMQ-dev] Intelmq + intelmq-manager from package distribution + own bots & confs
Hi Mika,
Am Montag 08 März 2021 14:13:09 schrieb Mika Silander:
I'd need to find an installation path to a reasonably stable production system for us.
version 2.3.0 is stable, so you'd probably should go for this one. (The big change vom 2.2 to 2.3 is that the Manager uses a new backend, called API, this may lead to a few minor defects that will usually be resolved quickly.)
So far I haven't installed the new 2.3.0 version, but usually someone just follows the installation requirements and yes, I'd go with native packages for a production setup (unless there are good reasons not to).
Regards, Bernhard
Hi Mika,
I'll try to response to both e-mails here. If I missed an aspect of the question(s), please forgive me :)
I see some options:
* Bring your code upstream (if'd like to get advice on that, let us know). Install the upstream version. or * Maintain your fork: Clone the certtools/intelmq Repo, using either master or maintenance branch as base. Add your non-publishable code there. Merge from upstream when a new version is ready. Install with `pip3 install -e path`. or * Do it manually, similar to how Intevation does it for the certbund-contact: https://github.com/Intevation/intelmq-certbund-contact/ (look at the postinst file in debian/): * Install the bots in the right place. That can be done with creating your own setup.py, which maps the executables to the modules, and installs the modules with the same directory structure as the upstream IntelMQ. * Add your bots to BOTS, so that they are visible in the IntelMQ Manager
You can have a look at https://github.com/jhemp/intelmq-tools but I don't know it's status and about the compatibility with 2.3.0 and 3.0.0
In IntelMQ 3.0 / the develop branch it's a bit easier, as you don't need to hassle with BOTS (once #1751 is merged). You still need to create the Python modules. As always, contributions are welcome :)
kind regards Sebastian
#1751: https://github.com/certtools/intelmq/pull/1751
On 3/9/21 9:18 AM, Mika Silander wrote:
Hi Bernhard,
Thanks for the advice. I also thought 2.3.0 may be a safer choice for me at the time being but I wanted to hear yours and Sebastian's opinions on this first. Still, my small problem remains: how to add my own bots and libraries to the production setup installed from the package management (in this case Ubuntu 20.04). I suppose there are again several bad ways of achieving this and a few good ones. Conf files are easy but having my own bots and libraries in use alongside the ones coming from package management e.g. subdirectories /usr/bin/ , /usr/lib/python3/dist-packages etc is a bit more complicated. I'd like to find an easy solution for this. The elegant (but tedious) option would be to include my bots and libraries into the process generating the intelmq packages but at the moment it feels like overkill. Anyway, happy to hear suggestions and ideas related to this.
Best regards, Mika
On 3/8/21 2:13 PM, Mika Silander wrote:
Hi Sebastian,
If it is not too much trouble, I'd like to hear the answer for both 2.3.0 and 3.0.0 alpha. A lot of effort has gone into my development, time flies, and I'd need to find an installation path to a reasonably stable production system for us. If you have to choose between the above two versions, provide the answer for stabler one, please.
Best regards, Mika
----- Original Message ----- From: "Sebastian Wagner" wagner@cert.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 8 March, 2021 15:07:13 Subject: Re: [IntelMQ-dev] Intelmq + intelmq-manager from package distribution + own bots & confs
Hi,
On 3/8/21 1:59 PM, Mika Silander wrote:
Sorry to bother again with something which is probably evident. Now that 3.0.0 is out I'd like
As it has an impact on the answer: Do you mean 2.3.0 or 3.0.0 alpha (develop branch)?
Sebastian
Hi Sebastian,
Thanks for spending time on this and outlining the potential options for integration.
Our bots are very much tailored to our particular needs so I don't see much value in bringing them upstream, unless you're interested in inventive bugs of course :-). There might be one bot that - after some modifications - could be perhaps of more general use. It maps IP addresses to abuse contact addresses and covers the case where we have knowledge of organizations' abuse contacts within single ASs, i.e. we can deduce a more specific abuse contact than what would be possible based on ASN only. You have to have knowledge of what IP networks belong to what organizations in a db, conf file or similar.
Maintaining our own fork: this is attractive but requires a lot more in terms of learning, I still consider myself novice with intelmq. When the install in this case is based on pip3, doesn't it also open the path to a virtual environment install of intelmq? I don't see a great risk in automatic system python3 package updates breaking an intelmq installation, but wouldn't a venv + pip3 install at least in theory protect even more against such problems?
The manual option is interesting because it seems requiring less familiarization effort from me but is there a risk that it also needs to be more often modified/adapted between intelmq (major) version upgrades? I'm now thinking for the long run, hoping that our team could use intelmq for years to come.
Best regards, Mika
----- Original Message ----- From: "Sebastian Wagner" wagner@cert.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Tuesday, 9 March, 2021 15:59:34 Subject: Re: [IntelMQ-dev] Intelmq + intelmq-manager from package distribution + own bots & confs
Hi Mika,
I'll try to response to both e-mails here. If I missed an aspect of the question(s), please forgive me :)
I see some options:
* Bring your code upstream (if'd like to get advice on that, let us know). Install the upstream version. or * Maintain your fork: Clone the certtools/intelmq Repo, using either master or maintenance branch as base. Add your non-publishable code there. Merge from upstream when a new version is ready. Install with `pip3 install -e path`. or * Do it manually, similar to how Intevation does it for the certbund-contact: https://github.com/Intevation/intelmq-certbund-contact/ (look at the postinst file in debian/): * Install the bots in the right place. That can be done with creating your own setup.py, which maps the executables to the modules, and installs the modules with the same directory structure as the upstream IntelMQ. * Add your bots to BOTS, so that they are visible in the IntelMQ Manager
You can have a look at https://github.com/jhemp/intelmq-tools but I don't know it's status and about the compatibility with 2.3.0 and 3.0.0
In IntelMQ 3.0 / the develop branch it's a bit easier, as you don't need to hassle with BOTS (once #1751 is merged). You still need to create the Python modules. As always, contributions are welcome :)
kind regards Sebastian
#1751: https://github.com/certtools/intelmq/pull/1751
On 3/9/21 9:18 AM, Mika Silander wrote:
Hi Bernhard,
Thanks for the advice. I also thought 2.3.0 may be a safer choice for me at the time being but I wanted to hear yours and Sebastian's opinions on this first. Still, my small problem remains: how to add my own bots and libraries to the production setup installed from the package management (in this case Ubuntu 20.04). I suppose there are again several bad ways of achieving this and a few good ones. Conf files are easy but having my own bots and libraries in use alongside the ones coming from package management e.g. subdirectories /usr/bin/ , /usr/lib/python3/dist-packages etc is a bit more complicated. I'd like to find an easy solution for this. The elegant (but tedious) option would be to include my bots and libraries into the process generating the intelmq packages but at the moment it feels like overkill. Anyway, happy to hear suggestions and ideas related to this.
Best regards, Mika
On 3/8/21 2:13 PM, Mika Silander wrote:
Hi Sebastian,
If it is not too much trouble, I'd like to hear the answer for both 2.3.0 and 3.0.0 alpha. A lot of effort has gone into my development, time flies, and I'd need to find an installation path to a reasonably stable production system for us. If you have to choose between the above two versions, provide the answer for stabler one, please.
Best regards, Mika
----- Original Message ----- From: "Sebastian Wagner" wagner@cert.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 8 March, 2021 15:07:13 Subject: Re: [IntelMQ-dev] Intelmq + intelmq-manager from package distribution + own bots & confs
Hi,
On 3/8/21 1:59 PM, Mika Silander wrote:
Sorry to bother again with something which is probably evident. Now that 3.0.0 is out I'd like
As it has an impact on the answer: Do you mean 2.3.0 or 3.0.0 alpha (develop branch)?
Sebastian
Hi Mika,
Am Mittwoch 10 März 2021 09:42:17 schrieb Mika Silander:
The manual option is interesting because it seems requiring less familiarization effort from me but is there a risk that it also needs to be more often modified/adapted between intelmq (major) version upgrades?
if you take a look at how we (at Intevation) did the packaging of the contact experts of https://github.com/Intevation/intelmq-certbund-contact/ this was quite stable for the last 4-5 years for Intelmq pre 1.0 up to 2.3.0 now. (Of course some minor adaptions, but not much, it was migrated to the repo above in 2018, but started 2016.)
This is part of the solution we call intelmq-cb-mailgen https://github.com/Intevation/intelmq-mailgen-release running until 2017.
Regards, Bernhard
Hi Bernhard,
Thanks for your comments. The amount of work you describe covering those 4-5 years is a lot less than what I had expected. I'll take a look at your solution and the cloning solution of the git repo.
Best regards, Mika
----- Original Message ----- From: "Bernhard Reiter" bernhard@intevation.de To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Wednesday, 10 March, 2021 10:58:45 Subject: Re: [IntelMQ-dev] Intelmq + intelmq-manager from package distribution + own bots & confs
Hi Mika,
Am Mittwoch 10 März 2021 09:42:17 schrieb Mika Silander:
The manual option is interesting because it seems requiring less familiarization effort from me but is there a risk that it also needs to be more often modified/adapted between intelmq (major) version upgrades?
if you take a look at how we (at Intevation) did the packaging of the contact experts of https://github.com/Intevation/intelmq-certbund-contact/ this was quite stable for the last 4-5 years for Intelmq pre 1.0 up to 2.3.0 now. (Of course some minor adaptions, but not much, it was migrated to the repo above in 2018, but started 2016.)
This is part of the solution we call intelmq-cb-mailgen https://github.com/Intevation/intelmq-mailgen-release running until 2017.
Regards, Bernhard
Hi Mika,
Am Mittwoch 10 März 2021 10:48:13 schrieb Mika Silander:
The amount of work you describe covering those 4-5 years is a lot less than what I had expected.
once set up, it was little maintenance work to keep up with IntelMQ for inserting our bots.
On a general note: we are using SCMs like hg or git to manage and partly also push manual configurations around for software we help to run or run ourselves. And if several people help to administrate a machine, we keep a logbook, where all actions done as root are noted down. E.g for installation and deinstallation of several package managers we even have a script that helps us to record stuff and fire up the /etc/logbuch.txt in the editor.: https://hg.intevation.de/adminton/file/tip/logbuch-tools/logbuch-installer (My point is, your mileage will vary and depends on your tools and administration habits.)
I'll take a look at your solution and the cloning solution of the git repo.
Feel free to ask or suggest stuff.
Best, Bernhard