Hey Folks,
Whilst checking the dependencies, I've found out that a lot of Bots have their own requirements-file.
From a packagers point of view, I think this is hard to maintain.
I think, we should discuss if it is reasonable to split the software into a CORE package, were all packages have the same dependencies, and several BOTS packages with their own dependencies.
This would create the additional need to rethink the BOTS json-file. It would be possible to have all JSON configs for the Bots in a bots.d directory and search for them in this directory.
I know, that this might create additional complexity in some points.
BR Dustin
On 2016/04/14 18:04, Dustin Demuth dustin.demuth@intevation.de wrote:
Hey Folks,
Whilst checking the dependencies, I've found out that a lot of Bots have their own requirements-file.
From a packagers point of view, I think this is hard to maintain.
I think, we should discuss if it is reasonable to split the software into a CORE package, were all packages have the same dependencies, and several BOTS packages with their own dependencies.
This would create the additional need to rethink the BOTS json-file. It would be possible to have all JSON configs for the Bots in a bots.d directory and search for them in this directory.
I know, that this might create additional complexity in some points.
sounds ok for me...
Sounds really good to me! I think its a good idea to re-organize the code and improve, like u said, the maintenance.
Dustin, is it possible for you to describe in some level of detail your plan in order to be quickly reviewed by community?
Thank you very much to come up with this idea, Best regards
On Thu, Apr 14, 2016 at 6:05 PM, L. Aaron Kaplan kaplan@cert.at wrote:
On 2016/04/14 18:04, Dustin Demuth dustin.demuth@intevation.de wrote:
Hey Folks,
Whilst checking the dependencies, I've found out that a lot of Bots have
their
own requirements-file.
From a packagers point of view, I think this is hard to maintain.
I think, we should discuss if it is reasonable to split the software
into a
CORE package, were all packages have the same dependencies, and several
BOTS
packages with their own dependencies.
This would create the additional need to rethink the BOTS json-file. It would be possible to have all JSON configs for the Bots in a bots.d directory and search for them in this directory.
I know, that this might create additional complexity in some points.
sounds ok for me...
Intelmq-dev mailing list Intelmq-dev@lists.cert.at http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
Some information that should help community follow your idea:
Repositories: - how many? - description for each one - examples of which kind of code each one will have
You mentioned some complexity regarding this changes. What kind of complexity do you see?
Suggestion ----- Regarding configs (e.g. intelmq configured with phishtank and autoshun feeds + dedup expert + file output bot):
bots.d\ collector.conf collector.phishtank.conf collector.autoshun.conf parser.conf parser.phishtank.conf parser.autoshun.conf expert.conf expert.dedup.conf output.conf output.archive.conf
collector.conf, parser.conf, expert.conf and output.conf are the default configurations for each type of bot which should be inherit for each type of bot. Example: phishtank parser will load all configs from parser.conf and if phishtank.parser.conf has some redefine parameters, this parameters should overwrite the default ones.
Regards
On Sat, Apr 16, 2016 at 4:12 PM, Tomás Lima synchroack@gmail.com wrote:
Sounds really good to me! I think its a good idea to re-organize the code and improve, like u said, the maintenance.
Dustin, is it possible for you to describe in some level of detail your plan in order to be quickly reviewed by community?
Thank you very much to come up with this idea, Best regards
On Thu, Apr 14, 2016 at 6:05 PM, L. Aaron Kaplan kaplan@cert.at wrote:
On 2016/04/14 18:04, Dustin Demuth dustin.demuth@intevation.de wrote:
Hey Folks,
Whilst checking the dependencies, I've found out that a lot of Bots
have their
own requirements-file.
From a packagers point of view, I think this is hard to maintain.
I think, we should discuss if it is reasonable to split the software
into a
CORE package, were all packages have the same dependencies, and several
BOTS
packages with their own dependencies.
This would create the additional need to rethink the BOTS json-file. It would be possible to have all JSON configs for the Bots in a bots.d directory and search for them in this directory.
I know, that this might create additional complexity in some points.
sounds ok for me...
Intelmq-dev mailing list Intelmq-dev@lists.cert.at http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
-- Tomás Lima* , * »-«* SYNchroACK *»-«
Hello Folks,
Tomás, thank you for your input. I'll describe my Ideas in more detail.
# Recent situation: Right now the bots-directory looks like this: intelmq/bots/ ./collectors/ ./experts/ ./outputs/ ./parsers/ ./BOTS
The file BOTS contains all data on bots available in the folders. Or enables the replication of those with differents URLs. It is not simple enough to add new bots to this file automagically. Removing them is even harder.
# Idea: When BOTS would be a directory looking like this: ./bots.d/ ./collectors/ ./experts/ ./outputs/ ./parsers/
It would be easier to maintain bots automatically. Just copy a collector-bot-config-file into bots.d/collectors/ to add a new collector. Removing them is also that easy.
Concerning your Questions: Am Samstag 16 April 2016 17:28:32 schrieben Sie:
Repositories:
- how many?
*Everything* that requires dependencies that are not met by the "CORE Application" (which needs to be defined), should be kept in a seperate repository. To facilitate development of new bots, clonable example-bots could be provided.
- description for each one
- examples of which kind of code each one will have
Sorry, I don't get that.
You mentioned some complexity regarding this changes. What kind of complexity do you see?
It's not one file anymore, you need to inspect several files and folders.
collector.conf, parser.conf, expert.conf and output.conf are the default configurations for each type of bot which should be inherit for each type of bot. Example: phishtank parser will load all configs from parser.conf and if phishtank.parser.conf has some redefine parameters, this parameters should overwrite the default ones.
I think using inheritance is too complex. Let's keep it simple.
Until now, I did not think about runtime and default configuration of the bots. This would also be reasonable.
BR Dustin
Dustin,
From my side, fully agree!
Cheers
On Mon, Apr 18, 2016 at 10:17 AM, Dustin Demuth <dustin.demuth@intevation.de
wrote:
Hello Folks,
Tomás, thank you for your input. I'll describe my Ideas in more detail.
# Recent situation: Right now the bots-directory looks like this: intelmq/bots/ ./collectors/ ./experts/ ./outputs/ ./parsers/ ./BOTS
The file BOTS contains all data on bots available in the folders. Or enables the replication of those with differents URLs. It is not simple enough to add new bots to this file automagically. Removing them is even harder.
# Idea: When BOTS would be a directory looking like this: ./bots.d/ ./collectors/ ./experts/ ./outputs/ ./parsers/
It would be easier to maintain bots automatically. Just copy a collector-bot-config-file into bots.d/collectors/ to add a new collector. Removing them is also that easy.
Concerning your Questions: Am Samstag 16 April 2016 17:28:32 schrieben Sie:
Repositories:
- how many?
*Everything* that requires dependencies that are not met by the "CORE Application" (which needs to be defined), should be kept in a seperate repository. To facilitate development of new bots, clonable example-bots could be provided.
- description for each one
- examples of which kind of code each one will have
Sorry, I don't get that.
You mentioned some complexity regarding this changes. What kind of complexity do you see?
It's not one file anymore, you need to inspect several files and folders.
collector.conf, parser.conf, expert.conf and output.conf are the default configurations for each type of bot which should be inherit for each type of bot. Example: phishtank parser will load all configs from parser.conf and if phishtank.parser.conf has some redefine parameters, this
parameters
should overwrite the default ones.
I think using inheritance is too complex. Let's keep it simple.
Until now, I did not think about runtime and default configuration of the bots. This would also be reasonable.
BR Dustin
Intelmq-dev mailing list Intelmq-dev@lists.cert.at http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
On 2016/04/18 15:04, Tomás Lima synchroack@gmail.com wrote:
Dustin,
From my side, fully agree!
+1 , same from me. It all makes sense IMHO.
Thanks for proposing this modular structure.
a.
Am Montag, 18. April 2016 19:11:22 schrieb L. Aaron Kaplan:
On 2016/04/18 15:04, Tomás Lima synchroack@gmail.com wrote:
From my side, fully agree!
+1 , same from me. It all makes sense IMHO.
As everybody agreed of doing a change here, I've created https://github.com/certtools/intelmq/issues/552 (Enable separate packaging of bots by allowing addition and removals to the config.)
Hi,
On 04/18/2016 11:17 AM, Dustin Demuth wrote:
collector.conf, parser.conf, expert.conf and output.conf are the default configurations for each type of bot which should be inherit for each type of bot. Example: phishtank parser will load all configs from parser.conf and if phishtank.parser.conf has some redefine parameters, this parameters should overwrite the default ones.
I think using inheritance is too complex. Let's keep it simple.
Until now, I did not think about runtime and default configuration of the bots. This would also be reasonable.
That's what Tomas addressed. Without a default or runtime config, every botconfig would have more than 25 fields more than now. Managing them manually would be horrible.
Sebastian
BR Dustin
Intelmq-dev mailing list Intelmq-dev@lists.cert.at http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
For reference: Issues related to this thread (or .conf files) - https://github.com/certtools/intelmq/issues/267 ! - https://github.com/certtools/intelmq/issues/368 - https://github.com/certtools/intelmq/issues/357
Am Dienstag 19 April 2016 10:43:58 schrieb Sebastian Wagner:
That's what Tomas addressed. Without a default or runtime config, every botconfig would have more than 25 fields more than now. Managing them manually would be horrible.
I agree, we need to give this a thougt. Can you briefly sketch the current setup in case I got it wrong, please?
From my understanding it is this way:
All Bots (should) inherit from lib/bots.py which:
1) Load DEFAULT Configuration via intelmq/__init__.py from etc/defaults.conf 2) Load RUNTIME Configuration via intelmq/__init__.py from etc/runtime.conf
When installing intelmq, a "default" configuration could be placed in /etc/intelmq/defaults.conf
whilst all configuration which will be altered by the manager should be stored in a directory like /var/lib/intelmq
Questions: What does startup.conf do? It's loaded in intelmqctl.py #224
I see it is loaded a second time in intelmqctl.py #241 ff But I don't get the point of it.
BR Dustin
Folks,
I'm not fully up to date about the intelmq configuration scheme, so I might be way off with this message.
On 20.04.2016 11:03, Dustin Demuth wrote:
Am Dienstag 19 April 2016 10:43:58 schrieb Sebastian Wagner:
That's what Tomas addressed. Without a default or runtime config, every botconfig would have more than 25 fields more than now. Managing them manually would be horrible.
I agree, we need to give this a thougt. Can you briefly sketch the current setup in case I got it wrong, please?
From my understanding it is this way:
All Bots (should) inherit from lib/bots.py which:
- Load DEFAULT Configuration via intelmq/__init__.py from etc/defaults.conf
- Load RUNTIME Configuration via intelmq/__init__.py from etc/runtime.conf
Aren't we running into the same issue as with the BOTS file? If installing a new bot requires manual configuration in a shared config-file then adding/removing bots cannot be easily done.
Here, too, I see a point in using a etc/defaults|runtime.d/* directory with small config files.
otmar
On 2016/04/20 11:04, Otmar Lendl lendl@cert.at wrote:
Folks,
I'm not fully up to date about the intelmq configuration scheme, so I might be way off with this message.
A tad bit ;-) Let me explain below.
On 20.04.2016 11:03, Dustin Demuth wrote:
Am Dienstag 19 April 2016 10:43:58 schrieb Sebastian Wagner:
That's what Tomas addressed. Without a default or runtime config, every botconfig would have more than 25 fields more than now. Managing them manually would be horrible.
I agree, we need to give this a thougt. Can you briefly sketch the current setup in case I got it wrong, please?
From my understanding it is this way:
All Bots (should) inherit from lib/bots.py which:
- Load DEFAULT Configuration via intelmq/__init__.py from etc/defaults.conf
- Load RUNTIME Configuration via intelmq/__init__.py from etc/runtime.conf
Aren't we running into the same issue as with the BOTS file? If
The BOTS file is simply a template of which bots exist in general. It is mainly used for the intelmq-manager to display which ones could get selected. And which parameters they have. This does not mean that the BOTS file always contains all defaults.conf parameters for every bot (not combinatorical explosion).
installing a new bot requires manual configuration in a shared config-file then adding/removing bots cannot be easily done.
Here, too, I see a point in using a etc/defaults|runtime.d/* directory with small config files.
otmar
// Otmar Lendl lendl@cert.at - T: +43 1 5056416 711 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
Intelmq-dev mailing list Intelmq-dev@lists.cert.at http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
Am Mittwoch, 20. April 2016 11:55:59 schrieb L. Aaron Kaplan:
Aren't we running into the same issue as with the BOTS file? If
The BOTS file is simply a template of which bots exist in general. It is mainly used for the intelmq-manager to display which ones could get selected. And which parameters they have.
In order to get this moving, I propose to tackle it in small steps.
What about implementing a solution for the contents of BOTS first? We keep the logic of etc/defaults.conf and etc/runtime.conf for this step. So no hiearchical configuration (for now) and then we take it from there.
Thinking about how to solve the BOTS file first: a) (proposed by Dusting) bots.d/ directory solution, means adding and removing filenames. b) adding an initialisation function to each module which returns the necessary information c) write a small packaging helper that adds/removes sections from BOTS
Some thoughts (being still new to the code base): b) seems attractive to me, because it keeps information close to where it is maintainted. Its main drawback probably is that is the largest change. Are there other drawbacks? c) has the drawback of requiring running code during (de)installation. And it does not add much over a).
Best, Bernhard
Hello,
for adding own bots to IntelMQ, I am using b) type of solution. Every BOT class contains Meta class, which include BOT type, name, description, default parameters and new harmonization fields. Then BOT is registered by setuptools entry_points functionality, so IntelMQ can generate the list of all BOTS.
You can find the example of implementation in https://gist.github.com/ondj/93d7d0e12fa07fe8205b24d76f258d46
Thet disadvantage of this solution is that for example default `collector_http` BOT has in current BOTS file multiple definitions for different services.
I am glad if you find an inspiration in my solution.
Best, Jakub
Dne 2016-06-30 17:55, Bernhard Reiter napsal:
Am Mittwoch, 20. April 2016 11:55:59 schrieb L. Aaron Kaplan:
Aren't we running into the same issue as with the BOTS file? If
The BOTS file is simply a template of which bots exist in general. It is mainly used for the intelmq-manager to display which ones could get selected. And which parameters they have.
In order to get this moving, I propose to tackle it in small steps.
What about implementing a solution for the contents of BOTS first? We keep the logic of etc/defaults.conf and etc/runtime.conf for this step. So no hiearchical configuration (for now) and then we take it from there.
Thinking about how to solve the BOTS file first: a) (proposed by Dusting) bots.d/ directory solution, means adding and removing filenames. b) adding an initialisation function to each module which returns the necessary information c) write a small packaging helper that adds/removes sections from BOTS
Some thoughts (being still new to the code base): b) seems attractive to me, because it keeps information close to where it is maintainted. Its main drawback probably is that is the largest change. Are there other drawbacks? c) has the drawback of requiring running code during (de)installation. And it does not add much over a).
Best, Bernhard _______________________________________________ Intelmq-dev mailing list Intelmq-dev@lists.cert.at http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
Hallo Folks,
Jakub, thank you very much for your approach with us.
Am Freitag 01 Juli 2016 11:57:42 schrieb j.onderka@nbu.cz:
for adding own bots to IntelMQ, I am using b) type of solution. Every BOT class contains Meta class, which include BOT type, name, description, default parameters and new harmonization fields. Then BOT is registered by setuptools entry_points functionality, so IntelMQ can generate the list of all BOTS.
As I'm not familiar with setuptools: I guess this requires a modification of setup.py for each bot?
BR Dustin
Hello
As I'm not familiar with setuptools: I guess this requires a modification of setup.py for each bot?
Yes, in this solution for each new bot it is necessary to add a line to setup.py. But maybe there is a solution to find bots automatically during install.
Jakub.
Dne 5.7.2016 v 11:37 Dustin Demuth napsal(a):
Hallo Folks,
Jakub, thank you very much for your approach with us.
Am Freitag 01 Juli 2016 11:57:42 schrieb j.onderka@nbu.cz:
for adding own bots to IntelMQ, I am using b) type of solution. Every BOT class contains Meta class, which include BOT type, name, description, default parameters and new harmonization fields. Then BOT is registered by setuptools entry_points functionality, so IntelMQ can generate the list of all BOTS.
As I'm not familiar with setuptools: I guess this requires a modification of setup.py for each bot?
BR Dustin
Intelmq-dev mailing list Intelmq-dev@lists.cert.at http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev