IntelMQ-dev

intelmq-dev@lists.cert.at

3 participants
255 discussions

IEP02: Introduce IntelMQ Mixins
by Birger Schacht 03 Mar '21

03 Mar '21

Dear IntelMQ developers, below I am writing about an idea that might make contributing to IntelMQ easier- it is a small change, but could have implications for various bots: # Introduce IntelMQ Mixins (IntelMQ Enhancement Proposal 02) I would like to lower the bar for adding additional bots by introducing mixins. Mixins are a concept coming from OOP theory and it is closely related to multiple inheritance. I think it could be a way to avoid a lot of duplicate code by outsourcing various functionalities to specific classes. ## Current situation At the moment there are two types of bots that could make use of mixins. The first are bots that make HTTP requests. Currently we make that easier for bots by providing two methods, `create_request_session` in intelmq/lib/utils.py and `set_request_parameters` as part of the bot module (intelmq/lib/bot.py) itself. When the bot wants to use a request session, it first uses `set_request_parameters` and then uses the session object returned by `create_request_session` to make the request. I think its a bit confusing to have methods that are related in two different modules and I think the Bot class itself should not contain any code that is not used by all the bots. The second type of bots that could make use of mixins are bots that use a cache. There are a lot of those that all use the `intelmq.lib.cache` module which provides a Cache object to work with. ## Improvement using mixins Using a Python class we could move everything related to HTTP requests to a class called HttpMixin. A bot that wants to use functionality from this Mixin just would have to inherit from this class, i.e.: ``` class MySuperBot(CollectorBot, HttpMixin): ``` The MySuperBot class would then have all the relevant attributes to use session objects and it would also have a private `__session` attribute to work with. Another upside of this approach is, that it is possible to generate a list of all the attributes a Bot has write access to (i.e. http_proxy, http_username, ssl_client_cert...) Similarly we could introduce a `CacheMixin` that allow access to private `__cache` attribute. If MySuperBot does not only want to make HTTP requests, but also have some caching mechanisms, it could simply inherit from both classes: ``` class MySuperBot(CollectorBot, HttpMixin, CacheMixin): ``` I propose to create a new module `intelmq.lib.mixins` that should be the home for the mixins. I have created a POC that implements the HttpMixin and uses it in the HTTPCollectorBot: https://github.com/certtools/intelmq/tree/schacht/http-mixin Mixins are also used in Django to add functionality to their class based Views, for more details see https://docs.djangoproject.com/en/3.1/topics/class-based-views/mixins/ A longer article about inheritance, MRO, mixins and Python that also talks about the implementation in Django can be found on https://www.thedigitalcatonline.com/blog/2020/03/27/mixin-classes-in-python/ cheers, Birger -- // Birger Schacht <schacht(a)cert.at> // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Installing new bots from devel host to production host
by Mika Silander 02 Mar '21

02 Mar '21

Hi, Bothering you again with a probably simple question. I've created half a dozen of bots on a development machine and now I'd like to create an installation system, Makefile or whatever to install those bots + configurations to the production host. It appears to me that the section "Update" (https://intelmq.readthedocs.io/en/latest/dev/guide.html#id12) contains the recommended steps to follow and assumes that one has the development environment (below /opt/dev_intelmq) available on the same host where one intends to run the production set up (under /opt/intelmq), correct? So, what do you recommend? Should I copy over the entire /opt/dev_intelmq to the production host and then "Update" there according to the above instructions or are there other ways to just copy/install the relevant bots+configurations from below /opt/dev_intelmq to the production host? Best regards and thanks again, Mika

3 6

Default crontab values for local lookup-databases update
by Sebastian Wagner 26 Feb '21

26 Feb '21

Dear devs, Thanks to Filip (CZ.NIC) IntelMQ comes now with an update mechanism for local lookup databases like TOR exit nodes, IP address to ASN ("ASN Lookup") and Maxmind GeoIP (IP address geolocation)[0]. Also, IntelMQ ships with update scripts for cron which are included in the deb/rpm packages as well. Currently the update scripts are scheduled as follows[1]: * TOR nodes: once per day. The database is very small. * Maxmind GeoIP: Once per week. Changes are scarce. * ASN Lookup: Every two hours. Big database, but the data is vital for subsequent routing of incidents. I'd like to hear your opinion if the default values are ok to ship with 2.3.0, especially for the last one. best regards, Sebastian [0]: https://github.com/certtools/intelmq/pull/1524 [1]: https://github.com/certtools/intelmq/blob/24f2355d0c549021a713c938d1d69a521… -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 2

Classification of malware itself in IntelMQ
by Sebastian Wagner 22 Feb '21

22 Feb '21

Dear IntelMQ community, sorry for cross-posting, but I think this topic should be discussed in a wider group. IntelMQ always followed the Reference Security Incident Taxonomy (short: RSIT)[0] and its predecessor for its 'classification.taxonomy/type' fields. The Classification column in the RSIT corresponds to our "classification.taxonomy" field, and the RSIT's second column (currently called Incident examples) corresponds to our "classification.type" field. "classification.identifier" is an optional third level free-text field to give more specific context.[1] Due to historical reasons and changes on both sides - IntelMQ as well as the RSIT -, IntelMQ's classification scheme deviated a bit from the RSIT over time. I'm working on aligning them again for 3.0, which works straightforward in most cases. But for one case, I need your input. The predecessor of the RSIT (the eCSIRT.net taxonomy)[2] used the malicious code taxonomy differently: To classify malware itself into categories, like virus, worm, trojan, etc. The RSIT never did that, as classifying malware is never unambiguous and there are plenty of existing classification scheme out there, which do this already. Also, the focus of the RSIT is different, as it classifies the incidents/events, not malware samples. And for this reason, IntelMQ had (until < 3.0.0) the classification.type "malware" in IntelMQ. Most of the usages were wrong anyway, and should have been infected-device, malware-distribution or something else anyway. There is only one usage in IntelMQ, which can not be changed. And that one is really about malware itself (or: the hashes of samples) as used in the GitHub Feed parser[3] and the FireEye Parser[4]. But the issue is more generic, as we need to decide anyway, how we want to deal with such malware-IoCs. A malware (hash) does not fit into the RSIT. It's neither an Infected System, a C2 Server, Malware Distribution nor Malware Configuration. It's just a malware (hash). I see four options: 1) Deviate from the RSIT and just use 'classification.taxonomy' = 'Malicious Code' and 'classification.type' = 'malware' 2) Deviate slightly less from the RSIT and use 'classification.taxonomy' = 'other' and 'classification.type' = 'malware' 3) Adhere strictly to the RSIT and use 'classification.taxonomy' = 'other' and 'classification.type' = 'other' and "classification.identifier" = 'malware' 4) IntelMQ does not support this use case In cases 1) and 2) "classification.identifier" could be used to specify what the event is about, e.g. "hash", or the malware family. I'm currently in favor of option 2), as we can keep the meaning of "Malicious Code" in sync with the RSIT and still support the use-case sufficiently. But my opinion could change during the discussion :) Do you see any more options than I listed above? What do you favor? best regards Sebastian [0]: https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/… [1]: https://intelmq.readthedocs.io/en/latest/dev/data-harmonization.html#classi… [2]: https://www.trusted-introducer.org/Incident-Classification-Taxonomy.pdf [3]: https://github.com/certtools/intelmq/blob/f7507ca2643fe8ddb3817c9be1209504e… [4]: https://github.com/certtools/intelmq/pull/1745 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

The IntelMQ 2.3.0 Release candidate waits to be tested
by Sebastian Wagner 22 Feb '21

22 Feb '21

Dear devs, We are in the final sprint of releasing the feature release version 2.3.0 and we are actively looking for testers! Ideally, we'd like to finish the 2.3.0 version at end of February. As Python 3.6 is required, support for Debian 9 and Ubuntu 16.04 had to be dropped. The list of supported operating systems is now: * CentOS 7 and 8 (with EPEL activated) * Debian 10 * Fedora 32, 33, Rawhide * openSUSE Leap 15.1, 15.2, Tumbleweed * Ubuntu: 18.04, 20.04, 20.10 (with universe activated) * Docker Engine: 18.x or higher The other major change is the split of the IntelMQ Manager into a separate API as backend (Thanks Birger) and the switch to Python for the latter (Thanks SUNET and Intevation). And thanks for all the numerous contributors who contributed greatly to the majorly extended feature set, by providing code, bug reports and other inputs. For all the other changes, please have a look at the changelog and the news file in the repository. I will give a more detailed overview of the changes in the final 2.3.0 announcement. https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md https://github.com/certtools/intelmq/blob/develop/NEWS.md To install the release candidate version from git, use the "maintenance" branch or 2.3.0.rc1 Tag (or if you are looking for more adventures, the "develop" branch,which we will use for the development of 3.0 now). For installation via pip, use `pip install --pre intelmq`, the version is called 2.3.0rc1 there. And for the packages, use the unstable repository: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… For the new beta docker image (Thanks Sebastian Waldbauer), install certat/intelmq-full: https://hub.docker.com/r/certat/intelmq-full Also summarized here: https://intelmq.readthedocs.io/en/latest/dev/guide.html#testing-pre-releases best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Advice for setting up tests for bot chains
by Mika Silander 19 Feb '21

19 Feb '21

Hi, While writing tests to individual bots seems quite straightforward, what would be the recommended way for writing (unit?) tests for a chain of bots? Ideally, I'd like to feed in individual test events to the first bot and only check what is returned by the last bot (event or whatever the expected outcome should be). I suppose there are many bad and clumsy ways of doing this so that's why I air this question on the dev list. Thanks for all help in advance, Mika

4 6

Bot behaviour in case of unrecoverable errors
by Mika Silander 16 Feb '21

16 Feb '21

Hi, Trying to assess what safeguards are sufficient: what happens when a bot has some internal failure and it "dies"? Will intelmq restart the bot automatically or will it be up to the admin of intelmq to manually restart it? And if automatic restarts is the norm, how could one stop the bot from processing new incoming messages if say, X consecutive failures like these have happened within the time frame of the last 5 minutes? By writing some log entries at bot startup and then making the bot itself analyze the log at every restart? I'm trying to make sure a burst of erroneous/malformed events are not accidentally forwarded by a malfunctioning or partially functioning bot. Cheers, Mika

2 6

Event rate limiting
by Mika Silander 28 Jan '21

28 Jan '21

Hi, Maybe this is answered somewhere already but I wasn't able to find any hints and I've just recently started developing bots so here's one for the list: I'd need to limit the rate at which events are processed to avoid swamping a back end system with requests. Big bursts of events should remain in the inqueue of such a rate limiting bot, and then be forwarded at a configurable rate e.g. X events/minute. Afaik, by default, bots attempt to process all the events in their inqueue without delay. How should one go about to create such a bot? Is it safe to put the bot process (=operating system process) to sleep for N seconds from within the bot's process method and then proceed, can we do busy-looping (not nice!), or, some other solution? Br, Mika

3 5

IEP01: IntelMQ Configuration Handling
by Birger Schacht 26 Jan '21

26 Jan '21

Dear IntelMQ developers and users, below are a couple of ideas how to (hopefully) make configuration of IntelMQ easier. Feel free to give feedback, voice concerns or simply ask if there is something unclear. We plan to evaluate the feedback that emerged in two weeks (after the christmas holidays). # IntelMQ Configuration Handling (IntelMQ Enhancement Proposal 01) ## Format ### JSON At the moment, the configuration format of IntelMQ is JSON[^1]. It is parsed using the Python json library, which is part of the Python Standard Library. The downside of JSON is, that is is hard to read and and write for humans and it cannot contain comments. [^1]: https://docs.python.org/3/library/index.html ### YAML There is a proposal[^2] to use YAML as the default configuration format. YAML provides way better readability for humans and YAML supports single line comments. There are two Python YAML libraries out there, the one being PyYAML[^3] and the other being ruamel.yaml[^4]. The former is a project by the YAML project itself. The latter is a fork of the former and had much more activity over the years and better support of the standard. It seems that pyyaml caught up in the last few years. We don't need any edge cases, so both libraries would be good for configuration files. According to this issue[^5] pyyaml does not support “editing YAML whilst maintaining comments”, which might be a deal breaker, but this issue is from 2016, this might have changed. On the other hand, IntelMQ does not edit configuration at the moment. pyyaml and ruamel.yaml are available as package in all relevant Linux distributions. [^2]: https://github.com/gethvi/intelmq/blob/ideas/docs/Ideas.md#changing-configu… [^3]: https://pyyaml.org/ [^4]: https://yaml.readthedocs.io/en/latest/ [^5]: https://github.com/yaml/pyyaml/issues/46 ### INI The Python Standard Library also ships configparser[^6], which is a “configuration language which provides a structure similar to what’s found in Microsoft Windows INI files”. The files can contain comments, it comes with a [DEFAULT] section, which can be used for default values and the configuration files can contain variables. One downside is that all the configurations are Strings, which means we would have to do parsing ourself. [^6]: https://docs.python.org/3/library/configparser.html ### toml Tom's Obvious, Minimal Language is another contender for the role of IntelMQs configuration file format. It looks similar to the INI file format, but comes with various data types. It also allows comments. There is a Python library[^7] that seems to be very active. toml is also used as the format for the proposed pyproject.toml file and by the rust community for their package configuration files. toml's syntax for dictionaries is hard to read/write, harder than with JSON. [^7]: https://pypi.org/project/toml/ ### Further information * The summary on file formats on the PEP518 proposition: https://www.python.org/dev/peps/pep-0518/#other-file-formats * At the moment we are leaning towards YAML. Regarding the library, we would choose ruamel.yaml, because it seems to have a more active upstream and it can retain comments when it modifies a yaml file. ## Storage This part is about the question where do we store the configuration?. The ideas document[^8] on GitHub already proposes to remove the pipeline.conf and specifying the destination pipelines in the individual bot configuration part. The declaration of the source queue can be dropped then as well, as it follows a rule anyway. In addition to that, to make the setup of IntelMQ easier, the defaults.conf should be dropped. Default values should be set in the Bot classes respectively in the IntelMQ process managers, but there is no need for a separate file. Another question is, if every bot should have their own configuration file. Some users wish to be able to start a bot without having to rely on IntelMQ, but at the moment, the bot gets the configuration from IntelMQ's runtime.conf. If we want to support the request to be able to pass individual configurations to bots, we could allow users to pass a separate configuration file to the bot (i.e. using `-c /path/to/config.$ext`). If that file is not set or does not contain the bots id, it is ignored and IntelMQ's runtime.conf is used as usual. If it does exists, the global runtime.conf is still parsed (if it exists - it should also be possible to run a bot without a runtime.conf) but only the values that are not set in the individual configuration file are considered. This individual configuration file would also allow a bot to be run in a docker environment without having to set any environment variables. This would make configuration handling probably easier, because then configuration settings could be stored in a file (and managed by a configuration management system) and the configuration file could contain comments. Proposal: * IntelMQ gets one global configuration file for all the bots and the pipeline.conf will be removed * This global configuration file is `${PREFIX}/etc/intelmq/intelmq.$ext`. If it does not exists or does not define any bots, IntelMQ should exit gracefully. The file extension depends on the chosen format. * The global configuration file contains an array of bot configurations with bot-ids as keys. * Every bot reads the global configuration file and extracts their own settings (as usual). * Every bot handles 0 to n `-c /path/to/configurationfile.$ext` flags, which are treated the same way as the global configuration file. The further ahead the configuration file in the commandline, the stronger the content (this allows us to have multiple non-global configuration files (i.e. for multiple groups)) Example: ``` > botcommand bot-id -c /etc/bots/botname.$ext -c /etc/bots/groups/group_foo.$ext ``` * Every bot also consults the environment and the values that are set their overwrite the values in any configuration file * There are also configuration files which list settings that are not bot specific, i.e. via a reserved key default (successor of the defaults.conf file) or group:id, those are also handled like other configuration files, but the bot does not compare its name to the key of the configuration. All the evaluated configuration formats provide the possibility to arrange the configuration parameters in hierarchies. To make the configuration files more readable, IntelMQ should make use of this hierarchy instead of denoting the different hierarchy levels with underscores. So instead of writing `http_proxy` the http parameter would have a childparameter proxy. For backwards compatibility and cases where the underscore does not imply hierarchy, the underscore notation will still work. In addition, IntelMQ should also make use of environment variables - those are still denoted using an underscore as delimiter and are prepended with `INTELMQ`: `INTELMQ_HTTP_PROXY`. [^8]: https://github.com/gethvi/intelmq/blob/ideas/docs/Ideas.md ### Caveats There are configuration settings, that do not really concern the bot- for example the type of process manager, that should be used to run the bot. In an ideal setup, the bot should be totally indifferent as to if it runs in a Docker container, on bare metal, in a SystemD unit file or with SupervisorD. This decision should only concern the tool managing all the bots (intelmqctl or in the future intelmq-api (which at the moment uses intelmqctl)). Another example is the enabled setting. At the moment, those are part of the individual bot configuration, but it might make sense to move them to a management.conf configuration file which is only for managing the individual bots, but not for configuring their parameters (this file would then also (for every bot) have a field that lists the configuration files the bot should consider when reading its configuration). On the other hand, this might make the configuration more complex again, now that we are trying to merge pipeline.conf and runtime.conf. We could also decide to make those configuration settings be part of the global configuration file, given that the individual bots should anyway simply ignore settings they do not know how to handle. ### Overriding by command line parameters If needed, a user can override specific bot settings using the -p switch (i.e. `-p redis_cache=example.com`). This should be easy to implement, in the best case scenario this is only one line of additional code in the Bot class. ### Examples A global configuration file with multiple bots /etc/intelmq/intelmq.yml ``` - shodan1: module: intelmq.bots.collectors.shodan.collector - mylittlebot23: module: intelmq.bots.expert.asn_lookup.expert http: proxy: http://myproxy.tld:80 - fop1: module: intelmq.bots.outputs.file output: filename: /dev/null ``` We can run a bot with intelmq-bot shodan1 which is the same as `intelmq-bot shodan1 -c /etc/intelmq/intelmq.yml` Another configuration file with multiple bots /root/intelmq-bots-managed-by-root: ``` - shodan2: module: intelmq.bots.collectors.shodan.collector - fop1: module: intelmq.bots.outputs.file output: filename: /var/log/fop1.log ``` We can run a bot with `intelmq-bot shodan2 -c /root/intelmq-bots-managed-by-root`; We can run a bot using `intelmq-bot fop1 -c /root/intelmq-bots-managed-by-root` which would then send output to `/var/log/fop1.log`. A configuration for a group in /etc/intelmq/collector-group.yml ``` - group:collectors http: proxy: http://thirdparty.proxy.tld:9000 ``` We can run a bot with intelmq-bot `mylittlebot23 -c /etc/intelmq/collector-group.yml` which uses the third-party proxy. ## Internal handling Every bot class defines their own settings as class variables. Every class variable has to be typed. Every class variable should be set to a reasonable default, otherwise None. The init of the (abstract) Bot class should load all the relevant configuration files and then overwrite the settings. If a setting is still None and the value of the setting is vital for the functionality of the bot, the bot should stop and emit a meaningful error message. For the most common types of settings, there should be Python objects to check the values. Value checking should only be done after all the configurations are merged. -- // Birger Schacht <schacht(a)cert.at> // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

6 14

Request for Testing and Feedback: Shadowserver API collector and parser
by Birger Schacht 19 Jan '21

19 Jan '21

Hi, Shadowserver nowadays not only sends out reports by Mail but also provides an API [0] to query reports. We recently implemented a Shadowserver Reports API collector bot [1] that downloads the reports from the API and feeds them into IntelMQ. To parse the downloaded feeds (in JSON format) we built upon the existing Shadowserver parsing logic and created a JSON parser [2] that's meant to be used together with the Shadowserver Reports API collector bot. If anyone is interested in testing the collector and parser that would be great- any feedback, bug reports or improvements are highly appreciated. To use the collector you will need a Report API Key, which you can request on [3]. cheers, Birger [0] https://www.shadowserver.org/what-we-do/network-reporting/api-reports-query/ [1] https://intelmq.readthedocs.io/en/latest/user/bots.html#shadowserver-report… [2] https://intelmq.readthedocs.io/en/latest/user/bots.html#shadowserver [3] https://www.shadowserver.org/contact/ -- // Birger Schacht <schacht(a)cert.at> // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

← Newer
1
...
9
10
11
12
13
14
15
...
26
Older →

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev