IntelMQ-dev February 2021

intelmq-dev@lists.cert.at

5 participants
5 discussions

Default crontab values for local lookup-databases update
by Sebastian Wagner 26 Feb '21

26 Feb '21

Dear devs, Thanks to Filip (CZ.NIC) IntelMQ comes now with an update mechanism for local lookup databases like TOR exit nodes, IP address to ASN ("ASN Lookup") and Maxmind GeoIP (IP address geolocation)[0]. Also, IntelMQ ships with update scripts for cron which are included in the deb/rpm packages as well. Currently the update scripts are scheduled as follows[1]: * TOR nodes: once per day. The database is very small. * Maxmind GeoIP: Once per week. Changes are scarce. * ASN Lookup: Every two hours. Big database, but the data is vital for subsequent routing of incidents. I'd like to hear your opinion if the default values are ok to ship with 2.3.0, especially for the last one. best regards, Sebastian [0]: https://github.com/certtools/intelmq/pull/1524 [1]: https://github.com/certtools/intelmq/blob/24f2355d0c549021a713c938d1d69a521… -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 2

Classification of malware itself in IntelMQ
by Sebastian Wagner 22 Feb '21

22 Feb '21

Dear IntelMQ community, sorry for cross-posting, but I think this topic should be discussed in a wider group. IntelMQ always followed the Reference Security Incident Taxonomy (short: RSIT)[0] and its predecessor for its 'classification.taxonomy/type' fields. The Classification column in the RSIT corresponds to our "classification.taxonomy" field, and the RSIT's second column (currently called Incident examples) corresponds to our "classification.type" field. "classification.identifier" is an optional third level free-text field to give more specific context.[1] Due to historical reasons and changes on both sides - IntelMQ as well as the RSIT -, IntelMQ's classification scheme deviated a bit from the RSIT over time. I'm working on aligning them again for 3.0, which works straightforward in most cases. But for one case, I need your input. The predecessor of the RSIT (the eCSIRT.net taxonomy)[2] used the malicious code taxonomy differently: To classify malware itself into categories, like virus, worm, trojan, etc. The RSIT never did that, as classifying malware is never unambiguous and there are plenty of existing classification scheme out there, which do this already. Also, the focus of the RSIT is different, as it classifies the incidents/events, not malware samples. And for this reason, IntelMQ had (until < 3.0.0) the classification.type "malware" in IntelMQ. Most of the usages were wrong anyway, and should have been infected-device, malware-distribution or something else anyway. There is only one usage in IntelMQ, which can not be changed. And that one is really about malware itself (or: the hashes of samples) as used in the GitHub Feed parser[3] and the FireEye Parser[4]. But the issue is more generic, as we need to decide anyway, how we want to deal with such malware-IoCs. A malware (hash) does not fit into the RSIT. It's neither an Infected System, a C2 Server, Malware Distribution nor Malware Configuration. It's just a malware (hash). I see four options: 1) Deviate from the RSIT and just use 'classification.taxonomy' = 'Malicious Code' and 'classification.type' = 'malware' 2) Deviate slightly less from the RSIT and use 'classification.taxonomy' = 'other' and 'classification.type' = 'malware' 3) Adhere strictly to the RSIT and use 'classification.taxonomy' = 'other' and 'classification.type' = 'other' and "classification.identifier" = 'malware' 4) IntelMQ does not support this use case In cases 1) and 2) "classification.identifier" could be used to specify what the event is about, e.g. "hash", or the malware family. I'm currently in favor of option 2), as we can keep the meaning of "Malicious Code" in sync with the RSIT and still support the use-case sufficiently. But my opinion could change during the discussion :) Do you see any more options than I listed above? What do you favor? best regards Sebastian [0]: https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/… [1]: https://intelmq.readthedocs.io/en/latest/dev/data-harmonization.html#classi… [2]: https://www.trusted-introducer.org/Incident-Classification-Taxonomy.pdf [3]: https://github.com/certtools/intelmq/blob/f7507ca2643fe8ddb3817c9be1209504e… [4]: https://github.com/certtools/intelmq/pull/1745 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

The IntelMQ 2.3.0 Release candidate waits to be tested
by Sebastian Wagner 22 Feb '21

22 Feb '21

Dear devs, We are in the final sprint of releasing the feature release version 2.3.0 and we are actively looking for testers! Ideally, we'd like to finish the 2.3.0 version at end of February. As Python 3.6 is required, support for Debian 9 and Ubuntu 16.04 had to be dropped. The list of supported operating systems is now: * CentOS 7 and 8 (with EPEL activated) * Debian 10 * Fedora 32, 33, Rawhide * openSUSE Leap 15.1, 15.2, Tumbleweed * Ubuntu: 18.04, 20.04, 20.10 (with universe activated) * Docker Engine: 18.x or higher The other major change is the split of the IntelMQ Manager into a separate API as backend (Thanks Birger) and the switch to Python for the latter (Thanks SUNET and Intevation). And thanks for all the numerous contributors who contributed greatly to the majorly extended feature set, by providing code, bug reports and other inputs. For all the other changes, please have a look at the changelog and the news file in the repository. I will give a more detailed overview of the changes in the final 2.3.0 announcement. https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md https://github.com/certtools/intelmq/blob/develop/NEWS.md To install the release candidate version from git, use the "maintenance" branch or 2.3.0.rc1 Tag (or if you are looking for more adventures, the "develop" branch,which we will use for the development of 3.0 now). For installation via pip, use `pip install --pre intelmq`, the version is called 2.3.0rc1 there. And for the packages, use the unstable repository: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… For the new beta docker image (Thanks Sebastian Waldbauer), install certat/intelmq-full: https://hub.docker.com/r/certat/intelmq-full Also summarized here: https://intelmq.readthedocs.io/en/latest/dev/guide.html#testing-pre-releases best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Advice for setting up tests for bot chains
by Mika Silander 19 Feb '21

19 Feb '21

Hi, While writing tests to individual bots seems quite straightforward, what would be the recommended way for writing (unit?) tests for a chain of bots? Ideally, I'd like to feed in individual test events to the first bot and only check what is returned by the last bot (event or whatever the expected outcome should be). I suppose there are many bad and clumsy ways of doing this so that's why I air this question on the dev list. Thanks for all help in advance, Mika

4 6

Bot behaviour in case of unrecoverable errors
by Mika Silander 16 Feb '21

16 Feb '21

Hi, Trying to assess what safeguards are sufficient: what happens when a bot has some internal failure and it "dies"? Will intelmq restart the bot automatically or will it be up to the admin of intelmq to manually restart it? And if automatic restarts is the norm, how could one stop the bot from processing new incoming messages if say, X consecutive failures like these have happened within the time frame of the last 5 minutes? By writing some log entries at bot startup and then making the bot itself analyze the log at every restart? I'm trying to make sure a burst of erroneous/malformed events are not accidentally forwarded by a malfunctioning or partially functioning bot. Cheers, Mika

2 6

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev February 2021