- IntelMQ-dev - CERT.at

Re: [IntelMQ-dev] [IntelMQ-users] Classification of malware itself in IntelMQ
by Sebastian Wagner 08 Mar '21

08 Mar '21

Hi, Thanks Filip for sharing your opinion. As there were no further comments on this topic, I'll implement option 2. Sebastian On 2/24/21 10:32 AM, Filip Pokorny wrote: > Hi Sebastian, > >> I'm currently in favor of option 2), as we can keep the meaning of "Malicious Code" in sync with the RSIT and still support the use-case sufficiently. But my opinion could change during the discussion :) > I agree. This combined with "malware.name", "malware.version" and > "malware.hash" seems good enough to describe malware for the IntelMQ > use-case. > > Best Regards, > Filip > > > On 2/22/21 11:24 AM, Sebastian Wagner wrote: >> Dear IntelMQ community, >> >> sorry for cross-posting, but I think this topic should be discussed in a >> wider group. >> >> IntelMQ always followed the Reference Security Incident Taxonomy (short: >> RSIT)[0] and its predecessor for its 'classification.taxonomy/type' >> fields. The Classification column in the RSIT corresponds to our >> "classification.taxonomy" field, and the RSIT's second column (currently >> called Incident examples) corresponds to our "classification.type" >> field. "classification.identifier" is an optional third level free-text >> field to give more specific context.[1] >> >> Due to historical reasons and changes on both sides - IntelMQ as well as >> the RSIT -, IntelMQ's classification scheme deviated a bit from the RSIT >> over time. I'm working on aligning them again for 3.0, which works >> straightforward in most cases. But for one case, I need your input. >> >> The predecessor of the RSIT (the eCSIRT.net taxonomy)[2] used the >> malicious code taxonomy differently: To classify malware itself into >> categories, like virus, worm, trojan, etc. The RSIT never did that, as >> classifying malware is never unambiguous and there are plenty of >> existing classification scheme out there, which do this already. Also, >> the focus of the RSIT is different, as it classifies the >> incidents/events, not malware samples. >> >> And for this reason, IntelMQ had (until < 3.0.0) the classification.type >> "malware" in IntelMQ. Most of the usages were wrong anyway, and should >> have been infected-device, malware-distribution or something else >> anyway. There is only one usage in IntelMQ, which can not be changed. >> And that one is really about malware itself (or: the hashes of samples) >> as used in the GitHub Feed parser[3] and the FireEye Parser[4]. But the >> issue is more generic, as we need to decide anyway, how we want to deal >> with such malware-IoCs. >> >> A malware (hash) does not fit into the RSIT. It's neither an Infected >> System, a C2 Server, Malware Distribution nor Malware Configuration. >> It's just a malware (hash). I see four options: >> >> 1) Deviate from the RSIT and just use 'classification.taxonomy' = >> 'Malicious Code' and 'classification.type' = 'malware' >> 2) Deviate slightly less from the RSIT and use 'classification.taxonomy' >> = 'other' and 'classification.type' = 'malware' >> 3) Adhere strictly to the RSIT and use 'classification.taxonomy' = >> 'other' and 'classification.type' = 'other' and >> "classification.identifier" = 'malware' >> 4) IntelMQ does not support this use case >> >> In cases 1) and 2) "classification.identifier" could be used to specify >> what the event is about, e.g. "hash", or the malware family. >> >> I'm currently in favor of option 2), as we can keep the meaning of >> "Malicious Code" in sync with the RSIT and still support the use-case >> sufficiently. But my opinion could change during the discussion :) >> >> Do you see any more options than I listed above? What do you favor? >> >> best regards >> Sebastian >> >> [0]: >> https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/… >> [1]: >> https://intelmq.readthedocs.io/en/latest/dev/data-harmonization.html#classi… >> [2]: https://www.trusted-introducer.org/Incident-Classification-Taxonomy.pdf >> [3]: >> https://github.com/certtools/intelmq/blob/f7507ca2643fe8ddb3817c9be1209504e… >> [4]: https://github.com/certtools/intelmq/pull/1745 >> >> >> -- >> // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 >> // CERT Austria - https://www.cert.at/ >> // Eine Initiative der nic.at GmbH - https://www.nic.at/ >> // Firmenbuchnummer 172568b, LG Salzburg >> >> >> _______________________________________________ >> IntelMQ-dev mailing list >> IntelMQ-dev(a)lists.cert.at >> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev >> -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ & API & Manager Version 2.3.0!
by Sebastian Wagner 04 Mar '21

04 Mar '21

Dear community, Another important intermediate step on the way to IntelMQ 3.0 is completed - IntelMQ 2.3.0 is really final as of today. Many thanks to all the contributors and supporters around the world - the major changes would never be possible without you! One thing you will immediately notice its a completely new component: the IntelMQ API. It originates from the IntelMQ Manager, but is a complete rewrite of it's backend in Python (finally!) financed by SUNET and realised by Intevation. We have then split the Backend off into a separate API. This means, that to run the Manager, you need the API as well. The installation instructions: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade instructions: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html All packages have been published to pypi, the deb/rpm-repositories and dockerhub. You can read a summary of the most important changes here: https://cert.at/en/blog/2021/3/intelmq-230-api-docker-shadowserver-reports-… The new or heavily changed bots are: * CZ.nic HAAS and PROKI Parsers, by Filip Pokorný and Edvard Rejthar (CSIRT.CZ) * ESET Collector and Parser, by Mikk Margus Möll (CERT.EE) * Kafka Collector, by Birger Schacht (CERT.at) * Key-Value Parser, by Karl-Johan Karlsson (Linköping University) * Request Tracker Output, by Marius Urkis (NRDCS.LT) * Shadowserver Reports API and JSON Parser, by Birger Schacht (CERT.at) * Splunk Saved Search Expert, by Karl-Johan Karlsson (Linköping University) * Threshold Expert, by Karl-Johan Karlsson (Linköping University) * Shadowserver CSV & JSON Parser: Support for the feeds MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS and fixes for existing feed mappings, by Sebastian Waldbauer and Sebastian Wagner (CERT.at) * HTTP collector: PGP signature check functionality, by sinus-x * Several Experts (1, 2, 3, 4): Integrated local database update mechanisms, by Filip Pokorný (CSIRT.CZ) Please find below the full changelog. best regards Sebastian IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the minimum supported Python version is 3.6. ### Configuration ### Core - `intelmq.lib.bot`: - `ParserBot.recover_line_json_stream`: Make `line` parameter optional, as it is not needed for this method (by Sebastian Wagner). - `Bot.argparser`: Added class method `_create_argparser` (returns `argparse.ArgumentParser`) for easy command line arguments parsing (PR#1586 by Filip Pokorný). - Runtime configuration does not necessarily need a parameter entry for each block. Previously at least an empty block was required (PR#1604 by Filip Pokorný). - Allow setting the pipeline host and the Redis cache host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer). - Better logging message for SIGHUP handling if the handling of the signal is not delayed (by Sebastian Wagner). - `intelmq.lib.upgrades`: - Add upgrade function for removal of *HPHosts Hosts file* feed and `intelmq.bots.parsers.hphosts` parser (#1559, by Sebastian Wagner). - `intelmq.lib.exceptions`: - `PipelineError`: Remove unused code to format exceptions (by Sebastian Wagner). - `intelmq.lib.utils`: - `create_request_session_from_bot`: - Changed bot argument to optional, uses defaults.conf as fallback, renamed to `create_request_session`. Name `create_request_session_from_bot` will be removed in version 3.0.0 (PR#1524 by Filip Pokorný). - Fixed setting of `http_verify_cert` from defaults configuration (PR#1758 by Birger Schacht). - `log`: Use `RotatingFileHandler` for allow log file rotation without external tools (PR#1637 by Vasek Bruzek). - `intelmq.lib.harmonization`: - The `IPAddress` type sanitation now accepts integer IP addresses and converts them to the string representation (by Sebastian Wagner). - `DateTime.parse_utc_isoformat`: Add parameter `return_datetime` to return `datetime` object instead of string ISO format (by Sebastian Wagner). - `DateTime.convert`: Fix `utc_isoformat` format, it pointed to a string and not a function, causing an exception when used (by Sebastian Wagner). - `DateTime.from_timestamp`: Ensure that time zone information (`+00:00`) is always present (by Sebastian Wagner). - `DateTime.__parse` now handles OverflowError exceptions from the dateutil library, happens for large numbers, e.g. telehpone numbers (by Sebastian Wagner). - `intelmq.lib.upgrades`: - Added upgrade function for CSV parser parameter misspelling (by Sebastian Wagner). - Check for existence of collector and parser for the obsolete Malware Domain List feed and raise warning if found (#1762, PR#1771 by Birger Schacht). ### Development - `intelmq.bin.intelmq_gen_docs`: - Add bot name to the resulting feed documentation (PR#1617 by Birger Schacht). - Merged into `docs/autogen.py` (PR#1622 by Birger Schacht). ### Bots #### Collectors - `intelmq.bots.collectors.eset.collector`: Added (PR#1554 by Mikk Margus Möll). - `intelmq.bots.collectors.http.collector_http`: - Added PGP signature check functionality (PR#1602 by sinus-x). - If status code is not 2xx, the request's and response's headers and body are logged in debug logging level (#1615, by Sebastian Wagner). - `intelmq.bots.collectors.kafka.collector`: Added (PR#1654 by Birger Schacht, closes #1634). - `intelmq.bots.collectors.xmpp.collector`: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht). - `intelmq.bots.collectors.shadowserver.collector_api`: - Added (#1683, PR#1700 by Birger Schacht). - Change file names in the report to `.json` instead of the original and wrong `.csv` (PR#1769 by Sebastian Wagner). - `intelmq.bots.collectors.mail`: Add content of the email's `Date` header as `extra.email_date` to the report in all email collectors (PR#1749 by aleksejsv and Sebastian Wagner). - `intelmq.bots.collectors.http.collector_http_stream`: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner). - `intelmq.bots.collectors.shodan.collector_stream`: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner). - `intelmq.bots.collectors.twitter.collector_twitter`: - Proper input validation in URLs using urllib. CWE-20, found by GitHub's CodeQL (PR#1754 by Sebastian Wagner). - Limit replacement ("pastebin.com", "pastebin.com/raw") to a maximum of one (PR#1754 by Sebastian Wagner). #### Parsers - `intelmq.bots.parsers.eset.parser`: Added (PR#1554 by Mikk Margus Möll). - Ignore invalid "NXDOMAIN" IP addresses (PR#1573 by Mikk Margus Möll). - `intelmq.bots.parsers.hphosts`: Removed, feed is unavailable (#1559, by Sebastian Wagner). - `intelmq.bots.parsers.cznic.parser_haas`: Added (PR#1560 by Filip Pokorný and Edvard Rejthar). - `intelmq.bots.parsers.cznic.parser_proki`: Added (PR#1599 by sinus-x). - `intelmq.bots.parsers.key_value.parser`: Added (PR#1607 by Karl-Johan Karlsson). - `intelmq.bots.parsers.generic.parser_csv`: Added new parameter `compose_fields` (by Sebastian Wagner). - `intelmq.bots.parsers.shadowserver.parser_json`: Added (PR#1700 by Birger Schacht). - `intelmq.bots.parsers.shadowserver.config`: - Fixed mapping for Block list feed to accept network ranges in CIDR notation (#1720, PR#1728 by Sebastian Waldbauer). - Added mapping for new feed MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS (#1716, #1726, #1733, PR#1732, PR#1735, PR#1736 by Sebastian Waldbauer). - Ignore value `0` for `source.asn` and `destination.asn` in all mappings to avoid parsing errors (PR#1769 by Sebastian Wagner). - `intelmq.bots.parsers.abusech.parser_ip`: Adapt to changes in the Feodo Tracker Botnet C2 IP Blocklist feed (PR#1741 by Thomas Bellus). - `intelmq.bots.parsers.malwaredomainlist`: Removed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht). #### Experts - `intelmq.bots.experts.rfc1918.expert`: - Add support for ASNs (PR#1557 by Mladen Markovic). - Speed improvements. - More output in debug logging mode (by Sebastian Wagner). - Checks parameter length on initialization and in check method (by Sebastian Wagner). - `intelmq.bots.experts.gethostbyname.expert`: - Added parameter `fallback_to_url` and set to True (PR#1586 by Edvard Rejthar). - Added parameter `gaierrors_to_ignore` to optionally ignore other `gethostbyname` errors (#1553). - Added parameter `overwrite` to optionally overwrite existing IP addresses (by Sebastian Wagner). - `intelmq.bots.experts.asn_lookup.expert`: - Added `--update-database` option (PR#1524 by Filip Pokorný). - The script `update-asn-data` is now deprecated and will be removed in version 3.0. - `intelmq.bots.experts.maxmind_geoip.expert`: - Added `--update-database` option (PR#1524 by Filip Pokorný). - Added `license_key` parameter (PR#1524 by Filip Pokorný). - The script `update-geoip-data` is now deprecated and will be removed in version 3.0. - `intelmq.bots.experts.tor_nodes.expert`: - Added `--update-database` option (PR#1524 by Filip Pokorný). - The script `update-tor-nodes` is now deprecated and will be removed in version 3.0. - `intelmq.bots.experts.recordedfuture_iprisk.expert`: - Added `--update-database` option (PR#1524 by Filip Pokorný). - Added `api_token` parameter (PR#1524 by Filip Pokorný). - The script `update-rfiprisk-data` is now deprecated and will be removed in version 3.0. - Added `intelmq.bots.experts.threshold` (PR#1608 by Karl-Johan Karlsson). - Added `intelmq.bots.experts.splunk_saved_search.expert` (PR#1666 by Karl-Johan Karlsson). - `intelmq.bots.experts.sieve.expert`: - Added possibility to give multiple queue names for the `path` directive (#1462, by Sebastian Wagner). - Added possibility to run actions without filtering expression (#1706, PR#1708 by Sebastian Waldbauer). - Added datetime math operations (#1680, PR#1696 by Sebastian Waldbauer). - `intelmq.bots.experts.maxmind_geoip.expert`: - Fixed handing over of `overwrite` parameter to `event.add` (PR#1743 by Birger Schacht). #### Outputs - `intelmq.bots.outputs.rt`: Added Request Tracker output bot (PR#1589 by Marius Urkis). - `intelmq.bots.outputs.xmpp.output`: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht). - `intelmq.bots.outputs.smtp.output`: Fix sending to multiple recipients when recipients are defined by event-data (#1759, PR#1760 by Sebastian Waldbauer and Sebastian Wagner). ### Documentation - Feeds: - Add ESET URL and Domain feeds (by Sebastian Wagner). - Remove unavailable *HPHosts Hosts file* feed (#1559 by Sebastian Wagner). - Added CZ.NIC HaaS feed (PR#1560 by Filip Pokorný and Edvard Rejthar). - Added CZ.NIC Proki feed (PR#1599 by sinus-x). - Updated Abuse.ch URLhaus feed (PT#1572 by Filip Pokorný). - Added CERT-BUND CB-Report Malware infections feed (PR#1598 by sinus-x and Sebastian Wagner). - Updated Turris Greylist feed with PGP verification information (by Sebastian Wagner). - Fixed parsing of the `public` field in the generated feeds documentation (PR#1641 by Birger Schacht). - Change the `rate_limit` parameter of some feeds from 2 days (129600 seconds) to one day (86400 seconds). - Update the cAPTure Ponmocup Domains feed documentation (PR#1574 by Filip Pokorný and Sebastian Wagner). - Added Shadowserver Reports API (by Sebastian Wagner). - Change the `rate_limit` parameter for many feeds from 2 days to the default one day (by Sebastian Wagner). - Removed Malware Domain List feed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht). - Bots: - Enhanced documentation of RFC1918 Expert (PR#1557 by Mladen Markovic and Sebastian Wagner). - Enhanced documentation of SQL Output (PR#1620 by Edvard Rejthar). - Updated documentation for MaxMind GeoIP, ASN Lookup, TOR Nodes and Recorded Future experts to reflect new `--update-database` option (PR#1524 by Filip Pokorný). - Added documentation for Shadowserver API collector and parser (PR#1700 by Birger Schacht and Sebastian Wagner). - Add n6 integration documentation (by Sebastian Wagner). - Moved 'Orphaned Queues' section from the FAQ to the intelmqctl documentation (by Sebastian Wagner). - Generate documentation using Sphinx (PR#1622 by Birger Schacht). - The documentation is now available at https://intelmq.readthedocs.io/en/latest/ - Refactor documentation and fix broken syntax (#1639, PRs #1638 #1640 #1642 by Birger Schacht). - Integrate intelmq-manager and intelmq-api user documentation to provide unified documentation place (PR#1714 & PR#1714 by Birger Schacht). ### Packaging - Fix paths in the packaged logcheck rules (by Sebastian Wagner). - Build the sphinx documentation on package build (PR#1701 by Birger Schacht). - Ignore non-zero exit-codes for the `intelmqctl check` call in postinst (#1748, by Sebastian Wagner). ### Tests - Added tests for `intelmq.lib.exceptions.PipelineError` (by Sebastian Wagner). - `intelmq.tests.bots.collectors.http_collector.test_collector`: Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner). - `intelmq.tests.bots.outputs.restapi.test_output`: - Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner). - Add a test for checking the response status code (by Sebastian Wagner). - `intelmq.tests.bots.collectors.mail.test_collector_url`: Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner). - `intelmq.tests.bots.experts.ripe.test_expert`: Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner). - The test flag (environment variable) `INTELMQ_TEST_LOCAL_WEB` is no longer used (by Sebastian Wagner). - Added tests for `intelmq.harmonization.DateTime.parse_utc_isoformat` and `convert_fuzzy` (by Sebastian Wagner). - Move from Travis to GitHub Actions (PR#1707 by Birger Schacht). - `intelmq.lib.test`: - `test_static_bot_check_method` checks the bot's static `check(parameters)` method for any exceptions, and a valid formatted return value (#1505, by Sebastian Wagner). - `setUpClass`: Skip tests if cache was requests with `use_cache` member, but Redis is deactivated with the environment variable `INTELMQ_SKIP_REDIS` (by Sebastian Wagner). - `intelmq.tests.bots.experts.cymru_whois.test_expert`: - Switch from `example.com` to `ns2.univie.ac.at` for hopefully more stable responses (#1730, PR#1731 by Sebastian Waldbauer). - Do not test for exact expected values in the 6to4 network test, as the values are changing regularly (by Sebastian Wagner). - `intelmq.tests.bots.parsers.abusech`: Remove tests cases of discontinued feeds (PR#1741 by Thomas Bellus). - Activate GitHub's CodeQL Code Analyzing tool as GitHub Action (by Sebastian Wagner). ### Tools - `intelmqdump`: - Check if given queue is configured upon recovery (#1433, PR#1587 by Mladen Markovic). - `intelmqctl`: - `intelmq list queues`: `--sum`, `--count`, `-s` flag for showing total count of messages (#1408, PR#1581 by Mladen Markovic). - `intelmq check`: Added a possibility to ignore queues from the orphaned queues check (by Sebastian Wagner). - Allow setting the pipeline host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer). ### Contrib - EventDB: - Add SQL script for keeping track of the oldest inserted/update "time.source" information (by Sebastian Wagner). - Cron Jobs: The script `intelmq-update-data` has been renamed to `intelmq-update-database` (by Filip Pokorný). - Dropped utterly outdated contrib modules (by Sebastian Wagner): - ansible - vagrant - vagrant-ansible - logrotate: - Do not use the deprecated "copytruncate" option as intelmq re-opens the log anyways (by Sebastian Wagner). - Set file permissions to `0644` (by Sebastian Wagner). ### Known issues - Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952). - Corrupt dump files when interrupted during writing (#870). - CSV line recovery forces Windows line endings (#1597). - intelmqdump: Honor logging_path variable (#1605). - Timeout error in mail URL fetcher (#1621). - AMQP pipeline: get_queues needs to check vhost of response (#1746). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IEP02: Introduce IntelMQ Mixins
by Birger Schacht 03 Mar '21

03 Mar '21

Dear IntelMQ developers, below I am writing about an idea that might make contributing to IntelMQ easier- it is a small change, but could have implications for various bots: # Introduce IntelMQ Mixins (IntelMQ Enhancement Proposal 02) I would like to lower the bar for adding additional bots by introducing mixins. Mixins are a concept coming from OOP theory and it is closely related to multiple inheritance. I think it could be a way to avoid a lot of duplicate code by outsourcing various functionalities to specific classes. ## Current situation At the moment there are two types of bots that could make use of mixins. The first are bots that make HTTP requests. Currently we make that easier for bots by providing two methods, `create_request_session` in intelmq/lib/utils.py and `set_request_parameters` as part of the bot module (intelmq/lib/bot.py) itself. When the bot wants to use a request session, it first uses `set_request_parameters` and then uses the session object returned by `create_request_session` to make the request. I think its a bit confusing to have methods that are related in two different modules and I think the Bot class itself should not contain any code that is not used by all the bots. The second type of bots that could make use of mixins are bots that use a cache. There are a lot of those that all use the `intelmq.lib.cache` module which provides a Cache object to work with. ## Improvement using mixins Using a Python class we could move everything related to HTTP requests to a class called HttpMixin. A bot that wants to use functionality from this Mixin just would have to inherit from this class, i.e.: ``` class MySuperBot(CollectorBot, HttpMixin): ``` The MySuperBot class would then have all the relevant attributes to use session objects and it would also have a private `__session` attribute to work with. Another upside of this approach is, that it is possible to generate a list of all the attributes a Bot has write access to (i.e. http_proxy, http_username, ssl_client_cert...) Similarly we could introduce a `CacheMixin` that allow access to private `__cache` attribute. If MySuperBot does not only want to make HTTP requests, but also have some caching mechanisms, it could simply inherit from both classes: ``` class MySuperBot(CollectorBot, HttpMixin, CacheMixin): ``` I propose to create a new module `intelmq.lib.mixins` that should be the home for the mixins. I have created a POC that implements the HttpMixin and uses it in the HTTPCollectorBot: https://github.com/certtools/intelmq/tree/schacht/http-mixin Mixins are also used in Django to add functionality to their class based Views, for more details see https://docs.djangoproject.com/en/3.1/topics/class-based-views/mixins/ A longer article about inheritance, MRO, mixins and Python that also talks about the implementation in Django can be found on https://www.thedigitalcatonline.com/blog/2020/03/27/mixin-classes-in-python/ cheers, Birger -- // Birger Schacht <schacht(a)cert.at> // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Installing new bots from devel host to production host
by Mika Silander 02 Mar '21

02 Mar '21

Hi, Bothering you again with a probably simple question. I've created half a dozen of bots on a development machine and now I'd like to create an installation system, Makefile or whatever to install those bots + configurations to the production host. It appears to me that the section "Update" (https://intelmq.readthedocs.io/en/latest/dev/guide.html#id12) contains the recommended steps to follow and assumes that one has the development environment (below /opt/dev_intelmq) available on the same host where one intends to run the production set up (under /opt/intelmq), correct? So, what do you recommend? Should I copy over the entire /opt/dev_intelmq to the production host and then "Update" there according to the above instructions or are there other ways to just copy/install the relevant bots+configurations from below /opt/dev_intelmq to the production host? Best regards and thanks again, Mika

3 6

Default crontab values for local lookup-databases update
by Sebastian Wagner 26 Feb '21

26 Feb '21

Dear devs, Thanks to Filip (CZ.NIC) IntelMQ comes now with an update mechanism for local lookup databases like TOR exit nodes, IP address to ASN ("ASN Lookup") and Maxmind GeoIP (IP address geolocation)[0]. Also, IntelMQ ships with update scripts for cron which are included in the deb/rpm packages as well. Currently the update scripts are scheduled as follows[1]: * TOR nodes: once per day. The database is very small. * Maxmind GeoIP: Once per week. Changes are scarce. * ASN Lookup: Every two hours. Big database, but the data is vital for subsequent routing of incidents. I'd like to hear your opinion if the default values are ok to ship with 2.3.0, especially for the last one. best regards, Sebastian [0]: https://github.com/certtools/intelmq/pull/1524 [1]: https://github.com/certtools/intelmq/blob/24f2355d0c549021a713c938d1d69a521… -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 2

Classification of malware itself in IntelMQ
by Sebastian Wagner 22 Feb '21

22 Feb '21

Dear IntelMQ community, sorry for cross-posting, but I think this topic should be discussed in a wider group. IntelMQ always followed the Reference Security Incident Taxonomy (short: RSIT)[0] and its predecessor for its 'classification.taxonomy/type' fields. The Classification column in the RSIT corresponds to our "classification.taxonomy" field, and the RSIT's second column (currently called Incident examples) corresponds to our "classification.type" field. "classification.identifier" is an optional third level free-text field to give more specific context.[1] Due to historical reasons and changes on both sides - IntelMQ as well as the RSIT -, IntelMQ's classification scheme deviated a bit from the RSIT over time. I'm working on aligning them again for 3.0, which works straightforward in most cases. But for one case, I need your input. The predecessor of the RSIT (the eCSIRT.net taxonomy)[2] used the malicious code taxonomy differently: To classify malware itself into categories, like virus, worm, trojan, etc. The RSIT never did that, as classifying malware is never unambiguous and there are plenty of existing classification scheme out there, which do this already. Also, the focus of the RSIT is different, as it classifies the incidents/events, not malware samples. And for this reason, IntelMQ had (until < 3.0.0) the classification.type "malware" in IntelMQ. Most of the usages were wrong anyway, and should have been infected-device, malware-distribution or something else anyway. There is only one usage in IntelMQ, which can not be changed. And that one is really about malware itself (or: the hashes of samples) as used in the GitHub Feed parser[3] and the FireEye Parser[4]. But the issue is more generic, as we need to decide anyway, how we want to deal with such malware-IoCs. A malware (hash) does not fit into the RSIT. It's neither an Infected System, a C2 Server, Malware Distribution nor Malware Configuration. It's just a malware (hash). I see four options: 1) Deviate from the RSIT and just use 'classification.taxonomy' = 'Malicious Code' and 'classification.type' = 'malware' 2) Deviate slightly less from the RSIT and use 'classification.taxonomy' = 'other' and 'classification.type' = 'malware' 3) Adhere strictly to the RSIT and use 'classification.taxonomy' = 'other' and 'classification.type' = 'other' and "classification.identifier" = 'malware' 4) IntelMQ does not support this use case In cases 1) and 2) "classification.identifier" could be used to specify what the event is about, e.g. "hash", or the malware family. I'm currently in favor of option 2), as we can keep the meaning of "Malicious Code" in sync with the RSIT and still support the use-case sufficiently. But my opinion could change during the discussion :) Do you see any more options than I listed above? What do you favor? best regards Sebastian [0]: https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/… [1]: https://intelmq.readthedocs.io/en/latest/dev/data-harmonization.html#classi… [2]: https://www.trusted-introducer.org/Incident-Classification-Taxonomy.pdf [3]: https://github.com/certtools/intelmq/blob/f7507ca2643fe8ddb3817c9be1209504e… [4]: https://github.com/certtools/intelmq/pull/1745 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

The IntelMQ 2.3.0 Release candidate waits to be tested
by Sebastian Wagner 22 Feb '21

22 Feb '21

Dear devs, We are in the final sprint of releasing the feature release version 2.3.0 and we are actively looking for testers! Ideally, we'd like to finish the 2.3.0 version at end of February. As Python 3.6 is required, support for Debian 9 and Ubuntu 16.04 had to be dropped. The list of supported operating systems is now: * CentOS 7 and 8 (with EPEL activated) * Debian 10 * Fedora 32, 33, Rawhide * openSUSE Leap 15.1, 15.2, Tumbleweed * Ubuntu: 18.04, 20.04, 20.10 (with universe activated) * Docker Engine: 18.x or higher The other major change is the split of the IntelMQ Manager into a separate API as backend (Thanks Birger) and the switch to Python for the latter (Thanks SUNET and Intevation). And thanks for all the numerous contributors who contributed greatly to the majorly extended feature set, by providing code, bug reports and other inputs. For all the other changes, please have a look at the changelog and the news file in the repository. I will give a more detailed overview of the changes in the final 2.3.0 announcement. https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md https://github.com/certtools/intelmq/blob/develop/NEWS.md To install the release candidate version from git, use the "maintenance" branch or 2.3.0.rc1 Tag (or if you are looking for more adventures, the "develop" branch,which we will use for the development of 3.0 now). For installation via pip, use `pip install --pre intelmq`, the version is called 2.3.0rc1 there. And for the packages, use the unstable repository: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… For the new beta docker image (Thanks Sebastian Waldbauer), install certat/intelmq-full: https://hub.docker.com/r/certat/intelmq-full Also summarized here: https://intelmq.readthedocs.io/en/latest/dev/guide.html#testing-pre-releases best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Advice for setting up tests for bot chains
by Mika Silander 19 Feb '21

19 Feb '21

Hi, While writing tests to individual bots seems quite straightforward, what would be the recommended way for writing (unit?) tests for a chain of bots? Ideally, I'd like to feed in individual test events to the first bot and only check what is returned by the last bot (event or whatever the expected outcome should be). I suppose there are many bad and clumsy ways of doing this so that's why I air this question on the dev list. Thanks for all help in advance, Mika

4 6

Bot behaviour in case of unrecoverable errors
by Mika Silander 16 Feb '21

16 Feb '21

Hi, Trying to assess what safeguards are sufficient: what happens when a bot has some internal failure and it "dies"? Will intelmq restart the bot automatically or will it be up to the admin of intelmq to manually restart it? And if automatic restarts is the norm, how could one stop the bot from processing new incoming messages if say, X consecutive failures like these have happened within the time frame of the last 5 minutes? By writing some log entries at bot startup and then making the bot itself analyze the log at every restart? I'm trying to make sure a burst of erroneous/malformed events are not accidentally forwarded by a malfunctioning or partially functioning bot. Cheers, Mika

2 6

Event rate limiting
by Mika Silander 28 Jan '21

28 Jan '21

Hi, Maybe this is answered somewhere already but I wasn't able to find any hints and I've just recently started developing bots so here's one for the list: I'd need to limit the rate at which events are processed to avoid swamping a back end system with requests. Big bursts of events should remain in the inqueue of such a rate limiting bot, and then be forwarded at a configurable rate e.g. X events/minute. Afaik, by default, bots attempt to process all the events in their inqueue without delay. How should one go about to create such a bot? Is it safe to put the bot process (=operating system process) to sleep for N seconds from within the bot's process method and then proceed, can we do busy-looping (not nice!), or, some other solution? Br, Mika

3 5

IEP01: IntelMQ Configuration Handling
by Birger Schacht 26 Jan '21

26 Jan '21

Dear IntelMQ developers and users, below are a couple of ideas how to (hopefully) make configuration of IntelMQ easier. Feel free to give feedback, voice concerns or simply ask if there is something unclear. We plan to evaluate the feedback that emerged in two weeks (after the christmas holidays). # IntelMQ Configuration Handling (IntelMQ Enhancement Proposal 01) ## Format ### JSON At the moment, the configuration format of IntelMQ is JSON[^1]. It is parsed using the Python json library, which is part of the Python Standard Library. The downside of JSON is, that is is hard to read and and write for humans and it cannot contain comments. [^1]: https://docs.python.org/3/library/index.html ### YAML There is a proposal[^2] to use YAML as the default configuration format. YAML provides way better readability for humans and YAML supports single line comments. There are two Python YAML libraries out there, the one being PyYAML[^3] and the other being ruamel.yaml[^4]. The former is a project by the YAML project itself. The latter is a fork of the former and had much more activity over the years and better support of the standard. It seems that pyyaml caught up in the last few years. We don't need any edge cases, so both libraries would be good for configuration files. According to this issue[^5] pyyaml does not support “editing YAML whilst maintaining comments”, which might be a deal breaker, but this issue is from 2016, this might have changed. On the other hand, IntelMQ does not edit configuration at the moment. pyyaml and ruamel.yaml are available as package in all relevant Linux distributions. [^2]: https://github.com/gethvi/intelmq/blob/ideas/docs/Ideas.md#changing-configu… [^3]: https://pyyaml.org/ [^4]: https://yaml.readthedocs.io/en/latest/ [^5]: https://github.com/yaml/pyyaml/issues/46 ### INI The Python Standard Library also ships configparser[^6], which is a “configuration language which provides a structure similar to what’s found in Microsoft Windows INI files”. The files can contain comments, it comes with a [DEFAULT] section, which can be used for default values and the configuration files can contain variables. One downside is that all the configurations are Strings, which means we would have to do parsing ourself. [^6]: https://docs.python.org/3/library/configparser.html ### toml Tom's Obvious, Minimal Language is another contender for the role of IntelMQs configuration file format. It looks similar to the INI file format, but comes with various data types. It also allows comments. There is a Python library[^7] that seems to be very active. toml is also used as the format for the proposed pyproject.toml file and by the rust community for their package configuration files. toml's syntax for dictionaries is hard to read/write, harder than with JSON. [^7]: https://pypi.org/project/toml/ ### Further information * The summary on file formats on the PEP518 proposition: https://www.python.org/dev/peps/pep-0518/#other-file-formats * At the moment we are leaning towards YAML. Regarding the library, we would choose ruamel.yaml, because it seems to have a more active upstream and it can retain comments when it modifies a yaml file. ## Storage This part is about the question where do we store the configuration?. The ideas document[^8] on GitHub already proposes to remove the pipeline.conf and specifying the destination pipelines in the individual bot configuration part. The declaration of the source queue can be dropped then as well, as it follows a rule anyway. In addition to that, to make the setup of IntelMQ easier, the defaults.conf should be dropped. Default values should be set in the Bot classes respectively in the IntelMQ process managers, but there is no need for a separate file. Another question is, if every bot should have their own configuration file. Some users wish to be able to start a bot without having to rely on IntelMQ, but at the moment, the bot gets the configuration from IntelMQ's runtime.conf. If we want to support the request to be able to pass individual configurations to bots, we could allow users to pass a separate configuration file to the bot (i.e. using `-c /path/to/config.$ext`). If that file is not set or does not contain the bots id, it is ignored and IntelMQ's runtime.conf is used as usual. If it does exists, the global runtime.conf is still parsed (if it exists - it should also be possible to run a bot without a runtime.conf) but only the values that are not set in the individual configuration file are considered. This individual configuration file would also allow a bot to be run in a docker environment without having to set any environment variables. This would make configuration handling probably easier, because then configuration settings could be stored in a file (and managed by a configuration management system) and the configuration file could contain comments. Proposal: * IntelMQ gets one global configuration file for all the bots and the pipeline.conf will be removed * This global configuration file is `${PREFIX}/etc/intelmq/intelmq.$ext`. If it does not exists or does not define any bots, IntelMQ should exit gracefully. The file extension depends on the chosen format. * The global configuration file contains an array of bot configurations with bot-ids as keys. * Every bot reads the global configuration file and extracts their own settings (as usual). * Every bot handles 0 to n `-c /path/to/configurationfile.$ext` flags, which are treated the same way as the global configuration file. The further ahead the configuration file in the commandline, the stronger the content (this allows us to have multiple non-global configuration files (i.e. for multiple groups)) Example: ``` > botcommand bot-id -c /etc/bots/botname.$ext -c /etc/bots/groups/group_foo.$ext ``` * Every bot also consults the environment and the values that are set their overwrite the values in any configuration file * There are also configuration files which list settings that are not bot specific, i.e. via a reserved key default (successor of the defaults.conf file) or group:id, those are also handled like other configuration files, but the bot does not compare its name to the key of the configuration. All the evaluated configuration formats provide the possibility to arrange the configuration parameters in hierarchies. To make the configuration files more readable, IntelMQ should make use of this hierarchy instead of denoting the different hierarchy levels with underscores. So instead of writing `http_proxy` the http parameter would have a childparameter proxy. For backwards compatibility and cases where the underscore does not imply hierarchy, the underscore notation will still work. In addition, IntelMQ should also make use of environment variables - those are still denoted using an underscore as delimiter and are prepended with `INTELMQ`: `INTELMQ_HTTP_PROXY`. [^8]: https://github.com/gethvi/intelmq/blob/ideas/docs/Ideas.md ### Caveats There are configuration settings, that do not really concern the bot- for example the type of process manager, that should be used to run the bot. In an ideal setup, the bot should be totally indifferent as to if it runs in a Docker container, on bare metal, in a SystemD unit file or with SupervisorD. This decision should only concern the tool managing all the bots (intelmqctl or in the future intelmq-api (which at the moment uses intelmqctl)). Another example is the enabled setting. At the moment, those are part of the individual bot configuration, but it might make sense to move them to a management.conf configuration file which is only for managing the individual bots, but not for configuring their parameters (this file would then also (for every bot) have a field that lists the configuration files the bot should consider when reading its configuration). On the other hand, this might make the configuration more complex again, now that we are trying to merge pipeline.conf and runtime.conf. We could also decide to make those configuration settings be part of the global configuration file, given that the individual bots should anyway simply ignore settings they do not know how to handle. ### Overriding by command line parameters If needed, a user can override specific bot settings using the -p switch (i.e. `-p redis_cache=example.com`). This should be easy to implement, in the best case scenario this is only one line of additional code in the Bot class. ### Examples A global configuration file with multiple bots /etc/intelmq/intelmq.yml ``` - shodan1: module: intelmq.bots.collectors.shodan.collector - mylittlebot23: module: intelmq.bots.expert.asn_lookup.expert http: proxy: http://myproxy.tld:80 - fop1: module: intelmq.bots.outputs.file output: filename: /dev/null ``` We can run a bot with intelmq-bot shodan1 which is the same as `intelmq-bot shodan1 -c /etc/intelmq/intelmq.yml` Another configuration file with multiple bots /root/intelmq-bots-managed-by-root: ``` - shodan2: module: intelmq.bots.collectors.shodan.collector - fop1: module: intelmq.bots.outputs.file output: filename: /var/log/fop1.log ``` We can run a bot with `intelmq-bot shodan2 -c /root/intelmq-bots-managed-by-root`; We can run a bot using `intelmq-bot fop1 -c /root/intelmq-bots-managed-by-root` which would then send output to `/var/log/fop1.log`. A configuration for a group in /etc/intelmq/collector-group.yml ``` - group:collectors http: proxy: http://thirdparty.proxy.tld:9000 ``` We can run a bot with intelmq-bot `mylittlebot23 -c /etc/intelmq/collector-group.yml` which uses the third-party proxy. ## Internal handling Every bot class defines their own settings as class variables. Every class variable has to be typed. Every class variable should be set to a reasonable default, otherwise None. The init of the (abstract) Bot class should load all the relevant configuration files and then overwrite the settings. If a setting is still None and the value of the setting is vital for the functionality of the bot, the bot should stop and emit a meaningful error message. For the most common types of settings, there should be Python objects to check the values. Value checking should only be done after all the configurations are merged. -- // Birger Schacht <schacht(a)cert.at> // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

6 14

Request for Testing and Feedback: Shadowserver API collector and parser
by Birger Schacht 19 Jan '21

19 Jan '21

Hi, Shadowserver nowadays not only sends out reports by Mail but also provides an API [0] to query reports. We recently implemented a Shadowserver Reports API collector bot [1] that downloads the reports from the API and feeds them into IntelMQ. To parse the downloaded feeds (in JSON format) we built upon the existing Shadowserver parsing logic and created a JSON parser [2] that's meant to be used together with the Shadowserver Reports API collector bot. If anyone is interested in testing the collector and parser that would be great- any feedback, bug reports or improvements are highly appreciated. To use the collector you will need a Report API Key, which you can request on [3]. cheers, Birger [0] https://www.shadowserver.org/what-we-do/network-reporting/api-reports-query/ [1] https://intelmq.readthedocs.io/en/latest/user/bots.html#shadowserver-report… [2] https://intelmq.readthedocs.io/en/latest/user/bots.html#shadowserver [3] https://www.shadowserver.org/contact/ -- // Birger Schacht <schacht(a)cert.at> // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ Docker Image (beta)
by Sebastian Waldbauer 15 Jan '21

15 Jan '21

====== IntelMQ Docker Image *(beta)* ====== *CAUTION* Please do not use this image in production yet! ===== Introduction ===== We are constantly trying to make the installation of IntelMQ easier for a wider user-base. Therefore we are currently working on an IntelMQ container[^1]. Because of the early stage, input from you is highly appreciated. We are currently targeting Docker Engine 18.x and higher. [^1] https://www.docker.com/resources/what-container ===== Current implementation ===== TL;DR: The image is a beta test, docker containers runs isolated, worth to test! *ATTENTION* The image provided is not fully Docker compliant. We are releasing the image to collect feedback and gather information for future improvements. The container should be considered as a beta version. There will likely be breaking changes in future versions of the container. The image is available on Dockerhub[^4]. You can find documentation on how to install and run the container in the IntelMQ documentation [^doc] The container comes preinstalled with IntelMQ Manager[^3], which will be the only way to control the IntelMQ installation in the container. You can't use `intelmqctl` outside the container to control IntelMQ in the container. At the moment, the image is shipped with the development version of IntelMQ[^2] and the last stable version 2.2.1 IntelMQ Manager[^3]. This means that the new IntelMQ API is //not// yet part of the Docker image. You can check the current included versions by using ```docker inspect --format '{{ index .Config.Labels "org.label-schema.vcs-ref"}}' certat/intelmq-full:1.0``` Build steps and deployment information is provided in IntelMQ Docker[^5] Repository. [^2] https://github.com/certtools/intelmq/ [^3] https://github.com/certtools/intelmq-manager [^4] https://hub.docker.com/repository/docker/certat/intelmq-full/general [^5] https://github.com/certat/intelmq-docker [^doc]: https://intelmq.readthedocs.io/en/latest/user/installation.html -- // Sebastian Waldbauer <waldbauer(a)cert.at> - T: +43 1 5056416 7202 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

An expert bot for Request Tracker's RT::IR
by Mika Silander 13 Jan '21

13 Jan '21

Hi, We recently decided to try IntelMQ with the intent to have it push security events into a Request Tracker (RT) instance. The events would thus be managed as RT::IR tickets within RT. We didn't manage to make the Request Tracker output bot working and we are not entirely sure whether it is because we have just missed something in its configuration or whether it has some other problem. Thus, what is the current status of this bot? Is it still usable with RT versions 4.x and 5.0.x ? Ideally, we'd like to have/create an RT + RT::IR output bot that uses the newer RT REST API 2.0. If there's anyone with similar endeavours, I'd be happy to hear from you. Br, Mika

2 3

IntelMQ 2.2.3 Christmas release
by Sebastian Wagner 23 Dec '20

23 Dec '20

Merry Christmas, dear community :) More or less last minute I decided to do a bugfix release before the holidays *really* start, because we already collected some fixed in the last weeks/months. There a no spectacular changes in this minor release, but the upcoming 2.3.0 will have some major changes for the IntelMQ Manager backend / the new IntelMQ API and the deprecation of Python 3.5. Installation documentation: https://github.com/certtools/intelmq/blob/2.2.3/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.2.3/docs/UPGRADING.md The deb/rpm packages will be available in the repositories in the next few hours. The NEWS: ### Harmonization A bug in the taxonomy expert did set the Taxonomy for the type `scanning` to `information gathering` whereas for the type `sniffing` and `social-engineering`, the taxonomy was correctly set to `information-gathering`. This inconsistency for the taxonomy `information-gathering` is now fixed, but the data eventually needs to fixed in data output (databases) as well. There are still some inconsistencies in the naming of the classification taxonomies and types, more fixes will come in version 3.0.0. See [issue #1409](https://github.com/certtools/intelmq/issues/1409). ### Postgres databases The following statements optionally update existing data. Please check if you did use these feed names and eventually adapt them for your setup! ```SQL UPDATE events SET "classification.taxonomy" = 'information-gathering' WHERE "classification.taxonomy" = 'information gathering'; ``` The full CHANGELOG: ### Documentation - Bots/Sieve expert: Add information about parenthesis in if-expressions (#1681, PR#1687 by Birger Schacht). ### Harmonization - See NEWS.md for information on a fixed bug in the taxonomy expert. ### Bots #### Collectors - `intelmq.bots.rt.collector_rt`: Log the size of the downloaded file in bytes on debug logging level. #### Parsers - `intelmq.bots.parsers.cymru.parser_cap_program`: - Add support for protocols 47 (GRE) and 59 (IPv6-NoNxt). - Add support for field `additional_asns` in optional information column. - `intelmq.bots.parsers.microsoft.parser_ctip`: - Fix mapping of `DestinationIpInfo.DestinationIpConnectionType` field (contained a typo). - Explicitly ignore field `DestinationIpInfo.DestinationIpv4Int` as the data is already in another field. - `intelmq.bots.parsers.generic.parser_csv`: - Ignore line having spaces or tabs only or comment having leading tabs or spaces (PR#1669 by Brajneesh). - Data fields containing `-` are now ignored and do not raise an exception anymore (#1651, PR#74 by Sebastian Waldbauer). #### Experts - `intelmq.bots.experts.taxonomy.expert`: Map type `scanner` to `information-gathering` instead of `information gathering`. See NEWS file for more information. ### Tests - Travis: Deactivate tests with optional requirements on Python 3.5, as the build fails because of abusix/querycontacts version conflicts on dnspython. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Corrupt dump files when interrupted during writing (#870). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 3.0, leaving CERT.at and the future of IntelMQ
by L. Aaron Kaplan 14 Nov '20

14 Nov '20

Dear incident handling automation tools list, dear IntelMQ folks, First of all, after extensive feedback from many of you, we do have a high level requirements document for IntelMQ 3.0. It's here: https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectu… This shall serve as a high level blueprint for IntelMQ 3.0 developments. Sebastian is working on prioritising individual items for CERT.at and then we will create individual GitHub issues and people (mostly at CERT.at) will be hacking away at it. Looking forward to this release. I'll be guiding this release however, I won't be working at CERT.at anymore starting on the 15th of Nov. Which brings me to an important conclusion: I thought long about it what we should do when core people leave CERT.at (as in my case, or... maybe Sebastian will leave one day or get run over by the famous bus which always seems to run over every team member according to manager's expectations ;-) ) In any case, the most solid approach seems to remember what IntelMQ actually is - a **community project**. It started as one, it is one , it will be one. In the last years, CERT.at did a lot of the heavy lifting and also a lot of the decisions on IntelMQ's future. However, with a couple of hundred (600?) installations worldwide, it would be wise to create an **advisory board/architecture board** for the future developments. I would envision a small-ish group of 4-8 people who take the responsibility of guiding the project for the next ~5 years. This means: - staying on top of current developments - coordinating with the other group members - coming up with a strategy and procedures (for example, compare with PEP, maybe a lightweight PEP approach is enough) etc. - ultimately, guiding the project It's work, for sure. You should have some passion for the project of course. I sent out a couple of invite requests to individuals but also would be interested to hear from you, if you would like to participate in such an effort. Hence, IntelMQ will become its own entity. And that's good, healthy and ensures a maximum benefit for many users. If you would like to be on that board, please send me an email. I'll guide it initially and get everything started. All the best, Aaron Kaplan (private email address for the future: aaron(a)lo-res.org) PS: we already have one or two companies offering development support for IntelMQ, I would like that they can thrive in this project as well - on a friendly basis. In the long run, this will make the project stronger. -- // L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ Tutorial and a new documentation page
by Sebastian Wagner 09 Nov '20

09 Nov '20

Dear community, I have two news items for you, both related to documentation: First, there is an IntelMQ Tutorial, which guides through various features and tools of IntelMQ. Lesson one introduces the architecture, concepts and terminology of the project. Lessons two and three delve hands-on into working with IntelMQ. Starting with installation and basic usage & configuration they go on to tackle progressively more advanced topics like using advanced features or changing the message queue software to be used. Solutions and explanations are offered for all tasks. In the last lesson you'll learn how to use intelmq-tools, a third-party software which makes customization of your IntelMQ instance much easier. We think that this kind of interactive online documentation is especially important nowadays when conferences and workshops cannot take place in real life. As for all other IntelMQ components, we welcome any contributions and feedback to the tutorial. -> https://github.com/certtools/intelmq-tutorial Second, we have a new IntelMQ Documentation page: We completely revised the way IntelMQ's documentation is presented: Instead of single files in the source-code repository, the best place to read the documentation is now intelmq.readthedocs.io. All pages are generated using Sphinx, the de facto standard tool for documentation. It features a better reading experience and a significantly improved navigation. Furthermore, the new page offers an integrated search as well as module index covering the complete code documentation If you find any bugs or have improvements, please let us know! -> https://intelmq.readthedocs.io/ best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

IntelMQ release 2.2.2
by Sebastian Wagner 28 Oct '20

28 Oct '20

Dear community, It's again long overdue for a new release and here it is finally. Since August we collected quite a few bugfixes - Thanks to all contributors! IntelMQ Installation documentation: https://github.com/certtools/intelmq/blob/2.2.2/docs/INSTALL.md IntelMQ Upgrade documentation: https://github.com/certtools/intelmq/blob/2.2.2/docs/UPGRADING.md *News for IntelMQ 2.2.2* ### Bots #### Cymru Whois Lookup The cache key calculation has been fixed. It previously led to duplicate keys for different IP addresses and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible. Therefore, this bot may take longer processing events than usual after applying this update. More details can be found in [issue #1592](https://github.com/certtools/intelmq/issues/1592). ### Harmonization #### Shadowserver Feed/Parser The feed "Blacklisted-IP" has been renamed by ShadowServer to "Blocklist". In IntelMQ, the old name can still be used in IntelMQ until version 3.0. *Changes for IntelMQ 2.2.2* ### Core - `intelmq.lib.upgrades`: - Add upgrade function for renamed Shadowserver feed name "Blacklisted-IP"/"Blocklist". ### Bots #### Parsers - `intelmq.bots.parsers.shadowserver`: - Rename "Blacklisted-IP" feed to "Blocklist", old name is still valid until IntelMQ version 3.0 (PR#1588 by Thomas Hungenberg). - Added support for the feeds `Accessible Radmin` and `CAIDA IP Spoofer` (PR#1600 by sinus-x). - `intelmq.bots.parsers.anubisnetworks.parser`: Fix parsing error where `dst.ip` was not equal to `comm.http.host`. - `intelmq/bots/parsers/danger_rulez/parser`: correctly skip malformed rows by defining variables before referencing (PR#1601 by Tomas Bellus). - `intelmq.bots.parsers.misp.parser: Fix MISP Event URL (#1619, PR#1618 by Nedfire23). - `intelmq.bots.parsers.microsoft.parser_ctip`: - Add support for `DestinationIpInfo.*` and `Signatures.Sha256` fields, used by the `ctip-c2` feed (PR#1623 by Mikk Margus Möll). - Use `extra.payload.text` for the feed's field `Payload` if the content cannot be decoded (PR#1610 by Giedrius Ramas). #### Experts - `intelmq.bots.experts.cymru_whois`: - Fix cache key calculation which previously led to duplicate keys and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible (#1592, PR#1606). - The bot now caches and logs (as level INFO) empty responses from Cymru (PR#1606). ### Documentation - README: - Add Core Infrastructure Initiative Best Practices Badge. - Bots: - Generic CSV Parser: Add note on escaping backslashes (#1579). - Remove section of non-existing "Copy Extra" Bot. - Explain taxonomy expert. - Add documentation on n6 parser. - Gethostbyname expert: Add documentation how errors are treated. - Feeds: - Fixed bot modules of Calidog CertStream feed. - Add information on Microsoft CTIP C2 feed. ### Packaging - In Debian packages, `intelmqctl check` and `intelmqctl upgrade-config` are executed in the postinst step (#1551, PR#1624 by Birger Schacht). ### Tests - `intelmq.tests.lib.test_pipeline`: Skip `TestAmqp.test_acknowledge` on Travis with Python 3.8. - `intelmq.tests.bots.outputs.elasticsearch.test_output`: Refresh index `intelmq` manually to fix random test failures (#1593, PR#1595 by Zach Stone). ### Tools - `intelmqctl check`: - For disabled bots which do not have any pipeline connections, do not raise an error, but only warning. - Fix check on source/destination queues for bots as well the orphaned queues. ### Contrib - Bash completion scripts: Check both `/opt/intelmq/` as well as LSB-paths (`/etc/intelmq/` and `/var/log/intelmq/`) for loading bot information (#1561, PR#1628 by Birger Schacht). ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Corrupt dump files when interrupted during writing (#870). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ & IntelMQ Manager releases 2.2.1
by Sebastian Wagner 30 Jul '20

30 Jul '20

Dear community, Today we have again a twin release 2.2.1 for both IntelMQ as well as IntelMQ Manager. This IntelMQ Manager version requires IntelMQ >= 2.2.1. There are currently issues with the signature in the package repositories for Debian/Ubuntu. I hope to get them resolved soon. IntelMQ Installation documentation: https://github.com/certtools/intelmq/blob/2.2.1/docs/INSTALL.md IntelMQ Upgrade documentation: https://github.com/certtools/intelmq/blob/2.2.1/docs/UPGRADING.md IntelMQ Manager Installation instructions: https://github.com/certtools/intelmq-manager/blob/2.2.1/docs/INSTALL.md *The changelog for IntelMQ Manager:* ### Backend - Fix loading paths from `intelmqctl` executable (PR #205 by Einar Felipe Lanfranco). ### Documentation - User Guide: - Add section on configuration paths. - Add section on named queues / paths. - Readme: - Update screenshots (#201, PR#207 by Mladen Markovic). ### Known issues * Graph jumps around on "Add edge" (#148). * wrong error message for new bots with existing ID (#152). * Monitor page: Automatic log refresh reset log page to first one (#190). *The News for IntelMQ:* ### Requirements #### MaxMind GeoIP Expert Bot The current python library versions of geoip (version 4) and maxminddb (version 2) no longer support Python 3.5. Keep older versions of these libraries if you are using this Python version. ### Configuration #### Abuse.ch URLHaus The current documented value for the `column` parameter was: ```json ['time.source', 'source.url', 'status', 'extra.urlhaus.threat_type', 'source.fqdn', 'source.ip', 'source.asn', 'source.geolocation.cc'] ``` Better is: ```json ['time.source', 'source.url', 'status', 'classification.type|__IGNORE__', 'source.fqdn|__IGNORE__', 'source.ip', 'source.asn', 'source.geolocation.cc'] ``` *And the changelog for IntelMQ:* ### Core - `intelmq.lib.upgrades`: - Add upgrade function for changed configuration of the feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný). - Add upgrade function for removal of *HPHosts Hosts file* feed and `intelmq.bots.parsers.hphosts` parser (#1559). - `intelmq.lib.harmonization`: - For IP Addresses, explicitly reject IPv6 addresses with scope ID (due to changed behavior in Python 3.9, #1550). ### Development - Ignore line length (E501) in code-style checks altogether. ### Bots #### Collectors - `intelmq.bots.collectors.misp`: Fix access to actual MISP object (PR#1548 by Tomas Bellus @tomas321) - `intelmq.bots.collectors.stomp`: Remove empty `client.pem` file. #### Parsers - `intelmq.bots.parsers.shadowserver.config`: - Add support for Accessible-CoAP feed (PR #1555 by Thomas Hungenberg). - Add support for Accessible-ARD feed (PR #1584 by Tomas Bellus @tomas321). - `intelmq.bots.parser.anubisnetworks.parser`: Ignore "TestSinkholingLoss" events, these are not intended to be sent out at all. - `intelmq.bots.parsers.generic.parser_csv`: Allow values of type dictionary for parameter `type_translation`. - `intelmq.bots.parsers.hphosts`: Removed, feed is unavailable (#1559). - `intelmq.bots.parsers.cymru.parser_cap_program`: Add support for comment "username" for "scanner" category. - `intelmq.bots.parsers.malwareurl.parser`: Check for valid FQDN and IP address in URL and IP address columns (PR#1585 by Marius Urkis). #### Experts - `intelmq.bots.experts.maxmind_geoip`: On Python < 3.6, require maxminddb < 2, as that version does no longer support Python 3.5. #### Outputs - `intelmq.bot.outputs.udp`: Fix error handling on sending, had a bug itself. ### Documentation - Feeds: - Update documentation of feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný). - Bots: - Overhaul of all bots' description fields (#1570). - User-Guide: - Overhaul pipeline configuration section and explain named queues better (#1577). ### Tests - `intelmq.tests.bots.experts.cymru`: Adapt `test_empty_result`, remove `test_unicode_as_name` and `test_country_question_mark` (#1576). ### Tools - `intelmq.bin.intelmq_gen_docs`: Format parameters of types lists with double quotes around values to produce conform JSON, ready to copy and paste the value into the IntelMQ Manager's bot parameter form. - `intelmq.bin.intelmqctl`: - `debug`: In JSON mode, use dictionaries instead of lists. - `debug`: Add `PATH` to the paths shown. - `check`: Show `$PATH` environment variable if executable cannot be found. ### Contrib - `malware_name_mapping`: Change MISP Threat Actors URL to new URL (branch master -> main) in download script. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Corrupt dump files when interrupted during writing (#870). - Bash completion scripts search in wrong directory in packages (#1561). - Cymru Expert: Wrong Cache-Key Calculation (#1592). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ Manager 2.2.0
by Sebastian Wagner 24 Jun '20

24 Jun '20

Dear community, To get the versions of intelmq and intelmq-manager in sync, we of course need a 2.2.0 for intelmq-manager too :) The number of changes is very short this time. However, a migration of the backend from PHP to Python, coded by Intevation and funded by SUNET, is in progress :) The deb/rpm packages are currently building, I will send a separate mail to intelmq-users once the 2.2.0 of intelmq and intelmq-manager packages have finally hit the repositories. This IntelMQ Manager version requires IntelMQ >= 2.2.0. Installation instructions: https://github.com/certtools/intelmq-manager/blob/2.2.0/docs/INSTALL.md Full changelog: ### Backend - `config`: Get file paths from `intelmctl debug --get-paths` if possible and fall back to hard-coded paths otherwise. Thereby environment variables influencing the paths are respected (#193). ### Pages #### About - Show output of `intelmqctl debug`. ### Documentation - Update release from intelmq's release documentation. - Update Installation documentation: Fix & update dependencies and supported operating systems. ### Packaging - Update default `positions.conf` to the default runtime/pipeline configuration of intelmq >= 2.1.1. ### Known issues * Missing CSRF protection (#111). * Graph jumps around on "Add edge" (#148). * wrong error message for new bots with existing ID (#152). * `ALLOWED_PATH=` violates CSP (#183). * Monitor page: Automatic log refresh reset log page to first one (#190). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 2.2.0 release
by Sebastian Wagner 18 Jun '20

18 Jun '20

Dear community, Today I finalized the 2.2.0 Feature release, more than half a year after 2.1.0. Thanks to all the contributors who made this possible! The release includes six new bots and seven heavily revised bots, and of course a lot of small changes to various bots. The full changelog can be found below. We dropped support for Python 3.4, that means Debian 8.0 and similar operating systems are no longer supported. Installation documentation: https://github.com/certtools/intelmq/blob/2.2.0/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.2.0/docs/UPGRADING.md ### Core - `__init__`: Changes to the path-handling, see [User Guide, section _/opt and LSB paths_](docs/User-Guide.md#opt-and-lsb-paths) for more information - The environment variable `INTELMQ_ROOT_DIR` can be used to set custom root directories instead of `/opt/intelmq/` (#805) in case of non LSB-path installations. - The environment variable `ROOT_DIR` can be used to set custom root directories instead of `/` (#805) in case of LSB-path installations. - `intelmq.lib.exceptions`: Added `MissingDependencyError` for show error messages about a missing library and how to install it (#1471). - Added optional parameter `installed` to show the installed version. - Added optional parameter `additional_text` to show arbitrary text. - Adding more type annotations for core libraries. - `intelmq.lib.pipeline.Pythonlist.sleep`: Drop deprecated method. - `intelmq.lib.utils`: `write_configuration`: Append a newline at end of configuration/file to allow proper comparisons & diffs. - `intelmq.lib.test`: `BotTestCase` drops privileges upon initialization (#1489). - `intelmq.lib.bot`: - New class `OutputBot`: - Method `export_event` to format/export events according to the parameters given by the user. - `ParserBot`: New methods `parse_json_stream` and `recover_line_json_stream`. - `ParserBot.recover_line_json`: Fix format by adding a list around the line data. - `Bot.send_message`: In debugging log level, the path to which the message is sent is now logged too. ### Bots - Bots with dependencies: Use of `intelmq.lib.exceptions.MissingDependencyError`. #### Collectors - `intelmq.bots.collectors.misp.collector`: Deprecate parameter `misp_verify` in favor of generic parameter `http_verify_cert`. - `intelmq.bots.collectors.tcp.collector`: Drop compatibility with Python 3.4. - `intelmq.bots.collectors.stomp.collector`: - Check the stomp.py version and show an error message if it does not match. - For stomp.py versions `>= 5.0.0` redirect the `stomp.PrintingListener` output to debug logging. - `intelmq.bots.collectors.microsoft.collector_azure`: Support current Python library `azure-storage-blob>= 12.0.0`, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details. - `intelmq.bots.collectors.amqp.collector_amqp`: Require `pika` minimum version 1.0. - `intelmq.bots.collectors.github_api.collector_github_contents_api`: Added (PR#1481). #### Parsers - `intelmq.bots.parsers.autoshun.parser`: Drop compatibility with Python 3.4. - `intelmq.bots.parsers.html_table.parser`: Drop compatibility with Python 3.4. - `intelmq.bots.parsers.shadowserver.parser`: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544). - `intelmq.bots.parsers.taichung.parser`: - Migrate to `ParserBot`. - Also parse geolocation information if available. - `intelmq.bots.parsers.cymru.parser_full_bogons`: - Migrate to `ParserBot`. - Add last updated information in raw. - `intelmq.bots.parsers.anubisnetworks.parser`: Add new parameter `use_malware_familiy_as_classification_identifier`. - `intelmq.bots.parsers.microsoft.parser_ctip`: Compatibility for new CTIP data format used provided by the Azure interface. - `intelmq.bots.parsers.cymru.parser_cap_program`: Support for `openresolver` type. - `intelmq.bots.parsers.github_feed.parser`: Added (PR#1481). - `intelmq.bots.parsers.urlvir.parser`: Removed, as the feed is discontinued (#1537). #### Experts - `intelmq.bots.experts.csv_converter`: Added as converter to CSV. - `intelmq.bots.experts.misp`: Added (PR#1475). - `intelmq.bots.experts.modify`: New parameter `maximum_matches`. #### Outputs - `intelmq.bots.outputs.amqptopic`: - Use `OutputBot` and `export_event`. - Allow formatting the routing key with event data by the new parameter `format_routing_key` (boolean). - `intelmq.bots.outputs.file`: Use `OutputBot` and `export_event`. - `intelmq.bots.outputs.files`: Use `OutputBot` and `export_event`. - `intelmq.bots.outputs.misp.output_feed`: Added, creates a MISP Feed (PR#1473). - `intelmq.bots.outputs.misp.output_api`: Added, pushes to MISP via the API (PR#1506, PR#1536). - `intelmq.bots.outputs.elasticsearch.output`: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (#1513). ### Documentation - Document usage of the `INTELMQ_ROOT_DIR` environment variable. - Added document on MISP integration possibilities. - Feeds: - Added "Full Bogons IPv6" feed. - Remove discontinued URLVir Feeds (#1537). ### Packaging - `setup.py` do not try to install any data to `/opt/intelmq/` as the behavior is inconsistent on various systems and with `intelmqsetup` we have a tool to create the structure and files anyway. - `debian/rules`: - Provide a blank state file in the package. - Patches: - Updated `fix-intelmq-paths.patch`. ### Tests - Travis: Use `intelmqsetup` here too. - Install required build dependencies for the Debian package build test. - This version is no longer automatically tested on Python `<` 3.5. - Also run the tests on Python 3.8. - Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8. - Added tests for the new bot `intelmq.bots.outputs.misp.output_feed` (#1473). - Added tests for the new bot `intelmq.bots.experts.misp.expert` (#1473). - Added tests for `intelmq.lib.exceptions`. - Added tests for `intelmq.lib.bot.OutputBot` and `intelmq.lib.bot.OutputBot.export_event`. - Added IPv6 tests for `intelmq.bots.parsers.cymru.parser_full_bogons`. - Added tests for `intelmq.lib.bot.ParserBot`'s new methods `parse_json_stream` and `recover_line_json_stream`. - `intelmq.tests.test_conf`: Set encoding to UTF-8 for reading the `feeds.yaml` file. ### Tools - `intelmqctl`: - `upgrade-config`: - Allow setting the state file location with the `--state-file` parameter. - Do not require a second run anymore, if the state file is newly created (#1491). - New parameter `no_backup`/`--no-backup` to skip creation of `.bak` files for state and configuration files. - Only require `psutil` for the `IntelMQProcessManager`, not for process manager independent calls like `upgrade-config` or `check`. - Add new command `debug` to output some information for debugging. Currently implemented: - paths - environment variables - `IntelMQController`: New argument `--no-file-logging` to disable logging to file. - If dropping privileges does not work, `intelmqctl` will now abort (#1489). - `intelmqsetup`: - Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions. - Call `intelmqctl upgrade-config` and add argument for the state file path (#1491). - `intelmq_generate_misp_objects_templates.py`: Tool to create a MISP object template (#1470). - `intelmqdump`: New parameter `-t` or `--truncate` to optionally give the maximum length of `raw` data to show, 0 for no truncating. ### Contrib - Added `development-tools`. - ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (#1513). - Malware Name Mapping Downloader: - New parameter `--mwnmp-ignore-adware`. - The parameter `--add-default` supports an optional parameter to define the default value. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Corrupt dump files when interrupted during writing (#870). best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Re: [Intelmq-dev] [Intelmq-users] IntelMQ vs AbuseIO
by Bernhard Reiter 09 Jun '20

09 Jun '20

Hi Patrick, Am Montag 01 Juni 2020 16:27:24 schrieb Patrick Forsberg: > Anyone have a quick comparison between AbuseIO and IntelMQ? > > I'm currently in the process of getting IntelMQ to work with > IntelMQ-Mailgen to be able to send out abuse-emails to our constituency > based on feeds like Shadowserver and since it seems like AbuseIO can do > just about the same I would like to know the pros and cons of the systems. for comparing AbuseIO and IntelMQ, I don't know AbuseIO enough. However if you are looking into IntelMQ Mailgen from the system we call intelmq-cb-mailgen, https://github.com/Intevation/intelmq-mailgen-release that is the one we've been developing for the CERT-Bund, so we can tell you more about it, maybe this helps with the comparison. The design idea is to be automated, flexible and high through-put. Thus there is a separation of concerns and several configuration places. You may have seen the overview diagram: https://raw.githubusercontent.com/Intevation/intelmq-mailgen/master/docs/no… There are rules within the IntelMQ export and additional notification formats scripts, those are quite flexible, so there is some learning curve. Once set up, there can be millions of events handled per day automatically with several people being able to add manual data to the contacts database. Feel free to ask us here or per direct email, if you have any questions or need help with setting it up. Best Regards, Bernhard -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

2.2.0 Release candidate 1
by Sebastian Wagner 30 May '20

30 May '20

Dear developers and pro-users, I have pushed a 2.2.0 pre-release to both PyPI as well as the unstable repository. If you have time & resources, or you are using the develop branch anyway, please test this version so we can ship the next stable release ready next week or the week afterwards, depending on the feedback. Please note that the current codebase is no longer tested with Python 3.4 and it may not work anymore with that Python version. For installation/upgrade with pip use the --pre parameter: pip install --pre intelmq Instructions for the deb & rpm unstable repository: https://software.opensuse.org/download/package?package=intelmq&project=home… best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 2.1.3 Bugfix release
by Sebastian Wagner 26 May '20

26 May '20

Dear community, We already collected a very long list of bug fixes since the last release, so it was time to mark the next milestone! As usual, you can find the list of changes below. The pre-built deb/rpm packages will hit the repositories very soon. Installation documentation: https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md Full changelog: ### Requirements - The python library `requests` is (again) listed as dependency of the core (#1519). ### Core - `intelmq.lib.upgrades`: - Harmonization upgrade: Also check and update regular expressions. - Add function to migrate the deprecated parameter `attach_unzip` to `extract_files` for the mail attachment collector. - Add function to migrate changed Taichung URL feed. - Check for discontinued Abuse.CH Zeus Tracker feed. - `intelmq.lib.bot`: - `ParserBot.recover_line`: Parameter `line` needs to be optional, fix usage of fallback value `self.current_line`. - `start`: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494). - `start`: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors. - `_dump_message`: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494). - `intelmq.lib.test`: - `BotTestCase.run_bot`: Add parameters `allowed_error_count` and `allowed_warning_count` to allow set the number per run, not per test class. - Set `source_pipeline_broker` and `destination_pipeline_broker` to `pythonlist` instead of the old `broker`, fixes `intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising`. - Fix test for (allowed) errors and warnings. - `intelmq.lib.exceptions`: - `InvalidKey`: Add `KeyError` as parent class. - `DecodingError`: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494). - `intelmq.lib.pipeline`: - Decode messages in `Pipeline.receive` not in the implementation's `_receive` so that the internal counter is correct in case of decoding errors (#1494). - `intelmq.lib.utils`: - `decode`: Raise new `DecodingError` if decoding fails. ### Harmonization - `protocol.transport`: Adapt regular expression to allow the value `nvp-ii` (protocol 11). ### Bots #### Collectors - `intelmq.bots.collectors.mail.collector_mail_attach`: - Fix handling of deprecated parameter name `attach_unzip`. - Fix handling of attachments without filenames (#1538). - `intelmq.bots.collectors.stomp.collector`: Fix compatibility with stomp.py versions `> 4.1.20` and catch errors on shutdown. - `intelmq.bots.collectors.microsoft`: - Update `REQUIREMENTS.txt` temporarily fixing deprecated Azure library (#1530, PR#1532). - `intelmq.bots.collectors.microsoft.collector_interflow`: Add method for printing the file list. #### Parsers - `intelmq.bots.parsers.cymru.parser_cap_program`: Support for protocol 11 (`nvp-ii`) and `conficker` type. - `intelmq.bots.parsers.taichung.parser`: Support more types/classifications: - Application Compromise: Apache vulnerability & SQL injections - Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks - C2 Sever: Attack controller - DDoS - DoS: DNS, DoS, Excess connection - IDS Alert / known vulnerability exploitation: backdoor - Malware: Malware Proxy - Warn on new unknown types. - `intelmq.bots.parsers.bitcash.parser`: Removed as feed is discontinued. - `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target`: Removed as feed is discontinued. - `intelmq.bots.parsers.malwaredomains.parser`: Correctly classify `C&C` and `phishing` events. - `intelmq.bots.parsers.shadowserver.parser`: More verbose error message for missing report specification (#1507). - `intelmq.bots.parsers.n6.parser_n6stomp`: Always add n6 field `name` as `malware.name` independent of `category`. - `intelmq.bots.parsers.anubisnetworks`: Update parser with new data format. - `intelmq.bots.parsers.bambenek`: Add new feed URLs with Host `faf.bambenekconsulting.com` (#1525, PR#1526). - `intelmq.bots.parsers.abusech.parser_ransomware`: Removed, as the feed is discontinued (#1537). - `intelmq.bots.parsers.nothink.parser`: Removed, as the feed is discontinued (#1537). - `intelmq.bots.parsers.n6.parser`: Remove not allowed characters in the name field for `malware.name` and write original value to `event_description.text` instead. #### Experts - `intelmq.bots.experts.cymru_whois.lib`: Fix parsing of AS names with Unicode characters. #### Outputs - `intelmq.bots.outputs.mongodb`: - Set default port 27017. - Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439). ### Documentation - Feeds: - Remove unavailable feed Abuse.CH Zeus Tracker. - Remove the field `status`, offline feeds should be removed. - Add a new field `public` to differentiate between private and public feeds. - Adding documentation URLs to nearly all feeds. - Remove unavailable Bitcash.cz feed. - Remove unavailable Fraunhofer DDos Attack feeds. - Remove unavailable feed Abuse.CH Ransomware Tracker (#1537). - Update information on Bambenek Feeds, many require a license now (#1525). - Remove discontinued Nothink Honeypot Feeds (#1537). - Developers Guide: Fix the instructions for `/opt/intelmq` file permissions. ### Packaging - Patches: `fix-logrotate-path.patch`: also include path to rotated file in patch. - Fix paths from `/opt` to LSB for `setup.py` and `contrib/logrotate/intelmq` in build process (#1500). - Add runtime dependency `debianutils` for the program `which`, which is required for `intelmqctl`. ### Tests - Dropping Travis tests for 3.4 as required libraries dropped 3.4 support. - `intelmq.tests.bots.experts.cymru_whois`: - Drop missing ASN test, does not work anymore. - IPv6 to IPv4 test: Test for two possible results. - `intelmq.lib.test`: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (#1342). - `intelmq.bots.collectors.tcp.test_collector`: Removing custom mocking and bot starting, not necessary anymore. - Added tests for `intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline`. - Fix and split `tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json`. - Added tests for invalid encodings in input messages in `intelmq.tests.lib.test_bot` and `intelmq.tests.lib.test_pipeline` (#1494). - Travis: Explicitly enable RabbitMQ management plugin. - `intelmq.tests.lib.test_message`: Fix usage of the parameter `blacklist` for Message hash tests (#1539). ### Tools - `intelmqsetup`: Copy missing BOTS file to IntelMQ's root directory (#1498). - `intelmq_gen_docs`: Feed documentation generation: Handle missing/empty parameters. - `intelmqctl`: - `IntelMQProcessManager`: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (#1492). - `IntelMQController`: Fix exit codes of `check` command for JSON output (now 0 on success and 1 on error, was swapped, #1520). - `intelmqdump`: - Handle base64-type messages for show, editor and recovery actions. ### Contrib - `intelmq/bots/experts/asn_lookup/update-asn-data`: Use `pyasn_util_download.py` to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, https://github.com/hadiasghari/pyasn/issues/62). ### Known issues - HTTP stream collector: retry on regular connection problems? (#1435). - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Reverse DNS: Only first record is used (#877). - Corrupt dump files when interrupted during writing (#870). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Roadmap: 2.2.0 release forecast?
by Bernhard Reiter 20 May '20

20 May '20

Hi Sebastian, do you have a rough timeframe for the 2.2.0 release? Best, Bernhard -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev