Dear community,
We already collected a very long list of bug fixes since the last release, so it was time to mark the next milestone! As usual, you can find the list of changes below. The pre-built deb/rpm packages will hit the repositories very soon.
Installation documentation: https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md
Full changelog:
### Requirements - The python library `requests` is (again) listed as dependency of the core (#1519).
### Core - `intelmq.lib.upgrades`: - Harmonization upgrade: Also check and update regular expressions. - Add function to migrate the deprecated parameter `attach_unzip` to `extract_files` for the mail attachment collector. - Add function to migrate changed Taichung URL feed. - Check for discontinued Abuse.CH Zeus Tracker feed. - `intelmq.lib.bot`: - `ParserBot.recover_line`: Parameter `line` needs to be optional, fix usage of fallback value `self.current_line`. - `start`: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494). - `start`: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors. - `_dump_message`: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494). - `intelmq.lib.test`: - `BotTestCase.run_bot`: Add parameters `allowed_error_count` and `allowed_warning_count` to allow set the number per run, not per test class. - Set `source_pipeline_broker` and `destination_pipeline_broker` to `pythonlist` instead of the old `broker`, fixes `intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising`. - Fix test for (allowed) errors and warnings. - `intelmq.lib.exceptions`: - `InvalidKey`: Add `KeyError` as parent class. - `DecodingError`: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494). - `intelmq.lib.pipeline`: - Decode messages in `Pipeline.receive` not in the implementation's `_receive` so that the internal counter is correct in case of decoding errors (#1494). - `intelmq.lib.utils`: - `decode`: Raise new `DecodingError` if decoding fails.
### Harmonization - `protocol.transport`: Adapt regular expression to allow the value `nvp-ii` (protocol 11).
### Bots #### Collectors - `intelmq.bots.collectors.mail.collector_mail_attach`: - Fix handling of deprecated parameter name `attach_unzip`. - Fix handling of attachments without filenames (#1538). - `intelmq.bots.collectors.stomp.collector`: Fix compatibility with stomp.py versions `> 4.1.20` and catch errors on shutdown. - `intelmq.bots.collectors.microsoft`: - Update `REQUIREMENTS.txt` temporarily fixing deprecated Azure library (#1530, PR#1532). - `intelmq.bots.collectors.microsoft.collector_interflow`: Add method for printing the file list.
#### Parsers - `intelmq.bots.parsers.cymru.parser_cap_program`: Support for protocol 11 (`nvp-ii`) and `conficker` type. - `intelmq.bots.parsers.taichung.parser`: Support more types/classifications: - Application Compromise: Apache vulnerability & SQL injections - Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks - C2 Sever: Attack controller - DDoS - DoS: DNS, DoS, Excess connection - IDS Alert / known vulnerability exploitation: backdoor - Malware: Malware Proxy - Warn on new unknown types. - `intelmq.bots.parsers.bitcash.parser`: Removed as feed is discontinued. - `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target`: Removed as feed is discontinued. - `intelmq.bots.parsers.malwaredomains.parser`: Correctly classify `C&C` and `phishing` events. - `intelmq.bots.parsers.shadowserver.parser`: More verbose error message for missing report specification (#1507). - `intelmq.bots.parsers.n6.parser_n6stomp`: Always add n6 field `name` as `malware.name` independent of `category`. - `intelmq.bots.parsers.anubisnetworks`: Update parser with new data format. - `intelmq.bots.parsers.bambenek`: Add new feed URLs with Host `faf.bambenekconsulting.com` (#1525, PR#1526). - `intelmq.bots.parsers.abusech.parser_ransomware`: Removed, as the feed is discontinued (#1537). - `intelmq.bots.parsers.nothink.parser`: Removed, as the feed is discontinued (#1537). - `intelmq.bots.parsers.n6.parser`: Remove not allowed characters in the name field for `malware.name` and write original value to `event_description.text` instead.
#### Experts - `intelmq.bots.experts.cymru_whois.lib`: Fix parsing of AS names with Unicode characters.
#### Outputs - `intelmq.bots.outputs.mongodb`: - Set default port 27017. - Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439).
### Documentation - Feeds: - Remove unavailable feed Abuse.CH Zeus Tracker. - Remove the field `status`, offline feeds should be removed. - Add a new field `public` to differentiate between private and public feeds. - Adding documentation URLs to nearly all feeds. - Remove unavailable Bitcash.cz feed. - Remove unavailable Fraunhofer DDos Attack feeds. - Remove unavailable feed Abuse.CH Ransomware Tracker (#1537). - Update information on Bambenek Feeds, many require a license now (#1525). - Remove discontinued Nothink Honeypot Feeds (#1537). - Developers Guide: Fix the instructions for `/opt/intelmq` file permissions.
### Packaging - Patches: `fix-logrotate-path.patch`: also include path to rotated file in patch. - Fix paths from `/opt` to LSB for `setup.py` and `contrib/logrotate/intelmq` in build process (#1500). - Add runtime dependency `debianutils` for the program `which`, which is required for `intelmqctl`.
### Tests - Dropping Travis tests for 3.4 as required libraries dropped 3.4 support. - `intelmq.tests.bots.experts.cymru_whois`: - Drop missing ASN test, does not work anymore. - IPv6 to IPv4 test: Test for two possible results. - `intelmq.lib.test`: Fix compatibility of logging capture with Python
= 3.7 by reworking the whole process (#1342).
- `intelmq.bots.collectors.tcp.test_collector`: Removing custom mocking and bot starting, not necessary anymore. - Added tests for `intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline`. - Fix and split `tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json`. - Added tests for invalid encodings in input messages in `intelmq.tests.lib.test_bot` and `intelmq.tests.lib.test_pipeline` (#1494). - Travis: Explicitly enable RabbitMQ management plugin. - `intelmq.tests.lib.test_message`: Fix usage of the parameter `blacklist` for Message hash tests (#1539).
### Tools - `intelmqsetup`: Copy missing BOTS file to IntelMQ's root directory (#1498). - `intelmq_gen_docs`: Feed documentation generation: Handle missing/empty parameters. - `intelmqctl`: - `IntelMQProcessManager`: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (#1492). - `IntelMQController`: Fix exit codes of `check` command for JSON output (now 0 on success and 1 on error, was swapped, #1520). - `intelmqdump`: - Handle base64-type messages for show, editor and recovery actions.
### Contrib - `intelmq/bots/experts/asn_lookup/update-asn-data`: Use `pyasn_util_download.py` to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, https://github.com/hadiasghari/pyasn/issues/62).
### Known issues - HTTP stream collector: retry on regular connection problems? (#1435). - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Reverse DNS: Only first record is used (#877). - Corrupt dump files when interrupted during writing (#870).