Dear community,
we again collected a bunch of bugfixes in the last weeks, coming almost one month after 2.1.0.
Install documentation: https://github.com/certtools/intelmq/blob/2.1.1/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.1.1/docs/UPGRADING.md
The full changelog:
### Configuration - Default configuration: - Remove discontinued feed "Feodo Tracker Domains" from default configuration. - Add "Feodo Tracker Browse" feed to default configuration.
### Core - `intelmq.lib.pipeline`: AMQP: using port 15672 as default (like RabbitMQ's defaults) for the monitoring interface for getting statistical data (`intelmqctl_rabbitmq_monitoring_url`). - `intelmq.lib.upgrades`: Added a generic upgrade function for harmonization, checking of all message types, it's fields and their types. - `intelmq.lib.utils`: - `TimeoutHTTPAdapter`: A subclass of `requests.adapters.HTTPAdapter` with the possibility to set the timeout per adapter. - `create_request_session_from_bot`: Use the `TimeoutHTTPAdapter` with the user-defined timeout. Previously the timeout was not functional.
### Bots #### Parsers - `intelmq.bots.parsers.shadowserver.parser`: Fix logging message if the parameter `feedname` is not present. - `intelmq.bots.parsers.shodan.parser`: Also add field `classification.identifier` (`'network-scan'`) in minimal mode. - `intelmq.bots.parsers.spamhaus.parser_cert`: Add support for category `'misc'`. - `intelmq.bots.parsers.cymru.parser_cap_program`: - Add support for phishing events without URL. - Add support for protocols >= 143 (unassigned, experiments, testing, reserved), saving the number to extra, as the data would be bogus. - `intelmq.bots.parsers.microsoft.parser_bingmurls`: - Save the `Tags` data as `source.geolocation.cc`.
#### Experts - `intelmq.bots.experts.modify.expert`: Fix bug with setting non-string values (#1460).
#### Outputs - `intelmq.bots.outputs.smtp`: - Allow non-existent field in text formatting by using a default value `None` instead of throwing errors. - Fix Authentication (#1464). - Fix sending to multiple recipients (#1464).
### Documentation - Feeds: - Fix configuration of `Feodo Tracker Browse` feed. - Bots: - Sieve expert: Document behavior of `!=` with lists.
### Tests - Adaption and extension of the test cases to the changes.
### Tools - `intelmq.bin.intelmqctl`: - check: Check if running the upgrade function for harmonization is necessary. - upgrade-config: Run the upgrade function for harmonization. - `intelmqctl restart` did throw an error as the message for restarting was not defined (#1465).
### Known issues - MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439) - ctl: shell colorizations are logged (#1436) - http stream collector: retry on regular connection problems? (#1435) - tests: capture logging with context manager (#1342) - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952) - n6 parser: mapping is modified within each run (#905) - reverse DNS: Only first record is used (#877) - Corrupt dump files when interrupted during writing (#870)