Dear community,
Today I released version 2.0.2 of intelmq with a bunch of bugfixes, see the changelog below. A feature release is coming soon too.
Install documentation: https://github.com/certtools/intelmq/blob/2.0.2/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.0.2/docs/UPGRADING.md
As always: read the NEWS file, upgrade according to the documentation and have fun! If you get any errors, please report them here or in the bug tracker.
Sebastian
Full changelog:
### Core - `intelmq.lib.bot.CollectorBot`: Support the deprecated parameter `feed` until version 2.2 as the documentation was not properly updated (#1445). - `intelmq.lib.bot.Bot`: - `_dump_message`: Wait for up to 60 seconds instead of 50 if the dump file is locked (the log message was said 60, but the code was for only 50). - `intelmq.lib.upgrades.v202_fixes` - Migration of deprecated parameter `feed` for Collectors. - Ripe expert parameter `query_ripe_stat_ip` was not correctly configured in `v110_deprecations`, now use `query_ripe_stat_asn` as default if it does not exist. - `intelmq.lib.upgrades.v110_deprecations`: Fix upgrade of ripe expert configuration. - `intelmq.lib.bot_debugger`: - Fix handling of empty messages generated by parser when user wanted to show the result by "--show-sent" flag. - Fix handling of sent messages for bots using the `path_permissive` paramter (#1453). - `intelmq.lib.pipeline.Amqp`: - use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used. - Reconnect once on sending messages if disconnect detected.
### Bots #### Collectors - `intelmq.bots.collectors.api.collector_api`: - Handle non-existing IO loop in shutdown. - Close socket on shutdown, fixes reloading. - Marked as non-threadable. - `intelmq.bots.collectors.rt.collector_rt`: Check for matching URLs if no `attachment_regex` is given. - `intelmq.bots.collectors.stomp.collector_stomp`: Handle disconnects by actively reconnecting.
#### Parsers - `intelmq.bots.cymru.parser_cap_program`: Fix parsing of the new `$certname_$date.txt` report format (#1443): - Support protocol ICMP. - Fix error message for unsupported protocols. - Support fields `destination_port_numbers`, `port`. - Support for all proxy types without ports. - Use Country Code of AS as `source.geolocation.cc`. - Support for 'scanner' and 'spam' categories. - Handle bogus lines with missing separator. - Fix bug preventing use of old format after using the new format. - Handle postfix ` (total_count:..)` for destination port numbers.
#### Experts - `intelmq.bots.experts.cymru_whois.expert`: Add optional parameter `overwrite`, current behavior was `True`, default if not given is `True` now, will change to `False` in 3.0.0 (#1452, #1455). - `intelmq.bots.experts.modify.expert`: Add optional parameter `overwrite`, current behavior was `True`, default if not given is `True` now, will change to `False` in 3.0.0 (#1452, #1455). - `intelmq.bots.experts.reverse_dns.expert`: Add optional parameter `overwrite`, current behavior was `True`, default if not given is `True` now, will change to `False` in 3.0.0 (#1452, #1455).
#### Outputs - `intelmq.bots.outputs.amqptopic.output`: use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.
### Packaging - Rules: - Exclude intelmqsetup tool in packages - Include update-rfiprisk-data in packages
### Tests - Tests for `intelmq.lib.upgrades.v202_fixes`. - Tests for `intelmq.lib.upgrades.v110_deprecations`. - Extended tests for `intelmq.bots.parser.cymru.parser_cap_program`.
### Tools - intelmqctl: - More and more precise logging messages for botnet starting and restarting, enable and disable. - No error message for disabled bots on botnet reload. - Fix `upgrade-conf` is state file is empty or not existing. - Use arpgarse's `store_true` action for flags instead of `store_const`. - If the loading of the defaults configuration failed, a variable definition was missing and causing an exception (#1456).
### Contrib - Check MK Statistics Cronjob: - Use `statistics_*` parameters. - Make file executable - Handle None values in `*.temporary.*` keys and treat them as 0. - systemd: - Add `PIDFile` parameter to service file.
### Known issues - MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439) - ctl: shell colorizations are logged (#1436) - http stream collector: retry on regular connection problems? (#1435) - tests: capture logging with context manager (#1342) - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952) - n6 parser: mapping is modified within each run (#905) - reverse DNS: Only first record is used (#877) - Corrupt dump files when interrupted during writing (#870)