[Intelmq-dev] IntelMQ 2.0.2 Bugfix release

[Sebastian Wagner]

Dear community,

Today I released version 2.0.2 of intelmq with a bunch of bugfixes, see
the changelog below. A feature release is coming soon too.

Install documentation:
https://github.com/certtools/intelmq/blob/2.0.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.0.2/docs/UPGRADING.md

As always: read the NEWS file, upgrade according to the documentation
and have fun! If you get any errors, please report them here or in the
bug tracker.

Sebastian

Full changelog:

### Core
- `intelmq.lib.bot.CollectorBot`: Support the deprecated parameter
`feed` until version 2.2 as the documentation was not properly updated
(#1445).
- `intelmq.lib.bot.Bot`:
  - `_dump_message`: Wait for up to 60 seconds instead of 50 if the dump
file is locked (the log message was said 60, but the code was for only 50).
- `intelmq.lib.upgrades.v202_fixes`
  - Migration of deprecated parameter `feed` for Collectors.
  - Ripe expert parameter `query_ripe_stat_ip` was not correctly
configured in `v110_deprecations`, now use `query_ripe_stat_asn` as
default if it does not exist.
- `intelmq.lib.upgrades.v110_deprecations`: Fix upgrade of ripe expert
configuration.
- `intelmq.lib.bot_debugger`:
  - Fix handling of empty messages generated by parser when user wanted
to show the result by "--show-sent" flag.
  - Fix handling of sent messages for bots using the `path_permissive`
paramter (#1453).
- `intelmq.lib.pipeline.Amqp`:
  - use default SSL context for client purposes, fixes compatibility
with python < 3.6 if TLS is used.
  - Reconnect once on sending messages if disconnect detected.

### Bots
#### Collectors
- `intelmq.bots.collectors.api.collector_api`:
  - Handle non-existing IO loop in shutdown.
  - Close socket on shutdown, fixes reloading.
  - Marked as non-threadable.
- `intelmq.bots.collectors.rt.collector_rt`: Check for matching URLs if
no `attachment_regex` is given.
- `intelmq.bots.collectors.stomp.collector_stomp`: Handle disconnects by
actively reconnecting.

#### Parsers
- `intelmq.bots.cymru.parser_cap_program`: Fix parsing of the new
`$certname_$date.txt` report format (#1443):
  - Support protocol ICMP.
  - Fix error message for unsupported protocols.
  - Support fields `destination_port_numbers`, `port`.
  - Support for all proxy types without ports.
  - Use Country Code of AS as `source.geolocation.cc`.
  - Support for 'scanner' and 'spam' categories.
  - Handle bogus lines with missing separator.
  - Fix bug preventing use of old format after using the new format.
  - Handle postfix ` (total_count:..)` for destination port numbers.

#### Experts
- `intelmq.bots.experts.cymru_whois.expert`: Add optional parameter
`overwrite`, current behavior was `True`, default if not given is `True`
now, will change to `False` in 3.0.0 (#1452, #1455).
- `intelmq.bots.experts.modify.expert`: Add optional parameter
`overwrite`, current behavior was `True`, default if not given is `True`
now, will change to `False` in 3.0.0 (#1452, #1455).
- `intelmq.bots.experts.reverse_dns.expert`: Add optional parameter
`overwrite`, current behavior was `True`, default if not given is `True`
now, will change to `False` in 3.0.0 (#1452, #1455).

#### Outputs
- `intelmq.bots.outputs.amqptopic.output`: use default SSL context for
client purposes, fixes compatibility with python < 3.6 if TLS is used.

### Packaging
- Rules:
  - Exclude intelmqsetup tool in packages
  - Include update-rfiprisk-data in packages

### Tests
- Tests for `intelmq.lib.upgrades.v202_fixes`.
- Tests for `intelmq.lib.upgrades.v110_deprecations`.
- Extended tests for `intelmq.bots.parser.cymru.parser_cap_program`.

### Tools
- intelmqctl:
  - More and more precise logging messages for botnet starting and
restarting, enable and disable.
  - No error message for disabled bots on botnet reload.
  - Fix `upgrade-conf` is state file is empty or not existing.
  - Use arpgarse's `store_true` action for flags instead of `store_const`.
  - If the loading of the defaults configuration failed, a variable
definition was missing and causing an exception (#1456).

### Contrib
- Check MK Statistics Cronjob:
  - Use `statistics_*` parameters.
  - Make file executable
  - Handle None values in `*.temporary.*` keys and treat them as 0.
- systemd:
  - Add `PIDFile` parameter to service file.

### Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo
versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 50564167201 
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20191014/f979bae4/attachment.sig>