[Intelmq-dev] IntelMQ 2.1.1 release

[Sebastian Wagner]

Dear community,

we again collected a bunch of bugfixes in the last weeks, coming almost
one month after 2.1.0.

Install documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/UPGRADING.md

The full changelog:

### Configuration
- Default configuration:
  - Remove discontinued feed "Feodo Tracker Domains" from default
configuration.
  - Add "Feodo Tracker Browse" feed to default configuration.

### Core
- `intelmq.lib.pipeline`: AMQP: using port 15672 as default (like
RabbitMQ's defaults) for the monitoring interface for getting
statistical data (`intelmqctl_rabbitmq_monitoring_url`).
- `intelmq.lib.upgrades`: Added a generic upgrade function for
harmonization, checking of all message types, it's fields and their types.
- `intelmq.lib.utils`:
  - `TimeoutHTTPAdapter`: A subclass of `requests.adapters.HTTPAdapter`
with the possibility to set the timeout per adapter.
  - `create_request_session_from_bot`: Use the `TimeoutHTTPAdapter` with
the user-defined timeout. Previously the timeout was not functional.

### Bots
#### Parsers
- `intelmq.bots.parsers.shadowserver.parser`: Fix logging message if the
parameter `feedname` is not present.
- `intelmq.bots.parsers.shodan.parser`: Also add field
`classification.identifier` (`'network-scan'`) in minimal mode.
- `intelmq.bots.parsers.spamhaus.parser_cert`: Add support for category
`'misc'`.
- `intelmq.bots.parsers.cymru.parser_cap_program`:
  - Add support for phishing events without URL.
  - Add support for protocols >= 143 (unassigned, experiments, testing,
reserved), saving the number to extra, as the data would be bogus.
- `intelmq.bots.parsers.microsoft.parser_bingmurls`:
  - Save the `Tags` data as `source.geolocation.cc`.

#### Experts
- `intelmq.bots.experts.modify.expert`: Fix bug with setting non-string
values (#1460).

#### Outputs
- `intelmq.bots.outputs.smtp`:
  - Allow non-existent field in text formatting by using a default value
`None` instead of throwing errors.
  - Fix Authentication (#1464).
  - Fix sending to multiple recipients (#1464).

### Documentation
- Feeds:
  - Fix configuration of `Feodo Tracker Browse` feed.
- Bots:
  - Sieve expert: Document behavior of `!=` with lists.

### Tests
- Adaption and extension of the test cases to the changes.

### Tools
- `intelmq.bin.intelmqctl`:
  - check: Check if running the upgrade function for harmonization is
necessary.
  - upgrade-config: Run the upgrade function for harmonization.
  - `intelmqctl restart` did throw an error as the message for
restarting was not defined (#1465).

### Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo
versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20191111/df07cbd8/attachment.sig>