[Intelmq-users] IntelMQ release 2.2.2

[Sebastian Wagner]

Dear community,

It's again long overdue for a new release and here it is finally. Since
August we collected quite a few bugfixes - Thanks to all contributors!

IntelMQ Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.2/docs/INSTALL.md
IntelMQ Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.2/docs/UPGRADING.md

*News for IntelMQ 2.2.2*

### Bots
#### Cymru Whois Lookup
The cache key calculation has been fixed. It previously led to duplicate
keys for different IP addresses and therefore wrong results in rare
cases. The cache key calculation is intentionally not
backwards-compatible. Therefore, this bot may take longer processing
events than usual after applying this update.
More details can be found in [issue
#1592](https://github.com/certtools/intelmq/issues/1592).

### Harmonization
#### Shadowserver Feed/Parser
The feed "Blacklisted-IP" has been renamed by ShadowServer to
"Blocklist". In IntelMQ, the old name can still be used in IntelMQ until
version 3.0.

*Changes for IntelMQ 2.2.2*

### Core
- `intelmq.lib.upgrades`:
  - Add upgrade function for renamed Shadowserver feed name
"Blacklisted-IP"/"Blocklist".

### Bots
#### Parsers
- `intelmq.bots.parsers.shadowserver`:
  - Rename "Blacklisted-IP" feed to "Blocklist", old name is still valid
until IntelMQ version 3.0 (PR#1588 by Thomas Hungenberg).
  - Added support for the feeds `Accessible Radmin` and `CAIDA IP
Spoofer` (PR#1600 by sinus-x).
- `intelmq.bots.parsers.anubisnetworks.parser`: Fix parsing error where
`dst.ip` was not equal to `comm.http.host`.
- `intelmq/bots/parsers/danger_rulez/parser`: correctly skip malformed
rows by defining variables before referencing (PR#1601 by Tomas Bellus).
- `intelmq.bots.parsers.misp.parser: Fix MISP Event URL (#1619, PR#1618
by Nedfire23).
- `intelmq.bots.parsers.microsoft.parser_ctip`:
  - Add support for `DestinationIpInfo.*` and `Signatures.Sha256`
fields, used by the `ctip-c2` feed (PR#1623 by Mikk Margus Möll).
  - Use `extra.payload.text` for the feed's field `Payload` if the
content cannot be decoded (PR#1610 by Giedrius Ramas).

#### Experts
- `intelmq.bots.experts.cymru_whois`:
  - Fix cache key calculation which previously led to duplicate keys and
therefore wrong results in rare cases. The cache key calculation is
intentionally not backwards-compatible (#1592, PR#1606).
  - The bot now caches and logs (as level INFO) empty responses from
Cymru (PR#1606).

### Documentation
- README:
  - Add Core Infrastructure Initiative Best Practices Badge.
- Bots:
  - Generic CSV Parser: Add note on escaping backslashes (#1579).
  - Remove section of non-existing "Copy Extra" Bot.
  - Explain taxonomy expert.
  - Add documentation on n6 parser.
  - Gethostbyname expert: Add documentation how errors are treated.
- Feeds:
  - Fixed bot modules of Calidog CertStream feed.
  - Add information on Microsoft CTIP C2 feed.

### Packaging
- In Debian packages, `intelmqctl check` and `intelmqctl upgrade-config`
are executed in the postinst step (#1551, PR#1624 by Birger Schacht).

### Tests
- `intelmq.tests.lib.test_pipeline`: Skip `TestAmqp.test_acknowledge` on
Travis with Python 3.8.
- `intelmq.tests.bots.outputs.elasticsearch.test_output`: Refresh index
`intelmq` manually to fix random test failures (#1593, PR#1595 by Zach
Stone).

### Tools
- `intelmqctl check`:
  - For disabled bots which do not have any pipeline connections, do not
raise an error, but only warning.
  - Fix check on source/destination queues for bots as well the orphaned
queues.

### Contrib
- Bash completion scripts: Check both `/opt/intelmq/` as well as
LSB-paths (`/etc/intelmq/` and `/var/log/intelmq/`) for loading bot
information (#1561, PR#1628 by Birger Schacht).

### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952).
- Corrupt dump files when interrupted during writing (#870).

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20201028/0ea97e92/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20201028/0ea97e92/attachment.sig>