[Intelmq-dev] IntelMQ & IntelMQ Manager releases 2.2.1

[Sebastian Wagner]

Dear community,

Today we have again a twin release 2.2.1 for both IntelMQ as well as
IntelMQ Manager. This IntelMQ Manager version requires IntelMQ >= 2.2.1.
There are currently issues with the signature in the package
repositories for Debian/Ubuntu. I hope to get them resolved soon.

IntelMQ Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/INSTALL.md
IntelMQ Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/UPGRADING.md
IntelMQ Manager Installation instructions:
https://github.com/certtools/intelmq-manager/blob/2.2.1/docs/INSTALL.md

*The changelog for IntelMQ Manager:*

### Backend
- Fix loading paths from `intelmqctl` executable (PR #205 by Einar
Felipe Lanfranco).

### Documentation
- User Guide:
  - Add section on configuration paths.
  - Add section on named queues / paths.
- Readme:
  - Update screenshots (#201, PR#207 by Mladen Markovic).

### Known issues
* Graph jumps around on "Add edge" (#148).
* wrong error message for new bots with existing ID (#152).
* Monitor page: Automatic log refresh reset log page to first one (#190).

*The News for IntelMQ:*

### Requirements
#### MaxMind GeoIP Expert Bot
The current python library versions of geoip (version 4) and maxminddb
(version 2) no longer support Python 3.5. Keep older versions of these
libraries if you are using this Python version.

### Configuration
#### Abuse.ch URLHaus

The current documented value for the `column` parameter was:
```json
['time.source', 'source.url', 'status', 'extra.urlhaus.threat_type',
'source.fqdn', 'source.ip', 'source.asn', 'source.geolocation.cc']
```
Better is:
```json
['time.source', 'source.url', 'status',
'classification.type|__IGNORE__', 'source.fqdn|__IGNORE__', 'source.ip',
'source.asn', 'source.geolocation.cc']
```

*And the changelog for IntelMQ:*

### Core
- `intelmq.lib.upgrades`:
  - Add upgrade function for changed configuration of the feed "Abuse.ch
URLHaus" (#1571, PR#1572 by Filip Pokorný).
  - Add upgrade function for removal of *HPHosts Hosts file* feed and
`intelmq.bots.parsers.hphosts` parser (#1559).
  - `intelmq.lib.harmonization`:
    - For IP Addresses, explicitly reject IPv6 addresses with scope ID
(due to changed behavior in Python 3.9, #1550).

### Development
- Ignore line length (E501) in code-style checks altogether.

### Bots
#### Collectors
- `intelmq.bots.collectors.misp`: Fix access to actual MISP object
(PR#1548 by Tomas Bellus @tomas321)
- `intelmq.bots.collectors.stomp`: Remove empty `client.pem` file.

#### Parsers
- `intelmq.bots.parsers.shadowserver.config`:
  - Add support for Accessible-CoAP feed (PR #1555 by Thomas Hungenberg).
  - Add support for Accessible-ARD feed (PR #1584 by Tomas Bellus
@tomas321).
- `intelmq.bots.parser.anubisnetworks.parser`: Ignore
"TestSinkholingLoss" events, these are not intended to be sent out at all.
- `intelmq.bots.parsers.generic.parser_csv`: Allow values of type
dictionary for parameter `type_translation`.
- `intelmq.bots.parsers.hphosts`: Removed, feed is unavailable (#1559).
- `intelmq.bots.parsers.cymru.parser_cap_program`: Add support for
comment "username" for "scanner" category.
- `intelmq.bots.parsers.malwareurl.parser`: Check for valid FQDN and IP
address in URL and IP address columns (PR#1585 by Marius Urkis).

#### Experts
- `intelmq.bots.experts.maxmind_geoip`: On Python < 3.6, require
maxminddb < 2, as that version does no longer support Python 3.5.

#### Outputs
- `intelmq.bot.outputs.udp`: Fix error handling on sending, had a bug
itself.

### Documentation
- Feeds:
  - Update documentation of feed "Abuse.ch URLHaus" (#1571, PR#1572 by
Filip Pokorný).
- Bots:
  - Overhaul of all bots' description fields (#1570).
- User-Guide:
  - Overhaul pipeline configuration section and explain named queues
better (#1577).

### Tests
- `intelmq.tests.bots.experts.cymru`: Adapt `test_empty_result`, remove
`test_unicode_as_name` and `test_country_question_mark` (#1576).

### Tools
- `intelmq.bin.intelmq_gen_docs`: Format parameters of types lists with
double quotes around values to produce conform JSON, ready to copy and
paste the value into the IntelMQ Manager's bot parameter form.
- `intelmq.bin.intelmqctl`:
  - `debug`: In JSON mode, use dictionaries instead of lists.
  - `debug`: Add `PATH` to the paths shown.
  - `check`: Show `$PATH` environment variable if executable cannot be
found.

### Contrib
- `malware_name_mapping`: Change MISP Threat Actors URL to new URL
(branch master -> main) in download script.

### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952).
- Corrupt dump files when interrupted during writing (#870).
- Bash completion scripts search in wrong directory in packages (#1561).
- Cymru Expert: Wrong Cache-Key Calculation (#1592).

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20200730/847b9e67/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20200730/847b9e67/attachment.sig>