[Intelmq-dev] IntelMQ 1.1.2 release

[Sebastian Wagner]

Dear community,

This is presumably the last bugfix release before the next feature
release. That is expected to be the version 2.0, not 1.2 in the next month.

The release will hit the deb/rpm repositories shortly.

Thanks to all contributors!

Installation instructions:
https://github.com/certtools/intelmq/blob/1.1.2/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.1.2/docs/UPGRADING.md

### Configuration
#### Feodotracker
 * The URL of the "Feodo Tracker IPs" feed has changed. The new one is `https://feodotracker.abuse.ch/downloads/ipblocklist.csv`. If you are using this feed, adapt your configuration accordingly. The parser has been updated to support the new format.
 * The feed "Feodo Tracker Domains" has been discontinued.

The full changelog:
### Core
- `intelmq.lib.bot`:
  - `Bot.__handle_sighup`: Handle exceptions in `shutdown` method of bots.

### Harmonization
- FQDN: Disallow `:` in FQDN values to prevent values like '10.0.0.1:8080' (#1235).

### Bots
#### Collectors
- `intelmq.bots.collectors.stomp.collector`
  - Fix name of shutdown method, was ineffective in the past.
  - Ignore `NotConnectedException` errors on disconnect during shutdown.
- `intelmq.bots.collectors.mail.collector_mail_url`: Decode body if it is bytes (#1367).
- `intelmq.bots.collectors.tcp.collector`: Timeout added. More stable version.

#### Parsers
- `intelmq.bots.parsers.shadowserver`:
  - Add support for the `Amplification-DDoS-Victim`, `HTTP-Scanners`, `ICS-Scanners` and `Accessible-Ubiquiti-Discovery-Service` feeds (#1368, #1383)
- `intelmq.bots.parsers.microsoft.parser_ctip`:
  - Workaround for mis-formatted data in `networkdestinationipv4` field (since 2019-03-14).
  - Ignore "hostname" ("destination.fqdn") if it contains invalid data.
- `intelmq.bots.parsers.shodan.parser`:
  - In `minimal_mode`:
    - Fix the parsing, previously only `source.geolocation.cc` and `extra.shodan` was correctly filled with information.
    - Add a `classification.type` = 'other' to all events.
    - Added tests for this mode.
  - Normal mode:
    - Fix the parsing of `timestamp` to `time.source in the normal mode, previously no timezone information has been added and thus every event raised an exception.
    - ISAKMP: Ignore `isakmp.aggressive`, as the content is same as `isakmp` or less.
- `intelmq.bots.parsers.abusech.parser_ip`: Re-structure the bot and support new format of the changed "Feodo Tracker Domains" feed.
- `intelmq.bots.parsers.n6.parser`:
  - Add parsing for fields "confidence", "expires" and "source".
  - Add support for type "bl-other" (category "other").

#### Experts
- `intelmq.bots.experts.sieve.expert`: Fix key definition to allow field names with numbers (`malware.hash.md5`/`sha1`, #1371).

#### Outputs
- `intelmq.bots.outputs.tcp.output`: Timeout added. When no separator used, awaits that every message is acknowledged by a simple "Ok" string to ensure more stability.

### Documentation
- Install: Update operating system versions
- Sieve Expert: Fix `elsif` -> `elif`.
- Rephrase the description of `time.*` fields.
- Feeds: New URL and format of the "Feodo Tracker IPs" feed. "Feodo Tracker Domains" has been discontinued.

### Packaging

### Tests
- Add missing `__init__.py` files in 4 bot's test directories. Previously these tests have never been executed.
- `intelmq.lib.test`: Allow bot test class names with an arbitrary postfix separated by an underscore. E.g. `TestShodanParserBot_minimal`.

### Tools
- intelmqctl:
  - status: Show commandline differences if a program with the expected PID could be found, but they do not match (previous output was `None`).
  - Use logging level from defauls configuration if possible, otherwise intelmq's internal default. Previously, DEBUG was used unconditionally.

### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is restarted (#952).
- stomp collector bot constantly uses 100% of CPU (#1364).

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20190325/6b900c21/attachment.sig>