[Intelmq-dev] IntelMQ 2.2.0 release
Sebastian Wagner
wagner at cert.at
Thu Jun 18 11:07:28 CEST 2020
Dear community,
Today I finalized the 2.2.0 Feature release, more than half a year after
2.1.0. Thanks to all the contributors who made this possible!
The release includes six new bots and seven heavily revised bots, and of
course a lot of small changes to various bots. The full changelog can be
found below. We dropped support for Python 3.4, that means Debian 8.0
and similar operating systems are no longer supported.
Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/UPGRADING.md
### Core
- `__init__`: Changes to the path-handling, see [User Guide, section
_/opt and LSB paths_](docs/User-Guide.md#opt-and-lsb-paths) for more
information
- The environment variable `INTELMQ_ROOT_DIR` can be used to set
custom root directories instead of `/opt/intelmq/` (#805) in case of non
LSB-path installations.
- The environment variable `ROOT_DIR` can be used to set custom root
directories instead of `/` (#805) in case of LSB-path installations.
- `intelmq.lib.exceptions`: Added `MissingDependencyError` for show
error messages about a missing library and how to install it (#1471).
- Added optional parameter `installed` to show the installed version.
- Added optional parameter `additional_text` to show arbitrary text.
- Adding more type annotations for core libraries.
- `intelmq.lib.pipeline.Pythonlist.sleep`: Drop deprecated method.
- `intelmq.lib.utils`: `write_configuration`: Append a newline at end of
configuration/file to allow proper comparisons & diffs.
- `intelmq.lib.test`: `BotTestCase` drops privileges upon initialization
(#1489).
- `intelmq.lib.bot`:
- New class `OutputBot`:
- Method `export_event` to format/export events according to the
parameters given by the user.
- `ParserBot`: New methods `parse_json_stream` and
`recover_line_json_stream`.
- `ParserBot.recover_line_json`: Fix format by adding a list around
the line data.
- `Bot.send_message`: In debugging log level, the path to which the
message is sent is now logged too.
### Bots
- Bots with dependencies: Use of
`intelmq.lib.exceptions.MissingDependencyError`.
#### Collectors
- `intelmq.bots.collectors.misp.collector`: Deprecate parameter
`misp_verify` in favor of generic parameter `http_verify_cert`.
- `intelmq.bots.collectors.tcp.collector`: Drop compatibility with
Python 3.4.
- `intelmq.bots.collectors.stomp.collector`:
- Check the stomp.py version and show an error message if it does not
match.
- For stomp.py versions `>= 5.0.0` redirect the
`stomp.PrintingListener` output to debug logging.
- `intelmq.bots.collectors.microsoft.collector_azure`: Support current
Python library `azure-storage-blob>= 12.0.0`, configuration is
incompatible and needs manual change. See NEWS file and bot's
documentation for more details.
- `intelmq.bots.collectors.amqp.collector_amqp`: Require `pika` minimum
version 1.0.
- `intelmq.bots.collectors.github_api.collector_github_contents_api`:
Added (PR#1481).
#### Parsers
- `intelmq.bots.parsers.autoshun.parser`: Drop compatibility with Python
3.4.
- `intelmq.bots.parsers.html_table.parser`: Drop compatibility with
Python 3.4.
- `intelmq.bots.parsers.shadowserver.parser`: Add support for MQTT and
Open-IPP feeds (PR#1512, PR#1544).
- `intelmq.bots.parsers.taichung.parser`:
- Migrate to `ParserBot`.
- Also parse geolocation information if available.
- `intelmq.bots.parsers.cymru.parser_full_bogons`:
- Migrate to `ParserBot`.
- Add last updated information in raw.
- `intelmq.bots.parsers.anubisnetworks.parser`: Add new parameter
`use_malware_familiy_as_classification_identifier`.
- `intelmq.bots.parsers.microsoft.parser_ctip`: Compatibility for new
CTIP data format used provided by the Azure interface.
- `intelmq.bots.parsers.cymru.parser_cap_program`: Support for
`openresolver` type.
- `intelmq.bots.parsers.github_feed.parser`: Added (PR#1481).
- `intelmq.bots.parsers.urlvir.parser`: Removed, as the feed is
discontinued (#1537).
#### Experts
- `intelmq.bots.experts.csv_converter`: Added as converter to CSV.
- `intelmq.bots.experts.misp`: Added (PR#1475).
- `intelmq.bots.experts.modify`: New parameter `maximum_matches`.
#### Outputs
- `intelmq.bots.outputs.amqptopic`:
- Use `OutputBot` and `export_event`.
- Allow formatting the routing key with event data by the new
parameter `format_routing_key` (boolean).
- `intelmq.bots.outputs.file`: Use `OutputBot` and `export_event`.
- `intelmq.bots.outputs.files`: Use `OutputBot` and `export_event`.
- `intelmq.bots.outputs.misp.output_feed`: Added, creates a MISP Feed
(PR#1473).
- `intelmq.bots.outputs.misp.output_api`: Added, pushes to MISP via the
API (PR#1506, PR#1536).
- `intelmq.bots.outputs.elasticsearch.output`: Dropped ElasticSearch
version 5 compatibility, added version 7 compatibility (#1513).
### Documentation
- Document usage of the `INTELMQ_ROOT_DIR` environment variable.
- Added document on MISP integration possibilities.
- Feeds:
- Added "Full Bogons IPv6" feed.
- Remove discontinued URLVir Feeds (#1537).
### Packaging
- `setup.py` do not try to install any data to `/opt/intelmq/` as the
behavior is inconsistent on various systems and with `intelmqsetup` we
have a tool to create the structure and files anyway.
- `debian/rules`:
- Provide a blank state file in the package.
- Patches:
- Updated `fix-intelmq-paths.patch`.
### Tests
- Travis: Use `intelmqsetup` here too.
- Install required build dependencies for the Debian package build test.
- This version is no longer automatically tested on Python `<` 3.5.
- Also run the tests on Python 3.8.
- Run the Debian packaging tests on Python 3.5 and the code-style test
on 3.8.
- Added tests for the new bot `intelmq.bots.outputs.misp.output_feed`
(#1473).
- Added tests for the new bot `intelmq.bots.experts.misp.expert` (#1473).
- Added tests for `intelmq.lib.exceptions`.
- Added tests for `intelmq.lib.bot.OutputBot` and
`intelmq.lib.bot.OutputBot.export_event`.
- Added IPv6 tests for `intelmq.bots.parsers.cymru.parser_full_bogons`.
- Added tests for `intelmq.lib.bot.ParserBot`'s new methods
`parse_json_stream` and `recover_line_json_stream`.
- `intelmq.tests.test_conf`: Set encoding to UTF-8 for reading the
`feeds.yaml` file.
### Tools
- `intelmqctl`:
- `upgrade-config`:
- Allow setting the state file location with the `--state-file`
parameter.
- Do not require a second run anymore, if the state file is newly
created (#1491).
- New parameter `no_backup`/`--no-backup` to skip creation of `.bak`
files for state and configuration files.
- Only require `psutil` for the `IntelMQProcessManager`, not for
process manager independent calls like `upgrade-config` or `check`.
- Add new command `debug` to output some information for debugging.
Currently implemented:
- paths
- environment variables
- `IntelMQController`: New argument `--no-file-logging` to disable
logging to file.
- If dropping privileges does not work, `intelmqctl` will now abort
(#1489).
- `intelmqsetup`:
- Add argument parsing and an option to skip setting file ownership,
possibly not requiring root permissions.
- Call `intelmqctl upgrade-config` and add argument for the state file
path (#1491).
- `intelmq_generate_misp_objects_templates.py`: Tool to create a MISP
object template (#1470).
- `intelmqdump`: New parameter `-t` or `--truncate` to optionally give
the maximum length of `raw` data to show, 0 for no truncating.
### Contrib
- Added `development-tools`.
- ElasticSearch: Dropped version 5 compatibility, added version 7
compatibility (#1513).
- Malware Name Mapping Downloader:
- New parameter `--mwnmp-ignore-adware`.
- The parameter `--add-default` supports an optional parameter to
define the default value.
### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952).
- Corrupt dump files when interrupted during writing (#870).
best regards
Sebastian
--
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20200618/40419770/attachment.sig>
More information about the Intelmq-dev
mailing list