[Intelmq-dev] IntelMQ 2.1.0 release

[Sebastian Wagner]

Dear community,

Given the vast amount of changes, additions and new features, it's time
to mark a new feature release! Thanks to all the contributors who
participate in this community project! IntelMQ gained a lot of new bots
and features in the last months.

Install documentation:
https://github.com/certtools/intelmq/blob/2.1.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.0/docs/UPGRADING.md

Full changelog:

### Core
- `intelmq.lib.harmonization`:
  - Use correct parent classes.
  - Add `DateTime.convert` as interface for all existing conversion
functions.
  - add `DateTime.convert_from_format`.
  - add `DateTime.convert_from_format_midnight`.
  - add `DateTime.convert_fuzzy`.
- `intelmq.lib.pipeline`:
  - Redis: Use single connection client if calling bot is not
multithreaded. Gives a small speed advantage.
  - Require the bot instance as parameter for all pipeline classes.
  - New internal variable `_has_message` to keep the state of the pipeline.
  - Split receive and acknowledge into public-facing and private methods.
  - Add `reject_message` method to the Pipeline class for explicit
requeue of messages.
  - AMQP:
    - Make exchange configurable.
    - If exchange is set, the queues are not declared, the queue name is
for routing used by exchanges.
- `intelmq.lib.bot`:
  - Log message after successful bot initialization, no log message
anymore for ready pipeline.
  - Use existing current message if receive is called and the current
message still exists.
  - Fix handling of received messaged after a sighup that happend during
a blocking receving connection using explicit rejection (#1438).
  - New method `_parse_common_parameters` called before `init` to parse
commonly used argument. Currently supported: `extract_files`.
- `intelmq.lib.test`:
  - Fix the tests broker by providing the testing pipeline.
- `intelmq.lib.utils`:
  - `unzip`:
    - new parameter `return_names` to optionally return the file names.
    - support for zip
    - new parameters `try_zip`, `try_gzip` and `try_tar` to control
which compressions are tried.
    - rewritten to an iterative approach
  - add `file_name_from_response` to extract a file name from a Response
object for downloaded files.
- `intelmq.lib.upgrades`: Added `v210_deprecations` for deprecated
parameters.

### Harmonization
- Add extra to reports.

### Bots
#### Collectors
- `intelmq.bots.collectors.http.collector_http`:
  - More extensive usage of `intelmq.lib.utils.unzip`.
  - Save the file names in the report if files have been extracted form
an archive.
- `intelmq.bots.collectors.rt.collector_rt`:
  - Save ticket information/metadata in the extra fields of the report.
  - Support for RT 3.8 and RT 4.4.
  - New parameters `extract_attachment` and `extract_download` for
generic archive extraction and consistency. The parameter
`unzip_attachment` is deprecated.
- `intelmq.bots.collectors.mail.*`: Save email information/metadata in
the extra fields of the report. See the bots documentation for a
complete list of provided data.
  - `intelmq.bots.collectors.mail.collector_mail_attach`: Check for
existence/validity of the `attach_regex` parameter.
  - Use the lib's `unzip` function for uncompressing attachments and use
the .
  - `intelmq.bots.collectors.mail.collector_mail_url`: Save the file
name of the downloaded file as `extra.file_name`.
- `intelmq.bots.collectors.amqp.collector_amqp`: New collector to
collect data from (remote) AMQP servers, for bot IntelMQ as well as
external data.
  - use default SSL context for client purposes, fixes compatibility
with python < 3.6 if TLS is used.

#### Parsers
- `intelmq.bot.parsers.html_table.parser`:
  * New parameter "html_parser".
  * Use time conversion functions directly from
`intelmq.lib.harmonization.DateTime.convert`.
  - Limit lxml dependency on 3.4 to < 4.4.0 (incompatibility).
- `intelmq.bots.parsers.netlab_360.parser`: Add support for hajime scanners.
- `intelmq.bots.parsers.hibp.parser_callback`: A new parser to parse
data retrieved from a HIBP Enterprise Subscription.
- `intelmq.bots.parsers.shadowserver.parser`:
  - Ability to detect the feed base on the reports's field
`extra.file_name`, so the parameter `feedname` is no longer required and
one configured parser can parse any feed (#1442).

#### Experts
- Add geohash expert.
- `intelmq.bot.experts.generic_db_lookup.expert`
  - new optional parameter `engine` with `postgresql` (default) and
`sqlite` (new) as possible values.

#### Outputs
- Add `intelmq.bots.outputs.touch.output`.
- `intelmq.bot.outputs.postgresql.output`:
  - deprecated in favor of `intelmq.bot.outputs.sql.output`
  - Compatibility shim will be available in the 2.x series.
- `intelmq.bot.outputs.sql.output` added generic SQL output bot.
Comparted to
  - new optional parameter `engine` with `postgresql` (default) and
`sqlite` (new) as possible values.
- `intelmq.bots.outputs.stomp.output`: New parameters
`message_hierarchical_output`, `message_jsondict_as_string`,
`message_with_type`, `single_key`.

### Documentation
- Feeds:
  - Add ViriBack feed.
  - Add Have I Been Pwned Enterprise Callback.
- `intelmq.tests.bots.outputs.amqptopic.test_output`: Added.
- Move the documentation of most bots from separate README files to the
central Bots.md and feeds.yaml files.

### Tests
- Travis:
  - Use UTC timezone.
- Tests for `utils.unzip`.
- Add a new asset: Zip archive with two files, same as with tar.gz archive.
- Added tests for the Mail Attachment & Mail URL collectors.
- Ignore logging-tests on Python 3.7 temporarily (#1342).

### Tools
- intelmqctl:
  - Use green and red text color for some interactive output to indicate
obvious errors or the absence of them.
- intelmqdump:
  - New edit action `v` to modify a message saved in the dump (#1284).

### Contrib
* malware name mapping:
  * Add support for MISP treat actors data, see it's README for more
information.
    * And handle empty synonyms in misp's galxies data.
  * Move apply-Script to the new EventDB directory
* EventDB: Scripts for applying malware name mapping and domain suffixes
to an EventDB.

### Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo
versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20191015/404f4bf1/attachment.sig>