[Intelmq-dev] IntelMQ 2.0.0 Beta 1 release

Sebastian Wagner wagner at cert.at
Wed Apr 10 15:29:40 CEST 2019 

Dear community,

The long awaited next major release is coming - in May. For now, there's a beta release as it still has a few bugs we should fix.

The release will hit the unstable deb/rpm repositories shortly:
https://software.opensuse.org/download.html?project=home:sebix:intelmq:unstable&package=intelmq

Thanks to all contributors who made IntelMQ what it is today!

Installation instructions:
https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/UPGRADING.md

The full changelog:

There are some features considered as beta and marked as such in the documentation, do not use them in production yet.

- upgraded all files to python3-only syntax, e.g. use `super()` instead of `super(..., ...)` in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer.

### Removals of deprecated code:
- Removed compatibility shim `intelmq.bots.collectors.n6.collector_stomp`, use `intelmq.bots.collectors.stomp.collector` instead (see #1124).
- Removed compatibility shim `intelmq.bots.parsers.cymru_full_bogons.parser`, use `intelmq.bots.parsers.cymru.parser_full_bogons` instead.
- Removed compatibility shim handing deprecated parameter `feed` for collectors. Use `name` instead.
- Removed deprecated and unused method `intelmq.lib.pipeline.Pipeline.sleep`.
- Removed support for deprecated parameter `query_ripe_stat` in `intelmq.bots.experts.ripe.expert`, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (#1291).
- Removed deprecated and unused function `intelmq.lib.utils.extract_tar`.

### Core
- `lib/pipeline`:
  - Allow setting the broker of source and destination independently.
  - Support for a new AMQP broker. See User Guide for configuration. (#1179)
- `lib/bot`:
  - Dump messages locks the dump file using unix file locks (#574).
  - Print idle/rate limit time also in human readable format (#1332).
  - `set_request_parameters`: Use `{}` as default proxy value instead of `None`. Allows updating of existing proxy dictionaries.
  - Bots drop privileges if they run as root.
  - Save statistics on successfully and failed processed messages in the redis database 3.
- `lib/utils`
  - Function `unzip` to extract files from gzipped and/or tar-archives.
  - New class `ListHandler`: new handler for logging purpose which saves the messages in a list.
  - Add function `seconds_to_human`.
  - Add function `drop_privileges`.
  - `parse_relative`: Strip string before parsing.
  - `parse_logline`: Do not convert the timestamps to UTC, leave them as is.
- `lib/cache`:
  - Allow ttl to be None explicitly.
  - Overwrite existing cache keys in the database instead of discarding the new data.
- `lib/bot`:
  - Basic, but easy-to-configure multi-threading using python's `threading` library. See the User-Guide for more information (#111, #186).
- `bin/intelmqctl`:
  - Support for Supervisor as process manager (#693, #1360).

### Harmonization

### Bots
#### Collectors
- added `intelmq.bots.parsers.opendxl.collector` (#1265).
- added `intelmq.bots.collectors.api`: collecting data using an HTTP API (#123, #1187).
- added `intelmq.bots.collectors.rsync` (#1286).
- `intelmq.bots.collectors.http.collector_http`:
  - Add support for uncompressing of gzipped-files (#1270).
  - Add time-delta support for time formatted URLs (#1366).
- `intelmq.collectors.blueliv.collector_crimeserver`: Allow setting the API URL by parameter (#1336).
- `intelmq.collectors.mail`:
  - Use internal lib for functionality.
  - Add `intelmq.bots.collectors.mail.collector_mail_body`.
  - Support for `ssl_ca_certificate` parameter (#1362).

#### Parsers
- added `intelmq.bots.parsers.mcafee.parser_atd` (#1265).
- `intelmq.bots.parsers.generic.parser_csv`:
  - New parameter `columns_required` to optionally ignore parse errors for columns.
- added `intelmq.bots.parsers.cert_eu.parser_csv` (#1287).
  - Do not overwrite the local `time.observation` with the data from the feed. The feed's field 'observation time' is now saved in the field `extra.cert_eu_time_observation`.
  - Fix parsing of `asn` (renamed to `source asn`, `source.asn` internally) and handle existing `feed.accuracy` for parsing `confidence`.
  - Update columns and mapping to current (2019-04-02) data.
- added `intelmq.bots.parsers.surbl.surbl`
- added `intelmq.bots.parsers.html_table` (#1381).
- `intelmq.bot.parsers.netlab_360.parser`: Handle empty lines containing blank characters (#1393).
- `intelmq.bots.parsers.n6.parser_n6stomp`: Handle events without IP addresses.
- `intelmq.bots.parsers.cymru.parser_cap_program`: Handle new feed format.
- `intelmq.bots.parsers.shadowserver`:
  - Add support for the `Accessible-FTP` feed (#1391).
- `intelmq.bots.parsers.dataplane.parser`:
  - Fix parse errors and log more context (#1396).
- added `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py` (#1373).

#### Experts
- added `intelmq.bots.experts.recordedfuture_iprisk` (#1267).
- added `intelmq.bots.experts.mcafee.expert_mar` (1265).
- renamed `intelmq.bots.experts.ripencc_abuse_contact.expert` to `intelmq.bots.experts.ripe.expert`, compatibility shim will be removed in version 3.0.
  - Added support for geolocation information in ripe expert with a new parameter `query_ripe_stat_geolocation` (#1317).
  - Restructurize the expert and de-duplicataion (#1384).
  - Handle '?' in geolocation country data (#1384).
- `intelmq.bots.experts.ripe.expert`:
  - Use a requests session (#1363).
  - Set the requests parameters once per session.
- `intelmq.bots.experts.maxmind_geoip.expert`: New parameter `use_registered` to use the registered country (#1344).
- `intelmq.bots.experts.filter.expert`: Support for paths (#1208).

#### Outputs
- added `intelmq.bots.experts.mcafee.output_esm` (1265).
- added `intelmq.bots.outputs.blackhole` (#1279).
- `intelmq.bots.outputs.restapi.expert`:
  - Set the requests parameters once per session.
- `intelmq.bots.outputs.redis`:
  - New parameter `hierarchichal_output` (#1388).
  - New parameter `with_type`.
- `intelmq.bots.outputs.amqptopic.output`: Compatibility with pika 1.0.0 (#1084, #1394).

### Documentation
- added documentation for feeds
  - CyberCrime Tracker
  - Feodo Tracker Latest
- Feeds: Document abuse.ch URLhaus feed (#1379).
- Install and Upgrading: Use `intelmqsetup` tool.

### Packaging

### Tests
- Add tests of AMQP broker.
- Travis: Change the ownership of `/opt/intelmq` to the current user.

### Tools
- `intelmqctl check`: Now uses the new `ListHandler` from utils to handle the logging in JSON output mode.
- `intelmqctl run`: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behaviour.
- `intelmqdump`: Inspecting dumps locks the dump file using unix file locks (#574).
- `intelmqctl`:
  - After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown.
- `intelmqsetup`: New tool for initialize an IntelMQ environment.

### Contrib
- `malware_name_mapping`:
  - Added the script `apply_mapping_eventdb.py` to apply the mapping to an eventdb.
  - Possibility to add local rules using the download tool.
- `check_mk`:
  - Added scripts for monitoring queues and statistics.

### Known issues
- Multi-threaded bots require multiple SIGTERMs (#1403)
- Stats can't be saved with AMQP if redis is password-protected (#1402)
- Update taxonomies to current RSIT and vice-versa (#1380)
- stomp collector bot constantly uses 100% of CPU (#1364)
- tests: capture logging with context manager (#1342)
- Consistent message counter log messages for all kind of bots (#1278)
- pymongo 3.0 deprecates used insert method  (#1063)
- pymongo >= 3.5: authentication changes  (#1062)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20190410/1527fa06/attachment.sig>

More information about the Intelmq-dev mailing list