Dear community,
The long awaited next major release is coming - in May. For now, there's a beta release as it still has a few bugs we should fix.
The release will hit the unstable deb/rpm repositories shortly: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unsta...
Thanks to all contributors who made IntelMQ what it is today!
Installation instructions: https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/UPGRADING.md
The full changelog:
There are some features considered as beta and marked as such in the documentation, do not use them in production yet.
- upgraded all files to python3-only syntax, e.g. use `super()` instead of `super(..., ...)` in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer.
### Removals of deprecated code: - Removed compatibility shim `intelmq.bots.collectors.n6.collector_stomp`, use `intelmq.bots.collectors.stomp.collector` instead (see #1124). - Removed compatibility shim `intelmq.bots.parsers.cymru_full_bogons.parser`, use `intelmq.bots.parsers.cymru.parser_full_bogons` instead. - Removed compatibility shim handing deprecated parameter `feed` for collectors. Use `name` instead. - Removed deprecated and unused method `intelmq.lib.pipeline.Pipeline.sleep`. - Removed support for deprecated parameter `query_ripe_stat` in `intelmq.bots.experts.ripe.expert`, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (#1291). - Removed deprecated and unused function `intelmq.lib.utils.extract_tar`.
### Core - `lib/pipeline`: - Allow setting the broker of source and destination independently. - Support for a new AMQP broker. See User Guide for configuration. (#1179) - `lib/bot`: - Dump messages locks the dump file using unix file locks (#574). - Print idle/rate limit time also in human readable format (#1332). - `set_request_parameters`: Use `{}` as default proxy value instead of `None`. Allows updating of existing proxy dictionaries. - Bots drop privileges if they run as root. - Save statistics on successfully and failed processed messages in the redis database 3. - `lib/utils` - Function `unzip` to extract files from gzipped and/or tar-archives. - New class `ListHandler`: new handler for logging purpose which saves the messages in a list. - Add function `seconds_to_human`. - Add function `drop_privileges`. - `parse_relative`: Strip string before parsing. - `parse_logline`: Do not convert the timestamps to UTC, leave them as is. - `lib/cache`: - Allow ttl to be None explicitly. - Overwrite existing cache keys in the database instead of discarding the new data. - `lib/bot`: - Basic, but easy-to-configure multi-threading using python's `threading` library. See the User-Guide for more information (#111, #186). - `bin/intelmqctl`: - Support for Supervisor as process manager (#693, #1360).
### Harmonization
### Bots #### Collectors - added `intelmq.bots.parsers.opendxl.collector` (#1265). - added `intelmq.bots.collectors.api`: collecting data using an HTTP API (#123, #1187). - added `intelmq.bots.collectors.rsync` (#1286). - `intelmq.bots.collectors.http.collector_http`: - Add support for uncompressing of gzipped-files (#1270). - Add time-delta support for time formatted URLs (#1366). - `intelmq.collectors.blueliv.collector_crimeserver`: Allow setting the API URL by parameter (#1336). - `intelmq.collectors.mail`: - Use internal lib for functionality. - Add `intelmq.bots.collectors.mail.collector_mail_body`. - Support for `ssl_ca_certificate` parameter (#1362).
#### Parsers - added `intelmq.bots.parsers.mcafee.parser_atd` (#1265). - `intelmq.bots.parsers.generic.parser_csv`: - New parameter `columns_required` to optionally ignore parse errors for columns. - added `intelmq.bots.parsers.cert_eu.parser_csv` (#1287). - Do not overwrite the local `time.observation` with the data from the feed. The feed's field 'observation time' is now saved in the field `extra.cert_eu_time_observation`. - Fix parsing of `asn` (renamed to `source asn`, `source.asn` internally) and handle existing `feed.accuracy` for parsing `confidence`. - Update columns and mapping to current (2019-04-02) data. - added `intelmq.bots.parsers.surbl.surbl` - added `intelmq.bots.parsers.html_table` (#1381). - `intelmq.bot.parsers.netlab_360.parser`: Handle empty lines containing blank characters (#1393). - `intelmq.bots.parsers.n6.parser_n6stomp`: Handle events without IP addresses. - `intelmq.bots.parsers.cymru.parser_cap_program`: Handle new feed format. - `intelmq.bots.parsers.shadowserver`: - Add support for the `Accessible-FTP` feed (#1391). - `intelmq.bots.parsers.dataplane.parser`: - Fix parse errors and log more context (#1396). - added `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py` (#1373).
#### Experts - added `intelmq.bots.experts.recordedfuture_iprisk` (#1267). - added `intelmq.bots.experts.mcafee.expert_mar` (1265). - renamed `intelmq.bots.experts.ripencc_abuse_contact.expert` to `intelmq.bots.experts.ripe.expert`, compatibility shim will be removed in version 3.0. - Added support for geolocation information in ripe expert with a new parameter `query_ripe_stat_geolocation` (#1317). - Restructurize the expert and de-duplicataion (#1384). - Handle '?' in geolocation country data (#1384). - `intelmq.bots.experts.ripe.expert`: - Use a requests session (#1363). - Set the requests parameters once per session. - `intelmq.bots.experts.maxmind_geoip.expert`: New parameter `use_registered` to use the registered country (#1344). - `intelmq.bots.experts.filter.expert`: Support for paths (#1208).
#### Outputs - added `intelmq.bots.experts.mcafee.output_esm` (1265). - added `intelmq.bots.outputs.blackhole` (#1279). - `intelmq.bots.outputs.restapi.expert`: - Set the requests parameters once per session. - `intelmq.bots.outputs.redis`: - New parameter `hierarchichal_output` (#1388). - New parameter `with_type`. - `intelmq.bots.outputs.amqptopic.output`: Compatibility with pika 1.0.0 (#1084, #1394).
### Documentation - added documentation for feeds - CyberCrime Tracker - Feodo Tracker Latest - Feeds: Document abuse.ch URLhaus feed (#1379). - Install and Upgrading: Use `intelmqsetup` tool.
### Packaging
### Tests - Add tests of AMQP broker. - Travis: Change the ownership of `/opt/intelmq` to the current user.
### Tools - `intelmqctl check`: Now uses the new `ListHandler` from utils to handle the logging in JSON output mode. - `intelmqctl run`: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behaviour. - `intelmqdump`: Inspecting dumps locks the dump file using unix file locks (#574). - `intelmqctl`: - After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown. - `intelmqsetup`: New tool for initialize an IntelMQ environment.
### Contrib - `malware_name_mapping`: - Added the script `apply_mapping_eventdb.py` to apply the mapping to an eventdb. - Possibility to add local rules using the download tool. - `check_mk`: - Added scripts for monitoring queues and statistics.
### Known issues - Multi-threaded bots require multiple SIGTERMs (#1403) - Stats can't be saved with AMQP if redis is password-protected (#1402) - Update taxonomies to current RSIT and vice-versa (#1380) - stomp collector bot constantly uses 100% of CPU (#1364) - tests: capture logging with context manager (#1342) - Consistent message counter log messages for all kind of bots (#1278) - pymongo 3.0 deprecates used insert method (#1063) - pymongo >= 3.5: authentication changes (#1062) - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952) - n6 parser: mapping is modified within each run (#905) - reverse DNS: Only first record is used (#877) - Corrupt dump files when interrupted during writing (#870)