Dear community,
Today I released the first maintenance version of the 1.1.x series. Thanks for all the contributions!
Installation instructions: https://github.com/certtools/intelmq/blob/1.1.1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.1/docs/UPGRADING.md
The full change log:
### Core - `lib/harmonization.py`: Change `parse_utc_isoformat` of `DateTime` class from private to public (related to #1322). - `lib/utils.py`: Add new function `object_pair_hook_bots`. - `lib.bot.py`: - `ParserBot`'s method `recover_line_csv` now also handles given `tempdata`. - `Bot.acknowledge_message()` deletes `__current_message` to free the memory, saves memory in idling parsers with big reports. - `start()`: Warn once per run if `error_dump_message` is set to false. - `Bot.start()`, `ParserBot.process()`: If errors happen on bots without destination pipeline, the `on_error` path has been queried and lead to an exception being raised. - `start()`: If `error_procedure` is pass and on pipeline errors, the bot retries forever (#1333). - `lib/message.py`: - Fix add('extra', ..., overwrite=True): old extra fields have not been deleted previously (#1335). - Do not ignore empty or ignored (as defined in `_IGNORED_VALUES`) values of `extra.*` fields for backwards compatibility (#1335). - `lib/pipeline.py` (`Redis.receive`): Wait in 1s steps if redis is busy loading its snapshot from disk (#1334).
### Default configuration - Set `error_dump_message` to true by default in `defaults.conf`. - Fixed typo in `defaults.conf`: `proccess_manager` -> `process_manager`
### Development - `bin/rewrite_config_files.py`: Fix ordering of BOTS file (#1327).
### Harmonization Update to 2018-09-26 version. New values are per taxonomy: - Taxonomy 'intrusions': - "application-compromise" - "burglary" - "privileged-account-compromise" - "unprivileged-account-compromise" - Taxonomy 'fraud': - "copyright" - "masquerade" - "unauthorized-use-of-resources" - Taxonomy 'information content security': - "data-loss" - Taxonomy 'vulnerable': - "ddos-amplifier" - "information-disclosure" - "potentially-unwanted-accessible" - "vulnerable-system" - "weak-crypto" - Taxonomy 'availability': - "dos" - "outage" - "sabotage" - Taxonomy 'abusive-content': - "harmful-speech" - "violence" - Taxonomy 'malicious code': - "malware-distribution" - Taxonomy 'information-gathering': - "social-engineering" - "sniffing" - Taxonomy 'information content security': - "Unauthorised-information-access" - "Unauthorised-information-modification"
### Bots #### Collectors - `intelmq.bots.collectors.http.collector_http`: - Fix parameter name `extract_files` in BOTS (#1331). - Fix handling of `extract_files` parameter if the value is an empty string. - Handle not installed dependency library `requests` gracefully. - Explain `extract_files` parameter in docs and use a sane default in BOTS file. - `intelmq.bots.collectors.mail.collector_mail_url`: - Handle HTTP status codes != 2xx the same as HTTP timeouts: No exception, but graceful handling. - Handle HTTP errors (bad status code and timeouts) with `error_procedure` == 'pass' but marking the mail as read and logging the error. - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.collectors.http.collector_http_stream`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.collectors.microsoft.collector_interflow`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.collectors.rt.collector_rt`: - Handle not installed dependency library `requests` gracefully. - added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (#1096). - Correctly check the version of the shodan library, it resulted in wrong comparisons with two digit numbers. - `intelmq.bots.collectors.microsoft.collector_interflow`: - Add check if Cache's TTL is big enough compared to `not_older_than` and throw an error otherwise.
#### Parsers - `intelmq.bots.parsers.misp`: Fix Object attribute (#1318). - `intelmq.bots.parsers.cymru.parser_cap_program`: - Add support for new format (extra data about botnet of 'bots'). - Handle AS number 0. - `intelmq.bots.parsers.shadowserver`: - Spam URL reports: remove `src_naics`, `src_sic` columns. - fix parsing of 'spam' events in ShadowServer's 'Botnet Drone Hadoop' Report (#1271). - Add support in parser to ignore some columns in config file by using `False` as intelmq key. - Add support for the `Outdated-DNSSEC-Key` and `Outdated-DNSSEC-Key-IPv6` feeds. - Add support for the `Accessible-Rsync` feed. - Document support for the `Open-LDAP-TCP` feed. - Add support for `Accessible-HTTP` and `Open-DB2-Discovery-Service` (#1349). - Add support for `Accessible-AFP` (#1351). - Add support for `Darknet` (#1353). - `intelmq.bots.parsers.generic.parser_csv`: If the `skip_header` parameter was set to `True`, the header was not part of the `raw` field as returned by the `recover_line` method. The header is now saved and handled correctly by the fixed recovery method. - `intelmq.bots.parsers.cleanmx.parser`: Use field `first` instead of `firsttime` for `time.source` (#1329, #1348). - `intelmq.bots.parsers.twitter.parser`: Support for `url-normalize` >= 1.4.1 and recommend it. Added new optional parameter `default_scheme`, passed to `url-normalize` (#1356).
#### Experts - `intelmq.bots.experts.national_cert_contact_certat.expert`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.experts.ripencc_abuse_contact.expert`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.experts.sieve.expert`: - check method: Add missing of the harmonization for the check, caused an error for every check. - Add text and more context to error messages. - README: Fix 'modify' to 'update' (#1340). - Handle empty rules file (#1343). - `intelmq.bots.experts.idea.expert`: Add mappings for new harmonization `classification.type` values, see above.
#### Outputs - `intelmq.bots.outputs.redis`: - Fix sending password to redis server. - Fix for redis-py >= 3.0.0: Convert Event to string explicitly (#1354). - Use `Redis` class instead of deprecated `StrictRedis` for redis-py
= 3.0.0 (#1355).
- `intelmq.bots.outputs.mongodb`: - New parameter `replacement_char` (default: `'_'`) for non-hierarchical output as dots in key names are not allowed (#1324, #1322). - Save value of fields `time.observation` and `time.source` as native datetime object, not as string (#1322). - `intelmq.bots.outputs.restapi.output`: - Handle not installed dependency library `requests` gracefully.
### Documentation - FAQ - Explanation and solution on orphaned queues. - Section on how and why to remove `raw` data. - Add or fix the tables of contents for all documentation files. - Feeds: - Fix Autoshun Feed URL (#1325). - Add parameters `name` and `provider` to `intelmq/etc/feeds.yaml`, `docs/Feeds.md` and `intelmq/bots/BOTS` (#1321). - Add SECURITY.md file.
### Packaging - Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320).
### Tests - `intelmq.tests.lib.test_bot`: Skip `test_logging_level_other` on python 3.7 because of unclear behavior related to copies of loggers (#1269). - `intelmq.tests.bots.collectors.rt.test_collector`: Remove test because the REST interface of the instance has been closed (see also https://github.com/CZ-NIC/python-rt/issues/28).
### Tools - `intelmqctl check`: Shows more detailed information on orphaned queues. - `intelmqctl`: - Correctly determine the status of bots started with `intelmqctl run`. - Fix output of errors during bot status determination, making it compatible to IntelMQ Manager. - `check` subcommand: Show bot ID for messages also in JSON output. - `run [bot-id] process -m [message]` works also with bots without a configured source pipeline (#1307).
### Contrib - elasticsearch/elasticmapper: Add tlp field (#1308). - `feeds-config-generator/intelmq_gen_feeds_conf`: - Add parameters to write resulting configuration directly to files (#1321). - Handle collector's `feed.name` and `feed.provider` (#1314).
### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). - Tests: capture logging with context manager (#1342). - stomp collector bot constantly uses 100% of CPU (#1364).