Dear community,
Today I released the first maintenance version of the 1.1.x series.
Thanks for all the contributions!
Installation instructions:
https://github.com/certtools/intelmq/blob/1.1.1/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.1.1/docs/UPGRADING.md
The full change log:
### Core
- `lib/harmonization.py`: Change `parse_utc_isoformat` of `DateTime`
class from private to public (related to #1322).
- `lib/utils.py`: Add new function `object_pair_hook_bots`.
- `lib.bot.py`:
- `ParserBot`'s method `recover_line_csv` now also handles given
`tempdata`.
- `Bot.acknowledge_message()` deletes `__current_message` to free the
memory, saves memory in idling parsers with big reports.
- `start()`: Warn once per run if `error_dump_message` is set to false.
- `Bot.start()`, `ParserBot.process()`: If errors happen on bots
without destination pipeline, the `on_error` path has been queried and
lead to an exception being raised.
- `start()`: If `error_procedure` is pass and on pipeline errors, the
bot retries forever (#1333).
- `lib/message.py`:
- Fix add('extra', ..., overwrite=True): old extra fields have not
been deleted previously (#1335).
- Do not ignore empty or ignored (as defined in `_IGNORED_VALUES`)
values of `extra.*` fields for backwards compatibility (#1335).
- `lib/pipeline.py` (`Redis.receive`): Wait in 1s steps if redis is busy
loading its snapshot from disk (#1334).
### Default configuration
- Set `error_dump_message` to true by default in `defaults.conf`.
- Fixed typo in `defaults.conf`: `proccess_manager` -> `process_manager`
### Development
- `bin/rewrite_config_files.py`: Fix ordering of BOTS file (#1327).
### Harmonization
Update to 2018-09-26 version. New values are per taxonomy:
- Taxonomy 'intrusions':
- "application-compromise"
- "burglary"
- "privileged-account-compromise"
- "unprivileged-account-compromise"
- Taxonomy 'fraud':
- "copyright"
- "masquerade"
- "unauthorized-use-of-resources"
- Taxonomy 'information content security':
- "data-loss"
- Taxonomy 'vulnerable':
- "ddos-amplifier"
- "information-disclosure"
- "potentially-unwanted-accessible"
- "vulnerable-system"
- "weak-crypto"
- Taxonomy 'availability':
- "dos"
- "outage"
- "sabotage"
- Taxonomy 'abusive-content':
- "harmful-speech"
- "violence"
- Taxonomy 'malicious code':
- "malware-distribution"
- Taxonomy 'information-gathering':
- "social-engineering"
- "sniffing"
- Taxonomy 'information content security':
- "Unauthorised-information-access"
- "Unauthorised-information-modification"
### Bots
#### Collectors
- `intelmq.bots.collectors.http.collector_http`:
- Fix parameter name `extract_files` in BOTS (#1331).
- Fix handling of `extract_files` parameter if the value is an empty
string.
- Handle not installed dependency library `requests` gracefully.
- Explain `extract_files` parameter in docs and use a sane default in
BOTS file.
- `intelmq.bots.collectors.mail.collector_mail_url`:
- Handle HTTP status codes != 2xx the same as HTTP timeouts: No
exception, but graceful handling.
- Handle HTTP errors (bad status code and timeouts) with
`error_procedure` == 'pass' but marking the mail as read and logging the
error.
- Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.collectors.http.collector_http_stream`:
- Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.collectors.microsoft.collector_interflow`:
- Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.collectors.rt.collector_rt`:
- Handle not installed dependency library `requests` gracefully.
- added `intelmq.bots.collectors.shodan.collector_stream` for collecting
shodan stream data (#1096).
- Correctly check the version of the shodan library, it resulted in
wrong comparisons with two digit numbers.
- `intelmq.bots.collectors.microsoft.collector_interflow`:
- Add check if Cache's TTL is big enough compared to `not_older_than`
and throw an error otherwise.
#### Parsers
- `intelmq.bots.parsers.misp`: Fix Object attribute (#1318).
- `intelmq.bots.parsers.cymru.parser_cap_program`:
- Add support for new format (extra data about botnet of 'bots').
- Handle AS number 0.
- `intelmq.bots.parsers.shadowserver`:
- Spam URL reports: remove `src_naics`, `src_sic` columns.
- fix parsing of 'spam' events in ShadowServer's 'Botnet Drone Hadoop'
Report (#1271).
- Add support in parser to ignore some columns in config file by using
`False` as intelmq key.
- Add support for the `Outdated-DNSSEC-Key` and
`Outdated-DNSSEC-Key-IPv6` feeds.
- Add support for the `Accessible-Rsync` feed.
- Document support for the `Open-LDAP-TCP` feed.
- Add support for `Accessible-HTTP` and `Open-DB2-Discovery-Service`
(#1349).
- Add support for `Accessible-AFP` (#1351).
- Add support for `Darknet` (#1353).
- `intelmq.bots.parsers.generic.parser_csv`: If the `skip_header`
parameter was set to `True`, the header was not part of the `raw` field
as returned by the `recover_line` method. The header is now saved and
handled correctly by the fixed recovery method.
- `intelmq.bots.parsers.cleanmx.parser`: Use field `first` instead of
`firsttime` for `time.source` (#1329, #1348).
- `intelmq.bots.parsers.twitter.parser`: Support for `url-normalize` >=
1.4.1 and recommend it. Added new optional parameter `default_scheme`,
passed to `url-normalize` (#1356).
#### Experts
- `intelmq.bots.experts.national_cert_contact_certat.expert`:
- Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.experts.ripencc_abuse_contact.expert`:
- Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.experts.sieve.expert`:
- check method: Add missing of the harmonization for the check, caused
an error for every check.
- Add text and more context to error messages.
- README: Fix 'modify' to 'update' (#1340).
- Handle empty rules file (#1343).
- `intelmq.bots.experts.idea.expert`: Add mappings for new harmonization
`classification.type` values, see above.
#### Outputs
- `intelmq.bots.outputs.redis`:
- Fix sending password to redis server.
- Fix for redis-py >= 3.0.0: Convert Event to string explicitly (#1354).
- Use `Redis` class instead of deprecated `StrictRedis` for redis-py
>= 3.0.0 (#1355).
- `intelmq.bots.outputs.mongodb`:
- New parameter `replacement_char` (default: `'_'`) for
non-hierarchical output as dots in key names are not allowed (#1324, #1322).
- Save value of fields `time.observation` and `time.source` as native
datetime object, not as string (#1322).
- `intelmq.bots.outputs.restapi.output`:
- Handle not installed dependency library `requests` gracefully.
### Documentation
- FAQ
- Explanation and solution on orphaned queues.
- Section on how and why to remove `raw` data.
- Add or fix the tables of contents for all documentation files.
- Feeds:
- Fix Autoshun Feed URL (#1325).
- Add parameters `name` and `provider` to `intelmq/etc/feeds.yaml`,
`docs/Feeds.md` and `intelmq/bots/BOTS` (#1321).
- Add SECURITY.md file.
### Packaging
- Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320).
### Tests
- `intelmq.tests.lib.test_bot`: Skip `test_logging_level_other` on
python 3.7 because of unclear behavior related to copies of loggers (#1269).
- `intelmq.tests.bots.collectors.rt.test_collector`: Remove test because
the REST interface of the instance has been closed (see also
https://github.com/CZ-NIC/python-rt/issues/28).
### Tools
- `intelmqctl check`: Shows more detailed information on orphaned queues.
- `intelmqctl`:
- Correctly determine the status of bots started with `intelmqctl run`.
- Fix output of errors during bot status determination, making it
compatible to IntelMQ Manager.
- `check` subcommand: Show bot ID for messages also in JSON output.
- `run [bot-id] process -m [message]` works also with bots without a
configured source pipeline (#1307).
### Contrib
- elasticsearch/elasticmapper: Add tlp field (#1308).
- `feeds-config-generator/intelmq_gen_feeds_conf`:
- Add parameters to write resulting configuration directly to files
(#1321).
- Handle collector's `feed.name` and `feed.provider` (#1314).
### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is restarted
(#952).
- Tests: capture logging with context manager (#1342).
- stomp collector bot constantly uses 100% of CPU (#1364).
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg