Dear community
I have just release a new bugfix release of IntelMQ.
Installation instructions: https://github.com/certtools/intelmq/blob/1.0.3/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.3/docs/UPGRADING.md
The released is published on PyPI, Github and the OpenBuildService (for rpm/deb packages). If you installed intelmq with a package manager, the new released will be installed automatically.
Full changelog:
### Contrib * logrotate: use sudo for postrotate script * cron-jobs: use the scripts in the bots' directories and link them (#1056, #1142)
### Core - `lib.harmonization`: Handle idna encoding error in FQDN sanitation (#1175, #1176). - `lib.bot`: - Bots stop when redis gives the error "OOM command not allowed when used memory > 'maxmemory'." (#1138). - warnings of bots are catched by the logger (#1074, #1113). - Fixed exitcodes 0 for graceful shutdowns . - better handling of problems with pipeline and especially it's initialization (#1178). - All parsers using `ParserBot`'s methods now log the sum of successfully parsed and failed lines at the end of each run (#1161).
### Harmonization - Rule for harmonization keys is enforced (#1104, #1141). - New allowed values for `classification.type`: `tor` & `leak` (see n6 parser below ).
### Bots #### Collectors - `bots.collectors.mail.collector_mail_attach`: Support attachment file parsing for imbox versions newer than 0.9.5 (#1134). - `bots.outputs.smtp.output`: Fix STARTTLS, threw an exception (#1152, #1153).
#### Parsers - All CSV parsers ignore NULL-bytes now, because the csv-library cannot handle it (#967, #1114). - `bots.experts.modify` default ruleset: changed conficker rule to catch more spellings. - `bots.parsers.shadowserver.parser`: Add Accessible Cisco Smart Install (#1122). - `bots.parsers.cleanmx.parser`: Handle new columns `first` and `last`, rewritten for XML feed. See NEWS.md for upgrade instructions (#1131, #1136, #1163). - `bots.parsers.n6.parser`: Fix classification mappings. See NEWS file for changes values (#738, #1127).
### Documentation - `Release.md` add release procedure documentation - `Bots.md`: fix example configuration for modify expert
### Tools - intelmqctl now exits with exit codes > 0 when errors happened or the operation was not successful. Also, the status operation exits with 1, if bots are stopped, but enabled. (#977, #1143) - `intelmctl check` checks for valid `run_mode` in runtime configuration (#1140).
### Tests - `tests.lib.test_pipeline`: Redis tests clear all queues before and after tests (#1086). - Repaired debian package build on travis (#1169). - Warnings are not allowed by default, an allowed count can be specified (#1129). - `tests.bots.experts.cymru_whois/abusix`: Skipped on travis because of ongoing problems.
### Packaging * cron jobs: fix paths of executables
### Known issues - `bots.collectors/outputs.xmpp` must be killed two times (#970). - When running bots with `intelmqctl run [bot-id]` the log level is always INFO (#1075). - `intelmqctl run [bot-id] message send [msg]` does only support Events, not Reports (#1077). - `python3 setup.py sdist` does not include static files in the resulting tarballs (#1146). - `bots.parsers.cleanmx.parser`: The cleanMX feed may have FQDNs as IPs in rare cases, such lines are dumped (#1162).
Sebastian