Dear developers and "pro-users",
In the current (beta) version and develop branch of intelmq there are
currently four optional features which are not considered stable yet and
thus marked as beta. All of them need more testing, development and your
feedback!
# AMQP as broker
It is now possible to use AMQP instead of Redis as broker. This is
especially useful if you need to send data between servers or even
organisations as RabbitMQ (one implementation of the AMQP protocol)
supports encryption, authentication etc. Also, you have some more
monitoring included as RabbitMQ provides statistics for all the queues.
However, it is expected to be slower than Redis as it has more overhead.
Documentation:
https://github.com/certtools/intelmq/blob/develop/docs/User-Guide.md#amqp-b…
# Supervisor as process manager
Instead of the internal pid-based process management, you can now use
supervisor.
Documentation:
https://github.com/certtools/intelmq/blob/develop/docs/User-Guide.md#using-…
# Multithreading
Basic multithreading for all bots has been implemented, you can use the
parameter `instances_threads` to spawn multiple threads of one bot.
See also the documentation and some cavecats in the User Guide:
https://github.com/certtools/intelmq/blob/develop/docs/User-Guide.md#multit…
One of the current bugs is that you need to kill a bot multiple times:
https://github.com/certtools/intelmq/issues/1403
# Statistics
While the statistcs function and code work fine, we are interested in
your feedback to improve this feature. The discussion take place here yet:
https://github.com/certtools/intelmq/issues/1274
So the "beta" character for this feature comes from the fact that it
might change. We also need some documentation on it.
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
The long awaited next major release is coming - in May. For now, there's a beta release as it still has a few bugs we should fix.
The release will hit the unstable deb/rpm repositories shortly:
https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst…
Thanks to all contributors who made IntelMQ what it is today!
Installation instructions:
https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/UPGRADING.md
The full changelog:
There are some features considered as beta and marked as such in the documentation, do not use them in production yet.
- upgraded all files to python3-only syntax, e.g. use `super()` instead of `super(..., ...)` in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer.
### Removals of deprecated code:
- Removed compatibility shim `intelmq.bots.collectors.n6.collector_stomp`, use `intelmq.bots.collectors.stomp.collector` instead (see #1124).
- Removed compatibility shim `intelmq.bots.parsers.cymru_full_bogons.parser`, use `intelmq.bots.parsers.cymru.parser_full_bogons` instead.
- Removed compatibility shim handing deprecated parameter `feed` for collectors. Use `name` instead.
- Removed deprecated and unused method `intelmq.lib.pipeline.Pipeline.sleep`.
- Removed support for deprecated parameter `query_ripe_stat` in `intelmq.bots.experts.ripe.expert`, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (#1291).
- Removed deprecated and unused function `intelmq.lib.utils.extract_tar`.
### Core
- `lib/pipeline`:
- Allow setting the broker of source and destination independently.
- Support for a new AMQP broker. See User Guide for configuration. (#1179)
- `lib/bot`:
- Dump messages locks the dump file using unix file locks (#574).
- Print idle/rate limit time also in human readable format (#1332).
- `set_request_parameters`: Use `{}` as default proxy value instead of `None`. Allows updating of existing proxy dictionaries.
- Bots drop privileges if they run as root.
- Save statistics on successfully and failed processed messages in the redis database 3.
- `lib/utils`
- Function `unzip` to extract files from gzipped and/or tar-archives.
- New class `ListHandler`: new handler for logging purpose which saves the messages in a list.
- Add function `seconds_to_human`.
- Add function `drop_privileges`.
- `parse_relative`: Strip string before parsing.
- `parse_logline`: Do not convert the timestamps to UTC, leave them as is.
- `lib/cache`:
- Allow ttl to be None explicitly.
- Overwrite existing cache keys in the database instead of discarding the new data.
- `lib/bot`:
- Basic, but easy-to-configure multi-threading using python's `threading` library. See the User-Guide for more information (#111, #186).
- `bin/intelmqctl`:
- Support for Supervisor as process manager (#693, #1360).
### Harmonization
### Bots
#### Collectors
- added `intelmq.bots.parsers.opendxl.collector` (#1265).
- added `intelmq.bots.collectors.api`: collecting data using an HTTP API (#123, #1187).
- added `intelmq.bots.collectors.rsync` (#1286).
- `intelmq.bots.collectors.http.collector_http`:
- Add support for uncompressing of gzipped-files (#1270).
- Add time-delta support for time formatted URLs (#1366).
- `intelmq.collectors.blueliv.collector_crimeserver`: Allow setting the API URL by parameter (#1336).
- `intelmq.collectors.mail`:
- Use internal lib for functionality.
- Add `intelmq.bots.collectors.mail.collector_mail_body`.
- Support for `ssl_ca_certificate` parameter (#1362).
#### Parsers
- added `intelmq.bots.parsers.mcafee.parser_atd` (#1265).
- `intelmq.bots.parsers.generic.parser_csv`:
- New parameter `columns_required` to optionally ignore parse errors for columns.
- added `intelmq.bots.parsers.cert_eu.parser_csv` (#1287).
- Do not overwrite the local `time.observation` with the data from the feed. The feed's field 'observation time' is now saved in the field `extra.cert_eu_time_observation`.
- Fix parsing of `asn` (renamed to `source asn`, `source.asn` internally) and handle existing `feed.accuracy` for parsing `confidence`.
- Update columns and mapping to current (2019-04-02) data.
- added `intelmq.bots.parsers.surbl.surbl`
- added `intelmq.bots.parsers.html_table` (#1381).
- `intelmq.bot.parsers.netlab_360.parser`: Handle empty lines containing blank characters (#1393).
- `intelmq.bots.parsers.n6.parser_n6stomp`: Handle events without IP addresses.
- `intelmq.bots.parsers.cymru.parser_cap_program`: Handle new feed format.
- `intelmq.bots.parsers.shadowserver`:
- Add support for the `Accessible-FTP` feed (#1391).
- `intelmq.bots.parsers.dataplane.parser`:
- Fix parse errors and log more context (#1396).
- added `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py` (#1373).
#### Experts
- added `intelmq.bots.experts.recordedfuture_iprisk` (#1267).
- added `intelmq.bots.experts.mcafee.expert_mar` (1265).
- renamed `intelmq.bots.experts.ripencc_abuse_contact.expert` to `intelmq.bots.experts.ripe.expert`, compatibility shim will be removed in version 3.0.
- Added support for geolocation information in ripe expert with a new parameter `query_ripe_stat_geolocation` (#1317).
- Restructurize the expert and de-duplicataion (#1384).
- Handle '?' in geolocation country data (#1384).
- `intelmq.bots.experts.ripe.expert`:
- Use a requests session (#1363).
- Set the requests parameters once per session.
- `intelmq.bots.experts.maxmind_geoip.expert`: New parameter `use_registered` to use the registered country (#1344).
- `intelmq.bots.experts.filter.expert`: Support for paths (#1208).
#### Outputs
- added `intelmq.bots.experts.mcafee.output_esm` (1265).
- added `intelmq.bots.outputs.blackhole` (#1279).
- `intelmq.bots.outputs.restapi.expert`:
- Set the requests parameters once per session.
- `intelmq.bots.outputs.redis`:
- New parameter `hierarchichal_output` (#1388).
- New parameter `with_type`.
- `intelmq.bots.outputs.amqptopic.output`: Compatibility with pika 1.0.0 (#1084, #1394).
### Documentation
- added documentation for feeds
- CyberCrime Tracker
- Feodo Tracker Latest
- Feeds: Document abuse.ch URLhaus feed (#1379).
- Install and Upgrading: Use `intelmqsetup` tool.
### Packaging
### Tests
- Add tests of AMQP broker.
- Travis: Change the ownership of `/opt/intelmq` to the current user.
### Tools
- `intelmqctl check`: Now uses the new `ListHandler` from utils to handle the logging in JSON output mode.
- `intelmqctl run`: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behaviour.
- `intelmqdump`: Inspecting dumps locks the dump file using unix file locks (#574).
- `intelmqctl`:
- After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown.
- `intelmqsetup`: New tool for initialize an IntelMQ environment.
### Contrib
- `malware_name_mapping`:
- Added the script `apply_mapping_eventdb.py` to apply the mapping to an eventdb.
- Possibility to add local rules using the download tool.
- `check_mk`:
- Added scripts for monitoring queues and statistics.
### Known issues
- Multi-threaded bots require multiple SIGTERMs (#1403)
- Stats can't be saved with AMQP if redis is password-protected (#1402)
- Update taxonomies to current RSIT and vice-versa (#1380)
- stomp collector bot constantly uses 100% of CPU (#1364)
- tests: capture logging with context manager (#1342)
- Consistent message counter log messages for all kind of bots (#1278)
- pymongo 3.0 deprecates used insert method (#1063)
- pymongo >= 3.5: authentication changes (#1062)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg