Dear community,
The time has come and IntelMQ 3.0 is final! We, as community, made loads of changes, smaller and bigger ones, and I really think that IntelMQ became more user-friendly, developer-friendly and feature-rich at the same time!
There are some major changes in this release, especially the in the field of the configuration and Internal Data format (previously: "harmonization"). For the configuration-part, the upgrade part should be automatic with `intelmqctl upgrade-config` as usual. For the Data format, carefully look at your bot configurations (filters, sieve, etc.) to update them. Adaptions in systems connected to IntelMQ, especially also databases might be necessary as well. The NEWS.md file give a summary of what has changed: https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-3...
We don't recommend to upgrade existing production instance of IntelMQ yet. We of course did testing, including the end-to-end tests, and have detailed release notes. But for critical systems, a delayed upgrade makes sense ;) Therefore the stable deb/rpm repositories don't contain the 3.0 release yet! Even though an upgrade of production systems is not yet recommended, extensive usage and testing of the new releases are very much welcome and required to get the necessary feedback for the next (maintenance) releases.
The releases are available via git, PyPI, Docker and the *unstable* deb/rpm repositories.
Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html IntelMQ API documentation: https://intelmq.readthedocs.io/en/maintenance/user/intelmq-api.html IntelMQ Manager documentation: https://intelmq.readthedocs.io/en/maintenance/user/intelmq-manager.html
NEWS/release notes of IntelMQ (Core): https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-3... Full Changelog of IntelMQ (Core): https://github.com/certtools/intelmq/blob/maintenance/CHANGELOG.md#300-2021-...
On a high level, these are the major changes compared to version 2.3.x (2.3.3 was released 2021-05-31):
In the core and Docker:
* Configuration rewrite including parameter loading and handling (IEP01), plus the required adoption of the API and Manager, by Birger Schacht (CERT.at). * Classification sync with RSIT, by Sebastian Wagner (CERT.at). * Removal of the BOTS file, by Sebastian Waldbauer (CERT.at). * Creation and maintenance of the Docker images by Sebastian Waldbauer (CERT.at). * Creation of Docker-instructions for development setups by Einar Lanfranco and Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar).
New and majorly enhanced bots:
* Added |intelmq.bots.collectors.fireeye|: A bot that collects indicators from Fireeye MAS appliances (PR#1745 by Christopher Schappelwein). * |intelmq.bots.collectors.api.collector_api|: Added UNIX socket capability (PR#1987 by Mikk Margus Möll, fixes #1986). * Added |intelmq.bots.parsers.fireeye|: A bot that parses hashes and URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein). * Added |intelmq.bots.experts.http.expert_status|: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR#1789 by Birger Schacht, fixes #1047 partly). * Added |intelmq.bots.experts.http.expert_content|: A bot that fetches an HTTP resource and checks if it contains a specific string (PR#1811 by Birger Schacht). * Added |intelmq.bots.experts.lookyloo.expert|: A bot that sends requests to a lookyloo instance & adds |screenshot_url| to the event (PR#1844 by Sebastian Waldbauer, fixes #1048). * Added |intelmq.bots.experts.rdap.expert|: A bot that checks the RDAP protocol for an abuse contact for a given domain (PR#1881 by Sebastian Waldbauer and Sebastian Wagner). * |intelmq.bots.experts.sieve.expert|: Major refactoring and lot's of new functionality New operators for working with various types (lists, sets, booleans, float, int), generic rule negation and nesting (PR#1895 by Mikk Margus Möll). * Added |intelmq.bots.experts.uwhoisd|: A bot that fetches the whois entry from a uwhois-instance (PR#1918 by Raphaël Vinot). * Added |intelmq.bots.experts.aggregate|: A bot that aggregate events based upon given fields & a timespan. (PR#1959 by Sebastian Waldbauer) * Added |intelmq.bots.experts.tuency|: A bot that queries the IntelMQ API of a tuency instance (PR#1857 by Sebastian Wagner, fixes #1856). * Added |intelmq.bots.outputs.templated_smtp| (PR#1901 by Karl-Johan Karlsson).
On the documentation front, these are the most important changes
* License and copyright information was added to all the bots (by Birger Schacht). * Added documentation on the EventDB (PR#1955 by Birger Schacht, PR#1985 by Sebastian Wagner). * Added TimescaleDB for time-series documentation (PR#1990 by Sebastian Waldbauer). * n6 interoperability documentation: Adding more graphs and illustrations (PR#1991 by Sebastian Wagner). * Added documentation on abuse-contact look-ups (PR#2021 by Sebastian Waldbauer and Sebastian Wagner).
And not to forget all the smaller changes and additions.
Thanks to (in random order)
Raphaël Vinot (circl.lu) Bernhard Reiter (intevation.de) Sebastian Wagner (CERT.AT) Filip Pokorný (CSIRT.CZ) Guillaume GRANJON de LÉPINEY (CERT XLM excellium-services.com) Mikk Margus Möll (CERT.ee) Alex Kaplan Thomas Hungenberg (CERT-BUND.DE) Einar Lanfranco (CERT-UNLP cert.unlp.edu.ar) Christopher Schappelwein (milCERT, BMLV.gv.at) Marcos Gonzalez (CSIRT-RD cncs.gob.do/csirt-rd/) Marius Karotkis (NRDCS.LT) Sebastian Waldbauer (CERT.AT) Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar) Karl-Johan Karlsson (Linköping University LIU.SE) Birger Schacht (CERT.AT)
... and all the contributors of previous releases and as well to all reporters, supporters, etc!
best regards Sebastian
Hi,
We recently published a blog post about the release, summarizing a few of the major changes: https://cert.at/en/blog/2021/8/intelmq-30-domain-based-workflow-ieps
Sebastian
On 7/5/21 5:59 PM, Sebastian Wagner wrote:
Dear community,
The time has come and IntelMQ 3.0 is final! We, as community, made loads of changes, smaller and bigger ones, and I really think that IntelMQ became more user-friendly, developer-friendly and feature-rich at the same time!
There are some major changes in this release, especially the in the field of the configuration and Internal Data format (previously: "harmonization"). For the configuration-part, the upgrade part should be automatic with `intelmqctl upgrade-config` as usual. For the Data format, carefully look at your bot configurations (filters, sieve, etc.) to update them. Adaptions in systems connected to IntelMQ, especially also databases might be necessary as well. The NEWS.md file give a summary of what has changed: https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-3...
We don't recommend to upgrade existing production instance of IntelMQ yet. We of course did testing, including the end-to-end tests, and have detailed release notes. But for critical systems, a delayed upgrade makes sense ;) Therefore the stable deb/rpm repositories don't contain the 3.0 release yet! Even though an upgrade of production systems is not yet recommended, extensive usage and testing of the new releases are very much welcome and required to get the necessary feedback for the next (maintenance) releases.
The releases are available via git, PyPI, Docker and the *unstable* deb/rpm repositories.
Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html IntelMQ API documentation: https://intelmq.readthedocs.io/en/maintenance/user/intelmq-api.html IntelMQ Manager documentation: https://intelmq.readthedocs.io/en/maintenance/user/intelmq-manager.html
NEWS/release notes of IntelMQ (Core): https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-3... Full Changelog of IntelMQ (Core): https://github.com/certtools/intelmq/blob/maintenance/CHANGELOG.md#300-2021-...
On a high level, these are the major changes compared to version 2.3.x (2.3.3 was released 2021-05-31):
In the core and Docker:
- Configuration rewrite including parameter loading and handling (IEP01), plus the required adoption of the API and Manager, by Birger Schacht (CERT.at).
- Classification sync with RSIT, by Sebastian Wagner (CERT.at).
- Removal of the BOTS file, by Sebastian Waldbauer (CERT.at).
- Creation and maintenance of the Docker images by Sebastian Waldbauer (CERT.at).
- Creation of Docker-instructions for development setups by Einar Lanfranco and Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar).
New and majorly enhanced bots:
- Added |intelmq.bots.collectors.fireeye|: A bot that collects indicators from Fireeye MAS appliances (PR#1745 by Christopher Schappelwein).
- |intelmq.bots.collectors.api.collector_api|: Added UNIX socket capability (PR#1987 by Mikk Margus Möll, fixes #1986).
- Added |intelmq.bots.parsers.fireeye|: A bot that parses hashes and URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein).
- Added |intelmq.bots.experts.http.expert_status|: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR#1789 by Birger Schacht, fixes #1047 partly).
- Added |intelmq.bots.experts.http.expert_content|: A bot that fetches an HTTP resource and checks if it contains a specific string (PR#1811 by Birger Schacht).
- Added |intelmq.bots.experts.lookyloo.expert|: A bot that sends requests to a lookyloo instance & adds |screenshot_url| to the event (PR#1844 by Sebastian Waldbauer, fixes #1048).
- Added |intelmq.bots.experts.rdap.expert|: A bot that checks the RDAP protocol for an abuse contact for a given domain (PR#1881 by Sebastian Waldbauer and Sebastian Wagner).
- |intelmq.bots.experts.sieve.expert|: Major refactoring and lot's of new functionality New operators for working with various types (lists, sets, booleans, float, int), generic rule negation and nesting (PR#1895 by Mikk Margus Möll).
- Added |intelmq.bots.experts.uwhoisd|: A bot that fetches the whois entry from a uwhois-instance (PR#1918 by Raphaël Vinot).
- Added |intelmq.bots.experts.aggregate|: A bot that aggregate events based upon given fields & a timespan. (PR#1959 by Sebastian Waldbauer)
- Added |intelmq.bots.experts.tuency|: A bot that queries the IntelMQ API of a tuency instance (PR#1857 by Sebastian Wagner, fixes #1856).
- Added |intelmq.bots.outputs.templated_smtp| (PR#1901 by Karl-Johan Karlsson).
On the documentation front, these are the most important changes
- License and copyright information was added to all the bots (by Birger Schacht).
- Added documentation on the EventDB (PR#1955 by Birger Schacht, PR#1985 by Sebastian Wagner).
- Added TimescaleDB for time-series documentation (PR#1990 by Sebastian Waldbauer).
- n6 interoperability documentation: Adding more graphs and illustrations (PR#1991 by Sebastian Wagner).
- Added documentation on abuse-contact look-ups (PR#2021 by Sebastian Waldbauer and Sebastian Wagner).
And not to forget all the smaller changes and additions.
Thanks to (in random order)
Raphaël Vinot (circl.lu) Bernhard Reiter (intevation.de) Sebastian Wagner (CERT.AT) Filip Pokorný (CSIRT.CZ) Guillaume GRANJON de LÉPINEY (CERT XLM excellium-services.com) Mikk Margus Möll (CERT.ee) Alex Kaplan Thomas Hungenberg (CERT-BUND.DE) Einar Lanfranco (CERT-UNLP cert.unlp.edu.ar) Christopher Schappelwein (milCERT, BMLV.gv.at) Marcos Gonzalez (CSIRT-RD cncs.gob.do/csirt-rd/) Marius Karotkis (NRDCS.LT) Sebastian Waldbauer (CERT.AT) Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar) Karl-Johan Karlsson (Linköping University LIU.SE) Birger Schacht (CERT.AT)
... and all the contributors of previous releases and as well to all reporters, supporters, etc!
best regards Sebastian
-- // Sebastian Wagner wagner@cert.at - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg