[IntelMQ-dev] IntelMQ 3.1 release

[L. Aaron Kaplan]

[TLP:WHITE, please pass this on, if appropriate]



Dear everyone,


 it took a bit of time but I am excited to announce that today we finally pushed out release 3.1 of the open source IntelMQ (https://github.com/certtools/intelmq) incident handling automation framework.


Special thanks go to Kamil, Sebix, Filip Pokorny (Gethvi) and Jason from Shadowserver (you all know who you are).
Also I would like to thank the 91 (!) contributors [1] who have been helping in many ways over the last year(s) with IntelMQ. This truly is became a community effort and I am very happy that we can push this project further and maintain IntelMQ in the long run as a community project.

IntelMQ is by now quite well respected as a "glue" / extract transform load (ETL) tool for fetching incident report feeds, processing them and sending them to the right place. It is widely used in Europe, in CERTs, SOCs and MSSPs.
And best of it, it is open source and freely available.


Again, we can not thank all the contributors enough.


So, what's in release 3.1?
===========================

(TL;DR: - find all things here: https://github.com/certtools/intelmq/blob/develop/NEWS.md)


Highlights:

* Excellent support for new Shadowserver feeds (thanks to Jason and Shadowserver). This is for sure the biggest part here. Shadowserver explicitly mentions IntelMQ for processing their feeds and will maintain the IntelMQ parsers for their feeds. 
* Big improvements for the dataplane.org parsers
* Big shodan parser improvements


Apart from that, a few other highlights:

* More CI/CD tests
* Tons of code clean-ups
* New bots such as the url expert
* Enhancements of existing bots:
  * SQL Output support MSSQL now,
  * GitHub collector,
  * Alienvault parser improvements,
  * Shodan parser improvements,
  * maxmind geoIP expert improvements, etc etc.
* Tons of behind the scenes bugfixes!
* Core changes (which you won't notice much) in the core part (pipeline class)

etc etc..

We also dropped some support for old, outdated versions (python 3.6 or older Debian and old RedHat systems). 
This slims down the code base a bit again.

As always, you can find all the nitty gritty details in the CHANGELOG [2] file and in the NEWS file [3] 



What's next?
==============

* API: We want to move the IntelMQ-API [4] to fastapi (thanks to Kamil). Due to minor issues with packaging, this is not happening today, but this is planned for next week. So, stay tuned. IntelMQ-api and intelmq will be both numbered with the same version 3.1.
* More user-feedback in the next few weeks.
* Update the docker images for intelmq 3.1
* Upcoming task-specific tutorials on intelmq.org (for example how to send data to Splunk, how to automate all shadowserver feeds, etc.). Courtesy of David Ruefenacht.




Should I update?
==================

Yes!  (but first on a DEV instance!)

If you run a production instance, please update on a dev instance first and cross check. Because every IntelMQ setup is different. We try to cover a lot in our testing[5] , but we can't give a guarantee that everything is completely covered, especially if there are local changes.
So, yes, please update, but tell us [6] in case something breaks in your DEV environment.

(But yes! you really should update - since you might really want to have all those Shadowserver and dataplane.org updates)



How to update?
===============

* Upgrading: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html
* Fresh installations: https://intelmq.readthedocs.io/en/maintenance/user/installation.html#

(Note that the docker installation might lag behind a bit)



Thank you and thanks to all the contributors!
Aaron Kaplan.
(for the whole IntelMQ team)



[1] https://github.com/certtools/intelmq/graphs/contributors
[2] https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md
[3] https://github.com/certtools/intelmq/blob/develop/NEWS.md
[4] https://github.com/certtools/intelmq-api
[5] https://intelmq.readthedocs.io/en/maintenance/dev/guide.html#testing
[6] https://github.com/certtools/intelmq/issues