- IntelMQ-users

IntelMQ 3.1 release
by L. Aaron Kaplan 10 Feb '23

10 Feb '23

[TLP:WHITE, please pass this on, if appropriate] Dear everyone, it took a bit of time but I am excited to announce that today we finally pushed out release 3.1 of the open source IntelMQ (https://github.com/certtools/intelmq) incident handling automation framework. Special thanks go to Kamil, Sebix, Filip Pokorny (Gethvi) and Jason from Shadowserver (you all know who you are). Also I would like to thank the 91 (!) contributors [1] who have been helping in many ways over the last year(s) with IntelMQ. This truly is became a community effort and I am very happy that we can push this project further and maintain IntelMQ in the long run as a community project. IntelMQ is by now quite well respected as a "glue" / extract transform load (ETL) tool for fetching incident report feeds, processing them and sending them to the right place. It is widely used in Europe, in CERTs, SOCs and MSSPs. And best of it, it is open source and freely available. Again, we can not thank all the contributors enough. So, what's in release 3.1? =========================== (TL;DR: - find all things here: https://github.com/certtools/intelmq/blob/develop/NEWS.md) Highlights: * Excellent support for new Shadowserver feeds (thanks to Jason and Shadowserver). This is for sure the biggest part here. Shadowserver explicitly mentions IntelMQ for processing their feeds and will maintain the IntelMQ parsers for their feeds. * Big improvements for the dataplane.org parsers * Big shodan parser improvements Apart from that, a few other highlights: * More CI/CD tests * Tons of code clean-ups * New bots such as the url expert * Enhancements of existing bots: * SQL Output support MSSQL now, * GitHub collector, * Alienvault parser improvements, * Shodan parser improvements, * maxmind geoIP expert improvements, etc etc. * Tons of behind the scenes bugfixes! * Core changes (which you won't notice much) in the core part (pipeline class) etc etc.. We also dropped some support for old, outdated versions (python 3.6 or older Debian and old RedHat systems). This slims down the code base a bit again. As always, you can find all the nitty gritty details in the CHANGELOG [2] file and in the NEWS file [3] What's next? ============== * API: We want to move the IntelMQ-API [4] to fastapi (thanks to Kamil). Due to minor issues with packaging, this is not happening today, but this is planned for next week. So, stay tuned. IntelMQ-api and intelmq will be both numbered with the same version 3.1. * More user-feedback in the next few weeks. * Update the docker images for intelmq 3.1 * Upcoming task-specific tutorials on intelmq.org (for example how to send data to Splunk, how to automate all shadowserver feeds, etc.). Courtesy of David Ruefenacht. Should I update? ================== Yes! (but first on a DEV instance!) If you run a production instance, please update on a dev instance first and cross check. Because every IntelMQ setup is different. We try to cover a lot in our testing[5] , but we can't give a guarantee that everything is completely covered, especially if there are local changes. So, yes, please update, but tell us [6] in case something breaks in your DEV environment. (But yes! you really should update - since you might really want to have all those Shadowserver and dataplane.org updates) How to update? =============== * Upgrading: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html * Fresh installations: https://intelmq.readthedocs.io/en/maintenance/user/installation.html# (Note that the docker installation might lag behind a bit) Thank you and thanks to all the contributors! Aaron Kaplan. (for the whole IntelMQ team) [1] https://github.com/certtools/intelmq/graphs/contributors [2] https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md [3] https://github.com/certtools/intelmq/blob/develop/NEWS.md [4] https://github.com/certtools/intelmq-api [5] https://intelmq.readthedocs.io/en/maintenance/dev/guide.html#testing [6] https://github.com/certtools/intelmq/issues

1 0

Certificate issues with 3.0.2
by Henrik Jensen 22 Nov '22

22 Nov '22

Hi, Are others than me experience problems with download of intelmq 3.0.2. We are informed that cerificate for intelmq cert expired rsa2048 2020-07-23 [SC] [expired: 2022-10-05]. Please inform me if so, thx. Bedste hilsner/regards Henrik Jensen - Sikkerhedskonsulent DeIC - DKCERT Tlf: +45 9351 0003 henrik.jensen(a)deic.dk

2 1

Support for EOL Python 3.6
by Kamil Mankowski 17 Nov '22

17 Nov '22

Hi all, the Python 3.6 is already end of life for a few months. Since some dependencies of IntelMQ have already started considering dropping support for the Python 3.6, it would be nice if we could go forward and stop officially supporting Python 3.6. I'd like to ask if there are still some users who use and want to support this version of Python. Dropping it would be most probably part of the 3.1 release. -- Best regards, Kamil Mankowski CERT.at GmbH www.cert.at

2 1

Modify expert get the value of data
by Guillaume GRANJON DE LEPINEY 19 Oct '22

19 Oct '22

Hello, This may be a silly question, but I can't find the answer. Is it possible to get the decoded value (not base 64) of my data in a configuration file of the bot intelmq.bots.experts.modify.expert? I would like to do something like that with the decoded value: [cid:image001.png@01D8E3A5.79043EE0] Regards, Guillaume GRANJON de LÉPINEY | ggranjon(a)excellium-services.be<mailto:ggranjon@excellium-services.be> | PGP Key ID: 0xE2FD5ED1<https://pgp.circl.lu/pks/lookup?search=0xE2FD5ED1&fingerprint=on&op=index> CERT-XLM | cert(a)excellium-services.com<mailto:cert@excellium-services.com> | PGP Key ID: 0xD74E5AC0<http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x67B311E5D7…> Excellium Services Belgium N.V. | Orion Bldg, Belgicastraat 13, B-1930 Zaventem, Belgium Mobile: +32 4 71 98 57 65 Emergency: +352 262 039 64 708 | emergency(a)excellium-services.com<mailto:emergency@excellium-services.com> | PGP Key ID: 0x42662EFE<https://excellium-services.com/assets/EMERGENCY_PKEY.asc> https://excellium-services.com/en/CERT-XLM/ https://www.trusted-introducer.org/directory/teams/cert-xlm.html https://www.first.org/members/teams/cert-xlm This email is confidential and may contain legally privileged information. If you are not the intended recipient, you should not copy, distribute, disclose or use the information it contains, please e-mail the sender immediately and delete this message from your system. Note: e-mails are susceptible to corruption, interception and unauthorised amendment; we do not accept liability for any such changes, or for their consequences. You should be aware that we may monitor your e-mails and their content. Excellium Services SA.

3 4

Issue with Intelmq SMTPoutput Bots
by Kossi DOH 18 Oct '22

18 Oct '22

Hi Community, Please I need helps solving this issue. The SMTP output bots for some time now is no more functioning. I think the issue begins after some upgrade of the server or the Intelmq. I can login to the mail server using web interface but when I run the command I received the below message. Can someone help? sudo -u www-data sudo -u intelmq intelmqctl run SMTP-Output-test process --show-sent --dry --msg '' Starting SMTP-Output-test... SMTP-Output-test: SMTPOutputBot initialized with id SMTP-Output-test and intelmq 3.0.2 and python 3.8.10 (default, Jun 22 2022, 20:18:18) as process 3335470. SMTP-Output-test: Bot is starting. SMTP-Output-test: Bot initialization completed. Traceback (most recent call last): File "/usr/bin/intelmqctl", line 11, in <module> load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main return x.run() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1048, in run retval, results = args.func(**args_dict) File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1058, in bot_run retval, results = self.bot_process_manager.bot_run(**kwargs) File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 186, in bot_run output = bd.run() File "/usr/lib/python3/dist-packages/intelmq/lib/bot_debugger.py", line 80, in run self._process(self.dryrun, self.msg, self.show) File "/usr/lib/python3/dist-packages/intelmq/lib/bot_debugger.py", line 164, in _process self.instance.process() File "/usr/lib/python3/dist-packages/intelmq/bots/outputs/smtp/output.py", line 68, in process smtp.login(user=self.smtp_username, password=self.smtp_password) File "/usr/lib/python3.8/smtplib.py", line 709, in login raise SMTPNotSupportedError( smtplib.SMTPNotSupportedError: SMTP AUTH extension not supported by server.

5 8

Re: [IntelMQ-users] Problems with intelMQ-Manager
by Henrik Jensen 09 Aug '22

09 Aug '22

Can anybody help me with the issue below? Bedste hilsner/regards Henrik Jensen DeIC - DKCERT Tlf: +45 9351 0003 henrik.jensen(a)deic.dk ________________________________ Fra: Henrik Jensen Sendt: 3. august 2022 15:51 Til: intelmq-dev-bounces(a)lists.cert.at Emne: Problems with intelMQ-Manager Hi, Can anybody help me out with an Little issue i have with intelMQ-manager. When i load the default page (index.html) i get nothing at the startup page (se below). Besides the page ask me to login, and i use the UID and PW i generated at time of installation, but i don't now if it's the problem. [cid:6bc258a9-a687-4daf-8808-5d13c2c26b48] And i get the following error massage: [cid:27b9a7f5-ef03-4042-a4fb-e7ead3f7456c] My installation is intelmq 3.0.2 and intelmq-manager 3.0.1 on an Ubuntu 20.04 (64 bit edition) Could anybody please help me out with this problem? Bedste hilsner/regards Henrik Jensen DeIC - DKCERT Tlf: +45 9351 0003 henrik.jensen(a)deic.dk

4 6

IntelMQ & API & Manager 3.1.0 Release candidate: Please test!
by Sebix 08 Aug '22

08 Aug '22

Dear community, about one year after the 3.0 release, 3.1 is just around the corner. Containing more than 200 commits in IntelMQ, this versions features numerous fixes and improvements from various contributors. The IntelMQ API did not change, the Manager did also receive two small bug fixes. Thanks to elsif2 the Shadowserver parser received massive updates (see Aaron's mail last week) and it needs testing. Therefore, we decided to do create a release candidate before the final "stable" 3.1.0 version. If you can spend some time on testing the new release, please do so and give feedback. Bear in mind that IntelMQ is a community project and has no sponsors, hence please consider contributing. The details you need to know before upgrading: https://github.com/certtools/intelmq/blob/3.1.0-rc1/NEWS.md Full list of changes: https://github.com/certtools/intelmq/blob/3.1.0-rc1/CHANGELOG.md The RC1 version is not available in the stable deb/rpm repository, only in the special "unstable" repository: https://intelmq.readthedocs.io/en/latest/dev/guide.html#testing-pre-releases https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… You can add the unstable repo alongside the stable repository. On PyPI the RC1 releases are marked as unstable and as such will only be installed if pip is called with intelmq==3.1.0rc1.post1 https://pypi.org/project/intelmq/3.1.0rc1.post1/ https://pypi.org/project/intelmq-api/3.1.0/ https://pypi.org/project/intelmq-manager/3.1.0rc1/ If you installed IntelMQ from git, just update the "develop" branch. The git tag is "3.1.0-rc1" best regards Sebastian -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://sebix.at/ ZVR 1510673578

1 0

heads-up: major code change in intelmq: shadowserver bots code base changed
by L. Aaron Kaplan 04 Aug '22

04 Aug '22

Dear intelmq users and -developers, yesterday I merged a major change to intelmq in the the shadowserver code base[1]. The big news is that from now on, Jason from shadowserver will maintain the intelmq shadowserver code base in intelmq (i.e., /intelmq/bots/{collectors,parsers}/shadowserver/*) directly, which is IMHO really good news for us. Why is this good news? First of all, this allows shadowserver to quickly add code to IntelMQ whenever they release a new scan-report. And they currently do roughly once per week. Sometimes more often! Secondly, the new shadowserver code uses the Shadowserver API [3] and JSON schema, which makes things easier. Finally, intelmq will be much more up to date and complete with respect to shadowserver's feeds. Please pay particular attention to the changes as outlined in the NEWS.md file here [2] Currently, this change is only in the "develop" branch [4]. What do we need from you? ========================= Testing feedback! If you have some time to try out the new shadowserver feeds, we would appreciate you trying out the "develop" branch with your shadowserver API key and tell us / this list / github's issue tracker [5] what worked and what did not work. After all, this is a large code change, and as always, there might be things which break unexpectedly. I would like to have this code well tested before we put it into the upcoming release 3.1. A *BIG* thank you goes out to shadowserver for offering the man-power, the help and all of the time they already put into improving the code base. And in my personal experience, shadowserver feeds are amongst the very best, highest quality data sources on vulnerable devices and IoCs in your network that you could get. So, in case you don't process their feeds yet, sign up here: https://www.shadowserver.org/what-we-do/network-reporting/get-reports/ (and it's even for free [6]). Thank you for your attention and best regards, Aaron Kaplan. [1] https://github.com/certtools/intelmq/pull/2227 [2] https://github.com/certtools/intelmq/pull/2227/files#diff-51920e95310ebfbc1… [3] https://www.shadowserver.org/what-we-do/network-reporting/api-documentation/ [4] https://github.com/certtools/intelmq/branches [5] https://github.com/certtools/intelmq/issues [6] though the urgently need sponsors for running their massive operations: https://www.shadowserver.org/sponsor/

1 0

Question about bots
by Thomas Rutström 10 Jun '22

10 Jun '22

Hello everybody, I hope this is the right mailing list for question like mine, if not please let me known so can find the right one instead. This is my first post and I have a little question about where to install bots. I am trying to install CERT-bund-Contact-Database bots but I run into : ModuleNotFoundError("No module named 'intelmq_certbund_contact'"). Its in the path /usr/local/bin and if I import it fr0m python it works but intelmq fails to load it every the time. Nothing is shown in the logs. /thomas

3 2

update on IEP04
by L. Aaron Kaplan 03 Jun '22

03 Jun '22

Dear IntelMQ community, in the last call on May 25th 2022, we converged on IEP04 (https://github.com/certtools/ieps/). Summary is that we will stay as close as possible to the AIL-exchange format which is fulfilling all our requirements for IntelMQ-to-IntelMQ communication. Also the UUID discussion was resolved. As part of the JTAN-hackathon at CIRCL, the AIL-exchange format was formalised as a JSON-Schema (https://github.com/certtools/ieps/blob/main/004/schema/schema.json) which will also allow us in IntelMQ to use it and to even verify data coming in to an IntelMQ instance. So, now that it is formalised, we'll see how to write collectors and output processors/bots for IntelMQ to IntelMQ communications. Thanks for the patience for IEP04 and I am happy that we have a solution now. Best, Aaron Kaplan.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users