- IntelMQ-users

IntelMQ 3.2 announcement
by L. Aaron Kaplan 20 Jul '23

20 Jul '23

IntelMQ 3.2 release announcement ================================== [TLP:WHITE, please pass this on, if appropriate] Dear everyone, We are excited to announce that today we can release 3.2.0 of the open source IntelMQ (https://github.com/certtools/intelmq) incident handling automation framework. Special thanks go to Kamil (CERT.at), Sebix, Intevation, Filip Pokorny (Gethvi) and Jason from Shadowserver (you all know who you are). Also I would like to thank the 93 contributors [1] who have been helping in many ways over the last year(s) with IntelMQ. Core members of IntelMQ are committed to continuing to develop, support and enhance IntelMQ in the long run. Especially since IntelMQ is a quite well adopted extract transform load (ETL) tool for fetching incident report feeds, processing them and sending them to the right place. It is widely used in Europe and in the whole world, in CERTs, SOCs and MSSPs. And best of it, it is open source and freely available. Again, we can not thank all the contributors enough. So, what's in release 3.2? =========================== Highlights: * The main change is that IntelMQ's code can now be used as a python module / library. That means: you can easily embed the hard work that IntelMQ parser writers did into your application. It's as simple as instantiating an IntelMQ bot in python and giving it data. Thanks go to Sebix. See also https://github.com/certtools/ieps/tree/main/007 * the IntelMQ API moved from hug to fastapi [4]. The API interface stayed the same. Thanks a lot to Kamil. * We have an updated intelmq developer version docker image. Apart from that, a few other highlights: * Shadowserver parser improvements. * New shadowserver data feeds are supported * Changes time_format parameter to use new TimeFormat class * new CIF3 output bot * intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar) As always, you can find all the nitty gritty details in the CHANGELOG [2] file and in the NEWS file [3] What's next? ============== * BSI funded a complete re-write of webinput-csv (https://github.com/Intevation/intelmq-webinput-csv), watch for an announcement soon. * Improved and re-done documentation via mkdocs (Thanks to Filip) * The upcoming next version 3.3 will have an update for the dynamic config loading of the shadowserver parser [4]. This will always give you the most current shadowserver feeds and parsers without having to wait for an official intelmq release. If you depend a lot on shadowserver feeds, it's ok to wait for the next release, which will be following very shortly. (And a big hooray to shadowserver for their help, we know that you need this feature quickly) We plan to release 3.3 in approx 1 week. Should I update? ================== Yes! **Note well**: we still have some issues with proper Debian packages for fastapi (a dependency of the new IntelMQ API server). This means re-packaging fastapi with the help of Debian developers or using the old hug API for now (no difference from a user perspective though). The deb-package for the new fastapi-based intelmq-api is currently only available on Debian 11. We are also sorting out an issue with the package build of intelmq on Debian 12: https://github.com/certtools/intelmq/issues/2384#issuecomment-1637799252 and issues on Ubuntu related to fastapi-Versions. Packages of the previous compatible hug-based intelmq-api remain the package repositories of affected distributions. Once these issues are sorted out, the new packages will become available for installation and upgrade in the respective repositories. For installations via pip or development setups, nothing needs to be kept in mind. How to update? =============== * Upgrading: https://intelmq.readthedocs.io/en/3.2.0/user/upgrade.html * Fresh installations: https://intelmq.readthedocs.io/en/3.2.0/user/installation.html (Note that the docker installation might lag behind a bit) Need help? =========== You can contact us on the mailing lists: * intelmq-dev for developers [5] * intelmq-users for users [6] See also: https://intelmq.readthedocs.io/en/3.2.0/user/support.html Thank you and thanks to all the contributors! Aaron Kaplan, Sebix, Kami, Filip (for the whole IntelMQ team) [1] https://github.com/certtools/intelmq/graphs/contributors [2] https://github.com/certtools/intelmq/blob/3.2.0/CHANGELOG.md [3] https://github.com/certtools/intelmq/blob/3.2.0/NEWS.md [4] https://github.com/certtools/intelmq/pull/2372 [5] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev [6] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users

1 0

Re: [IntelMQ-users] MailURLCollector bot not working
by Sebix 01 Jun '23

01 Jun '23

On 6/1/23 01:28, Renz Delmor Zabala wrote: > Dear sir, > > Thank you for the reply, it's Gmail. Although Gmail's IMAP is a bit specific, imbox, the library IntelMQ uses for IMAP access, has no related known bugs for Gmail (https://github.com/martinrusev/imbox/issues?q=is%3Aissue+gmail+is%3Aopen). So it will be hard to guess the reason or a solution for the issues without debugging it. You can open an issue upstream at imbox as well. Sorry that there is no direct and easy solution, but the problem is also outside of IntelMQ, but in imbox or Gmail. > May I add, about me installing in the host, are you referring to the > part where I cloned it in the root folder? No, I referred to "1. With pip from PyPI.". best regards Sebastian Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnoloy.org/ ZVR 1510673578 On 6/1/23 01:28, Renz Delmor Zabala wrote: > Dear sir, > > Thank you for the reply, it's Gmail. > > On Wed, May 31, 2023 at 7:45 PM Sebix <sebix(a)sebix.at> wrote: > > Dear Renz, > > > On 5/31/23 09:01, Renz Delmor Zabala via IntelMQ-users wrote: >> With pip from PyPI. >> >> 1. Installed Docker-Compose using Docker’s repository. >> 2. Installed Docker Engine using the apt repository. >> 3. Cloned intelmq-docker to the root folder. >> 4. Performed docker compose pull and up. >> > > If you use intelmq-docker, you don't need to install IntelMQ with > PyPI on the host. > >> imaplib.IMAP4.error: UID command error: BAD [b'Could not parse >> command'] > > Which IMAP server/provider are you using? > > best regards > Sebastian > > Institute for Common Good Technology > gemeinnütziger Kulturverein - nonprofit cultural society > https://sebix.at/ > ZVR 1510673578 >

1 0

MailURLCollector bot not working
by Renz Delmor Zabala 31 May '23

31 May '23

Hello, May I ask for your expertise on how to make this bot working, any prerequisite softwares that I'm not aware of that might've missed in your installation instructions? Please help, thanks. *Linux Ubuntu 20.04.6 LTS (Focal Fossa)* *Installation:* 1. With pip from PyPI. 2. Installed Docker-Compose using Docker’s repository. 3. Installed Docker Engine using the apt repository. 4. Cloned intelmq-docker to the root folder. 5. Performed docker compose pull and up. 6. Now able to access in 127.0.0.1:1337 7. Using MailURLCollector => Genericcsvparser => DeduplicatorExpert => File-output 2023-05-31 05:31:43,766 - MailURL-Collector - INFO - MailURLCollectorBot initialized with id MailURL-Collector and intelmq 3.1.0 and python 3.9.2 (default, Feb 28 2021, 17:03:44) as process 1792. 2023-05-31 05:31:43,767 - MailURL-Collector - INFO - Bot is starting. 2023-05-31 05:31:44,095 - MailURL-Collector - INFO - Bot initialization completed. 2023-05-31 05:31:47,922 - MailURL-Collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py", line 319, in start self.process() File "/usr/local/lib/python3.9/dist-packages/intelmq/bots/collectors/mail/_lib.py", line 67, in process for uid, message in emails: File "/usr/local/lib/python3.9/dist-packages/imbox/messages.py", line 55, in _fetch_email_list yield uid, self._fetch_email(uid) File "/usr/local/lib/python3.9/dist-packages/imbox/messages.py", line 42, in _fetch_email return fetch_email_by_uid(uid=uid, File "/usr/local/lib/python3.9/dist-packages/imbox/parser.py", line 150, in fetch_email_by_uid message, data = connection.uid('fetch', uid, '(BODY.PEEK[] FLAGS)') File "/usr/lib/python3.9/imaplib.py", line 890, in uid typ, dat = self._simple_command(name, command, *args) File "/usr/lib/python3.9/imaplib.py", line 1230, in _simple_command return self._command_complete(name, self._command(name, *args)) File "/usr/lib/python3.9/imaplib.py", line 1055, in _command_complete raise self.error('%s command error: %s %s' % (name, typ, data)) imaplib.IMAP4.error: UID command error: BAD [b'Could not parse command'] 2023-05-31 05:31:47,994 - MailURL-Collector - INFO - Bot will continue in 15 seconds. 2023-05-31 05:31:56,408 - MailURL-Collector - INFO - Received SIGTERM. 2023-05-31 05:31:56,409 - MailURL-Collector - INFO - Bot stopped.

2 1

Question: Changing default configuration path for non-deb installations
by Kamil Mankowski 06 Apr '23

06 Apr '23

Hi all, I wanted to ask regarding an idea from the issue https://github.com/certtools/intelmq/issues/2349 At the moment, the Debian package has the default configuration path in /etc/intelmq/ and keeps logs in /var/log/intelmq/, however the pure Python package prefers /opt/intelmq/etc/ and /opt/intelmq/var/log/ This difference is introduced during building DEB packages and described here: https://intelmq.readthedocs.io/en/develop/user/configuration-management.htm… I want to suggest the unification, and keeping the first version (Linux standard paths, /etc/intelmq and /var/log/intelmq/) as the default in any case. The second schema would be still available as well as custom path, after setting the appropriate environment variable. This change could affect development environments and installations made with Python packages. For Docker and Debian packages, there would be no changes. Do you feel it makes sense, or the impact (or other reason) would be too big? -- Best regards, Kamil Mankowski CERT.at GmbH www.cert.at

1 0

IntelMQ 3.1 release
by L. Aaron Kaplan 10 Feb '23

10 Feb '23

[TLP:WHITE, please pass this on, if appropriate] Dear everyone, it took a bit of time but I am excited to announce that today we finally pushed out release 3.1 of the open source IntelMQ (https://github.com/certtools/intelmq) incident handling automation framework. Special thanks go to Kamil, Sebix, Filip Pokorny (Gethvi) and Jason from Shadowserver (you all know who you are). Also I would like to thank the 91 (!) contributors [1] who have been helping in many ways over the last year(s) with IntelMQ. This truly is became a community effort and I am very happy that we can push this project further and maintain IntelMQ in the long run as a community project. IntelMQ is by now quite well respected as a "glue" / extract transform load (ETL) tool for fetching incident report feeds, processing them and sending them to the right place. It is widely used in Europe, in CERTs, SOCs and MSSPs. And best of it, it is open source and freely available. Again, we can not thank all the contributors enough. So, what's in release 3.1? =========================== (TL;DR: - find all things here: https://github.com/certtools/intelmq/blob/develop/NEWS.md) Highlights: * Excellent support for new Shadowserver feeds (thanks to Jason and Shadowserver). This is for sure the biggest part here. Shadowserver explicitly mentions IntelMQ for processing their feeds and will maintain the IntelMQ parsers for their feeds. * Big improvements for the dataplane.org parsers * Big shodan parser improvements Apart from that, a few other highlights: * More CI/CD tests * Tons of code clean-ups * New bots such as the url expert * Enhancements of existing bots: * SQL Output support MSSQL now, * GitHub collector, * Alienvault parser improvements, * Shodan parser improvements, * maxmind geoIP expert improvements, etc etc. * Tons of behind the scenes bugfixes! * Core changes (which you won't notice much) in the core part (pipeline class) etc etc.. We also dropped some support for old, outdated versions (python 3.6 or older Debian and old RedHat systems). This slims down the code base a bit again. As always, you can find all the nitty gritty details in the CHANGELOG [2] file and in the NEWS file [3] What's next? ============== * API: We want to move the IntelMQ-API [4] to fastapi (thanks to Kamil). Due to minor issues with packaging, this is not happening today, but this is planned for next week. So, stay tuned. IntelMQ-api and intelmq will be both numbered with the same version 3.1. * More user-feedback in the next few weeks. * Update the docker images for intelmq 3.1 * Upcoming task-specific tutorials on intelmq.org (for example how to send data to Splunk, how to automate all shadowserver feeds, etc.). Courtesy of David Ruefenacht. Should I update? ================== Yes! (but first on a DEV instance!) If you run a production instance, please update on a dev instance first and cross check. Because every IntelMQ setup is different. We try to cover a lot in our testing[5] , but we can't give a guarantee that everything is completely covered, especially if there are local changes. So, yes, please update, but tell us [6] in case something breaks in your DEV environment. (But yes! you really should update - since you might really want to have all those Shadowserver and dataplane.org updates) How to update? =============== * Upgrading: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html * Fresh installations: https://intelmq.readthedocs.io/en/maintenance/user/installation.html# (Note that the docker installation might lag behind a bit) Thank you and thanks to all the contributors! Aaron Kaplan. (for the whole IntelMQ team) [1] https://github.com/certtools/intelmq/graphs/contributors [2] https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md [3] https://github.com/certtools/intelmq/blob/develop/NEWS.md [4] https://github.com/certtools/intelmq-api [5] https://intelmq.readthedocs.io/en/maintenance/dev/guide.html#testing [6] https://github.com/certtools/intelmq/issues

1 0

Certificate issues with 3.0.2
by Henrik Jensen 22 Nov '22

22 Nov '22

Hi, Are others than me experience problems with download of intelmq 3.0.2. We are informed that cerificate for intelmq cert expired rsa2048 2020-07-23 [SC] [expired: 2022-10-05]. Please inform me if so, thx. Bedste hilsner/regards Henrik Jensen - Sikkerhedskonsulent DeIC - DKCERT Tlf: +45 9351 0003 henrik.jensen(a)deic.dk

2 1

Support for EOL Python 3.6
by Kamil Mankowski 17 Nov '22

17 Nov '22

Hi all, the Python 3.6 is already end of life for a few months. Since some dependencies of IntelMQ have already started considering dropping support for the Python 3.6, it would be nice if we could go forward and stop officially supporting Python 3.6. I'd like to ask if there are still some users who use and want to support this version of Python. Dropping it would be most probably part of the 3.1 release. -- Best regards, Kamil Mankowski CERT.at GmbH www.cert.at

2 1

Modify expert get the value of data
by Guillaume GRANJON DE LEPINEY 19 Oct '22

19 Oct '22

Hello, This may be a silly question, but I can't find the answer. Is it possible to get the decoded value (not base 64) of my data in a configuration file of the bot intelmq.bots.experts.modify.expert? I would like to do something like that with the decoded value: [cid:image001.png@01D8E3A5.79043EE0] Regards, Guillaume GRANJON de LÉPINEY | ggranjon(a)excellium-services.be<mailto:ggranjon@excellium-services.be> | PGP Key ID: 0xE2FD5ED1<https://pgp.circl.lu/pks/lookup?search=0xE2FD5ED1&fingerprint=on&op=index> CERT-XLM | cert(a)excellium-services.com<mailto:cert@excellium-services.com> | PGP Key ID: 0xD74E5AC0<http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x67B311E5D7…> Excellium Services Belgium N.V. | Orion Bldg, Belgicastraat 13, B-1930 Zaventem, Belgium Mobile: +32 4 71 98 57 65 Emergency: +352 262 039 64 708 | emergency(a)excellium-services.com<mailto:emergency@excellium-services.com> | PGP Key ID: 0x42662EFE<https://excellium-services.com/assets/EMERGENCY_PKEY.asc> https://excellium-services.com/en/CERT-XLM/ https://www.trusted-introducer.org/directory/teams/cert-xlm.html https://www.first.org/members/teams/cert-xlm This email is confidential and may contain legally privileged information. If you are not the intended recipient, you should not copy, distribute, disclose or use the information it contains, please e-mail the sender immediately and delete this message from your system. Note: e-mails are susceptible to corruption, interception and unauthorised amendment; we do not accept liability for any such changes, or for their consequences. You should be aware that we may monitor your e-mails and their content. Excellium Services SA.

3 4

Issue with Intelmq SMTPoutput Bots
by Kossi DOH 18 Oct '22

18 Oct '22

Hi Community, Please I need helps solving this issue. The SMTP output bots for some time now is no more functioning. I think the issue begins after some upgrade of the server or the Intelmq. I can login to the mail server using web interface but when I run the command I received the below message. Can someone help? sudo -u www-data sudo -u intelmq intelmqctl run SMTP-Output-test process --show-sent --dry --msg '' Starting SMTP-Output-test... SMTP-Output-test: SMTPOutputBot initialized with id SMTP-Output-test and intelmq 3.0.2 and python 3.8.10 (default, Jun 22 2022, 20:18:18) as process 3335470. SMTP-Output-test: Bot is starting. SMTP-Output-test: Bot initialization completed. Traceback (most recent call last): File "/usr/bin/intelmqctl", line 11, in <module> load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main return x.run() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1048, in run retval, results = args.func(**args_dict) File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1058, in bot_run retval, results = self.bot_process_manager.bot_run(**kwargs) File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 186, in bot_run output = bd.run() File "/usr/lib/python3/dist-packages/intelmq/lib/bot_debugger.py", line 80, in run self._process(self.dryrun, self.msg, self.show) File "/usr/lib/python3/dist-packages/intelmq/lib/bot_debugger.py", line 164, in _process self.instance.process() File "/usr/lib/python3/dist-packages/intelmq/bots/outputs/smtp/output.py", line 68, in process smtp.login(user=self.smtp_username, password=self.smtp_password) File "/usr/lib/python3.8/smtplib.py", line 709, in login raise SMTPNotSupportedError( smtplib.SMTPNotSupportedError: SMTP AUTH extension not supported by server.

5 8

Re: [IntelMQ-users] Problems with intelMQ-Manager
by Henrik Jensen 09 Aug '22

09 Aug '22

Can anybody help me with the issue below? Bedste hilsner/regards Henrik Jensen DeIC - DKCERT Tlf: +45 9351 0003 henrik.jensen(a)deic.dk ________________________________ Fra: Henrik Jensen Sendt: 3. august 2022 15:51 Til: intelmq-dev-bounces(a)lists.cert.at Emne: Problems with intelMQ-Manager Hi, Can anybody help me out with an Little issue i have with intelMQ-manager. When i load the default page (index.html) i get nothing at the startup page (se below). Besides the page ask me to login, and i use the UID and PW i generated at time of installation, but i don't now if it's the problem. [cid:6bc258a9-a687-4daf-8808-5d13c2c26b48] And i get the following error massage: [cid:27b9a7f5-ef03-4042-a4fb-e7ead3f7456c] My installation is intelmq 3.0.2 and intelmq-manager 3.0.1 on an Ubuntu 20.04 (64 bit edition) Could anybody please help me out with this problem? Bedste hilsner/regards Henrik Jensen DeIC - DKCERT Tlf: +45 9351 0003 henrik.jensen(a)deic.dk

4 6

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users