- IntelMQ-users

IntelMQ not working
by Jovana Gjurovska 14 Feb '24

14 Feb '24

Dear, I had successfully installed intelmq-docker on a linux virtual machine and it worked perfectly till 31 of January. Now I have a problem. Docker is running and it is active but I can not docker-compose down and docker-compose up -d and I do not know why. Here are the errors I am getting: docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2024-02-14 09:31:20 UTC; 21min ago TriggeredBy: ● docker.socket Docs: https://docs.docker.com Main PID: 158179 (dockerd) Tasks: 14 Memory: 32.6M CPU: 8.801s CGroup: /system.slice/docker.service └─158179 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock 3 services are failed: UNIT LOAD ACTIVE SUB DESCRIPTION ● intelmq-api.service loaded failed failed Gunicorn deamon to serve the IntelMQ API ● logrotate.service loaded failed failed Rotate log files ● nginx.service loaded failed failed A high performance web server and a reverse proxy server When I type docker compose down this error appears: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 446, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 441, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse response.begin() File "/usr/lib/python3.10/http/client.py", line 318, in begin version, status, reason = self._read_status() File "/usr/lib/python3.10/http/client.py", line 279, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/lib/python3.10/socket.py", line 705, in readinto return self._sock.recv_into(b) Can you please help me solve the problem? Best regards, Jovana Gjurovska. Jovana Gjurovska EdTech Specialist Mobile: +389 (77) 836-658 Email: mailto:stefan.jovanovski@sigmoidgroup.com Sigmoid DOO, member of Sigmoid Group No 22/1 - 302, Str. 1615, Cevahir Sky City, Tower B, Floor 33, 1000 Skopje, Macedonia https://www.sigmoidgroup.com/ --- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

2 2

Shadowserver parser: Bad mapping for malware events
by Thomas Hungenberg 24 Jan '24

24 Jan '24

Hi all, the parsers for malware events provided by different sources usually store the malware name in malware.name and classification.identifier is left blank (or set to the feed's name). When using the malware name mapping, a harmonized malware name is subsequently written to classification.identifier. So finally you have the original name in malware.name and the harmonized name in classification.identifier. Formerly (in the version we initially provided), the Shadowserver parser also stored the malware name in malware.name, see e.g. <https://github.com/certtools/intelmq/blob/c61ff2fd4232d6937f3815377b75f682a…> line 387 However, for some time the Shadowserver parser now writes the malware name ("infection") to classification.identifier and "family" to malware.name instead. This is bad for several reasons: - it is not consistent with parsers for other malware feeds - it breaks deduplicators matching on malware.name - the malware name mapping overwrites classification.identifier with the value of "family" (which often is empty) Here is a patch (for the version included with IntelMQ 3.2.1) to fix this problem and make malware events parsed by the Shadowserver parser consistent with other parsers again: =============================================== diff --git a/_config.py.orig b/_config.py index bea3d0c..431bcb9 100644 --- a/_config.py.orig +++ b/_config.py @@ -867,10 +867,9 @@ event_sinkhole = { ('source.port', 'src_port', convert_int), ], 'optional_fields': [ - ('classification.identifier', 'infection', validate_to_none), - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('source.asn', 'src_asn', invalidate_zero), ('source.geolocation.cc', 'src_geo'), @@ -899,6 +898,7 @@ event_sinkhole = { 'constant_fields': { 'classification.taxonomy': 'malicious-code', 'classification.type': 'infected-system', + 'classification.identifier': 'sinkhole-events', }, } @@ -944,10 +944,9 @@ event_sinkhole_http = { ('source.port', 'src_port', convert_int), ], 'optional_fields': [ - ('classification.identifier', 'tag'), - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('source.asn', 'src_asn', invalidate_zero), ('source.geolocation.cc', 'src_geo'), @@ -982,6 +981,7 @@ event_sinkhole_http = { 'constant_fields': { 'classification.taxonomy': 'malicious-code', 'classification.type': 'infected-system', + 'classification.identifier': 'sinkhole-http-events', 'protocol.application': 'http', }, } @@ -992,9 +992,9 @@ event_sinkhole_http_referer = { ('time.source', 'timestamp', add_UTC_to_timestamp), ], 'optional_fields': [ - ('malware.name', 'family', validate_to_none), + ('malware.name', 'infection', validate_to_none), + ('extra.', 'family', validate_to_none), ('extra.', 'tag', validate_to_none), - ('extra.', 'infection', validate_to_none), ('protocol.transport', 'protocol'), ('extra.', 'http_referer_ip', validate_ip), ('extra.', 'http_referer_port', convert_int), =============================================== Kind regards Thomas

1 0

Shadowserver parser flooding logs
by Thomas Hungenberg 24 Jan '24

24 Jan '24

Hello, I noticed that our logs for bots based on the Shadowserver parser grew VERY large. Reason for this is that with recent versions of the parser, a WARNING is logged for every _optional_ key not found in the feed data. As the optional key "sic" is no longer included in any feed, at least one WARNING is logged for each and every event processed by Shadowserver parsers. :-/ This looks like: 2024-01-16 06:41:55,551 - shadowserver-parser-xyz - WARNING - Optional key 'sic' not found in feed 'XYZ'. Possible change in data format or misconfiguration. A quick fix is to disable logging of the WARNINGs in .../intelmq/bots/parsers/shadowserver/parser.py. diff --git a/parser.py.orig b/parser.py index 70ba3b4..2ad29ac 100644 --- a/parser.py.orig +++ b/parser.py @@ -140,8 +140,8 @@ class ShadowserverParserBot(ParserBot): intelmqkey, shadowkey = item[:2] if shadowkey not in fields: if shadowkey not in row: # key does not exist in data (not even in the header) - self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data' - ' format or misconfiguration.'.format(shadowkey, self.feedname)) + # self.logger.warning('Optional key {!r} not found in feed {!r}. Possible change in data' + # ' format or misconfiguration.'.format(shadowkey, self.feedname)) continue else: # key is used twice fields.append(shadowkey) Or replace "self.logger.warning" by "self.logger.debug" if not running the bot in debug mode. Kind regards Thomas

2 3

IntelMQ & Kubernetes
by Jessica Schumacher 17 Jan '24

17 Jan '24

Hello all, we’ve been using IntelMQ for a long time at Switch and moved IntelMQ on Docker a few years ago. Now we want to take the “next” step to Kubernetes. Does anyone have experience with running IntelMQ on Kubernetes? At the moment we are just at the brainstorming phase and thinking about the design. Especially how we should split it up into different pots and redundancy. Any feedback about experiences or inputs are highly appreciated. Kind regards, Jessica — Switch Jessica Schumacher, Switch CERT Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland phone +41 44 268 15 15, direct +41 44 268 16 33 jessica.schumacher(a)switch.ch, http://www.switch.ch

2 1

Introducing proposals to IntelMQ Data Format (IEP008 and 9)
by Kamil Mankowski 27 Nov '23

27 Nov '23

Dear IntelMQ Developers & Users, I'd like to propose two changes to the IntelMQ data format to make it more suitable for current use cases. 1. IEP008 - add constituency identifier (this has already been shared, but only on the developer list): https://github.com/certtools/ieps/pull/9 2. IEP009 - add product and vulnerability information: https://github.com/certtools/ieps/pull/10 I'd like to hear your feedback on these suggestions. Feel free to comment here or on the Github pull requests. My plan is to implement them, unless rejected, by December. -- Best regards, Kamil Mankowski CERT.at GmbH www.cert.at

2 3

IntelMQ 3.2.1 release with two important bug fixes
by Sebix 28 Aug '23

28 Aug '23

Dear IntelMQ community, Together with CERT.at we just released IntelMQ version 3.2.1 containing two important bug fixes. 1. A bug in the bot's core library: > Fixes an issue which prevented bots from stopping gracefully after reloading. > As logrotate reloads all bots regularly, this bug affects most IntelMQ installations. 2. A bug in the Reverse DNS Expert bot: > Until IntelMQ version 3.2.0, the bot incorrectly cached and re-used results for /24 networks instead of single IP addresses. > If the bot retrieved the PTR for `192.0.43.7`, it was cached for `192.0.43.0/24` and used for all IP addresses in this range, for example for `192.0.43.8`. > IntelMQ version 3.2.1 fixes this issue. > The bugfix will correctly increase the cache sizes and decrease the performance, as less (incorrect) data is re-used. Upgrading: https://intelmq.readthedocs.io/en/develop/user/upgrade.html Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html The PyPI package and git tag/GitHub release are published. The new deb-/rpm-packages will hit the repositories in the next few minutes and the updated Docker image follows soon. best regards Sebastian -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://sebix.at/ ZVR 1510673578

1 0

IntelMQ 3.2 announcement
by L. Aaron Kaplan 20 Jul '23

20 Jul '23

IntelMQ 3.2 release announcement ================================== [TLP:WHITE, please pass this on, if appropriate] Dear everyone, We are excited to announce that today we can release 3.2.0 of the open source IntelMQ (https://github.com/certtools/intelmq) incident handling automation framework. Special thanks go to Kamil (CERT.at), Sebix, Intevation, Filip Pokorny (Gethvi) and Jason from Shadowserver (you all know who you are). Also I would like to thank the 93 contributors [1] who have been helping in many ways over the last year(s) with IntelMQ. Core members of IntelMQ are committed to continuing to develop, support and enhance IntelMQ in the long run. Especially since IntelMQ is a quite well adopted extract transform load (ETL) tool for fetching incident report feeds, processing them and sending them to the right place. It is widely used in Europe and in the whole world, in CERTs, SOCs and MSSPs. And best of it, it is open source and freely available. Again, we can not thank all the contributors enough. So, what's in release 3.2? =========================== Highlights: * The main change is that IntelMQ's code can now be used as a python module / library. That means: you can easily embed the hard work that IntelMQ parser writers did into your application. It's as simple as instantiating an IntelMQ bot in python and giving it data. Thanks go to Sebix. See also https://github.com/certtools/ieps/tree/main/007 * the IntelMQ API moved from hug to fastapi [4]. The API interface stayed the same. Thanks a lot to Kamil. * We have an updated intelmq developer version docker image. Apart from that, a few other highlights: * Shadowserver parser improvements. * New shadowserver data feeds are supported * Changes time_format parameter to use new TimeFormat class * new CIF3 output bot * intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar) As always, you can find all the nitty gritty details in the CHANGELOG [2] file and in the NEWS file [3] What's next? ============== * BSI funded a complete re-write of webinput-csv (https://github.com/Intevation/intelmq-webinput-csv), watch for an announcement soon. * Improved and re-done documentation via mkdocs (Thanks to Filip) * The upcoming next version 3.3 will have an update for the dynamic config loading of the shadowserver parser [4]. This will always give you the most current shadowserver feeds and parsers without having to wait for an official intelmq release. If you depend a lot on shadowserver feeds, it's ok to wait for the next release, which will be following very shortly. (And a big hooray to shadowserver for their help, we know that you need this feature quickly) We plan to release 3.3 in approx 1 week. Should I update? ================== Yes! **Note well**: we still have some issues with proper Debian packages for fastapi (a dependency of the new IntelMQ API server). This means re-packaging fastapi with the help of Debian developers or using the old hug API for now (no difference from a user perspective though). The deb-package for the new fastapi-based intelmq-api is currently only available on Debian 11. We are also sorting out an issue with the package build of intelmq on Debian 12: https://github.com/certtools/intelmq/issues/2384#issuecomment-1637799252 and issues on Ubuntu related to fastapi-Versions. Packages of the previous compatible hug-based intelmq-api remain the package repositories of affected distributions. Once these issues are sorted out, the new packages will become available for installation and upgrade in the respective repositories. For installations via pip or development setups, nothing needs to be kept in mind. How to update? =============== * Upgrading: https://intelmq.readthedocs.io/en/3.2.0/user/upgrade.html * Fresh installations: https://intelmq.readthedocs.io/en/3.2.0/user/installation.html (Note that the docker installation might lag behind a bit) Need help? =========== You can contact us on the mailing lists: * intelmq-dev for developers [5] * intelmq-users for users [6] See also: https://intelmq.readthedocs.io/en/3.2.0/user/support.html Thank you and thanks to all the contributors! Aaron Kaplan, Sebix, Kami, Filip (for the whole IntelMQ team) [1] https://github.com/certtools/intelmq/graphs/contributors [2] https://github.com/certtools/intelmq/blob/3.2.0/CHANGELOG.md [3] https://github.com/certtools/intelmq/blob/3.2.0/NEWS.md [4] https://github.com/certtools/intelmq/pull/2372 [5] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev [6] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users

1 0

Re: [IntelMQ-users] MailURLCollector bot not working
by Sebix 01 Jun '23

01 Jun '23

On 6/1/23 01:28, Renz Delmor Zabala wrote: > Dear sir, > > Thank you for the reply, it's Gmail. Although Gmail's IMAP is a bit specific, imbox, the library IntelMQ uses for IMAP access, has no related known bugs for Gmail (https://github.com/martinrusev/imbox/issues?q=is%3Aissue+gmail+is%3Aopen). So it will be hard to guess the reason or a solution for the issues without debugging it. You can open an issue upstream at imbox as well. Sorry that there is no direct and easy solution, but the problem is also outside of IntelMQ, but in imbox or Gmail. > May I add, about me installing in the host, are you referring to the > part where I cloned it in the root folder? No, I referred to "1. With pip from PyPI.". best regards Sebastian Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnoloy.org/ ZVR 1510673578 On 6/1/23 01:28, Renz Delmor Zabala wrote: > Dear sir, > > Thank you for the reply, it's Gmail. > > On Wed, May 31, 2023 at 7:45 PM Sebix <sebix(a)sebix.at> wrote: > > Dear Renz, > > > On 5/31/23 09:01, Renz Delmor Zabala via IntelMQ-users wrote: >> With pip from PyPI. >> >> 1. Installed Docker-Compose using Docker’s repository. >> 2. Installed Docker Engine using the apt repository. >> 3. Cloned intelmq-docker to the root folder. >> 4. Performed docker compose pull and up. >> > > If you use intelmq-docker, you don't need to install IntelMQ with > PyPI on the host. > >> imaplib.IMAP4.error: UID command error: BAD [b'Could not parse >> command'] > > Which IMAP server/provider are you using? > > best regards > Sebastian > > Institute for Common Good Technology > gemeinnütziger Kulturverein - nonprofit cultural society > https://sebix.at/ > ZVR 1510673578 >

1 0

MailURLCollector bot not working
by Renz Delmor Zabala 31 May '23

31 May '23

Hello, May I ask for your expertise on how to make this bot working, any prerequisite softwares that I'm not aware of that might've missed in your installation instructions? Please help, thanks. *Linux Ubuntu 20.04.6 LTS (Focal Fossa)* *Installation:* 1. With pip from PyPI. 2. Installed Docker-Compose using Docker’s repository. 3. Installed Docker Engine using the apt repository. 4. Cloned intelmq-docker to the root folder. 5. Performed docker compose pull and up. 6. Now able to access in 127.0.0.1:1337 7. Using MailURLCollector => Genericcsvparser => DeduplicatorExpert => File-output 2023-05-31 05:31:43,766 - MailURL-Collector - INFO - MailURLCollectorBot initialized with id MailURL-Collector and intelmq 3.1.0 and python 3.9.2 (default, Feb 28 2021, 17:03:44) as process 1792. 2023-05-31 05:31:43,767 - MailURL-Collector - INFO - Bot is starting. 2023-05-31 05:31:44,095 - MailURL-Collector - INFO - Bot initialization completed. 2023-05-31 05:31:47,922 - MailURL-Collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py", line 319, in start self.process() File "/usr/local/lib/python3.9/dist-packages/intelmq/bots/collectors/mail/_lib.py", line 67, in process for uid, message in emails: File "/usr/local/lib/python3.9/dist-packages/imbox/messages.py", line 55, in _fetch_email_list yield uid, self._fetch_email(uid) File "/usr/local/lib/python3.9/dist-packages/imbox/messages.py", line 42, in _fetch_email return fetch_email_by_uid(uid=uid, File "/usr/local/lib/python3.9/dist-packages/imbox/parser.py", line 150, in fetch_email_by_uid message, data = connection.uid('fetch', uid, '(BODY.PEEK[] FLAGS)') File "/usr/lib/python3.9/imaplib.py", line 890, in uid typ, dat = self._simple_command(name, command, *args) File "/usr/lib/python3.9/imaplib.py", line 1230, in _simple_command return self._command_complete(name, self._command(name, *args)) File "/usr/lib/python3.9/imaplib.py", line 1055, in _command_complete raise self.error('%s command error: %s %s' % (name, typ, data)) imaplib.IMAP4.error: UID command error: BAD [b'Could not parse command'] 2023-05-31 05:31:47,994 - MailURL-Collector - INFO - Bot will continue in 15 seconds. 2023-05-31 05:31:56,408 - MailURL-Collector - INFO - Received SIGTERM. 2023-05-31 05:31:56,409 - MailURL-Collector - INFO - Bot stopped.

2 1

Question: Changing default configuration path for non-deb installations
by Kamil Mankowski 06 Apr '23

06 Apr '23

Hi all, I wanted to ask regarding an idea from the issue https://github.com/certtools/intelmq/issues/2349 At the moment, the Debian package has the default configuration path in /etc/intelmq/ and keeps logs in /var/log/intelmq/, however the pure Python package prefers /opt/intelmq/etc/ and /opt/intelmq/var/log/ This difference is introduced during building DEB packages and described here: https://intelmq.readthedocs.io/en/develop/user/configuration-management.htm… I want to suggest the unification, and keeping the first version (Linux standard paths, /etc/intelmq and /var/log/intelmq/) as the default in any case. The second schema would be still available as well as custom path, after setting the appropriate environment variable. This change could affect development environments and installations made with Python packages. For Docker and Debian packages, there would be no changes. Do you feel it makes sense, or the impact (or other reason) would be too big? -- Best regards, Kamil Mankowski CERT.at GmbH www.cert.at

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users