- IntelMQ-users

IEP04: The choice of the UUID-format
by Sebastian Wagner 07 Sep '21

07 Sep '21

Dear allies, The discussion around the IEP04 proposal, adding meta-information to IntelMQ messages, has stalled over the last months - first because of the time-intensive IntelMQ 3.0 release preparations and then because of the vacation season. Here is the current proposal: https://github.com/certtools/ieps/tree/main/004#readme Aaron, Sebastian Waldbauer and myself worked on it over the summer and also identified two open issues to be discussed: 1. The exact format of the meta-information and how to name and structure the fields. AIL made the first move and now uses a format similar to the previously proposed Variant "A". The IEP04 document contains the current proposal which is in line with the AIL format: https://github.com/certtools/ieps/tree/main/004#user-content-variant-ail If there are no other proposals, this will most probably the way to go. 2. The format of the UUID format which we want to uniquely identify IntelMQ events. We don't necessarily need to use the UUIDv4 format which represents pure randomness, but also other options which include the time and are even /time-sortable/. Sebastian Waldbauer analysed a couple of options and summarised his results in this document: https://github.com/certtools/ieps/blob/main/004/UUID.md Please let us know your opinion on the different UUID options. cheers Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

IntelMQ & API & Manager release 3.0.1 and upgrade note
by Sebastian Wagner 02 Sep '21

02 Sep '21

Dear community, Over the past two months, IntelMQ contributors had no summer pause, but did the final finish for IntelMQ 3.0. A special thanks goes to Mikk Margus Möll (CERT.ee) who has put tremendous efforts in the IntelMQ Manager tackling structural and usability issues, mainly in the JavaScript-components! The deb/rpm repositories did not receive the 3.0.0 release at beginning of July to get more experience with the major changes before doing automatic upgrades, but now they deliver the brand-new 3.0.1 version. Please note, that the automatic upgrade procedures may still not be fully smooth. Just now, we have noticed, that the packages contain a small flaw, which harms the upgrade experience: The packages ship a default configuration (the file is now called `runtime.yaml`), but only if the file does not exist before - for new installations. But now in this special case, we renamed the configuration from `runtime.conf` to `runtime.yaml` and therefore, the new - default shipped - configuration takes precedence. I hope the following commands and hints will be of help to you. # remove the runtime configuration shipped by the package (can be called /etc/intelmq/runtime.*) and rename your original one to /etc/intelmq/runtime.yaml # the previously used runtime.conf can be used as drop-in to runtime.yaml (YAML is backwards-compatible with JSON) sudo -u intelmq intelmqctl upgrade-config -f -u v300_pipeline_file_removal sudo -u intelmq intelmqctl upgrade-config -f -u v300_defaults_file_removal sudo -u intelmq intelmqctl upgrade-config -f -u v301_deprecations The last three steps are important to merge the defaults and pipeline configuration into the new combined configuration file Please do not hesitate to ask. The deb-packages are also already available for the newly released Debian 11 Bullseye. We are not planning a bugfix release until the 3.1.0 release, so that one will be the next version to be released. Here's a short summary of what happened during the summer: - various fixes related to the IEP001 implementation (IEP001 was the change configuration format and merge of files, rewrite oft the internal parameter-handling) - removal of the malwaredomains feed and parser, because it does not exist anymore - Various fixes in the Shadowserver Parser and support for new reports: Vulnerable SMTP Server, Microsoft Sinkhole Events Report & Microsoft Sinkhole HTTP Events Report, Honeypot HTTP Scan - SMTP Output: Added Content-Disposition-header to the attachment, fixing the display in MS Outlook clients (as reported and dicussed on the Mailinglist). - Heavy refactoring of IntelMQ-Manager's JavaScript parts to fix errors and usability issues. If you are interested in developing on IntelMQ and you don't know where to start, have a look at the dev guide an the issues labeled "good first issue": https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3A… We are especially welcoming contributions to the documentation! You can read the full changelogs here: - https://github.com/certtools/intelmq/releases/tag/3.0.1 - https://github.com/certtools/intelmq-api/releases/tag/3.0.1 - https://github.com/certtools/intelmq-manager/releases/tag/3.0.1 https://cert.at/en/blog/2021/9/intelmq-301-release https://twitter.com/CERT_at/status/1433475188381806594 btw: There's new contact management portal called "tuency" for administrating abuse contacts available, which can be used in conjunction with IntelMQ. Read more about its features here: https://cert.at/en/blog/2021/9/tuency-constituency-portal-for-iocs-and-certs https://gitlab.com/intevation/tuency/tuency -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Intelmq-docker v.2.3.3 API
by Jessica Schumacher 25 Aug '21

25 Aug '21

Hi all, I recently looked into intelmq-docker [1] and really liked the concept of it. Especially since we currently have IntelMQ in a dockerized setup running, it looks really promising. Thank you for the work you put into it so far! I do have a question though or more of an issue I ran into. I am currently working with `intelmq-full-dev` to implement our own bots. I use the docker image `certat/intelmq-full:2.3.3` so I am using IntelMQ Version 2.3.3 in my setup. I get the following error messages in IntelMQ Manager: The file does indeed not exist at this url shown in the image but can be accessed via: http://127.0.0.1:1337/intelmq/v1/api/config?file=positions <http://127.0.0.1:1337/intelmq/v1/api/config?file=positions> My guess is, that an older IntelMQ API is installed where only one of those methods to access the files works. At the moment version 2.3.1 of the IntelMQ API is automatically installed when setting up the docker container. If I compare the api.py of 2.3.1 [2] with the current one [3], I can see, that entries for “runtime” or “positions” for example are missing in version 2.3.1; therefor the error. I am not entirely sure if this has something to do with this [4] issue on Github. Is this something that can be fixed on the docker image from your site? Or am I doing something wrong here? Thank you for your help. Kind regards, Jessica [1] https://github.com/certat/intelmq-docker <https://github.com/certat/intelmq-docker> [2] https://github.com/certtools/intelmq-api/blob/2.3.1/intelmq_api/api.py <https://github.com/certtools/intelmq-api/blob/2.3.1/intelmq_api/api.py> [3] https://github.com/certtools/intelmq-api/blob/develop/intelmq_api/api.py <https://github.com/certtools/intelmq-api/blob/develop/intelmq_api/api.py> [4] https://github.com/certat/intelmq-docker/issues/8 <https://github.com/certat/intelmq-docker/issues/8> -- SWITCH Jessica Schumacher, SWITCH-CERT Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland phone +41 44 268 15 15, direct +41 44 268 16 33 jessica.schumacher(a)switch.ch, http://www.switch.ch

2 2

Unable to send CSV file using Intelmq
by Kossi DOH 22 Aug '21

22 Aug '21

Dear Community Please I need help to configure the SMTP bots of Intelmq to send CSV file as describe in the description. I check the MIME and I is not the blocking point. The Information needed to be sent in CSV is either in the ATT00001 file and contrary to all the Intelmq mail I do receive mine is not .txt. Can someone please help me to solve this issue? This is an example of the file sent after the configuration: [cid:image001.png@01D77335.33F95C40] Cordialement / Best Regards, Kossi DOH Analyste Cyber Securite CYBER DEFENSE AFRICA S.A.S. Mobile: +228 70 54 93 26 https://cert.tg This information is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Unauthorised use of this information by a person or entity other than the intended recipient is prohibited by law. If you received this by mistake, please immediately contact the sender by email or phone and delete this information from any computer. Thank you.

4 9

[IntelMQ] Deduplication on an optional field
by Guillaume GRANJON DE LEPINEY 06 Aug '21

06 Aug '21

Hello everyone, I wonder if there is a simple way to use a Deduplicator bot on an optional field. Indeed, I noticed when I apply the deduplicator on an optional field that the null value must be entered in the redis because all messages (except the first one) that do not contain the field are dropped. Is there a workaround please? I could work around this problem by adding two Sieve bots at the exit of the precedent bot that would jump the Deduplicator bot if the message doesn't have the field, but I don't find that to be optimal. Thus, I am open to any proposal that could help me. Regards, Guillaume GRANJON de LÉPINEY | ggranjon(a)excellium-services.be<mailto:ggranjon@excellium-services.be> | PGP Key ID: 0xE2FD5ED1<https://pgp.circl.lu/pks/lookup?search=0xE2FD5ED1&fingerprint=on&op=index> CERT-XLM Incident Handler @ excellium-services.com<https://excellium-services.com/> CERT-XLM | cert(a)excellium-services.com<mailto:cert@excellium-services.com> | PGP Key ID: 0xD74E5AC0<http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x67B311E5D7…> Excellium Services Belgium N.V. | Orion Bldg, Belgicastraat 13, B-1930 Zaventem, Belgium Mobile: +32 4 71 98 57 65 Emergency: +352 262 039 64 708 | emergency(a)excellium-services.com<mailto:emergency@excellium-services.com> | PGP Key ID: 0x42662EFE<https://excellium-services.com/assets/EMERGENCY_PKEY.asc> This email is confidential and may contain legally privileged information. If you are not the intended recipient, you should not copy, distribute, disclose or use the information it contains, please e-mail the sender immediately and delete this message from your system. Note: e-mails are susceptible to corruption, interception and unauthorised amendment; we do not accept liability for any such changes, or for their consequences. You should be aware that we may monitor your e-mails and their content. Excellium Services SA.

2 3

IntelMQ & API & Manager Release 3.0
by Sebastian Wagner 05 Aug '21

05 Aug '21

Dear community, The time has come and IntelMQ 3.0 is final! We, as community, made loads of changes, smaller and bigger ones, and I really think that IntelMQ became more user-friendly, developer-friendly and feature-rich at the same time! There are some major changes in this release, especially the in the field of the configuration and Internal Data format (previously: "harmonization"). For the configuration-part, the upgrade part should be automatic with `intelmqctl upgrade-config` as usual. For the Data format, carefully look at your bot configurations (filters, sieve, etc.) to update them. Adaptions in systems connected to IntelMQ, especially also databases might be necessary as well. The NEWS.md file give a summary of what has changed: https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-… We don't recommend to upgrade existing production instance of IntelMQ yet. We of course did testing, including the end-to-end tests, and have detailed release notes. But for critical systems, a delayed upgrade makes sense ;) Therefore the stable deb/rpm repositories don't contain the 3.0 release yet! Even though an upgrade of production systems is not yet recommended, extensive usage and testing of the new releases are very much welcome and required to get the necessary feedback for the next (maintenance) releases. The releases are available via git, PyPI, Docker and the *unstable* deb/rpm repositories. Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html IntelMQ API documentation: https://intelmq.readthedocs.io/en/maintenance/user/intelmq-api.html IntelMQ Manager documentation: https://intelmq.readthedocs.io/en/maintenance/user/intelmq-manager.html NEWS/release notes of IntelMQ (Core): https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-… Full Changelog of IntelMQ (Core): https://github.com/certtools/intelmq/blob/maintenance/CHANGELOG.md#300-2021… On a high level, these are the major changes compared to version 2.3.x (2.3.3 was released 2021-05-31): In the core and Docker: * Configuration rewrite including parameter loading and handling (IEP01), plus the required adoption of the API and Manager, by Birger Schacht (CERT.at). * Classification sync with RSIT, by Sebastian Wagner (CERT.at). * Removal of the BOTS file, by Sebastian Waldbauer (CERT.at). * Creation and maintenance of the Docker images by Sebastian Waldbauer (CERT.at). * Creation of Docker-instructions for development setups by Einar Lanfranco and Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar). New and majorly enhanced bots: * Added |intelmq.bots.collectors.fireeye|: A bot that collects indicators from Fireeye MAS appliances (PR#1745 by Christopher Schappelwein). * |intelmq.bots.collectors.api.collector_api|: Added UNIX socket capability (PR#1987 by Mikk Margus Möll, fixes #1986). * Added |intelmq.bots.parsers.fireeye|: A bot that parses hashes and URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein). * Added |intelmq.bots.experts.http.expert_status|: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR#1789 by Birger Schacht, fixes #1047 partly). * Added |intelmq.bots.experts.http.expert_content|: A bot that fetches an HTTP resource and checks if it contains a specific string (PR#1811 by Birger Schacht). * Added |intelmq.bots.experts.lookyloo.expert|: A bot that sends requests to a lookyloo instance & adds |screenshot_url| to the event (PR#1844 by Sebastian Waldbauer, fixes #1048). * Added |intelmq.bots.experts.rdap.expert|: A bot that checks the RDAP protocol for an abuse contact for a given domain (PR#1881 by Sebastian Waldbauer and Sebastian Wagner). * |intelmq.bots.experts.sieve.expert|: Major refactoring and lot's of new functionality New operators for working with various types (lists, sets, booleans, float, int), generic rule negation and nesting (PR#1895 by Mikk Margus Möll). * Added |intelmq.bots.experts.uwhoisd|: A bot that fetches the whois entry from a uwhois-instance (PR#1918 by Raphaël Vinot). * Added |intelmq.bots.experts.aggregate|: A bot that aggregate events based upon given fields & a timespan. (PR#1959 by Sebastian Waldbauer) * Added |intelmq.bots.experts.tuency|: A bot that queries the IntelMQ API of a tuency instance (PR#1857 by Sebastian Wagner, fixes #1856). * Added |intelmq.bots.outputs.templated_smtp| (PR#1901 by Karl-Johan Karlsson). On the documentation front, these are the most important changes * License and copyright information was added to all the bots (by Birger Schacht). * Added documentation on the EventDB (PR#1955 by Birger Schacht, PR#1985 by Sebastian Wagner). * Added TimescaleDB for time-series documentation (PR#1990 by Sebastian Waldbauer). * n6 interoperability documentation: Adding more graphs and illustrations (PR#1991 by Sebastian Wagner). * Added documentation on abuse-contact look-ups (PR#2021 by Sebastian Waldbauer and Sebastian Wagner). And not to forget all the smaller changes and additions. Thanks to (in random order) Raphaël Vinot (circl.lu) Bernhard Reiter (intevation.de) Sebastian Wagner (CERT.AT) Filip Pokorný (CSIRT.CZ) Guillaume GRANJON de LÉPINEY (CERT XLM excellium-services.com) Mikk Margus Möll (CERT.ee) Alex Kaplan Thomas Hungenberg (CERT-BUND.DE) Einar Lanfranco (CERT-UNLP cert.unlp.edu.ar) Christopher Schappelwein (milCERT, BMLV.gv.at) Marcos Gonzalez (CSIRT-RD cncs.gob.do/csirt-rd/) Marius Karotkis (NRDCS.LT) Sebastian Waldbauer (CERT.AT) Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar) Karl-Johan Karlsson (Linköping University LIU.SE) Birger Schacht (CERT.AT) ... and all the contributors of previous releases and as well to all reporters, supporters, etc! best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

Re: [IntelMQ-users] [IHAP] Intelmq SMTP bots not sending CSV in mail:: Need Help
by Sebastian Wagner 07 Jun '21

07 Jun '21

Hi, On 6/7/21 10:04 AM, Ángel González Berdasco wrote: > El lun, 07-06-2021 a las 09:49 +0200, Sebastian Wagner escribió: >> >> Hi, >> >> The SMTP output bot uses these headers for the CSV attachment: >> >> Content-Type: text/csv; charset="us-ascii" >> MIME-Version: 1.0 >> Content-Transfer-Encoding: 7bit >> >> Maybe Outlook needs something else? >> >> Sebastian >> > > I think the issue is the lack of name, so it would need a header like > > Content-Disposition: attachment; filename="somefeed.csv" > > Best regards > > > ¹ https://datatracker.ietf.org/doc/html/rfc6266 Thanks for the hint. It looks like this header needs to be "manually" added: https://docs.python.org/3/library/email.compat32-message.html#email.message… here: https://github.com/certtools/intelmq/blob/a51276054a7c5f0744ccb90cda54a03ee… Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

Intelmq SMTP bots not sending CSV in mail:: Need Help
by Kossi DOH 07 Jun '21

07 Jun '21

Hi Intelmq Community I will be very grateful if someone can help me solve this issue. I finished the configuration of the Intelmq to send mail using SMTP bots but all I try the attachment is not CSV but is as below: [cid:image001.png@01D75788.E1D91CB0] Even though the ATT file has all the required information the receiver will not be familiar with this file. How can I get the CSV file sent by mail? Cordialement / Best Regards, Kossi DOH Analyste Cyber Securite CYBER DEFENSE AFRICA S.A.S. Mobile: +228 70 54 93 26 www.cda.tg<http://www.cda.tg> This information is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Unauthorised use of this information by a person or entity other than the intended recipient is prohibited by law. If you received this by mistake, please immediately contact the sender by email or phone and delete this information from any computer. Thank you.

3 2

IntelMQ 2.3.3 release
by Sebastian Wagner 31 May '21

31 May '21

Dear community, While the development of the next major version 3.0.0 of IntelMQ is in the final spurt, the current version 2.3.3 marks the end of the 2.x development cycle. Beside small error corrections it comes with support for a few new Shadowserver feeds (https://www.shadowserver.org/news/changes-in-sinkhole-and-honeypot-report-t…). Please find below the list of changes. Thanks to all contributors for the issues reported and pull requests! The new version is already available on GitHub, PyPI, the deb+rpm repositories and DockerHub. Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html ### Core - `intelmq.lib.upgrade`: - Added `v233_feodotracker_browse` for Abuse.ch Feodotracker Browse parser configuration adaption (PR#1941 by Sebastian Wagner). ### Bots #### Parsers - `intelmq.bots.parsers.microsoft.parser_ctip`: - Add support for new field `SourceIpInfo.SourceIpv4Int` (PR#1940 by Sebastian Wagner). - Fix mapping of "ConnectionType" fields, this is not `protocol.application`. Now mapped to `extra.*.connection_type` (PR#1940 by Sebastian Wagner). - `intelmq.bots.parsers.shadowserver._config`: - Add support for the new feeds *Honeypot-Amplification-DDoS-Events*, *Honeypot-Brute-Force-Events*, *Honeypot-Darknet*, *IP-Spoofer-Events*, *Sinkhole-Events*, *Sinkhole-HTTP-Events*, *Vulnerable-Exchange-Server*, *Sinkhole-Events-HTTP-Referer* (PR#1950, PR#1952, PR#1953, PR#1954, PR#1970 by Birger Schacht and Sebastian Wagner, PR#1971 by Mikk Margus Möll). #### Experts - `intelmq.bots.experts.splunk_saved_search.expert`: - fixed erroneous string formatting (PR#1960 by Karl-Johan Karlsson). #### Outputs - `intelmq.bots.outputs.smtp.output`: - Handle empty "fieldnames" parameter by sending no attachment (PR#1932 by Sebastian Wagner). ### Documentation - Feeds: - Fixed Abuse.ch Feodotracker Browse parser configuration (PR#1941 by Sebastian Wagner fixes #1938). ### Tests - `intelmq.bots.parsers.html_table`: - Added testcase for Abuse.ch Feodotracker Browse (PR#1941 by Sebastian Wagner). ### Tools - intelmqsetup: - Set ownershop of state file path and its parent directory (PR#1911 by Sebastian Wagner). ### Known issues - ParserBot: erroneous raw line recovery in error handling (#1850). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

REDIS SERVER STOP
by hatibu chande 20 May '21

20 May '21

Hi, My redis server stopped working after successfully installing intelmq. The following logs are from the redis server. pam_unix(sudo:session): session opened for user intelmq by (uid=0) pam_unix(sudo:session): session closed for user intelmq Can anyone help me. Regards, Hatibu.

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users