- IntelMQ-users

Fwd: Fwd: Invitation to an IntelMQ in depth training on February 23th, 14:30h CET
by Sebix 09 Feb '22

09 Feb '22

-------- Forwarded Message -------- Subject: [IntelMQ-dev] Fwd: Invitation to an IntelMQ in depth training on February 23th, 14:30h CET Date: Thu, 3 Feb 2022 16:12:26 +0100 From: Otmar Lendl via IntelMQ-dev <intelmq-dev(a)lists.cert.at> Reply-To: Otmar Lendl <lendl(a)cert.at> To: ihap(a)lists.trusted-introducer.org <ihap(a)lists.trusted-introducer.org>, intelmq-dev(a)lists.cert.at

1 0

Taxii collector bot
by Joaquin Cabrera 18 Jan '22

18 Jan '22

Dear community, I'm looking for a collector bot to retrieve data from a taxii server, but i couldn't find any. We are tying to use intelMQ as our main tool to collect all security feeds and one of them is a taxii feed. Does anyone have this kind of scenario? Best regards, Joaquín Cabrera CERTuy - AGESIC Torre Ejecutiva Anexo Liniers 1280 piso 1 Tel: (+598) 2901 2929 Int. 8509 (11.000) Montevideo – URUGUAY www.agesic.gub.uy

4 5

[IntelMQ] Question about harmonization.conf
by Jonathan SCOUPREMAN 17 Dec '21

17 Dec '21

Greetings IntelMQ community, I'm looking for information about the harmonization.conf file located in /etc. In this file I see two sections: «event», and «report». Could you please tell me the difference? Unfortunately, I was not able to find hints in the current documentation. Also, as I see «report», does that mean IntelMQ has a reporting capability ? As far as I know IntelMQ doesn't provide a reporting capability, hence my doubt. ?? Thanks, Jonathan -- Jonathan SCOUPREMAN | jscoupreman(a)excellium-services.lu<mailto:jscoupreman@excellium-services.lu> | PGP Key ID: 0xAD971C07<http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x6802c48ead…> CERT-XLM | cert(a)excellium-services.com<mailto:cert@excellium-services.com> | PGP Key ID: 0xD74E5AC0<http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x67B311E5D7…> CERT-XLM Incident Handler @ excellium-services.com<https://excellium-services.com/> Excellium Services S.A. | 5 rue Goell L-5326 Contern Mobile: +352 691 982 790 Emergency: +352 262 039 64 708 | emergency(a)excellium-services.com<mailto:emergency@excellium-services.com> | PGP Key ID: 0x42662EFE<https://excellium-services.com/assets/EMERGENCY_PKEY.asc> This email is confidential and may contain legally privileged information. If you are not the intended recipient, you should not copy, distribute, disclose or use the information it contains, please e-mail the sender immediately and delete this message from your system. Note: e-mails are susceptible to corruption, interception and unauthorised amendment; we do not accept liability for any such changes, or for their consequences. You should be aware that we may monitor your e-mails and their content. Excellium Services SA.

3 3

Invitation to an IntelMQ Demo/Workshop on Dec. 9th, 13h CET
by Sebastian Waldbauer 25 Nov '21

25 Nov '21

Hey IntelMQ-Users :wave: I'd like to invite you to CERT.at's IntelMQ presentation. The presentation will be about 3 topics: - A short summary of what IntelMQ is & how it works under the hood - See it in action how CERT.at uses it - IntelMQ Toolset & Ecosystem Date/Time: 2021-12-09 13:00 - 14:30 UTC+1 (CET) Location: Zoom: https://us06web.zoom.us/j/84397883968?pwd=dm4xSFdlTUZUdmV2dVlFeGk3ZlRZdz09 Meeting-ID: 843 9788 3968 Code: 031551 Presenter: Sebastian Waldbauer No registration is required :) Kind regards, Sebastian -- // Sebastian Waldbauer <waldbauer(a)cert.at> - T: +43 1 5056416 7202 // CERT.at GmbH - https://www.cert.at/ // Firmenbuchnummer 561772k, LG Wien

1 0

IntelMQ 3.0.2 release fixing performance issues
by Sebastian Wagner 10 Sep '21

10 Sep '21

Dear community, Last week I said in the announcement for 3.0.1, that we didn't plan any further bugfix release before 3.1.0. I'm now proving myself wrong. Narzhan reported general performance issues and we investigated the problem on our own instances. We found two problems which different effects. It could be the case that you didn't even notice any performance drop, as both bugs affected only certain bots or situations. - When any collector creates a new report, it initializes an object of type Report. Since five years there was a bug, that this always loaded the harmonization.conf, instead of loading it just once at the start (which is - correctly - the case for new events). This was never a problem, as the Python's built-in json module was fast enough, but ruamel.yaml is slower. If a collector didn't massively create reports, that bug was not obvious, especially under testing conditions. And the bug was only relevant, if stream data is processed (for example with the Shodan stream collector). - Two bots, the STOMP collector and the API collector, don't have a proper process method, because they are using threading. A thread retrieves data asynchronously and sends the data. Normally, the process-method is running in an endless-loop, with sleeping rate_limit seconds between. If rate_limit is 0 or not defined, this would result in high load, therefore these bots defined the __collector_empty_process attribute which should have prohibited the endless loop. But due to class-restructuring, the attribute which would have been to be set was _Bot__collector_empty_process, and the flag was not effective. That did not only cause a loop, but as bot statistics are written to redis after every process-call, that caused continuous connections to redis. We've now renamed the flag to _collector_empty_process and fixed the two bots. The flag __is_multithreadable was affected analogous, but that should not have cause performance issues in particular. Additionally, the bot statistics can now be disabled by setting |statistics_host| to null (previously this caused an exception).| | The new version is on GitHub and PyPI, very soon in the deb/rpm repositories. The Docker image will follow on Monday or earlier. You can read the full changelog here: https://github.com/certtools/intelmq/releases/tag/3.0.2 have a nice weekend Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IEP04: The choice of the UUID-format
by Sebastian Wagner 07 Sep '21

07 Sep '21

Dear allies, The discussion around the IEP04 proposal, adding meta-information to IntelMQ messages, has stalled over the last months - first because of the time-intensive IntelMQ 3.0 release preparations and then because of the vacation season. Here is the current proposal: https://github.com/certtools/ieps/tree/main/004#readme Aaron, Sebastian Waldbauer and myself worked on it over the summer and also identified two open issues to be discussed: 1. The exact format of the meta-information and how to name and structure the fields. AIL made the first move and now uses a format similar to the previously proposed Variant "A". The IEP04 document contains the current proposal which is in line with the AIL format: https://github.com/certtools/ieps/tree/main/004#user-content-variant-ail If there are no other proposals, this will most probably the way to go. 2. The format of the UUID format which we want to uniquely identify IntelMQ events. We don't necessarily need to use the UUIDv4 format which represents pure randomness, but also other options which include the time and are even /time-sortable/. Sebastian Waldbauer analysed a couple of options and summarised his results in this document: https://github.com/certtools/ieps/blob/main/004/UUID.md Please let us know your opinion on the different UUID options. cheers Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

IntelMQ & API & Manager release 3.0.1 and upgrade note
by Sebastian Wagner 02 Sep '21

02 Sep '21

Dear community, Over the past two months, IntelMQ contributors had no summer pause, but did the final finish for IntelMQ 3.0. A special thanks goes to Mikk Margus Möll (CERT.ee) who has put tremendous efforts in the IntelMQ Manager tackling structural and usability issues, mainly in the JavaScript-components! The deb/rpm repositories did not receive the 3.0.0 release at beginning of July to get more experience with the major changes before doing automatic upgrades, but now they deliver the brand-new 3.0.1 version. Please note, that the automatic upgrade procedures may still not be fully smooth. Just now, we have noticed, that the packages contain a small flaw, which harms the upgrade experience: The packages ship a default configuration (the file is now called `runtime.yaml`), but only if the file does not exist before - for new installations. But now in this special case, we renamed the configuration from `runtime.conf` to `runtime.yaml` and therefore, the new - default shipped - configuration takes precedence. I hope the following commands and hints will be of help to you. # remove the runtime configuration shipped by the package (can be called /etc/intelmq/runtime.*) and rename your original one to /etc/intelmq/runtime.yaml # the previously used runtime.conf can be used as drop-in to runtime.yaml (YAML is backwards-compatible with JSON) sudo -u intelmq intelmqctl upgrade-config -f -u v300_pipeline_file_removal sudo -u intelmq intelmqctl upgrade-config -f -u v300_defaults_file_removal sudo -u intelmq intelmqctl upgrade-config -f -u v301_deprecations The last three steps are important to merge the defaults and pipeline configuration into the new combined configuration file Please do not hesitate to ask. The deb-packages are also already available for the newly released Debian 11 Bullseye. We are not planning a bugfix release until the 3.1.0 release, so that one will be the next version to be released. Here's a short summary of what happened during the summer: - various fixes related to the IEP001 implementation (IEP001 was the change configuration format and merge of files, rewrite oft the internal parameter-handling) - removal of the malwaredomains feed and parser, because it does not exist anymore - Various fixes in the Shadowserver Parser and support for new reports: Vulnerable SMTP Server, Microsoft Sinkhole Events Report & Microsoft Sinkhole HTTP Events Report, Honeypot HTTP Scan - SMTP Output: Added Content-Disposition-header to the attachment, fixing the display in MS Outlook clients (as reported and dicussed on the Mailinglist). - Heavy refactoring of IntelMQ-Manager's JavaScript parts to fix errors and usability issues. If you are interested in developing on IntelMQ and you don't know where to start, have a look at the dev guide an the issues labeled "good first issue": https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3A… We are especially welcoming contributions to the documentation! You can read the full changelogs here: - https://github.com/certtools/intelmq/releases/tag/3.0.1 - https://github.com/certtools/intelmq-api/releases/tag/3.0.1 - https://github.com/certtools/intelmq-manager/releases/tag/3.0.1 https://cert.at/en/blog/2021/9/intelmq-301-release https://twitter.com/CERT_at/status/1433475188381806594 btw: There's new contact management portal called "tuency" for administrating abuse contacts available, which can be used in conjunction with IntelMQ. Read more about its features here: https://cert.at/en/blog/2021/9/tuency-constituency-portal-for-iocs-and-certs https://gitlab.com/intevation/tuency/tuency -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Intelmq-docker v.2.3.3 API
by Jessica Schumacher 25 Aug '21

25 Aug '21

Hi all, I recently looked into intelmq-docker [1] and really liked the concept of it. Especially since we currently have IntelMQ in a dockerized setup running, it looks really promising. Thank you for the work you put into it so far! I do have a question though or more of an issue I ran into. I am currently working with `intelmq-full-dev` to implement our own bots. I use the docker image `certat/intelmq-full:2.3.3` so I am using IntelMQ Version 2.3.3 in my setup. I get the following error messages in IntelMQ Manager: The file does indeed not exist at this url shown in the image but can be accessed via: http://127.0.0.1:1337/intelmq/v1/api/config?file=positions <http://127.0.0.1:1337/intelmq/v1/api/config?file=positions> My guess is, that an older IntelMQ API is installed where only one of those methods to access the files works. At the moment version 2.3.1 of the IntelMQ API is automatically installed when setting up the docker container. If I compare the api.py of 2.3.1 [2] with the current one [3], I can see, that entries for “runtime” or “positions” for example are missing in version 2.3.1; therefor the error. I am not entirely sure if this has something to do with this [4] issue on Github. Is this something that can be fixed on the docker image from your site? Or am I doing something wrong here? Thank you for your help. Kind regards, Jessica [1] https://github.com/certat/intelmq-docker <https://github.com/certat/intelmq-docker> [2] https://github.com/certtools/intelmq-api/blob/2.3.1/intelmq_api/api.py <https://github.com/certtools/intelmq-api/blob/2.3.1/intelmq_api/api.py> [3] https://github.com/certtools/intelmq-api/blob/develop/intelmq_api/api.py <https://github.com/certtools/intelmq-api/blob/develop/intelmq_api/api.py> [4] https://github.com/certat/intelmq-docker/issues/8 <https://github.com/certat/intelmq-docker/issues/8> -- SWITCH Jessica Schumacher, SWITCH-CERT Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland phone +41 44 268 15 15, direct +41 44 268 16 33 jessica.schumacher(a)switch.ch, http://www.switch.ch

2 2

Unable to send CSV file using Intelmq
by Kossi DOH 22 Aug '21

22 Aug '21

Dear Community Please I need help to configure the SMTP bots of Intelmq to send CSV file as describe in the description. I check the MIME and I is not the blocking point. The Information needed to be sent in CSV is either in the ATT00001 file and contrary to all the Intelmq mail I do receive mine is not .txt. Can someone please help me to solve this issue? This is an example of the file sent after the configuration: [cid:image001.png@01D77335.33F95C40] Cordialement / Best Regards, Kossi DOH Analyste Cyber Securite CYBER DEFENSE AFRICA S.A.S. Mobile: +228 70 54 93 26 https://cert.tg This information is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Unauthorised use of this information by a person or entity other than the intended recipient is prohibited by law. If you received this by mistake, please immediately contact the sender by email or phone and delete this information from any computer. Thank you.

4 9

[IntelMQ] Deduplication on an optional field
by Guillaume GRANJON DE LEPINEY 06 Aug '21

06 Aug '21

Hello everyone, I wonder if there is a simple way to use a Deduplicator bot on an optional field. Indeed, I noticed when I apply the deduplicator on an optional field that the null value must be entered in the redis because all messages (except the first one) that do not contain the field are dropped. Is there a workaround please? I could work around this problem by adding two Sieve bots at the exit of the precedent bot that would jump the Deduplicator bot if the message doesn't have the field, but I don't find that to be optimal. Thus, I am open to any proposal that could help me. Regards, Guillaume GRANJON de LÉPINEY | ggranjon(a)excellium-services.be<mailto:ggranjon@excellium-services.be> | PGP Key ID: 0xE2FD5ED1<https://pgp.circl.lu/pks/lookup?search=0xE2FD5ED1&fingerprint=on&op=index> CERT-XLM Incident Handler @ excellium-services.com<https://excellium-services.com/> CERT-XLM | cert(a)excellium-services.com<mailto:cert@excellium-services.com> | PGP Key ID: 0xD74E5AC0<http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x67B311E5D7…> Excellium Services Belgium N.V. | Orion Bldg, Belgicastraat 13, B-1930 Zaventem, Belgium Mobile: +32 4 71 98 57 65 Emergency: +352 262 039 64 708 | emergency(a)excellium-services.com<mailto:emergency@excellium-services.com> | PGP Key ID: 0x42662EFE<https://excellium-services.com/assets/EMERGENCY_PKEY.asc> This email is confidential and may contain legally privileged information. If you are not the intended recipient, you should not copy, distribute, disclose or use the information it contains, please e-mail the sender immediately and delete this message from your system. Note: e-mails are susceptible to corruption, interception and unauthorised amendment; we do not accept liability for any such changes, or for their consequences. You should be aware that we may monitor your e-mails and their content. Excellium Services SA.

2 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users