IntelMQ-dev

intelmq-dev@lists.cert.at

1 participants
247 discussions

IntelMQ & API & Manager Release 3.0
by Sebastian Wagner 05 Aug '21

05 Aug '21

Dear community, The time has come and IntelMQ 3.0 is final! We, as community, made loads of changes, smaller and bigger ones, and I really think that IntelMQ became more user-friendly, developer-friendly and feature-rich at the same time! There are some major changes in this release, especially the in the field of the configuration and Internal Data format (previously: "harmonization"). For the configuration-part, the upgrade part should be automatic with `intelmqctl upgrade-config` as usual. For the Data format, carefully look at your bot configurations (filters, sieve, etc.) to update them. Adaptions in systems connected to IntelMQ, especially also databases might be necessary as well. The NEWS.md file give a summary of what has changed: https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-… We don't recommend to upgrade existing production instance of IntelMQ yet. We of course did testing, including the end-to-end tests, and have detailed release notes. But for critical systems, a delayed upgrade makes sense ;) Therefore the stable deb/rpm repositories don't contain the 3.0 release yet! Even though an upgrade of production systems is not yet recommended, extensive usage and testing of the new releases are very much welcome and required to get the necessary feedback for the next (maintenance) releases. The releases are available via git, PyPI, Docker and the *unstable* deb/rpm repositories. Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html IntelMQ API documentation: https://intelmq.readthedocs.io/en/maintenance/user/intelmq-api.html IntelMQ Manager documentation: https://intelmq.readthedocs.io/en/maintenance/user/intelmq-manager.html NEWS/release notes of IntelMQ (Core): https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-… Full Changelog of IntelMQ (Core): https://github.com/certtools/intelmq/blob/maintenance/CHANGELOG.md#300-2021… On a high level, these are the major changes compared to version 2.3.x (2.3.3 was released 2021-05-31): In the core and Docker: * Configuration rewrite including parameter loading and handling (IEP01), plus the required adoption of the API and Manager, by Birger Schacht (CERT.at). * Classification sync with RSIT, by Sebastian Wagner (CERT.at). * Removal of the BOTS file, by Sebastian Waldbauer (CERT.at). * Creation and maintenance of the Docker images by Sebastian Waldbauer (CERT.at). * Creation of Docker-instructions for development setups by Einar Lanfranco and Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar). New and majorly enhanced bots: * Added |intelmq.bots.collectors.fireeye|: A bot that collects indicators from Fireeye MAS appliances (PR#1745 by Christopher Schappelwein). * |intelmq.bots.collectors.api.collector_api|: Added UNIX socket capability (PR#1987 by Mikk Margus Möll, fixes #1986). * Added |intelmq.bots.parsers.fireeye|: A bot that parses hashes and URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein). * Added |intelmq.bots.experts.http.expert_status|: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR#1789 by Birger Schacht, fixes #1047 partly). * Added |intelmq.bots.experts.http.expert_content|: A bot that fetches an HTTP resource and checks if it contains a specific string (PR#1811 by Birger Schacht). * Added |intelmq.bots.experts.lookyloo.expert|: A bot that sends requests to a lookyloo instance & adds |screenshot_url| to the event (PR#1844 by Sebastian Waldbauer, fixes #1048). * Added |intelmq.bots.experts.rdap.expert|: A bot that checks the RDAP protocol for an abuse contact for a given domain (PR#1881 by Sebastian Waldbauer and Sebastian Wagner). * |intelmq.bots.experts.sieve.expert|: Major refactoring and lot's of new functionality New operators for working with various types (lists, sets, booleans, float, int), generic rule negation and nesting (PR#1895 by Mikk Margus Möll). * Added |intelmq.bots.experts.uwhoisd|: A bot that fetches the whois entry from a uwhois-instance (PR#1918 by Raphaël Vinot). * Added |intelmq.bots.experts.aggregate|: A bot that aggregate events based upon given fields & a timespan. (PR#1959 by Sebastian Waldbauer) * Added |intelmq.bots.experts.tuency|: A bot that queries the IntelMQ API of a tuency instance (PR#1857 by Sebastian Wagner, fixes #1856). * Added |intelmq.bots.outputs.templated_smtp| (PR#1901 by Karl-Johan Karlsson). On the documentation front, these are the most important changes * License and copyright information was added to all the bots (by Birger Schacht). * Added documentation on the EventDB (PR#1955 by Birger Schacht, PR#1985 by Sebastian Wagner). * Added TimescaleDB for time-series documentation (PR#1990 by Sebastian Waldbauer). * n6 interoperability documentation: Adding more graphs and illustrations (PR#1991 by Sebastian Wagner). * Added documentation on abuse-contact look-ups (PR#2021 by Sebastian Waldbauer and Sebastian Wagner). And not to forget all the smaller changes and additions. Thanks to (in random order) Raphaël Vinot (circl.lu) Bernhard Reiter (intevation.de) Sebastian Wagner (CERT.AT) Filip Pokorný (CSIRT.CZ) Guillaume GRANJON de LÉPINEY (CERT XLM excellium-services.com) Mikk Margus Möll (CERT.ee) Alex Kaplan Thomas Hungenberg (CERT-BUND.DE) Einar Lanfranco (CERT-UNLP cert.unlp.edu.ar) Christopher Schappelwein (milCERT, BMLV.gv.at) Marcos Gonzalez (CSIRT-RD cncs.gob.do/csirt-rd/) Marius Karotkis (NRDCS.LT) Sebastian Waldbauer (CERT.AT) Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar) Karl-Johan Karlsson (Linköping University LIU.SE) Birger Schacht (CERT.AT) ... and all the contributors of previous releases and as well to all reporters, supporters, etc! best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

How does one avoid reprocessing mail attachments already processed?
by Mika Silander 04 Aug '21

04 Aug '21

Hi all, We are using the mail collector bot to read email attachments over IMAPS. It works fine but rereads all the messages in the message folder even if it had processed them already. Is there a way to configure the mail bot to remove the messages once it has processed them? I didn't find a hint concerning this in the bot's documentation nor the sources. What approaches do people use to avoid reprocessing messages? Br, Mika

2 3

Path/permission problem with intelmq-manager (v2.3.1-2) ?
by Mika Silander 02 Aug '21

02 Aug '21

Hi all, Returning to an earlier problem that surfaced after having updated intelmq to version 2.3.3-1: intelmq-manager starts, but when opening the configuration tab, the network of bots doesn't get rendered. Looking at the output of the browser's developer console one can see successful retrieval of https://ourhost/configs.html that in turn initiates the successful retrieval of a number of CSS, PNG and JS files. Next we have five failures (404 Not found): https://ourhost/intelmq/v1/api/config?file=bots https://ourhost/intelmq/v1/api/config?file=defaults https://ourhost/intelmq/v1/api/config?file=runtime https://ourhost/intelmq/v1/api/config?file=pipeline https://ourhost/intelmq/v1/api/config?file=positions Then a few more successful retrievals of PNGs and a GIF. After this we end up in an infinite loop that tries unsuccessfully (404 Not found) to fetch https://ourhost/intelmq/v1/api/queues-and-status once every second. The error the browser shows repeatedly is "Error loading bot queues information:". What is common to all failures above is that they seem to be due to jquery-actions. Any ideas as to where the problem could be? I suspect some path and/or permission problem but that's the first guess. Br, Mika For reference, these are the package versions on our intelmq Ubuntu 20 LTS host: intelmq 2.3.3-1 intelmq-api 2.3.1-1 intelmq-manager 2.3.1-2

2 3

Preventing lost events when stopping intelmq
by Mika Silander 30 Jul '21

30 Jul '21

Hi, Returning to a similar issue but from a different angle: for maintenance I'd like to be able to cleanly shutdown the server running intelmq. Is there a way to guarantee that none of the bots is in a processing state (i.e. processing an event in the process method) before server shutdown? Can "intelmqctl stop" for example stop the bot chain in such a way that none of the bots is in the midst of processing an event? If not, what would be the best approach for achieving this? Br, Mika

2 5

Re: [IntelMQ-dev] [IntelMQ-users] Unable to send CSV file using Intelmq
by Bernhard Reiter 13 Jul '21

13 Jul '21

Hi Kossi, Am Montag 12 Juli 2021 18:25:05 schrieb Kossi DOH via IntelMQ-users: > I will work on it with and see what it gives. If I understand well this > issue is related to Microsoft Outlook. Sebastian also created the following issue, which you can add information, help and subscribe to, if you like: https://github.com/certtools/intelmq/issues/2018 Regards, Bernhard -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

IntelMQ & API & Manager 3.0.0 Release candidate
by Sebastian Wagner 26 Jun '21

26 Jun '21

Dear community, The 3.0.0 stable release is already to be released mid next week, so I've created 3.0.0 Release candidate versions for IntelMQ, IntelMQ API and the IntelMQ Manager. Thanks everyone who contributed to IntelMQ making this major release possible! This is a big step for the IntelMQ project and we all can be proud of it. All unit-tests and the end-to-end integration tests[0] succeed without any problems. However, I have two bad news: One bug discovered prevents the use of inline comments in the configuration file - or actually the workaround for a related bug: https://github.com/certtools/intelmq/issues/2003 Unfortunately there are open issues in the IntelMQ Manager[1] most importantly one bug which prevents bots to be connected with queues[2]. Therefore, I am hesitant releasing a stable version of the Manager in this state. If you can code JavaScript and have some free time available, please consider contributing. /The IntelMQ project needs your contribution/. IntelMQ has some known issues as well[3]. Before trying out the new code base, read the NEWS.md file, it describes a few notable major changes[4]. I don't recommend upgrading production instances yet! The unstable repositories with deb & rpm packages are updated and contain the 3.0.0 RC1 versions: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… If you want to try them out with vagrant, you can start here[0] best regards Sebastian [0]: https://github.com/schacht-certat/intelmq-vagrant/pull/19 [1]: https://github.com/certtools/intelmq-manager/issues?q=is%3Aopen+is%3Aissue+… [2]: https://github.com/certtools/intelmq-manager/issues/260 [3]: https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3A… [4]: https://github.com/certtools/intelmq/blob/develop/NEWS.md#user-content-300-… -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq-manager coughs after upgrade from intelmq 2.3.2 to 2.3.3
by Mika Silander 07 Jun '21

07 Jun '21

Hi, I upgraded intelmq 2.3.2 to 2.3.3 on Ubuntu 20 LTS and after that intelmq-manager shows a strange error when accessing https:/ourhost/intelmq-manager/configs.html. When one clicks open the error message (displayed in the upper right hand corner of the intelmq-manager view), a page appears saying: 09:30:23 : Error loading bot queues information : then displays the contents of the normal configs.html page and after that ?> SyntaxError: Unexpected token < in JSON at line 3. It also seems some endless javascript loop is started and the above error lines + configs.html page gets repeated several times over in a single web page. The previous installation of intelmq 2.3.2 worked fine. All ideas for troubleshooting and fixing this would be much appreciated. For reference, our exact setup is now: intelmq 2.3.3-1 intelmq-api 2.3.1-1 intelmq-manager 2.3.1-2 Br, Mika

2 2

IntelMQ 2.3.3 release
by Sebastian Wagner 31 May '21

31 May '21

Dear community, While the development of the next major version 3.0.0 of IntelMQ is in the final spurt, the current version 2.3.3 marks the end of the 2.x development cycle. Beside small error corrections it comes with support for a few new Shadowserver feeds (https://www.shadowserver.org/news/changes-in-sinkhole-and-honeypot-report-t…). Please find below the list of changes. Thanks to all contributors for the issues reported and pull requests! The new version is already available on GitHub, PyPI, the deb+rpm repositories and DockerHub. Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html ### Core - `intelmq.lib.upgrade`: - Added `v233_feodotracker_browse` for Abuse.ch Feodotracker Browse parser configuration adaption (PR#1941 by Sebastian Wagner). ### Bots #### Parsers - `intelmq.bots.parsers.microsoft.parser_ctip`: - Add support for new field `SourceIpInfo.SourceIpv4Int` (PR#1940 by Sebastian Wagner). - Fix mapping of "ConnectionType" fields, this is not `protocol.application`. Now mapped to `extra.*.connection_type` (PR#1940 by Sebastian Wagner). - `intelmq.bots.parsers.shadowserver._config`: - Add support for the new feeds *Honeypot-Amplification-DDoS-Events*, *Honeypot-Brute-Force-Events*, *Honeypot-Darknet*, *IP-Spoofer-Events*, *Sinkhole-Events*, *Sinkhole-HTTP-Events*, *Vulnerable-Exchange-Server*, *Sinkhole-Events-HTTP-Referer* (PR#1950, PR#1952, PR#1953, PR#1954, PR#1970 by Birger Schacht and Sebastian Wagner, PR#1971 by Mikk Margus Möll). #### Experts - `intelmq.bots.experts.splunk_saved_search.expert`: - fixed erroneous string formatting (PR#1960 by Karl-Johan Karlsson). #### Outputs - `intelmq.bots.outputs.smtp.output`: - Handle empty "fieldnames" parameter by sending no attachment (PR#1932 by Sebastian Wagner). ### Documentation - Feeds: - Fixed Abuse.ch Feodotracker Browse parser configuration (PR#1941 by Sebastian Wagner fixes #1938). ### Tests - `intelmq.bots.parsers.html_table`: - Added testcase for Abuse.ch Feodotracker Browse (PR#1941 by Sebastian Wagner). ### Tools - intelmqsetup: - Set ownershop of state file path and its parent directory (PR#1911 by Sebastian Wagner). ### Known issues - ParserBot: erroneous raw line recovery in error handling (#1850). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IEP04: IntelMQ Data Format - Meta-Information
by Sebastian Waldbauer 17 May '21

17 May '21

Dear IntelMQ Developers and Users, nowadays security incidents are more important than 10 years ago. As IntelMQ can be used as core element for automated security incident handling, we would like to provide a way to share information with other intelmq instances. This proposal is also an alternative to IEP03 insofar as solving the "multiple values" is possible by using UUIDs so "link" related events in a backwards-compatible manner. If you're interested, please let us know, so we could organize a hackathon for further discussions about the specification of the meta-information. Previously this idea was discussed in [0] and [1]. [0] https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectu… [1] https://github.com/certtools/intelmq/issues/1521 # IEP04: Internal Data Format: Meta Information and Data Exchange To ease data exchange between two or more IntelMQ instances, adding some meta-information to the events can make this sharing easier in certain regards. "Linking" events could be based on the same theory as `git` using it - with parent hashes ( we would call it UUID ). ### TL;DR Communication between one or more IntelMQ instances & exchange data with a backwards-compatible format. P2P or centralized architecture is a big topic, which has to be discussed after the format is being set. ### Why is metadata important? Short and simple. To avoid race conditions & being able to discard/drop already processed events from other instances. ### Meta information Metadata is used to transfer some general data, which is not likely related to the event itself. It's more or less just an information to keep events clear & sortable. A message could look like: { "meta": { "version": 1, # protocol version, so we are allowed to fallback to old versions too "uuid": { current: "cert_at:aaaa-bbbb-cccc-dddd" # format to be decided parent: "cert_at:xxxx-yyyy-zzzz-ffff" # format to be discussed, if not set -> current is the parent uuid }, "type": "event", "format": "intelmq", # i. e. this field could contain "n6" or "idea", so the receiving component can decode on demand. }, "payload": { # normal intelmq data "source.ip": "127.0.0.1", "source.fqdn": "example.com", "raw": base64-blob } } Tell us your opinion about adding non-standardized meta-information fields ( i. e. RTIR ticket number, origin, other local contact informationen ... and so on ) #### The UUID For the UUID there are multiple options: 1. Generate a random 128 bit UUID 2. A list of entities, which dealt with this event already. For example if an event was passed on from cert-at to cert-ee, the field could look like `!cert-at!cert-ee`. A message sending loop can be detected if the own name is already in this field upon reception. 3. Using CyCat: `publisher-short-name:project-short-name:UUID`. For example: `cert-at:intelmq:72ddb00c-2d0a-4eea-b7ac-ae122b8e6c3b`, or `cert-pl:n6:f60c9fb9-81f9-4e0b-8a44-ea41326a15b3`. Some more research and discussion is required before the implementation of this option. Have a look at https://www.cycat.org/services/concept/ for more details. 4. A hash: A benefit using a hash is that we're able to recalculate them on every intelmq instance. ### Exporting events to other systems In IntelMQ 2.x the events only comprise of the "payload" and no meta information. For local storages like file output or databases, the meta information may not be relevant in some use-cases. So it needs to be possible to export events *without* meta information, which is also the backwards-compatible behaviour. The "type" field exists in the current format as "__type" in the flat payload structure. In the output bots there's currently a boolean parameter `message_with_type` to include the field `__type` in the "export". For optionally exporting meta-information like uuid or format, a similar logic could be used. ### How can data exchange work? This now depends on how IntelMQ instances can communicate, either Peer-to-peer or via a central data hub. Both of them do have pro's and con's. #### P2P ( Peer 2 Peer ) Decentralized network + Less downtimes: A downtime of one instance, does not affect the whole network + Better privacy: data is not shared to an unrelated instance + More secure: data can optionally be encrypted (key-exchange between instances?) + Decentralized and local maintenance ~ Network latency depends on server locations - Networking issues may occur How would data exchange looks like between two instances: 1) Instance A has events which should be relayed to Instance B & C, because they're not sure who the actually receiver should be 2) Instance A ensures all messages have a UUID 3) Instance A sends the data to Instance B & Instance C 4) Instance B checks the data & they're sure that the data should be for Instance C 5) Instance C receives data from Instance A & Instance B 6) Instance C checks the UUID, which is the same & drops the package from Instance B #### (Central) Data hub + Less maintenance: Is maintained by the hub administrator + Central data storage (reports can optionally be cached to be downloaded later) ~ Central data analysis (e.g. statistics) is possible ~ Network latency depends on server locations - point of failure: if network problems occur, no exchange is possible As already seen above, data exchange here would be less complicated. The sending may look like: 1) Instance A has events which should be relayed to Instance B (e.g. different country) 2) Instance A ensures all messages have a UUID 3) Instance A sends these messages to the data hub The reception side can look like: 1) Instance B connects to central instance 2) Instance B queries and downloads all available messages 3) Upon reception, all messages are de-duplicated based on the UUID: a) If the UUID is already known, discard the message b) If the UUID has not been seen before, continue with processing To sum up, both exchange variants are useful. More research is needed, i. e. a mixed infrastructure with centralized parts but can be decentralized too. However, this shall not be neither the purpose nor the aim of this IEP. -- // Sebastian Waldbauer <waldbauer(a)cert.at> - T: +43 1 5056416 7202 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

7 17

question pandas
by L. Aaron Kaplan 16 May '21

16 May '21

Hi intelmq-dev, while thinking through a lot (and IEP04), I was wondering if there was ever the need to process intelmq data (DHO/IDF) via standard pandas dataframe like syntax? Would that be appealing for developers? I myself started to use pandas on and off more and more over the last years. And it is the de-facto data science tool/library for python (together with numpy). Best, Aaron.

2 1

← Newer
1
...
5
6
7
8
9
10
11
...
25
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev