IntelMQ-dev May 2021

intelmq-dev@lists.cert.at

6 participants
5 discussions

IntelMQ 2.3.3 release
by Sebastian Wagner 31 May '21

31 May '21

Dear community, While the development of the next major version 3.0.0 of IntelMQ is in the final spurt, the current version 2.3.3 marks the end of the 2.x development cycle. Beside small error corrections it comes with support for a few new Shadowserver feeds (https://www.shadowserver.org/news/changes-in-sinkhole-and-honeypot-report-t…). Please find below the list of changes. Thanks to all contributors for the issues reported and pull requests! The new version is already available on GitHub, PyPI, the deb+rpm repositories and DockerHub. Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html ### Core - `intelmq.lib.upgrade`: - Added `v233_feodotracker_browse` for Abuse.ch Feodotracker Browse parser configuration adaption (PR#1941 by Sebastian Wagner). ### Bots #### Parsers - `intelmq.bots.parsers.microsoft.parser_ctip`: - Add support for new field `SourceIpInfo.SourceIpv4Int` (PR#1940 by Sebastian Wagner). - Fix mapping of "ConnectionType" fields, this is not `protocol.application`. Now mapped to `extra.*.connection_type` (PR#1940 by Sebastian Wagner). - `intelmq.bots.parsers.shadowserver._config`: - Add support for the new feeds *Honeypot-Amplification-DDoS-Events*, *Honeypot-Brute-Force-Events*, *Honeypot-Darknet*, *IP-Spoofer-Events*, *Sinkhole-Events*, *Sinkhole-HTTP-Events*, *Vulnerable-Exchange-Server*, *Sinkhole-Events-HTTP-Referer* (PR#1950, PR#1952, PR#1953, PR#1954, PR#1970 by Birger Schacht and Sebastian Wagner, PR#1971 by Mikk Margus Möll). #### Experts - `intelmq.bots.experts.splunk_saved_search.expert`: - fixed erroneous string formatting (PR#1960 by Karl-Johan Karlsson). #### Outputs - `intelmq.bots.outputs.smtp.output`: - Handle empty "fieldnames" parameter by sending no attachment (PR#1932 by Sebastian Wagner). ### Documentation - Feeds: - Fixed Abuse.ch Feodotracker Browse parser configuration (PR#1941 by Sebastian Wagner fixes #1938). ### Tests - `intelmq.bots.parsers.html_table`: - Added testcase for Abuse.ch Feodotracker Browse (PR#1941 by Sebastian Wagner). ### Tools - intelmqsetup: - Set ownershop of state file path and its parent directory (PR#1911 by Sebastian Wagner). ### Known issues - ParserBot: erroneous raw line recovery in error handling (#1850). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IEP04: IntelMQ Data Format - Meta-Information
by Sebastian Waldbauer 17 May '21

17 May '21

Dear IntelMQ Developers and Users, nowadays security incidents are more important than 10 years ago. As IntelMQ can be used as core element for automated security incident handling, we would like to provide a way to share information with other intelmq instances. This proposal is also an alternative to IEP03 insofar as solving the "multiple values" is possible by using UUIDs so "link" related events in a backwards-compatible manner. If you're interested, please let us know, so we could organize a hackathon for further discussions about the specification of the meta-information. Previously this idea was discussed in [0] and [1]. [0] https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectu… [1] https://github.com/certtools/intelmq/issues/1521 # IEP04: Internal Data Format: Meta Information and Data Exchange To ease data exchange between two or more IntelMQ instances, adding some meta-information to the events can make this sharing easier in certain regards. "Linking" events could be based on the same theory as `git` using it - with parent hashes ( we would call it UUID ). ### TL;DR Communication between one or more IntelMQ instances & exchange data with a backwards-compatible format. P2P or centralized architecture is a big topic, which has to be discussed after the format is being set. ### Why is metadata important? Short and simple. To avoid race conditions & being able to discard/drop already processed events from other instances. ### Meta information Metadata is used to transfer some general data, which is not likely related to the event itself. It's more or less just an information to keep events clear & sortable. A message could look like: { "meta": { "version": 1, # protocol version, so we are allowed to fallback to old versions too "uuid": { current: "cert_at:aaaa-bbbb-cccc-dddd" # format to be decided parent: "cert_at:xxxx-yyyy-zzzz-ffff" # format to be discussed, if not set -> current is the parent uuid }, "type": "event", "format": "intelmq", # i. e. this field could contain "n6" or "idea", so the receiving component can decode on demand. }, "payload": { # normal intelmq data "source.ip": "127.0.0.1", "source.fqdn": "example.com", "raw": base64-blob } } Tell us your opinion about adding non-standardized meta-information fields ( i. e. RTIR ticket number, origin, other local contact informationen ... and so on ) #### The UUID For the UUID there are multiple options: 1. Generate a random 128 bit UUID 2. A list of entities, which dealt with this event already. For example if an event was passed on from cert-at to cert-ee, the field could look like `!cert-at!cert-ee`. A message sending loop can be detected if the own name is already in this field upon reception. 3. Using CyCat: `publisher-short-name:project-short-name:UUID`. For example: `cert-at:intelmq:72ddb00c-2d0a-4eea-b7ac-ae122b8e6c3b`, or `cert-pl:n6:f60c9fb9-81f9-4e0b-8a44-ea41326a15b3`. Some more research and discussion is required before the implementation of this option. Have a look at https://www.cycat.org/services/concept/ for more details. 4. A hash: A benefit using a hash is that we're able to recalculate them on every intelmq instance. ### Exporting events to other systems In IntelMQ 2.x the events only comprise of the "payload" and no meta information. For local storages like file output or databases, the meta information may not be relevant in some use-cases. So it needs to be possible to export events *without* meta information, which is also the backwards-compatible behaviour. The "type" field exists in the current format as "__type" in the flat payload structure. In the output bots there's currently a boolean parameter `message_with_type` to include the field `__type` in the "export". For optionally exporting meta-information like uuid or format, a similar logic could be used. ### How can data exchange work? This now depends on how IntelMQ instances can communicate, either Peer-to-peer or via a central data hub. Both of them do have pro's and con's. #### P2P ( Peer 2 Peer ) Decentralized network + Less downtimes: A downtime of one instance, does not affect the whole network + Better privacy: data is not shared to an unrelated instance + More secure: data can optionally be encrypted (key-exchange between instances?) + Decentralized and local maintenance ~ Network latency depends on server locations - Networking issues may occur How would data exchange looks like between two instances: 1) Instance A has events which should be relayed to Instance B & C, because they're not sure who the actually receiver should be 2) Instance A ensures all messages have a UUID 3) Instance A sends the data to Instance B & Instance C 4) Instance B checks the data & they're sure that the data should be for Instance C 5) Instance C receives data from Instance A & Instance B 6) Instance C checks the UUID, which is the same & drops the package from Instance B #### (Central) Data hub + Less maintenance: Is maintained by the hub administrator + Central data storage (reports can optionally be cached to be downloaded later) ~ Central data analysis (e.g. statistics) is possible ~ Network latency depends on server locations - point of failure: if network problems occur, no exchange is possible As already seen above, data exchange here would be less complicated. The sending may look like: 1) Instance A has events which should be relayed to Instance B (e.g. different country) 2) Instance A ensures all messages have a UUID 3) Instance A sends these messages to the data hub The reception side can look like: 1) Instance B connects to central instance 2) Instance B queries and downloads all available messages 3) Upon reception, all messages are de-duplicated based on the UUID: a) If the UUID is already known, discard the message b) If the UUID has not been seen before, continue with processing To sum up, both exchange variants are useful. More research is needed, i. e. a mixed infrastructure with centralized parts but can be decentralized too. However, this shall not be neither the purpose nor the aim of this IEP. -- // Sebastian Waldbauer <waldbauer(a)cert.at> - T: +43 1 5056416 7202 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

7 17

question pandas
by L. Aaron Kaplan 16 May '21

16 May '21

Hi intelmq-dev, while thinking through a lot (and IEP04), I was wondering if there was ever the need to process intelmq data (DHO/IDF) via standard pandas dataframe like syntax? Would that be appealing for developers? I myself started to use pandas on and off more and more over the last years. And it is the de-facto data science tool/library for python (together with numpy). Best, Aaron.

2 1

RFH: intelmq-manager adaptions to new intelmq-api
by Birger Schacht 04 May '21

04 May '21

Hi *, tl;dr: we need your help with adapting the intelmq-manager (Javascript) to the new API endpoints. due to the changes described in IEP01 regarding the configuration files, there are a lot of adaptions necessary in the IntelMQ API and in the IntelMQ Manager. We already updated the API accordingly and removed most of the file handling logic and replaced the direct file access with dedicated endpoints for reading and writing the configuration. All the configuration settings for IntelMQ are now stored in the runtime.yaml file and there is a correspoinding endpoint /api/runtime for GET and POST which works with JSON. This replaces the default.conf, runtime.conf and pipeline.conf configuration files. While we were at it, we also created endpoints for the harmonization file (/api/harmonization, GET) and the positions file (/api/positions, GET and POST). Now some of the intelmq-manager logic needs to be updated, to work with the new endpoints. Integrating the harmonziation and positions endpoints shouldn't be a problem, but the pipeline configuration was now merged with the runtime so this might need some adaptions in the overall structure of the Javascript code. Is anyone here experienced enough in Javascript to tackle that? A first step would probably be to modify the `load_configuration` method in the runtime.js file and remove traces of the legacy configuration files and then use the runtime config to create the app.edges dict using an updated version of the `read_pipeline_conf` function... Any help with this task would be appreciated. cheers, Birger -- // Birger Schacht <schacht(a)cert.at> // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Redis failing to start after intelmq* package update
by Mika Silander 04 May '21

04 May '21

Hi, Our intelmq installation on Ubuntu 20.04 LTS was auto-updated to the versions: Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-===============-============-============-=======================================> ii intelmq 2.3.2-1 all Solution for IT security teams for coll> ii intelmq-api 2.3.1-1 all HUG based API for the intelmq project ii intelmq-manager 2.3.1-2 all Graphical interface to manage configura For some reason intelmq-manager now shows a problem with accessing redis queues as it tries to run sudo -u www-data sudo -u intelmq intelmqctl --type json list queues-and-status Traceback (most recent call last): File "/usr/lib/python3/dist-packages/intelmq/lib/pipeline.py", line 286, in count_queued_messages queue_dict[queue] = self.pipe.llen(queue) File "/usr/lib/python3/dist-packages/redis/client.py", line 1671, in llen return self.execute_command('LLEN', name) File "/usr/lib/python3/dist-packages/redis/client.py", line 836, in execute_command conn = self.connection or pool.get_connection(command_name, **options) File "/usr/lib/python3/dist-packages/redis/connection.py", line 1073, in get_connection connection.connect() File "/usr/lib/python3/dist-packages/redis/connection.py", line 544, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused. systemctl status redis.service indicates there's a problem starting redis: May 03 15:48:20 ourhost systemd[1]: Failed to start Advanced key-value store. May 03 15:48:20 ourhost systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 5. May 03 15:48:20 ourhost systemd[1]: Stopped Advanced key-value store. May 03 15:48:20 ourhost systemd[1]: redis-server.service: Start request repeated too quickly. May 03 15:48:20 ourhost systemd[1]: redis-server.service: Failed with result 'exit-code'. May 03 15:48:20 ourhost systemd[1]: Failed to start Advanced key-value store. Don't know whether this has anything to do with the package update or is this an independent problem but hints as to how to remedy this would be welcome. Br, Mika

3 5

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev May 2021