IntelMQ-dev

intelmq-dev@lists.cert.at

6 participants
232 discussions

IEP03: IntelMQ Data Format - Multiple Values
by Sebastian Waldbauer 22 Apr '21

22 Apr '21

Dear IntelMQ Developers and Users, an evaluation of current challenges with the internal data format led to the idea of allowing multiple values for one field in IntelMQ 3.0 (scheduled for June 2021)[0]. The idea is described below, including various advantages and disadvantages. We appreciate your input, opinion and analysis of further implications on this idea. We plan to evaluate the feedback that emerged in two weeks. [0] https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectu… "The new IDF shall support (sorted) lists of IPs, domains, taxonomy categories, etc. By convention the most relevant item in such a list MUST be the first item in the sorted list." https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectu…: n6-system "Since the new IDF shall support multiple values, mapping to n6 should be rather easy." ## Use-cases ### Network information IntelMQ's format currently allows for *exactly one* value per field. For example, every event can have *one* `source.ip` and *one* `source.fqdn`. In some use-cases, multiple values can be useful, for example when querying DNS information. One domain (`source.fqdn`) can point to multiple IP addresses (`source.ip`). The other way round, multiple domains point to the same IP address is also very common. The use-case first appeared was that one IP address can be part of multiple Autonomous systems (`source.asn`).[1][2][3] See the examples below in section Format. [1]: "Multiple ASNs/networks per IP? #543" https://github.com/certtools/intelmq/issues/543 [2]: "BOT: DNS lookup #373" https://github.com/certtools/intelmq/issues/373 [3]: "reverse DNS: Only first record is used "https://github.com/certtools/intelmq/issues/877 ### Classification Another use-case is to use multiple classifications.[5] For example, if a website was hacked and used for a phishing page, it can be assigned two classifications: For the hacking: Taxonomy: information-content-security, type: unauthorised-modification-of-information For the phishing page: Taxonomy: fraud, type: phishing Another example are reachable networks services, which should not be accessible by the internet. Shadowserver provides a lot of this data. Open XDMCP instances are both DDoS amplifiers and Potentially unwanted accessible systems. Therefore both classifications apply: Taxonomy: vulnerable, type: ddos-amplifier Taxonomy: vulnerable, type: potentially-unwanted-accessible-system A list of all fields on the RSIT can be found in the RSIT repository[6] [5]: https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/… [6]: https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/… ## Format Some examples: {"source.ip": ["192.0.43.8"], "source.asn": [16876, 40528]} {"source.ip": ["10.0.0.1", "10.0.0.2"], "source.url": ["http://example.com/", "http://example.net"]} {"classification.taxonomy": ["information-content-security", "fraud"], "classification.type": ["unauthorised-modification-of-information", "phishing"], "source.url": ["http://example.com/"], "source.ip": ["10.0.0.1", "10.0.0.2"]} In the bots' code multiple values need to be taken car of. For example, instead of: ip_addr = event["source.ip"] # do stuff it is necessary to loop over the values: for ip_addr in event["source.ip"]: # do stuff This logic is required for *all* fields which can have multiple values, therefore nested loops may be necessary. Everything which processes IntelMQ data needs to be adapted, including data bases. See the "Disadvantages" section below. ### Optional back-conversion ("value-explosion") One variant/option of this IEP is to create a conversion layer from the new multi-value format to the old one-value format by creating multiple events with only one value per field. Using this conversion, compatibility with external components can be kept, while the advantages only exist inside the IntelMQ core (ie. the bots). Examples: {"source.ip": ["127.0.0.1", "127.0.0.2"], "source.fqdn": ["example.com"]} -> {"source.ip": "127.0.0.1", "source.fqdn": ["example.com"]}, {"source.ip": "127.0.0.2", "source.fqdn": ["example.com"]} {"source.ip": ["127.0.0.1", "127.0.0.2"], "source.fqdn": ["example.com", "example.org"]} -> {"source.ip": "127.0.0.1", "source.fqdn": "example.com"}, {"source.ip": "127.0.0.1", "source.fqdn": "example.org"}, {"source.ip": "127.0.0.2", "source.fqdn": "example.com"}, {"source.ip": "127.0.0.2", "source.fqdn": "example.org"} ### What will change? We'll change the behaviour of the current IntelMQ internal parsing process, i. e. you'll be able to add multiple IP addresses to on field, which will be handled as multiple events, but merged into one event. This will allow us to combine i. e. a domain with multiple IP addresses to one event. ### Advantages Supporting multiple values allows us to add multiple IP addresses to one event. As opposed to using multiple events with nearly similar data, the multiple-value approach reduces data duplication and has less overhead, while on the other hand the complexity increases. If multiple events would be used instead, related events would need to be linked together by other means (see section Alternative below). ### Disadvantages (breaking behaviour) The complexity in IntelMQ and all linked components increases without doubt. All components dealing with the IntelMQ-data need to be adapted to deal with multiple values. This includes all bots, but IntelMQ administrators need to adapt their configurations (e.g. filters, etc.) as well. Without the explosion-variant, all connected databases need to be adapted (e.g. PostgreSQL, SQLite, Elastic, MongoDB etc.) additionally and all software which is processing data from IntelMQ need to be adapted. PostgreSQL support arrays for columns, but the scheme conversion can be complex and resource-hungry. IntelMQ followed the KISS ("keep it simple, stupid")[4] principle from its beginning. It is disputable if multiple values breaks with this principle. [4]: https://en.wikipedia.org/wiki/KISS_principle ## Alternatives An alternative to using multiple values per field is to set unique identifiers (e.g. UUID) per event and let events with the same origin have the same "parent" identifier. This way, related events can be linked and compatibility is easier. Relating the events to each other requires extra steps although, but keeps the KISS principle. This approach will be described in IEP04. To solve the use-case of multiple classifications per event, the primary and most important classification can be used instead of multiple ones. A possible solution for the classification use-case above would be to some sort of tagging - in short "tags". I. e. { "source.ip": ["192.0.43.8"], "source.asn": [16876, 40528], "tags": ["ddos-amplifier", "info-disclosure", "mirai-botnet"] } ## Other IoC processing formats For reference, we describe the formats of other IoC-processing systems similar to IntelMQ. Both formats, IDEA and n6 do support multiple values in different kinds. If you know of other similar formats supporting multiple values, please speak up! ### "IDEA" The IDEA-format, used by CESNET-developed Warden, supports multiple values for some fields. But the data format structure differs clearly from IntelMQ's, as you can see in the example below. The classification is defined per address and network ranges are possible as addresses, what is not supported in IntelMQ. IDEA was designed from scratch to overcome disadvantages of Warden's previous data format. Example: "Source": [ { "Type": ["Phishing"], "IP4": ["192.168.0.2-192.168.0.5", "192.168.0.10/25"], "IP6": ["2001:0db8:0000:0000:0000:ff00:0042::/112"], "Hostname": ["example.com"], "URL": ["http://example.com/cgi-bin/killemall"], "Proto": ["tcp", "http"], "AttachHand": ["att1"], "Netname": ["ripe:IANA-CBLK-RESERVED1"] } ], "Target": [ { "Type": ["Backscatter", "OriginSpam"], "Email": ["innocent(a)example.com"], "Spoofed": true }, { "IP4": ["10.2.2.0/24"], "Anonymised": true } ] Upstream documentation: https://idea.cesnet.cz/en/index https://warden.cesnet.cz/en/index ### n6 In the n6 format, the addr field is a list of arrays with `ip`, `asn`, `cc` and `dir` fields. `addr` is similar to IntelMQ's `source` namespace, but the size of `addr` is much lower and the "direction" of the address is given by a field inside the addr item. Example: [{"ipv6": "abcd::1", "cc": "PL", "asn": 12345, "dir": "dst"}] Upstream documentation: https://n6sdk.readthedocs.io/en/latest/tutorial.html#field-class-addressfie… https://n6sdk.readthedocs.io/en/latest/tutorial.html#field-class-extendedad… -- // Sebastian Waldbauer <waldbauer(a)cert.at> - T: +43 1 5056416 7202 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

4 7

regarding IEP03
by L. Aaron Kaplan 22 Apr '21

22 Apr '21

Hi everyone, I wanted to send back a better write-up of my stance on IEP03 ("multiple values" in the IntelMQ internal data format). Alas, I was quite busy and I am sprinting to push out some code for our deadline at work. However, let me summarise it: I think the IEP03 is very well written, thank you a lot for this! Thinking this through was important and I think Sebastian Waldbauer did a great job. Reading it, I realised that my initial proposal of having multiple values is really breaking the KISS principle of IntelMQ in a bad way. Worse than I had thought. So, I am thinking of retracting the proposal. However, .... https://github.com/certtools/ieps/tree/main/003#alternatives has a good core in it. If we have multiple values, instead of doing the n x m complexity explosion, we link different events (JSON rows) together via UUIDs this gives us what we need: * UUIDs help with deduplication! That's important when linking IntelMQ instances! * lower complexity / keep the KISS principle * consumers can ignore the UUID-linking if it's not relevant for them (f.ex enrichment processes/bots) * we can still represent linked events. I would like to add one little but important thing for the UUID linking idea: add a "link-type". Examples for link-types: * parent-child event * grouping types (all of these events belong to the same report) etc. With this triplet information , we are close to RDF (left-side, type, right-side) and thus we can (future-proof) represent any type of relation. A list of valid types needs to be documented in the IDF format page of course. So, I think with that, we can go ahead. Thanks, a. PS: and sorry that my feedback came a bit late, as said - code sprints.

2 1

IntelMQ -> feed MISP
by Bernhard Reiter 16 Apr '21

16 Apr '21

Hi Drupad, Am Donnerstag 15 April 2021 17:56:18 schrieb Soni, Drupad: > Also I want your help in setting up misp output feed as below. your image showed that you want all events to go into MISP as well using https://github.com/certtools/intelmq/blob/develop/intelmq/bots/outputs/misp… > Feed is working fine adding feed in misp doesn't show any feeds there. > I am not sure what is the gap here. Me neither, my experience with MISP is limited, there are many functions and ways to manually use MISP. When following the documentation, I could make the api work, but I've not tested the feed. One possibility you have is to ask the MISP people about how to further analyse the situation (Please give them all the details.) Best Regards, Bernhard -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

3 2

IntelMQ Data format (IEP03 & IEP04) hackathon 2021-04-20 10h UTC (12h CEST)
by Sebastian Wagner 15 Apr '21

15 Apr '21

Dear community, The selected date for the IEP03 & IEP04 (data format) hackathon is: Thursday 22th April, 10h-12h UTC / 12h-14h CEST Location: https://jitsi.cert.ee/GreenGalaxiesDanceSeldom Unfortunately, there was no suitable date for all people who entered their interest in the poll. The main goal of the hackathon is to discuss the IEP03 and IEP04 proposal on changes to the data format and get a decision how to proceed. Here are the direct links to the propsals: IEP03 Multiple Values: https://lists.cert.at/pipermail/intelmq-dev/2021-March/000425.html IEP04 Meta Information and Data Exchange: https://lists.cert.at/pipermail/intelmq-dev/2021-March/000426.html see you next week, Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

A spooky SSLSocket making tests fail?
by Mika Silander 14 Apr '21

14 Apr '21

Hi, Something fails in my unit tests for my output bot: I have a test following this outline: def test_ok_events(self): for event in [ firstevent, secondevent ]: self.input_message = event self.run_bot(parameters={ 'logging_level': 'DEBUG', ... some other bot specific parameters here ... }, iterations=1, allowed_error_count=0, allowed_warning_count=0) Above firstevent and secondevent are JSON structures that conform to the Event class. The first event is handled ok, but the second seems to get handled but in the end the run_bot method in /opt/dev_intelmq/intelmq/lib/test.py claims things are not ok. Next, I raise allowed_warning_count to 4 (other bot parameters remain untouched) and the same thing happens although the bot progresses a bit further down the run_bot method and then prints the following traceback message in my IDE's console: --- Failure Traceback (most recent call last): File "/opt/dev_intelmq/intelmq/tests/bots/outputs/mybot/test_output.py", line ABC, in test_ok_events }, allowed_warning_count=4) File "/opt/dev_intelmq/intelmq/lib/test.py, line 356, in run_bot ''.format(fields['message'])) AssertionError: False is not true : Logline "/opt/dev_intelmq/intelmq/lib/test.py:232: ResourceWarning: unclosed <ssl.SSLSocket ... >" does not end with .? or !. --- I guess the unclosed socket is due to my bot using Sessions from the requests module to send data to a back end service and this is not taken down between the two invocations of the run_bot method. Any ideas how I should modify the test to take this session down before the second event is tested? Or, maybe better still, how should I modify my bot to close this session down cleanly at exit? By overriding a suitable stop or shutdown method and closing the session inside this overriding method? Thanks again, Mika

1 1

IntelMQ Data format Hackathon (IEP03 & IEP04): scheduling
by Sebastian Wagner 12 Apr '21

12 Apr '21

Dear community, Thank you for your input and interest on the proposed changes for IntelMQ's internal data format IEP03 and IEP04. To get the discussion forward and hopefully to get some conclusions, we propose to do a small *hackathon*. We think that this would be a great opportunity as well to bring together the IntelMQ community. If you are interested, please add your availability here until Tuesday 20th April: https://www.termino.gv.at/meet/en/p/219061bf603a2dbb91efe8af61ee4718-64494 The poll offers several 2h-slots in the week 19th-23rd April, we plan to select a date on Wednesday 21th April and announce it here, together with a location. kind regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 2

Intelmq-manager unable to save configuration
by Mika Silander 08 Apr '21

08 Apr '21

Hi again, Our installation has progressed to the point where we have defined our own set of: /etc/intelmq/runtime.conf /etc/intelmq/pipeline.conf However, when having opened intelmq-manager's Configuration tab and pressing "Save configuration" there, it says "Could not write file". How can I debug this? I don't seem to find a suitable log/error file in order to find the cause of the problem. As always, all help is appreciated, Mika (intelmq 2.3.1 + intelmq-manager + intelmq-api on Ubuntu 20.04 LTS)

2 2

intelmq-manager authentication issue on Ubuntu 20.04 LTS
by Mika Silander 08 Apr '21

08 Apr '21

Hi, Seems I have a configuration error somewhere in intelmq-manager's setup. I have configured basic authentication for this service, apache asks for the account and password and shows the start view of intelmq-manager. The start view says "Not logged in" and provides a login button below. If I press the button and use the same credentials, it still says "not logged in". Any hints? intelmq-manager, intelmq and intelmq-api were all installed from package repos for Ubuntu 20.04 LTS. Cheers, Mika

3 10

Generating the deb packages for intelmq 2.3.1
by Mika Silander 07 Apr '21

07 Apr '21

Hi, A quick one: how am I supposed to build the distribution packages of intelmq 2.3.1? I cloned the intelmq github repo, then: git checkout maintenance cd debian but then running e.g. "debuild -us -uc" first stumbles on python3-cerberus dependency missing and there's no package in Ubuntu 18.04 LTS for installing it. I can remove this dependency (it was only relevant for documenting, right?) but after that other errors surface like "no upstream tarball found ...". Any hints as to where to look for solutions? Br, Mika

3 3

IntelMQ & API & Manager 2.3.1
by Sebastian Wagner 25 Mar '21

25 Mar '21

Dear community, This IntelMQ version did not gain any major features and is a sole product maintenance release. It covers various minor error corrections in the IntelMQ software, and usability enhancements for the new API. Some notable changes are listed below. Thanks again to all contributors! The installation instructions: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade instructions: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html All packages have been published to PyPI. The deb/rpm-repositories and dockerhub are following on Friday. I'm trying to summarize the changes here, the full changelogs are linked below. The Cymru CAP Parser has been adapted to the new format for events of the category "bruteforce" (PR#1795 by Sebastian Wagner, CERT.at, fixes 1794). The Shodan Parser now supports nested conversions, improved protocol detection and has a majorly extended parser mapping (PR#1821 by Mikk Markus Möll, CERT.EE). The feed documentation gained a missing description for the Shodan Country Stream (by Sebastian Wagner, CERT.at). On the documentation front, the ecosystem document received revised sections on intelmq-cb-mailgen and fody (PR#1792 by Bernhard Reiter, Intevation). A new section in the documentation summarizes hardware requirements (PR#1811 by Sebastian Wagner, CERT.at). Minor enhancements and adaptions in the tests. # IntelMQ API-related changes The IntelMQ API documentation now has more details on the required write permission for the session database file (PR#1798 by Birger Schacht, CERT.at, fixes intelmq-api#23). See https://intelmq.readthedocs.io/en/maintenance/user/intelmq-api.html Session database permission errors: Catch the exception in the code and add a hint to check the permissions of both the file and the directory (PR#25 by Birger Schacht, CERT.at, fixes #23). The tool `intelmqsetup`, which is part of the installation routine for manual IntelMQ installations, is now able to automatically create the required directory layout and file permissions for the IntelMQ API (PR#1787 by Sebastian Wagner, CERT.at, fixes #1783) and as well covers the webserver and sudoers configuration for IntelMQ API and IntelMQ Manager (PR#1805 by Sebastian Wagner, CERT.at, fixes #1803). # IntelMQ Manager-related changes The required authentication token in save-data request of the "Configuration" tab is now sent to the backend (PR#245 by Marcos Gonzalez, CNCSRD-DO, fixes #244). On two locations the link destinations have been fixed, they still linked to the old PHP-based backend URLs: "Clear Configuration" link destination (PR#249 by Sebastian Wagner, CERT.at, fixes #248) and link to configuration page of bot on the monitor page (PR#249 by Sebastian Wagner, CERT.at, fixes #248). # Full changelogs: IntelMQ: https://github.com/certtools/intelmq/releases/tag/2.3.1 IntelMQ API: https://github.com/certtools/intelmq-api/releases/tag/2.3.1 IntelMQ Manager: https://github.com/certtools/intelmq-manager/releases/tag/2.3.1 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

← Newer
1
...
5
6
7
8
9
10
11
...
24
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev