[IntelMQ-dev] [IntelMQ-users] IEP03: IntelMQ Data Format - Multiple Values

[Sebastian Waldbauer]

Hey List & Hey Pavel!

Thanks for your awesome feedback on the IEP03 :)

On 3/31/21 5:27 PM, Pavel Kácha wrote:

> Hello,
>
>     again a few notes based on Idea experience. :)
>
>> From: Sebastian Waldbauer <waldbauer at cert.at>, Date: bře 30, 2021
>>
>> ## Use-cases
>> ### Network information
>> IntelMQ's format currently allows for *exactly one* value per field. For
>> example, every event can have *one* `source.ip` and *one* `source.fqdn`. In
>> some use-cases, multiple values can be useful, for example when querying DNS
>> information. One domain (`source.fqdn`) can point to multiple IP addresses
>> (`source.ip`). The other way round, multiple domains point to the same 
IP
>> address is also very common. The use-case first appeared was that one IP
>> address can be part of multiple Autonomous systems (`source.asn`).[1][2][3]
>     Do all source.fgdn have to correspond with source.ip and source.asn?
>
>     Consider:
>
>     source.ip: [78.128.216.141, 2001:718:ff05:202::141, 78.128.211.46, 2001:718:1:1f:50:56ff:feee:46]
>     source.fqdn: [idea.cesnet.cz, www.cesnet.cz, cesnet.cz]
>
>     Relation of which IPs correspond to which FQDNs is lost here.
>
>     If it's not to be lost, you need another level of nesting/indirection -
> or you can _require_ for all fields to correspond, split events accordingly
> where it's not the case and implement both this and also variation of IEP04
> (where you may face cartesian explosion problem I mentioned in reaction
> there). Like something akin to:
>
> Event 1
>     meta.uuid.current: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
>     source.ip: [78.128.216.141, 2001:718:ff05:202::141]
>     source.fqdn: [idea.cesnet.cz]
>
> Event 2
>     meta.uuid.current: bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb
>     meta.uuid.parent: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
>     source.ip: [78.128.211.46, 2001:718:1:1f:50:56ff:feee:46]
>     source.fqdn: [www.cesnet.cz, cesnet.cz]

IMHO, I wouldn't use multiple values in source fields as intelmq data 
gets more complex and will break the KISS principle. Using the parent 
uuid would solve this problem of nesting, as you can use uuids to 
connect similar events to each other. I'd propose to use multiple values 
in fields where it doenst get too complex like `tags`. Tags can be used 
to add specified tags like campaigns.

>> ### Classification
> ...
>> ## Format
>> {"classification.taxonomy": ["information-content-security", "fraud"],
>> "classification.type": ["unauthorised-modification-of-information",
>> "phishing"]
>     I believe (feel free to correct me) that RSIT does not preclude usage of
> just first level category in cases where second level is ambiguous or
> unknown, so in two array format you could solve it for example like:
>
>     {
>        "classification.taxonomy": ["information-content-security", "fraud"],
>        "classification.type": [null, "phishing"]
>
>     In Idea we went for "merged" field, here it might look like:
>
>     classification: [
>        "information-content-security.unauthorised-modification-of-information",
>        "fraud.phishing"
>     ]
>
>     or considering missing second level:
>
>     classification: [
>        "information-content-security",
>        "fraud.phishing"
>     ]

Agree with the "tagging" style, as it can contain a lot of information & 
can be set per event.

I'd suggest to discuss this in an hackathon, so there might be more 
input & use-case on that one.

-- 
// Sebastian Waldbauer <waldbauer at cert.at> - T: +43 1 5056416 7202
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0x1917C49891C4BF92.asc
Type: application/pgp-keys
Size: 3909 bytes
Desc: OpenPGP public key
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210406/edc554e2/attachment.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210406/edc554e2/attachment.sig>