Hi,
IntelMQ uses dictionaries to represent messages (in python, json etc.). We use a flat and unnested structure, which is one of the first design goals made in the very beginning of IntelMQ AFAIK. E.g. we have field names like "source.ip"
But there's also another possible representation, which is implemented in IntelMQ: nested structures. E.g.: flat: {"classification.type": "unknown", "source.asn": 456, "source.ip": "127.0.0.1"} nested: {"classification": {"type": "unknown"}, "source": {"ip": "127.0.0.1", "asn": 456}}
The first is used everywhere except: The messages to_json and to_dict methods, which use the nested format by default. These methods are used in these output bots: file, xmpp, restapi, mongodb, intelmqmailer
I think, that this is a wrong default. The default should be something which can be directly interpreted by IntelMQ: the flat structure.
Proposal: make flat default and nested optional (for the function and the bots)
Sebastian
On 09 Sep 2016, at 12:55, Sebastian Wagner wagner@cert.at wrote:
Hi,
IntelMQ uses dictionaries to represent messages (in python, json etc.). We use a flat and unnested structure, which is one of the first design goals made in the very beginning of IntelMQ AFAIK. E.g. we have field names like "source.ip"
But there's also another possible representation, which is implemented in IntelMQ: nested structures. E.g.: flat: {"classification.type": "unknown", "source.asn": 456, "source.ip": "127.0.0.1"} nested: {"classification": {"type": "unknown"}, "source": {"ip": "127.0.0.1", "asn": 456}}
The first is used everywhere except: The messages to_json and to_dict methods, which use the nested format by default. These methods are used in these output bots: file, xmpp, restapi, mongodb, intelmqmailer
I think, that this is a wrong default. The default should be something which can be directly interpreted by IntelMQ: the flat structure.
I fully agree.
Proposal: make flat default and nested optional (for the function and the bots)
+1
Why keep the nested optional?
Am Freitag 09 September 2016 12:55:17 schrieb Sebastian Wagner:
I think, that this is a wrong default. The default should be something which can be directly interpreted by IntelMQ: the flat structure.
From my perspective: I agree that there should be one format used everywhere in intelmq. flat sound good as this format, but I haven't checked. Flat may not cut it, if we have several entries or possible name collisions.
Bernhard