Hi,
All I wanted to have in the first place is an easy way to find out if an event has any extra.* field. I.e. in python: 'extra' in event. But this does not work, because there is no such field, only extra.foo. So, what about changing the __contains__ method of the Message class? It could check for sub-fields if necessary.
Using the (wrong) algorithm to check if some existing key starts with (in this example) `extra` it turns out this also leads to the convenient situation that you now can't add e.g. `extra.error.message` if `extra.error` was previously defined. And: the alienvault otx parser actually did this with extra.pulse
This is good, but also prevents adding `extra.errormessage` if `extra.error` exists. The correct algorithm is to check if some field starts with `extra.`. But using this assumption does not prevent hierarchy conflicts.
IMO, we want both: 1) a simple way to check if any field of a group is (not) present 2) prevention to add keys which conflict with others in the hierarchy, i.e. `extra.error.message` must not be added if `extra.error` is present.
We could adapt the contains-operator that e.g. a search for `extra.` means: "Is there any field in the extra.* domain?". It's an invalid key name anyway[1]. And for adding new fields: check if there is some value in a higher hierarchy, if so, fail. Overwriting is not allowed in this case.
Any thoughts on this?
Sebastian
P.S.: In the long rung we can probably save the fields hierarchically internally too, would avoid some of these shortcomings. We also had problems with the hierarchy of malware.hash in the past: https://github.com/certtools/intelmq/issues/732
[1]: actually it is allowed currently, but it shouldn't and does not make sense, see https://github.com/certtools/intelmq/issues/1104