[Intelmq-dev] contains-operator and hierarchies of messages

[Sebastian Wagner]

Hi,

All I wanted to have in the first place is an easy way to find out if an
event has any extra.* field. I.e. in python: 'extra' in event.
But this does not work, because there is no such field, only extra.foo.
So, what about changing the __contains__ method of the Message class? It
could check for sub-fields if necessary.

Using the (wrong) algorithm to check if some existing key starts with
(in this example) `extra` it turns out this also leads to the convenient
situation that you now can't add e.g. `extra.error.message` if
`extra.error` was previously defined.
And: the alienvault otx parser actually did this with extra.pulse

This is good, but also prevents adding `extra.errormessage` if
`extra.error` exists. The correct algorithm is to check if some field
starts with `extra.`. But using this assumption does not prevent
hierarchy conflicts.

IMO, we want both:
1) a simple way to check if any field of a group is (not) present
2) prevention to add keys which conflict with others in the hierarchy,
i.e. `extra.error.message` must not be added if `extra.error` is present.

We could adapt the contains-operator that e.g. a search for `extra.`
means: "Is there any field in the extra.* domain?". It's an invalid key
name anyway[1].
And for adding new fields: check if there is some value in a higher
hierarchy, if so, fail. Overwriting is not allowed in this case.

Any thoughts on this?

Sebastian

P.S.: In the long rung we can probably save the fields hierarchically
internally too, would avoid some of these shortcomings.
We also had problems with the hierarchy of malware.hash in the past:
https://github.com/certtools/intelmq/issues/732

[1]: actually it is allowed currently, but it shouldn't and does not
make sense, see https://github.com/certtools/intelmq/issues/1104

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20171010/b3c9706a/attachment.sig>