[IntelMQ-dev] IEP04: The choice of the UUID-format

[Sebastian Wagner]

Hi,

On 9/8/21 7:13 AM, Chris Horsley wrote:
> On 8/09/2021 1:34 am, Sebastian Wagner wrote:
>>> But don't we need to have a timestamp in the meta-data ?
>>> I mean something like this;
>>>
>>> {
>>>     "format": "intelmq",
>>>     "version": 1,
>>>     "type": "event",
>>>     "meta": {
>>>         "intelmq:uuid": "<event-uuid-1>",
>>> 	"intelmq:uuid_org": "<org-uuid-1>",
>>> 	"intelmq:timestamp": "<creation time of this message>",  <== here
>>> 	:
>> Every IntelMQ message should already have a /time.source/ field in
>> the payload, so I'm not sure if it's necessary to have it in the
>> metadata as well explicitly. And that overlaps with the next topic:
>
> Not specifically for IntelMQ, but I tend to break an event message
> into at least three timestamps (but possibly more depending on event
> type):
>
> * actual occurrence time of reported security event (time.source as
> I'd understand it)
> * event package original creation time (the suggested
> meta.intelmq:timestamp here, which I'd possibly rename to
> meta.intelmq:creation_timestamp or similar)
> * event package system ingestion time (time.observation?)
>
Thinking again about the timestamps: Wouldn't it be better to (also?)
put the date of the /message creation/ (and/or last updated) into the
meta field? That would often be identical to the time.observation if the
data is fetched from a feed*, and if the message is passed on from a
different system - could also be from another organisation! - it is
inherited.

On the other hand, I think that the meta-field should be as small as
possible, because it adds a significant amount of relative data size to
the event which has /enormous/ impacts on memory consumption. I don't
want to specifically advocate time-based UUID formats, but using them
would add the message creation time for no overhead costs.

As an aside: To keep the memory consumption and processing-times low, we
could also implement a minimal-meta-mode, which only adds the
meta-fields when the message is passed on to other systems and
exclusively keeps the parts of meta which are absolutely necessary for
inter-bot communication (in particular the type and event UUID)

cheers
Sebastian

* not if the data comes from another system, e.g. mail or ticketing
system, in that cases time.observation is the reception timestamp of the
source

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210916/7759512d/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210916/7759512d/attachment.sig>