Hi,

On 8/09/2021 1:34 am, Sebastian Wagner wrote:

But don't we need to have a timestamp in the meta-data ?
I mean something like this;

{
    "format": "intelmq",
    "version": 1,
    "type": "event",
    "meta": {
        "intelmq:uuid": "<event-uuid-1>",
	"intelmq:uuid_org": "<org-uuid-1>",
	"intelmq:timestamp": "<creation time of this message>",  <== here
	:

Every IntelMQ message should already have a time.source field in the payload, so I'm not sure if it's necessary to have it in the metadata as well explicitly. And that overlaps with the next topic:

Not specifically for IntelMQ, but I tend to break an event message into at least three timestamps (but possibly more depending on event type):

* actual occurrence time of reported security event (time.source as I'd understand it)
* event package original creation time (the suggested meta.intelmq:timestamp here, which I'd possibly rename to meta.intelmq:creation_timestamp or similar)
* event package system ingestion time (time.observation?)

Thinking again about the timestamps: Wouldn't it be better to (also?) put the date of the message creation (and/or last updated) into the meta field? That would often be identical to the time.observation if the data is fetched from a feed*, and if the message is passed on from a different system - could also be from another organisation! - it is inherited.

On the other hand, I think that the meta-field should be as small as possible, because it adds a significant amount of relative data size to the event which has enormous impacts on memory consumption. I don't want to specifically advocate time-based UUID formats, but using them would add the message creation time for no overhead costs.

As an aside: To keep the memory consumption and processing-times low, we could also implement a minimal-meta-mode, which only adds the meta-fields when the message is passed on to other systems and exclusively keeps the parts of meta which are absolutely necessary for inter-bot communication (in particular the type and event UUID)

cheers
Sebastian

* not if the data comes from another system, e.g. mail or ticketing system, in that cases time.observation is the reception timestamp of the source

-- 
// Sebastian Wagner <wagner@cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg