Hi,
On 9/8/21 7:13 AM, Chris Horsley wrote:
On 8/09/2021 1:34 am, Sebastian Wagner wrote:
But don't we need to have a timestamp in the meta-data ? I mean something like this;
{ "format": "intelmq", "version": 1, "type": "event", "meta": { "intelmq:uuid": "<event-uuid-1>", "intelmq:uuid_org": "<org-uuid-1>", "intelmq:timestamp": "<creation time of this message>", <== here :
Every IntelMQ message should already have a /time.source/ field in the payload, so I'm not sure if it's necessary to have it in the metadata as well explicitly. And that overlaps with the next topic:
Not specifically for IntelMQ, but I tend to break an event message into at least three timestamps (but possibly more depending on event type):
- actual occurrence time of reported security event (time.source as
I'd understand it)
- event package original creation time (the suggested
meta.intelmq:timestamp here, which I'd possibly rename to meta.intelmq:creation_timestamp or similar)
- event package system ingestion time (time.observation?)
Thinking again about the timestamps: Wouldn't it be better to (also?) put the date of the /message creation/ (and/or last updated) into the meta field? That would often be identical to the time.observation if the data is fetched from a feed*, and if the message is passed on from a different system - could also be from another organisation! - it is inherited.
On the other hand, I think that the meta-field should be as small as possible, because it adds a significant amount of relative data size to the event which has /enormous/ impacts on memory consumption. I don't want to specifically advocate time-based UUID formats, but using them would add the message creation time for no overhead costs.
As an aside: To keep the memory consumption and processing-times low, we could also implement a minimal-meta-mode, which only adds the meta-fields when the message is passed on to other systems and exclusively keeps the parts of meta which are absolutely necessary for inter-bot communication (in particular the type and event UUID)
cheers Sebastian
* not if the data comes from another system, e.g. mail or ticketing system, in that cases time.observation is the reception timestamp of the source