<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,<br>
</p>
<div class="moz-cite-prefix">On 9/8/21 7:13 AM, Chris Horsley wrote:<br>
</div>
<blockquote type="cite"
cite="mid:c3daa645-13e7-f344-c590-3b2b66115840@csirtfoundry.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div class="moz-cite-prefix">On 8/09/2021 1:34 am, Sebastian
Wagner wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1d2f5a23-5ef6-49cd-9f90-d8cdae6f3009@cert.at">
<blockquote type="cite"
cite="mid:20210907.094028.1714939875838089049.moto@kawasaki3.org">
<pre class="moz-quote-pre" wrap="">But don't we need to have a timestamp in the meta-data ?
I mean something like this;
{
"format": "intelmq",
"version": 1,
"type": "event",
"meta": {
"intelmq:uuid": "<event-uuid-1>",
"intelmq:uuid_org": "<org-uuid-1>",
"intelmq:timestamp": "<creation time of this message>", <== here
:</pre>
</blockquote>
Every IntelMQ message should already have a <i>time.source</i>
field in the payload, so I'm not sure if it's necessary to have
it in the metadata as well explicitly. And that overlaps with
the next topic:<br>
</blockquote>
<p>Not specifically for IntelMQ, but I tend to break an event
message into at least three timestamps (but possibly more
depending on event type):<br>
</p>
<p>* actual occurrence time of reported security event
(time.source as I'd understand it)<br>
* event package original creation time (the suggested
meta.intelmq:timestamp here, which I'd possibly rename to
meta.intelmq:creation_timestamp or similar)<br>
* event package system ingestion time (time.observation?)</p>
</blockquote>
<p>Thinking again about the timestamps: Wouldn't it be better to
(also?) put the date of the <i>message creation</i> (and/or last
updated) into the meta field? That would often be identical to the
time.observation if the data is fetched from a feed*, and if the
message is passed on from a different system - could also be from
another organisation! - it is inherited.<br>
</p>
<p>On the other hand, I think that the meta-field should be as small
as possible, because it adds a significant amount of relative data
size to the event which has <i>enormous</i> impacts on memory
consumption. I don't want to specifically advocate time-based UUID
formats, but using them would add the message creation time for no
overhead costs.</p>
<p>As an aside: To keep the memory consumption and processing-times
low, we could also implement a minimal-meta-mode, which only adds
the meta-fields when the message is passed on to other systems and
exclusively keeps the parts of meta which are absolutely necessary
for inter-bot communication (in particular the type and event
UUID)</p>
<p>cheers<br>
Sebastian<br>
</p>
<p>* not if the data comes from another system, e.g. mail or
ticketing system, in that cases time.observation is the reception
timestamp of the source<br>
</p>
<pre class="moz-signature" cols="72">--
// Sebastian Wagner <a class="moz-txt-link-rfc2396E" href="mailto:wagner@cert.at"><wagner@cert.at></a> - T: +43 676 898 298 7201
// CERT Austria - <a class="moz-txt-link-freetext" href="https://www.cert.at/">https://www.cert.at/</a>
// Eine Initiative der nic.at GmbH - <a class="moz-txt-link-freetext" href="https://www.nic.at/">https://www.nic.at/</a>
// Firmenbuchnummer 172568b, LG Salzburg</pre>
</body>
</html>