Dear list,
in pull request #944 (netlab 360 enh [0]) by navtej an issue came up which can't be solved trivially:
The feed Netlab 360 DGA[1] - which is already included in intelmq - provides a validity time frame for each domain. Most of those (~90%) end in 2030 while the start date is the current day at 00:00. So both start and end time are artificial. And the source claims the event is valid in the future, which is a very odd. And does it actually make sense to forward this kind of information? Also, we can't really handle this time information using the current harmonization.
One idea would be to set time.source to time.observation if the time.source is in the future. So time.source <= time.observation does always apply.
What do you think?
Sebastian
[0]: https://github.com/certtools/intelmq/pull/944 [1]: http://data.netlab.360.com/feeds/dga/dga.txt - attention, quite big! The domains at the beginning have a very near end date.
Any thoughts on this?
On 04/21/2017 03:42 PM, Sebastian Wagner wrote:
Dear list,
in pull request #944 (netlab 360 enh [0]) by navtej an issue came up which can't be solved trivially:
The feed Netlab 360 DGA[1] - which is already included in intelmq - provides a validity time frame for each domain. Most of those (~90%) end in 2030 while the start date is the current day at 00:00. So both start and end time are artificial. And the source claims the event is valid in the future, which is a very odd. And does it actually make sense to forward this kind of information? Also, we can't really handle this time information using the current harmonization.
One idea would be to set time.source to time.observation if the time.source is in the future. So time.source <= time.observation does always apply.
What do you think?
Sebastian
[1]: http://data.netlab.360.com/feeds/dga/dga.txt - attention, quite big! The domains at the beginning have a very near end date.
Intelmq-dev mailing list Intelmq-dev@lists.cert.at http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
I appreciate your comments on this topic. This problem is still unresolved.
On 06/19/2017 02:13 PM, Sebastian Wagner wrote:
Any thoughts on this?
On 04/21/2017 03:42 PM, Sebastian Wagner wrote:
Dear list,
in pull request #944 (netlab 360 enh [0]) by navtej an issue came up which can't be solved trivially:
The feed Netlab 360 DGA[1] - which is already included in intelmq - provides a validity time frame for each domain. Most of those (~90%) end in 2030 while the start date is the current day at 00:00. So both start and end time are artificial. And the source claims the event is valid in the future, which is a very odd. And does it actually make sense to forward this kind of information? Also, we can't really handle this time information using the current harmonization.
One idea would be to set time.source to time.observation if the time.source is in the future. So time.source <= time.observation does always apply.
What do you think?
Sebastian
[1]: http://data.netlab.360.com/feeds/dga/dga.txt - attention, quite big! The domains at the beginning have a very near end date.
Intelmq-dev mailing list Intelmq-dev@lists.cert.at http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
-- // Sebastian Wagner wagner@cert.at - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
Intelmq-dev mailing list Intelmq-dev@lists.cert.at http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
Hi,
Thank you Sebastian to raise this to ML.
My perspective: 100% agree with your suggestion. This case is the same as a stream feed like AnubisNetworks. AnubisNetworks sends a event to the stream per each bot (infected machine) connection to C&C and the timestamp is the connection time. If we assume that netlab306 bot sees (like you mention) a future timestamp as timestamp.now(), we just need to handle that events like we do with AnubisNetworks parser.
TL;DR: 100% agree, lets fix it!
Cheers
-------- Original Message -------- Subject: Re: [Intelmq-dev] handling of time frames Local Time: August 14, 2017 2:23 PM UTC Time: August 14, 2017 1:23 PM From: wagner@cert.at To: intelmq-dev@lists.cert.at intelmq-dev@lists.cert.at
I appreciate your comments on this topic. This problem is still unresolved.
On 06/19/2017 02:13 PM, Sebastian Wagner wrote:
Any thoughts on this?
On 04/21/2017 03:42 PM, Sebastian Wagner wrote:
Dear list,
in pull request #944 (netlab 360 enh [0]) by navtej an issue came up which can't be solved trivially:
The feed Netlab 360 DGA[1] - which is already included in intelmq - provides a validity time frame for each domain. Most of those (~90%) end in 2030 while the start date is the current day at 00:00. So both start and end time are artificial. And the source claims the event is valid in the future, which is a very odd. And does it actually make sense to forward this kind of information? Also, we can't really handle this time information using the current harmonization.
One idea would be to set time.source to time.observation if the time.source is in the future. So time.source <= time.observation does always apply.
What do you think?
Sebastian
- attention, quite
big! The domains at the beginning have a very near end date.
Intelmq-dev mailing list Intelmq-dev@lists.cert.at
-- // Sebastian Wagner [wagner@cert.at](mailto:wagner@cert.at)
- T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
Intelmq-dev mailing list Intelmq-dev@lists.cert.at
-- // Sebastian Wagner [wagner@cert.at](mailto:wagner@cert.at)
- T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg