Hi,
Is there a way of updating outputs such as databases when IOCs expire? Don't want to spend time re-inventing the wheel.
Thanks.
Hi,
How do IOCs expire?
Sebastian
On 12/09/2018 03.22, joanna@scate.tech wrote:
Hi,
Is there a way of updating outputs such as databases when IOCs expire? Don't want to spend time re-inventing the wheel.
Thanks.
On 12 Sep 2018, at 10:23, Sebastian Wagner wagner@cert.at wrote:
Hi,
How do IOCs expire?
Well I can imagine a scenario where you fetch for example IP addresses via intelMQ from a blacklist and you want to expire them at some point (to be defined by the blacklist and/or the user of intelmq).
So, I do see a use-case here.
Sebastian
On 12/09/2018 03.22, joanna@scate.tech wrote:
Hi,
Is there a way of updating outputs such as databases when IOCs expire? Don't want to spend time re-inventing the wheel.
Thanks.
-- // Sebastian Wagner wagner@cert.at - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
-- Listen-Einstellungen: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
-- // L. Aaron Kaplan kaplan@cert.at - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
Expiring indicators needs some careful thought in my experience.
There are some threat intelligence platforms which have a well-integrated way to do this using a relevancy half-life time per feed or indicator. If the half-life you set for a feed is one month, it starts at 100% relevancy, after one month it's 50%, after two months it's 25% etc.
Over time, indicators get a less relevant score, but are not deleted by default. Sometimes, you might want to do a search for all indicators over all time (e.g. you're coming up with the complete history for an ASN / registrar / URL pattern). Other times, you might want to only export IP addresses with a time relevancy score over 70% to your network appliance to keep the list small and useful.
The trick is that different types of indicators from different feeds probably need different expiry windows. There might also be different use cases for the same data where you want to filter based on timeliness / relevancy.
Chris
On 12 Sep 2018, at 8:15 pm, L. Aaron Kaplan kaplan@cert.at wrote:
Signed PGP part
On 12 Sep 2018, at 10:23, Sebastian Wagner wagner@cert.at wrote:
Hi,
How do IOCs expire?
Well I can imagine a scenario where you fetch for example IP addresses via intelMQ from a blacklist and you want to expire them at some point (to be defined by the blacklist and/or the user of intelmq).
So, I do see a use-case here.
Sebastian
On 12/09/2018 03.22, joanna@scate.tech wrote:
Hi,
Is there a way of updating outputs such as databases when IOCs expire? Don't want to spend time re-inventing the wheel.
Thanks.
-- // Sebastian Wagner wagner@cert.at - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
-- Listen-Einstellungen: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
-- // L. Aaron Kaplan kaplan@cert.at - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
On 14 Sep 2018, at 03:28, Chris Horsley chris.horsley@csirtfoundry.com wrote:
Expiring indicators needs some careful thought in my experience.
There are some threat intelligence platforms which have a well-integrated way to do this using a relevancy half-life time per feed or indicator. If the half-life you set for a feed is one month, it starts at 100% relevancy, after one month it's 50%, after two months it's 25% etc.
Over time, indicators get a less relevant score, but are not deleted by default. Sometimes, you might want to do a search for all indicators over all time (e.g. you're coming up with the complete history for an ASN / registrar / URL pattern). Other times, you might want to only export IP addresses with a time relevancy score over 70% to your network appliance to keep the list small and useful.
The trick is that different types of indicators from different feeds probably need different expiry windows. There might also be different use cases for the same data where you want to filter based on timeliness / relevancy.
Totally agree that this would be a smart way to do it. Probably even a very generic way (again, that depends on your use case). But I am not sure if its in scope of IntelMQ since that just fetches data, processes, filters & enriches it and sends it somewhere. Anyway, I wanted to say that I do understand the wish for expiring (whatever your method might be) IoCs. But I am not sure if it fits the model and architecture of intelMQ.
Interesting discussion.
Joanna, what are you trying to achieve with expiring IoCs? (just to understand your use case better).
Chris
On 12 Sep 2018, at 8:15 pm, L. Aaron Kaplan kaplan@cert.at wrote:
Signed PGP part
On 12 Sep 2018, at 10:23, Sebastian Wagner wagner@cert.at wrote:
Hi,
How do IOCs expire?
Well I can imagine a scenario where you fetch for example IP addresses via intelMQ from a blacklist and you want to expire them at some point (to be defined by the blacklist and/or the user of intelmq).
So, I do see a use-case here.
Sebastian
On 12/09/2018 03.22, joanna@scate.tech wrote:
Hi,
Is there a way of updating outputs such as databases when IOCs expire? Don't want to spend time re-inventing the wheel.
Thanks.
-- // Sebastian Wagner wagner@cert.at - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
-- Listen-Einstellungen: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
-- // L. Aaron Kaplan kaplan@cert.at - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
-- Listen-Einstellungen: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
-- // L. Aaron Kaplan kaplan@cert.at - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
+1 to doing this outside of IntelMQ. Ageing out indicators will need continually scheduled queries into whatever datastore is being used, and given there are different ways to do the ageing as we've discussed, better if this is standalone I think.
On 14 Sep 2018, at 7:44 pm, L. Aaron Kaplan kaplan@cert.at wrote:
On 14 Sep 2018, at 03:28, Chris Horsley chris.horsley@csirtfoundry.com wrote:
Expiring indicators needs some careful thought in my experience.
There are some threat intelligence platforms which have a well-integrated way to do this using a relevancy half-life time per feed or indicator. If the half-life you set for a feed is one month, it starts at 100% relevancy, after one month it's 50%, after two months it's 25% etc.
Over time, indicators get a less relevant score, but are not deleted by default. Sometimes, you might want to do a search for all indicators over all time (e.g. you're coming up with the complete history for an ASN / registrar / URL pattern). Other times, you might want to only export IP addresses with a time relevancy score over 70% to your network appliance to keep the list small and useful.
The trick is that different types of indicators from different feeds probably need different expiry windows. There might also be different use cases for the same data where you want to filter based on timeliness / relevancy.
Totally agree that this would be a smart way to do it. Probably even a very generic way (again, that depends on your use case). But I am not sure if its in scope of IntelMQ since that just fetches data, processes, filters & enriches it and sends it somewhere. Anyway, I wanted to say that I do understand the wish for expiring (whatever your method might be) IoCs. But I am not sure if it fits the model and architecture of intelMQ.
Interesting discussion.
Joanna, what are you trying to achieve with expiring IoCs? (just to understand your use case better).
Chris
On 12 Sep 2018, at 8:15 pm, L. Aaron Kaplan kaplan@cert.at wrote:
Signed PGP part
On 12 Sep 2018, at 10:23, Sebastian Wagner wagner@cert.at wrote:
Hi,
How do IOCs expire?
Well I can imagine a scenario where you fetch for example IP addresses via intelMQ from a blacklist and you want to expire them at some point (to be defined by the blacklist and/or the user of intelmq).
So, I do see a use-case here.
Sebastian
On 12/09/2018 03.22, joanna@scate.tech wrote:
Hi,
Is there a way of updating outputs such as databases when IOCs expire? Don't want to spend time re-inventing the wheel.
Thanks.
-- // Sebastian Wagner wagner@cert.at - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
-- Listen-Einstellungen: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
-- // L. Aaron Kaplan kaplan@cert.at - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
-- Listen-Einstellungen: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
-- // L. Aaron Kaplan kaplan@cert.at - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg