Dear devs,
Thanks to Filip (CZ.NIC) IntelMQ comes now with an update mechanism for local lookup databases like TOR exit nodes, IP address to ASN ("ASN Lookup") and Maxmind GeoIP (IP address geolocation)[0]. Also, IntelMQ ships with update scripts for cron which are included in the deb/rpm packages as well.
Currently the update scripts are scheduled as follows[1]:
* TOR nodes: once per day. The database is very small. * Maxmind GeoIP: Once per week. Changes are scarce. * ASN Lookup: Every two hours. Big database, but the data is vital for subsequent routing of incidents.
I'd like to hear your opinion if the default values are ok to ship with 2.3.0, especially for the last one.
best regards, Sebastian
[0]: https://github.com/certtools/intelmq/pull/1524 [1]: https://github.com/certtools/intelmq/blob/24f2355d0c549021a713c938d1d69a5213...
Hi Sebastian,
thanks for bringing this up. I just wanted to add that the default values were chosen based on how often the datasource is updated. For the GeoLite datasouce it is once a week (on Tuesday). The ASN datasource is updated and published every two hours and it is a rather large database. For Tor nodes datasource I couldn't find how often it is updated, therefore updating the database once per day seemed reasonable.
Unsurprisingly I feel these default values are ok to ship with 2.3.0 as they always provide the user with the latest data. Also the user doesn't need to lookup how often the database needs to be updated to have the latest data. However if the user wants to update the databases with lower frequency they are free to do so.
Best regards, Filip
On 2/17/21 7:05 PM, Sebastian Wagner wrote:
Dear devs,
Thanks to Filip (CZ.NIC) IntelMQ comes now with an update mechanism for local lookup databases like TOR exit nodes, IP address to ASN ("ASN Lookup") and Maxmind GeoIP (IP address geolocation)[0]. Also, IntelMQ ships with update scripts for cron which are included in the deb/rpm packages as well.
Currently the update scripts are scheduled as follows[1]:
- TOR nodes: once per day. The database is very small.
- Maxmind GeoIP: Once per week. Changes are scarce.
- ASN Lookup: Every two hours. Big database, but the data is vital for
subsequent routing of incidents.
I'd like to hear your opinion if the default values are ok to ship with 2.3.0, especially for the last one.
best regards, Sebastian
IntelMQ-dev mailing list IntelMQ-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
Dear Filip,
On 2/18/21 10:52 AM, Filip Pokorný wrote:
thanks for bringing this up. I just wanted to add that the default values were chosen based on how often the datasource is updated. For the GeoLite datasouce it is once a week (on Tuesday). The ASN datasource is updated and published every two hours and it is a rather large database. For Tor nodes datasource I couldn't find how often it is updated, therefore updating the database once per day seemed reasonable.
Thanks, I added the reasons to the comments in the crontab file: https://github.com/certtools/intelmq/commit/da257c9f61e7dac1106ff72b7eda4e43...
I was thinking about the additional load on the server-side as well. But that only affects routeviews.org (University of Oregon) and I assume they can handle it.
Btw, I added a database update functionality for the domain suffix expert today: https://github.com/certtools/intelmq/commit/1b987c0573d7eb29a3a18d3fba40c452...
best regards Sebastian