I noticed the geoip expert bot crashed on our IntelMQ instance.
In the logs I found:
# tail /opt/intelmq/var/log/maxmind-geoip-expert.log
maxminddb.errors.InvalidDatabaseError: Error opening database file (/opt/intelmq/var/lib/bots/maxmind_geoip/GeoLite2-City.mmdb). Is this a valid MaxMind DB file? 2017-07-28 09:23:00,674 - maxmind-geoip-expert - INFO - Bot stopped.
It looks like the database file was only partially downloaded with the cron-based update last night. :-/
I was able to fix this by running /usr/bin/update-geoip-data
To avoid this, I think the updater should verify the downloaded database before overwriting the file.
Also, it looks like the script /usr/bin/update-geoip-data simply passes on any parameters to "mv" which is probably not a good idea:
# /usr/bin/update-geoip-data --help Usage: mv [OPTION]... [-T] SOURCE DEST or: mv [OPTION]... SOURCE... DIRECTORY or: mv [OPTION]... -t DIRECTORY SOURCE... Rename SOURCE to DEST, or move SOURCE(s) to DIRECTORY.
- Thomas
CERT-Bund Incident Response & Malware Analysis Team
Hi,
You are using this file: https://github.com/certtools/intelmq/blob/develop/intelmq/bots/experts/maxmi...
On 07/28/2017 10:00 AM, Thomas Hungenberg wrote:
To avoid this, I think the updater should verify the downloaded database before overwriting the file.
The script already as `set -e`, so it should exit on errors -> for curl everything was fine, we need to invoke the maxminddb:
python3 -c "import maxminddb; maxminddb.open_database('GeoLite2-City.mmdb')"
The non-zero exit code in case of en exception will then cause the abort of the script.
any parameters to "mv" which is probably not a good idea:
# /usr/bin/update-geoip-data --help Usage: mv [OPTION]... [-T] SOURCE DEST or: mv [OPTION]... SOURCE... DIRECTORY or: mv [OPTION]... -t DIRECTORY SOURCE... Rename SOURCE to DEST, or move SOURCE(s) to DIRECTORY.
Reading the script, this seems to be intentional, as the first parameter has to be the destination file. But we could catch -h and --help here to print this instruction.
Thanks for raising this issue.
Sebastian
P.S.: for pyasn we use the pyasn's tools for download and conversion, this is is probably not a problem there.