Dear community,
Last week I said in the announcement for 3.0.1, that we didn't plan any further bugfix release before 3.1.0. I'm now proving myself wrong. Narzhan reported general performance issues and we investigated the problem on our own instances.
We found two problems which different effects. It could be the case that you didn't even notice any performance drop, as both bugs affected only certain bots or situations.
- When any collector creates a new report, it initializes an object of type Report. Since five years there was a bug, that this always loaded the harmonization.conf, instead of loading it just once at the start (which is - correctly - the case for new events). This was never a problem, as the Python's built-in json module was fast enough, but ruamel.yaml is slower. If a collector didn't massively create reports, that bug was not obvious, especially under testing conditions. And the bug was only relevant, if stream data is processed (for example with the Shodan stream collector).
- Two bots, the STOMP collector and the API collector, don't have a proper process method, because they are using threading. A thread retrieves data asynchronously and sends the data. Normally, the process-method is running in an endless-loop, with sleeping rate_limit seconds between. If rate_limit is 0 or not defined, this would result in high load, therefore these bots defined the __collector_empty_process attribute which should have prohibited the endless loop. But due to class-restructuring, the attribute which would have been to be set was _Bot__collector_empty_process, and the flag was not effective. That did not only cause a loop, but as bot statistics are written to redis after every process-call, that caused continuous connections to redis. We've now renamed the flag to _collector_empty_process and fixed the two bots. The flag __is_multithreadable was affected analogous, but that should not have cause performance issues in particular. Additionally, the bot statistics can now be disabled by setting |statistics_host| to null (previously this caused an exception).| |
The new version is on GitHub and PyPI, very soon in the deb/rpm repositories. The Docker image will follow on Monday or earlier.
You can read the full changelog here: https://github.com/certtools/intelmq/releases/tag/3.0.2
have a nice weekend Sebastian