[IntelMQ-dev] IntelMQ 3.0.2 release fixing performance issues

[Sebastian Wagner]

Dear community,

Last week I said in the announcement for 3.0.1, that we didn't plan any
further bugfix release before 3.1.0. I'm now proving myself wrong.
Narzhan reported general performance issues and we investigated the
problem on our own instances.

We found two problems which different effects. It could be the case that
you didn't even notice any performance drop, as both bugs affected only
certain bots or situations.

- When any collector creates a new report, it initializes an object of
type Report. Since five years there was a bug, that this always loaded
the harmonization.conf, instead of loading it just once at the start
(which is - correctly - the case for new events). This was never a
problem, as the Python's built-in json module was fast enough, but
ruamel.yaml is slower. If a collector didn't massively create reports,
that bug was not obvious, especially under testing conditions. And the
bug was only relevant, if stream data is processed (for example with the
Shodan stream collector).

- Two bots, the STOMP collector and the API collector, don't have a
proper process method, because they are using threading. A thread
retrieves data asynchronously and sends the data. Normally, the
process-method is running in an endless-loop, with sleeping rate_limit
seconds between. If rate_limit is 0 or not defined, this would result in
high load, therefore these bots defined the __collector_empty_process
attribute which should have prohibited the endless loop. But due to
class-restructuring, the attribute which would have been to be set was
_Bot__collector_empty_process, and the flag was not effective. That did
not only cause a loop, but as bot statistics are written to redis after
every process-call, that caused continuous connections to redis. We've
now renamed the flag to _collector_empty_process and fixed the two bots.
The flag __is_multithreadable was affected analogous, but that should
not have cause performance issues in particular.
  Additionally, the bot statistics can now be disabled by setting
|statistics_host| to null (previously this caused an exception).|
|

The new version is on GitHub and PyPI, very soon in the deb/rpm
repositories. The Docker image will follow on Monday or earlier.

You can read the full changelog here:
https://github.com/certtools/intelmq/releases/tag/3.0.2

have a nice weekend
Sebastian

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210910/226d6031/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210910/226d6031/attachment.sig>