<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dear community,</p>
<p>Last week I said in the announcement for 3.0.1, that we didn't
plan any further bugfix release before 3.1.0. I'm now proving
myself wrong. Narzhan reported general performance issues and we
investigated the problem on our own instances.<br>
</p>
<p>We found two problems which different effects. It could be the
case that you didn't even notice any performance drop, as both
bugs affected only certain bots or situations.</p>
<p>- When any collector creates a new report, it initializes an
object of type <tt>Report</tt>. Since five years there was a bug,
that this always loaded the <tt>harmonization.conf</tt>, instead
of loading it just once at the start (which is - correctly - the
case for new events). This was never a problem, as the Python's
built-in <tt>json</tt> module was fast enough, but <tt>ruamel.yaml</tt>
is slower. If a collector didn't massively create reports, that
bug was not obvious, especially under testing conditions. And the
bug was only relevant, if stream data is processed (for example
with the Shodan stream collector).</p>
<p>- Two bots, the STOMP collector and the API collector, don't have
a proper <tt>process</tt> method, because they are using
threading. A thread retrieves data asynchronously and sends the
data. Normally, the process-method is running in an endless-loop,
with sleeping <tt>rate_limit</tt> seconds between. If <tt>rate_limit</tt>
is 0 or not defined, this would result in high load, therefore
these bots defined the <tt>__collector_empty_process</tt>
attribute which should have prohibited the endless loop. But due
to class-restructuring, the attribute which would have been to be
set was <tt>_Bot__collector_empty_process</tt>, and the flag was
not effective. That did not only cause a loop, but as bot
statistics are written to redis after every process-call, that
caused continuous connections to redis. We've now renamed the flag
to <tt>_collector_empty_process</tt> and fixed the two bots. The
flag <tt>__is_multithreadable</tt> was affected analogous, but
that should not have cause performance issues in particular.<br>
Additionally, the bot statistics can now be disabled by setting
<code>statistics_host</code> to null (previously this caused an
exception).<code><br>
</code></p>
<p>The new version is on GitHub and PyPI, very soon in the deb/rpm
repositories. The Docker image will follow on Monday or earlier.<br>
</p>
<p>You can read the full changelog here:<br>
<a class="moz-txt-link-freetext"
href="https://github.com/certtools/intelmq/releases/tag/3.0.2"
moz-do-not-send="true">https://github.com/certtools/intelmq/releases/tag/3.0.2</a></p>
<p>have a nice weekend<br>
Sebastian<br>
</p>
<pre class="moz-signature" cols="72">--
// Sebastian Wagner <a class="moz-txt-link-rfc2396E" href="mailto:wagner@cert.at" moz-do-not-send="true"><wagner@cert.at></a> - T: +43 676 898 298 7201
// CERT Austria - <a class="moz-txt-link-freetext" href="https://www.cert.at/" moz-do-not-send="true">https://www.cert.at/</a>
// Eine Initiative der nic.at GmbH - <a class="moz-txt-link-freetext" href="https://www.nic.at/" moz-do-not-send="true">https://www.nic.at/</a>
// Firmenbuchnummer 172568b, LG Salzburg</pre>
</body>
</html>