<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi,</p>
<p>The traceback in the email shows an exception in a custom bot.
Without the code, it's hard to say what's going on.</p>
<p>The exceptions attached do contain the following redis error
message:<br>
> redis.exceptions.BusyLoadingError: Redis is loading the
dataset in memory<br>
<br>
Looks like redis is just starting. In this case we could wait (up
to a maximum time) as long as this error occurs and then continue.
Still requires someone to implement it.<br>
I opened <a class="moz-txt-link-freetext" href="https://github.com/certtools/intelmq/issues/1334">https://github.com/certtools/intelmq/issues/1334</a> for it.<br>
</p>
<p>Sebastian<br>
</p>
<div class="moz-cite-prefix">On 11/10/2018 08.47, Vaclav Bruzek
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAAQS+TcBNJe5RtJ0n7WFFCe7KRTfC=Z3TuM+8+rZaguDdiQLCA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">Hi Sebastian,
<div>I've added the Redis exception to the attachment. That is
the case that I would expect that the bot would keep trying
to connect to Redis and not give up and exit. </div>
<div><br>
</div>
<div>I use continuous run mode for all bots. </div>
<div><br>
</div>
<div>I've also extracted the example of the other behaviour,
that is exiting without logging that the bot stopped. That
is indeed what I meant (your last point), that the bot logs
the exception but doesn't log the line "Bot stopped" and
stops, which is what status check is reproting.</div>
<div>
<div>¨</div>
<div>2018-10-02 02:43:13,744 - output - ERROR - Bot has
found a problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.0-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.0-py3.6.egg/intelmq/bots/outputs/bot/output.py",
line 67, in process</div>
<div> status = self.db_check()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.0-py3.6.egg/intelmq/bots/outputs/bot/output.py",
line 53, in db_check</div>
<div> payload =
self.connection_blacklist.get(key).decode("utf-8",
errors="ignore")</div>
<div>AttributeError: 'NoneType' object has no attribute
'decode'</div>
<div>2018-10-02 02:43:13,744 - output - INFO - Current
Message(event): {"some event"}.</div>
<div>2018-10-02 02:43:13,745 - output - INFO - Bot will
continue in 0 seconds.</div>
<div>2018-10-02 02:43:35,997 - whitelist-output - ERROR -
Bot has found a problem.</div>
<div>Traceback (most recent call last):</div>
<div><br>
</div>
<div>AttributeError: 'NoneType' object has no attribute
'decode'</div>
<div>2018-10-02 02:43:35,998 - whitelist-output - INFO -
Current Message(event): {'feed.accuracy': 100.0, '<a
href="http://feed.name" moz-do-not-send="true">feed.name</a>':
'whalebone', 'feed.url': '<a
href="http://wb-whitelist.azurewebsites.net/whitelist.txt"
moz-do-not-send="true">http://wb-whitelist.azurewebsites.net/whitelist.txt</a>',
'time.observation': '2018-10-02T02:41:58+00:00',
'source.fqdn': '<a href="http://com.bd"
moz-do-not-send="true">com.bd</a>', 'raw':
'Y29tLmJkDQo='}.</div>
<div>2018-10-02 02:43:35,998 - whitelist-output - INFO -
Dumping message from pipeline to dump file.</div>
</div>
<div><br>
</div>
<div><br clear="all">
<div>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">Sincerely,
<div>Václav Brůžek</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Wed, 10 Oct 2018 at 15:36, Sebastian Wagner
<<a href="mailto:wagner@cert.at" moz-do-not-send="true">wagner@cert.at</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Václav,</p>
<p>I can't estimate the implications of the docker usage on
redis and intelmq.</p>
<p>Concerning the redis problem: There were no changes in
the code handling redis problems and the only case when
intelmq's bots do not log anything is when there are not
enough resources to shutdown cleanly (memory, disk). Even
then, there's output on stdout. You could log stdout and
see if there are any errors shown at the end.</p>
<p>Concerning the error handling and sudden stops: There
haven't been code changes too. Do you use the scheduled
run mode? If the error_procedure is pass and there are
pipeline problems, the bot stops (in bot.py search for
"error_procedure: pass and pipeline problem"). AFAIR the
reasoning for this was/is that if the bot would not stop,
the pipeline would be kind of DOS'ed. But as problems with
memory and snapshots in redis are handled better now, that
could be relaxed. I'll do some experiments.</p>
<p>Concerning "encounters an exception and logs nothing but
status check reports that the bot is not running": How do
you know that the bot encountered an exception if nothing
is logged? Is the bot then still running or not?</p>
<p>Sebastian<br>
</p>
<div class="m_-2514805637409056439moz-cite-prefix">On
09/10/2018 12.58, Vaclav Bruzek wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi,
<div>no there are no modification to the intelmq code.
The situation occurs at my custom bots as well as the
default ones. As an example of this behaviour:
recently Redis broker wasn't available for some time,
as a result almost all bots stopped without any log
message indicating that the bot stopped.</div>
<div><br clear="all">
<div>
<div dir="ltr"
class="m_-2514805637409056439gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">Sincerely,
<div>Václav Brůžek</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, 9 Oct 2018 at 12:01, Sebastian
Wagner <<a href="mailto:wagner@cert.at"
target="_blank" moz-do-not-send="true">wagner@cert.at</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi,</p>
<p>I didn't know of any problems yet. Do you use any
custom modifications in the code? If yes, which?</p>
<p>Sebastian<br>
</p>
<div
class="m_-2514805637409056439m_8019457949410273893moz-cite-prefix">On
09/10/2018 10.42, Vaclav Bruzek wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi,
<div>since upgrading to version 1.1.0 it
became quite a big problem the
stability of bots. Often it happens
that bot encounters an exception and
logs that the bot is stopped or
encounters an exception and logs
nothing but status check reports that
the bot is not running. I'm using the
'error_procedure' parameter set to
'pass' (with error_max_retries
and error_retry_delay set to 0) and
I've always thought that this is a
sort of 'run forever' parameter that
even when exception occurs the bot
will keep on doing its job. I'm using
intelmq in Docker environment with
ubuntu 18.04 as base.</div>
<div><br clear="all">
<div>
<div dir="ltr"
class="m_-2514805637409056439m_8019457949410273893gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">Sincerely,
<div>Václav Brůžek</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset
class="m_-2514805637409056439m_8019457949410273893mimeAttachmentHeader"></fieldset>
</blockquote>
<pre class="m_-2514805637409056439m_8019457949410273893moz-signature" cols="72">--
// Sebastian Wagner <a class="m_-2514805637409056439m_8019457949410273893moz-txt-link-rfc2396E" href="mailto:wagner@cert.at" target="_blank" moz-do-not-send="true"><wagner@cert.at></a> - T: +43 1 5056416 7201
// CERT Austria - <a class="m_-2514805637409056439m_8019457949410273893moz-txt-link-freetext" href="https://www.cert.at/" target="_blank" moz-do-not-send="true">https://www.cert.at/</a>
// Eine Initiative der <a href="http://nic.at" target="_blank" moz-do-not-send="true">nic.at</a> GmbH - <a class="m_-2514805637409056439m_8019457949410273893moz-txt-link-freetext" href="https://www.nic.at/" target="_blank" moz-do-not-send="true">https://www.nic.at/</a>
// Firmenbuchnummer 172568b, LG Salzburg</pre>
</div>
</blockquote>
</div>
</blockquote>
<pre class="m_-2514805637409056439moz-signature" cols="72">--
// Sebastian Wagner <a class="m_-2514805637409056439moz-txt-link-rfc2396E" href="mailto:wagner@cert.at" target="_blank" moz-do-not-send="true"><wagner@cert.at></a> - T: +43 1 5056416 7201
// CERT Austria - <a class="m_-2514805637409056439moz-txt-link-freetext" href="https://www.cert.at/" target="_blank" moz-do-not-send="true">https://www.cert.at/</a>
// Eine Initiative der <a href="http://nic.at" target="_blank" moz-do-not-send="true">nic.at</a> GmbH - <a class="m_-2514805637409056439moz-txt-link-freetext" href="https://www.nic.at/" target="_blank" moz-do-not-send="true">https://www.nic.at/</a>
// Firmenbuchnummer 172568b, LG Salzburg</pre>
</div>
</blockquote>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
// Sebastian Wagner <a class="moz-txt-link-rfc2396E" href="mailto:wagner@cert.at"><wagner@cert.at></a> - T: +43 1 5056416 7201
// CERT Austria - <a class="moz-txt-link-freetext" href="https://www.cert.at/">https://www.cert.at/</a>
// Eine Initiative der nic.at GmbH - <a class="moz-txt-link-freetext" href="https://www.nic.at/">https://www.nic.at/</a>
// Firmenbuchnummer 172568b, LG Salzburg</pre>
</body>
</html>