<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>I found the cause of this and the fix is already in the
repository and will be part of the next releases (2.0.1 bugfix
release to be expected next week)</p>
<p><a class="moz-txt-link-freetext" href="https://github.com/certtools/intelmq/issues/1431">https://github.com/certtools/intelmq/issues/1431</a><br>
<a class="moz-txt-link-freetext" href="https://github.com/certtools/intelmq/commit/e644396a7c1ebb5d36e796335099dfc1deb783c1">https://github.com/certtools/intelmq/commit/e644396a7c1ebb5d36e796335099dfc1deb783c1</a><br>
</p>
<div class="moz-cite-prefix">On 3/11/19 3:33 PM, Vaclav Bruzek
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAAQS+Tfm7TdzNPjESP3V39Pb2AnLujXk=i=5oqTY5_ToopEGOw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi Sebastian,
<div>my default configuration is and bot
configurations are following, hope it helps:</div>
<div>
<pre style="color:rgb(0,0,0);white-space:pre-wrap">{
"accuracy": 0,
"broker": "redis",
"error_procedure": "pass",
"error_max_retries": 3,
"error_log_message": true,
"error_log_exception": true,
"error_dump_message": true,
"error_retry_delay": 15,
"rate_limit": 0,
"load_balance": true,
"source_pipeline_host": "redis",
"source_pipeline_port": "6379",
"source_pipeline_db": "2",
"source_pipeline_password": null,
"destination_pipeline_host": "redis",
"destination_pipeline_port": "6379",
"destination_pipeline_db": "2",
"destination_pipeline_password": null,
"http_proxy": null,
"https_proxy": null,
"http_user_agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36",
"http_timeout_max_tries": 3,
"http_timeout_sec": 30,
"http_verify_cert": true,
"log_processed_messages_count": 500,
"log_processed_messages_seconds": 900,
"logging_level": "INFO",
"logging_path": "/opt/intelmq/var/log/",
"logging_handler": "file",
"logging_syslog": "/dev/log",
"proccess_manager": "intelmq"
}</pre>
</div>
<div>With configuration for the bot:</div>
<div>
<pre style="color:rgb(0,0,0);white-space:pre-wrap"> "phishtank-collector": {
"description": "PhishTank Collector is the bot responsible to get the report from source of information.",
"group": "Collector",
"module": "intelmq.bots.collectors.http.collector_http",
"name": "PhishTank",
"parameters": {
"accuracy": 10,
"error_dump_message": true,
"error_log_message": true,
"error_max_retries": 1,
"error_procedure": "pass",
"error_retry_delay": 60,
"feed": "phishtank",
"http_url": "<a href="https://data.phishtank.com/data/online-valid.csv" moz-do-not-send="true">https://data.phishtank.com/data/online-valid.csv</a>",
"rate_limit": 86600
}
},</pre>
</div>
<div><br clear="all">
<div>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">Sincerely,
<div>Václav Brůžek</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, 11 Mar 2019 at 14:59,
Sebastian Wagner <<a href="mailto:wagner@cert.at"
moz-do-not-send="true">wagner@cert.at</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>HI Vaclav,<br>
</p>
<p>What's your default configuration and the runtime
configuration for this bot?</p>
<p>Sebastian<br>
</p>
<div class="gmail-m_4098019383956860312moz-cite-prefix">On
05.03.19 12:31, Vaclav Bruzek wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi mailinglist,
<div>I've went over intelmq logs today and that
some bots keep retrying indefinitely when they
encounter a failure. However each of those bots
have <b>error_max_retries</b> set to 1 (and <b>error_retry_delay
</b>to 60). Which in my mind should limit the
bot to just try one more time and than give up
and sleep (if sleep is set up) however this is
not happening. Am I doing something wrong or
does this functionality work differently? My
version of intelmq is 1.1.1 in Docker on Ubuntu
18.04 as base.</div>
<div><br>
</div>
<div>Following is an example from logs os
collector collecting from phishtank:</div>
<div>
<div>2019-03-01 20:07:51,747 -
phishtank-collector - INFO - Downloading
report from '<a
href="https://data.phishtank.com/data/online-valid.csv"
target="_blank" moz-do-not-send="true">https://data.phishtank.com/data/online-valid.csv</a>'.</div>
<div>2019-03-01 20:07:52,139 -
phishtank-collector - ERROR - Bot has found a
problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process</div>
<div> raise ValueError('HTTP response status
code was %i.' % resp.status_code)</div>
<div>ValueError: HTTP response status code was
509.</div>
<div>2019-03-01 20:07:52,140 -
phishtank-collector - INFO - Current
Message(event): None.</div>
<div>2019-03-01 20:07:52,140 -
phishtank-collector - INFO - Downloading
report from '<a
href="https://data.phishtank.com/data/online-valid.csv"
target="_blank" moz-do-not-send="true">https://data.phishtank.com/data/online-valid.csv</a>'.</div>
<div>2019-03-01 20:07:52,298 -
phishtank-collector - ERROR - Bot has found a
problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process</div>
<div> raise ValueError('HTTP response status
code was %i.' % resp.status_code)</div>
<div>ValueError: HTTP response status code was
509.</div>
<div>2019-03-01 20:07:52,299 -
phishtank-collector - INFO - Current
Message(event): None.</div>
<div>2019-03-01 20:07:52,299 -
phishtank-collector - INFO - Bot will continue
in 60 seconds.</div>
<div>2019-03-01 20:08:52,359 -
phishtank-collector - INFO - Downloading
report from '<a
href="https://data.phishtank.com/data/online-valid.csv"
target="_blank" moz-do-not-send="true">https://data.phishtank.com/data/online-valid.csv</a>'.</div>
<div>2019-03-01 20:08:52,654 -
phishtank-collector - ERROR - Bot has found a
problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process</div>
<div> raise ValueError('HTTP response status
code was %i.' % resp.status_code)</div>
<div>ValueError: HTTP response status code was
509.</div>
<div>2019-03-01 20:08:52,656 -
phishtank-collector - INFO - Current
Message(event): None.</div>
<div>2019-03-01 20:08:52,656 -
phishtank-collector - INFO - Downloading
report from '<a
href="https://data.phishtank.com/data/online-valid.csv"
target="_blank" moz-do-not-send="true">https://data.phishtank.com/data/online-valid.csv</a>'.</div>
<div>2019-03-01 20:08:52,831 -
phishtank-collector - ERROR - Bot has found a
problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process</div>
<div> raise ValueError('HTTP response status
code was %i.' % resp.status_code)</div>
<div>ValueError: HTTP response status code was
509.</div>
<div>2019-03-01 20:08:52,832 -
phishtank-collector - INFO - Current
Message(event): None.</div>
<div>2019-03-01 20:08:52,832 -
phishtank-collector - INFO - Bot will continue
in 60 seconds.</div>
<div>2019-03-01 20:09:52,891 -
phishtank-collector - INFO - Downloading
report from '<a
href="https://data.phishtank.com/data/online-valid.csv"
target="_blank" moz-do-not-send="true">https://data.phishtank.com/data/online-valid.csv</a>'.</div>
<div>2019-03-01 20:09:53,138 -
phishtank-collector - ERROR - Bot has found a
problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process</div>
<div> raise ValueError('HTTP response status
code was %i.' % resp.status_code)</div>
<div>ValueError: HTTP response status code was
509.</div>
<div>2019-03-01 20:09:53,139 -
phishtank-collector - INFO - Current
Message(event): None.</div>
<div>2019-03-01 20:09:53,139 -
phishtank-collector - INFO - Downloading
report from '<a
href="https://data.phishtank.com/data/online-valid.csv"
target="_blank" moz-do-not-send="true">https://data.phishtank.com/data/online-valid.csv</a>'.</div>
<div>2019-03-01 20:09:53,315 -
phishtank-collector - ERROR - Bot has found a
problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process</div>
<div> raise ValueError('HTTP response status
code was %i.' % resp.status_code)</div>
<div>ValueError: HTTP response status code was
509.</div>
<div>2019-03-01 20:09:53,317 -
phishtank-collector - INFO - Current
Message(event): None.</div>
<div>2019-03-01 20:09:53,318 -
phishtank-collector - INFO - Bot will continue
in 60 seconds.</div>
</div>
<div><br>
</div>
<div><br clear="all">
<div>
<div dir="ltr"
class="gmail-m_4098019383956860312gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">Sincerely,
<div>Václav Brůžek</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset
class="gmail-m_4098019383956860312mimeAttachmentHeader"></fieldset>
</blockquote>
<pre class="gmail-m_4098019383956860312moz-signature" cols="72">--
// Sebastian Wagner <a class="gmail-m_4098019383956860312moz-txt-link-rfc2396E" href="mailto:wagner@cert.at" target="_blank" moz-do-not-send="true"><wagner@cert.at></a> - T: +43 1 5056416 7201
// CERT Austria - <a class="gmail-m_4098019383956860312moz-txt-link-freetext" href="https://www.cert.at/" target="_blank" moz-do-not-send="true">https://www.cert.at/</a>
// Eine Initiative der <a href="http://nic.at" target="_blank" moz-do-not-send="true">nic.at</a> GmbH - <a class="gmail-m_4098019383956860312moz-txt-link-freetext" href="https://www.nic.at/" target="_blank" moz-do-not-send="true">https://www.nic.at/</a>
// Firmenbuchnummer 172568b, LG Salzburg</pre>
</div>
</blockquote>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
// Sebastian Wagner <a class="moz-txt-link-rfc2396E" href="mailto:wagner@cert.at"><wagner@cert.at></a> - T: +43 1 5056416 7201
// CERT Austria - <a class="moz-txt-link-freetext" href="https://www.cert.at/">https://www.cert.at/</a>
// Eine Initiative der nic.at GmbH - <a class="moz-txt-link-freetext" href="https://www.nic.at/">https://www.nic.at/</a>
// Firmenbuchnummer 172568b, LG Salzburg</pre>
</body>
</html>