<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Sebastian,<div>my default configuration is and bot configurations are following, hope it helps:</div><div><pre style="color:rgb(0,0,0);white-space:pre-wrap">{
"accuracy": 0,
"broker": "redis",
"error_procedure": "pass",
"error_max_retries": 3,
"error_log_message": true,
"error_log_exception": true,
"error_dump_message": true,
"error_retry_delay": 15,
"rate_limit": 0,
"load_balance": true,
"source_pipeline_host": "redis",
"source_pipeline_port": "6379",
"source_pipeline_db": "2",
"source_pipeline_password": null,
"destination_pipeline_host": "redis",<br> "destination_pipeline_port": "6379",<br> "destination_pipeline_db": "2",
"destination_pipeline_password": null,
"http_proxy": null,
"https_proxy": null,
"http_user_agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36",
"http_timeout_max_tries": 3,
"http_timeout_sec": 30,
"http_verify_cert": true,
"log_processed_messages_count": 500,
"log_processed_messages_seconds": 900,
"logging_level": "INFO",
"logging_path": "/opt/intelmq/var/log/",
"logging_handler": "file",
"logging_syslog": "/dev/log",
"proccess_manager": "intelmq"
}</pre></div><div>With configuration for the bot:</div><div><pre style="color:rgb(0,0,0);white-space:pre-wrap"> "phishtank-collector": {
"description": "PhishTank Collector is the bot responsible to get the report from source of information.",
"group": "Collector",
"module": "intelmq.bots.collectors.http.collector_http",
"name": "PhishTank",
"parameters": {
"accuracy": 10,
"error_dump_message": true,
"error_log_message": true,
"error_max_retries": 1,
"error_procedure": "pass",
"error_retry_delay": 60,
"feed": "phishtank",
"http_url": "<a href="https://data.phishtank.com/data/online-valid.csv">https://data.phishtank.com/data/online-valid.csv</a>",
"rate_limit": 86600
}
},</pre></div><div><br clear="all"><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr">Sincerely,<div>Václav Brůžek</div></div></div></div></div></div><br></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 11 Mar 2019 at 14:59, Sebastian Wagner <<a href="mailto:wagner@cert.at">wagner@cert.at</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>HI Vaclav,<br>
</p>
<p>What's your default configuration and the runtime configuration
for this bot?</p>
<p>Sebastian<br>
</p>
<div class="gmail-m_4098019383956860312moz-cite-prefix">On 05.03.19 12:31, Vaclav Bruzek wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi mailinglist,
<div>I've went over intelmq logs today and that some bots
keep retrying indefinitely when they encounter a
failure. However each of those bots have <b>error_max_retries</b>
set to 1 (and <b>error_retry_delay </b>to 60). Which
in my mind should limit the bot to just try one more
time and than give up and sleep (if sleep is set up)
however this is not happening. Am I doing something
wrong or does this functionality work differently? My
version of intelmq is 1.1.1 in Docker on Ubuntu 18.04 as
base.</div>
<div><br>
</div>
<div>Following is an example from logs os collector
collecting from phishtank:</div>
<div>
<div>2019-03-01 20:07:51,747 - phishtank-collector -
INFO - Downloading report from '<a href="https://data.phishtank.com/data/online-valid.csv" target="_blank">https://data.phishtank.com/data/online-valid.csv</a>'.</div>
<div>2019-03-01 20:07:52,139 - phishtank-collector -
ERROR - Bot has found a problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process</div>
<div> raise ValueError('HTTP response status code was
%i.' % resp.status_code)</div>
<div>ValueError: HTTP response status code was 509.</div>
<div>2019-03-01 20:07:52,140 - phishtank-collector -
INFO - Current Message(event): None.</div>
<div>2019-03-01 20:07:52,140 - phishtank-collector -
INFO - Downloading report from '<a href="https://data.phishtank.com/data/online-valid.csv" target="_blank">https://data.phishtank.com/data/online-valid.csv</a>'.</div>
<div>2019-03-01 20:07:52,298 - phishtank-collector -
ERROR - Bot has found a problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process</div>
<div> raise ValueError('HTTP response status code was
%i.' % resp.status_code)</div>
<div>ValueError: HTTP response status code was 509.</div>
<div>2019-03-01 20:07:52,299 - phishtank-collector -
INFO - Current Message(event): None.</div>
<div>2019-03-01 20:07:52,299 - phishtank-collector -
INFO - Bot will continue in 60 seconds.</div>
<div>2019-03-01 20:08:52,359 - phishtank-collector -
INFO - Downloading report from '<a href="https://data.phishtank.com/data/online-valid.csv" target="_blank">https://data.phishtank.com/data/online-valid.csv</a>'.</div>
<div>2019-03-01 20:08:52,654 - phishtank-collector -
ERROR - Bot has found a problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process</div>
<div> raise ValueError('HTTP response status code was
%i.' % resp.status_code)</div>
<div>ValueError: HTTP response status code was 509.</div>
<div>2019-03-01 20:08:52,656 - phishtank-collector -
INFO - Current Message(event): None.</div>
<div>2019-03-01 20:08:52,656 - phishtank-collector -
INFO - Downloading report from '<a href="https://data.phishtank.com/data/online-valid.csv" target="_blank">https://data.phishtank.com/data/online-valid.csv</a>'.</div>
<div>2019-03-01 20:08:52,831 - phishtank-collector -
ERROR - Bot has found a problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process</div>
<div> raise ValueError('HTTP response status code was
%i.' % resp.status_code)</div>
<div>ValueError: HTTP response status code was 509.</div>
<div>2019-03-01 20:08:52,832 - phishtank-collector -
INFO - Current Message(event): None.</div>
<div>2019-03-01 20:08:52,832 - phishtank-collector -
INFO - Bot will continue in 60 seconds.</div>
<div>2019-03-01 20:09:52,891 - phishtank-collector -
INFO - Downloading report from '<a href="https://data.phishtank.com/data/online-valid.csv" target="_blank">https://data.phishtank.com/data/online-valid.csv</a>'.</div>
<div>2019-03-01 20:09:53,138 - phishtank-collector -
ERROR - Bot has found a problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process</div>
<div> raise ValueError('HTTP response status code was
%i.' % resp.status_code)</div>
<div>ValueError: HTTP response status code was 509.</div>
<div>2019-03-01 20:09:53,139 - phishtank-collector -
INFO - Current Message(event): None.</div>
<div>2019-03-01 20:09:53,139 - phishtank-collector -
INFO - Downloading report from '<a href="https://data.phishtank.com/data/online-valid.csv" target="_blank">https://data.phishtank.com/data/online-valid.csv</a>'.</div>
<div>2019-03-01 20:09:53,315 - phishtank-collector -
ERROR - Bot has found a problem.</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start</div>
<div> self.process()</div>
<div> File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process</div>
<div> raise ValueError('HTTP response status code was
%i.' % resp.status_code)</div>
<div>ValueError: HTTP response status code was 509.</div>
<div>2019-03-01 20:09:53,317 - phishtank-collector -
INFO - Current Message(event): None.</div>
<div>2019-03-01 20:09:53,318 - phishtank-collector -
INFO - Bot will continue in 60 seconds.</div>
</div>
<div><br>
</div>
<div><br clear="all">
<div>
<div dir="ltr" class="gmail-m_4098019383956860312gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">Sincerely,
<div>Václav Brůžek</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="gmail-m_4098019383956860312mimeAttachmentHeader"></fieldset>
</blockquote>
<pre class="gmail-m_4098019383956860312moz-signature" cols="72">--
// Sebastian Wagner <a class="gmail-m_4098019383956860312moz-txt-link-rfc2396E" href="mailto:wagner@cert.at" target="_blank"><wagner@cert.at></a> - T: +43 1 5056416 7201
// CERT Austria - <a class="gmail-m_4098019383956860312moz-txt-link-freetext" href="https://www.cert.at/" target="_blank">https://www.cert.at/</a>
// Eine Initiative der <a href="http://nic.at" target="_blank">nic.at</a> GmbH - <a class="gmail-m_4098019383956860312moz-txt-link-freetext" href="https://www.nic.at/" target="_blank">https://www.nic.at/</a>
// Firmenbuchnummer 172568b, LG Salzburg</pre>
</div>
</blockquote></div>