HI Vaclav,

What's your default configuration and the runtime configuration for this bot?

Sebastian

On 05.03.19 12:31, Vaclav Bruzek wrote:
Hi mailinglist,
I've went over intelmq logs today and that some bots keep retrying indefinitely when they encounter a failure. However each of those bots have error_max_retries set to 1 (and error_retry_delay to 60). Which in my mind should limit the bot to just try one more time and than give up and sleep (if sleep is set up) however this is not happening. Am I doing something wrong or does this functionality work differently? My version of intelmq is 1.1.1 in Docker on Ubuntu 18.04 as base.

Following is an example from logs os collector collecting from phishtank:
2019-03-01 20:07:51,747 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:07:52,139 - phishtank-collector - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start
    self.process()
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process
    raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:07:52,140 - phishtank-collector - INFO - Current Message(event): None.
2019-03-01 20:07:52,140 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:07:52,298 - phishtank-collector - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start
    self.process()
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process
    raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:07:52,299 - phishtank-collector - INFO - Current Message(event): None.
2019-03-01 20:07:52,299 - phishtank-collector - INFO - Bot will continue in 60 seconds.
2019-03-01 20:08:52,359 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:08:52,654 - phishtank-collector - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start
    self.process()
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process
    raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:08:52,656 - phishtank-collector - INFO - Current Message(event): None.
2019-03-01 20:08:52,656 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:08:52,831 - phishtank-collector - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start
    self.process()
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process
    raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:08:52,832 - phishtank-collector - INFO - Current Message(event): None.
2019-03-01 20:08:52,832 - phishtank-collector - INFO - Bot will continue in 60 seconds.
2019-03-01 20:09:52,891 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:09:53,138 - phishtank-collector - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start
    self.process()
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process
    raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:09:53,139 - phishtank-collector - INFO - Current Message(event): None.
2019-03-01 20:09:53,139 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:09:53,315 - phishtank-collector - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start
    self.process()
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process
    raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:09:53,317 - phishtank-collector - INFO - Current Message(event): None.
2019-03-01 20:09:53,318 - phishtank-collector - INFO - Bot will continue in 60 seconds.


Sincerely,
Václav Brůžek

-- 
// Sebastian Wagner <wagner@cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg