Hi mailinglist,
I've went over intelmq logs today and that some bots keep retrying
indefinitely when they encounter a failure. However each of those bots have
*error_max_retries* set to 1 (and *error_retry_delay *to 60). Which in my
mind should limit the bot to just try one more time and than give up and
sleep (if sleep is set up) however this is not happening. Am I doing
something wrong or does this functionality work differently? My version of
intelmq is 1.1.1 in Docker on Ubuntu 18.04 as base.
Following is an example from logs os collector collecting from phishtank:
2019-03-01 20:07:51,747 - phishtank-collector - INFO - Downloading report
from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:07:52,139 - phishtank-collector - ERROR - Bot has found a
problem.
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start
self.process()
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process
raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:07:52,140 - phishtank-collector - INFO - Current
Message(event): None.
2019-03-01 20:07:52,140 - phishtank-collector - INFO - Downloading report
from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:07:52,298 - phishtank-collector - ERROR - Bot has found a
problem.
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start
self.process()
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process
raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:07:52,299 - phishtank-collector - INFO - Current
Message(event): None.
2019-03-01 20:07:52,299 - phishtank-collector - INFO - Bot will continue in
60 seconds.
2019-03-01 20:08:52,359 - phishtank-collector - INFO - Downloading report
from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:08:52,654 - phishtank-collector - ERROR - Bot has found a
problem.
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start
self.process()
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process
raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:08:52,656 - phishtank-collector - INFO - Current
Message(event): None.
2019-03-01 20:08:52,656 - phishtank-collector - INFO - Downloading report
from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:08:52,831 - phishtank-collector - ERROR - Bot has found a
problem.
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start
self.process()
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process
raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:08:52,832 - phishtank-collector - INFO - Current
Message(event): None.
2019-03-01 20:08:52,832 - phishtank-collector - INFO - Bot will continue in
60 seconds.
2019-03-01 20:09:52,891 - phishtank-collector - INFO - Downloading report
from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:09:53,138 - phishtank-collector - ERROR - Bot has found a
problem.
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start
self.process()
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process
raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:09:53,139 - phishtank-collector - INFO - Current
Message(event): None.
2019-03-01 20:09:53,139 - phishtank-collector - INFO - Downloading report
from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:09:53,315 - phishtank-collector - ERROR - Bot has found a
problem.
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start
self.process()
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process
raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:09:53,317 - phishtank-collector - INFO - Current
Message(event): None.
2019-03-01 20:09:53,318 - phishtank-collector - INFO - Bot will continue in
60 seconds.
Sincerely,
Václav Brůžek