[Intelmq-users] Max retries issue
Sebastian Wagner
wagner at cert.at
Thu Aug 1 21:48:41 CEST 2019
I found the cause of this and the fix is already in the repository and
will be part of the next releases (2.0.1 bugfix release to be expected
next week)
https://github.com/certtools/intelmq/issues/1431
https://github.com/certtools/intelmq/commit/e644396a7c1ebb5d36e796335099dfc1deb783c1
On 3/11/19 3:33 PM, Vaclav Bruzek wrote:
> Hi Sebastian,
> my default configuration is and bot configurations are following, hope
> it helps:
> {
> "accuracy": 0,
> "broker": "redis",
> "error_procedure": "pass",
> "error_max_retries": 3,
> "error_log_message": true,
> "error_log_exception": true,
> "error_dump_message": true,
> "error_retry_delay": 15,
> "rate_limit": 0,
> "load_balance": true,
> "source_pipeline_host": "redis",
> "source_pipeline_port": "6379",
> "source_pipeline_db": "2",
> "source_pipeline_password": null,
> "destination_pipeline_host": "redis",
> "destination_pipeline_port": "6379",
> "destination_pipeline_db": "2",
> "destination_pipeline_password": null,
> "http_proxy": null,
> "https_proxy": null,
> "http_user_agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36",
> "http_timeout_max_tries": 3,
> "http_timeout_sec": 30,
> "http_verify_cert": true,
> "log_processed_messages_count": 500,
> "log_processed_messages_seconds": 900,
> "logging_level": "INFO",
> "logging_path": "/opt/intelmq/var/log/",
> "logging_handler": "file",
> "logging_syslog": "/dev/log",
> "proccess_manager": "intelmq"
> }
> With configuration for the bot:
> "phishtank-collector": {
> "description": "PhishTank Collector is the bot responsible to get the report from source of information.",
> "group": "Collector",
> "module": "intelmq.bots.collectors.http.collector_http",
> "name": "PhishTank",
> "parameters": {
> "accuracy": 10,
> "error_dump_message": true,
> "error_log_message": true,
> "error_max_retries": 1,
> "error_procedure": "pass",
> "error_retry_delay": 60,
> "feed": "phishtank",
> "http_url": "https://data.phishtank.com/data/online-valid.csv",
> "rate_limit": 86600
> }
> },
>
> Sincerely,
> Václav Brůžek
>
>
> On Mon, 11 Mar 2019 at 14:59, Sebastian Wagner <wagner at cert.at
> <mailto:wagner at cert.at>> wrote:
>
> HI Vaclav,
>
> What's your default configuration and the runtime configuration
> for this bot?
>
> Sebastian
>
> On 05.03.19 12:31, Vaclav Bruzek wrote:
>> Hi mailinglist,
>> I've went over intelmq logs today and that some bots keep
>> retrying indefinitely when they encounter a failure. However each
>> of those bots have *error_max_retries* set to 1 (and
>> *error_retry_delay *to 60). Which in my mind should limit the bot
>> to just try one more time and than give up and sleep (if sleep is
>> set up) however this is not happening. Am I doing something wrong
>> or does this functionality work differently? My version of
>> intelmq is 1.1.1 in Docker on Ubuntu 18.04 as base.
>>
>> Following is an example from logs os collector collecting from
>> phishtank:
>> 2019-03-01 20:07:51,747 - phishtank-collector - INFO -
>> Downloading report from
>> 'https://data.phishtank.com/data/online-valid.csv'.
>> 2019-03-01 20:07:52,139 - phishtank-collector - ERROR - Bot has
>> found a problem.
>> Traceback (most recent call last):
>> File
>> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
>> line 167, in start
>> self.process()
>> File
>> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
>> line 76, in process
>> raise ValueError('HTTP response status code was %i.' %
>> resp.status_code)
>> ValueError: HTTP response status code was 509.
>> 2019-03-01 20:07:52,140 - phishtank-collector - INFO - Current
>> Message(event): None.
>> 2019-03-01 20:07:52,140 - phishtank-collector - INFO -
>> Downloading report from
>> 'https://data.phishtank.com/data/online-valid.csv'.
>> 2019-03-01 20:07:52,298 - phishtank-collector - ERROR - Bot has
>> found a problem.
>> Traceback (most recent call last):
>> File
>> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
>> line 167, in start
>> self.process()
>> File
>> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
>> line 76, in process
>> raise ValueError('HTTP response status code was %i.' %
>> resp.status_code)
>> ValueError: HTTP response status code was 509.
>> 2019-03-01 20:07:52,299 - phishtank-collector - INFO - Current
>> Message(event): None.
>> 2019-03-01 20:07:52,299 - phishtank-collector - INFO - Bot will
>> continue in 60 seconds.
>> 2019-03-01 20:08:52,359 - phishtank-collector - INFO -
>> Downloading report from
>> 'https://data.phishtank.com/data/online-valid.csv'.
>> 2019-03-01 20:08:52,654 - phishtank-collector - ERROR - Bot has
>> found a problem.
>> Traceback (most recent call last):
>> File
>> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
>> line 167, in start
>> self.process()
>> File
>> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
>> line 76, in process
>> raise ValueError('HTTP response status code was %i.' %
>> resp.status_code)
>> ValueError: HTTP response status code was 509.
>> 2019-03-01 20:08:52,656 - phishtank-collector - INFO - Current
>> Message(event): None.
>> 2019-03-01 20:08:52,656 - phishtank-collector - INFO -
>> Downloading report from
>> 'https://data.phishtank.com/data/online-valid.csv'.
>> 2019-03-01 20:08:52,831 - phishtank-collector - ERROR - Bot has
>> found a problem.
>> Traceback (most recent call last):
>> File
>> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
>> line 167, in start
>> self.process()
>> File
>> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
>> line 76, in process
>> raise ValueError('HTTP response status code was %i.' %
>> resp.status_code)
>> ValueError: HTTP response status code was 509.
>> 2019-03-01 20:08:52,832 - phishtank-collector - INFO - Current
>> Message(event): None.
>> 2019-03-01 20:08:52,832 - phishtank-collector - INFO - Bot will
>> continue in 60 seconds.
>> 2019-03-01 20:09:52,891 - phishtank-collector - INFO -
>> Downloading report from
>> 'https://data.phishtank.com/data/online-valid.csv'.
>> 2019-03-01 20:09:53,138 - phishtank-collector - ERROR - Bot has
>> found a problem.
>> Traceback (most recent call last):
>> File
>> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
>> line 167, in start
>> self.process()
>> File
>> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
>> line 76, in process
>> raise ValueError('HTTP response status code was %i.' %
>> resp.status_code)
>> ValueError: HTTP response status code was 509.
>> 2019-03-01 20:09:53,139 - phishtank-collector - INFO - Current
>> Message(event): None.
>> 2019-03-01 20:09:53,139 - phishtank-collector - INFO -
>> Downloading report from
>> 'https://data.phishtank.com/data/online-valid.csv'.
>> 2019-03-01 20:09:53,315 - phishtank-collector - ERROR - Bot has
>> found a problem.
>> Traceback (most recent call last):
>> File
>> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
>> line 167, in start
>> self.process()
>> File
>> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
>> line 76, in process
>> raise ValueError('HTTP response status code was %i.' %
>> resp.status_code)
>> ValueError: HTTP response status code was 509.
>> 2019-03-01 20:09:53,317 - phishtank-collector - INFO - Current
>> Message(event): None.
>> 2019-03-01 20:09:53,318 - phishtank-collector - INFO - Bot will
>> continue in 60 seconds.
>>
>>
>> Sincerely,
>> Václav Brůžek
>>
> --
> // Sebastian Wagner <wagner at cert.at> <mailto:wagner at cert.at> - T: +43 1 5056416 7201
> // CERT Austria - https://www.cert.at/
> // Eine Initiative der nic.at <http://nic.at> GmbH - https://www.nic.at/
> // Firmenbuchnummer 172568b, LG Salzburg
>
--
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20190801/a809fba2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20190801/a809fba2/attachment.sig>
More information about the Intelmq-users
mailing list