[Intelmq-users] Max retries issue
Vaclav Bruzek
vasek.bruzek at gmail.com
Mon Mar 11 15:33:12 CET 2019
Hi Sebastian,
my default configuration is and bot configurations are following, hope it
helps:
{
"accuracy": 0,
"broker": "redis",
"error_procedure": "pass",
"error_max_retries": 3,
"error_log_message": true,
"error_log_exception": true,
"error_dump_message": true,
"error_retry_delay": 15,
"rate_limit": 0,
"load_balance": true,
"source_pipeline_host": "redis",
"source_pipeline_port": "6379",
"source_pipeline_db": "2",
"source_pipeline_password": null,
"destination_pipeline_host": "redis",
"destination_pipeline_port": "6379",
"destination_pipeline_db": "2",
"destination_pipeline_password": null,
"http_proxy": null,
"https_proxy": null,
"http_user_agent": "Mozilla/5.0 (Windows NT 6.1)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0
Safari/537.36",
"http_timeout_max_tries": 3,
"http_timeout_sec": 30,
"http_verify_cert": true,
"log_processed_messages_count": 500,
"log_processed_messages_seconds": 900,
"logging_level": "INFO",
"logging_path": "/opt/intelmq/var/log/",
"logging_handler": "file",
"logging_syslog": "/dev/log",
"proccess_manager": "intelmq"
}
With configuration for the bot:
"phishtank-collector": {
"description": "PhishTank Collector is the bot responsible to
get the report from source of information.",
"group": "Collector",
"module": "intelmq.bots.collectors.http.collector_http",
"name": "PhishTank",
"parameters": {
"accuracy": 10,
"error_dump_message": true,
"error_log_message": true,
"error_max_retries": 1,
"error_procedure": "pass",
"error_retry_delay": 60,
"feed": "phishtank",
"http_url": "https://data.phishtank.com/data/online-valid.csv",
"rate_limit": 86600
}
},
Sincerely,
Václav Brůžek
On Mon, 11 Mar 2019 at 14:59, Sebastian Wagner <wagner at cert.at> wrote:
> HI Vaclav,
>
> What's your default configuration and the runtime configuration for this
> bot?
>
> Sebastian
> On 05.03.19 12:31, Vaclav Bruzek wrote:
>
> Hi mailinglist,
> I've went over intelmq logs today and that some bots keep retrying
> indefinitely when they encounter a failure. However each of those bots have
> *error_max_retries* set to 1 (and *error_retry_delay *to 60). Which in my
> mind should limit the bot to just try one more time and than give up and
> sleep (if sleep is set up) however this is not happening. Am I doing
> something wrong or does this functionality work differently? My version of
> intelmq is 1.1.1 in Docker on Ubuntu 18.04 as base.
>
> Following is an example from logs os collector collecting from phishtank:
> 2019-03-01 20:07:51,747 - phishtank-collector - INFO - Downloading report
> from 'https://data.phishtank.com/data/online-valid.csv'.
> 2019-03-01 20:07:52,139 - phishtank-collector - ERROR - Bot has found a
> problem.
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
> line 167, in start
> self.process()
> File
> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
> line 76, in process
> raise ValueError('HTTP response status code was %i.' %
> resp.status_code)
> ValueError: HTTP response status code was 509.
> 2019-03-01 20:07:52,140 - phishtank-collector - INFO - Current
> Message(event): None.
> 2019-03-01 20:07:52,140 - phishtank-collector - INFO - Downloading report
> from 'https://data.phishtank.com/data/online-valid.csv'.
> 2019-03-01 20:07:52,298 - phishtank-collector - ERROR - Bot has found a
> problem.
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
> line 167, in start
> self.process()
> File
> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
> line 76, in process
> raise ValueError('HTTP response status code was %i.' %
> resp.status_code)
> ValueError: HTTP response status code was 509.
> 2019-03-01 20:07:52,299 - phishtank-collector - INFO - Current
> Message(event): None.
> 2019-03-01 20:07:52,299 - phishtank-collector - INFO - Bot will continue
> in 60 seconds.
> 2019-03-01 20:08:52,359 - phishtank-collector - INFO - Downloading report
> from 'https://data.phishtank.com/data/online-valid.csv'.
> 2019-03-01 20:08:52,654 - phishtank-collector - ERROR - Bot has found a
> problem.
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
> line 167, in start
> self.process()
> File
> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
> line 76, in process
> raise ValueError('HTTP response status code was %i.' %
> resp.status_code)
> ValueError: HTTP response status code was 509.
> 2019-03-01 20:08:52,656 - phishtank-collector - INFO - Current
> Message(event): None.
> 2019-03-01 20:08:52,656 - phishtank-collector - INFO - Downloading report
> from 'https://data.phishtank.com/data/online-valid.csv'.
> 2019-03-01 20:08:52,831 - phishtank-collector - ERROR - Bot has found a
> problem.
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
> line 167, in start
> self.process()
> File
> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
> line 76, in process
> raise ValueError('HTTP response status code was %i.' %
> resp.status_code)
> ValueError: HTTP response status code was 509.
> 2019-03-01 20:08:52,832 - phishtank-collector - INFO - Current
> Message(event): None.
> 2019-03-01 20:08:52,832 - phishtank-collector - INFO - Bot will continue
> in 60 seconds.
> 2019-03-01 20:09:52,891 - phishtank-collector - INFO - Downloading report
> from 'https://data.phishtank.com/data/online-valid.csv'.
> 2019-03-01 20:09:53,138 - phishtank-collector - ERROR - Bot has found a
> problem.
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
> line 167, in start
> self.process()
> File
> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
> line 76, in process
> raise ValueError('HTTP response status code was %i.' %
> resp.status_code)
> ValueError: HTTP response status code was 509.
> 2019-03-01 20:09:53,139 - phishtank-collector - INFO - Current
> Message(event): None.
> 2019-03-01 20:09:53,139 - phishtank-collector - INFO - Downloading report
> from 'https://data.phishtank.com/data/online-valid.csv'.
> 2019-03-01 20:09:53,315 - phishtank-collector - ERROR - Bot has found a
> problem.
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
> line 167, in start
> self.process()
> File
> "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
> line 76, in process
> raise ValueError('HTTP response status code was %i.' %
> resp.status_code)
> ValueError: HTTP response status code was 509.
> 2019-03-01 20:09:53,317 - phishtank-collector - INFO - Current
> Message(event): None.
> 2019-03-01 20:09:53,318 - phishtank-collector - INFO - Bot will continue
> in 60 seconds.
>
>
> Sincerely,
> Václav Brůžek
>
> --
> // Sebastian Wagner <wagner at cert.at> <wagner at cert.at> - T: +43 1 5056416 7201
> // CERT Austria - https://www.cert.at/
> // Eine Initiative der nic.at GmbH - https://www.nic.at/
> // Firmenbuchnummer 172568b, LG Salzburg
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20190311/0065437b/attachment.html>
More information about the Intelmq-users
mailing list