[Intelmq-dev] Bot problem

[Vaclav Bruzek]

Hi,
I've recently encountered a problem with the rest-api-output bot. The bot
sometimes timeouts because the consumer endpoint is unavailable. I've
thought that the error_procedure -> pass would solve the problem and the
bot would retry after several seconds (immediately in current set-up).
However the bot tries once and than stops I thought that max retiries
discards the event but in this case it seems that it shuts down the bot
itself. There are no modifications to the code of the code. Following is
the runtime configuration without the targets configuration. The exception
is quite large so I've added it to the attachment.

"rest-api-0-output":{
   "group":"Output",
   "module":"intelmq.bots.outputs.restapi.output",
   "name":"REST API",
   "parameters":{
      "error_dump_message":true,
      "error_log_message":true,
      "error_max_retries":1,
      "error_procedure":"pass",
      "error_retry_delay":0,
      "hierarchical_output":true,
      "use_json":true
   }
},

Sincerely,
Václav Brůžek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20190114/2d3e5ed8/attachment.html>
-------------- next part --------------
2019-01-12 13:37:42,453 - rest-api-0-output - INFO - Current Message(event): {'feed.name': 'bambenek', 'time.observation': '2019-01-12T10:40:24+00:00', 'source.fqdn': 'dvzosagabardinedazyx.com', 'classification.type': 'c&c', 'raw': 'ZHZ6b3NhZ2FiYXJkaW5lZGF6eXguY29tLERvbWFpbiB1c2VkIGJ5IGJhbmpvcmkgLSBub3QgZGF0ZSBzZWVkZWQsMjAxOS0wMS0xMixodHRwOi8vb3NpbnQuYmFtYmVuZWtjb25zdWx0aW5nLmNvbS9tYW51YWwvYmFuam9yaS50eHQNCg==', 'feed.accuracy': 90, 'classification.taxonomy': 'malicious code'}.
2019-01-12 13:37:42,454 - rest-api-0-output - INFO - Bot will continue in 0 seconds.
2019-01-12 13:38:12,487 - rest-api-0-output - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/requests/packages/urllib3/connectionpool.py", line 355, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/requests/packages/urllib3/connectionpool.py", line 564, in urlopen
    _pool=self, _stacktrace=stacktrace)
  File "/usr/local/lib/python3.6/dist-packages/requests/packages/urllib3/util/retry.py", line 245, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/dist-packages/requests/packages/urllib3/packages/six.py", line 310, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/requests/packages/urllib3/connectionpool.py", line 518, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python3.6/dist-packages/requests/packages/urllib3/connectionpool.py", line 357, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.6/dist-packages/requests/packages/urllib3/connectionpool.py", line 285, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='', port=8080): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.0a1-py3.6.egg/intelmq/lib/bot.py", line 163, in start
    self.process()
  File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.0a1-py3.6.egg/intelmq/bots/outputs/restapi/output.py", line 36, in process
    **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 504, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 433, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='', port=8080): Read timed out. (read timeout=30)
2019-01-12 13:38:12,488 - rest-api-0-output - INFO - Current Message(event): {'feed.name': 'bambenek', 'time.observation': '2019-01-12T10:40:24+00:00', 'source.fqdn': 'dvzosagabardinedazyx.com', 'classification.type': 'c&c', 'raw': 'ZHZ6b3NhZ2FiYXJkaW5lZGF6eXguY29tLERvbWFpbiB1c2VkIGJ5IGJhbmpvcmkgLSBub3QgZGF0ZSBzZWVkZWQsMjAxOS0wMS0xMixodHRwOi8vb3NpbnQuYmFtYmVuZWtjb25zdWx0aW5nLmNvbS9tYW51YWwvYmFuam9yaS50eHQNCg==', 'feed.accuracy': 90, 'classification.taxonomy': 'malicious code'}.
2019-01-12 13:38:12,489 - rest-api-0-output - INFO - Dumping message from pipeline to dump file.