Hi mailinglist,
I've went over intelmq logs today and that some bots keep retrying
indefinitely when they encounter a failure. However each of those bots have
*error_max_retries* set to 1 (and *error_retry_delay *to 60). Which in my
mind should limit the bot to just try one more time and than give up and
sleep (if sleep is set up) however this is not happening. Am I doing
something wrong or does this functionality work differently? My version of
intelmq is 1.1.1 in Docker on Ubuntu 18.04 as base.
Following is an example from logs os collector collecting from phishtank:
2019-03-01 20:07:51,747 - phishtank-collector - INFO - Downloading report
from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:07:52,139 - phishtank-collector - ERROR - Bot has found a
problem.
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start
self.process()
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process
raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:07:52,140 - phishtank-collector - INFO - Current
Message(event): None.
2019-03-01 20:07:52,140 - phishtank-collector - INFO - Downloading report
from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:07:52,298 - phishtank-collector - ERROR - Bot has found a
problem.
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start
self.process()
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process
raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:07:52,299 - phishtank-collector - INFO - Current
Message(event): None.
2019-03-01 20:07:52,299 - phishtank-collector - INFO - Bot will continue in
60 seconds.
2019-03-01 20:08:52,359 - phishtank-collector - INFO - Downloading report
from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:08:52,654 - phishtank-collector - ERROR - Bot has found a
problem.
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start
self.process()
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process
raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:08:52,656 - phishtank-collector - INFO - Current
Message(event): None.
2019-03-01 20:08:52,656 - phishtank-collector - INFO - Downloading report
from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:08:52,831 - phishtank-collector - ERROR - Bot has found a
problem.
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start
self.process()
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process
raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:08:52,832 - phishtank-collector - INFO - Current
Message(event): None.
2019-03-01 20:08:52,832 - phishtank-collector - INFO - Bot will continue in
60 seconds.
2019-03-01 20:09:52,891 - phishtank-collector - INFO - Downloading report
from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:09:53,138 - phishtank-collector - ERROR - Bot has found a
problem.
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start
self.process()
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process
raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:09:53,139 - phishtank-collector - INFO - Current
Message(event): None.
2019-03-01 20:09:53,139 - phishtank-collector - INFO - Downloading report
from 'https://data.phishtank.com/data/online-valid.csv'.
2019-03-01 20:09:53,315 - phishtank-collector - ERROR - Bot has found a
problem.
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py",
line 167, in start
self.process()
File
"/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py",
line 76, in process
raise ValueError('HTTP response status code was %i.' % resp.status_code)
ValueError: HTTP response status code was 509.
2019-03-01 20:09:53,317 - phishtank-collector - INFO - Current
Message(event): None.
2019-03-01 20:09:53,318 - phishtank-collector - INFO - Bot will continue in
60 seconds.
Sincerely,
Václav Brůžek
Dear community,
This is presumably the last bugfix release before the next feature
release. That is expected to be the version 2.0, not 1.2 in the next month.
The release will hit the deb/rpm repositories shortly.
Thanks to all contributors!
Installation instructions:
https://github.com/certtools/intelmq/blob/1.1.2/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.1.2/docs/UPGRADING.md
### Configuration
#### Feodotracker
* The URL of the "Feodo Tracker IPs" feed has changed. The new one is `https://feodotracker.abuse.ch/downloads/ipblocklist.csv`. If you are using this feed, adapt your configuration accordingly. The parser has been updated to support the new format.
* The feed "Feodo Tracker Domains" has been discontinued.
The full changelog:
### Core
- `intelmq.lib.bot`:
- `Bot.__handle_sighup`: Handle exceptions in `shutdown` method of bots.
### Harmonization
- FQDN: Disallow `:` in FQDN values to prevent values like '10.0.0.1:8080' (#1235).
### Bots
#### Collectors
- `intelmq.bots.collectors.stomp.collector`
- Fix name of shutdown method, was ineffective in the past.
- Ignore `NotConnectedException` errors on disconnect during shutdown.
- `intelmq.bots.collectors.mail.collector_mail_url`: Decode body if it is bytes (#1367).
- `intelmq.bots.collectors.tcp.collector`: Timeout added. More stable version.
#### Parsers
- `intelmq.bots.parsers.shadowserver`:
- Add support for the `Amplification-DDoS-Victim`, `HTTP-Scanners`, `ICS-Scanners` and `Accessible-Ubiquiti-Discovery-Service` feeds (#1368, #1383)
- `intelmq.bots.parsers.microsoft.parser_ctip`:
- Workaround for mis-formatted data in `networkdestinationipv4` field (since 2019-03-14).
- Ignore "hostname" ("destination.fqdn") if it contains invalid data.
- `intelmq.bots.parsers.shodan.parser`:
- In `minimal_mode`:
- Fix the parsing, previously only `source.geolocation.cc` and `extra.shodan` was correctly filled with information.
- Add a `classification.type` = 'other' to all events.
- Added tests for this mode.
- Normal mode:
- Fix the parsing of `timestamp` to `time.source in the normal mode, previously no timezone information has been added and thus every event raised an exception.
- ISAKMP: Ignore `isakmp.aggressive`, as the content is same as `isakmp` or less.
- `intelmq.bots.parsers.abusech.parser_ip`: Re-structure the bot and support new format of the changed "Feodo Tracker Domains" feed.
- `intelmq.bots.parsers.n6.parser`:
- Add parsing for fields "confidence", "expires" and "source".
- Add support for type "bl-other" (category "other").
#### Experts
- `intelmq.bots.experts.sieve.expert`: Fix key definition to allow field names with numbers (`malware.hash.md5`/`sha1`, #1371).
#### Outputs
- `intelmq.bots.outputs.tcp.output`: Timeout added. When no separator used, awaits that every message is acknowledged by a simple "Ok" string to ensure more stability.
### Documentation
- Install: Update operating system versions
- Sieve Expert: Fix `elsif` -> `elif`.
- Rephrase the description of `time.*` fields.
- Feeds: New URL and format of the "Feodo Tracker IPs" feed. "Feodo Tracker Domains" has been discontinued.
### Packaging
### Tests
- Add missing `__init__.py` files in 4 bot's test directories. Previously these tests have never been executed.
- `intelmq.lib.test`: Allow bot test class names with an arbitrary postfix separated by an underscore. E.g. `TestShodanParserBot_minimal`.
### Tools
- intelmqctl:
- status: Show commandline differences if a program with the expected PID could be found, but they do not match (previous output was `None`).
- Use logging level from defauls configuration if possible, otherwise intelmq's internal default. Previously, DEBUG was used unconditionally.
### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is restarted (#952).
- stomp collector bot constantly uses 100% of CPU (#1364).
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg