Dear community,
we again collected a bunch of bugfixes in the last weeks, coming almost
one month after 2.1.0.
Install documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/UPGRADING.md
The full changelog:
### Configuration
- Default configuration:
- Remove discontinued feed "Feodo Tracker Domains" from default
configuration.
- Add "Feodo Tracker Browse" feed to default configuration.
### Core
- `intelmq.lib.pipeline`: AMQP: using port 15672 as default (like
RabbitMQ's defaults) for the monitoring interface for getting
statistical data (`intelmqctl_rabbitmq_monitoring_url`).
- `intelmq.lib.upgrades`: Added a generic upgrade function for
harmonization, checking of all message types, it's fields and their types.
- `intelmq.lib.utils`:
- `TimeoutHTTPAdapter`: A subclass of `requests.adapters.HTTPAdapter`
with the possibility to set the timeout per adapter.
- `create_request_session_from_bot`: Use the `TimeoutHTTPAdapter` with
the user-defined timeout. Previously the timeout was not functional.
### Bots
#### Parsers
- `intelmq.bots.parsers.shadowserver.parser`: Fix logging message if the
parameter `feedname` is not present.
- `intelmq.bots.parsers.shodan.parser`: Also add field
`classification.identifier` (`'network-scan'`) in minimal mode.
- `intelmq.bots.parsers.spamhaus.parser_cert`: Add support for category
`'misc'`.
- `intelmq.bots.parsers.cymru.parser_cap_program`:
- Add support for phishing events without URL.
- Add support for protocols >= 143 (unassigned, experiments, testing,
reserved), saving the number to extra, as the data would be bogus.
- `intelmq.bots.parsers.microsoft.parser_bingmurls`:
- Save the `Tags` data as `source.geolocation.cc`.
#### Experts
- `intelmq.bots.experts.modify.expert`: Fix bug with setting non-string
values (#1460).
#### Outputs
- `intelmq.bots.outputs.smtp`:
- Allow non-existent field in text formatting by using a default value
`None` instead of throwing errors.
- Fix Authentication (#1464).
- Fix sending to multiple recipients (#1464).
### Documentation
- Feeds:
- Fix configuration of `Feodo Tracker Browse` feed.
- Bots:
- Sieve expert: Document behavior of `!=` with lists.
### Tests
- Adaption and extension of the test cases to the changes.
### Tools
- `intelmq.bin.intelmqctl`:
- check: Check if running the upgrade function for harmonization is
necessary.
- upgrade-config: Run the upgrade function for harmonization.
- `intelmqctl restart` did throw an error as the message for
restarting was not defined (#1465).
### Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo
versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
After updating to version 2.1.0 the file collector raise exceptions.InvalidKey(key).
How can I find out what the problem is, this bot has worked very well under 1.1.1?
Please help
Regards Majid
Bot has found a problem.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start
self.process()
File "/usr/local/lib/python3.5/dist-packages/intelmq/bots/collectors/file/collector_file.py", line 66, in process
template.add("extra.file_name", f)
File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/message.py", line 234, in add
raise exceptions.InvalidKey(key)
intelmq.lib.exceptions.InvalidKey: invalid key 'extra.file_name'
Majid Salehi Ghamsari
Wiss. Mitarbeiter Digital Public Services
Tel: +49 30 3463-7118
Fax: +49 30 3463-99-7118
E-Mail: majid.salehi.ghamsari(a)fokus.fraunhofer.de
Fraunhofer-Institut
für Offene Kommunikationssysteme FOKUS
Kaiserin-Augusta-Allee 31
10589 Berlin
http://www.fokus.fraunhofer.de