Dear community,
Version 2.1.2, a bugfix release, is out which contains various bugfixes.
Install documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/UPGRADING.md
The full changelog:
### Core
- `__init__`: Resolve absolute path for `STATE_FILE_PATH` variable
(resolves `..`).
- `intelmq.lib.utils`:
- log: Do not raise an exception if logging to neither file nor syslog
is requested.
- logging StreamHandler: Colorize all warning and error messages red.
- logging FileHandler: Strip all shell colorizations from the messages
(#1436).
- `intelmq.lib.message`:
- `Message.to_json`: Set `sort_keys=True` to get reproducible results.
- `drop_privileges`: Handle situations where the user or group
`intelmq` does not exist.
- `intelmq.lib.pipeline`:
- `Amqp._send` and `Amqp._acknowledge`: Log traceback in debug mode in
case of errors and necessary re-connections.
- `Amqp._acknowledge`: Reset delivery tag if acknowledge was successful.
### Bots
#### Collectors
- `intelmq.bots.collectors.misp.collector`:
- Add compatibility with current pymisp versions and versions released
after January 2020 (PR #1468).
#### Parsers
- `intelmq.bots.parsers.shadowserver.config`: Add some missing fields
for the feed `accessible-rdp` (#1463).
- `intelmq.bots.parsers.shadowserver.parser`:
- Feed-detection based on file names: The prefixed date is optional now.
- Feed-detection based on file names: Re-detect feed for every report
received (#1493).
#### Experts
- `intelmq.bots.experts.national_cert_contact_certat`: Handle empty
responses by server (#1467).
- `intelmq.bots.experts.maxmind_geoip`: The script `update-geoip-data`
now requires a license key as second parameter because of upstream
changes (#1484)).
#### Outputs
- `intelmq.bots.outputs.restapi.output`: Fix logging of response body if
response status code was not ok.
### Documentation
- Remove some hardcoded `/opt/intelmq/` paths from code comments and
program outputs.
### Packaging
- debian/rules: Only replace `/opt/intelmq/` with LSB-paths in some
certain files, not the whole tree, avoiding wrong replacements.
- debian/rules and debian/intelmq.install: Do install the examples
configuration directly instead of working around the abandoned examples
directory.
### Tests
- `lib/test_utils`: Skip some tests on Python 3.4 because
`contextlib.redirect_stdout` and `contextlib.redirect_sterr` are not
supported on this version.
- Travis: Stop running tests with all optional dependencies on Python
3.4, as more and more libraries are dropping support for it. Tests on
the core and code without non-optional requirements are not affected.
- `tests.bots.parsers.html_table`: Make tests independent of current year.
### Tools
- `intelmqctl upgrade-config`: Fix missing substitution in error message
"State file %r is not writable.".
### Known issues
- bots trapped in endless loop if decoding of raw message fails (#1494)
- intelmqctl status of processes: need to check bot id too (#1492)
- MongoDB authentication: compatibility on different MongoDB and pymongo
versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear IntelMQ users,
Sebix mentioned something that might be relevant for everyone using IntelMQ (or actually maxmind for the matter).
(quote) "MaxMind has always been committed to an individual’s right to privacy on the internet. We welcome the burgeoning privacy regulations, such as GDPR and CCPA, for the benefit they can provide to internet citizens. However, these new legislative measures place restrictions that impact our ability to continue distributing our GeoLite2 databases on a public page under the Creative Commons Attribution-ShareAlike 4.0 International License."
Maxmind has decided to change the download mechanism of the maxmind GeoLite 2 database. You now need to be registered and need a license key.
Since I know that many IntelMQ users rely on maxmind for geolocation, you might want to register there and get a new license key and make sure it can be used for the most recent database version.
More info at https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-us…
All the best and a happy 2020!
Aaron.
--
// L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
we again collected a bunch of bugfixes in the last weeks, coming almost
one month after 2.1.0.
Install documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/UPGRADING.md
The full changelog:
### Configuration
- Default configuration:
- Remove discontinued feed "Feodo Tracker Domains" from default
configuration.
- Add "Feodo Tracker Browse" feed to default configuration.
### Core
- `intelmq.lib.pipeline`: AMQP: using port 15672 as default (like
RabbitMQ's defaults) for the monitoring interface for getting
statistical data (`intelmqctl_rabbitmq_monitoring_url`).
- `intelmq.lib.upgrades`: Added a generic upgrade function for
harmonization, checking of all message types, it's fields and their types.
- `intelmq.lib.utils`:
- `TimeoutHTTPAdapter`: A subclass of `requests.adapters.HTTPAdapter`
with the possibility to set the timeout per adapter.
- `create_request_session_from_bot`: Use the `TimeoutHTTPAdapter` with
the user-defined timeout. Previously the timeout was not functional.
### Bots
#### Parsers
- `intelmq.bots.parsers.shadowserver.parser`: Fix logging message if the
parameter `feedname` is not present.
- `intelmq.bots.parsers.shodan.parser`: Also add field
`classification.identifier` (`'network-scan'`) in minimal mode.
- `intelmq.bots.parsers.spamhaus.parser_cert`: Add support for category
`'misc'`.
- `intelmq.bots.parsers.cymru.parser_cap_program`:
- Add support for phishing events without URL.
- Add support for protocols >= 143 (unassigned, experiments, testing,
reserved), saving the number to extra, as the data would be bogus.
- `intelmq.bots.parsers.microsoft.parser_bingmurls`:
- Save the `Tags` data as `source.geolocation.cc`.
#### Experts
- `intelmq.bots.experts.modify.expert`: Fix bug with setting non-string
values (#1460).
#### Outputs
- `intelmq.bots.outputs.smtp`:
- Allow non-existent field in text formatting by using a default value
`None` instead of throwing errors.
- Fix Authentication (#1464).
- Fix sending to multiple recipients (#1464).
### Documentation
- Feeds:
- Fix configuration of `Feodo Tracker Browse` feed.
- Bots:
- Sieve expert: Document behavior of `!=` with lists.
### Tests
- Adaption and extension of the test cases to the changes.
### Tools
- `intelmq.bin.intelmqctl`:
- check: Check if running the upgrade function for harmonization is
necessary.
- upgrade-config: Run the upgrade function for harmonization.
- `intelmqctl restart` did throw an error as the message for
restarting was not defined (#1465).
### Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo
versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
After updating to version 2.1.0 the file collector raise exceptions.InvalidKey(key).
How can I find out what the problem is, this bot has worked very well under 1.1.1?
Please help
Regards Majid
Bot has found a problem.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start
self.process()
File "/usr/local/lib/python3.5/dist-packages/intelmq/bots/collectors/file/collector_file.py", line 66, in process
template.add("extra.file_name", f)
File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/message.py", line 234, in add
raise exceptions.InvalidKey(key)
intelmq.lib.exceptions.InvalidKey: invalid key 'extra.file_name'
Majid Salehi Ghamsari
Wiss. Mitarbeiter Digital Public Services
Tel: +49 30 3463-7118
Fax: +49 30 3463-99-7118
E-Mail: majid.salehi.ghamsari(a)fokus.fraunhofer.de
Fraunhofer-Institut
für Offene Kommunikationssysteme FOKUS
Kaiserin-Augusta-Allee 31
10589 Berlin
http://www.fokus.fraunhofer.de
Dear community,
after installing intelmq from git. Ii get still a lot of errors. How can I fix these?
Calling intelmq upgrade fixes some of error but not all of them.
Majid Salehi
git clone https://github.com/certtools/intelmq.git /opt/dev_intelmq
cd /opt/dev_intelmq
git fetch
git checkout tags/2.1.0 -b 2.1.0
pip3 install .
sudo -u intelmq /usr/local/bin/intelmqctl upgrade-config
Found previous version 1.1.1 in state file.
Upgrading to version 1.1.2.
Search for discontinued feodotracker domains feed: Upgrade failed: The discontinued feed "Feodo Tracker Domains" has been found as bo t 'abusech-feodo-domains-collector'. Remove it yourself please.
Some migration did not succeed or manual intervention is needed. Look at the output above. Afterwards, re-run this program.
root@csp-intelmq:/scripts# sudo -u intelmq /usr/local/bin/intelmqctl upgrade-config
Found previous version 1.1.1 in state file.
Upgrading to version 1.1.2.
Search for discontinued feodotracker domains feed: Upgrade failed: The discontinued feed "Feodo Tracker Domains" has been found as bo t 'abusech-feodo-domains-collector'. Remove it yourself please.
Some migration did not succeed or manual intervention is needed. Look at the output above. Afterwards, re-run this program.
root@csp-intelmq:/scripts# sudo -u intelmq /usr/local/bin/intelmqctl check
Reading configuration files.
Checking defaults configuration.
Checking runtime configuration.
Checking runtime and pipeline configuration.
Checking harmonization configuration.
Checking for bots.
Upgrade function v200_defaults_statistics not completed (successfully). Please run 'intelmqctl upgrade-config'.
Upgrade function v200_defaults_broker not completed (successfully). Please run 'intelmqctl upgrade-config'.
Upgrade function v112_feodo_tracker_domains not completed (successfully). Please run 'intelmqctl upgrade-config'.
Upgrade function v200_defaults_ssl_ca_certificate not completed (successfully). Please run 'intelmqctl upgrade-config'.
Upgrade function v202_fixes not completed (successfully). Please run 'intelmqctl upgrade-config'.
Upgrade function v210_deprecations not completed (successfully). Please run 'intelmqctl upgrade-config'.
No issues found.
Dear community,
Given the vast amount of changes, additions and new features, it's time
to mark a new feature release! Thanks to all the contributors who
participate in this community project! IntelMQ gained a lot of new bots
and features in the last months.
Install documentation:
https://github.com/certtools/intelmq/blob/2.1.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.0/docs/UPGRADING.md
Full changelog:
### Core
- `intelmq.lib.harmonization`:
- Use correct parent classes.
- Add `DateTime.convert` as interface for all existing conversion
functions.
- add `DateTime.convert_from_format`.
- add `DateTime.convert_from_format_midnight`.
- add `DateTime.convert_fuzzy`.
- `intelmq.lib.pipeline`:
- Redis: Use single connection client if calling bot is not
multithreaded. Gives a small speed advantage.
- Require the bot instance as parameter for all pipeline classes.
- New internal variable `_has_message` to keep the state of the pipeline.
- Split receive and acknowledge into public-facing and private methods.
- Add `reject_message` method to the Pipeline class for explicit
requeue of messages.
- AMQP:
- Make exchange configurable.
- If exchange is set, the queues are not declared, the queue name is
for routing used by exchanges.
- `intelmq.lib.bot`:
- Log message after successful bot initialization, no log message
anymore for ready pipeline.
- Use existing current message if receive is called and the current
message still exists.
- Fix handling of received messaged after a sighup that happend during
a blocking receving connection using explicit rejection (#1438).
- New method `_parse_common_parameters` called before `init` to parse
commonly used argument. Currently supported: `extract_files`.
- `intelmq.lib.test`:
- Fix the tests broker by providing the testing pipeline.
- `intelmq.lib.utils`:
- `unzip`:
- new parameter `return_names` to optionally return the file names.
- support for zip
- new parameters `try_zip`, `try_gzip` and `try_tar` to control
which compressions are tried.
- rewritten to an iterative approach
- add `file_name_from_response` to extract a file name from a Response
object for downloaded files.
- `intelmq.lib.upgrades`: Added `v210_deprecations` for deprecated
parameters.
### Harmonization
- Add extra to reports.
### Bots
#### Collectors
- `intelmq.bots.collectors.http.collector_http`:
- More extensive usage of `intelmq.lib.utils.unzip`.
- Save the file names in the report if files have been extracted form
an archive.
- `intelmq.bots.collectors.rt.collector_rt`:
- Save ticket information/metadata in the extra fields of the report.
- Support for RT 3.8 and RT 4.4.
- New parameters `extract_attachment` and `extract_download` for
generic archive extraction and consistency. The parameter
`unzip_attachment` is deprecated.
- `intelmq.bots.collectors.mail.*`: Save email information/metadata in
the extra fields of the report. See the bots documentation for a
complete list of provided data.
- `intelmq.bots.collectors.mail.collector_mail_attach`: Check for
existence/validity of the `attach_regex` parameter.
- Use the lib's `unzip` function for uncompressing attachments and use
the .
- `intelmq.bots.collectors.mail.collector_mail_url`: Save the file
name of the downloaded file as `extra.file_name`.
- `intelmq.bots.collectors.amqp.collector_amqp`: New collector to
collect data from (remote) AMQP servers, for bot IntelMQ as well as
external data.
- use default SSL context for client purposes, fixes compatibility
with python < 3.6 if TLS is used.
#### Parsers
- `intelmq.bot.parsers.html_table.parser`:
* New parameter "html_parser".
* Use time conversion functions directly from
`intelmq.lib.harmonization.DateTime.convert`.
- Limit lxml dependency on 3.4 to < 4.4.0 (incompatibility).
- `intelmq.bots.parsers.netlab_360.parser`: Add support for hajime scanners.
- `intelmq.bots.parsers.hibp.parser_callback`: A new parser to parse
data retrieved from a HIBP Enterprise Subscription.
- `intelmq.bots.parsers.shadowserver.parser`:
- Ability to detect the feed base on the reports's field
`extra.file_name`, so the parameter `feedname` is no longer required and
one configured parser can parse any feed (#1442).
#### Experts
- Add geohash expert.
- `intelmq.bot.experts.generic_db_lookup.expert`
- new optional parameter `engine` with `postgresql` (default) and
`sqlite` (new) as possible values.
#### Outputs
- Add `intelmq.bots.outputs.touch.output`.
- `intelmq.bot.outputs.postgresql.output`:
- deprecated in favor of `intelmq.bot.outputs.sql.output`
- Compatibility shim will be available in the 2.x series.
- `intelmq.bot.outputs.sql.output` added generic SQL output bot.
Comparted to
- new optional parameter `engine` with `postgresql` (default) and
`sqlite` (new) as possible values.
- `intelmq.bots.outputs.stomp.output`: New parameters
`message_hierarchical_output`, `message_jsondict_as_string`,
`message_with_type`, `single_key`.
### Documentation
- Feeds:
- Add ViriBack feed.
- Add Have I Been Pwned Enterprise Callback.
- `intelmq.tests.bots.outputs.amqptopic.test_output`: Added.
- Move the documentation of most bots from separate README files to the
central Bots.md and feeds.yaml files.
### Tests
- Travis:
- Use UTC timezone.
- Tests for `utils.unzip`.
- Add a new asset: Zip archive with two files, same as with tar.gz archive.
- Added tests for the Mail Attachment & Mail URL collectors.
- Ignore logging-tests on Python 3.7 temporarily (#1342).
### Tools
- intelmqctl:
- Use green and red text color for some interactive output to indicate
obvious errors or the absence of them.
- intelmqdump:
- New edit action `v` to modify a message saved in the dump (#1284).
### Contrib
* malware name mapping:
* Add support for MISP treat actors data, see it's README for more
information.
* And handle empty synonyms in misp's galxies data.
* Move apply-Script to the new EventDB directory
* EventDB: Scripts for applying malware name mapping and domain suffixes
to an EventDB.
### Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo
versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
The new 2.1.0 version of the intelmq manager improves the error
reporting in the frontend and allows to connect collectors to experts
and experts to parsers.
Thanks to Edvard for his work on this tool!
Installation instructions:
https://github.com/certtools/intelmq-manager/blob/2.0.0/docs/INSTALL.md
Full changelog:
### Backend
- Fix mispelling of the environmental variable
`INTELMQ_MANGER_CONTROLER_CMD` to `INTELMQ_MANGER_CONTROLLER_CMD` (you
might be required to add the double 'l' to your nginx/apache server
configuration)
- When displaying a command to be replicated by debugging user, the
string "sudo -u {webserver user}" string is prepended so that linux user
do not have to bother with sudoing himself on the commonly used user
"www-data" – which often can't be sudoed to (no bash provided due to
good security measures). (Used in monitor and error reporting.)
### Frontend
- Error reporting
- Click will enlarge the dialog that contains much more useful info,
notably the very command that failed so that it can be easily reproduced.
- Error messages are shuffled only when minimized, not when maximized.
That would disturb the user trying to read the details.
- Invalid Syntax Error message removed. Till now, all error messages
generated the string that JSON received is invalid – that wasn't needed,
we knew it's invalid because it contained string message.
- Double click does not close log window anymore since it would
interfere with the user trying to select whole text by mouse.
- Escape minimizes the reporting.
- For common seen errors, a tip is displayed (preferable with a link
to the Github manual).
### Pages
#### Configuration
- Node group Collector now may connect to Expert and Expert can connect
to Parser, however you receive a warning that it is not very common.
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
Today I released version 2.0.2 of intelmq with a bunch of bugfixes, see
the changelog below. A feature release is coming soon too.
Install documentation:
https://github.com/certtools/intelmq/blob/2.0.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.0.2/docs/UPGRADING.md
As always: read the NEWS file, upgrade according to the documentation
and have fun! If you get any errors, please report them here or in the
bug tracker.
Sebastian
Full changelog:
### Core
- `intelmq.lib.bot.CollectorBot`: Support the deprecated parameter
`feed` until version 2.2 as the documentation was not properly updated
(#1445).
- `intelmq.lib.bot.Bot`:
- `_dump_message`: Wait for up to 60 seconds instead of 50 if the dump
file is locked (the log message was said 60, but the code was for only 50).
- `intelmq.lib.upgrades.v202_fixes`
- Migration of deprecated parameter `feed` for Collectors.
- Ripe expert parameter `query_ripe_stat_ip` was not correctly
configured in `v110_deprecations`, now use `query_ripe_stat_asn` as
default if it does not exist.
- `intelmq.lib.upgrades.v110_deprecations`: Fix upgrade of ripe expert
configuration.
- `intelmq.lib.bot_debugger`:
- Fix handling of empty messages generated by parser when user wanted
to show the result by "--show-sent" flag.
- Fix handling of sent messages for bots using the `path_permissive`
paramter (#1453).
- `intelmq.lib.pipeline.Amqp`:
- use default SSL context for client purposes, fixes compatibility
with python < 3.6 if TLS is used.
- Reconnect once on sending messages if disconnect detected.
### Bots
#### Collectors
- `intelmq.bots.collectors.api.collector_api`:
- Handle non-existing IO loop in shutdown.
- Close socket on shutdown, fixes reloading.
- Marked as non-threadable.
- `intelmq.bots.collectors.rt.collector_rt`: Check for matching URLs if
no `attachment_regex` is given.
- `intelmq.bots.collectors.stomp.collector_stomp`: Handle disconnects by
actively reconnecting.
#### Parsers
- `intelmq.bots.cymru.parser_cap_program`: Fix parsing of the new
`$certname_$date.txt` report format (#1443):
- Support protocol ICMP.
- Fix error message for unsupported protocols.
- Support fields `destination_port_numbers`, `port`.
- Support for all proxy types without ports.
- Use Country Code of AS as `source.geolocation.cc`.
- Support for 'scanner' and 'spam' categories.
- Handle bogus lines with missing separator.
- Fix bug preventing use of old format after using the new format.
- Handle postfix ` (total_count:..)` for destination port numbers.
#### Experts
- `intelmq.bots.experts.cymru_whois.expert`: Add optional parameter
`overwrite`, current behavior was `True`, default if not given is `True`
now, will change to `False` in 3.0.0 (#1452, #1455).
- `intelmq.bots.experts.modify.expert`: Add optional parameter
`overwrite`, current behavior was `True`, default if not given is `True`
now, will change to `False` in 3.0.0 (#1452, #1455).
- `intelmq.bots.experts.reverse_dns.expert`: Add optional parameter
`overwrite`, current behavior was `True`, default if not given is `True`
now, will change to `False` in 3.0.0 (#1452, #1455).
#### Outputs
- `intelmq.bots.outputs.amqptopic.output`: use default SSL context for
client purposes, fixes compatibility with python < 3.6 if TLS is used.
### Packaging
- Rules:
- Exclude intelmqsetup tool in packages
- Include update-rfiprisk-data in packages
### Tests
- Tests for `intelmq.lib.upgrades.v202_fixes`.
- Tests for `intelmq.lib.upgrades.v110_deprecations`.
- Extended tests for `intelmq.bots.parser.cymru.parser_cap_program`.
### Tools
- intelmqctl:
- More and more precise logging messages for botnet starting and
restarting, enable and disable.
- No error message for disabled bots on botnet reload.
- Fix `upgrade-conf` is state file is empty or not existing.
- Use arpgarse's `store_true` action for flags instead of `store_const`.
- If the loading of the defaults configuration failed, a variable
definition was missing and causing an exception (#1456).
### Contrib
- Check MK Statistics Cronjob:
- Use `statistics_*` parameters.
- Make file executable
- Handle None values in `*.temporary.*` keys and treat them as 0.
- systemd:
- Add `PIDFile` parameter to service file.
### Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo
versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
I just released version 2.0.1 of IntelMQ, the first bugfix release for
the 2.x series, published in May. This is also the first 2.0 release
which will hit the deb/rpm stable repositories. If you are using them,
you'll receive the update in the next few hours.
The version further introduces a new intelmqctl command: intelmqctl
upgrade-conf. This command takes care of changed configurations, or
newly added parameters. Call it after every upgrade, as documented in
the Upgrade documentation linked below. The tools updates the
configuration if necessary and creates backup files on it's own. If it
can't fix the issues automatically, it will show you an error message. A
side effect is, that intelmq now saves a "state" of itself in
/var/lib/intelmq/state.json or /opt/intelmq/var/lib/state.json depending
on your setup.
Install documentation:
https://github.com/certtools/intelmq/blob/2.0.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.0.1/docs/UPGRADING.md
As always: read the NEWS file, upgrade according to the documentation
and have fun! If you get any errors, please report them here or in the
bug tracker.
The full changelog:
### Core
- `intelmq.lib.harmonization`:
- `IPAddress`: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
- All types: Handle `None` for validation and sanitation gracefully.
- `intelmq.lib.bot`:
- fix parameters of ParserBot and CollectorBot constructors, allowing
`intelmqctl run` with these bots again (#1414).
- Also run `rate_limit` after retry counter reset (#1431).
- `__version_info__`:
- is now available in the top level module.
- uses integer values now instead of strings for numerical version parts
- Also provide (empty) `ROOT_DIR` for non-pip installations.
- `intelmq.lib.upgrades`: New library file `upgrades` with upgrade
functions.
- `intelmq.lib.utils`:
- New function `setup_list_logging` for intelmqctl check an possibly
others.
- Fix return values (#1423).
- New function `version_smaller` for version comparisons.
- New function `lazy_int` for version conversions.
- `parse_logline`: Handle thread IDs.
- `log` takes a new argument `logging_level_stream` for the logging
level of the console handler.
- New constant `LOG_FORMAT_SIMPLE`, used by intelmqctl.
- New function `write_configuration` to write dicts to files in the
correct json formatting.
- New function `create_request_session_from_bot`.
- `intelmq.lib.pipeline`:
- AMQP:
- Actually use `source/destination_pipeline_amqp_virtual_host`
parameter.
- Support for SSL with `source/destination_pipeline_ssl` parameter.
- pipeline base class: add missing dummy methods.
- Add missing return types.
- Redis: Evaluate return parameter of queue/key deletion.
- Variable `STATE_FILE_PATH` added.
### Development
- `intelmq.bin.intelmq_gen_docs`: For yaml use `safe_load` instead of
unsafe `load`.
### Harmonization
- IPAddress type: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
- TLP: Sanitation handles now more cases: case-insensitive prefixes and
arbitrary whitespace between the prefix and the value (#1420).
### Bots
#### Collectors
- `intelmq.bots.collectors.http.collector_http`: Use
`utils.create_request_session_from_bot`.
- `intelmq.bots.collectors.http.collector_http_stream`: Use
`utils.create_request_session_from_bot` and thus fix some retries on
connection timeouts.
- `intelmq.bots.collectors.mail.collector_mail_url`: Use
`utils.create_request_session_from_bot`.
- `intelmq.bots.collectors.microsoft.collector_interflow`: Use
`utils.create_request_session_from_bot` and thus fix retries on
connection timeouts.
- `intelmq.bots.collectors.rt.collector_rt`: Use
`utils.create_request_session_from_bot` and thus fix retries on
connection timeouts.
- `intelmq.bots.collectors.twitter.collector_twitter`: Use
`utils.create_request_session_from_bot` and thus fix retries on
connection timeouts for non-twitter connections.
#### Parsers
- `intelmq.bots.parsers.n6.parser_n6stomp`: use `malware-generic`
instead of `generic-n6-drone` for unknown infected system events.
- `intelmq.bots.parsers.abusech.parser_ip`: Support LastOnline column in
feodo feed (#1400) and use it for `time.source` if available.
- Use lower case malware names as default, should not make a
difference in practice.
- Fix handling of CSV header for feodotracker (#1417, #1418).
- `intelmq.bots.parsers.netlab_360.parser`: Detect feeds with `https://`
too.
#### Experts
- `intelmq.bots.experts.generic_db_lookup`: Recommend psycopg2-binary
package.
- `intelmq.bots.experts.modify.expert`:
- Compile regular expressions (all string rules) at initialization,
improves the speed.
- Warn about old configuration style deprecation.
- `intelmq.bots.experts.do_portal.expert`:
- Use `utils.create_request_session_from_bot` and thus fix retries on
connection timeouts (#1432).
- Treat "502 Bad Gateway" as timeout which can be retried.
- `intelmq.bots.experts.ripe.expert`: Use
`utils.create_request_session_from_bot` and thus fix retries on
connection timeouts.
- `intelmq.bots.experts.url2fqdn.expert`: Support for IP addresses in
hostnames (#1416).
- `intelmq.bots.experts.national_cert_contact_certat.expert`: Use
`utils.create_request_session_from_bot` and thus fix retries on
connection timeouts.
#### Outputs
- `intelmq.bots.outputs.postgresql`: Recommend psycopg2-binary package.
- `intelmq.bots.outputs.amqptopic`:
- Shutdown: Close connection only if connection exists.
- Add support for pika > 1. Pika changed the way it indicates
(Non-)Acknowledgments of sent messages.
- Gracefully handle unroutable messages and give advice.
- Support for connections without authentication.
- Replace deprecated parameter `type` with `exchange_type` for
`exchange_declare`, supporting pika >= 0.11 (#1425).
- New parameters `message_hierarchical_output`, `message_with_type`,
`message_jsondict_as_string`.
- New parameter `use_ssl` for SSL connections.
- New parameter `single_key` for sending single fields instead of the
full event.
- `intelmq.bots.outputs.mongodb.output`: Support for pymongo >= 3.0.0
(#1063, PR#1421).
- `intelmq.bots.outputs.file`: `time.*` field serialization: support for
microseconds.
- `intelmq.bots.outputs.mongodb.output`: Support for authentication in
pymongo >= 3.5 (#1062).
- `intelmq.bots.outputs.restapi.output`: Use
`utils.create_request_session_from_bot` and thus fix retries on
connection timeouts.
### Documentation
- Add certbund-contact to the ecosystem document.
- Rename the IDEA expert to "IDEA Converter".
- Add the new configuration upgrade function to the docs.
- User Guide:
- Clarify on Uninstallation
### Packaging
- Do not execute the tcp collector tests during debian and ubuntu builds
as they fail there.
### Tests
- `intelmq.lib.test`: Disable statistics for test runs of bots.
- `contrib.malware_name_mapping`: Added tests.
- Travis: Also run tests of contrib.
### Tools
- `intelmqsetup`: Only change directory ownerships if necessary.
- `intelmqctl`:
- Provide new command `upgrade-conf` to uprade configuration to a
newer version.
- Makes backups of configurations files on its own.
- Also checks for previously skipped or new functions of older
versions and catches up.
- Provides logging level on class layer.
- Fix `-q` flag for `intelmqctl list queues` by renaming its
alternative name to `--non-zero` to avoid a name collision with the
global `--quiet` parameter.
- For console output the string `intelmqctl: ` at the beginning of
each line is no longer present.
- `check`: Support for the state file added. Checks if it exists and
all upgrade functions have been executed successfully.
- Waits for up to 2 seconds when stopping a bot (#1434).
- Exits early on restart when stopping a bot did not work (#1434).
- `intelmqctl run process -m` debugging: Mock acknowledge method if
incoming message is mocked too, otherwise a different message is
acknowledged.
- Queue listing for AMQP: Support non-default monitoring URLs, see
User-Guide.
### Contrib
* logcheck rules: Adapt ignore rule to cover the instance IDs of bot names.
* malware name mapping:
- Ignore lines in mapping starting with '#'.
- Optionally include malpedia data.
- Fix command line parsing for not arguments (#1427).
- bash-completion: Support for `intelmqctl upgrade-config` added.
### Known issues
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
On the maintenance/develop branches there's a new functionality to fix
issues in the configuration, or "upgrade" the configuration. This was
some times needed in the past and only covered by entries in the NEWS
file. This includes cases like:
* Discontinued feeds
* Errors in the shipped configuration
* changed configuration syntax
* renamed parameters
* new needed default parameters
* etc.
This function also introduces the "state" file, where intelmq can keep
track of which upgrades have been performed. This would already have
been useful in the past. This also makes debugging problems easier and
maybe we can even introduce a full upgrade function some day...
So, why I am writing here is: This needs testing! Although the programs
makes backups itself, make a backup of your configuration nevertheless.
Then call
> intelmqctl upgrade-conf
And follow the output!
If you want to know what exactly happens, have a look at the
intelmq/lib/upgrades.py and intelmq/bin/intelmqctl.py files
Have fun and please report back your experiences
Sebastian
P.S.: I'd like to finish the 2.0.1 release in the next days/week, so
this will be part of it.
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg