[IntelMQ-dev] IntelMQ & API & Manager Version 2.3.0!

[Sebastian Wagner]

Dear community,

Another important intermediate step on the way to IntelMQ 3.0 is
completed - IntelMQ 2.3.0 is really final as of today. Many thanks to
all the contributors and supporters around the world - the major changes
would never be possible without you!

One thing you will immediately notice its a completely new component:
the IntelMQ API. It originates from the IntelMQ Manager, but is a
complete rewrite of it's backend in Python (finally!) financed by SUNET
and realised by Intevation. We have then split the Backend off into a
separate API. This means, that to run the Manager, you need the API as well.

The installation instructions:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade instructions:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

All packages have been published to pypi, the deb/rpm-repositories and
dockerhub.

You can read a summary of the most important changes here:
https://cert.at/en/blog/2021/3/intelmq-230-api-docker-shadowserver-reports-api-documentation

The new or heavily changed bots are:

* CZ.nic HAAS and PROKI Parsers, by Filip Pokorný and Edvard Rejthar
(CSIRT.CZ)
* ESET Collector and Parser, by Mikk Margus Möll (CERT.EE)
* Kafka Collector, by Birger Schacht (CERT.at)
* Key-Value Parser, by Karl-Johan Karlsson (Linköping University)
* Request Tracker Output, by Marius Urkis (NRDCS.LT)
* Shadowserver Reports API and JSON Parser, by Birger Schacht (CERT.at)
* Splunk Saved Search Expert, by Karl-Johan Karlsson (Linköping University)
* Threshold Expert, by Karl-Johan Karlsson (Linköping University)
* Shadowserver CSV & JSON Parser: Support for the feeds MSRDPUDP,
Vulnerable-HTTP, Sinkhole DNS and fixes for existing feed mappings, by
Sebastian Waldbauer and Sebastian Wagner (CERT.at)
* HTTP collector: PGP signature check functionality, by sinus-x
* Several Experts (1, 2, 3, 4): Integrated local database update
mechanisms, by Filip Pokorný (CSIRT.CZ)

Please find below the full changelog.

best regards
Sebastian

IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu
16.04), the minimum supported Python version is 3.6.

### Configuration

### Core
- `intelmq.lib.bot`:
  - `ParserBot.recover_line_json_stream`: Make `line` parameter
optional, as it is not needed for this method (by Sebastian Wagner).
  - `Bot.argparser`: Added class method `_create_argparser` (returns
`argparse.ArgumentParser`) for easy command line arguments parsing
(PR#1586 by Filip Pokorný).
  - Runtime configuration does not necessarily need a parameter entry
for each block. Previously at least an empty block was required (PR#1604
by Filip Pokorný).
  - Allow setting the pipeline host and the Redis cache host by
environment variables for docker usage (PR#1669 by Sebastian Waldbauer).
  - Better logging message for SIGHUP handling if the handling of the
signal is not delayed (by Sebastian Wagner).
- `intelmq.lib.upgrades`:
  - Add upgrade function for removal of *HPHosts Hosts file* feed and
`intelmq.bots.parsers.hphosts` parser (#1559, by Sebastian Wagner).
- `intelmq.lib.exceptions`:
  - `PipelineError`: Remove unused code to format exceptions (by
Sebastian Wagner).
- `intelmq.lib.utils`:
  - `create_request_session_from_bot`:
    - Changed bot argument to optional, uses defaults.conf as fallback,
renamed to `create_request_session`. Name
`create_request_session_from_bot` will be removed in version 3.0.0
(PR#1524 by Filip Pokorný).
    - Fixed setting of `http_verify_cert` from defaults configuration
(PR#1758 by Birger Schacht).
  - `log`: Use `RotatingFileHandler` for allow log file rotation without
external tools (PR#1637 by Vasek Bruzek).
- `intelmq.lib.harmonization`:
  - The `IPAddress` type sanitation now accepts integer IP addresses and
converts them to the string representation (by Sebastian Wagner).
  - `DateTime.parse_utc_isoformat`: Add parameter `return_datetime` to
return `datetime` object instead of string ISO format (by Sebastian Wagner).
  - `DateTime.convert`: Fix `utc_isoformat` format, it pointed to a
string and not a function, causing an exception when used (by Sebastian
Wagner).
  - `DateTime.from_timestamp`: Ensure that time zone information
(`+00:00`) is always present (by Sebastian Wagner).
  - `DateTime.__parse` now handles OverflowError exceptions from the
dateutil library, happens for large numbers, e.g. telehpone numbers (by
Sebastian Wagner).
- `intelmq.lib.upgrades`:
  - Added upgrade function for CSV parser parameter misspelling (by
Sebastian Wagner).
  - Check for existence of collector and parser for the obsolete Malware
Domain List feed and raise warning if found (#1762, PR#1771 by Birger
Schacht).

### Development
- `intelmq.bin.intelmq_gen_docs`:
  - Add bot name to the resulting feed documentation (PR#1617 by Birger
Schacht).
  - Merged into `docs/autogen.py` (PR#1622 by Birger Schacht).

### Bots
#### Collectors
- `intelmq.bots.collectors.eset.collector`: Added (PR#1554 by Mikk
Margus Möll).
- `intelmq.bots.collectors.http.collector_http`:
  - Added PGP signature check functionality (PR#1602 by sinus-x).
  - If status code is not 2xx, the request's and response's headers and
body are logged in debug logging level (#1615, by Sebastian Wagner).
- `intelmq.bots.collectors.kafka.collector`: Added (PR#1654 by Birger
Schacht, closes #1634).
- `intelmq.bots.collectors.xmpp.collector`: Marked as deprecated, see
https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html
(#1614, PR#1685 by Birger Schacht).
- `intelmq.bots.collectors.shadowserver.collector_api`:
  - Added (#1683, PR#1700 by Birger Schacht).
  - Change file names in the report to `.json` instead of the original
and wrong `.csv` (PR#1769 by Sebastian Wagner).
- `intelmq.bots.collectors.mail`: Add content of the email's `Date`
header as `extra.email_date` to the report in all email collectors
(PR#1749 by aleksejsv and Sebastian Wagner).
- `intelmq.bots.collectors.http.collector_http_stream`: Retry on common
connection issues without raising exceptions (#1435, PR#1747 by
Sebastian Waldbauer and Sebastian Wagner).
- `intelmq.bots.collectors.shodan.collector_stream`: Retry on common
connection issues without raising exceptions (#1435, PR#1747 by
Sebastian Waldbauer and Sebastian Wagner).
- `intelmq.bots.collectors.twitter.collector_twitter`:
  - Proper input validation in URLs using urllib. CWE-20, found by
GitHub's CodeQL (PR#1754 by Sebastian Wagner).
  - Limit replacement ("pastebin.com", "pastebin.com/raw") to a maximum
of one (PR#1754 by Sebastian Wagner).

#### Parsers
- `intelmq.bots.parsers.eset.parser`: Added (PR#1554 by Mikk Margus Möll).
  - Ignore invalid "NXDOMAIN" IP addresses (PR#1573 by Mikk Margus Möll).
- `intelmq.bots.parsers.hphosts`: Removed, feed is unavailable (#1559,
by Sebastian Wagner).
- `intelmq.bots.parsers.cznic.parser_haas`: Added (PR#1560 by Filip
Pokorný and Edvard Rejthar).
- `intelmq.bots.parsers.cznic.parser_proki`: Added (PR#1599 by sinus-x).
- `intelmq.bots.parsers.key_value.parser`: Added (PR#1607 by Karl-Johan
Karlsson).
- `intelmq.bots.parsers.generic.parser_csv`: Added new parameter
`compose_fields` (by Sebastian Wagner).
- `intelmq.bots.parsers.shadowserver.parser_json`: Added (PR#1700 by
Birger Schacht).
- `intelmq.bots.parsers.shadowserver.config`:
  - Fixed mapping for Block list feed to accept network ranges in CIDR
notation (#1720, PR#1728 by Sebastian Waldbauer).
  - Added mapping for new feed MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS
(#1716, #1726, #1733, PR#1732, PR#1735, PR#1736 by Sebastian Waldbauer).
  - Ignore value `0` for `source.asn` and `destination.asn` in all
mappings to avoid parsing errors (PR#1769 by Sebastian Wagner).
- `intelmq.bots.parsers.abusech.parser_ip`: Adapt to changes in the
Feodo Tracker Botnet C2 IP Blocklist feed (PR#1741 by Thomas Bellus).
- `intelmq.bots.parsers.malwaredomainlist`: Removed, as the feed is
obsolete (#1762, PR#1771 by Birger Schacht).

#### Experts
- `intelmq.bots.experts.rfc1918.expert`:
  - Add support for ASNs (PR#1557 by Mladen Markovic).
  - Speed improvements.
  - More output in debug logging mode (by Sebastian Wagner).
  - Checks parameter length on initialization and in check method (by
Sebastian Wagner).
- `intelmq.bots.experts.gethostbyname.expert`:
  - Added parameter `fallback_to_url` and set to True (PR#1586 by Edvard
Rejthar).
  - Added parameter `gaierrors_to_ignore` to optionally ignore other
`gethostbyname` errors (#1553).
  - Added parameter `overwrite` to optionally overwrite existing IP
addresses (by Sebastian Wagner).
- `intelmq.bots.experts.asn_lookup.expert`:
  - Added `--update-database` option (PR#1524 by Filip Pokorný).
  - The script `update-asn-data` is now deprecated and will be removed
in version 3.0.
- `intelmq.bots.experts.maxmind_geoip.expert`:
  - Added `--update-database` option (PR#1524 by Filip Pokorný).
  - Added `license_key` parameter (PR#1524 by Filip Pokorný).
  - The script `update-geoip-data` is now deprecated and will be removed
in version 3.0.
- `intelmq.bots.experts.tor_nodes.expert`:
  - Added `--update-database` option (PR#1524 by Filip Pokorný).
  - The script `update-tor-nodes` is now deprecated and will be removed
in version 3.0.
- `intelmq.bots.experts.recordedfuture_iprisk.expert`:
  - Added `--update-database` option (PR#1524 by Filip Pokorný).
  - Added `api_token` parameter (PR#1524 by Filip Pokorný).
  - The script `update-rfiprisk-data` is now deprecated and will be
removed in version 3.0.
- Added `intelmq.bots.experts.threshold` (PR#1608 by Karl-Johan Karlsson).
- Added `intelmq.bots.experts.splunk_saved_search.expert` (PR#1666 by
Karl-Johan Karlsson).
- `intelmq.bots.experts.sieve.expert`:
  - Added possibility to give multiple queue names for the `path`
directive (#1462, by Sebastian Wagner).
  - Added possibility to run actions without filtering expression
(#1706, PR#1708 by Sebastian Waldbauer).
  - Added datetime math operations (#1680, PR#1696 by Sebastian Waldbauer).
- `intelmq.bots.experts.maxmind_geoip.expert`:
  - Fixed handing over of `overwrite` parameter to `event.add` (PR#1743
by Birger Schacht).

#### Outputs
- `intelmq.bots.outputs.rt`: Added Request Tracker output bot (PR#1589
by Marius Urkis).
- `intelmq.bots.outputs.xmpp.output`: Marked as deprecated, see
https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html
(#1614, PR#1685 by Birger Schacht).
- `intelmq.bots.outputs.smtp.output`: Fix sending to multiple recipients
when recipients are defined by event-data (#1759, PR#1760 by Sebastian
Waldbauer and Sebastian Wagner).

### Documentation
- Feeds:
  - Add ESET URL and Domain feeds (by Sebastian Wagner).
  - Remove unavailable *HPHosts Hosts file* feed (#1559 by Sebastian
Wagner).
  - Added CZ.NIC HaaS feed (PR#1560 by Filip Pokorný and Edvard Rejthar).
  - Added CZ.NIC Proki feed (PR#1599 by sinus-x).
  - Updated Abuse.ch URLhaus feed (PT#1572 by Filip Pokorný).
  - Added CERT-BUND CB-Report Malware infections feed (PR#1598 by
sinus-x and Sebastian Wagner).
  - Updated Turris Greylist feed with PGP verification information (by
Sebastian Wagner).
  - Fixed parsing of the `public` field in the generated feeds
documentation (PR#1641 by Birger Schacht).
  - Change the `rate_limit` parameter of some feeds from 2 days (129600
seconds) to one day (86400 seconds).
  - Update the cAPTure Ponmocup Domains feed documentation (PR#1574 by
Filip Pokorný and Sebastian Wagner).
  - Added Shadowserver Reports API (by Sebastian Wagner).
  - Change the `rate_limit` parameter for many feeds from 2 days to the
default one day (by Sebastian Wagner).
  - Removed Malware Domain List feed, as the feed is obsolete (#1762,
PR#1771 by Birger Schacht).
- Bots:
  - Enhanced documentation of RFC1918 Expert (PR#1557 by Mladen Markovic
and Sebastian Wagner).
  - Enhanced documentation of SQL Output (PR#1620 by Edvard Rejthar).
  - Updated documentation for MaxMind GeoIP, ASN Lookup, TOR Nodes and
Recorded Future experts to reflect new `--update-database` option
(PR#1524 by Filip Pokorný).
  - Added documentation for Shadowserver API collector and parser
(PR#1700 by Birger Schacht and Sebastian Wagner).
- Add n6 integration documentation (by Sebastian Wagner).
- Moved 'Orphaned Queues' section from the FAQ to the intelmqctl
documentation (by Sebastian Wagner).
- Generate documentation using Sphinx (PR#1622 by Birger Schacht).
  - The documentation is now available at
https://intelmq.readthedocs.io/en/latest/
  - Refactor documentation and fix broken syntax (#1639, PRs #1638 #1640
#1642 by Birger Schacht).
- Integrate intelmq-manager and intelmq-api user documentation to
provide unified documentation place (PR#1714 & PR#1714 by Birger Schacht).

### Packaging
- Fix paths in the packaged logcheck rules (by Sebastian Wagner).
- Build the sphinx documentation on package build (PR#1701 by Birger
Schacht).
- Ignore non-zero exit-codes for the `intelmqctl check` call in postinst
(#1748, by Sebastian Wagner).

### Tests
- Added tests for `intelmq.lib.exceptions.PipelineError` (by Sebastian
Wagner).
- `intelmq.tests.bots.collectors.http_collector.test_collector`: Use
`requests_mock` to mock all requests and do not require a local
webserver (by Sebastian Wagner).
- `intelmq.tests.bots.outputs.restapi.test_output`:
  - Use `requests_mock` to mock all requests and do not require a local
webserver (by Sebastian Wagner).
  - Add a test for checking the response status code (by Sebastian Wagner).
- `intelmq.tests.bots.collectors.mail.test_collector_url`: Use
`requests_mock` to mock all requests and do not require a local
webserver (by Sebastian Wagner).
- `intelmq.tests.bots.experts.ripe.test_expert`: Use `requests_mock` to
mock all requests and do not require a local webserver (by Sebastian
Wagner).
- The test flag (environment variable) `INTELMQ_TEST_LOCAL_WEB` is no
longer used (by Sebastian Wagner).
- Added tests for `intelmq.harmonization.DateTime.parse_utc_isoformat`
and `convert_fuzzy` (by Sebastian Wagner).
- Move from Travis to GitHub Actions (PR#1707 by Birger Schacht).
- `intelmq.lib.test`:
  - `test_static_bot_check_method` checks the bot's static
`check(parameters)` method for any exceptions, and a valid formatted
return value (#1505, by Sebastian Wagner).
  - `setUpClass`: Skip tests if cache was requests with `use_cache`
member, but Redis is deactivated with the environment variable
`INTELMQ_SKIP_REDIS` (by Sebastian Wagner).
- `intelmq.tests.bots.experts.cymru_whois.test_expert`:
  - Switch from `example.com` to `ns2.univie.ac.at` for hopefully more
stable responses (#1730, PR#1731 by Sebastian Waldbauer).
  - Do not test for exact expected values in the 6to4 network test, as
the values are changing regularly (by Sebastian Wagner).
- `intelmq.tests.bots.parsers.abusech`: Remove tests cases of
discontinued feeds (PR#1741 by Thomas Bellus).
- Activate GitHub's CodeQL Code Analyzing tool as GitHub Action (by
Sebastian Wagner).

### Tools
- `intelmqdump`:
    - Check if given queue is configured upon recovery (#1433, PR#1587
by Mladen Markovic).
- `intelmqctl`:
  - `intelmq list queues`: `--sum`, `--count`, `-s` flag for showing
total count of messages (#1408, PR#1581 by Mladen Markovic).
  - `intelmq check`: Added a possibility to ignore queues from the
orphaned queues check (by Sebastian Wagner).
  - Allow setting the pipeline host by environment variables for docker
usage (PR#1669 by Sebastian Waldbauer).

### Contrib
- EventDB:
  - Add SQL script for keeping track of the oldest inserted/update
"time.source" information (by Sebastian Wagner).
- Cron Jobs: The script `intelmq-update-data` has been renamed to
`intelmq-update-database` (by Filip Pokorný).
- Dropped utterly outdated contrib modules (by Sebastian Wagner):
  - ansible
  - vagrant
  - vagrant-ansible
- logrotate:
  - Do not use the deprecated "copytruncate" option as intelmq re-opens
the log anyways (by Sebastian Wagner).
  - Set file permissions to `0644` (by Sebastian Wagner).

### Known issues
- Bots started with IntelMQ-API/Manager stop when the webserver is
restarted (#952).
- Corrupt dump files when interrupted during writing (#870).
- CSV line recovery forces Windows line endings (#1597).
- intelmqdump: Honor logging_path variable (#1605).
- Timeout error in mail URL fetcher (#1621).
- AMQP pipeline: get_queues needs to check vhost of response (#1746).

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210304/aeda827e/attachment.sig>