Dear IntelMQ users
The latest IntelMQ release, version 3.5.0, introduced a new field named
'severity' with its possible values critical, high, medium, low, info,
and undefined.
If you have incoming data that places the severity information in the
'extra.severity' field, but you want it in the 'severity' field, you can
use a workaround using a Modify expert.
For example, this is necessary for all data coming from Shadowserver.
Howto:
1. Add a Modify Expert, e.g., named 'severity-modify-expert', to your
dataflow and put it somewhere after the parsers.
2. Set the configuration path to e.g.
/var/lib/intelmq/bots/modify/severity.conf
3. Put the content of the attached file into this file.
4. Start the bot and reload it right before the new
'severity-modify-expert' is introduced. Thisis doneautomatically if
you use the IntelMQ Manager.
The attached configuration for the Modify expert contains two rules:
1. If the 'extra.severity' field exists (is not empty), but the
'severity' field does not (is empty), use the value of the latter
for the first.
2. Otherwise, if both fields do not exist (are empty), set the default
value 'undefined'
Of course, you can also add these two rules to an existing Modify expert
to keep your configuration small.
To test if your newly updated configuration works, you can use the
commands below.
Please let us know if this is helpful to you or if you have any further
questions.
Best regards
Sebastian
The test commands:
intelmqctl run severity-expert process --dryrun --show-sent --msg '{}'
-> this results in the fallback {"severity": "undefined"}
intelmqctl run severity-expert process --dryrun --show-sent --msg
'{"extra.severity": "high"}'
-> this results in {"extra.severity": "high", "severity": "high"}
intelmqctl run severity-expert process --dryrun --show-sent --msg
'{"extra.severity": "high", "severity": "low"}'
-> this results in no change: {"extra.severity": "high", "severity": "low"}
--
Institute for Common Good Technology
gemeinnütziger Kulturverein - nonprofit cultural society
https://commongoodtechnology.org/
ZVR 1510673578
Dear IntelMQ community
The new IntelMQ release 3.5.0 brings support for a few new data fields
that many of you have requested:
* severity
* product.full_name, product.name, product.vendor, product.version,
product.vulnerabilities (IEP009: IntelMQ Data Format: Describe
product and mark vulnerabilities
https://github.com/certtools/ieps/tree/main/009)
* constituency (IEP008: IntelMQ Data Format: Constituency field
https://github.com/certtools/ieps/tree/main/008)
If you use a database to store events, update the schema of theevents
table with the SQL statements shown below.
If you use deb- or rpm-packages, you have two new packages available:
intelmq-contrib and intelmq-autostart. The first contains stufffrom the
repositories' contrib/ folder: some eventdb scripts, an example
extension package, the feeds config generator, malware name mapping
scripts, and systemd scripts.The autostart package contains systemd
units to start IntelMQ at boot time and then run it periodically.
The full list of changes is very long, belowyou can read the full list
of changes.
Thanks to all who contributed to this version, whether through user
feedback or code.
This version contains code contributions from Edvard Rejthar, Kamil
Mańkowski, Karl-Johan Karlsson, Lukas Heindl, Mikk Margus Möll, Nakul
Rajpal, Sebastian Wagner, Timde Boer.
The 3.5.0 version is available at GitHub, PyPI andin the package
repositories:
https://github.com/certtools/intelmq/releases/tag/3.5.0https://pypi.org/project/intelmq/https://docs.intelmq.org/latest/admin/installation/linux-packages/
The complete change log:
## Requirements
Python `>=3.9` is now required, which is available on all platforms
supported by IntelMQ.
## Data Format
To save new fields from IntelMQ Data Format in existing PostgreSQL
instances, the following schema update is necessary:
```sql
CREATE TYPE severity_enum AS ENUM (
'critical',
'high',
'medium',
'low',
'info',
'undefined'
);
ALTER TABLE events ADD "product.full_name" text;
ALTER TABLE events ADD "product.name" text;
ALTER TABLE events ADD "product.vendor" text;
ALTER TABLE events ADD "product.version" text;
ALTER TABLE events ADD "product.vulnerabilities" text;
ALTER TABLE events ADD severity varchar(10);
ALTER TABLE events ADD "constituency" text;
UPDATE events SET severity = (extra ->> 'severity')::severity_enum;
```
Optionally remove the severity field from the extra fields in existing
entries:
```sql
UPDATE events SET extra = extra - 'severity';
```
To switch to the more efficient data type `jsonb` instead of `json`, use
the following SQL statement. Data is preserved. JSONB also has more
query and data manipulation features than plain JSON.
```sql
ALTER TABLE events
ALTER COLUMN "extra" SET DATA TYPE jsonb;
```
## Configuration
- New parameter `stop_retry_limit` (PR#2598 by Lukas Heindl).
## Core
- Drop support for Python 3.8 (fixes #2616, PR#2617 by Sebastian Wagner).
- `intelmq.lib.splitreports`: Handle bot parameter `chunk_size` values
empty string, due to missing parameter typing checks (PR#2604 by
Sebastian Wagner).
- `intelmq.lib.mixins.sql` Add Support for MySQL (PR#2625 by Karl-Johan
Karlsson).
- New parameter `stop_retry_limit` to gracefully handle stopping bots
which take longer to shutdown(PR#2598 by Lukas Heindl, fixes #2595).
- `intelmq.lib.datatypes`: Remove unneeded Dict39 alias (PR#2639 by
Nakul Rajpal, fixes #2635)
- `intelmq.lib.mixins.http`: Only set HTTP header 'Authorization' if
username or password are set and are not both empty stringas they are by
default in the Manager (fixes #2590, PR#2634 by Sebastian Wagner).
- `intelmq.lib.message.Message.from_dict`: Do not modify the dict
parameter by adding the `__type` field and raise an error when type is
not determinable (PR#2545 by Sebastian Wagner).
- `intelmq.lib.mixins.cache.CacheMixin` was extended to support
temporarystoring messages in a cache queue
(PR#2509 by Kamil Mankowski).
## Data Format
- Implementing [IEP009](https://github.com/certtools/ieps/tree/main/009)
introducingfields to identify products and vulnerabilities:
`product.full_name`, `product.name`, `product.vendor`,
`product.version`, `product.vulnerabilities`. To store in existing
PostgreSQL instances, a schema update is necessary, see firstsection.
- added `severity` field to help with triaging received events (PR#2575
by Kamil Mańkowski). To allow saving the field in PostgreSQLdatabase in
existing installations, the schema update shown in the first section is
necessary.
- Implementing [IEP008](https://github.com/certtools/ieps/tree/main/008)
introducingthe `constituency` field for easier identification in
multi-constituency setups. (PR#2573 by Kamil Mańkowski). To use in
current PostgreSQL installations, a schema update is necessary, see
firstsection.
## Bots
## Collectors
- `intelmq.bots.collectors.mail.collector_mail_attach`: Decrypt GPG
attachments (PR#2623 by Edvard Rejthar).
- `intelmq.bots.collectors.mail.collector_mail_attach`: Allow empty
attachments (PR#2647 by Edvard Rejthar).
- `intelmq.bots.collectors.shodan.collector_alert`: Added a new
collector to query the Shodan Alert API (PR#2618 by Sebastian Wagner and
Malawi CERT).
- Remove `intelmq.bots.collectors.blueliv` as it uses an unmaintained
library, does not work any moreandbreaks other CI tests (fixes #2593,
PR#2632 by Sebastian Wagner).
## Parsers
- `intelmq.bots.parsers.cymru.parser_cap_program`: Add mapping for TOR
and ipv6-icmpprotocol (PR#2621 by Mikk Margus Möll).
- Remove `intelmq.bots.parsers.blueliv` as it is obsolete with the
removed collector (PR#2632 by Sebastian Wagner).
- `intelmq.bots.parser.json.parser`:
- Support data containing lists of JSON Events (PR#2545 by Tim de Boer).
- Add default `classification.type` with value `undetermined` if
input data has nowclassification itself (PR#2545 by Sebastian Wagner).
## Experts
- `intelmq.bots.experts.asn_lookup.expert`:
- Print URLs to stdout only in verbose mode (PR#2591 by Sebastian
Wagner).
- Check for database file existence and writability (fixes #2566).
- Use database path matching to installation type (PR#2606 by
Sebastian Wagner).
- `intelmq.bots.experts.fake.expert`:
- Use database path matching to installation type (PR#2606 by
Sebastian Wagner).
- Add new mode `random_single_value` (PR#2601 by Sebastian Wagner).
- `intelmq.bots.experts.sieve.expert`: Test for textX dependency in
self-check (PR#2605 by Sebastian Wagner).
- `intelmq.bots.experts.trusted_introducer_lookup.expert`: Change to new
TI database URL (fixes #2620, PR#2633 by Sebastian Wagner).
## Outputs
- `intelmq.bots.outputs.smtp_batch.output`:
- Add newparameter `additional_grouping_keys` for an enhanced email
batching feature.
- Add newparameter `templating` for additional template variables.
- Add new parameter `allowed_fieldnames` for csvfield specification.
- Add new parameter `fieldnames_translation` for naming csvheaders
(PR#2610 by Lukas Heindl, fixes #2586).
- `intelmq.bots.outputs.sql.output`: Add Support for MySQL (PR#2625 by
Karl-Johan Karlsson).
## Documentation
- Fix and refresh links to mailing lists (PR#2609 by Kamil Mańkowski)
- `Aggregate Bot`: Add illustration graphics (PR#2612 by Sebastian Wagner).
## Packaging
- Replace `/opt/intelmq` example paths in bots with variable
`VAR_STATE_PATH` for correct paths in LSB-path setups likewith packages
(PR#2587 by Sebastian Wagner).
- New deb-package `intelmq-contrib` with all `contrib/` scripts and
documentation (PR#2614 by Sebastian Wagner).
- New deb-package `intelmq-autostart` containing systemd services and
timers to start all enabled IntelMQ bots at boot and periodically
(PR#2638 by Sebastian Wagner).
## Tests
- `intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge`: Skip on
all Python versions when running on CI (PR#2602 by Sebastian Wagner).
- `.github/workflows/codespell.yml`, `debian-package.yml`,
`regexploit.yml`: Upgrade to `ubuntu-latest` runners (PR#2602 by
Sebastian Wagner).
- `intelmq.test.test_conf`: With changed behaviourin ruamel.yaml on line
wrapping since version 0.18.13, only test the parsabiltyof
`runtime.yaml` (PR#2619 by Sebastian Wagner).
- `intelmq.test.BotTestCase.test_static_bot_check_method`: Remove
debugging stub raising for all non-empty checks (PR#2622 by Sebastian
Wagner).
## Tools
- `intelmq.bin.intelmq_psql_initdb`: Use `JSONB` type by default,
Postgres supports it since version 9 (PR#2597 by Sebastian Wagner).
- `intelmq.bin.rewrite_config_files`: Removed obsolete JSON
configuration file rewriter (PR#2613 by Sebastian Wagner).
- `intelmq/lib/bot_debugger.py`: Fix overwriting the runtime logging
level by command line parameter (PR#2603 by Sebastian Wagner, fixes #2563).
- `intelmq.bin.intelmqctl`: Fix bot log level filtering (PR#2607 by
Sebastian Wagner, fixes #2596).
## Contrib
- Bash Completion: Adapt to YAML-style runtime configuration (PR#2642 by
Sebastian Wagner, fixes #2094).
- Remove `prettyprint` script, use `jq` instead (PR#2551 by Sebastian
Wagner).
## Known issues
Thisis shortlist of the most important known issues. The fulllist can be
retrievedfrom
[GitHub](https://github.com/certtools/intelmq/labels/bug?page=2&q=is%3Aopen+….
- stomp.py 8.2.0+ breaks the version check in stomp bots (#2600).
- Traceback when calling intelmqdump without write access to the log
file (#2529).
- pyyaml PendingDeprecationWarning: you should no longer specify
'unsafe' -> test failure (#2489).
- `intelmq.parsers.html_table` may not process invalid URLs in patched
Python version due to changes in `urllib` (#2382).
- Breaking changes in 'rt' 3.0 library (#2367).
- Type error with SQL output bot's `prepare_values` returning list
instead of tuple(#2255).
- `intelmq_psql_initdb` does not work for SQLite (#2202).
- intelmqsetup: should install a default state file (#2175).
- Misp Expert - Crash if misp event already exist(#2170).
- Spamhaus CERT parser uses wrongfield (#2165).
- Custom headers ignored in HTTPCollectorBot (#2150).
- intelmqctl log: parsing syslog does not work (#2097).
- Bots started with IntelMQ-API/Manager stop when the webserveris
restarted (#952).
- Corrupt dump files when interrupted during writing (#870).
--
Institute for Common Good Technology
gemeinnütziger Kulturverein - nonprofit cultural society
https://commongoodtechnology.org/
ZVR 1510673578