Dear community,
I have two news items for you, both related to documentation:
First, there is an IntelMQ Tutorial, which guides through various
features and tools of IntelMQ.
Lesson one introduces the architecture, concepts and terminology of the
project. Lessons two and three delve hands-on into working with IntelMQ.
Starting with installation and basic usage & configuration they go on to
tackle progressively more advanced topics like using advanced features
or changing the message queue software to be used. Solutions and
explanations are offered for all tasks. In the last lesson you'll learn
how to use intelmq-tools, a third-party software which makes
customization of your IntelMQ instance much easier.
We think that this kind of interactive online documentation is
especially important nowadays when conferences and workshops cannot take
place in real life.
As for all other IntelMQ components, we welcome any contributions and
feedback to the tutorial.
-> https://github.com/certtools/intelmq-tutorial
Second, we have a new IntelMQ Documentation page:
We completely revised the way IntelMQ's documentation is presented:
Instead of single files in the source-code repository, the best place to
read the documentation is now intelmq.readthedocs.io. All pages are
generated using Sphinx, the de facto standard tool for documentation. It
features a better reading experience and a significantly improved
navigation. Furthermore, the new page offers an integrated search as
well as module index covering the complete code documentation
If you find any bugs or have improvements, please let us know!
-> https://intelmq.readthedocs.io/
best regards
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
It's again long overdue for a new release and here it is finally. Since
August we collected quite a few bugfixes - Thanks to all contributors!
IntelMQ Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.2/docs/INSTALL.md
IntelMQ Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.2/docs/UPGRADING.md
*News for IntelMQ 2.2.2*
### Bots
#### Cymru Whois Lookup
The cache key calculation has been fixed. It previously led to duplicate
keys for different IP addresses and therefore wrong results in rare
cases. The cache key calculation is intentionally not
backwards-compatible. Therefore, this bot may take longer processing
events than usual after applying this update.
More details can be found in [issue
#1592](https://github.com/certtools/intelmq/issues/1592).
### Harmonization
#### Shadowserver Feed/Parser
The feed "Blacklisted-IP" has been renamed by ShadowServer to
"Blocklist". In IntelMQ, the old name can still be used in IntelMQ until
version 3.0.
*Changes for IntelMQ 2.2.2*
### Core
- `intelmq.lib.upgrades`:
- Add upgrade function for renamed Shadowserver feed name
"Blacklisted-IP"/"Blocklist".
### Bots
#### Parsers
- `intelmq.bots.parsers.shadowserver`:
- Rename "Blacklisted-IP" feed to "Blocklist", old name is still valid
until IntelMQ version 3.0 (PR#1588 by Thomas Hungenberg).
- Added support for the feeds `Accessible Radmin` and `CAIDA IP
Spoofer` (PR#1600 by sinus-x).
- `intelmq.bots.parsers.anubisnetworks.parser`: Fix parsing error where
`dst.ip` was not equal to `comm.http.host`.
- `intelmq/bots/parsers/danger_rulez/parser`: correctly skip malformed
rows by defining variables before referencing (PR#1601 by Tomas Bellus).
- `intelmq.bots.parsers.misp.parser: Fix MISP Event URL (#1619, PR#1618
by Nedfire23).
- `intelmq.bots.parsers.microsoft.parser_ctip`:
- Add support for `DestinationIpInfo.*` and `Signatures.Sha256`
fields, used by the `ctip-c2` feed (PR#1623 by Mikk Margus Möll).
- Use `extra.payload.text` for the feed's field `Payload` if the
content cannot be decoded (PR#1610 by Giedrius Ramas).
#### Experts
- `intelmq.bots.experts.cymru_whois`:
- Fix cache key calculation which previously led to duplicate keys and
therefore wrong results in rare cases. The cache key calculation is
intentionally not backwards-compatible (#1592, PR#1606).
- The bot now caches and logs (as level INFO) empty responses from
Cymru (PR#1606).
### Documentation
- README:
- Add Core Infrastructure Initiative Best Practices Badge.
- Bots:
- Generic CSV Parser: Add note on escaping backslashes (#1579).
- Remove section of non-existing "Copy Extra" Bot.
- Explain taxonomy expert.
- Add documentation on n6 parser.
- Gethostbyname expert: Add documentation how errors are treated.
- Feeds:
- Fixed bot modules of Calidog CertStream feed.
- Add information on Microsoft CTIP C2 feed.
### Packaging
- In Debian packages, `intelmqctl check` and `intelmqctl upgrade-config`
are executed in the postinst step (#1551, PR#1624 by Birger Schacht).
### Tests
- `intelmq.tests.lib.test_pipeline`: Skip `TestAmqp.test_acknowledge` on
Travis with Python 3.8.
- `intelmq.tests.bots.outputs.elasticsearch.test_output`: Refresh index
`intelmq` manually to fix random test failures (#1593, PR#1595 by Zach
Stone).
### Tools
- `intelmqctl check`:
- For disabled bots which do not have any pipeline connections, do not
raise an error, but only warning.
- Fix check on source/destination queues for bots as well the orphaned
queues.
### Contrib
- Bash completion scripts: Check both `/opt/intelmq/` as well as
LSB-paths (`/etc/intelmq/` and `/var/log/intelmq/`) for loading bot
information (#1561, PR#1628 by Birger Schacht).
### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952).
- Corrupt dump files when interrupted during writing (#870).
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
Today we have again a twin release 2.2.1 for both IntelMQ as well as
IntelMQ Manager. This IntelMQ Manager version requires IntelMQ >= 2.2.1.
There are currently issues with the signature in the package
repositories for Debian/Ubuntu. I hope to get them resolved soon.
IntelMQ Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/INSTALL.md
IntelMQ Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/UPGRADING.md
IntelMQ Manager Installation instructions:
https://github.com/certtools/intelmq-manager/blob/2.2.1/docs/INSTALL.md
*The changelog for IntelMQ Manager:*
### Backend
- Fix loading paths from `intelmqctl` executable (PR #205 by Einar
Felipe Lanfranco).
### Documentation
- User Guide:
- Add section on configuration paths.
- Add section on named queues / paths.
- Readme:
- Update screenshots (#201, PR#207 by Mladen Markovic).
### Known issues
* Graph jumps around on "Add edge" (#148).
* wrong error message for new bots with existing ID (#152).
* Monitor page: Automatic log refresh reset log page to first one (#190).
*The News for IntelMQ:*
### Requirements
#### MaxMind GeoIP Expert Bot
The current python library versions of geoip (version 4) and maxminddb
(version 2) no longer support Python 3.5. Keep older versions of these
libraries if you are using this Python version.
### Configuration
#### Abuse.ch URLHaus
The current documented value for the `column` parameter was:
```json
['time.source', 'source.url', 'status', 'extra.urlhaus.threat_type',
'source.fqdn', 'source.ip', 'source.asn', 'source.geolocation.cc']
```
Better is:
```json
['time.source', 'source.url', 'status',
'classification.type|__IGNORE__', 'source.fqdn|__IGNORE__', 'source.ip',
'source.asn', 'source.geolocation.cc']
```
*And the changelog for IntelMQ:*
### Core
- `intelmq.lib.upgrades`:
- Add upgrade function for changed configuration of the feed "Abuse.ch
URLHaus" (#1571, PR#1572 by Filip Pokorný).
- Add upgrade function for removal of *HPHosts Hosts file* feed and
`intelmq.bots.parsers.hphosts` parser (#1559).
- `intelmq.lib.harmonization`:
- For IP Addresses, explicitly reject IPv6 addresses with scope ID
(due to changed behavior in Python 3.9, #1550).
### Development
- Ignore line length (E501) in code-style checks altogether.
### Bots
#### Collectors
- `intelmq.bots.collectors.misp`: Fix access to actual MISP object
(PR#1548 by Tomas Bellus @tomas321)
- `intelmq.bots.collectors.stomp`: Remove empty `client.pem` file.
#### Parsers
- `intelmq.bots.parsers.shadowserver.config`:
- Add support for Accessible-CoAP feed (PR #1555 by Thomas Hungenberg).
- Add support for Accessible-ARD feed (PR #1584 by Tomas Bellus
@tomas321).
- `intelmq.bots.parser.anubisnetworks.parser`: Ignore
"TestSinkholingLoss" events, these are not intended to be sent out at all.
- `intelmq.bots.parsers.generic.parser_csv`: Allow values of type
dictionary for parameter `type_translation`.
- `intelmq.bots.parsers.hphosts`: Removed, feed is unavailable (#1559).
- `intelmq.bots.parsers.cymru.parser_cap_program`: Add support for
comment "username" for "scanner" category.
- `intelmq.bots.parsers.malwareurl.parser`: Check for valid FQDN and IP
address in URL and IP address columns (PR#1585 by Marius Urkis).
#### Experts
- `intelmq.bots.experts.maxmind_geoip`: On Python < 3.6, require
maxminddb < 2, as that version does no longer support Python 3.5.
#### Outputs
- `intelmq.bot.outputs.udp`: Fix error handling on sending, had a bug
itself.
### Documentation
- Feeds:
- Update documentation of feed "Abuse.ch URLHaus" (#1571, PR#1572 by
Filip Pokorný).
- Bots:
- Overhaul of all bots' description fields (#1570).
- User-Guide:
- Overhaul pipeline configuration section and explain named queues
better (#1577).
### Tests
- `intelmq.tests.bots.experts.cymru`: Adapt `test_empty_result`, remove
`test_unicode_as_name` and `test_country_question_mark` (#1576).
### Tools
- `intelmq.bin.intelmq_gen_docs`: Format parameters of types lists with
double quotes around values to produce conform JSON, ready to copy and
paste the value into the IntelMQ Manager's bot parameter form.
- `intelmq.bin.intelmqctl`:
- `debug`: In JSON mode, use dictionaries instead of lists.
- `debug`: Add `PATH` to the paths shown.
- `check`: Show `$PATH` environment variable if executable cannot be
found.
### Contrib
- `malware_name_mapping`: Change MISP Threat Actors URL to new URL
(branch master -> main) in download script.
### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952).
- Corrupt dump files when interrupted during writing (#870).
- Bash completion scripts search in wrong directory in packages (#1561).
- Cymru Expert: Wrong Cache-Key Calculation (#1592).
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
To get the versions of intelmq and intelmq-manager in sync, we of course
need a 2.2.0 for intelmq-manager too :) The number of changes is very
short this time. However, a migration of the backend from PHP to Python,
coded by Intevation and funded by SUNET, is in progress :)
The deb/rpm packages are currently building, I will send a separate mail
to intelmq-users once the 2.2.0 of intelmq and intelmq-manager packages
have finally hit the repositories.
This IntelMQ Manager version requires IntelMQ >= 2.2.0.
Installation instructions:
https://github.com/certtools/intelmq-manager/blob/2.2.0/docs/INSTALL.md
Full changelog:
### Backend
- `config`: Get file paths from `intelmctl debug --get-paths` if
possible and fall back to hard-coded paths otherwise. Thereby
environment variables influencing the paths are respected (#193).
### Pages
#### About
- Show output of `intelmqctl debug`.
### Documentation
- Update release from intelmq's release documentation.
- Update Installation documentation: Fix & update dependencies and
supported operating systems.
### Packaging
- Update default `positions.conf` to the default runtime/pipeline
configuration of intelmq >= 2.1.1.
### Known issues
* Missing CSRF protection (#111).
* Graph jumps around on "Add edge" (#148).
* wrong error message for new bots with existing ID (#152).
* `ALLOWED_PATH=` violates CSP (#183).
* Monitor page: Automatic log refresh reset log page to first one (#190).
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
Today I finalized the 2.2.0 Feature release, more than half a year after
2.1.0. Thanks to all the contributors who made this possible!
The release includes six new bots and seven heavily revised bots, and of
course a lot of small changes to various bots. The full changelog can be
found below. We dropped support for Python 3.4, that means Debian 8.0
and similar operating systems are no longer supported.
Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/UPGRADING.md
### Core
- `__init__`: Changes to the path-handling, see [User Guide, section
_/opt and LSB paths_](docs/User-Guide.md#opt-and-lsb-paths) for more
information
- The environment variable `INTELMQ_ROOT_DIR` can be used to set
custom root directories instead of `/opt/intelmq/` (#805) in case of non
LSB-path installations.
- The environment variable `ROOT_DIR` can be used to set custom root
directories instead of `/` (#805) in case of LSB-path installations.
- `intelmq.lib.exceptions`: Added `MissingDependencyError` for show
error messages about a missing library and how to install it (#1471).
- Added optional parameter `installed` to show the installed version.
- Added optional parameter `additional_text` to show arbitrary text.
- Adding more type annotations for core libraries.
- `intelmq.lib.pipeline.Pythonlist.sleep`: Drop deprecated method.
- `intelmq.lib.utils`: `write_configuration`: Append a newline at end of
configuration/file to allow proper comparisons & diffs.
- `intelmq.lib.test`: `BotTestCase` drops privileges upon initialization
(#1489).
- `intelmq.lib.bot`:
- New class `OutputBot`:
- Method `export_event` to format/export events according to the
parameters given by the user.
- `ParserBot`: New methods `parse_json_stream` and
`recover_line_json_stream`.
- `ParserBot.recover_line_json`: Fix format by adding a list around
the line data.
- `Bot.send_message`: In debugging log level, the path to which the
message is sent is now logged too.
### Bots
- Bots with dependencies: Use of
`intelmq.lib.exceptions.MissingDependencyError`.
#### Collectors
- `intelmq.bots.collectors.misp.collector`: Deprecate parameter
`misp_verify` in favor of generic parameter `http_verify_cert`.
- `intelmq.bots.collectors.tcp.collector`: Drop compatibility with
Python 3.4.
- `intelmq.bots.collectors.stomp.collector`:
- Check the stomp.py version and show an error message if it does not
match.
- For stomp.py versions `>= 5.0.0` redirect the
`stomp.PrintingListener` output to debug logging.
- `intelmq.bots.collectors.microsoft.collector_azure`: Support current
Python library `azure-storage-blob>= 12.0.0`, configuration is
incompatible and needs manual change. See NEWS file and bot's
documentation for more details.
- `intelmq.bots.collectors.amqp.collector_amqp`: Require `pika` minimum
version 1.0.
- `intelmq.bots.collectors.github_api.collector_github_contents_api`:
Added (PR#1481).
#### Parsers
- `intelmq.bots.parsers.autoshun.parser`: Drop compatibility with Python
3.4.
- `intelmq.bots.parsers.html_table.parser`: Drop compatibility with
Python 3.4.
- `intelmq.bots.parsers.shadowserver.parser`: Add support for MQTT and
Open-IPP feeds (PR#1512, PR#1544).
- `intelmq.bots.parsers.taichung.parser`:
- Migrate to `ParserBot`.
- Also parse geolocation information if available.
- `intelmq.bots.parsers.cymru.parser_full_bogons`:
- Migrate to `ParserBot`.
- Add last updated information in raw.
- `intelmq.bots.parsers.anubisnetworks.parser`: Add new parameter
`use_malware_familiy_as_classification_identifier`.
- `intelmq.bots.parsers.microsoft.parser_ctip`: Compatibility for new
CTIP data format used provided by the Azure interface.
- `intelmq.bots.parsers.cymru.parser_cap_program`: Support for
`openresolver` type.
- `intelmq.bots.parsers.github_feed.parser`: Added (PR#1481).
- `intelmq.bots.parsers.urlvir.parser`: Removed, as the feed is
discontinued (#1537).
#### Experts
- `intelmq.bots.experts.csv_converter`: Added as converter to CSV.
- `intelmq.bots.experts.misp`: Added (PR#1475).
- `intelmq.bots.experts.modify`: New parameter `maximum_matches`.
#### Outputs
- `intelmq.bots.outputs.amqptopic`:
- Use `OutputBot` and `export_event`.
- Allow formatting the routing key with event data by the new
parameter `format_routing_key` (boolean).
- `intelmq.bots.outputs.file`: Use `OutputBot` and `export_event`.
- `intelmq.bots.outputs.files`: Use `OutputBot` and `export_event`.
- `intelmq.bots.outputs.misp.output_feed`: Added, creates a MISP Feed
(PR#1473).
- `intelmq.bots.outputs.misp.output_api`: Added, pushes to MISP via the
API (PR#1506, PR#1536).
- `intelmq.bots.outputs.elasticsearch.output`: Dropped ElasticSearch
version 5 compatibility, added version 7 compatibility (#1513).
### Documentation
- Document usage of the `INTELMQ_ROOT_DIR` environment variable.
- Added document on MISP integration possibilities.
- Feeds:
- Added "Full Bogons IPv6" feed.
- Remove discontinued URLVir Feeds (#1537).
### Packaging
- `setup.py` do not try to install any data to `/opt/intelmq/` as the
behavior is inconsistent on various systems and with `intelmqsetup` we
have a tool to create the structure and files anyway.
- `debian/rules`:
- Provide a blank state file in the package.
- Patches:
- Updated `fix-intelmq-paths.patch`.
### Tests
- Travis: Use `intelmqsetup` here too.
- Install required build dependencies for the Debian package build test.
- This version is no longer automatically tested on Python `<` 3.5.
- Also run the tests on Python 3.8.
- Run the Debian packaging tests on Python 3.5 and the code-style test
on 3.8.
- Added tests for the new bot `intelmq.bots.outputs.misp.output_feed`
(#1473).
- Added tests for the new bot `intelmq.bots.experts.misp.expert` (#1473).
- Added tests for `intelmq.lib.exceptions`.
- Added tests for `intelmq.lib.bot.OutputBot` and
`intelmq.lib.bot.OutputBot.export_event`.
- Added IPv6 tests for `intelmq.bots.parsers.cymru.parser_full_bogons`.
- Added tests for `intelmq.lib.bot.ParserBot`'s new methods
`parse_json_stream` and `recover_line_json_stream`.
- `intelmq.tests.test_conf`: Set encoding to UTF-8 for reading the
`feeds.yaml` file.
### Tools
- `intelmqctl`:
- `upgrade-config`:
- Allow setting the state file location with the `--state-file`
parameter.
- Do not require a second run anymore, if the state file is newly
created (#1491).
- New parameter `no_backup`/`--no-backup` to skip creation of `.bak`
files for state and configuration files.
- Only require `psutil` for the `IntelMQProcessManager`, not for
process manager independent calls like `upgrade-config` or `check`.
- Add new command `debug` to output some information for debugging.
Currently implemented:
- paths
- environment variables
- `IntelMQController`: New argument `--no-file-logging` to disable
logging to file.
- If dropping privileges does not work, `intelmqctl` will now abort
(#1489).
- `intelmqsetup`:
- Add argument parsing and an option to skip setting file ownership,
possibly not requiring root permissions.
- Call `intelmqctl upgrade-config` and add argument for the state file
path (#1491).
- `intelmq_generate_misp_objects_templates.py`: Tool to create a MISP
object template (#1470).
- `intelmqdump`: New parameter `-t` or `--truncate` to optionally give
the maximum length of `raw` data to show, 0 for no truncating.
### Contrib
- Added `development-tools`.
- ElasticSearch: Dropped version 5 compatibility, added version 7
compatibility (#1513).
- Malware Name Mapping Downloader:
- New parameter `--mwnmp-ignore-adware`.
- The parameter `--add-default` supports an optional parameter to
define the default value.
### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952).
- Corrupt dump files when interrupted during writing (#870).
best regards
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi Patrick,
Am Montag 01 Juni 2020 16:27:24 schrieb Patrick Forsberg:
> Anyone have a quick comparison between AbuseIO and IntelMQ?
>
> I'm currently in the process of getting IntelMQ to work with
> IntelMQ-Mailgen to be able to send out abuse-emails to our constituency
> based on feeds like Shadowserver and since it seems like AbuseIO can do
> just about the same I would like to know the pros and cons of the systems.
for comparing AbuseIO and IntelMQ, I don't know AbuseIO enough.
However if you are looking into IntelMQ Mailgen from
the system we call intelmq-cb-mailgen,
https://github.com/Intevation/intelmq-mailgen-release
that is the one we've been developing for the CERT-Bund,
so we can tell you more about it, maybe this helps with the comparison.
The design idea is to be automated, flexible and high through-put.
Thus there is a separation of concerns and several configuration places.
You may have seen the overview diagram:
https://raw.githubusercontent.com/Intevation/intelmq-mailgen/master/docs/no…
There are rules within the IntelMQ export and additional notification formats
scripts, those are quite flexible, so there is some learning curve.
Once set up, there can be millions of events handled per day automatically
with several people being able to add manual data to the contacts database.
Feel free to ask us here or per direct email, if you have any questions or
need help with setting it up.
Best Regards,
Bernhard
--
www.intevation.de/~bernhard +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
Dear developers and pro-users,
I have pushed a 2.2.0 pre-release to both PyPI as well as the unstable
repository. If you have time & resources, or you are using the develop
branch anyway, please test this version so we can ship the next stable
release ready next week or the week afterwards, depending on the feedback.
Please note that the current codebase is no longer tested with Python
3.4 and it may not work anymore with that Python version.
For installation/upgrade with pip use the --pre parameter: pip install
--pre intelmq
Instructions for the deb & rpm unstable repository:
https://software.opensuse.org/download/package?package=intelmq&project=home…
best regards
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
We already collected a very long list of bug fixes since the last
release, so it was time to mark the next milestone! As usual, you can
find the list of changes below. The pre-built deb/rpm packages will hit
the repositories very soon.
Installation documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md
Full changelog:
### Requirements
- The python library `requests` is (again) listed as dependency of the
core (#1519).
### Core
- `intelmq.lib.upgrades`:
- Harmonization upgrade: Also check and update regular expressions.
- Add function to migrate the deprecated parameter `attach_unzip` to
`extract_files` for the mail attachment collector.
- Add function to migrate changed Taichung URL feed.
- Check for discontinued Abuse.CH Zeus Tracker feed.
- `intelmq.lib.bot`:
- `ParserBot.recover_line`: Parameter `line` needs to be optional, fix
usage of fallback value `self.current_line`.
- `start`: Handle decoding errors in the pipeline different so that
the bot is not stuck in an endless loop (#1494).
- `start`: Only acknowledge a message in case of errors, if we
actually had a message to dump, which is not the case for collectors.
- `_dump_message`: Dump messages with encoding errors base64 encoded,
not in JSON format as it's not possible to decode them (#1494).
- `intelmq.lib.test`:
- `BotTestCase.run_bot`: Add parameters `allowed_error_count` and
`allowed_warning_count` to allow set the number per run, not per test class.
- Set `source_pipeline_broker` and `destination_pipeline_broker` to
`pythonlist` instead of the old `broker`, fixes
`intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising`.
- Fix test for (allowed) errors and warnings.
- `intelmq.lib.exceptions`:
- `InvalidKey`: Add `KeyError` as parent class.
- `DecodingError`: Added, string representation has all relevant
information on the decoding error, including encoding, reason and the
affected string (#1494).
- `intelmq.lib.pipeline`:
- Decode messages in `Pipeline.receive` not in the implementation's
`_receive` so that the internal counter is correct in case of decoding
errors (#1494).
- `intelmq.lib.utils`:
- `decode`: Raise new `DecodingError` if decoding fails.
### Harmonization
- `protocol.transport`: Adapt regular expression to allow the value
`nvp-ii` (protocol 11).
### Bots
#### Collectors
- `intelmq.bots.collectors.mail.collector_mail_attach`:
- Fix handling of deprecated parameter name `attach_unzip`.
- Fix handling of attachments without filenames (#1538).
- `intelmq.bots.collectors.stomp.collector`: Fix compatibility with
stomp.py versions `> 4.1.20` and catch errors on shutdown.
- `intelmq.bots.collectors.microsoft`:
- Update `REQUIREMENTS.txt` temporarily fixing deprecated Azure
library (#1530, PR#1532).
- `intelmq.bots.collectors.microsoft.collector_interflow`: Add method
for printing the file list.
#### Parsers
- `intelmq.bots.parsers.cymru.parser_cap_program`: Support for protocol
11 (`nvp-ii`) and `conficker` type.
- `intelmq.bots.parsers.taichung.parser`: Support more
types/classifications:
- Application Compromise: Apache vulnerability & SQL injections
- Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH &
SIP attacks
- C2 Sever: Attack controller
- DDoS
- DoS: DNS, DoS, Excess connection
- IDS Alert / known vulnerability exploitation: backdoor
- Malware: Malware Proxy
- Warn on new unknown types.
- `intelmq.bots.parsers.bitcash.parser`: Removed as feed is discontinued.
- `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc` and
`intelmq.bots.parsers.fraunhofer.parser_ddosattack_target`: Removed as
feed is discontinued.
- `intelmq.bots.parsers.malwaredomains.parser`: Correctly classify `C&C`
and `phishing` events.
- `intelmq.bots.parsers.shadowserver.parser`: More verbose error message
for missing report specification (#1507).
- `intelmq.bots.parsers.n6.parser_n6stomp`: Always add n6 field `name`
as `malware.name` independent of `category`.
- `intelmq.bots.parsers.anubisnetworks`: Update parser with new data format.
- `intelmq.bots.parsers.bambenek`: Add new feed URLs with Host
`faf.bambenekconsulting.com` (#1525, PR#1526).
- `intelmq.bots.parsers.abusech.parser_ransomware`: Removed, as the feed
is discontinued (#1537).
- `intelmq.bots.parsers.nothink.parser`: Removed, as the feed is
discontinued (#1537).
- `intelmq.bots.parsers.n6.parser`: Remove not allowed characters in the
name field for `malware.name` and write original value to
`event_description.text` instead.
#### Experts
- `intelmq.bots.experts.cymru_whois.lib`: Fix parsing of AS names with
Unicode characters.
#### Outputs
- `intelmq.bots.outputs.mongodb`:
- Set default port 27017.
- Use different authentication mechanisms per MongoDB server version
to fix compatibility with server version >= 3.4 (#1439).
### Documentation
- Feeds:
- Remove unavailable feed Abuse.CH Zeus Tracker.
- Remove the field `status`, offline feeds should be removed.
- Add a new field `public` to differentiate between private and public
feeds.
- Adding documentation URLs to nearly all feeds.
- Remove unavailable Bitcash.cz feed.
- Remove unavailable Fraunhofer DDos Attack feeds.
- Remove unavailable feed Abuse.CH Ransomware Tracker (#1537).
- Update information on Bambenek Feeds, many require a license now
(#1525).
- Remove discontinued Nothink Honeypot Feeds (#1537).
- Developers Guide: Fix the instructions for `/opt/intelmq` file
permissions.
### Packaging
- Patches: `fix-logrotate-path.patch`: also include path to rotated file
in patch.
- Fix paths from `/opt` to LSB for `setup.py` and
`contrib/logrotate/intelmq` in build process (#1500).
- Add runtime dependency `debianutils` for the program `which`, which is
required for `intelmqctl`.
### Tests
- Dropping Travis tests for 3.4 as required libraries dropped 3.4 support.
- `intelmq.tests.bots.experts.cymru_whois`:
- Drop missing ASN test, does not work anymore.
- IPv6 to IPv4 test: Test for two possible results.
- `intelmq.lib.test`: Fix compatibility of logging capture with Python
>= 3.7 by reworking the whole process (#1342).
- `intelmq.bots.collectors.tcp.test_collector`: Removing custom mocking
and bot starting, not necessary anymore.
- Added tests for
`intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline`.
- Fix and split
`tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json`.
- Added tests for invalid encodings in input messages in
`intelmq.tests.lib.test_bot` and `intelmq.tests.lib.test_pipeline` (#1494).
- Travis: Explicitly enable RabbitMQ management plugin.
- `intelmq.tests.lib.test_message`: Fix usage of the parameter
`blacklist` for Message hash tests (#1539).
### Tools
- `intelmqsetup`: Copy missing BOTS file to IntelMQ's root directory
(#1498).
- `intelmq_gen_docs`: Feed documentation generation: Handle
missing/empty parameters.
- `intelmqctl`:
- `IntelMQProcessManager`: For the status of running bots also check
the bot ID of the commandline and ignore the path of the executable (#1492).
- `IntelMQController`: Fix exit codes of `check` command for JSON
output (now 0 on success and 1 on error, was swapped, #1520).
- `intelmqdump`:
- Handle base64-type messages for show, editor and recovery actions.
### Contrib
- `intelmq/bots/experts/asn_lookup/update-asn-data`: Use
`pyasn_util_download.py` to download the data instead from RIPE, which
cannot be parsed currently (#1517, PR#1518,
https://github.com/hadiasghari/pyasn/issues/62).
### Known issues
- HTTP stream collector: retry on regular connection problems? (#1435).
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952).
- Reverse DNS: Only first record is used (#877).
- Corrupt dump files when interrupted during writing (#870).
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi Sebastian,
do you have a rough timeframe for the 2.2.0 release?
Best,
Bernhard
--
www.intevation.de/~bernhard +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
Hi IntelMQ-Devs,
wondering what an OutputBot should do, if it cannot
put an event to the output, because of the event itself.
Should it do something like
self.logger.warning("event does not meet criteria for output")
self._dump_message()
# to place the event in a dump file for later inspection
self.acknowledge_message()
?
Background: the
https://github.com/certtools/intelmq/blob/develop/intelmq/bots/outputs/misp…
seems to get events that it cannot insert into MISP, because
some fields necessary in the intelmq event are not filled with values.
If the bot detects this, it can skip the event,
but it seems a good idea to preserve enough info how the empty values came to
be.
The alternatives to dumping would be
a) write out the event in the log using self.logger
b) just ignore the event
Thanks,
Bernhard
--
www.intevation.de/~bernhard +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
