Dear allies,
The discussion around the IEP04 proposal, adding meta-information to
IntelMQ messages, has stalled over the last months - first because of
the time-intensive IntelMQ 3.0 release preparations and then because of
the vacation season.
Here is the current proposal:
https://github.com/certtools/ieps/tree/main/004#readme
Aaron, Sebastian Waldbauer and myself worked on it over the summer and
also identified two open issues to be discussed:
1. The exact format of the meta-information and how to name and
structure the fields. AIL made the first move and now uses a format
similar to the previously proposed Variant "A". The IEP04 document
contains the current proposal which is in line with the AIL format:
https://github.com/certtools/ieps/tree/main/004#user-content-variant-ail
If there are no other proposals, this will most probably the way to go.
2. The format of the UUID format which we want to uniquely identify
IntelMQ events. We don't necessarily need to use the UUIDv4 format which
represents pure randomness, but also other options which include the
time and are even /time-sortable/. Sebastian Waldbauer analysed a couple
of options and summarised his results in this document:
https://github.com/certtools/ieps/blob/main/004/UUID.md
Please let us know your opinion on the different UUID options.
cheers
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
Last week I said in the announcement for 3.0.1, that we didn't plan any
further bugfix release before 3.1.0. I'm now proving myself wrong.
Narzhan reported general performance issues and we investigated the
problem on our own instances.
We found two problems which different effects. It could be the case that
you didn't even notice any performance drop, as both bugs affected only
certain bots or situations.
- When any collector creates a new report, it initializes an object of
type Report. Since five years there was a bug, that this always loaded
the harmonization.conf, instead of loading it just once at the start
(which is - correctly - the case for new events). This was never a
problem, as the Python's built-in json module was fast enough, but
ruamel.yaml is slower. If a collector didn't massively create reports,
that bug was not obvious, especially under testing conditions. And the
bug was only relevant, if stream data is processed (for example with the
Shodan stream collector).
- Two bots, the STOMP collector and the API collector, don't have a
proper process method, because they are using threading. A thread
retrieves data asynchronously and sends the data. Normally, the
process-method is running in an endless-loop, with sleeping rate_limit
seconds between. If rate_limit is 0 or not defined, this would result in
high load, therefore these bots defined the __collector_empty_process
attribute which should have prohibited the endless loop. But due to
class-restructuring, the attribute which would have been to be set was
_Bot__collector_empty_process, and the flag was not effective. That did
not only cause a loop, but as bot statistics are written to redis after
every process-call, that caused continuous connections to redis. We've
now renamed the flag to _collector_empty_process and fixed the two bots.
The flag __is_multithreadable was affected analogous, but that should
not have cause performance issues in particular.
Additionally, the bot statistics can now be disabled by setting
|statistics_host| to null (previously this caused an exception).|
|
The new version is on GitHub and PyPI, very soon in the deb/rpm
repositories. The Docker image will follow on Monday or earlier.
You can read the full changelog here:
https://github.com/certtools/intelmq/releases/tag/3.0.2
have a nice weekend
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
Over the past two months, IntelMQ contributors had no summer pause, but
did the final finish for IntelMQ 3.0.
A special thanks goes to Mikk Margus Möll (CERT.ee) who has put
tremendous efforts in the IntelMQ Manager tackling structural and
usability issues, mainly in the JavaScript-components!
The deb/rpm repositories did not receive the 3.0.0 release at beginning
of July to get more experience with the major changes before doing
automatic upgrades, but now they deliver the brand-new 3.0.1 version.
Please note, that the automatic upgrade procedures may still not be
fully smooth. Just now, we have noticed, that the packages contain a
small flaw, which harms the upgrade experience: The packages ship a
default configuration (the file is now called `runtime.yaml`), but only
if the file does not exist before - for new installations. But now in
this special case, we renamed the configuration from `runtime.conf` to
`runtime.yaml` and therefore, the new - default shipped - configuration
takes precedence. I hope the following commands and hints will be of
help to you.
# remove the runtime configuration shipped by the package (can be called
/etc/intelmq/runtime.*) and rename your original one to
/etc/intelmq/runtime.yaml
# the previously used runtime.conf can be used as drop-in to
runtime.yaml (YAML is backwards-compatible with JSON)
sudo -u intelmq intelmqctl upgrade-config -f -u v300_pipeline_file_removal
sudo -u intelmq intelmqctl upgrade-config -f -u v300_defaults_file_removal
sudo -u intelmq intelmqctl upgrade-config -f -u v301_deprecations
The last three steps are important to merge the defaults and pipeline
configuration into the new combined configuration file
Please do not hesitate to ask.
The deb-packages are also already available for the newly released
Debian 11 Bullseye.
We are not planning a bugfix release until the 3.1.0 release, so that
one will be the next version to be released.
Here's a short summary of what happened during the summer:
- various fixes related to the IEP001 implementation (IEP001 was the
change configuration format and merge of files, rewrite oft the internal
parameter-handling)
- removal of the malwaredomains feed and parser, because it does not
exist anymore
- Various fixes in the Shadowserver Parser and support for new reports:
Vulnerable SMTP Server, Microsoft Sinkhole Events Report & Microsoft
Sinkhole HTTP Events Report, Honeypot HTTP Scan
- SMTP Output: Added Content-Disposition-header to the attachment,
fixing the display in MS Outlook clients (as reported and dicussed on
the Mailinglist).
- Heavy refactoring of IntelMQ-Manager's JavaScript parts to fix errors
and usability issues.
If you are interested in developing on IntelMQ and you don't know where
to start, have a look at the dev guide an the issues labeled "good first
issue":
https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3A…
We are especially welcoming contributions to the documentation!
You can read the full changelogs here:
- https://github.com/certtools/intelmq/releases/tag/3.0.1
- https://github.com/certtools/intelmq-api/releases/tag/3.0.1
- https://github.com/certtools/intelmq-manager/releases/tag/3.0.1https://cert.at/en/blog/2021/9/intelmq-301-releasehttps://twitter.com/CERT_at/status/1433475188381806594
btw:
There's new contact management portal called "tuency" for administrating
abuse contacts available, which can be used in conjunction with IntelMQ.
Read more about its features here:
https://cert.at/en/blog/2021/9/tuency-constituency-portal-for-iocs-and-certshttps://gitlab.com/intevation/tuency/tuency
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg