Hey IntelMQ-Users :wave:
I'd like to invite you to CERT.at's IntelMQ presentation.
The presentation will be about 3 topics:
- A short summary of what IntelMQ is & how it works under the hood
- See it in action how CERT.at uses it
- IntelMQ Toolset & Ecosystem
Date/Time: 2021-12-09 13:00 - 14:30 UTC+1 (CET)
Location: Zoom:
https://us06web.zoom.us/j/84397883968?pwd=dm4xSFdlTUZUdmV2dVlFeGk3ZlRZdz09
Meeting-ID: 843 9788 3968
Code: 031551
Presenter: Sebastian Waldbauer
No registration is required :)
Kind regards,
Sebastian
--
// Sebastian Waldbauer <waldbauer(a)cert.at> - T: +43 1 5056416 7202
// CERT.at GmbH - https://www.cert.at/
// Firmenbuchnummer 561772k, LG Wien
Dear community,
Last week I said in the announcement for 3.0.1, that we didn't plan any
further bugfix release before 3.1.0. I'm now proving myself wrong.
Narzhan reported general performance issues and we investigated the
problem on our own instances.
We found two problems which different effects. It could be the case that
you didn't even notice any performance drop, as both bugs affected only
certain bots or situations.
- When any collector creates a new report, it initializes an object of
type Report. Since five years there was a bug, that this always loaded
the harmonization.conf, instead of loading it just once at the start
(which is - correctly - the case for new events). This was never a
problem, as the Python's built-in json module was fast enough, but
ruamel.yaml is slower. If a collector didn't massively create reports,
that bug was not obvious, especially under testing conditions. And the
bug was only relevant, if stream data is processed (for example with the
Shodan stream collector).
- Two bots, the STOMP collector and the API collector, don't have a
proper process method, because they are using threading. A thread
retrieves data asynchronously and sends the data. Normally, the
process-method is running in an endless-loop, with sleeping rate_limit
seconds between. If rate_limit is 0 or not defined, this would result in
high load, therefore these bots defined the __collector_empty_process
attribute which should have prohibited the endless loop. But due to
class-restructuring, the attribute which would have been to be set was
_Bot__collector_empty_process, and the flag was not effective. That did
not only cause a loop, but as bot statistics are written to redis after
every process-call, that caused continuous connections to redis. We've
now renamed the flag to _collector_empty_process and fixed the two bots.
The flag __is_multithreadable was affected analogous, but that should
not have cause performance issues in particular.
Additionally, the bot statistics can now be disabled by setting
|statistics_host| to null (previously this caused an exception).|
|
The new version is on GitHub and PyPI, very soon in the deb/rpm
repositories. The Docker image will follow on Monday or earlier.
You can read the full changelog here:
https://github.com/certtools/intelmq/releases/tag/3.0.2
have a nice weekend
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear allies,
The discussion around the IEP04 proposal, adding meta-information to
IntelMQ messages, has stalled over the last months - first because of
the time-intensive IntelMQ 3.0 release preparations and then because of
the vacation season.
Here is the current proposal:
https://github.com/certtools/ieps/tree/main/004#readme
Aaron, Sebastian Waldbauer and myself worked on it over the summer and
also identified two open issues to be discussed:
1. The exact format of the meta-information and how to name and
structure the fields. AIL made the first move and now uses a format
similar to the previously proposed Variant "A". The IEP04 document
contains the current proposal which is in line with the AIL format:
https://github.com/certtools/ieps/tree/main/004#user-content-variant-ail
If there are no other proposals, this will most probably the way to go.
2. The format of the UUID format which we want to uniquely identify
IntelMQ events. We don't necessarily need to use the UUIDv4 format which
represents pure randomness, but also other options which include the
time and are even /time-sortable/. Sebastian Waldbauer analysed a couple
of options and summarised his results in this document:
https://github.com/certtools/ieps/blob/main/004/UUID.md
Please let us know your opinion on the different UUID options.
cheers
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
Over the past two months, IntelMQ contributors had no summer pause, but
did the final finish for IntelMQ 3.0.
A special thanks goes to Mikk Margus Möll (CERT.ee) who has put
tremendous efforts in the IntelMQ Manager tackling structural and
usability issues, mainly in the JavaScript-components!
The deb/rpm repositories did not receive the 3.0.0 release at beginning
of July to get more experience with the major changes before doing
automatic upgrades, but now they deliver the brand-new 3.0.1 version.
Please note, that the automatic upgrade procedures may still not be
fully smooth. Just now, we have noticed, that the packages contain a
small flaw, which harms the upgrade experience: The packages ship a
default configuration (the file is now called `runtime.yaml`), but only
if the file does not exist before - for new installations. But now in
this special case, we renamed the configuration from `runtime.conf` to
`runtime.yaml` and therefore, the new - default shipped - configuration
takes precedence. I hope the following commands and hints will be of
help to you.
# remove the runtime configuration shipped by the package (can be called
/etc/intelmq/runtime.*) and rename your original one to
/etc/intelmq/runtime.yaml
# the previously used runtime.conf can be used as drop-in to
runtime.yaml (YAML is backwards-compatible with JSON)
sudo -u intelmq intelmqctl upgrade-config -f -u v300_pipeline_file_removal
sudo -u intelmq intelmqctl upgrade-config -f -u v300_defaults_file_removal
sudo -u intelmq intelmqctl upgrade-config -f -u v301_deprecations
The last three steps are important to merge the defaults and pipeline
configuration into the new combined configuration file
Please do not hesitate to ask.
The deb-packages are also already available for the newly released
Debian 11 Bullseye.
We are not planning a bugfix release until the 3.1.0 release, so that
one will be the next version to be released.
Here's a short summary of what happened during the summer:
- various fixes related to the IEP001 implementation (IEP001 was the
change configuration format and merge of files, rewrite oft the internal
parameter-handling)
- removal of the malwaredomains feed and parser, because it does not
exist anymore
- Various fixes in the Shadowserver Parser and support for new reports:
Vulnerable SMTP Server, Microsoft Sinkhole Events Report & Microsoft
Sinkhole HTTP Events Report, Honeypot HTTP Scan
- SMTP Output: Added Content-Disposition-header to the attachment,
fixing the display in MS Outlook clients (as reported and dicussed on
the Mailinglist).
- Heavy refactoring of IntelMQ-Manager's JavaScript parts to fix errors
and usability issues.
If you are interested in developing on IntelMQ and you don't know where
to start, have a look at the dev guide an the issues labeled "good first
issue":
https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3A…
We are especially welcoming contributions to the documentation!
You can read the full changelogs here:
- https://github.com/certtools/intelmq/releases/tag/3.0.1
- https://github.com/certtools/intelmq-api/releases/tag/3.0.1
- https://github.com/certtools/intelmq-manager/releases/tag/3.0.1https://cert.at/en/blog/2021/9/intelmq-301-releasehttps://twitter.com/CERT_at/status/1433475188381806594
btw:
There's new contact management portal called "tuency" for administrating
abuse contacts available, which can be used in conjunction with IntelMQ.
Read more about its features here:
https://cert.at/en/blog/2021/9/tuency-constituency-portal-for-iocs-and-certshttps://gitlab.com/intevation/tuency/tuency
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear Community
Please I need help to configure the SMTP bots of Intelmq to send CSV file as describe in the description. I check the MIME and I is not the blocking point. The Information needed to be sent in CSV is either in the ATT00001 file and contrary to all the Intelmq mail I do receive mine is not .txt.
Can someone please help me to solve this issue?
This is an example of the file sent after the configuration:
[cid:image001.png@01D77335.33F95C40]
Cordialement / Best Regards,
Kossi DOH
Analyste Cyber Securite
CYBER DEFENSE AFRICA S.A.S.
Mobile: +228 70 54 93 26
https://cert.tg
This information is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Unauthorised use of this information by a person or entity other than the intended recipient is prohibited by law. If you received this by mistake, please immediately contact the sender by email or phone and delete this information from any computer. Thank you.
Hello everyone,
I wonder if there is a simple way to use a Deduplicator bot on an optional field. Indeed, I noticed when I apply the deduplicator on an optional field that the null value must be entered in the redis because all messages (except the first one) that do not contain the field are dropped.
Is there a workaround please?
I could work around this problem by adding two Sieve bots at the exit of the precedent bot that would jump the Deduplicator bot if the message doesn't have the field, but I don't find that to be optimal. Thus, I am open to any proposal that could help me.
Regards,
Guillaume GRANJON de LÉPINEY | ggranjon(a)excellium-services.be<mailto:ggranjon@excellium-services.be> | PGP Key ID: 0xE2FD5ED1<https://pgp.circl.lu/pks/lookup?search=0xE2FD5ED1&fingerprint=on&op=index>
CERT-XLM Incident Handler @ excellium-services.com<https://excellium-services.com/>
CERT-XLM | cert(a)excellium-services.com<mailto:cert@excellium-services.com> | PGP Key ID: 0xD74E5AC0<http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x67B311E5D7…>
Excellium Services Belgium N.V. | Orion Bldg, Belgicastraat 13, B-1930 Zaventem, Belgium
Mobile: +32 4 71 98 57 65
Emergency: +352 262 039 64 708 | emergency(a)excellium-services.com<mailto:emergency@excellium-services.com> | PGP Key ID: 0x42662EFE<https://excellium-services.com/assets/EMERGENCY_PKEY.asc>
This email is confidential and may contain legally privileged information. If you are not the intended recipient, you should not copy, distribute, disclose or use the information it contains, please e-mail the sender immediately and delete this message from your system. Note: e-mails are susceptible to corruption, interception and unauthorised amendment; we do not accept liability for any such changes, or for their consequences. You should be aware that we may monitor your e-mails and their content. Excellium Services SA.
Dear community,
The time has come and IntelMQ 3.0 is final! We, as community, made loads
of changes, smaller and bigger ones, and I really think that IntelMQ
became more user-friendly, developer-friendly and feature-rich at the
same time!
There are some major changes in this release, especially the in the
field of the configuration and Internal Data format (previously:
"harmonization"). For the configuration-part, the upgrade part should be
automatic with `intelmqctl upgrade-config` as usual. For the Data
format, carefully look at your bot configurations (filters, sieve, etc.)
to update them. Adaptions in systems connected to IntelMQ, especially
also databases might be necessary as well. The NEWS.md file give a
summary of what has changed:
https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-…
We don't recommend to upgrade existing production instance of IntelMQ
yet. We of course did testing, including the end-to-end tests, and have
detailed release notes. But for critical systems, a delayed upgrade
makes sense ;)
Therefore the stable deb/rpm repositories don't contain the 3.0 release
yet! Even though an upgrade of production systems is not yet
recommended, extensive usage and testing of the new releases are very
much welcome and required to get the necessary feedback for the next
(maintenance) releases.
The releases are available via git, PyPI, Docker and the *unstable*
deb/rpm repositories.
Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html
IntelMQ API documentation:
https://intelmq.readthedocs.io/en/maintenance/user/intelmq-api.html
IntelMQ Manager documentation:
https://intelmq.readthedocs.io/en/maintenance/user/intelmq-manager.html
NEWS/release notes of IntelMQ (Core):
https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-…
Full Changelog of IntelMQ (Core):
https://github.com/certtools/intelmq/blob/maintenance/CHANGELOG.md#300-2021…
On a high level, these are the major changes compared to version 2.3.x
(2.3.3 was released 2021-05-31):
In the core and Docker:
* Configuration rewrite including parameter loading and handling
(IEP01), plus the required adoption of the API and Manager, by
Birger Schacht (CERT.at).
* Classification sync with RSIT, by Sebastian Wagner (CERT.at).
* Removal of the BOTS file, by Sebastian Waldbauer (CERT.at).
* Creation and maintenance of the Docker images by Sebastian Waldbauer
(CERT.at).
* Creation of Docker-instructions for development setups by Einar
Lanfranco and Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar).
New and majorly enhanced bots:
* Added |intelmq.bots.collectors.fireeye|: A bot that collects
indicators from Fireeye MAS appliances (PR#1745 by Christopher
Schappelwein).
* |intelmq.bots.collectors.api.collector_api|: Added UNIX socket
capability (PR#1987 by Mikk Margus Möll, fixes #1986).
* Added |intelmq.bots.parsers.fireeye|: A bot that parses hashes and
URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein).
* Added |intelmq.bots.experts.http.expert_status|: A bot that fetches
the HTTP Status for a given URI and adds it to the message (PR#1789
by Birger Schacht, fixes #1047 partly).
* Added |intelmq.bots.experts.http.expert_content|: A bot that fetches
an HTTP resource and checks if it contains a specific string
(PR#1811 by Birger Schacht).
* Added |intelmq.bots.experts.lookyloo.expert|: A bot that sends
requests to a lookyloo instance & adds |screenshot_url| to the event
(PR#1844 by Sebastian Waldbauer, fixes #1048).
* Added |intelmq.bots.experts.rdap.expert|: A bot that checks the RDAP
protocol for an abuse contact for a given domain (PR#1881 by
Sebastian Waldbauer and Sebastian Wagner).
* |intelmq.bots.experts.sieve.expert|: Major refactoring and lot's of
new functionality New operators for working with various types
(lists, sets, booleans, float, int), generic rule negation and
nesting (PR#1895 by Mikk Margus Möll).
* Added |intelmq.bots.experts.uwhoisd|: A bot that fetches the whois
entry from a uwhois-instance (PR#1918 by Raphaël Vinot).
* Added |intelmq.bots.experts.aggregate|: A bot that aggregate events
based upon given fields & a timespan. (PR#1959 by Sebastian Waldbauer)
* Added |intelmq.bots.experts.tuency|: A bot that queries the IntelMQ
API of a tuency instance (PR#1857 by Sebastian Wagner, fixes #1856).
* Added |intelmq.bots.outputs.templated_smtp| (PR#1901 by Karl-Johan
Karlsson).
On the documentation front, these are the most important changes
* License and copyright information was added to all the bots (by
Birger Schacht).
* Added documentation on the EventDB (PR#1955 by Birger Schacht,
PR#1985 by Sebastian Wagner).
* Added TimescaleDB for time-series documentation (PR#1990 by
Sebastian Waldbauer).
* n6 interoperability documentation: Adding more graphs and
illustrations (PR#1991 by Sebastian Wagner).
* Added documentation on abuse-contact look-ups (PR#2021 by Sebastian
Waldbauer and Sebastian Wagner).
And not to forget all the smaller changes and additions.
Thanks to (in random order)
Raphaël Vinot (circl.lu)
Bernhard Reiter (intevation.de)
Sebastian Wagner (CERT.AT)
Filip Pokorný (CSIRT.CZ)
Guillaume GRANJON de LÉPINEY (CERT XLM excellium-services.com)
Mikk Margus Möll (CERT.ee)
Alex Kaplan
Thomas Hungenberg (CERT-BUND.DE)
Einar Lanfranco (CERT-UNLP cert.unlp.edu.ar)
Christopher Schappelwein (milCERT, BMLV.gv.at)
Marcos Gonzalez (CSIRT-RD cncs.gob.do/csirt-rd/)
Marius Karotkis (NRDCS.LT)
Sebastian Waldbauer (CERT.AT)
Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar)
Karl-Johan Karlsson (Linköping University LIU.SE)
Birger Schacht (CERT.AT)
... and all the contributors of previous releases and as well to all
reporters, supporters, etc!
best regards
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi,
On 6/7/21 10:04 AM, Ángel González Berdasco wrote:
> El lun, 07-06-2021 a las 09:49 +0200, Sebastian Wagner escribió:
>>
>> Hi,
>>
>> The SMTP output bot uses these headers for the CSV attachment:
>>
>> Content-Type: text/csv; charset="us-ascii"
>> MIME-Version: 1.0
>> Content-Transfer-Encoding: 7bit
>>
>> Maybe Outlook needs something else?
>>
>> Sebastian
>>
>
> I think the issue is the lack of name, so it would need a header like
>
> Content-Disposition: attachment; filename="somefeed.csv"
>
> Best regards
>
>
> ¹ https://datatracker.ietf.org/doc/html/rfc6266
Thanks for the hint. It looks like this header needs to be "manually" added:
https://docs.python.org/3/library/email.compat32-message.html#email.message…
here:
https://github.com/certtools/intelmq/blob/a51276054a7c5f0744ccb90cda54a03ee…
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi Intelmq Community
I will be very grateful if someone can help me solve this issue.
I finished the configuration of the Intelmq to send mail using SMTP bots but all I try the attachment is not CSV but is as below:
[cid:image001.png@01D75788.E1D91CB0]
Even though the ATT file has all the required information the receiver will not be familiar with this file. How can I get the CSV file sent by mail?
Cordialement / Best Regards,
Kossi DOH
Analyste Cyber Securite
CYBER DEFENSE AFRICA S.A.S.
Mobile: +228 70 54 93 26
www.cda.tg<http://www.cda.tg>
This information is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Unauthorised use of this information by a person or entity other than the intended recipient is prohibited by law. If you received this by mistake, please immediately contact the sender by email or phone and delete this information from any computer. Thank you.