Greetings IntelMQ community,
I'm looking for information about the harmonization.conf file located in /etc. In this file I see two sections: «event», and «report». Could you please tell me the difference? Unfortunately, I was not able to find hints in the current documentation.
Also, as I see «report», does that mean IntelMQ has a reporting capability ? As far as I know IntelMQ doesn't provide a reporting capability, hence my doubt. ??
Thanks, Jonathan
-- Jonathan SCOUPREMAN | jscoupreman@excellium-services.lumailto:jscoupreman@excellium-services.lu | PGP Key ID: 0xAD971C07http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x6802c48ead971c07 CERT-XLM | cert@excellium-services.commailto:cert@excellium-services.com | PGP Key ID: 0xD74E5AC0http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x67B311E5D74E5AC0 CERT-XLM Incident Handler @ excellium-services.comhttps://excellium-services.com/ Excellium Services S.A. | 5 rue Goell L-5326 Contern Mobile: +352 691 982 790 Emergency: +352 262 039 64 708 | emergency@excellium-services.commailto:emergency@excellium-services.com | PGP Key ID: 0x42662EFEhttps://excellium-services.com/assets/EMERGENCY_PKEY.asc
This email is confidential and may contain legally privileged information. If you are not the intended recipient, you should not copy, distribute, disclose or use the information it contains, please e-mail the sender immediately and delete this message from your system. Note: e-mails are susceptible to corruption, interception and unauthorised amendment; we do not accept liability for any such changes, or for their consequences. You should be aware that we may monitor your e-mails and their content. Excellium Services SA.
Hi Jonathan.
A report is a JSON document that is emitted by a collector bot. This document contains fields listed in the "report" section of harmonization.conf. Packed inside the report there is information about 1-N security events. Once the collector has the report ready, it sends it to a parser bot. The parser bot then splits the report document into events. An event is a JSON document describing a single security event. The fields in an event are listed in the "event" section of harmonization.conf. After the split, the parser bot sends the events one by one to the next configured expert bot in the bot chain. Eventually, the last expert bot in the chain passes the event to an output bot. The output bots could be considered as the components that provide the "reporting capability" you mention as they typically forward/store the processed event into a system outside of intelmq.
Harmonized fields are in other words, "standardised" over all reports and events intelmq processes because there are rules as to what values they are allowed to contain. Beyond these we have many non-harmonised fields that may contain (almost) arbitrary values. These non-harmonised fields are typically named "extra.*" describing the event of the corresponding security feed in more detail. Basically, these contain data that is specific to feeds, not data common to all feeds and thus not "harmonizable".
I'll stop here in the hope the above helps. More seasoned developers may continue from hereon (and correct me along the way if necessary).
Br, Mika
From: "intelmq-users" intelmq-users@lists.cert.at To: "intelmq-users" intelmq-users@lists.cert.at Sent: Friday, 17 December, 2021 14:12:50 Subject: [IntelMQ-users] [IntelMQ] Question about harmonization.conf
Greetings IntelMQ community,
I’m looking for information about the harmonization.conf file located in /etc. In this file I see two sections: «event», and «report».
Could you please tell me the difference? Unfortunately, I was not able to find hints in the current documentation.
Also, as I see «report», does that mean IntelMQ has a reporting capability ? As far as I know IntelMQ doesn’t provide a reporting capability, hence my doubt. 😊
Thanks,
Jonathan
Dear Jonathan, dear Mika,
On 12/17/21 1:12 PM, Jonathan SCOUPREMAN via IntelMQ-users wrote:
Could you please tell me the difference? Unfortunately, I was not able
to find hints in the current documentation.
Yeah, that part is missing in the docs, it was only part of (my) presentations/workshops. Needless to say: PRs for the Docs and other contributions are always welcome :)
Also, as I see «report», does that mean IntelMQ has a reporting
capability ? As far as I know IntelMQ doesn’t provide a reporting capability, hence my doubt. 😊
Depends very much on your understand of reporting.
On 12/17/21 3:17 PM, Mika Silander wrote:
I'll stop here in the hope the above helps. More seasoned developers may continue from hereon (and correct me along the way if necessary).
Excellent answer, haven't found anything to correct ;)
Small additions for context:
The report contains just raw data fetched from the source, plus some metadata of the collection process which may be needed or useful afterwards.
In IntelMQ 3.0 we have renamed the "Data Harmonization Ontology" to "IntelMQ/Internal Data Format" (IDF). The choice to name the data format "Harmonization" comes from the fact that, in the beginnings of IntelMQ back in 2015, Thomas Lima harmonized different format that were out there, most importantly AbuseHelper and ElasticSearch's ECS. The file name harmonization.conf is just a leftover from the recent renaming, which we intentionally left as is to minimize the migration burden and wait for other potential refactorings in this area (e.g. moving the file elsewhere, changing the format, completely obsolete it, ... whatever).
Sebastian
Hi Mika, Sebastian,
As always, it is a pleasure to hear from you!
Thank you for both your answers, they are very helpful. 😊
Have a nice evening/weekend, Jonathan
-- Jonathan SCOUPREMAN | jscoupreman@excellium-services.lumailto:jscoupreman@excellium-services.lu | PGP Key ID: 0xAD971C07http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x6802c48ead971c07 CERT-XLM | cert@excellium-services.commailto:cert@excellium-services.com | PGP Key ID: 0xD74E5AC0http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x67B311E5D74E5AC0 CERT-XLM Incident Handler @ excellium-services.comhttps://excellium-services.com/ Excellium Services S.A. | 5 rue Goell L-5326 Contern Mobile: +352 691 982 790 Emergency: +352 262 039 64 708 | emergency@excellium-services.commailto:emergency@excellium-services.com | PGP Key ID: 0x42662EFEhttps://excellium-services.com/assets/EMERGENCY_PKEY.asc
From: IntelMQ-users intelmq-users-bounces@lists.cert.at On Behalf Of Sebix Sent: vendredi 17 décembre 2021 16:19 To: intelmq-users intelmq-users@lists.cert.at Subject: Re: [IntelMQ-users] [IntelMQ] Question about harmonization.conf
You don't often get email from sebix@sebix.atmailto:sebix@sebix.at. Learn why this is importanthttp://aka.ms/LearnAboutSenderIdentification
Dear Jonathan, dear Mika,
On 12/17/21 1:12 PM, Jonathan SCOUPREMAN via IntelMQ-users wrote:
Could you please tell me the difference? Unfortunately, I was not able to find hints in the current documentation.
Yeah, that part is missing in the docs, it was only part of (my) presentations/workshops. Needless to say: PRs for the Docs and other contributions are always welcome :)
Also, as I see «report», does that mean IntelMQ has a reporting capability ? As far as I know IntelMQ doesn’t provide a reporting capability, hence my doubt. 😊
Depends very much on your understand of reporting. On 12/17/21 3:17 PM, Mika Silander wrote: I'll stop here in the hope the above helps. More seasoned developers may continue from hereon (and correct me along the way if necessary).
Excellent answer, haven't found anything to correct ;)
Small additions for context:
The report contains just raw data fetched from the source, plus some metadata of the collection process which may be needed or useful afterwards.
In IntelMQ 3.0 we have renamed the "Data Harmonization Ontology" to "IntelMQ/Internal Data Format" (IDF). The choice to name the data format "Harmonization" comes from the fact that, in the beginnings of IntelMQ back in 2015, Thomas Lima harmonized different format that were out there, most importantly AbuseHelper and ElasticSearch's ECS. The file name harmonization.conf is just a leftover from the recent renaming, which we intentionally left as is to minimize the migration burden and wait for other potential refactorings in this area (e.g. moving the file elsewhere, changing the format, completely obsolete it, ... whatever).
Sebastian
This email is confidential and may contain legally privileged information. If you are not the intended recipient, you should not copy, distribute, disclose or use the information it contains, please e-mail the sender immediately and delete this message from your system. Note: e-mails are susceptible to corruption, interception and unauthorised amendment; we do not accept liability for any such changes, or for their consequences. You should be aware that we may monitor your e-mails and their content. Excellium Services SA.