IntelMQ-dev

intelmq-dev@lists.cert.at

4 participants
243 discussions

Proposed mapping for Badsecrets (IPv4 and IPv6)
by elsif 09 Sep '25

09 Sep '25

Hello, Below is the proposed mapping for a new report as documented at https://www.shadowserver.org/what-we-do/network-reporting/badsecrets-report/. Please let me know if you have any changes before September 13th. Regards, Jason { "constant_fields" : { "classification.identifier" : "badsecret", "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system", "protocol.application" : "http" }, "feed_name" : "IPv6-Badsecrets", "file_name" : "scan6_badsecrets", "optional_fields" : [ [ "extra.", "severity", "validate_to_none" ], [ "protocol.transport", "protocol" ], [ "source.reverse_dns", "hostname" ], [ "extra.", "tag" ], [ "source.asn", "asn", "invalidate_zero" ], [ "source.geolocation.cc", "geo" ], [ "source.geolocation.region", "region" ], [ "source.geolocation.city", "city" ], [ "extra.source.naics", "naics", "invalidate_zero" ], [ "extra.", "hostname_source", "validate_to_none" ], [ "extra.source.sector", "sector", "validate_to_none" ], [ "extra.", "badsecret_location", "validate_to_none" ], [ "extra.", "badsecret_module", "validate_to_none" ], [ "extra.", "badsecret_type", "validate_to_none" ], [ "extra.", "badsecret_product", "validate_to_none" ], [ "extra.", "http", "validate_to_none" ], [ "extra.", "http_code", "convert_int" ], [ "extra.", "server", "validate_to_none" ], [ "extra.", "request_path", "validate_to_none" ], [ "extra.", "cert_serial_number", "validate_to_none" ], [ "extra.", "subject_common_name", "validate_to_none" ], [ "extra.", "issuer_common_name", "validate_to_none" ], [ "extra.", "subject_organization_name", "validate_to_none" ], [ "extra.", "issuer_organization_name", "validate_to_none" ], [ "extra.", "sha1_fingerprint", "validate_to_none" ], [ "extra.", "sha256_fingerprint", "validate_to_none" ] ], "required_fields" : [ [ "time.source", "timestamp", "add_UTC_to_timestamp" ], [ "source.ip", "ip", "validate_ip" ], [ "source.port", "port", "convert_int" ] ], "url" : "https://www.shadowserver.org/what-we-do/network-reporting/badsecrets-report/" } { "constant_fields" : { "classification.identifier" : "badsecret", "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system", "protocol.application" : "http" }, "feed_name" : "Badsecrets", "file_name" : "scan_badsecrets", "optional_fields" : [ [ "extra.", "severity", "validate_to_none" ], [ "protocol.transport", "protocol" ], [ "source.reverse_dns", "hostname" ], [ "extra.", "tag", "validate_to_none" ], [ "source.asn", "asn", "invalidate_zero" ], [ "source.geolocation.cc", "geo" ], [ "source.geolocation.region", "region" ], [ "source.geolocation.city", "city" ], [ "extra.source.naics", "naics", "invalidate_zero" ], [ "extra.", "hostname_source", "validate_to_none" ], [ "extra.source.sector", "sector", "validate_to_none" ], [ "extra.", "badsecret_location", "validate_to_none" ], [ "extra.", "badsecret_module", "validate_to_none" ], [ "extra.", "badsecret_type", "validate_to_none" ], [ "extra.", "badsecret_product", "validate_to_none" ], [ "extra.", "http", "validate_to_none" ], [ "extra.", "http_code", "convert_int" ], [ "extra.", "server", "validate_to_none" ], [ "extra.", "request_path", "validate_to_none" ], [ "extra.", "cert_serial_number", "validate_to_none" ], [ "extra.", "subject_common_name", "validate_to_none" ], [ "extra.", "issuer_common_name", "validate_to_none" ], [ "extra.", "subject_organization_name", "validate_to_none" ], [ "extra.", "issuer_organization_name", "validate_to_none" ], [ "extra.", "sha1_fingerprint", "validate_to_none" ], [ "extra.", "sha256_fingerprint", "validate_to_none" ] ], "required_fields" : [ [ "time.source", "timestamp", "add_UTC_to_timestamp" ], [ "source.ip", "ip", "validate_ip" ], [ "source.port", "port", "convert_int" ] ], "url" : "https://www.shadowserver.org/what-we-do/network-reporting/badsecrets-report/" }

4 7

Proposed mapping for Accessible GPRS Tunneling Protocol (GTP)
by elsif 09 Sep '25

09 Sep '25

Hello, Below is the proposed mapping for a new report as documented at https://www.shadowserver.org/what-we-do/network-reporting/accessible-gprs-t…. Please let me know if you have any changes before August 26th. Regards, Jason { "constant_fields" : { "classification.identifier" : "accessible-gtp", "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system" }, "feed_name" : "Accessible-GPRS-Tunneling-Protocol-GTP", "file_name" : "scan_gtp", "optional_fields" : [ [ "extra.", "teid", "convert_int" ], [ "extra.", "sequence", "convert_int" ], [ "extra.", "severity", "validate_to_none" ], [ "protocol.transport", "protocol" ], [ "source.reverse_dns", "hostname" ], [ "extra.", "tag", "validate_to_none" ], [ "source.asn", "asn", "invalidate_zero" ], [ "source.geolocation.cc", "geo" ], [ "source.geolocation.region", "region" ], [ "source.geolocation.city", "city" ], [ "extra.source.naics", "naics", "invalidate_zero" ], [ "extra.", "hostname_source", "validate_to_none" ], [ "extra.source.sector", "sector", "validate_to_none" ], [ "extra.", "flags", "validate_to_none" ], [ "extra.", "message_type", "validate_to_none" ], [ "extra.", "message_type_text", "validate_to_none" ], [ "extra.", "message_length", "convert_int" ], [ "extra.", "recovery", "validate_to_none" ], [ "extra.", "response_size", "convert_int" ], [ "extra.", "raw_response", "validate_to_none" ] ], "required_fields" : [ [ "time.source", "timestamp", "add_UTC_to_timestamp" ], [ "source.ip", "ip", "validate_ip" ], [ "source.port", "port", "convert_int" ] ], "url" : "https://www.shadowserver.org/what-we-do/network-reporting/accessible-gprs-t…" }

3 2

Proposed mapping for Accessible ISAKMP (IPv4 and IPv6)
by elsif 05 Sep '25

05 Sep '25

Hello, Below is the proposed mapping for a new report as documented at https://www.shadowserver.org/what-we-do/network-reporting/accessible-isakmp…. Please let me know if you have any changes before September 11th. Regards, Jason { "constant_fields" : { "classification.identifier" : "accessible-ike", "classification.taxonomy" : "other", "classification.type" : "other", "protocol.application" : "ipsec" }, "feed_name" : "Accessible-ISAKMP", "file_name" : "population_isakmp", "optional_fields" : [ [ "extra.", "spi_size", "convert_int" ], [ "extra.", "severity", "validate_to_none" ], [ "protocol.transport", "protocol" ], [ "source.reverse_dns", "hostname" ], [ "extra.", "tag" ], [ "source.asn", "asn", "invalidate_zero" ], [ "source.geolocation.cc", "geo" ], [ "source.geolocation.region", "region" ], [ "source.geolocation.city", "city" ], [ "extra.source.naics", "naics", "invalidate_zero" ], [ "extra.", "hostname_source", "validate_to_none" ], [ "extra.source.sector", "sector", "validate_to_none" ], [ "extra.", "initiator_spi", "validate_to_none" ], [ "extra.", "responder_spi", "validate_to_none" ], [ "extra.", "next_payload", "validate_to_none" ], [ "extra.", "exchange_type", "validate_to_none" ], [ "extra.", "flags", "validate_to_none" ], [ "extra.", "message_id", "validate_to_none" ], [ "extra.", "next_payload2", "validate_to_none" ], [ "extra.", "domain_of_interpretation", "validate_to_none" ], [ "extra.", "protocol_id", "validate_to_none" ], [ "extra.", "notify_message_type", "validate_to_none" ], [ "extra.", "response_size", "convert_int" ], [ "extra.", "amplification", "convert_float" ] ], "required_fields" : [ [ "time.source", "timestamp", "add_UTC_to_timestamp" ], [ "source.ip", "ip", "validate_ip" ], [ "source.port", "port", "convert_int" ] ], "url" : "https://www.shadowserver.org/what-we-do/network-reporting/accessible-isakmp…" } { "constant_fields" : { "classification.identifier" : "accessible-ike", "classification.taxonomy" : "other", "classification.type" : "other", "protocol.application" : "ipsec" }, "feed_name" : "IPv6-Accessible-ISAKMP", "file_name" : "population6_isakmp", "optional_fields" : [ [ "extra.", "spi_size", "convert_int" ], [ "extra.", "severity", "validate_to_none" ], [ "protocol.transport", "protocol" ], [ "source.reverse_dns", "hostname" ], [ "extra.", "tag" ], [ "source.asn", "asn", "invalidate_zero" ], [ "source.geolocation.cc", "geo" ], [ "source.geolocation.region", "region" ], [ "source.geolocation.city", "city" ], [ "extra.source.naics", "naics", "invalidate_zero" ], [ "extra.", "hostname_source", "validate_to_none" ], [ "extra.source.sector", "sector", "validate_to_none" ], [ "extra.", "initiator_spi", "validate_to_none" ], [ "extra.", "responder_spi", "validate_to_none" ], [ "extra.", "next_payload", "validate_to_none" ], [ "extra.", "exchange_type", "validate_to_none" ], [ "extra.", "flags", "validate_to_none" ], [ "extra.", "message_id", "validate_to_none" ], [ "extra.", "next_payload2", "validate_to_none" ], [ "extra.", "domain_of_interpretation", "validate_to_none" ], [ "extra.", "protocol_id", "validate_to_none" ], [ "extra.", "notify_message_type", "validate_to_none" ], [ "extra.", "response_size", "convert_int" ], [ "extra.", "amplification", "convert_float" ] ], "required_fields" : [ [ "time.source", "timestamp", "add_UTC_to_timestamp" ], [ "source.ip", "ip", "validate_ip" ], [ "source.port", "port", "convert_int" ] ], "url" : "https://www.shadowserver.org/what-we-do/network-reporting/accessible-isakmp…" }

2 2

Proposed mapping for Accessible NTRIP
by elsif 27 Aug '25

27 Aug '25

Hello, Below is the proposed mapping for a new report as documented at https://www.shadowserver.org/what-we-do/network-reporting/accessible-ntrip-…. Please let me know if you have any changes before September 2nd. Regards, Jason { "constant_fields" : { "classification.identifier" : "accessible-ntrip", "classification.taxonomy" : "other", "classification.type" : "other", "protocol.application" : "ntrip" }, "feed_name" : "Accessible-NTRIP", "file_name" : "population_ntrip", "optional_fields" : [ [ "extra.", "authentication", "convert_bool" ], [ "extra.", "fee", "convert_bool" ], [ "extra.", "bit_rate", "convert_int" ], [ "extra.", "severity", "validate_to_none" ], [ "protocol.transport", "protocol" ], [ "source.reverse_dns", "hostname" ], [ "extra.", "tag" ], [ "source.asn", "asn", "invalidate_zero" ], [ "source.geolocation.cc", "geo" ], [ "source.geolocation.region", "region" ], [ "source.geolocation.city", "city" ], [ "extra.source.naics", "naics", "invalidate_zero" ], [ "extra.", "hostname_source", "validate_to_none" ], [ "extra.source.sector", "sector", "validate_to_none" ], [ "extra.", "ntrip", "validate_to_none" ], [ "extra.", "ntrip_code", "validate_to_none" ], [ "extra.", "ntrip_reason", "validate_to_none" ], [ "extra.", "content_type", "validate_to_none" ], [ "extra.", "connection", "validate_to_none" ], [ "extra.", "server", "validate_to_none" ], [ "extra.", "content_length", "convert_int" ], [ "extra.", "ntrip_version", "validate_to_none" ], [ "extra.", "ntrip_date", "validate_to_none" ], [ "extra.", "ntrip_type", "validate_to_none" ], [ "extra.", "mount_point", "validate_to_none" ], [ "extra.", "identifier", "validate_to_none" ], [ "extra.", "format", "validate_to_none" ], [ "extra.", "format_details", "validate_to_none" ], [ "extra.", "carrier", "validate_to_none" ], [ "extra.", "nav_system", "validate_to_none" ], [ "extra.", "network", "validate_to_none" ], [ "extra.", "ntrip_country", "validate_to_none" ], [ "extra.", "ntrip_latitude", "validate_to_none" ], [ "extra.", "ntrip_longitude", "validate_to_none" ], [ "extra.", "nmea", "validate_to_none" ], [ "extra.", "solution", "validate_to_none" ], [ "extra.", "generator", "validate_to_none" ], [ "extra.", "compression", "validate_to_none" ], [ "extra.", "misc", "validate_to_none" ], [ "extra.", "header_base64", "validate_to_none" ], [ "extra.", "body_base64", "validate_to_none" ] ], "required_fields" : [ [ "time.source", "timestamp", "add_UTC_to_timestamp" ], [ "source.ip", "ip", "validate_ip" ], [ "source.port", "port", "convert_int" ] ], "url" : "https://www.shadowserver.org/what-we-do/network-reporting/accessible-ntrip-…" }

2 1

Shadowserver: proposed mapping change for blocklist.source
by elsif 18 Jul '25

18 Jul '25

Hello, I would like to propose a change to the mapping of the 'source' field in the blocklist report which is currently mapped to 'extra.source' in IntelMQ to 'extra.source_name'. This change is necessary to avoid conflicts with the elasticsearch mapping of the 'extra.source' field where it is commonly an object instead of a string. Reports details can be found here: https://www.shadowserver.org/what-we-do/network-reporting/blocklist-report/ Regards, Jason

2 2

Problem installing intelmq-manager 3.3.0-1 to Ubuntu 24 LTS
by Mika Silander 04 Apr '25

04 Apr '25

Hi, I'm trying to set up intelmq-manager to a fresh Ubuntu 24 LTS but I keep bumping into this: E: Failed to fetch http://some-mirror-host-name-here/pub/opensuse/repositories/home%3A/sebix%3… File has unexpected size (3960404 != 3960834). Mirror sync in progress? [IP: some-mirror-hosts-IP-here 80] Hashes of expected file: - SHA256:5269aabe03adb09d03df161294dd41fe1798f3e9b4fefbb88647e13672093cbe - SHA1:2c028c8bf346d0591747d00cad5501b6e18a7e79 [weak] - MD5Sum:fbcee8e105b081b5c5edf635f5c15626 [weak] - Filesize:3960834 [weak] It's quite seldom I've encountered a mismatch in file size checks so is the package somehow broken or have I missed something? Br, Mika The information in this email may be confidential and is intended solely for the use of the individual or entity to whom it is intended. If you are not the intended recipient of this message, please delete the message and notify the sender immediately. For information on how we process personal data and our contact information, please see CSC's website: Privacy<https://csc.fi/en/privacy> T?m?n s?hk?postin tiedot voivat olla luottamuksellisia ja ne on tarkoitettu yksinomaan sen henkil?n tai yhteis?n k?ytt??n, jolle ne on osoitettu. Jos et ole viestiss? tarkoitettu vastaanottaja, tuhoa viesti ja ilmoita asiasta v?litt?m?sti viestin l?hett?j?lle. Tietoja henkil?tietojen ja yhteystietojen k?sittelyst? l?yd?t CSC:n verkkosivuilta: Tietosuoja<https://csc.fi/tietosuoja>

1 0

A question on Shadowserver's Sinkhole HTTP Referer Events feed
by Mika Silander 02 Apr '25

02 Apr '25

Hi, We received one sinkhole http referer event that got flagged as not belonging to our constituency. At a closer look, it turned out that intelmq's destination.ip field (see event4_sinkhole_http_referer in https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/inte…<https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/inte…>) contains the IP the malware tried to access (?) and then the IP of the machine possibly infected by the malware ended up in extra.http_referer_ip. Everything fine and coherent with the feed's documentation, no complaints. To deduce to whom in our constituency an event belongs, we have a bot that first looks at the source.ip field and if this does not match, then we try the destination.ip field. This heuristic works fine with all feeds but in the above case it fails. Is there a chance the report field http_referer_ip gets mapped one day in intelmq.json to intelmq's source.ip field (like in the majority of other feeds) instead of extra.http_referer_ip? Just asking since I'd like to keep things simpler and avoid work-arounds. Br, Mika The information in this email may be confidential and is intended solely for the use of the individual or entity to whom it is intended. If you are not the intended recipient of this message, please delete the message and notify the sender immediately. For information on how we process personal data and our contact information, please see CSC's website: Privacy<https://csc.fi/en/privacy> T?m?n s?hk?postin tiedot voivat olla luottamuksellisia ja ne on tarkoitettu yksinomaan sen henkil?n tai yhteis?n k?ytt??n, jolle ne on osoitettu. Jos et ole viestiss? tarkoitettu vastaanottaja, tuhoa viesti ja ilmoita asiasta v?litt?m?sti viestin l?hett?j?lle. Tietoja henkil?tietojen ja yhteystietojen k?sittelyst? l?yd?t CSC:n verkkosivuilta: Tietosuoja<https://csc.fi/tietosuoja>

5 5

Re: A question on Shadowserver's Sinkhole HTTP Referer Events feed
by Mika Silander 31 Mar '25

31 Mar '25

Hi Sebastian, Thanks for a quick response. It appears the IPv6 "sibling" of the feed, i.e. the event6_sinkhole_http_referer also needs the same small change of field names (extra.http_referer_ip -> source.ip). I haven't made a thorough check, so I'm not sure if these are the only ones. Br, Mika ________________________________ From: Sebastian Wagner Sent: Monday, March 31, 2025 11.41 To: intelmq-dev(a)lists.cert.at Cc: Mika Silander; elsif Subject: Re: [IntelMQ-dev] A question on Shadowserver's Sinkhole HTTP Referer Events feed Hi Mika I agree with you. As per IntelMQ's guidelines/documentation (https://docs.intelmq.org/latest/user/event/#meaning-of-source-and-destinati…) the data we care about is always in the source part, and the IP address in source.ip. Jason, can you please adapt the mapping? best regards Sebastian On 3/31/25 10:08 AM, Mika Silander via IntelMQ-dev wrote: Hi, We received one sinkhole http referer event that got flagged as not belonging to our constituency. At a closer look, it turned out that intelmq's destination.ip field (see event4_sinkhole_http_referer in https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/inte…<https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/inte…>) contains the IP the malware tried to access (?) and then the IP of the machine possibly infected by the malware ended up in extra.http_referer_ip. Everything fine and coherent with the feed's documentation, no complaints. To deduce to whom in our constituency an event belongs, we have a bot that first looks at the source.ip field and if this does not match, then we try the destination.ip field. This heuristic works fine with all feeds but in the above case it fails. Is there a chance the report field http_referer_ip gets mapped one day in intelmq.json to intelmq's source.ip field (like in the majority of other feeds) instead of extra.http_referer_ip? Just asking since I'd like to keep things simpler and avoid work-arounds. Br, Mika The information in this email may be confidential and is intended solely for the use of the individual or entity to whom it is intended. If you are not the intended recipient of this message, please delete the message and notify the sender immediately. For information on how we process personal data and our contact information, please see CSC's website: Privacy<https://csc.fi/en/privacy> Tämän sähköpostin tiedot voivat olla luottamuksellisia ja ne on tarkoitettu yksinomaan sen henkilön tai yhteisön käyttöön, jolle ne on osoitettu. Jos et ole viestissä tarkoitettu vastaanottaja, tuhoa viesti ja ilmoita asiasta välittömästi viestin lähettäjälle. Tietoja henkilötietojen ja yhteystietojen käsittelystä löydät CSC:n verkkosivuilta: Tietosuoja<https://csc.fi/tietosuoja> _______________________________________________ IntelMQ-dev mailing list -- intelmq-dev(a)lists.cert.at<mailto:intelmq-dev@lists.cert.at> To unsubscribe send an email to intelmq-dev-leave(a)lists.cert.at<mailto:intelmq-dev-leave@lists.cert.at> -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578

1 0

IntelMQ 3.4.0 with new bots and support for Ubuntu 24.04
by Sebix 14 Mar '25

14 Mar '25

Hello everyone! As we all know, Friday evenings are a great time to release a new software version, and there it is, the log awaited feature release 3.4.0! :) Version 3.4.0 of IntelMQ brings changes to 16 bots and two new bots. It contains changes of four contributors: Kamil Mankowski, Sebastian Wagner, Radek Vyhnal and Frank Westers. A big thanks goes to CSIRT.LI for making this release possible! Please refer to https://docs.intelmq.org/latest/admin/upgrade/ for upgrade instructions The release is available in the GitHub repository, on PyPI and in the deb-package repositories. Read the full NEWS and changelog here: https://github.com/certtools/intelmq/blob/develop/NEWS.md#340-feature-relea… https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md#340-feature-… The most important changes potentially requiring administration attention: - Requirements: Python 3.8 or newer is required. - /CIF 3 API Output/ is deprecated - /Twitter Collector/ is removed (was dysfunctional) - The /Twitter Parser/ is renamed to /IoC Extractor Parser/ (/intelmq.bots.parsers.ioc_extractor/). - Packages are now also available for Ubuntu 24.04. To upgrade an Ubuntu 22.04 installation to 24.04 please refer to the Ubuntu documentation: https://documentation.ubuntu.com/server/how-to/software/upgrade-your-releas… Please refer to the NEWS file linked above for more details on these changes. We encourage you to share your feedback with us, also positive news about seamless upgrades :) We don't hope you experience any abnormal behavior, but if you do, please report it to us via GitHub or e-mail. best regards Sebastian for the IntelMQ maintainer group: Aaron, Kamil, Sebastian -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578

1 0

Implementing proposed severity, constituency and product.* fields
by Kamil Mankowski 03 Mar '25

03 Mar '25

Dear IntelMQ users & developers, To follow up with changes to IntelMQ Data Format proposed over a year ago (have I heard "long To-Do list"?), I have opened following PRs: https://github.com/certtools/intelmq/pull/2575 - severity field https://github.com/certtools/intelmq/pull/2573 - constituency field (IEP008) https://github.com/certtools/intelmq/pull/2574 - product.* fields (IEP009) I'd like to ask you to look if the proposed version still answers yours needs, and eventually support merging them ;) I hope we could agree on merging soon and see them in next minor release. -- Best regards // Kamil Mańkowski <mankowski(a)cert.at> - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev