IntelMQ-dev

intelmq-dev@lists.cert.at

1 participants
244 discussions

Question on a few feeds' field mappings in Shadowserver parser
by Mika Silander 12 Nov '24

12 Nov '24

Hi, We discovered a few Shadowserver feeds with field mappings that look somewhat odd. We use the feed mapping file https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/inte… and our Shadowserver parser bot is configured to update its own copy of this file dynamically. Our intelmq and its Shadowserver parser bot is the one from intelmq git repo branch release-3.3.1. In the intelmq.json file mentioned above, the Sandbox URL feed defines the optional input field "user_agent" to be parsed on output to "user_agent" (right?): [ "user_agent", "user_agent", "validate_to_none" ], However, the parser bot appears to output "extra.user_agent" instead. The other mapping that seemed odd was in Sinkhole Events HTTP IPv4 & IPv6 (and in Microsoft Sinkhole Events HTTP IPv4): [ "destination.url", "http_url", "convert_http_host_and_url", true ], I interpret here that the optional input field "http_url" should be mapped by the Shadowserver parser bot to "destination.url" on output, but we seem to get it mapped to "extra.http_url" instead. Is this a hickup in intelmq.json, the parser or have I (again) missed something? Anyone else seeing this? Best regards, Mika

3 4

About Shadowserver's generic Special report
by Mika Silander 03 Oct '24

03 Oct '24

Hi, We received today a Shadowserver report that gets mapped into the feed "Special" by the Shadowserver parser bot. It's fine to try to inform about vulnerabilities asap, but as this report was unknown to our checker bot, it was put on hold. The corresponding email is named "Vulnerable CUPS Special Report" which already gives an idea of what the report speaks about. This report does not seem to be documented on Shadowserver's own pages under https://www.shadowserver.org/what-we-do/network-reporting (yet?). So, here's a request: could someone who is able to update https://interchange.shadowserver.org/intelmq/v1/schema/shadowserver-schema.… turn the special report into a more specific feed definition, e.g. the report file name could be "scan_cups" (and "scan6_cups" for IPv6) and the "feed.name" field could be e.g. "Vulnerable-CUPS-Server" or similar? Another option is to create a report of its own for vulnerable CUPS servers and leave "special" as the catch-all alongside. Thank you. Br, Mika P.S: The usual disclaimer: I hope I have not misunderstood anything in the the aforementioned specs.

3 6

RFC new compromised_iot report
by elsif 24 Sep '24

24 Sep '24

Hello, We have a new report that will begin tomorrow. https://www.shadowserver.org/what-we-do/network-reporting/compromised-iot-r… Please let me know if you have any recommended changes for the following mapping for the report. Regards, Jason "compromised_iot" : { "constant_fields" : { "classification.identifier" : "compromised-iot", "classification.taxonomy" : "intrusions", "classification.type" : "system-compromise" }, "feed_name" : "Compromised-IoT-Device", "file_name" : "compromised_iot", "optional_fields" : [ [ "malware.name", "family", "validate_to_none" ], [ "extra.", "severity", "validate_to_none" ], [ "protocol.transport", "protocol" ], [ "source.reverse_dns", "hostname" ], [ "extra.", "tag" ], [ "source.asn", "asn", "invalidate_zero" ], [ "source.geolocation.cc", "geo" ], [ "source.geolocation.region", "region" ], [ "source.geolocation.city", "city" ], [ "extra.source.naics", "naics", "invalidate_zero" ], [ "extra.", "hostname_source", "validate_to_none" ], [ "extra.source.sector", "sector", "validate_to_none" ], [ "extra.", "device_vendor", "validate_to_none" ], [ "extra.", "device_type", "validate_to_none" ], [ "extra.", "device_model", "validate_to_none" ], [ "extra.", "device_version", "validate_to_none" ], [ "extra.", "source_type", "validate_to_none" ], [ "event_description.text", "category", "validate_to_none" ], [ "status", "status" ], [ "extra.", "detail", "validate_to_none" ], [ "extra.", "public_source", "validate_to_none" ], [ "source.account", "account", "validate_to_none" ], [ "extra.", "server_host_key", "validate_to_none" ], [ "extra.", "malpubkey_sha256", "validate_to_none" ] ], "required_fields" : [ [ "time.source", "timestamp", "add_UTC_to_timestamp" ], [ "source.ip", "ip", "validate_ip" ], [ "source.port", "port", "convert_int" ] ], "url" : "https://www.shadowserver.org/what-we-do/network-reporting/compromised-iot-r…" },

2 1

IntelMQ 3.3.1 bugfix release
by Sebix 03 Sep '24

03 Sep '24

Dear IntelMQ community, users, developers, and Incident Response teams! We are excited to announce the release of IntelMQ version 3.3.1, which includes important bug fixes. The updated version is available on PyPI, in the git repository, and the deb/rpm repositories. Please see the list of all changes below. Documentation: https://doc.intelmq.org/ Source code: https://github.com/certtools/intelmq Thanks to all contributors to this release, in alphabetical order: * DigitalTrustCenter * Edvard Rejthar (CSIRT.CZ) * elsif2 (Shadowserver Foundation) * Kamil Mankowski (CERT.at) * Mikk Margus Möll (CERT.ee) * Sebastian Wagner (Institute for Common Good Technology, Intevation & BSI) The full list of changes: Core ==== - `intelmq.lib.utils.drop_privileges`: When IntelMQ is called as `root` and dropping the privileges to user `intelmq`, also set the non-primary groups associated with the `intelmq` user. Makes the behaviour of running intelmqctl as `root` closer to the behaviour of `sudo -u intelmq ...` (PR#2507 by Mikk Margus Möll). - `intelmq.lib.utils.unzip`: Ignore directories themselves when extracting data to prevent the extraction of empty data for a directory entries (PR#2512 by Kamil Mankowski). Bots ==== Collectors ---------- - `intelmq.bots.collectors.shadowserver.collector_reports_api.py`: - Added support for the types parameter to be either a string or a list (PR#2495 by elsif2). - Refactored to utilize the type field returned by the API to match the requested types instead of a sub-string match on the filename. - Fixed timezone issue for collecting reports (PR#2506 by elsif2). - Fixed behaviour if parameter `reports` value is empty string, behave the same way as not set, not like no report (PR#2523 by Sebastian Wagner). - `intelmq.bots.collectors.shodan.collector_stream` (PR#2492 by Mikk Margus Möll): - Add `alert` parameter to Shodan stream collector to allow fetching streams by configured alert ID - `intelmq.bots.collectors.mail._lib`: Remove deprecated parameter `attach_unzip` from default parameters (PR#2511 by Sebastian Wagner). Parsers ------- - `intelmq.bots.parsers.shadowserver._config`: - Fetch schema before first run (PR#2482 by elsif2, fixes #2480). - `intelmq.bots.parsers.dataplane.parser`: Use ` | ` as field delimiter, fix parsing of AS names including `|` (PR#2488 by DigitalTrustCenter). - all parsers: add `copy_collector_provided_fields` parameter allowing copying additional fields from the report, e.g. `extra.file_name`. (PR#2513 by Kamil Mankowski). Experts ------- - `intelmq.bots.experts.sieve.expert`: - For `:contains`, `=~` and `!~`, convert the value to string before matching avoiding an exception. If the value is a dict, convert the value to JSON (PR#2500 by Sebastian Wagner). - Add support for variables in Sieve scripts (PR#2514 by Mikk Margus Möll, fixes #2486). - `intelmq.bots.experts.filter.expert`: - Treat value `false` for parameter `filter_regex` as false (PR#2499 by Sebastian Wagner). Outputs ------- - `intelmq.bots.outputs.misp.output_feed`: Handle failures if saved current event wasn't saved or is incorrect (PR by Kamil Mankowski). - `intelmq.bots.outputs.smtp_batch.output`: Documentation on multiple recipients added (PR#2501 by Edvard Rejthar). Documentation ============= - Bots: Clarify some section of Mail collectors and the Generic CSV Parser (PR#2510 by Sebastian Wagner). -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578

1 0

Re: [IntelMQ-dev] Small unpatching issue during build of intelmq v3.3.0
by Mika Silander 04 Apr '24

04 Apr '24

-- Hi Sebastian, all, Thanks for your reply. I spent a good while digging through my Ubuntu's python3 packages and it seems some of the system python's packages were corrupt for God knows what reason (well, probably because of me). Fixing the mess was a pain but I can now confirm that, quilt unpatching succeeds as well as the "dpkg-buildpackage -us -uc -b -d" so the build runs as expected. Br, Mika ----- Original Message ----- From: "Sebix" <sebix(a)sebix.at> To: "mika silander" <mika.silander(a)csc.fi>, "intelmq-dev" <intelmq-dev(a)lists.cert.at> Sent: Thursday, 4 April, 2024 12:12:02 Subject: Re: [IntelMQ-dev] Small unpatching issue during build of intelmq v3.3.0 Hi Mika, On 4/4/24 10:33, Mika Silander via IntelMQ-dev wrote: > Gave a try on intelmq v3.3.0 (cloned from github and on the develop branch) today and > got this when running "dpkg-buildpackage -us -uc -b -d" on a Ubuntu 22.04 LTS: > > dpkg-buildpackage: warning: debian/changelog(l5): ignoring invalid week day 'Fr' > [...] > LINE: -- Aaron Kaplan <aaron(a)lo-res.org> Fr, 01 Mar 2024 14:19:00 +0100 Side note: This warning will be fixed with https://github.com/certtools/intelmq/pull/2490 > Patch fix-intelmq-paths.patch does not remove cleanly (refresh it or enforce with -f) > dh_quilt_unpatch: error: quilt --quiltrc /dev/null pop -a || test $? = 2 returned exit code 1 > make: *** [debian/rules:20: clean] Error 1 > dpkg-buildpackage: error: fakeroot debian/rules clean subprocess returned exit status 2 > > What buttons, levers, cranks and the like should be turned to get passed the patching > issue? I'd prefer not to have to enforce the thing as suggested. I don't have a solution, just a few observations and thoughts: The patch applies in our package builds on Ubuntu 22.04 (see https://build.opensuse.org/build/home:sebix:intelmq/xUbuntu_22.04/x86_64/in…) and in the GitHub CI tests using buildah (https://github.com/certtools/intelmq/actions/runs/8551898452/job/2343187443…) The latter uses dpkg-buildpackage -us -uc -b (without -d). But -d, --no-check-builddeps shouldn't make a difference for the patching. Neither the patch, nor the affected file were changed in the release (last changes of both were in 2021). best regards Sebastian -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578

1 0

Small unpatching issue during build of intelmq v3.3.0
by mika.silander＠csc.fi 04 Apr '24

04 Apr '24

-- Hi, Gave a try on intelmq v3.3.0 (cloned from github and on the develop branch) today and got this when running "dpkg-buildpackage -us -uc -b -d" on a Ubuntu 22.04 LTS: dpkg-buildpackage: warning: debian/changelog(l5): ignoring invalid week day 'Fr' LINE: -- Aaron Kaplan <aaron(a)lo-res.org> Fr, 01 Mar 2024 14:19:00 +0100 dpkg-buildpackage: info: source package intelmq dpkg-buildpackage: info: source version 3.3.0-1 dpkg-buildpackage: info: source distribution stable dpkg-buildpackage: info: source changed by Aaron Kaplan <aaron(a)lo-res.org> dpkg-buildpackage: info: host architecture amd64 dpkg-source --before-build . dpkg-source: warning: intelmq_v2/debian/changelog(l5): ignoring invalid week day 'Fr' LINE: -- Aaron Kaplan <aaron(a)lo-res.org> Fr, 01 Mar 2024 14:19:00 +0100 fakeroot debian/rules clean dh clean --with python3,sphinxdoc --without python2 --buildsystem=pybuild --with quilt --with systemd dh: warning: Compatibility levels before 10 are deprecated (level 9 in use) dh_auto_clean -O--buildsystem=pybuild dh_auto_clean: warning: Compatibility levels before 10 are deprecated (level 9 in use) I: pybuild base:239: python3.10 setup.py clean running clean removing '/home/mika/intelmqwork/intelmq_v2/.pybuild/cpython3_3.10_intelmq/build' (and everything under it) 'build/bdist.linux-x86_64' does not exist -- can't clean it 'build/scripts-3.10' does not exist -- can't clean it dh_quilt_unpatch -O--buildsystem=pybuild Patch fix-intelmq-paths.patch does not remove cleanly (refresh it or enforce with -f) dh_quilt_unpatch: error: quilt --quiltrc /dev/null pop -a || test $? = 2 returned exit code 1 make: *** [debian/rules:20: clean] Error 1 dpkg-buildpackage: error: fakeroot debian/rules clean subprocess returned exit status 2 What buttons, levers, cranks and the like should be turned to get passed the patching issue? I'd prefer not to have to enforce the thing as suggested. Br, Mika

2 1

test, please ignore
by L. Aaron Kaplan 03 Apr '24

03 Apr '24

test test

1 0

Proposed classification for new loop-dos report
by elsif 20 Mar '24

20 Mar '24

We are in the process of creating a new daily report based on the previous loop-dos special report: https://www.shadowserver.org/what-we-do/network-reporting/loop-dos-special-… I would like to propose the following constant_fields: classification.taxonomy = vulnerable classification.type = vulnerable-system protocol.application = application Where the application would be tftp or dns for example.

4 5

intelmq 3.3.0 release
by L. Aaron Kaplan 01 Mar '24

01 Mar '24

Dear intelmq users, developers and IR automation people, we are very happy to announce (finally announce!) IntelMQ 3.3.0 See: https://github.com/certtools/intelmq Important changes =================== The most relevant changes are: 1. Documentation ------------------ great new documentation (mkdocs) at https://docs.intelmq.org/ (Thanks a lot to Filip/ @gethvi! Amazing work) Documentation has been updated and restructured into User, Administrator and Developer Guide. The documentation is easier to navigate, slicker and just way more readable now. We know that the user-experience with a framework such as IntelMQ is very important. Good documentation helps a lot here. Please do check it out and give us feedback! 2. dynamic shadowserver reports and -parser ------------------------------------------- Lots of thanks to Shadowserver / @elsif2 for his continuous and dedicated contributions to make IntelMQ much better with shadowserver feeds. Motivation for this: Shadowserver adds new scans on a nearly weekly basis. IntelMQ's release cycle and the need for a stable release could not keep up with this high intensity of shadowserver parser changes. We therefore (thanks to @eslif2) move the shadowserver reports collector and parser to a new, dynamic system. It can: • fetch the shadowserver schema from shadowserver (https://interchange.shadowserver.org/intelmq/v1/schema) • dynamically collect new reports (see also https://docs.intelmq.org/latest/user/bots/?h=shadow#shadowserver-reports-api) • parse the new reports You can find all about the new shadowserver reports in IntelMQ here: https://docs.intelmq.org/latest/user/bots/#shadowserver-reports-api and https://docs.intelmq.org/latest/user/bots/#shadowserver 3. More ------------ And of course, lots of changes in the background and the core. Big thanks to Kamil and Sebix! You can find the full details here: https://github.com/certtools/intelmq/blob/develop/NEWS.md (short version) and here: https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md 4. Future of IntelMQ and release cycles ---------------------------------------- We know that the last one, two years, IntelMQ releases were a bit sporadic. We plan to change this and make it more deterministic. We therefore commit to 2 releases per year + two release candidates (intermediate, BETA) releases: Release 1: in quarter 1: March. Release 2: release-candidate in Q2 (end of June). Release 3: Sept Release 4: release-candidate release in Q4 (Dec). Also, in order to secure the future sustainability of IntelMQ, we will join the "ossbase" Open Source Security Software alliance. The idea here is to join a bigger foundation (similarly structured to the Apache Foundation) which will be created together with CIRCL.lu. Thanks Alexandre for continuing to push this idea. Being a part of ossbase will allow us to support IntelMQ in the future years while keeping the project independent, alive, kicking and a great community project. And that's what relevant. Stay tuned for updates here. 5. Thank you -------------- And finally and again: a big THANK YOU for all the committers, contributors, bug reporters, etc.!!! Without this community, we would not exist. You are mentioned here: https://github.com/certtools/intelmq/blob/develop/AUTHORS https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md https://github.com/certtools/intelmq/issues

1 0

Shadowserver parser extract_cve_from_tag
by Thomas Hungenberg 08 Feb '24

08 Feb '24

Hi, like for other Shadowserver reports, we are sending email notifications to affected network owners in Germany based on the Vulnerable-HTTP report. Unlike other reports (like Open-Portmapper), the Vulnerable-HTTP report contains information on different kinds of software/systems. So we are sending the notifications with a different template for each software/system. For some software/systems (like Fortinet or VMWare), the report contains information on different vulnerabilities (CVEs) and each host included in the report can be affected by one or more of them as listed in "tag", e.g. "cve-2019-5544;cve-2020-3992;cve-2021-21974;vmware" "cve-2020-3992;cve-2021-21974;vmware" "cve-2019-5544;vmware" "cve-2021-21974;vmware" "cve-2021-21972;cve-2023-34048;vcenter-dcerpc-exposed;vmware" "cve-2021-21972;cve-2023-20892;cve-2023-34048;vmware" So we need to tell the recipients of our notifications to which of the vulnerabilities their host is exactly vulnerable to. We do this by adding a column "cve" in addition to "ip", "port", etc. to the CSV data included with our notifications. However, including the raw "tag" attribute from the Shadowserver report would confuse the recipients as "tag" often includes additional information not relevant in the context of the specific report like "ssl-poodle", e.g. "cve-2024-23897;jenkins;ssl;ssl-poodle" "cve-2023-46805;cve-2024-21887;ivanti-connect-secure;ssl;ssl-freak;vpn" So we added a function "extract_cve_from_tag" to our local Shadowserver parser _config.py (included with IntelMQ 3.2.1) which returns a sorted comma separated list of CVEs included in "tag", e.g. extract_cve_from_tag("cve-2023-46805;cve-2024-21887;ivanti-connect-secure;ssl;ssl-freak;vpn") -> "cve-2023-46805,cve-2024-21887" The result is stored in "extra.cve" by the parser scan_http_vulnerable = { 'optional_fields': [ ('protocol.transport', 'protocol'), ('source.reverse_dns', 'hostname'), ('extra.', 'tag', validate_to_none), ('extra.cve', 'tag', extract_cve_from_tag), and this way can later easily be included in the CSV data generated for the notifications. I think this might be useful for others as well and would be glad if it could be added to the parser code and schema for the next release so we would not need to patch it locally after each update of the parser/schema. I have just created a pull request for this: https://github.com/certtools/intelmq/pull/2457 @elsif: If the PR is accepted, would you like to change the schema for scan_http_vulnerable and scan6_http_vulnerable like this? [ "extra.", "tag", "validate_to_none" ], + [ + "extra.cve", + "tag", + "extract_cve_from_tag" + ], Kind regards Thomas

3 6

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev