IntelMQ-dev

intelmq-dev@lists.cert.at

3 participants
255 discussions

Re: [IntelMQ-dev] Question on a few feeds' field mappings in Shadowserver parser
by Mika Silander 14 Nov '24

14 Nov '24

Hi, Ok, thanks. I'll consider that. Br, Mika ----- Original Message ----- From: "elsif" <elsif(a)shadowserver.org> To: "Mika Silander" <mika.silander(a)csc.fi> Sent: Tuesday, 12 November, 2024 17:03:15 Subject: Re: [IntelMQ-dev] Question on a few feeds' field mappings in Shadowserver parser Hello, Please create an issue (https://github.com/certtools/intelmq/issues/new/choose) for your proposed changes on how the parser should operate as this would be a departure from how it has historically functioned. The IPv6 scan6_ntp and scan6_ssl_poodle feeds use the more specific extra.source.naics (10449) and extra.source.sector (10459) fields while the IPv4 scan_ntp and scan_ssl_poodle feeds use extra.naics (21262) and extra.sector (21272) for backwards compatibility. Regards, Jason On 11/12/24 1:07 AM, Mika Silander wrote > Hi Jason & all, > > Thanks for the clarification. I had taken intelmq.json for a spec but, based on what you tell, it's more like a memorandum of intent. The more undocumented special cases that lurk between the lines in intelmq.json, the less useful it becomes. My strong recommendation here would be, please, document all feed input fields with their corresponding output fields. This should be a one-to-one mapping and be respected by the Shadowserver parser bot. No other fields should magically be added by the parser. Otherwise, the only way to check for event coherence is through event sampling - something I'd dearly like to avoid. > > What comes to the cases I brought up for discussion, imho there are two ways of solving them to make intelmq.json a true spec. Document both possible output field names as optional, or, better, merge the two possible field names into one and improve the parser's converter function to handle both cases. > > Following the latter option, the example of "http_url" could be output by the parser in a single field named "extra.partial_or_complete_url" (getting thus rid of the lottery between "extra.http_url" and "destination.url") and making the converter function in the parser accept both partial and complete URLs. In other words, we'd have something like the following for Sinkhole Events HTTP and friends: > > [ > "extra.", > "partial_or_complete_url", > "convert_to_possibly_incomplete_url" > ], > > For Sandbox URL, we could stick to "extra.user_agent" without exceptions: > > [ > "extra.", > "user_agent", > "validate_to_none" > ], > > This morning's harvest from our checker is: > > Shadowserver/NTP-Version: extra.source.naics, extra.source.sector > Shadowserver/SSL-POODLE-Vulnerable-Servers IPv4: extra.source.naics, extra.source.sector > > but there is no trace of these fields in intelmq.json :-(. Maybe you could include them as well in the next round of schema updates? > > And the above was only intended for improving things, not a criticism. I, like the rest of the security community, appreciate and value the work of Shadowserver. > > Br, Mika > > ----- Original Message ----- > From: "elsif" <elsif(a)shadowserver.org> > To: "intelmq-dev" <intelmq-dev(a)lists.cert.at> > Sent: Monday, 11 November, 2024 18:08:06 > Subject: Re: [IntelMQ-dev] Question on a few feeds' field mappings in Shadowserver parser > > Hello, > > I will update the user_agent in the Sandbox URL mapping to be more > explicit in the next schema update. > > The reason that the http_url is commonly not mapped to destination.url > for the Sinkhole Events HTTP reports is that the value is not a fully > qualified URL such as "/index.php" without any other context which fails > the validation for the type "URL" as specified in the harmonization > configuration. When the value fails validation it is added to extra > instead. > > Regards, > > Jason > > On 11/10/24 11:06 PM, Mika Silander via IntelMQ-dev wrote: >> Hi, >> >> We discovered a few Shadowserver feeds with field mappings that look somewhat odd. We use the feed mapping file https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/inte… and our Shadowserver parser bot is configured to update its own copy of this file dynamically. Our intelmq and its Shadowserver parser bot is the one from intelmq git repo branch release-3.3.1. >> >> In the intelmq.json file mentioned above, the Sandbox URL feed defines the optional input field "user_agent" to be parsed on output to "user_agent" (right?): >> >> [ >> "user_agent", >> "user_agent", >> "validate_to_none" >> ], >> >> However, the parser bot appears to output "extra.user_agent" instead. >> >> The other mapping that seemed odd was in Sinkhole Events HTTP IPv4 & IPv6 (and in Microsoft Sinkhole Events HTTP IPv4): >> >> [ >> "destination.url", >> "http_url", >> "convert_http_host_and_url", >> true >> ], >> >> I interpret here that the optional input field "http_url" should be mapped by the Shadowserver parser bot to "destination.url" on output, but we seem to get it mapped to "extra.http_url" instead. >> >> Is this a hickup in intelmq.json, the parser or have I (again) missed something? Anyone else seeing this? >> >> Best regards, Mika >> _______________________________________________ >> IntelMQ-dev mailing list >> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://docs.intelmq.org/ > _______________________________________________ > IntelMQ-dev mailing list > https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://docs.intelmq.org/

1 0

Question on a few feeds' field mappings in Shadowserver parser
by Mika Silander 12 Nov '24

12 Nov '24

Hi, We discovered a few Shadowserver feeds with field mappings that look somewhat odd. We use the feed mapping file https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/inte… and our Shadowserver parser bot is configured to update its own copy of this file dynamically. Our intelmq and its Shadowserver parser bot is the one from intelmq git repo branch release-3.3.1. In the intelmq.json file mentioned above, the Sandbox URL feed defines the optional input field "user_agent" to be parsed on output to "user_agent" (right?): [ "user_agent", "user_agent", "validate_to_none" ], However, the parser bot appears to output "extra.user_agent" instead. The other mapping that seemed odd was in Sinkhole Events HTTP IPv4 & IPv6 (and in Microsoft Sinkhole Events HTTP IPv4): [ "destination.url", "http_url", "convert_http_host_and_url", true ], I interpret here that the optional input field "http_url" should be mapped by the Shadowserver parser bot to "destination.url" on output, but we seem to get it mapped to "extra.http_url" instead. Is this a hickup in intelmq.json, the parser or have I (again) missed something? Anyone else seeing this? Best regards, Mika

3 4

About Shadowserver's generic Special report
by Mika Silander 03 Oct '24

03 Oct '24

Hi, We received today a Shadowserver report that gets mapped into the feed "Special" by the Shadowserver parser bot. It's fine to try to inform about vulnerabilities asap, but as this report was unknown to our checker bot, it was put on hold. The corresponding email is named "Vulnerable CUPS Special Report" which already gives an idea of what the report speaks about. This report does not seem to be documented on Shadowserver's own pages under https://www.shadowserver.org/what-we-do/network-reporting (yet?). So, here's a request: could someone who is able to update https://interchange.shadowserver.org/intelmq/v1/schema/shadowserver-schema.… turn the special report into a more specific feed definition, e.g. the report file name could be "scan_cups" (and "scan6_cups" for IPv6) and the "feed.name" field could be e.g. "Vulnerable-CUPS-Server" or similar? Another option is to create a report of its own for vulnerable CUPS servers and leave "special" as the catch-all alongside. Thank you. Br, Mika P.S: The usual disclaimer: I hope I have not misunderstood anything in the the aforementioned specs.

3 6

RFC new compromised_iot report
by elsif 24 Sep '24

24 Sep '24

Hello, We have a new report that will begin tomorrow. https://www.shadowserver.org/what-we-do/network-reporting/compromised-iot-r… Please let me know if you have any recommended changes for the following mapping for the report. Regards, Jason "compromised_iot" : { "constant_fields" : { "classification.identifier" : "compromised-iot", "classification.taxonomy" : "intrusions", "classification.type" : "system-compromise" }, "feed_name" : "Compromised-IoT-Device", "file_name" : "compromised_iot", "optional_fields" : [ [ "malware.name", "family", "validate_to_none" ], [ "extra.", "severity", "validate_to_none" ], [ "protocol.transport", "protocol" ], [ "source.reverse_dns", "hostname" ], [ "extra.", "tag" ], [ "source.asn", "asn", "invalidate_zero" ], [ "source.geolocation.cc", "geo" ], [ "source.geolocation.region", "region" ], [ "source.geolocation.city", "city" ], [ "extra.source.naics", "naics", "invalidate_zero" ], [ "extra.", "hostname_source", "validate_to_none" ], [ "extra.source.sector", "sector", "validate_to_none" ], [ "extra.", "device_vendor", "validate_to_none" ], [ "extra.", "device_type", "validate_to_none" ], [ "extra.", "device_model", "validate_to_none" ], [ "extra.", "device_version", "validate_to_none" ], [ "extra.", "source_type", "validate_to_none" ], [ "event_description.text", "category", "validate_to_none" ], [ "status", "status" ], [ "extra.", "detail", "validate_to_none" ], [ "extra.", "public_source", "validate_to_none" ], [ "source.account", "account", "validate_to_none" ], [ "extra.", "server_host_key", "validate_to_none" ], [ "extra.", "malpubkey_sha256", "validate_to_none" ] ], "required_fields" : [ [ "time.source", "timestamp", "add_UTC_to_timestamp" ], [ "source.ip", "ip", "validate_ip" ], [ "source.port", "port", "convert_int" ] ], "url" : "https://www.shadowserver.org/what-we-do/network-reporting/compromised-iot-r…" },

2 1

IntelMQ 3.3.1 bugfix release
by Sebix 03 Sep '24

03 Sep '24

Dear IntelMQ community, users, developers, and Incident Response teams! We are excited to announce the release of IntelMQ version 3.3.1, which includes important bug fixes. The updated version is available on PyPI, in the git repository, and the deb/rpm repositories. Please see the list of all changes below. Documentation: https://doc.intelmq.org/ Source code: https://github.com/certtools/intelmq Thanks to all contributors to this release, in alphabetical order: * DigitalTrustCenter * Edvard Rejthar (CSIRT.CZ) * elsif2 (Shadowserver Foundation) * Kamil Mankowski (CERT.at) * Mikk Margus Möll (CERT.ee) * Sebastian Wagner (Institute for Common Good Technology, Intevation & BSI) The full list of changes: Core ==== - `intelmq.lib.utils.drop_privileges`: When IntelMQ is called as `root` and dropping the privileges to user `intelmq`, also set the non-primary groups associated with the `intelmq` user. Makes the behaviour of running intelmqctl as `root` closer to the behaviour of `sudo -u intelmq ...` (PR#2507 by Mikk Margus Möll). - `intelmq.lib.utils.unzip`: Ignore directories themselves when extracting data to prevent the extraction of empty data for a directory entries (PR#2512 by Kamil Mankowski). Bots ==== Collectors ---------- - `intelmq.bots.collectors.shadowserver.collector_reports_api.py`: - Added support for the types parameter to be either a string or a list (PR#2495 by elsif2). - Refactored to utilize the type field returned by the API to match the requested types instead of a sub-string match on the filename. - Fixed timezone issue for collecting reports (PR#2506 by elsif2). - Fixed behaviour if parameter `reports` value is empty string, behave the same way as not set, not like no report (PR#2523 by Sebastian Wagner). - `intelmq.bots.collectors.shodan.collector_stream` (PR#2492 by Mikk Margus Möll): - Add `alert` parameter to Shodan stream collector to allow fetching streams by configured alert ID - `intelmq.bots.collectors.mail._lib`: Remove deprecated parameter `attach_unzip` from default parameters (PR#2511 by Sebastian Wagner). Parsers ------- - `intelmq.bots.parsers.shadowserver._config`: - Fetch schema before first run (PR#2482 by elsif2, fixes #2480). - `intelmq.bots.parsers.dataplane.parser`: Use ` | ` as field delimiter, fix parsing of AS names including `|` (PR#2488 by DigitalTrustCenter). - all parsers: add `copy_collector_provided_fields` parameter allowing copying additional fields from the report, e.g. `extra.file_name`. (PR#2513 by Kamil Mankowski). Experts ------- - `intelmq.bots.experts.sieve.expert`: - For `:contains`, `=~` and `!~`, convert the value to string before matching avoiding an exception. If the value is a dict, convert the value to JSON (PR#2500 by Sebastian Wagner). - Add support for variables in Sieve scripts (PR#2514 by Mikk Margus Möll, fixes #2486). - `intelmq.bots.experts.filter.expert`: - Treat value `false` for parameter `filter_regex` as false (PR#2499 by Sebastian Wagner). Outputs ------- - `intelmq.bots.outputs.misp.output_feed`: Handle failures if saved current event wasn't saved or is incorrect (PR by Kamil Mankowski). - `intelmq.bots.outputs.smtp_batch.output`: Documentation on multiple recipients added (PR#2501 by Edvard Rejthar). Documentation ============= - Bots: Clarify some section of Mail collectors and the Generic CSV Parser (PR#2510 by Sebastian Wagner). -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578

1 0

Re: [IntelMQ-dev] Small unpatching issue during build of intelmq v3.3.0
by Mika Silander 04 Apr '24

04 Apr '24

-- Hi Sebastian, all, Thanks for your reply. I spent a good while digging through my Ubuntu's python3 packages and it seems some of the system python's packages were corrupt for God knows what reason (well, probably because of me). Fixing the mess was a pain but I can now confirm that, quilt unpatching succeeds as well as the "dpkg-buildpackage -us -uc -b -d" so the build runs as expected. Br, Mika ----- Original Message ----- From: "Sebix" <sebix(a)sebix.at> To: "mika silander" <mika.silander(a)csc.fi>, "intelmq-dev" <intelmq-dev(a)lists.cert.at> Sent: Thursday, 4 April, 2024 12:12:02 Subject: Re: [IntelMQ-dev] Small unpatching issue during build of intelmq v3.3.0 Hi Mika, On 4/4/24 10:33, Mika Silander via IntelMQ-dev wrote: > Gave a try on intelmq v3.3.0 (cloned from github and on the develop branch) today and > got this when running "dpkg-buildpackage -us -uc -b -d" on a Ubuntu 22.04 LTS: > > dpkg-buildpackage: warning: debian/changelog(l5): ignoring invalid week day 'Fr' > [...] > LINE: -- Aaron Kaplan <aaron(a)lo-res.org> Fr, 01 Mar 2024 14:19:00 +0100 Side note: This warning will be fixed with https://github.com/certtools/intelmq/pull/2490 > Patch fix-intelmq-paths.patch does not remove cleanly (refresh it or enforce with -f) > dh_quilt_unpatch: error: quilt --quiltrc /dev/null pop -a || test $? = 2 returned exit code 1 > make: *** [debian/rules:20: clean] Error 1 > dpkg-buildpackage: error: fakeroot debian/rules clean subprocess returned exit status 2 > > What buttons, levers, cranks and the like should be turned to get passed the patching > issue? I'd prefer not to have to enforce the thing as suggested. I don't have a solution, just a few observations and thoughts: The patch applies in our package builds on Ubuntu 22.04 (see https://build.opensuse.org/build/home:sebix:intelmq/xUbuntu_22.04/x86_64/in…) and in the GitHub CI tests using buildah (https://github.com/certtools/intelmq/actions/runs/8551898452/job/2343187443…) The latter uses dpkg-buildpackage -us -uc -b (without -d). But -d, --no-check-builddeps shouldn't make a difference for the patching. Neither the patch, nor the affected file were changed in the release (last changes of both were in 2021). best regards Sebastian -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578

1 0

Small unpatching issue during build of intelmq v3.3.0
by mika.silander＠csc.fi 04 Apr '24

04 Apr '24

-- Hi, Gave a try on intelmq v3.3.0 (cloned from github and on the develop branch) today and got this when running "dpkg-buildpackage -us -uc -b -d" on a Ubuntu 22.04 LTS: dpkg-buildpackage: warning: debian/changelog(l5): ignoring invalid week day 'Fr' LINE: -- Aaron Kaplan <aaron(a)lo-res.org> Fr, 01 Mar 2024 14:19:00 +0100 dpkg-buildpackage: info: source package intelmq dpkg-buildpackage: info: source version 3.3.0-1 dpkg-buildpackage: info: source distribution stable dpkg-buildpackage: info: source changed by Aaron Kaplan <aaron(a)lo-res.org> dpkg-buildpackage: info: host architecture amd64 dpkg-source --before-build . dpkg-source: warning: intelmq_v2/debian/changelog(l5): ignoring invalid week day 'Fr' LINE: -- Aaron Kaplan <aaron(a)lo-res.org> Fr, 01 Mar 2024 14:19:00 +0100 fakeroot debian/rules clean dh clean --with python3,sphinxdoc --without python2 --buildsystem=pybuild --with quilt --with systemd dh: warning: Compatibility levels before 10 are deprecated (level 9 in use) dh_auto_clean -O--buildsystem=pybuild dh_auto_clean: warning: Compatibility levels before 10 are deprecated (level 9 in use) I: pybuild base:239: python3.10 setup.py clean running clean removing '/home/mika/intelmqwork/intelmq_v2/.pybuild/cpython3_3.10_intelmq/build' (and everything under it) 'build/bdist.linux-x86_64' does not exist -- can't clean it 'build/scripts-3.10' does not exist -- can't clean it dh_quilt_unpatch -O--buildsystem=pybuild Patch fix-intelmq-paths.patch does not remove cleanly (refresh it or enforce with -f) dh_quilt_unpatch: error: quilt --quiltrc /dev/null pop -a || test $? = 2 returned exit code 1 make: *** [debian/rules:20: clean] Error 1 dpkg-buildpackage: error: fakeroot debian/rules clean subprocess returned exit status 2 What buttons, levers, cranks and the like should be turned to get passed the patching issue? I'd prefer not to have to enforce the thing as suggested. Br, Mika

2 1

test, please ignore
by L. Aaron Kaplan 03 Apr '24

03 Apr '24

test test

1 0

Proposed classification for new loop-dos report
by elsif 20 Mar '24

20 Mar '24

We are in the process of creating a new daily report based on the previous loop-dos special report: https://www.shadowserver.org/what-we-do/network-reporting/loop-dos-special-… I would like to propose the following constant_fields: classification.taxonomy = vulnerable classification.type = vulnerable-system protocol.application = application Where the application would be tftp or dns for example.

4 5

intelmq 3.3.0 release
by L. Aaron Kaplan 01 Mar '24

01 Mar '24

Dear intelmq users, developers and IR automation people, we are very happy to announce (finally announce!) IntelMQ 3.3.0 See: https://github.com/certtools/intelmq Important changes =================== The most relevant changes are: 1. Documentation ------------------ great new documentation (mkdocs) at https://docs.intelmq.org/ (Thanks a lot to Filip/ @gethvi! Amazing work) Documentation has been updated and restructured into User, Administrator and Developer Guide. The documentation is easier to navigate, slicker and just way more readable now. We know that the user-experience with a framework such as IntelMQ is very important. Good documentation helps a lot here. Please do check it out and give us feedback! 2. dynamic shadowserver reports and -parser ------------------------------------------- Lots of thanks to Shadowserver / @elsif2 for his continuous and dedicated contributions to make IntelMQ much better with shadowserver feeds. Motivation for this: Shadowserver adds new scans on a nearly weekly basis. IntelMQ's release cycle and the need for a stable release could not keep up with this high intensity of shadowserver parser changes. We therefore (thanks to @eslif2) move the shadowserver reports collector and parser to a new, dynamic system. It can: • fetch the shadowserver schema from shadowserver (https://interchange.shadowserver.org/intelmq/v1/schema) • dynamically collect new reports (see also https://docs.intelmq.org/latest/user/bots/?h=shadow#shadowserver-reports-api) • parse the new reports You can find all about the new shadowserver reports in IntelMQ here: https://docs.intelmq.org/latest/user/bots/#shadowserver-reports-api and https://docs.intelmq.org/latest/user/bots/#shadowserver 3. More ------------ And of course, lots of changes in the background and the core. Big thanks to Kamil and Sebix! You can find the full details here: https://github.com/certtools/intelmq/blob/develop/NEWS.md (short version) and here: https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md 4. Future of IntelMQ and release cycles ---------------------------------------- We know that the last one, two years, IntelMQ releases were a bit sporadic. We plan to change this and make it more deterministic. We therefore commit to 2 releases per year + two release candidates (intermediate, BETA) releases: Release 1: in quarter 1: March. Release 2: release-candidate in Q2 (end of June). Release 3: Sept Release 4: release-candidate release in Q4 (Dec). Also, in order to secure the future sustainability of IntelMQ, we will join the "ossbase" Open Source Security Software alliance. The idea here is to join a bigger foundation (similarly structured to the Apache Foundation) which will be created together with CIRCL.lu. Thanks Alexandre for continuing to push this idea. Being a part of ossbase will allow us to support IntelMQ in the future years while keeping the project independent, alive, kicking and a great community project. And that's what relevant. Stay tuned for updates here. 5. Thank you -------------- And finally and again: a big THANK YOU for all the committers, contributors, bug reporters, etc.!!! Without this community, we would not exist. You are mentioned here: https://github.com/certtools/intelmq/blob/develop/AUTHORS https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md https://github.com/certtools/intelmq/issues

1 0

← Newer
1
2
3
4
5
6
7
...
26
Older →

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev