[IntelMQ-dev] Shadowserver parser extract_cve_from_tag

[Thomas Hungenberg]

Hi,

like for other Shadowserver reports, we are sending email notifications to
affected network owners in Germany based on the Vulnerable-HTTP report.
Unlike other reports (like Open-Portmapper), the Vulnerable-HTTP report
contains information on different kinds of software/systems.
So we are sending the notifications with a different template for each
software/system.

For some software/systems (like Fortinet or VMWare), the report contains
information on different vulnerabilities (CVEs) and each host included in
the report can be affected by one or more of them as listed in "tag", e.g.

"cve-2019-5544;cve-2020-3992;cve-2021-21974;vmware"
"cve-2020-3992;cve-2021-21974;vmware"
"cve-2019-5544;vmware"
"cve-2021-21974;vmware"
"cve-2021-21972;cve-2023-34048;vcenter-dcerpc-exposed;vmware"
"cve-2021-21972;cve-2023-20892;cve-2023-34048;vmware"

So we need to tell the recipients of our notifications to which of the
vulnerabilities their host is exactly vulnerable to.
We do this by adding a column "cve" in addition to "ip", "port", etc.
to the CSV data included with our notifications.

However, including the raw "tag" attribute from the Shadowserver report
would confuse the recipients as "tag" often includes additional information
not relevant in the context of the specific report like "ssl-poodle", e.g.

"cve-2024-23897;jenkins;ssl;ssl-poodle"
"cve-2023-46805;cve-2024-21887;ivanti-connect-secure;ssl;ssl-freak;vpn"

So we added a function "extract_cve_from_tag" to our local Shadowserver parser
_config.py (included with IntelMQ 3.2.1) which returns a sorted comma separated
list of CVEs included in "tag", e.g.

extract_cve_from_tag("cve-2023-46805;cve-2024-21887;ivanti-connect-secure;ssl;ssl-freak;vpn")
-> "cve-2023-46805,cve-2024-21887"

The result is stored in "extra.cve" by the parser

scan_http_vulnerable = {
     'optional_fields': [
         ('protocol.transport', 'protocol'),
         ('source.reverse_dns', 'hostname'),
         ('extra.', 'tag', validate_to_none),
         ('extra.cve', 'tag', extract_cve_from_tag),

and this way can later easily be included in the CSV data generated for
the notifications.

I think this might be useful for others as well and would be glad if it
could be added to the parser code and schema for the next release
so we would not need to patch it locally after each update of the parser/schema.

I have just created a pull request for this:
https://github.com/certtools/intelmq/pull/2457


@elsif: If the PR is accepted, would you like to change the schema
for scan_http_vulnerable and scan6_http_vulnerable like this?

          [
             "extra.",
             "tag",
             "validate_to_none"
          ],
+        [
+           "extra.cve",
+           "tag",
+           "extract_cve_from_tag"
+        ],


Kind regards
Thomas