[IntelMQ-dev] IEP009 (was: Shadowserver parser extract_cve_from_tag)

[Thomas Hungenberg]

Hi Kamil,

I like your proposal very much!

However, "product.product" sounds a bit confusing.
Maybe this can be changed to "product.cpename" or something like that?


      - Thomas


On 05.02.24 10:12, Kamil Mankowski via IntelMQ-dev wrote:
> Hey,
> 
> We definitely need to handle vulnerabilities in IntelMQ better. I have already proposed an IEP to introduce a vulnerability field in the 
> harmonization, together with related product information: https://github.com/certtools/ieps/pull/10/files What do you think about that proposal?
> 
> We currently use a little different approach than just copy CVE numbers: we have an additional bot to split events and create them in the way one CVE 
> = one event, with a CVE as identifier. This is later used to compare with other sources, so we can get the information about vulnerability from 
> different sources, but send just a one notification.
> 
> Best regards
> 
> // Kamil Mańkowski <mankowski at cert.at> - T: +43 676 898 298 7204
> // CERT Austria - https://www.cert.at/
> // CERT.at GmbH, FB-Nr. 561772k, HG Wien
> 
> On 2/5/24 10:03, Thomas Hungenberg via IntelMQ-dev wrote:
>> Hi,
>>
>> like for other Shadowserver reports, we are sending email notifications to
>> affected network owners in Germany based on the Vulnerable-HTTP report.
>> Unlike other reports (like Open-Portmapper), the Vulnerable-HTTP report
>> contains information on different kinds of software/systems.
>> So we are sending the notifications with a different template for each
>> software/system.
>>
>> For some software/systems (like Fortinet or VMWare), the report contains
>> information on different vulnerabilities (CVEs) and each host included in
>> the report can be affected by one or more of them as listed in "tag", e.g.
>>
>> "cve-2019-5544;cve-2020-3992;cve-2021-21974;vmware"
>> "cve-2020-3992;cve-2021-21974;vmware"
>> "cve-2019-5544;vmware"
>> "cve-2021-21974;vmware"
>> "cve-2021-21972;cve-2023-34048;vcenter-dcerpc-exposed;vmware"
>> "cve-2021-21972;cve-2023-20892;cve-2023-34048;vmware"
>>
>> So we need to tell the recipients of our notifications to which of the
>> vulnerabilities their host is exactly vulnerable to.
>> We do this by adding a column "cve" in addition to "ip", "port", etc.
>> to the CSV data included with our notifications.
>>
>> However, including the raw "tag" attribute from the Shadowserver report
>> would confuse the recipients as "tag" often includes additional information
>> not relevant in the context of the specific report like "ssl-poodle", e.g.
>>
>> "cve-2024-23897;jenkins;ssl;ssl-poodle"
>> "cve-2023-46805;cve-2024-21887;ivanti-connect-secure;ssl;ssl-freak;vpn"
>>
>> So we added a function "extract_cve_from_tag" to our local Shadowserver parser
>> _config.py (included with IntelMQ 3.2.1) which returns a sorted comma separated
>> list of CVEs included in "tag", e.g.
>>
>> extract_cve_from_tag("cve-2023-46805;cve-2024-21887;ivanti-connect-secure;ssl;ssl-freak;vpn")
>> -> "cve-2023-46805,cve-2024-21887"
>>
>> The result is stored in "extra.cve" by the parser
>>
>> scan_http_vulnerable = {
>>      'optional_fields': [
>>          ('protocol.transport', 'protocol'),
>>          ('source.reverse_dns', 'hostname'),
>>          ('extra.', 'tag', validate_to_none),
>>          ('extra.cve', 'tag', extract_cve_from_tag),
>>
>> and this way can later easily be included in the CSV data generated for
>> the notifications.
>>
>> I think this might be useful for others as well and would be glad if it
>> could be added to the parser code and schema for the next release
>> so we would not need to patch it locally after each update of the parser/schema.
>>
>> I have just created a pull request for this:
>> https://github.com/certtools/intelmq/pull/2457
>>
>>
>> @elsif: If the PR is accepted, would you like to change the schema
>> for scan_http_vulnerable and scan6_http_vulnerable like this?
>>
>>           [
>>              "extra.",
>>              "tag",
>>              "validate_to_none"
>>           ],
>> +        [
>> +           "extra.cve",
>> +           "tag",
>> +           "extract_cve_from_tag"
>> +        ],
>>
>>
>> Kind regards
>> Thomas
>> _______________________________________________
>> IntelMQ-dev mailing list
>> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
>> https://intelmq.readthedocs.io/
> 
> _______________________________________________
> IntelMQ-dev mailing list
> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
> https://intelmq.readthedocs.io/