Folks,
I've been looking at some of the data that we recently processed with our IntelMQ setup (leading to https://cert.at/de/aktuelles/2023/8/verwundbare-webserver-status-in-osterrei...), and I found that we need a few changes to IntelMQ to better process these kind of feeds.
What happened? Initially, our focus with IntelMQ was on botnet drones which were detected via sinkholing. That's why we have the source.*, destination.*, and malware.* parameters in the data model. A C2 communication was the main inspiration.
These days, a majority of the data feeds we process are generated by scanning the Internet. That can be Shodan, that can be Shadowserver, that can be from our own local scans. This is fine and good, it gives us valuable telemetry on what's going on in our constituency.
But the fields needed to store the data in a consistent way are missing in the current iteration of the IDF. I don't want to pack that into extra.* or any non-standard field, I think we should come to an agreement what those elements are and how they should be handled.
Here are some of the data fields I'm missing:
What did the scanning find?
* vendor * product * software version
Are there any problems with it?
* Vulnerability identifier (CVE or other, potentially multi-valued, which is another can of worms) * Severity information (e.g. associated CVSS score) * CWE IDs (https://cwe.mitre.org/index.html)
Generic tagging
* I see e.g. "iot", "ics" or similar ones popping up.
As a minimum I think we need "vendor", "product" and "vulnerability".
What do you all think?
otmar
-----------
References:
There is e.g.:
os string Operating system platform string product string Name of the software that powers the service. vendor string
https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-http-re...
info on the vendor and CVEs is stored in the tag field.
The right way to do this is to write up a IEP (intelmq enhancement proposal) in GitHub. It’s simple. You can basically reuse the proposal below.
This idea aligns nicely with some extensions which I need for processing new shadowserver feeds. We could add these fields in one go.
i would propose you copy & paste the idea below, specify the fields in detail (data type) in the IEP, have a few sentences if this affects older installations (I assume it does not since it’s additional fields only) and then we wait a to be defined silence procedure on the mailing list and we (Kamil? Cert.at? Some one else?) implement it.
I will then add the fields for shadowserver to the IEP and we can do it in one go.
Best, Aaron.
--- Mobile
On 17.08.2023, at 17:32, Otmar Lendl via IntelMQ-dev intelmq-dev@lists.cert.at wrote:
Folks,
I've been looking at some of the data that we recently processed with our IntelMQ setup (leading to https://cert.at/de/aktuelles/2023/8/verwundbare-webserver-status-in-osterrei...), and I found that we need a few changes to IntelMQ to better process these kind of feeds.
What happened? Initially, our focus with IntelMQ was on botnet drones which were detected via sinkholing. That's why we have the source.*, destination.*, and malware.* parameters in the data model. A C2 communication was the main inspiration.
These days, a majority of the data feeds we process are generated by scanning the Internet. That can be Shodan, that can be Shadowserver, that can be from our own local scans. This is fine and good, it gives us valuable telemetry on what's going on in our constituency.
But the fields needed to store the data in a consistent way are missing in the current iteration of the IDF. I don't want to pack that into extra.* or any non-standard field, I think we should come to an agreement what those elements are and how they should be handled.
Here are some of the data fields I'm missing:
What did the scanning find?
- vendor
- product
- software version
Are there any problems with it?
- Vulnerability identifier (CVE or other, potentially multi-valued, which is another can of worms)
- Severity information (e.g. associated CVSS score)
- CWE IDs (https://cwe.mitre.org/index.html)
Generic tagging
- I see e.g. "iot", "ics" or similar ones popping up.
As a minimum I think we need "vendor", "product" and "vulnerability".
What do you all think?
otmar
References:
There is e.g.:
os string Operating system platform string product string Name of the software that powers the service. vendor string
https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-http-re...
info on the vendor and CVEs is stored in the tag field.
-- // Otmar Lendl lendl@cert.at - T: +43 1 5056416 711 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
On 17.08.2023, at 18:34, L. Aaron Kaplan aaron@lo-res.org wrote:
The right way to do this is to write up a IEP (intelmq enhancement proposal) in GitHub. It’s simple. You can basically reuse the proposal below.
Forgot to add: here are some examples: https://github.com/certtools/ieps/tree/main/
As said, I will add a new field for shadowserver as well here.
Best, a.
This idea aligns nicely with some extensions which I need for processing new shadowserver feeds. We could add these fields in one go.
i would propose you copy & paste the idea below, specify the fields in detail (data type) in the IEP, have a few sentences if this affects older installations (I assume it does not since it’s additional fields only) and then we wait a to be defined silence procedure on the mailing list and we (Kamil? Cert.at? Some one else?) implement it.
I will then add the fields for shadowserver to the IEP and we can do it in one go.
Best, Aaron.
Mobile
On 17.08.2023, at 17:32, Otmar Lendl via IntelMQ-dev intelmq-dev@lists.cert.at wrote:
Folks,
I've been looking at some of the data that we recently processed with our IntelMQ setup (leading to https://cert.at/de/aktuelles/2023/8/verwundbare-webserver-status-in-osterrei...), and I found that we need a few changes to IntelMQ to better process these kind of feeds.
What happened? Initially, our focus with IntelMQ was on botnet drones which were detected via sinkholing. That's why we have the source.*, destination.*, and malware.* parameters in the data model. A C2 communication was the main inspiration.
These days, a majority of the data feeds we process are generated by scanning the Internet. That can be Shodan, that can be Shadowserver, that can be from our own local scans. This is fine and good, it gives us valuable telemetry on what's going on in our constituency.
But the fields needed to store the data in a consistent way are missing in the current iteration of the IDF. I don't want to pack that into extra.* or any non-standard field, I think we should come to an agreement what those elements are and how they should be handled.
Here are some of the data fields I'm missing:
What did the scanning find?
- vendor
- product
- software version
Are there any problems with it?
- Vulnerability identifier (CVE or other, potentially multi-valued, which is another can of worms)
- Severity information (e.g. associated CVSS score)
- CWE IDs (https://cwe.mitre.org/index.html)
Generic tagging
- I see e.g. "iot", "ics" or similar ones popping up.
As a minimum I think we need "vendor", "product" and "vulnerability".
What do you all think?
otmar
References:
There is e.g.:
os string Operating system platform string product string Name of the software that powers the service. vendor string
https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-http-re...
info on the vendor and CVEs is stored in the tag field.
-- // Otmar Lendl lendl@cert.at - T: +43 1 5056416 711 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Let's go with an IEP for that. @Otmar – please prepare some requirements & goals, we can also discuss it internally. We have a lot of data in extra.* namespace, and I've also seen that sometimes it duplicates the standard IDF. I'm also thinking that in the future it may be helpful to introduce some versioning to the IDF - but I don't have any clear vision to this at the moment.
Best regards, Kamil Mankowski CERT.at GmbH www.cert.at
On 8/17/23 20:34, L. Aaron Kaplan wrote:
On 17.08.2023, at 18:34, L. Aaron Kaplan aaron@lo-res.org wrote:
The right way to do this is to write up a IEP (intelmq enhancement proposal) in GitHub. It’s simple. You can basically reuse the proposal below.
Forgot to add: here are some examples: https://github.com/certtools/ieps/tree/main/
As said, I will add a new field for shadowserver as well here.
Best, a.
This idea aligns nicely with some extensions which I need for processing new shadowserver feeds. We could add these fields in one go.
i would propose you copy & paste the idea below, specify the fields in detail (data type) in the IEP, have a few sentences if this affects older installations (I assume it does not since it’s additional fields only) and then we wait a to be defined silence procedure on the mailing list and we (Kamil? Cert.at? Some one else?) implement it.
I will then add the fields for shadowserver to the IEP and we can do it in one go.
Best, Aaron.
Mobile
On 17.08.2023, at 17:32, Otmar Lendl via IntelMQ-dev intelmq-dev@lists.cert.at wrote:
Folks,
I've been looking at some of the data that we recently processed with our IntelMQ setup (leading to https://cert.at/de/aktuelles/2023/8/verwundbare-webserver-status-in-osterrei...), and I found that we need a few changes to IntelMQ to better process these kind of feeds.
What happened? Initially, our focus with IntelMQ was on botnet drones which were detected via sinkholing. That's why we have the source.*, destination.*, and malware.* parameters in the data model. A C2 communication was the main inspiration.
These days, a majority of the data feeds we process are generated by scanning the Internet. That can be Shodan, that can be Shadowserver, that can be from our own local scans. This is fine and good, it gives us valuable telemetry on what's going on in our constituency.
But the fields needed to store the data in a consistent way are missing in the current iteration of the IDF. I don't want to pack that into extra.* or any non-standard field, I think we should come to an agreement what those elements are and how they should be handled.
Here are some of the data fields I'm missing:
What did the scanning find?
- vendor
- product
- software version
Are there any problems with it?
- Vulnerability identifier (CVE or other, potentially multi-valued, which is another can of worms)
- Severity information (e.g. associated CVSS score)
- CWE IDs (https://cwe.mitre.org/index.html)
Generic tagging
- I see e.g. "iot", "ics" or similar ones popping up.
As a minimum I think we need "vendor", "product" and "vulnerability".
What do you all think?
otmar
References:
There is e.g.:
os string Operating system platform string product string Name of the software that powers the service. vendor string
https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-http-re...
info on the vendor and CVEs is stored in the tag field.
-- // Otmar Lendl lendl@cert.at - T: +43 1 5056416 711 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Hi,
On 8/21/23 10:41, Kamil Mankowski via IntelMQ-dev wrote:
I'm also thinking that in the future it may be helpful to introduce some versioning to the IDF - but I don't have any clear vision to this at the moment.
We already have an IEP on Metadata including Versioning: https://github.com/certtools/ieps/tree/main/004 It's waiting for someone to implement it.
best regards Sebastian