Hi,
Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed.
This is very important as our setup (and most probably others as well) heavily depends on known classification identifiers like "open-rdp" and classification types from the initial parsing of events up to notification_rules and formats/templates for mailgen. So with a change of a classification attribute lots of scripts and configs would need to be changed as well.
Looking at the current schema, I see the classification identifiers are still correct for some feeds for both IPv4 and IPv6 like here:
"scan_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
"scan6_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
However, for other feeds the classification identifier has been kept correctly for IPv4 like here:
"scan_rdp" : { "constant_fields" : { "classification.identifier" : "open-rdp",
"compromised_website" : { "constant_fields" : { "classification.identifier" : "compromised-website",
but for IPv6 it has changed to the name of the feed:
"scan6_rdp" : { "constant_fields" : { "classification.identifier" : "scan6-rdp", <- should be "open-rdp"
"compromised_website6" : { "constant_fields" : { "classification.identifier" : "compromised-website6", <- should be "compromised-website"
The classification.identifier should describe the incident (like "open-rdp") and not the source (like "scan6-rdp").
May I ask you to check and adjust all classification identifiers and types in the schema so they are consistent with the ones generated by the current _config.py?
Thanks a lot for all your work on the new schema based parser!
Kind regards Thomas
Hello,
Proposed changes are attached. Please let me know if you agree with the changes or have any alterations.
Regards
On 1/31/24 7:05 AM, Thomas Hungenberg wrote:
Hi,
Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed.
This is very important as our setup (and most probably others as well) heavily depends on known classification identifiers like "open-rdp" and classification types from the initial parsing of events up to notification_rules and formats/templates for mailgen. So with a change of a classification attribute lots of scripts and configs would need to be changed as well.
Looking at the current schema, I see the classification identifiers are still correct for some feeds for both IPv4 and IPv6 like here:
"scan_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
"scan6_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
However, for other feeds the classification identifier has been kept correctly for IPv4 like here:
"scan_rdp" : { "constant_fields" : { "classification.identifier" : "open-rdp",
"compromised_website" : { "constant_fields" : { "classification.identifier" : "compromised-website",
but for IPv6 it has changed to the name of the feed:
"scan6_rdp" : { "constant_fields" : { "classification.identifier" : "scan6-rdp", <- should be "open-rdp"
"compromised_website6" : { "constant_fields" : { "classification.identifier" : "compromised-website6", <- should be "compromised-website"
The classification.identifier should describe the incident (like "open-rdp") and not the source (like "scan6-rdp").
May I ask you to check and adjust all classification identifiers and types in the schema so they are consistent with the ones generated by the current _config.py?
Thanks a lot for all your work on the new schema based parser!
Kind regards Thomas
Hi,
thanks a lot for your prompt response and sorry for the delay on my side.
The changes look good!
However, I have made a few additional changes:
1) Make classification.identifier for honeypot_ics_scan consistent with other honeypot scans: ===================== "event_honeypot_ics_scan" : { "constant_fields" : { - "classification.identifier" : "ics", + "classification.identifier" : "honeypot-ics-scan", =====================
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
2) Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for accessible-bgp and accessible-msmq.
Not included in old _config.py, so no need to document.
3) Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for open-mysql, open-postgres, open-couchdb, open-epmd.
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
4) Correct classification.identifier for vulnerable-http: ===================== "scan_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http", + "classification.identifier" : "vulnerable-http",
"scan6_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http", + "classification.identifier" : "vulnerable-http", =====================
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Please find the updates intelmq.json attached.
Kind regards Thomas
On 31.01.24 16:42, elsif wrote:
Hello,
Proposed changes are attached. Please let me know if you agree with the changes or have any alterations.
Regards
On 1/31/24 7:05 AM, Thomas Hungenberg wrote:
Hi,
Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed.
This is very important as our setup (and most probably others as well) heavily depends on known classification identifiers like "open-rdp" and classification types from the initial parsing of events up to notification_rules and formats/templates for mailgen. So with a change of a classification attribute lots of scripts and configs would need to be changed as well.
Looking at the current schema, I see the classification identifiers are still correct for some feeds for both IPv4 and IPv6 like here:
"scan_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
"scan6_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
However, for other feeds the classification identifier has been kept correctly for IPv4 like here:
"scan_rdp" : { "constant_fields" : { "classification.identifier" : "open-rdp",
"compromised_website" : { "constant_fields" : { "classification.identifier" : "compromised-website",
but for IPv6 it has changed to the name of the feed:
"scan6_rdp" : { "constant_fields" : { "classification.identifier" : "scan6-rdp", <- should be "open-rdp"
"compromised_website6" : { "constant_fields" : { "classification.identifier" : "compromised-website6", <- should be "compromised-website"
The classification.identifier should describe the incident (like "open-rdp") and not the source (like "scan6-rdp").
May I ask you to check and adjust all classification identifiers and types in the schema so they are consistent with the ones generated by the current _config.py?
Thanks a lot for all your work on the new schema based parser!
Kind regards Thomas
Hello,
Attached is the revised schema which matches yours except population6_msmq was updated to match population_msmq. Please let me know if you have any other changes.
As we add new reports I will post the draft schema here for review so that we can get the classifications set correctly.
Regards
On 2/2/24 2:48 AM, Thomas Hungenberg wrote:
Hi,
thanks a lot for your prompt response and sorry for the delay on my side.
The changes look good!
However, I have made a few additional changes:
Make classification.identifier for honeypot_ics_scan consistent with other honeypot scans: ===================== "event_honeypot_ics_scan" : { "constant_fields" : { - "classification.identifier" : "ics",
+ "classification.identifier" : "honeypot-ics-scan",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for accessible-bgp and accessible-msmq.
Not included in old _config.py, so no need to document.
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for open-mysql, open-postgres, open-couchdb, open-epmd.
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Correct classification.identifier for vulnerable-http:
"scan_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http", + "classification.identifier" : "vulnerable-http",
"scan6_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http",
+ "classification.identifier" : "vulnerable-http",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Please find the updates intelmq.json attached.
Kind regards Thomas
On 31.01.24 16:42, elsif wrote:
Hello,
Proposed changes are attached. Please let me know if you agree with the changes or have any alterations.
Regards
On 1/31/24 7:05 AM, Thomas Hungenberg wrote:
Hi,
Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed.
This is very important as our setup (and most probably others as well) heavily depends on known classification identifiers like "open-rdp" and classification types from the initial parsing of events up to notification_rules and formats/templates for mailgen. So with a change of a classification attribute lots of scripts and configs would need to be changed as well.
Looking at the current schema, I see the classification identifiers are still correct for some feeds for both IPv4 and IPv6 like here:
"scan_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
"scan6_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
However, for other feeds the classification identifier has been kept correctly for IPv4 like here:
"scan_rdp" : { "constant_fields" : { "classification.identifier" : "open-rdp",
"compromised_website" : { "constant_fields" : { "classification.identifier" : "compromised-website",
but for IPv6 it has changed to the name of the feed:
"scan6_rdp" : { "constant_fields" : { "classification.identifier" : "scan6-rdp", <- should be "open-rdp"
"compromised_website6" : { "constant_fields" : { "classification.identifier" : "compromised-website6", <- should be "compromised-website"
The classification.identifier should describe the incident (like "open-rdp") and not the source (like "scan6-rdp").
May I ask you to check and adjust all classification identifiers and types in the schema so they are consistent with the ones generated by the current _config.py?
Thanks a lot for all your work on the new schema based parser!
Kind regards Thomas
Hello,
On 02.02.24 17:08, elsif wrote:
Attached is the revised schema which matches yours except population6_msmq was updated to match population_msmq.
Thanks for updating population6_msmq as well!
As we add new reports I will post the draft schema here for review so that we can get the classifications set correctly.
Excellent!
Kind regards Thomas
Hi, thanks for the changes and reviews! They looks good to me too!
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/2/24 11:48, Thomas Hungenberg via IntelMQ-dev wrote:
Hi,
thanks a lot for your prompt response and sorry for the delay on my side.
The changes look good!
However, I have made a few additional changes:
Make classification.identifier for honeypot_ics_scan consistent with other honeypot scans: ===================== "event_honeypot_ics_scan" : { "constant_fields" : { - "classification.identifier" : "ics",
+ "classification.identifier" : "honeypot-ics-scan",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for accessible-bgp and accessible-msmq.
Not included in old _config.py, so no need to document.
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for open-mysql, open-postgres, open-couchdb, open-epmd.
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Correct classification.identifier for vulnerable-http:
"scan_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http", + "classification.identifier" : "vulnerable-http",
"scan6_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http",
+ "classification.identifier" : "vulnerable-http",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Please find the updates intelmq.json attached.
Kind regards Thomas
On 31.01.24 16:42, elsif wrote:
Hello,
Proposed changes are attached. Please let me know if you agree with the changes or have any alterations.
Regards
On 1/31/24 7:05 AM, Thomas Hungenberg wrote:
Hi,
Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed.
This is very important as our setup (and most probably others as well) heavily depends on known classification identifiers like "open-rdp" and classification types from the initial parsing of events up to notification_rules and formats/templates for mailgen. So with a change of a classification attribute lots of scripts and configs would need to be changed as well.
Looking at the current schema, I see the classification identifiers are still correct for some feeds for both IPv4 and IPv6 like here:
"scan_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
"scan6_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
However, for other feeds the classification identifier has been kept correctly for IPv4 like here:
"scan_rdp" : { "constant_fields" : { "classification.identifier" : "open-rdp",
"compromised_website" : { "constant_fields" : { "classification.identifier" : "compromised-website",
but for IPv6 it has changed to the name of the feed:
"scan6_rdp" : { "constant_fields" : { "classification.identifier" : "scan6-rdp", <- should be "open-rdp"
"compromised_website6" : { "constant_fields" : { "classification.identifier" : "compromised-website6", <- should be "compromised-website"
The classification.identifier should describe the incident (like "open-rdp") and not the source (like "scan6-rdp").
May I ask you to check and adjust all classification identifiers and types in the schema so they are consistent with the ones generated by the current _config.py?
Thanks a lot for all your work on the new schema based parser!
Kind regards Thomas
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Or rather not fully - as @gethvi brought to my attention that most of "Accessible" or "Open" feeds should be rather classified as "potentially-unwanted-accessible" according to the taxonomy (https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/b...) - instead of vulnerable-system or other.
For many cases we have our own classification enforced - I'm attaching an extract from my configuration to compare with the original schema. It's a YAML list used to generate the final configuration later.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/5/24 10:15, Kamil Mankowski wrote:
Hi, thanks for the changes and reviews! They looks good to me too!
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/2/24 11:48, Thomas Hungenberg via IntelMQ-dev wrote:
Hi,
thanks a lot for your prompt response and sorry for the delay on my side.
The changes look good!
However, I have made a few additional changes:
Make classification.identifier for honeypot_ics_scan consistent with other honeypot scans: ===================== "event_honeypot_ics_scan" : { "constant_fields" : { - "classification.identifier" : "ics",
+ "classification.identifier" : "honeypot-ics-scan",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for accessible-bgp and accessible-msmq.
Not included in old _config.py, so no need to document.
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for open-mysql, open-postgres, open-couchdb, open-epmd.
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Correct classification.identifier for vulnerable-http:
"scan_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http", + "classification.identifier" : "vulnerable-http",
"scan6_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http",
+ "classification.identifier" : "vulnerable-http",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Please find the updates intelmq.json attached.
Kind regards Thomas
On 31.01.24 16:42, elsif wrote:
Hello,
Proposed changes are attached. Please let me know if you agree with the changes or have any alterations.
Regards
On 1/31/24 7:05 AM, Thomas Hungenberg wrote:
Hi,
Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed.
This is very important as our setup (and most probably others as well) heavily depends on known classification identifiers like "open-rdp" and classification types from the initial parsing of events up to notification_rules and formats/templates for mailgen. So with a change of a classification attribute lots of scripts and configs would need to be changed as well.
Looking at the current schema, I see the classification identifiers are still correct for some feeds for both IPv4 and IPv6 like here:
"scan_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
"scan6_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
However, for other feeds the classification identifier has been kept correctly for IPv4 like here:
"scan_rdp" : { "constant_fields" : { "classification.identifier" : "open-rdp",
"compromised_website" : { "constant_fields" : { "classification.identifier" : "compromised-website",
but for IPv6 it has changed to the name of the feed:
"scan6_rdp" : { "constant_fields" : { "classification.identifier" : "scan6-rdp", <- should be "open-rdp"
"compromised_website6" : { "constant_fields" : { "classification.identifier" : "compromised-website6", <- should be "compromised-website"
The classification.identifier should describe the incident (like "open-rdp") and not the source (like "scan6-rdp").
May I ask you to check and adjust all classification identifiers and types in the schema so they are consistent with the ones generated by the current _config.py?
Thanks a lot for all your work on the new schema based parser!
Kind regards Thomas
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Should I make further changes to the schema or should the proposed version be published?
On 2/5/24 1:46 AM, Kamil Mankowski via IntelMQ-dev wrote:
Or rather not fully - as @gethvi brought to my attention that most of "Accessible" or "Open" feeds should be rather classified as "potentially-unwanted-accessible" according to the taxonomy (https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/b...)
- instead of vulnerable-system or other.
For many cases we have our own classification enforced - I'm attaching an extract from my configuration to compare with the original schema. It's a YAML list used to generate the final configuration later.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/5/24 10:15, Kamil Mankowski wrote:
Hi, thanks for the changes and reviews! They looks good to me too!
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/2/24 11:48, Thomas Hungenberg via IntelMQ-dev wrote:
Hi,
thanks a lot for your prompt response and sorry for the delay on my side.
The changes look good!
However, I have made a few additional changes:
Make classification.identifier for honeypot_ics_scan consistent with other honeypot scans: ===================== "event_honeypot_ics_scan" : { "constant_fields" : { - "classification.identifier" : "ics",
+ "classification.identifier" : "honeypot-ics-scan",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for accessible-bgp and accessible-msmq.
Not included in old _config.py, so no need to document.
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for open-mysql, open-postgres, open-couchdb, open-epmd.
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Correct classification.identifier for vulnerable-http:
"scan_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http", + "classification.identifier" : "vulnerable-http",
"scan6_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http",
+ "classification.identifier" : "vulnerable-http",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Please find the updates intelmq.json attached.
Kind regards Thomas
On 31.01.24 16:42, elsif wrote:
Hello,
Proposed changes are attached. Please let me know if you agree with the changes or have any alterations.
Regards
On 1/31/24 7:05 AM, Thomas Hungenberg wrote:
Hi,
Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed.
This is very important as our setup (and most probably others as well) heavily depends on known classification identifiers like "open-rdp" and classification types from the initial parsing of events up to notification_rules and formats/templates for mailgen. So with a change of a classification attribute lots of scripts and configs would need to be changed as well.
Looking at the current schema, I see the classification identifiers are still correct for some feeds for both IPv4 and IPv6 like here:
"scan_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
"scan6_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
However, for other feeds the classification identifier has been kept correctly for IPv4 like here:
"scan_rdp" : { "constant_fields" : { "classification.identifier" : "open-rdp",
"compromised_website" : { "constant_fields" : { "classification.identifier" : "compromised-website",
but for IPv6 it has changed to the name of the feed:
"scan6_rdp" : { "constant_fields" : { "classification.identifier" : "scan6-rdp", <- should be "open-rdp"
"compromised_website6" : { "constant_fields" : { "classification.identifier" : "compromised-website6", <- should be "compromised-website"
The classification.identifier should describe the incident (like "open-rdp") and not the source (like "scan6-rdp").
May I ask you to check and adjust all classification identifiers and types in the schema so they are consistent with the ones generated by the current _config.py?
Thanks a lot for all your work on the new schema based parser!
Kind regards Thomas
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IMHO we should not make further significant changes to the schema now for the upcoming release but discuss in detail for which feeds the classification.type should be changed from "vulnerable-system" to "potentially-unwanted-accessible" and where the old classification.identifiers open-* should be changed to accessible-* in future versions of the schema.
- Thomas
On 05.02.24 16:38, elsif wrote:
Should I make further changes to the schema or should the proposed version be published?
On 2/5/24 1:46 AM, Kamil Mankowski via IntelMQ-dev wrote:
Or rather not fully - as @gethvi brought to my attention that most of "Accessible" or "Open" feeds should be rather classified as "potentially-unwanted-accessible" according to the taxonomy (https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/b...) - instead of vulnerable-system or other.
For many cases we have our own classification enforced - I'm attaching an extract from my configuration to compare with the original schema. It's a YAML list used to generate the final configuration later.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/5/24 10:15, Kamil Mankowski wrote:
Hi, thanks for the changes and reviews! They looks good to me too!
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/2/24 11:48, Thomas Hungenberg via IntelMQ-dev wrote:
Hi,
thanks a lot for your prompt response and sorry for the delay on my side.
The changes look good!
However, I have made a few additional changes:
Make classification.identifier for honeypot_ics_scan consistent with other honeypot scans: ===================== "event_honeypot_ics_scan" : { "constant_fields" : { - "classification.identifier" : "ics",
+ "classification.identifier" : "honeypot-ics-scan",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for accessible-bgp and accessible-msmq.
Not included in old _config.py, so no need to document.
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for open-mysql, open-postgres, open-couchdb, open-epmd.
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Correct classification.identifier for vulnerable-http:
"scan_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http", + "classification.identifier" : "vulnerable-http",
"scan6_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http",
+ "classification.identifier" : "vulnerable-http",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Please find the updates intelmq.json attached.
Kind regards Thomas
On 31.01.24 16:42, elsif wrote:
Hello,
Proposed changes are attached. Please let me know if you agree with the changes or have any alterations.
Regards
On 1/31/24 7:05 AM, Thomas Hungenberg wrote:
Hi,
Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed.
This is very important as our setup (and most probably others as well) heavily depends on known classification identifiers like "open-rdp" and classification types from the initial parsing of events up to notification_rules and formats/templates for mailgen. So with a change of a classification attribute lots of scripts and configs would need to be changed as well.
Looking at the current schema, I see the classification identifiers are still correct for some feeds for both IPv4 and IPv6 like here:
"scan_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
"scan6_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
However, for other feeds the classification identifier has been kept correctly for IPv4 like here:
"scan_rdp" : { "constant_fields" : { "classification.identifier" : "open-rdp",
"compromised_website" : { "constant_fields" : { "classification.identifier" : "compromised-website",
but for IPv6 it has changed to the name of the feed:
"scan6_rdp" : { "constant_fields" : { "classification.identifier" : "scan6-rdp", <- should be "open-rdp"
"compromised_website6" : { "constant_fields" : { "classification.identifier" : "compromised-website6", <- should be "compromised-website"
The classification.identifier should describe the incident (like "open-rdp") and not the source (like "scan6-rdp").
May I ask you to check and adjust all classification identifiers and types in the schema so they are consistent with the ones generated by the current _config.py?
Thanks a lot for all your work on the new schema based parser!
Kind regards Thomas
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
I second that. Especially when schema is now independent, let's stay with the last proposal, and speak about taxonomy separately. I haven't had time yet to deeply compare which we have differently.
When it comes to identifiers changes, I would be very conservative. They can be used for filtering, and as so - changing them is potentially dangerous. I second fixes about IPv6, those were more misleading than helping, but for the rest - we need to be careful and announce the change.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/6/24 11:11, Thomas Hungenberg wrote:
IMHO we should not make further significant changes to the schema now for the upcoming release but discuss in detail for which feeds the classification.type should be changed from "vulnerable-system" to "potentially-unwanted-accessible" and where the old classification.identifiers open-* should be changed to accessible-* in future versions of the schema.
- Thomas
On 05.02.24 16:38, elsif wrote:
Should I make further changes to the schema or should the proposed version be published?
On 2/5/24 1:46 AM, Kamil Mankowski via IntelMQ-dev wrote:
Or rather not fully - as @gethvi brought to my attention that most of "Accessible" or "Open" feeds should be rather classified as "potentially-unwanted-accessible" according to the taxonomy (https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/b...) - instead of vulnerable-system or other.
For many cases we have our own classification enforced - I'm attaching an extract from my configuration to compare with the original schema. It's a YAML list used to generate the final configuration later.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/5/24 10:15, Kamil Mankowski wrote:
Hi, thanks for the changes and reviews! They looks good to me too!
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/2/24 11:48, Thomas Hungenberg via IntelMQ-dev wrote:
Hi,
thanks a lot for your prompt response and sorry for the delay on my side.
The changes look good!
However, I have made a few additional changes:
Make classification.identifier for honeypot_ics_scan consistent with other honeypot scans: ===================== "event_honeypot_ics_scan" : { "constant_fields" : { - "classification.identifier" : "ics",
+ "classification.identifier" : "honeypot-ics-scan",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for accessible-bgp and accessible-msmq.
Not included in old _config.py, so no need to document.
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for open-mysql, open-postgres, open-couchdb, open-epmd.
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Correct classification.identifier for vulnerable-http:
"scan_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http", + "classification.identifier" : "vulnerable-http",
"scan6_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http",
+ "classification.identifier" : "vulnerable-http",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Please find the updates intelmq.json attached.
Kind regards Thomas
On 31.01.24 16:42, elsif wrote:
Hello,
Proposed changes are attached. Please let me know if you agree with the changes or have any alterations.
Regards
On 1/31/24 7:05 AM, Thomas Hungenberg wrote: > Hi, > > Sebastian (sebix) told me it was agreed that with the translation > from the current parser _config.py (included with IntelMQ 3.2.1) > to the new schema, no classification.* attributes will be changed. > > This is very important as our setup (and most probably others as > well) > heavily depends on known classification identifiers like "open-rdp" > and classification types from the initial parsing of events up to > notification_rules and formats/templates for mailgen. > So with a change of a classification attribute lots of scripts and > configs would need to be changed as well. > > Looking at the current schema, I see the classification identifiers > are still correct for some feeds for both IPv4 and IPv6 like here: > > "scan_dns" : { > "constant_fields" : { > "classification.identifier" : "dns-open-resolver", > > "scan6_dns" : { > "constant_fields" : { > "classification.identifier" : "dns-open-resolver", > > > However, for other feeds the classification identifier has been kept > correctly for IPv4 like here: > > "scan_rdp" : { > "constant_fields" : { > "classification.identifier" : "open-rdp", > > "compromised_website" : { > "constant_fields" : { > "classification.identifier" : "compromised-website", > > > but for IPv6 it has changed to the name of the feed: > > "scan6_rdp" : { > "constant_fields" : { > "classification.identifier" : "scan6-rdp", <- should be > "open-rdp" > > "compromised_website6" : { > "constant_fields" : { > "classification.identifier" : "compromised-website6", <- > should be "compromised-website" > > > The classification.identifier should describe the incident (like > "open-rdp") > and not the source (like "scan6-rdp"). > > May I ask you to check and adjust all classification identifiers > and types > in the schema so they are consistent with the ones generated by > the current > _config.py? > > > Thanks a lot for all your work on the new schema based parser! > > > Kind regards > Thomas >
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
The updated schema has been published.
On 2/6/24 4:42 AM, Kamil Mankowski wrote:
I second that. Especially when schema is now independent, let's stay with the last proposal, and speak about taxonomy separately. I haven't had time yet to deeply compare which we have differently.
When it comes to identifiers changes, I would be very conservative. They can be used for filtering, and as so - changing them is potentially dangerous. I second fixes about IPv6, those were more misleading than helping, but for the rest - we need to be careful and announce the change.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/6/24 11:11, Thomas Hungenberg wrote:
IMHO we should not make further significant changes to the schema now for the upcoming release but discuss in detail for which feeds the classification.type should be changed from "vulnerable-system" to "potentially-unwanted-accessible" and where the old classification.identifiers open-* should be changed to accessible-* in future versions of the schema.
- Thomas
On 05.02.24 16:38, elsif wrote:
Should I make further changes to the schema or should the proposed version be published?
On 2/5/24 1:46 AM, Kamil Mankowski via IntelMQ-dev wrote:
Or rather not fully - as @gethvi brought to my attention that most of "Accessible" or "Open" feeds should be rather classified as "potentially-unwanted-accessible" according to the taxonomy (https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/b...)
- instead of vulnerable-system or other.
For many cases we have our own classification enforced - I'm attaching an extract from my configuration to compare with the original schema. It's a YAML list used to generate the final configuration later.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/5/24 10:15, Kamil Mankowski wrote:
Hi, thanks for the changes and reviews! They looks good to me too!
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/2/24 11:48, Thomas Hungenberg via IntelMQ-dev wrote:
Hi,
thanks a lot for your prompt response and sorry for the delay on my side.
The changes look good!
However, I have made a few additional changes:
Make classification.identifier for honeypot_ics_scan consistent with other honeypot scans: ===================== "event_honeypot_ics_scan" : { "constant_fields" : { - "classification.identifier" : "ics",
+ "classification.identifier" : "honeypot-ics-scan",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for accessible-bgp and accessible-msmq.
Not included in old _config.py, so no need to document.
Change classification.taxonomy and classification.type from
"classification.taxonomy" : "other", "classification.type" : "other",
to "classification.taxonomy" : "vulnerable", "classification.type" : "vulnerable-system",
for open-mysql, open-postgres, open-couchdb, open-epmd.
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Correct classification.identifier for vulnerable-http:
"scan_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http", + "classification.identifier" : "vulnerable-http",
"scan6_http_vulnerable" : { "constant_fields" : { - "classification.identifier" : "accessible-http",
+ "classification.identifier" : "vulnerable-http",
This change should be documented here: https://github.com/The-Shadowserver-Foundation/report_schema/blob/main/compl...
Please find the updates intelmq.json attached.
Kind regards Thomas
On 31.01.24 16:42, elsif wrote: > Hello, > > Proposed changes are attached. Please let me know if you agree > with the changes or have any alterations. > > Regards > > On 1/31/24 7:05 AM, Thomas Hungenberg wrote: >> Hi, >> >> Sebastian (sebix) told me it was agreed that with the translation >> from the current parser _config.py (included with IntelMQ 3.2.1) >> to the new schema, no classification.* attributes will be changed. >> >> This is very important as our setup (and most probably others >> as well) >> heavily depends on known classification identifiers like >> "open-rdp" >> and classification types from the initial parsing of events up to >> notification_rules and formats/templates for mailgen. >> So with a change of a classification attribute lots of scripts and >> configs would need to be changed as well. >> >> Looking at the current schema, I see the classification >> identifiers >> are still correct for some feeds for both IPv4 and IPv6 like here: >> >> "scan_dns" : { >> "constant_fields" : { >> "classification.identifier" : "dns-open-resolver", >> >> "scan6_dns" : { >> "constant_fields" : { >> "classification.identifier" : "dns-open-resolver", >> >> >> However, for other feeds the classification identifier has been >> kept >> correctly for IPv4 like here: >> >> "scan_rdp" : { >> "constant_fields" : { >> "classification.identifier" : "open-rdp", >> >> "compromised_website" : { >> "constant_fields" : { >> "classification.identifier" : "compromised-website", >> >> >> but for IPv6 it has changed to the name of the feed: >> >> "scan6_rdp" : { >> "constant_fields" : { >> "classification.identifier" : "scan6-rdp", <- should >> be "open-rdp" >> >> "compromised_website6" : { >> "constant_fields" : { >> "classification.identifier" : "compromised-website6", >> <- should be "compromised-website" >> >> >> The classification.identifier should describe the incident >> (like "open-rdp") >> and not the source (like "scan6-rdp"). >> >> May I ask you to check and adjust all classification >> identifiers and types >> in the schema so they are consistent with the ones generated by >> the current >> _config.py? >> >> >> Thanks a lot for all your work on the new schema based parser! >> >> >> Kind regards >> Thomas >>
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
On 06.02.24 13:42, Kamil Mankowski wrote:
When it comes to identifiers changes, I would be very conservative. They can be used for filtering, and as so - changing them is potentially dangerous. > I second fixes about IPv6, those were more misleading than helping, but for the rest - we need to be careful and announce the change.
Yes, our IntelMQ setup with mailgen etc. also heavily depends on the known classification identifiers. That is why I asked not to change them with the switch to the dynamic schema.
However, Shadowserver renamed some "old" feeds from "open-*" to "accessible-*" some years ago (e.g. "open-telnet" -> "accessible-telnet"). So far, we have not adopted those changes for the classification identifiers but still use "open-telnet" etc. for "old" feeds. On the other hand, for newer feeds like "accessible-ftp" we use the classification identifier "accessible-ftp". So we have "open-telnet" but "accessible-ftp" which is not consistent.
We should probably discuss which services are "open" and which ones are "accessible" and change the classification identifiers accordingly.
Of course, all those changes need to be documented in the CHANGELOG and we should provide SQL UPDATE statements in NEWS.md like for the changes in version 3.0.0.
- Thomas
Hi all,
thanks for the lively discussion first of all, because it shows that these things matter :)
However, my question currently is: green light for release or not? What do you feel Thomas? Do you feel we need more time to discover things which would create a lot of havoc at different installations ? Or do you feel that these changes are so minimal that it's just part of the regular attentive update cycle? We can put that into the CHANGELOG and NEWS file of course. Would that be enough?
Because I have been delaying the release a bit for the discussion to settle and in order to make sure that there are as few surprises as possible for everyone with the dynamic schema update.
Best, Aaron.
On 08.02.2024, at 10:59, Thomas Hungenberg via IntelMQ-dev intelmq-dev@lists.cert.at wrote:
On 06.02.24 13:42, Kamil Mankowski wrote:
When it comes to identifiers changes, I would be very conservative. They can be used for filtering, and as so - changing them is potentially dangerous. > I second fixes about IPv6, those were more misleading than helping, but for the rest - we need to be careful and announce the change.
Yes, our IntelMQ setup with mailgen etc. also heavily depends on the known classification identifiers. That is why I asked not to change them with the switch to the dynamic schema.
However, Shadowserver renamed some "old" feeds from "open-*" to "accessible-*" some years ago (e.g. "open-telnet" -> "accessible-telnet"). So far, we have not adopted those changes for the classification identifiers but still use "open-telnet" etc. for "old" feeds. On the other hand, for newer feeds like "accessible-ftp" we use the classification identifier "accessible-ftp". So we have "open-telnet" but "accessible-ftp" which is not consistent.
We should probably discuss which services are "open" and which ones are "accessible" and change the classification identifiers accordingly.
Of course, all those changes need to be documented in the CHANGELOG and we should provide SQL UPDATE statements in NEWS.md like for the changes in version 3.0.0.
- Thomas
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
From my perspective, it shouldn't delay the release.
We should clearly note the change of ShadowServer parser, especially that it can include important data changes and link to the parser's changelog - and let users decide, whether they feel ready to upgrade their instances or not. We can note that there is ongoing discussion about improving identifiers and point to where people can participate.
In my opinion, the current delay has already more painful consequences for users than a potential profit of delaying it further. In currently published release, the STOMP collector doesn't work with n6, new ShadowServer feeds are not supported, schema of the rest of them is outdated. New release brings also a possibility for easier extending with custom bots, without dealing with modifying IntelMQ manually.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/8/24 11:11, L. Aaron Kaplan wrote:
Hi all,
thanks for the lively discussion first of all, because it shows that these things matter :)
However, my question currently is: green light for release or not? What do you feel Thomas? Do you feel we need more time to discover things which would create a lot of havoc at different installations ? Or do you feel that these changes are so minimal that it's just part of the regular attentive update cycle? We can put that into the CHANGELOG and NEWS file of course. Would that be enough?
Because I have been delaying the release a bit for the discussion to settle and in order to make sure that there are as few surprises as possible for everyone with the dynamic schema update.
Best, Aaron.
On 08.02.2024, at 10:59, Thomas Hungenberg via IntelMQ-dev intelmq-dev@lists.cert.at wrote:
On 06.02.24 13:42, Kamil Mankowski wrote:
When it comes to identifiers changes, I would be very conservative. They can be used for filtering, and as so - changing them is potentially dangerous. > I second fixes about IPv6, those were more misleading than helping, but for the rest - we need to be careful and announce the change.
Yes, our IntelMQ setup with mailgen etc. also heavily depends on the known classification identifiers. That is why I asked not to change them with the switch to the dynamic schema.
However, Shadowserver renamed some "old" feeds from "open-*" to "accessible-*" some years ago (e.g. "open-telnet" -> "accessible-telnet"). So far, we have not adopted those changes for the classification identifiers but still use "open-telnet" etc. for "old" feeds. On the other hand, for newer feeds like "accessible-ftp" we use the classification identifier "accessible-ftp". So we have "open-telnet" but "accessible-ftp" which is not consistent.
We should probably discuss which services are "open" and which ones are "accessible" and change the classification identifiers accordingly.
Of course, all those changes need to be documented in the CHANGELOG and we should provide SQL UPDATE statements in NEWS.md like for the changes in version 3.0.0.
- Thomas
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
I agree with everything being said, personally. But I would like to hear Thomas' opinion as well since they shovel *a lot* of data through intelmq.
On 08.02.2024, at 11:21, Kamil Mankowski mankowski@cert.at wrote:
From my perspective, it shouldn't delay the release.
We should clearly note the change of ShadowServer parser, especially that it can include important data changes and link to the parser's changelog - and let users decide, whether they feel ready to upgrade their instances or not. We can note that there is ongoing discussion about improving identifiers and point to where people can participate.
In my opinion, the current delay has already more painful consequences for users than a potential profit of delaying it further. In currently published release, the STOMP collector doesn't work with n6, new ShadowServer feeds are not supported, schema of the rest of them is outdated. New release brings also a possibility for easier extending with custom bots, without dealing with modifying IntelMQ manually.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/8/24 11:11, L. Aaron Kaplan wrote:
Hi all, thanks for the lively discussion first of all, because it shows that these things matter :) However, my question currently is: green light for release or not? What do you feel Thomas? Do you feel we need more time to discover things which would create a lot of havoc at different installations ? Or do you feel that these changes are so minimal that it's just part of the regular attentive update cycle? We can put that into the CHANGELOG and NEWS file of course. Would that be enough? Because I have been delaying the release a bit for the discussion to settle and in order to make sure that there are as few surprises as possible for everyone with the dynamic schema update. Best, Aaron.
On 08.02.2024, at 10:59, Thomas Hungenberg via IntelMQ-dev intelmq-dev@lists.cert.at wrote:
On 06.02.24 13:42, Kamil Mankowski wrote:
When it comes to identifiers changes, I would be very conservative. They can be used for filtering, and as so - changing them is potentially dangerous. > I second fixes about IPv6, those were more misleading than helping, but for the rest - we need to be careful and announce the change.
Yes, our IntelMQ setup with mailgen etc. also heavily depends on the known classification identifiers. That is why I asked not to change them with the switch to the dynamic schema.
However, Shadowserver renamed some "old" feeds from "open-*" to "accessible-*" some years ago (e.g. "open-telnet" -> "accessible-telnet"). So far, we have not adopted those changes for the classification identifiers but still use "open-telnet" etc. for "old" feeds. On the other hand, for newer feeds like "accessible-ftp" we use the classification identifier "accessible-ftp". So we have "open-telnet" but "accessible-ftp" which is not consistent.
We should probably discuss which services are "open" and which ones are "accessible" and change the classification identifiers accordingly.
Of course, all those changes need to be documented in the CHANGELOG and we should provide SQL UPDATE statements in NEWS.md like for the changes in version 3.0.0.
- Thomas
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Hi Aaron,
green light for the release from my side.
Thanks for delaying it for the recent discussions.
- Thomas
On 08.02.24 11:11, L. Aaron Kaplan wrote:
Hi all,
thanks for the lively discussion first of all, because it shows that these things matter :)
However, my question currently is: green light for release or not? What do you feel Thomas? Do you feel we need more time to discover things which would create a lot of havoc at different installations ? Or do you feel that these changes are so minimal that it's just part of the regular attentive update cycle? We can put that into the CHANGELOG and NEWS file of course. Would that be enough?
Because I have been delaying the release a bit for the discussion to settle and in order to make sure that there are as few surprises as possible for everyone with the dynamic schema update.
Best, Aaron.
On 08.02.2024, at 10:59, Thomas Hungenberg via IntelMQ-dev intelmq-dev@lists.cert.at wrote:
On 06.02.24 13:42, Kamil Mankowski wrote:
When it comes to identifiers changes, I would be very conservative. They can be used for filtering, and as so - changing them is potentially dangerous. > I second fixes about IPv6, those were more misleading than helping, but for the rest - we need to be careful and announce the change.
Yes, our IntelMQ setup with mailgen etc. also heavily depends on the known classification identifiers. That is why I asked not to change them with the switch to the dynamic schema.
However, Shadowserver renamed some "old" feeds from "open-*" to "accessible-*" some years ago (e.g. "open-telnet" -> "accessible-telnet"). So far, we have not adopted those changes for the classification identifiers but still use "open-telnet" etc. for "old" feeds. On the other hand, for newer feeds like "accessible-ftp" we use the classification identifier "accessible-ftp". So we have "open-telnet" but "accessible-ftp" which is not consistent.
We should probably discuss which services are "open" and which ones are "accessible" and change the classification identifiers accordingly.
Of course, all those changes need to be documented in the CHANGELOG and we should provide SQL UPDATE statements in NEWS.md like for the changes in version 3.0.0.
- Thomas
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Super ! Thanks :)
I think it's important that we do this in a coordinated fashion. Thanks :)
Best, Aaron.
On 08.02.2024, at 11:32, Thomas Hungenberg th@cert-bund.de wrote:
Hi Aaron,
green light for the release from my side.
Thanks for delaying it for the recent discussions.
- ThomasOn 08.02.24 11:11, L. Aaron Kaplan wrote:
Hi all, thanks for the lively discussion first of all, because it shows that these things matter :) However, my question currently is: green light for release or not? What do you feel Thomas? Do you feel we need more time to discover things which would create a lot of havoc at different installations ? Or do you feel that these changes are so minimal that it's just part of the regular attentive update cycle? We can put that into the CHANGELOG and NEWS file of course. Would that be enough? Because I have been delaying the release a bit for the discussion to settle and in order to make sure that there are as few surprises as possible for everyone with the dynamic schema update. Best, Aaron.
On 08.02.2024, at 10:59, Thomas Hungenberg via IntelMQ-dev intelmq-dev@lists.cert.at wrote:
On 06.02.24 13:42, Kamil Mankowski wrote:
When it comes to identifiers changes, I would be very conservative. They can be used for filtering, and as so - changing them is potentially dangerous. > I second fixes about IPv6, those were more misleading than helping, but for the rest - we need to be careful and announce the change.
Yes, our IntelMQ setup with mailgen etc. also heavily depends on the known classification identifiers. That is why I asked not to change them with the switch to the dynamic schema.
However, Shadowserver renamed some "old" feeds from "open-*" to "accessible-*" some years ago (e.g. "open-telnet" -> "accessible-telnet"). So far, we have not adopted those changes for the classification identifiers but still use "open-telnet" etc. for "old" feeds. On the other hand, for newer feeds like "accessible-ftp" we use the classification identifier "accessible-ftp". So we have "open-telnet" but "accessible-ftp" which is not consistent.
We should probably discuss which services are "open" and which ones are "accessible" and change the classification identifiers accordingly.
Of course, all those changes need to be documented in the CHANGELOG and we should provide SQL UPDATE statements in NEWS.md like for the changes in version 3.0.0.
- Thomas
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
For the sake of completeness:
On 1/31/24 16:05, Thomas Hungenberg wrote:
Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed.
I stated this based on https://docs.intelmq.org/latest/user/bots/#shadowserver
*Schema contract*
Once set in the schema, the |classification.identifier|, |classification.taxonomy|, and |classification.type| fields will remain static for a specific report.
The schema revision history is maintained at https://github.com/The-Shadowserver-Foundation/report_schema/.
which was a result of a discussion on the Pull Request (https://github.com/certtools/intelmq/pull/2372)
Thank you Thomas and elsif for analyzing and fixing the schema!
best regards
Hi,
I agree with Thomas that the classification should describe content and not the source (IPv4 vs IPv6).
Based on this I also noticed something else regarding the schema: I believe all the Accessible-SERVICE feeds classified as "vulnerable-system" should be actually classified as "potentially-unwanted-accessible". They are not vulnerable per se, they are just exposing a service to the internet which is usually exposed by mistake.
Best regards,
Filip Pokorný CSIRT.CZ
On 1/31/24 16:05, Thomas Hungenberg via IntelMQ-dev wrote:
Hi,
Sebastian (sebix) told me it was agreed that with the translation from the current parser _config.py (included with IntelMQ 3.2.1) to the new schema, no classification.* attributes will be changed.
This is very important as our setup (and most probably others as well) heavily depends on known classification identifiers like "open-rdp" and classification types from the initial parsing of events up to notification_rules and formats/templates for mailgen. So with a change of a classification attribute lots of scripts and configs would need to be changed as well.
Looking at the current schema, I see the classification identifiers are still correct for some feeds for both IPv4 and IPv6 like here:
"scan_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
"scan6_dns" : { "constant_fields" : { "classification.identifier" : "dns-open-resolver",
However, for other feeds the classification identifier has been kept correctly for IPv4 like here:
"scan_rdp" : { "constant_fields" : { "classification.identifier" : "open-rdp",
"compromised_website" : { "constant_fields" : { "classification.identifier" : "compromised-website",
but for IPv6 it has changed to the name of the feed:
"scan6_rdp" : { "constant_fields" : { "classification.identifier" : "scan6-rdp", <- should be "open-rdp"
"compromised_website6" : { "constant_fields" : { "classification.identifier" : "compromised-website6", <- should be "compromised-website"
The classification.identifier should describe the incident (like "open-rdp") and not the source (like "scan6-rdp").
May I ask you to check and adjust all classification identifiers and types in the schema so they are consistent with the ones generated by the current _config.py?
Thanks a lot for all your work on the new schema based parser!
Kind regards Thomas
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/