On 12 Apr 2017, at 12:54, Sebastian Wagner wagner@cert.at wrote:
Hi,
we have this IP vs FQDN problem in some parsers, not only the shadowserver. Stripping the port there can be simply achieved by use a conversion function. But the main problem is IP/FQDN.
Instead of implementing the logic in many parsers we could add this "intelligence" in the libs.
I am not sure if I like that approach. Usually the particularities of the "messiness" are best placed in the parser. Even if the logic repeats itself a bit amongst different parsers. We could of course have a function in lib/ to clear this up, but then each parser which thinks it needs that cleanup part must call the cleanup function in lib/. But: other parsers MUST NOT call that cleanup function.
Because the http host dest fields might contain totally different (crap) in other feeds. So... I would *not* try to impose a default behaviour for all parsers here.
I believe the shadowserver parser should be extended in a way as Thomas suggested.
One possibility: If the parser tries to add an FQDN as IP, save the IP. But I don't like this simple approach as this implicitness raises other problems.
yup
Other possibility: Use a new "logic" (actually non-existing) field, e.g. `destination.host-info`,
how about calling it destination.http_host ?
same applies to source. If some data is added to this field, the data will be parsed and added to ip, fqdn, port (,network?)
Example 1: event['destination.host-info'] = 'example.com:8080' results in: {'destination.fqdn': 'example.com', 'destination.port': 8080} Example2: event['destination.host-info'] = '10.0.0.1' results in: {'source.ip': '10.0.0.1'}
but again, if you have destination.http_host there, then again it would make sense to parse it and put the info into destination.ip, destination.port etc, ...
Sebastian
-- // Sebastian Wagner wagner@cert.at - T: +43 1 50564167201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
Intelmq-dev mailing list Intelmq-dev@lists.cert.at http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
-- // L. Aaron Kaplan kaplan@cert.at - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg