[Intelmq-dev] Support of shadowserver feeds

[Otmar Lendl]

Dustin, All,

On 16.06.2016 12:27, Dustin Demuth wrote:
> 
> The file contains a bunch of mappings of the feeds below.
> We are not sure if the mappings are correct.
> 
> Can someone verify this and, if possible, remove the appropriate todos, or 
> correct the mapping?

I've been maintaining parsers (in a different context) for Shadowserver
feeds for the last 3 years. Based on that experience a few comments:

* Don't assume that the field-names will stay constant. Be prepared to
support logic like "use 'ip' or 'srcip' for the IntelMQ 'source.ip'".

For the Drone feed, I e.g. have the following mapping rules in our old
system:

# Mapping from local CSV column names to eventDB column names
        $self->{eventdb_map} = {
                asn             => "reported_asn",
                ip              => "src_ip",
                hostname        => "src_hostname",
                port            => "src_port",
                cc              => "dst_ip",
                cc_ip           => "dst_ip",
                cc_port         => "dst_port",
                cc_dns          => "dst_fqdn",
                timestamp       => "ts",
                url             => "dst_url",
                geo             => "reported_iso2cc",
                infection       => "malware",
                machine_name    => "local_hostname",
# older names
                "Timestamp"     => "ts",
                "Drone"         => "src_ip",
                "ASN"           => "reported_asn",
                "Geo"           => "reported_iso2cc",
                "Hostname"      => "src_hostname",
                "C&C"           => "dst_ip",
                "C&C DNS"       => "dst_fqdn",
                "C&C Port"      => "dst_port",
                "Infection"     => "malware",
                };


* I see you support a fixup-function for each attribute. Yes, this is
needed but potentially not good enough. The reason is that you might
need to manipulate multiple fields together, e.g. it varies by feed
whether C&C URLs are transmitted as full URL or split up in
proto/port/hostname/path. If you want to unify these fields, a single
function per attribute will not do.

Here is code from one of my parsers (not shadowserver, this is for
Virustracker) to demonstrate this point.

>         if (exists($row->{reported_asn})) {
>                 $row->{reported_asn}  =~ s/^AS(\d+)\s*.*/$1/;
>         }
> 
>         if (($row->{Type} eq 'HTTP') and $row->{RequestPath} and $row->{dst_fqdn}
>                 and $row->{"dst_port"} and ($row->{"dst_port"} =~ /^80|443$/)) {
>                 $row->{RequestPath} =~ s,^/?,/,;        # make sure request starts with /
>                 $row->{"dst_url"} =
>                                 (($row->{"dst_port"} eq '443') ? 'https://' : 'http://')
>                                 . ($row->{dst_fqdn} ? $row->{dst_fqdn} : "")
>                                 . ($row->{RequestPath} ? $row->{RequestPath} : "")
>                                 ;
>         }
> 
> # for udp p2p botnets, the destinatin IP address is encoded in the Domain parameter
>         if (($row->{Type} =~ /P2P/) and  $row->{dst_fqdn} and ($row->{"dst_fqdn"} =~ /^([\d.]+):/)) {
>                 $row->{"dst_ip"} = $1;
>                 delete($row->{"dst_fqdn"});
>         }
> 
> # move IP-addresses from fqdn to ip field
>         if ($row->{dst_fqdn} and ($row->{"dst_fqdn"} =~ /^([\d.]+)$/)) {
>                 $row->{"dst_ip"} = $1;
>                 delete($row->{"dst_fqdn"});
>         }

HTH,

otmar
-- 
// Otmar Lendl <lendl at cert.at> - T: +43 1 5056416 711
// CERT Austria - http://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20160616/221d65b5/attachment.sig>