[Intelmq-dev] Output format to syslog/splunk (PR#503)

[Sebastian Wagner]

Hallo all,

Maybe someone on the IHAP-list has some experience with splunk in
special or syslog and CEF in general and could help us here?

We want to send intelmq-events to a remote syslog instance for further
processing in CEF or json-format. How could we transform the data to CEF
and maintain parseability. See my email below and the PR and the
discussion there for more details.
This is the PR in question: https://github.com/certtools/intelmq/pull/503

According to this specification[0] I found,
 - | should be escaped to \|
 - \ should be escaped to \\
 - = should be escaped to \= (we don't use that though)
 - newlines should be \n (is \\n meant?)

But we do have arbitrary strings (including \0 and others). It seems CEF
does not allow these values.

Sebastian

[0]:
https://kc.mcafee.com/resources/sites/MCAFEE/content/live/CORP_KNOWLEDGEBASE/78000/KB78712/en_US/CEF_White_Paper_20100722.pdf

On 05/10/2016 11:01 AM, Sebastian Wagner wrote:
> Dear developers, contributors, users, etc.
>
> Pedro Reis (@pedromreis) opened a pull request for an UDP output bot,
> which can be used to send events to a syslog daemon (and then picked
> up by further processing software).
> The implementation has the following features:
>
>   * Output formats are JSON or delimited by a configurable character
>   * a optional header (at beginning of the line) can be set
>   * `raw` field can be dropped
>
> I can see some potential problems with the 'delimited'-method here:
>
>   * Strings can contain the delimiter itself, which breaks parsing.
>   * Strings can contain arbitrary characters like \0 or \n which
>     breaks everything
>
> Possible solutions could be:
>
>   * ignore the problem as it's maybe not relevant
>   * escape all problematic characters (solves problem with \n)
>   * quote strings (solves problem with delimiters in strings)
>   * strip non-printable characters
>   * drop fields with non-printable characters
>   * encode strings in base64
>
>
> As you may have possible applications for this bot or you have
> experience with events in syslog, I would appreciate some feedback
> from you.
>
> Sebastian
>
> -- 
> // Sebastian Wagner <wagner at cert.at> - T: +43 1 50564167201 
> // CERT Austria - http://www.cert.at/
> // Eine Initiative der nic.at GmbH - http://www.nic.at/
> // Firmenbuchnummer 172568b, LG Salzburg
>
>
> _______________________________________________
> Intelmq-dev mailing list
> Intelmq-dev at lists.cert.at
> http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 50564167201 
// CERT Austria - http://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20160524/78a67635/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20160524/78a67635/attachment.sig>