[Intelmq-dev] Data Harmonization - Fields with multiple values

[Sebastian Wagner]

Hi,

On 11/03/2017 06:26 AM, Knight, Alexander wrote:
>
> At the Deepsec conference Sebastian mentioned updating the
> harmonization to allow for fields with multiple values. Has this issue
> been progressed at all?
>
The use case was the field abuse_contact which could be a list and then
be concatenated (if necessary) with commas.
Technically it is not hard to do it. In the develop branch I already
have something similar (and more complex): a dictionary type named JSONDict.
So, not directly, but some changes that should make a change easier.

There are some questions popping up that need to be clarified first:
* How to define the types of the values inside the list? E.g. for the
abuse_contact it has to be a list of strings/email addresses
* How should the "API" look like, or in other words: what should happen
for the in and setitem-operations etc
* When should the list be converted to a string (or maybe also a
JSON-list)? E.g. for postgres output the abuse_contact could either be a
json-list or a comma separated list, depending on the table's
definition, but for NoSQL-databases and files it can be just the list
itself.

And: what use cases do we have? That's good to know before thinking
about how we implement that all:
>
> We will require multiple values for some fields in our events,
>
What is in these fields? (type and/or example values) Where do you put
that that and how do you want to work with in (inside intelmq)?

I'd like to hear opinions of other users and developers too!

Sebastian
P.S.: I do have specific ideas, but don't want to bias others ;)

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20171108/a296ecb1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20171108/a296ecb1/attachment.sig>