[Intelmq-dev] Aggregating events within IntelMQ

[Bernhard Reiter]

Am Freitag 21 Oktober 2016 18:31:10 schrieb L. Aaron Kaplan:
> > an aggregation of events within IntelMQ might be a
> > reasonable thing to do.
>
> I am not sure if an aggregation *within* intelmq makes sense.
> The classical way would be to do an aggregation from a datastore/DB after
> intelmq puts it there.

The aggregation for email notifications in abuse handling is special. 
We are seeing this while building the solution for CERT-Bund.

It is not the need of data collection for analysis, but just sending out one 
email. So the time-frame is short.

> I *highly* recommend to take a serious look at other ETL and aggregation
> tools and processes and then come back to this discussion. Intelmq was not
> made for aggregation.

In a data flow sense, the deduplicator already "aggregates".
Some abuse handling decisions will depend on seeing several sources
report something in the future, for this they will need to wait a bit, maybe
just a few minutes like the deduplicator.

The main question is: How many typical intelmq setups will want to have 
functionality that sends out an email? If many are, than email should be part 
of the core intelmq experience. And email means aggregated at least
for a few minutes, otherwise it is too much overhead.

Best Regards,
Bernhard

-- 
www.intevation.de/~bernhard   +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20161024/ab624a39/attachment.sig>