[Intelmq-dev] Aggregating events within IntelMQ

[L. Aaron Kaplan]

> On 21 Oct 2016, at 15:06, Dustin Demuth <dustin.demuth at intevation.de> wrote:
> 
> 
> Dear IntelMQ-Devs,
> 
> whilst analysing our current setup and possible requirements, we discovered
> that an aggregation of events within IntelMQ might be a reasonable thing to
> do.

I am not sure if an aggregation *within* intelmq makes sense.
The classical way would be to do an aggregation from a datastore/DB after intelmq puts it there.


We risk feature creep if we do that in intelmq!

I am involved with another project [1] where we explicitly deal with large amounts of data.
We intentionally decided against the aggregation within the ETL part (extract transform load) - the equivalent of intelmq. There we process ~ 1 TB of data.

I *highly* recommend to take a serious look at other ETL and aggregation tools and processes and then come back to this discussion.
Intelmq was not made for aggregation. Please let's keep these things separated or at least not in the core part of intelmq.
If aggregation makes sense for you within intelmq, no one is going to stop you. But I don't want to see that feature in the core part. Because it's a different tool.

My 2 cents,
a.




[1] http://cybergreen-stats.herokuapp.com/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20161021/06103230/attachment.sig>