[Intelmq-dev] Reports larger than 500 MB in IntelMQ

[Dustin Demuth]

Dear All,

seems we have a solution for this problem now.

Bernhard has created a solution to split large csv-reports into chunks [1].

To do so, the collectors (in this case the "Mail-URL-Collector" which is the 
only one affected for our use case) is extended with  `generate_reports()` 
from `intelmq.lib.splitreports`.

The collector can be extended with two parameters. Those are `chunk_size`, 
determining the size of each chunk (I don't know the unit yet, seems to be 
bytes), and `chunk_replicate_header` which replicates the first line of the 
file.


From my short look at the code, I see that splitreports cannot process lines 
which are comments (you might have seen those starting with a # sign).

Should this be integrated?



Am Montag 25 Juli 2016 15:02:55 schrieb Dustin Demuth:

> I'm looking forward to see the solution we are creating right now.
> As of this writing I have not looked into detail.
> I'll report to the list when I know more.
>


[1] https://github.com/Intevation/intelmq/tree/dev-split-csv-reports


-- 
dustin.demuth at intevation.de  https://intevation.de/   OpenPGP key: B40D2EFF
Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998
Geschäftsführer:   Frank Koormann,  Bernhard Reiter,  Dr. Jan-Oliver Wagner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20160729/c8ab6ecc/attachment.sig>