Dear All,
seems we have a solution for this problem now.
Bernhard has created a solution to split large csv-reports into chunks [1].
To do so, the collectors (in this case the "Mail-URL-Collector" which is the only one affected for our use case) is extended with `generate_reports()` from `intelmq.lib.splitreports`.
The collector can be extended with two parameters. Those are `chunk_size`, determining the size of each chunk (I don't know the unit yet, seems to be bytes), and `chunk_replicate_header` which replicates the first line of the file.
From my short look at the code, I see that splitreports cannot process lines which are comments (you might have seen those starting with a # sign).
Should this be integrated?
Am Montag 25 Juli 2016 15:02:55 schrieb Dustin Demuth:
I'm looking forward to see the solution we are creating right now. As of this writing I have not looked into detail. I'll report to the list when I know more.
[1] https://github.com/Intevation/intelmq/tree/dev-split-csv-reports