[Intelmq-dev] Reports larger than 500 MB in IntelMQ

[Jakub Onderka]

> Another option that we discussed here is to get rid of the base64 encoding.
> This saves quite some space as well!
>
> Redis would be ok with UTF-8 (actually binary stuff).
Currently, it is not possible because data are serialized into JSON, which does not support binary data. So this change requires change serialization format, for example to msgpack, which supports binary data, it is smaller than JSON, probably faster in Python (https://gist.github.com/justinfx/3174062), is supported in Redis scripts and for example in Redis Desktop Manager too.

Jakub


Dne 26.7.2016 v 15:32 L. Aaron Kaplan napsal(a):
>> On 25 Jul 2016, at 15:02, Dustin Demuth <dustin.demuth at intevation.de> wrote:
>>
>>
>> Dear Otmar,
>> thank you very much for your valuable input.
>>
>> Am Freitag 22 Juli 2016 22:47:05 schrieb Otmar Lendl:
>>> A sensible approach is to add some sort of "split" option to the
>>> collector bots.
>> We've already discussed this option here at intevation.
>> The current trend is that we will do something like this.
>>
>> I'm looking forward to see the solution we are creating right now.
>> As of this writing I have not looked into detail.
>> I'll report to the list when I know more.
>>
> Another option that we discussed here is to get rid of the base64 encoding.
> This saves quite some space as well!
>
> Redis would be ok with UTF-8 (actually binary stuff).
>
>
>
>
> _______________________________________________
> Intelmq-dev mailing list
> Intelmq-dev at lists.cert.at
> http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev

-- 
Jakub Onderka
Národní bezpečnostní úřad
Národní centrum kybernetické bezpečnosti
www.govcert.cz


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20160726/22f3ce75/attachment.sig>