[Intelmq-dev] Reports larger than 500 MB in IntelMQ

[L. Aaron Kaplan]

> On 26 Jul 2016, at 15:51, Jakub Onderka <j.onderka at nbu.cz> wrote:
> 
>> Another option that we discussed here is to get rid of the base64 encoding.
>> This saves quite some space as well!
>> 
>> Redis would be ok with UTF-8 (actually binary stuff).
> Currently, it is not possible because data are serialized into JSON, which does not support binary data.
Sure, that's a intelmq specific thing. Redis per se would accept most anything.

> So this change requires change serialization format, for example to msgpack, which supports binary data, it is smaller than JSON, probably faster in Python (https://gist.github.com/justinfx/3174062), is supported in Redis scripts and for example in Redis Desktop Manager too.

Please note: I was not saying we should be changing the intelmq (JSON) format.
Just saying that Sebix and me discussed that theoretically we could get rid of the extra base64 encoding.
This would save extra space (when the 512 MB redis key / value limit is the issue).

Best,
a.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20160726/21e463ff/attachment.sig>