Another option that we discussed here is to get rid of the base64 encoding. This saves quite some space as well!
Redis would be ok with UTF-8 (actually binary stuff).
Currently, it is not possible because data are serialized into JSON, which does not support binary data. So this change requires change serialization format, for example to msgpack, which supports binary data, it is smaller than JSON, probably faster in Python (https://gist.github.com/justinfx/3174062), is supported in Redis scripts and for example in Redis Desktop Manager too.
Jakub
Dne 26.7.2016 v 15:32 L. Aaron Kaplan napsal(a):
On 25 Jul 2016, at 15:02, Dustin Demuth dustin.demuth@intevation.de wrote:
Dear Otmar, thank you very much for your valuable input.
Am Freitag 22 Juli 2016 22:47:05 schrieb Otmar Lendl:
A sensible approach is to add some sort of "split" option to the collector bots.
We've already discussed this option here at intevation. The current trend is that we will do something like this.
I'm looking forward to see the solution we are creating right now. As of this writing I have not looked into detail. I'll report to the list when I know more.
Another option that we discussed here is to get rid of the base64 encoding. This saves quite some space as well!
Redis would be ok with UTF-8 (actually binary stuff).
Intelmq-dev mailing list Intelmq-dev@lists.cert.at http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev