[Intelmq-dev] Reports larger than 500 MB in IntelMQ

[L. Aaron Kaplan]

> On 22 Jul 2016, at 09:02, Dustin Demuth <dustin.demuth at intevation.de> wrote:
> 
> Dear all,
> 
> currently we are facing the problem, that IntelMQ is not capable of handling
> large reports ( > 500 MB) when using Redis as a Message Queuing System.
> First we thought this might get fixed in most recent redis versions (see:
> [1]), but apparently this is not the case.
> 
> 
> 
> Are you aware of setups of IntelMQ which use a different backend, like ZMQ?
> 
> Do you know of performance or other issues which occurred when using other
> Queuing Systems?
> 
> 
> 
We originally had RabbitMQ as a message queueing system.
Redis was quicker. But there is no reason why we could not additionally add zmq as a possible backend (or rabbitMQ). That's why the pipeline.py module provides an abstraction.

Best,
a.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20160722/ea31dff8/attachment.sig>