-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi, thanks for the response.
Can you include information about your OS version, arch? Also no of bots running etc.
# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
# uname -a Linux IntelMQ 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
At the time of taking down below redis info memory, 6 blocklist.de collectors were running, including their parsers and 8 expert bots.
redis /var/run/redis/redis.sock> info memory # Memory used_memory:3317891992 used_memory_human:3.09G used_memory_rss:3666083840 used_memory_rss_human:3.41G used_memory_peak:3317891992 used_memory_peak_human:3.09G used_memory_peak_perc:100.00% used_memory_overhead:5421164 used_memory_startup:765624 used_memory_dataset:3312470828 used_memory_dataset_perc:99.86% total_system_memory:14713245696 total_system_memory_human:13.70G used_memory_lua:37888 used_memory_lua_human:37.00K maxmemory:6442450944 maxmemory_human:6.00G maxmemory_policy:volatile-lru mem_fragmentation_ratio:1.10 mem_allocator:jemalloc-4.0.3 active_defrag_running:0 lazyfree_pending_objects:0
Also I noticed that every blocklist.de collector throws this error:
2017-12-14 09:08:51,693 - blocklistde-apache-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 144, in start self.process() File "/usr/lib/python3/dist-packages/intelmq/bots/collectors/http/collector_h ttp.py", line 54, in process ''.format(resp.status_code)) ValueError: HTTP response status code was 502. 2017-12-14 09:25:16,978 - blocklistde-apache-collector - WARNING - Ignoring report without raw field. Possible bug or misconfiguration of this bot.
But the data were processed to their parsers.
Kind regards,
- -- Tomislav
On 13.12.2017 17:04, Navtej Singh wrote:
6GB should be enough in most cases. Can you include information about your OS version, arch? Also no of bots running etc. Also do check if your config changes have been applied to runing redis instance, you can use 'config get' https://redis.io/commands/config-get
On Wed, Dec 13, 2017 at 8:47 PM, Tomislav Protega tomislav.protega@cert.hr wrote:
Hi,
since I'm new to Redis i would kindly ask for your assistance to issue which is related to this error:
intelmq.lib.exceptions.PipelineError: pipeline failed - ResponseError("OOM command not allowed when used memory > 'maxmemory'.", )
Currently I have around 15GB memory allocated to intelmq virtual instanc e.
In redis.conf in section "LIMITS" I have set "maxmemory" to 6GB and "maxmemory-policy volatile-lru"
I'm using scheduled run mode for collectors to run during the day at different times with time spaces between them. For example from 9-10h I have set "blocklist.de" collectors to fetch data. After more then hour redis memory limit was reached and the above error shows. I also tried with 10GB memory limit and the same thing happened.
So I would kindly ask for some advise what to change to avoid this situation?
Thanks in advance.
Kind regards,
-- Tomislav
-- Listen-Einstellungen: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users