-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
since I'm new to Redis i would kindly ask for your assistance to issue which is related to this error:
intelmq.lib.exceptions.PipelineError: pipeline failed - ResponseError("OOM command not allowed when used memory > 'maxmemory'.", )
Currently I have around 15GB memory allocated to intelmq virtual instanc e.
In redis.conf in section "LIMITS" I have set "maxmemory" to 6GB and "maxmemory-policy volatile-lru"
I'm using scheduled run mode for collectors to run during the day at different times with time spaces between them. For example from 9-10h I have set "blocklist.de" collectors to fetch data. After more then hour redis memory limit was reached and the above error shows. I also tried with 10GB memory limit and the same thing happened.
So I would kindly ask for some advise what to change to avoid this situation?
Thanks in advance.
Kind regards,
- -- Tomislav
6GB should be enough in most cases. Can you include information about your OS version, arch? Also no of bots running etc. Also do check if your config changes have been applied to runing redis instance, you can use 'config get' https://redis.io/commands/config-get
On Wed, Dec 13, 2017 at 8:47 PM, Tomislav Protega tomislav.protega@cert.hr wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
since I'm new to Redis i would kindly ask for your assistance to issue which is related to this error:
intelmq.lib.exceptions.PipelineError: pipeline failed - ResponseError("OOM command not allowed when used memory > 'maxmemory'.", )
Currently I have around 15GB memory allocated to intelmq virtual instanc e.
In redis.conf in section "LIMITS" I have set "maxmemory" to 6GB and "maxmemory-policy volatile-lru"
I'm using scheduled run mode for collectors to run during the day at different times with time spaces between them. For example from 9-10h I have set "blocklist.de" collectors to fetch data. After more then hour redis memory limit was reached and the above error shows. I also tried with 10GB memory limit and the same thing happened.
So I would kindly ask for some advise what to change to avoid this situation?
Thanks in advance.
Kind regards,
Tomislav
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2
iEYEARECAAYFAloxRG8ACgkQrREm8+n2Xc+uLwCggTkOK1E06gQGi4IGgjsO/Cc5 rTYAoPYo+PDhKPLzqD8YzFRwEjrLjHER =vUYT
-----END PGP SIGNATURE-----
Listen-Einstellungen: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi, thanks for the response.
Can you include information about your OS version, arch? Also no of bots running etc.
# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
# uname -a Linux IntelMQ 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
At the time of taking down below redis info memory, 6 blocklist.de collectors were running, including their parsers and 8 expert bots.
redis /var/run/redis/redis.sock> info memory # Memory used_memory:3317891992 used_memory_human:3.09G used_memory_rss:3666083840 used_memory_rss_human:3.41G used_memory_peak:3317891992 used_memory_peak_human:3.09G used_memory_peak_perc:100.00% used_memory_overhead:5421164 used_memory_startup:765624 used_memory_dataset:3312470828 used_memory_dataset_perc:99.86% total_system_memory:14713245696 total_system_memory_human:13.70G used_memory_lua:37888 used_memory_lua_human:37.00K maxmemory:6442450944 maxmemory_human:6.00G maxmemory_policy:volatile-lru mem_fragmentation_ratio:1.10 mem_allocator:jemalloc-4.0.3 active_defrag_running:0 lazyfree_pending_objects:0
Also I noticed that every blocklist.de collector throws this error:
2017-12-14 09:08:51,693 - blocklistde-apache-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 144, in start self.process() File "/usr/lib/python3/dist-packages/intelmq/bots/collectors/http/collector_h ttp.py", line 54, in process ''.format(resp.status_code)) ValueError: HTTP response status code was 502. 2017-12-14 09:25:16,978 - blocklistde-apache-collector - WARNING - Ignoring report without raw field. Possible bug or misconfiguration of this bot.
But the data were processed to their parsers.
Kind regards,
- -- Tomislav
On 13.12.2017 17:04, Navtej Singh wrote:
6GB should be enough in most cases. Can you include information about your OS version, arch? Also no of bots running etc. Also do check if your config changes have been applied to runing redis instance, you can use 'config get' https://redis.io/commands/config-get
On Wed, Dec 13, 2017 at 8:47 PM, Tomislav Protega tomislav.protega@cert.hr wrote:
Hi,
since I'm new to Redis i would kindly ask for your assistance to issue which is related to this error:
intelmq.lib.exceptions.PipelineError: pipeline failed - ResponseError("OOM command not allowed when used memory > 'maxmemory'.", )
Currently I have around 15GB memory allocated to intelmq virtual instanc e.
In redis.conf in section "LIMITS" I have set "maxmemory" to 6GB and "maxmemory-policy volatile-lru"
I'm using scheduled run mode for collectors to run during the day at different times with time spaces between them. For example from 9-10h I have set "blocklist.de" collectors to fetch data. After more then hour redis memory limit was reached and the above error shows. I also tried with 10GB memory limit and the same thing happened.
So I would kindly ask for some advise what to change to avoid this situation?
Thanks in advance.
Kind regards,
-- Tomislav
-- Listen-Einstellungen: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
Hi,
With the redis-command "info memory" you can show details about the memory usage, see https://redis.io/commands/INFO
On 12/13/2017 04:17 PM, Tomislav Protega wrote:
intelmq.lib.exceptions.PipelineError: pipeline failed - ResponseError("OOM command not allowed when used memory > 'maxmemory'.", )
We should actually stop the bot if that happens, letting the bot run does not make the situation any better.
For example from 9-10h I have set "blocklist.de" collectors to fetch data.
Are they all stuck in the queue? How many events are there per queue? Are there probably some non-empty queues you no longer use? You can display all queues with:
redis-cli -n 2 keys *
Sebastian
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
We should actually stop the bot if that happens, letting the bot run does not make the situation any better.
Maybe this could be partial problem, because I noticed the bot collector doesn't stop after first run as it should according to description in User-Guide at https://github.com/certtools/intelmq/blob/develop/docs/User-Guide.md
I also noticed that "run_mode": "scheduled", in runtime.conf is misplaced under "parameters": which shouldn't be.
I'll fix this and let you back the results.
Regards,
- -- Tomislav
On 13.12.2017 17:13, Sebastian Wagner wrote:
Hi,
With the redis-command "info memory" you can show details about the memory usage, see https://redis.io/commands/INFO
On 12/13/2017 04:17 PM, Tomislav Protega wrote:
intelmq.lib.exceptions.PipelineError: pipeline failed - ResponseError("OOM command not allowed when used memory > 'maxmemory'.", )
We should actually stop the bot if that happens, letting the bot run does not make the situation any better.
For example from 9-10h I have set "blocklist.de" collectors to fetch data.
Are they all stuck in the queue? How many events are there per queue? Are there probably some non-empty queues you no longer use? You can display all queues with:
redis-cli -n 2 keys *
Sebastian
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
just confirming that for now I fix the issue. As I mentioned yesterday, collector bots weren't stopping after first ru n. I made corrections in runtime.conf and it's fine.
Thanks for tips.
- -- Tomislav
On 14.12.2017 12:34, Tomislav Protega wrote:
Hi,
We should actually stop the bot if that happens, letting the bot run does not make the situation any better.
Maybe this could be partial problem, because I noticed the bot collector doesn't stop after first run as it should according to description in User-Guide at https://github.com/certtools/intelmq/blob/develop/docs/User-Guide.md
I also noticed that "run_mode": "scheduled", in runtime.conf is
misplaced under "parameters": which shouldn't be.
I'll fix this and let you back the results.
Regards,
-- Tomislav
On 13.12.2017 17:13, Sebastian Wagner wrote:
Hi,
With the redis-command "info memory" you can show details about the memory usage, see https://redis.io/commands/INFO
On 12/13/2017 04:17 PM, Tomislav Protega wrote:
intelmq.lib.exceptions.PipelineError: pipeline failed - ResponseError("OOM command not allowed when used memory > 'maxmemory'.", )
We should actually stop the bot if that happens, letting the bot run does not make the situation any better.
For example from 9-10h I have set "blocklist.de" collectors to fetch data.
Are they all stuck in the queue? How many events are there per queue? Are there probably some non-empty queues you no longer use? You can display all queues with:
redis-cli -n 2 keys *
Sebastian
Hi,
On 12/14/2017 12:34 PM, Tomislav Protega wrote:
I also noticed that "run_mode": "scheduled", in runtime.conf is misplaced under "parameters": which shouldn't be.
I opened a PR for the intelmq-manager fixing handling of this parameter. Because the current version 0.3 just deleted that parameter (if saved). I did not check how previous versions deal with it.
Sebastian
https://github.com/certtools/intelmq-manager/pull/151