[Intelmq-users] intelmq-redis experience (OOM error)

[Tomislav Protega]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,
thanks for the response.

> Can you include information about your OS version, arch? Also no of
> bots running etc.

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"

# uname -a
Linux IntelMQ 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC
2017 x86_64 x86_64 x86_64 GNU/Linux

At the time of taking down below redis info memory, 6 blocklist.de
collectors were running, including their parsers and 8 expert bots.

redis /var/run/redis/redis.sock> info memory
# Memory
used_memory:3317891992
used_memory_human:3.09G
used_memory_rss:3666083840
used_memory_rss_human:3.41G
used_memory_peak:3317891992
used_memory_peak_human:3.09G
used_memory_peak_perc:100.00%
used_memory_overhead:5421164
used_memory_startup:765624
used_memory_dataset:3312470828
used_memory_dataset_perc:99.86%
total_system_memory:14713245696
total_system_memory_human:13.70G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:6442450944
maxmemory_human:6.00G
maxmemory_policy:volatile-lru
mem_fragmentation_ratio:1.10
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0


Also I noticed that every blocklist.de collector throws this error:

2017-12-14 09:08:51,693 - blocklistde-apache-collector - ERROR - Bot
has found a problem.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 144,
in start
    self.process()
  File
"/usr/lib/python3/dist-packages/intelmq/bots/collectors/http/collector_h
ttp.py",
line 54, in process
    ''.format(resp.status_code))
ValueError: HTTP response status code was 502.
2017-12-14 09:25:16,978 - blocklistde-apache-collector - WARNING -
Ignoring report without raw field. Possible bug or misconfiguration of
this bot.

But the data were processed to their parsers.


Kind regards,

- --
Tomislav



On 13.12.2017 17:04, Navtej Singh wrote:
> 6GB should be enough in most cases. Can you include information
> about your OS version, arch? Also no of bots running etc. Also do
> check if your config changes have been applied to runing redis 
> instance, you can use 'config get' 
> https://redis.io/commands/config-get
> 
> On Wed, Dec 13, 2017 at 8:47 PM, Tomislav Protega
> <tomislav.protega at cert.hr> wrote:
> 
> Hi,
> 
> since I'm new to Redis i would kindly ask for your assistance to
> issue which is related to this error:
> 
> intelmq.lib.exceptions.PipelineError: pipeline failed - 
> ResponseError("OOM command not allowed when used memory >
> 'maxmemory'.", )
> 
> Currently I have around 15GB memory allocated to intelmq virtual
> instanc e.
> 
> In redis.conf in section "LIMITS" I have set "maxmemory" to 6GB
> and "maxmemory-policy volatile-lru"
> 
> I'm using scheduled run mode for collectors to run during the day
> at different times with time spaces between them. For example from
> 9-10h I have set "blocklist.de" collectors to fetch data. After
> more then hour redis memory limit was reached and the above error
> shows. I also tried with 10GB memory limit and the same thing
> happened.
> 
> So I would kindly ask for some advise what to change to avoid this 
> situation?
> 
> 
> Thanks in advance.
> 
> 
> Kind regards,
> 
> -- Tomislav
> 
> 
>> -- Listen-Einstellungen: 
>> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
>> 
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAloyR+gACgkQrREm8+n2Xc/2qwCfYRigJ9LlrKxQAeJ302em9Aop
YSoAmgKE/ZYQyDICP9bFYOZQZvxcqcdB
=B2b4
-----END PGP SIGNATURE-----