-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
since I'm new to Redis i would kindly ask for your assistance
to issue which is related to this error:
intelmq.lib.exceptions.PipelineError: pipeline failed -
ResponseError("OOM command not allowed when used memory > 'maxmemory'.",
)
Currently I have around 15GB memory allocated to intelmq virtual instanc
e.
In redis.conf in section "LIMITS" I have set
"maxmemory" to 6GB and
"maxmemory-policy volatile-lru"
I'm using scheduled run mode for collectors to run during the day at
different times with time spaces between them.
For example from 9-10h I have set "blocklist.de" collectors to fetch
data. After more then hour redis memory limit was reached and the
above error shows. I also tried with 10GB memory limit and the same
thing happened.
So I would kindly ask for some advise what to change to avoid this
situation?
Thanks in advance.
Kind regards,
- --
Tomislav
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iEYEARECAAYFAloxRG8ACgkQrREm8+n2Xc+uLwCggTkOK1E06gQGi4IGgjsO/Cc5
rTYAoPYo+PDhKPLzqD8YzFRwEjrLjHER
=vUYT
-----END PGP SIGNATURE-----
Dear community,
I just pushed the version 1.0.2 to pypi and the build servers.
Installation documentation:
https://github.com/certtools/intelmq/blob/1.0.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md
### Core
- `lib.message.add`: parameter force has finally been removed, should
have been gone in 1.0.0.rc1 already
### Bots
- `collectors.mail.collector_mail_url`: Fix bug which prevented marking
emails seen due to disconnects from server (#852).
- `parsers.spamhaus.parser_cert`: Handle/ignore 'AS?' in feed (#1111)
### Packaging
- The following changes have been in effect for the built packages
already since version 1.0.0
- Support building for more distributions, now supported: CentOS 7,
Debian 8 and 9, Fedora 25 and 26, RHEL 7, openSUSE Leap 42.2 and 42.3
and Tumbleweed, Ubuntu 14.04 and 16.04
- Use LSB-paths for created packages (/etc/intelmq/, /var/lib/intelmq/,
/run/intelmq/) (#470). Does does not affect installations with
setuptools/pip.
- Change the debian package format from native to quilt
- Fix problems in postint and postrm scripts
- Use systemd-tmpfile for creation of /run/intelmq/
### Documentation
- Add disclaimer on maxmind database in bot documentation and code and
the cron-job (#1110)
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi,
We need some consistent behavior for extracting files of downloaded
archives. For this, I'd like to hear some opinions from users. What do
you want to be able to configure, what should be done automatically? Do
you want it to be automatic and still have the possibility to override?
There are some possible settings:
* Do extraction at all
* What to extract? Some files vs everything. Can be combined with above
* archive type. Could be guessed from filename extension or mimetype.
The latter is as not trivial in python as I expected :/
Background:
The HTTP collector can currently extract files from zip-files on the
fly. There is no parameter for this, all files will be passed on as
separate reports.
The RT collector can extract zip on the fly if the parameter
`unzip_attachment` is true.
PR#1095[0] adds the ability to extract files for tar.gz archives
including a parameter `extract_files` to give a list of filenames to be
extracted. And all files will be extracted if the parameter is simply True.
Sebastian
[0]: https://github.com/certtools/intelmq/pull/1095
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear users and contributors,
Yesterday I release version 0.3 and today 0.3.1 (containing a fix for a
bug preventing the saving of files).
This release contains a lot of exciting usability fixes and
enhancements. See the changelog below for a full list. We are getting
close to a stable release now!
Please refer to the installation docs. Deb and rpm packages are
available. Note that for the deb-packages, you need to set group
permissions on the configuration files first.
This is the changelog of 0.3:
* Partly support for CentOS/RHEL 7 (#55, #103)
* Note on security considerations in Readme to avoid misunderstandings
* Show versions of intelmq and intelmq manager on about page
* Update vis.js to current version
### Configuration
* interface for defaults.conf (#45)
* drag&drop (#105, #41)
* fix #96
* save buttons starts blinking after changes (#41)
* Allow redrawing of botnet on demand
* Save/load position of bots in/from /opt/intelmq/etc/manager/positions.conf
File needs to be writeable
* parameters from defaults are shown for new bots (#107)
* parameters are grouped by type: generic, runtime, defaults
* better feedback on errors with backend (#69, #99)
* pressing ESC in forms equals to pressing the cancel button
* Edit node window is now much bigger
* pressing enter in 'add key' window equals to pressing ok button
### Management
* Reload and restart have been added as actions on bots and the whole
botnet (#114)
* A click on the bot name opens the monitor page of the bot
### Monitor
* clearing queues is possible in general and specific view for all
queues (#54)
### Backend
* Fix regex checks on bot ids and log line number in controller, they
have not been effective
* fix overflow in extended message box (#49)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
I just released the bug fix release IntelMQ 1.0.1.
The existing bugs which have not been fixed in time have been moved to
the 1.0.2 milestone (10 bugs).
For upgrade instructions look at the documentation:
https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md
Updated packages have been built and are available in the repositories.
From the changelog:
### Documentation
- Feeds: use more https:// URLs
- minor fixes
### Bots
- bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for
rest.db.ripe.net
- bots/outputs/file/output.py: properly close the file handle on shutdown
### Core
- lib/bot: Bots will now log the used intelmq version at startup
### Tools
- intelmqctl: To check the status of a bot, the comandline of the
running process is compared to the actual executable of the bot.
Otherwise unrelated programs with the same PID are detected as running bot.
- intelmqctl: the "enable", "disable", "check", "clear" commands now
support the JSON output
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
forwarding here
> Begin forwarded message:
>
> From: Sebastian Wagner <wagner(a)cert.at>
> Subject: [Intelmq-dev] Published release candidate 1.0.1
> Date: 23 August 2017 at 16:56:23 GMT+2
> To: "intelmq-dev(a)lists.cert.at" <intelmq-dev(a)lists.cert.at>
>
> I just published the release candidate for the next bugfix release
> 1.0.1. You can expect the final release next week / end of august.
>
> Changelog:
>
> ### Documentation
> - Feeds: use more https:// URLs
> - minor fixes
>
> ### Bots
> - bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for
> rest.db.ripe.net
> - bots/outputs/file/output.py: properly close the file handle on shutdown
>
> ### Core
> - lib/bot: Bots will now log the used intelmq version at startup
>
> ### Tools
> - intelmqctl: To check the status of a bot, the comandline of the
> running process is compared to the actual executable of the bot.
> Otherwise unrelated programs with the same PID are detected as running bot.
> - intelmqctl: enable, disable, check, clear now support the JSON output
>
> --
> // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
> // CERT Austria - https://www.cert.at/
> // Eine Initiative der nic.at GmbH - https://www.nic.at/
> // Firmenbuchnummer 172568b, LG Salzburg
>
>
> _______________________________________________
> Intelmq-dev mailing list
> Intelmq-dev(a)lists.cert.at
> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
--
// L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear colleagues and CERT-ies, dear abuse handling teams,
(sorry for big x-posting)
we are happy to announce (finally!) the official 1.0 release of our IntelMQ tool.
What is it?
===========
IntelMQ [1] is a free open source tool initially developed by CERT.pt and CERT.at to automatically handle and process the many incident reports (mostly shadowserver and similar) that we receive.
At CERT.at it processes many thousands of events per day.
But first of all, let me thank all the contributors and the different teams involved in this collaborative open source effort! Starting with CERT.pt (Tomas Lima and Mauro), BSI, Intevation (Bernhard, Dustin), CZ.NIC, CESNET, CERT Australia, CERT.ee, the IHAP [2] group and many many others. You coded, helped, discussed and attended the regular IHAP meetings which allowed us to discuss your wishes and requirements. This all - combined with the testing and coding efforts that many of you contributed - finally gave us version 1.0. today.
We counted at least 45 contributors.
IntelMQ has been running quite stable at CERT.at for nearly a year now and we are processing the bulk of the incident reports with it.
Of course, a 1.0 version always begs for some 1.0.1 bugfixes :) So therefore we would like to ask you to report any bugs or change requests on github's issue tracker [3].
Where can I get it?
===================
Follow the instructions in https://github.com/certtools/intelmq/tree/master/docs
Note that we also have (.deb, .rpm) packages for download. [10]
Future plans
=============
We now tagged the master branch "1.0.0". This will remain stable now.
We also started with a new "develop" branch which will become the 1.1 and 2.0 releases in the future.
You can read more about our branching strategy here [4]
Development will continue towards 1.1 with a set of wishes and requests that we received. You can view them in the issue tracker.
CSP integration
===============
Some of you already know that IntelMQ is a tool included into the "Core Service Platform" (CSP) as part of the CSIRT network [5].
We are very proud to offer our open source solution to the CSP.
Integration into your incident handling automation
==================================================
If you want to integrate IntelMQ into your incident handling automation environment, please note that you might want to use further tools such as "mail-gen" [6] or "intelmqcli" [7] (residing in separate repositories) which connect your ticket system (OTRS or RT) with IntelMQ.
In case you have questions, we have
* an IRC channel on freenode.net (#intelmq)
* a users mailing list [8]
* a developers mailing list [9]
Thanks again everyone who participated in this open source solution!
& feel free to (re-)tweet #intelmq
L. Aaron Kaplan and Sebastian Wagner,
CERT.at
[1] https://github.com/certtools/intelmq/
[2] incident handling automation project.
[3] https://github.com/certtools/intelmq/issues
[4] https://github.com/certtools/intelmq/blob/master/docs/Developers-Guide.md#r…
[5] https://www.enisa.europa.eu/news/enisa-news/2nd-informal-meeting-of-csirt-n…
[6] https://github.com/Intevation/intelmq-mailgen
[7] https://github.com/certat/intelmq
[8] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
[9] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
[10] https://software.opensuse.org//download.html?project=home%3Asebix%3Aintelmq…
--
// L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi,
On 08/01/2017 06:44 PM, Vaclav Bruzek wrote:
> #opt/intelmq/etc# intelmqctl start xxx-parser
> intelmqctl: Running intelmqctl as root is highly discouraged!
> intelmqctl: Starting xxx-parser...
> intelmqctl: lastline-parser is running.
> #/opt/intelmq/etc# intelmqctl stop xxx-parser
> intelmqctl: Running intelmqctl as root is highly discouraged!
> intelmqctl: xxx-parser was NOT RUNNING.
Please do not run intelmqctl as root, you may get a lot of wrong
permissions. Please check them all in /opt/intelmq/ (and all
sub-directories) before continuing.
Then check the logfiles and - if that does not reveal anything -
otherwise start the both with `intelmqctl run bot-id`. See also
https://github.com/certtools/intelmq/blob/develop/docs/FAQ.md#my-bots-died-…
and
https://github.com/certtools/intelmq/blob/develop/docs/intelmqctl.mdDistrib…
ID: elementary
> Description: elementary OS 0.4.1 Loki
something base on 16.04
> intelmq (1.0.0.dev8)
You missed the rc1
> Intelmqctl is located in /usr/local/bin/ next to all intelmq.bots.*
> files (now with my custom bot file definitions)
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg