Dear community
I have just release a new bugfix release of IntelMQ.
Installation instructions:
https://github.com/certtools/intelmq/blob/1.0.3/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.0.3/docs/UPGRADING.md
The released is published on PyPI, Github and the OpenBuildService (for
rpm/deb packages). If you installed intelmq with a package manager, the
new released will be installed automatically.
Full changelog:
### Contrib
* logrotate: use sudo for postrotate script
* cron-jobs: use the scripts in the bots' directories and link them
(#1056, #1142)
### Core
- `lib.harmonization`: Handle idna encoding error in FQDN sanitation
(#1175, #1176).
- `lib.bot`:
- Bots stop when redis gives the error "OOM command not allowed when
used memory > 'maxmemory'." (#1138).
- warnings of bots are catched by the logger (#1074, #1113).
- Fixed exitcodes 0 for graceful shutdowns .
- better handling of problems with pipeline and especially it's
initialization (#1178).
- All parsers using `ParserBot`'s methods now log the sum of
successfully parsed and failed lines at the end of each run (#1161).
### Harmonization
- Rule for harmonization keys is enforced (#1104, #1141).
- New allowed values for `classification.type`: `tor` & `leak` (see n6
parser below ).
### Bots
#### Collectors
- `bots.collectors.mail.collector_mail_attach`: Support attachment file
parsing for imbox versions newer than 0.9.5 (#1134).
- `bots.outputs.smtp.output`: Fix STARTTLS, threw an exception (#1152,
#1153).
#### Parsers
- All CSV parsers ignore NULL-bytes now, because the csv-library cannot
handle it (#967, #1114).
- `bots.experts.modify` default ruleset: changed conficker rule to catch
more spellings.
- `bots.parsers.shadowserver.parser`: Add Accessible Cisco Smart Install
(#1122).
- `bots.parsers.cleanmx.parser`: Handle new columns `first` and `last`,
rewritten for XML feed. See NEWS.md for upgrade instructions (#1131,
#1136, #1163).
- `bots.parsers.n6.parser`: Fix classification mappings. See NEWS file
for changes values (#738, #1127).
### Documentation
- `Release.md` add release procedure documentation
- `Bots.md`: fix example configuration for modify expert
### Tools
- intelmqctl now exits with exit codes > 0 when errors happened or the
operation was not successful. Also, the status operation exits with 1,
if bots are stopped, but enabled. (#977, #1143)
- `intelmctl check` checks for valid `run_mode` in runtime configuration
(#1140).
### Tests
- `tests.lib.test_pipeline`: Redis tests clear all queues before and
after tests (#1086).
- Repaired debian package build on travis (#1169).
- Warnings are not allowed by default, an allowed count can be specified
(#1129).
- `tests.bots.experts.cymru_whois/abusix`: Skipped on travis because of
ongoing problems.
### Packaging
* cron jobs: fix paths of executables
### Known issues
- `bots.collectors/outputs.xmpp` must be killed two times (#970).
- When running bots with `intelmqctl run [bot-id]` the log level is
always INFO (#1075).
- `intelmqctl run [bot-id] message send [msg]` does only support Events,
not Reports (#1077).
- `python3 setup.py sdist` does not include static files in the
resulting tarballs (#1146).
- `bots.parsers.cleanmx.parser`: The cleanMX feed may have FQDNs as IPs
in rare cases, such lines are dumped (#1162).
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hello everybody,
I have not much experience with intelmq nor python packages managments so while while implementing a couple of new bots, I have experienced the following issues:
1- According to the documentation the best way to proceed is to clone the repo and install intelmq via “pip3 install -e .”. The issue here is that it needs to be done from root, otherwise the install fails. However once it is installed the user intelmq cannot see the intelmq installation (pip3 list) since it is only installed for root.
2- If I Install from pip3 directly (not using a local repo) intelmq works just fine but then, new bot creation fails since according to the documentation new bots needs to be generated reinstalling intelmq. Trying to manually generate the binaries for the bots is not recommended according to previous emails (in the archive).
3- Using virtual env allowed me to install intelmq as recommended for developers (“pip3 install -e .”.) and new bots are generated appropriately . However, some issues appear with the pipeline connection and nowhere in the documentation it is suggested.
My question then is, what is the recommended and verified way of implementing new bots for intelmq? Starting from the installation of intelmq to testing the new bots.
My environment: VM with Ubuntu 16.04
Thanks in advance for your time!
Best regards,
Leandro Velasco
---
Rijnland 4c
1948 RL Beverwijk
Leandro.Velasco(a)dearbytes.nl<mailto:Leandro.Velasco@dearbytes.nl>
www.dearbytes.nl<http://www.dearbytes.nl>
Hi,
recently I came up into elasticsearch parsing exception.
Dump is attached below.
It only happens when it processes data from Blueliv Crimeserver and
Shadowserver-Open-XDMCP collectors.
Not so far ago my elasticsearch output bot didn't throw that exception.
Currently I'm using intelmq 1.0.2 and intelmq-manager 0.3.1, all
installed from .deb package and python client elasticsearch 6.0.0.
Anyone experienced the same?
Thanks for the efforts.
Regards,
--
Tomislav
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
since I'm new to Redis i would kindly ask for your assistance
to issue which is related to this error:
intelmq.lib.exceptions.PipelineError: pipeline failed -
ResponseError("OOM command not allowed when used memory > 'maxmemory'.",
)
Currently I have around 15GB memory allocated to intelmq virtual instanc
e.
In redis.conf in section "LIMITS" I have set
"maxmemory" to 6GB and
"maxmemory-policy volatile-lru"
I'm using scheduled run mode for collectors to run during the day at
different times with time spaces between them.
For example from 9-10h I have set "blocklist.de" collectors to fetch
data. After more then hour redis memory limit was reached and the
above error shows. I also tried with 10GB memory limit and the same
thing happened.
So I would kindly ask for some advise what to change to avoid this
situation?
Thanks in advance.
Kind regards,
- --
Tomislav
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iEYEARECAAYFAloxRG8ACgkQrREm8+n2Xc+uLwCggTkOK1E06gQGi4IGgjsO/Cc5
rTYAoPYo+PDhKPLzqD8YzFRwEjrLjHER
=vUYT
-----END PGP SIGNATURE-----
Dear community,
I just pushed the version 1.0.2 to pypi and the build servers.
Installation documentation:
https://github.com/certtools/intelmq/blob/1.0.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md
### Core
- `lib.message.add`: parameter force has finally been removed, should
have been gone in 1.0.0.rc1 already
### Bots
- `collectors.mail.collector_mail_url`: Fix bug which prevented marking
emails seen due to disconnects from server (#852).
- `parsers.spamhaus.parser_cert`: Handle/ignore 'AS?' in feed (#1111)
### Packaging
- The following changes have been in effect for the built packages
already since version 1.0.0
- Support building for more distributions, now supported: CentOS 7,
Debian 8 and 9, Fedora 25 and 26, RHEL 7, openSUSE Leap 42.2 and 42.3
and Tumbleweed, Ubuntu 14.04 and 16.04
- Use LSB-paths for created packages (/etc/intelmq/, /var/lib/intelmq/,
/run/intelmq/) (#470). Does does not affect installations with
setuptools/pip.
- Change the debian package format from native to quilt
- Fix problems in postint and postrm scripts
- Use systemd-tmpfile for creation of /run/intelmq/
### Documentation
- Add disclaimer on maxmind database in bot documentation and code and
the cron-job (#1110)
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi,
We need some consistent behavior for extracting files of downloaded
archives. For this, I'd like to hear some opinions from users. What do
you want to be able to configure, what should be done automatically? Do
you want it to be automatic and still have the possibility to override?
There are some possible settings:
* Do extraction at all
* What to extract? Some files vs everything. Can be combined with above
* archive type. Could be guessed from filename extension or mimetype.
The latter is as not trivial in python as I expected :/
Background:
The HTTP collector can currently extract files from zip-files on the
fly. There is no parameter for this, all files will be passed on as
separate reports.
The RT collector can extract zip on the fly if the parameter
`unzip_attachment` is true.
PR#1095[0] adds the ability to extract files for tar.gz archives
including a parameter `extract_files` to give a list of filenames to be
extracted. And all files will be extracted if the parameter is simply True.
Sebastian
[0]: https://github.com/certtools/intelmq/pull/1095
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear users and contributors,
Yesterday I release version 0.3 and today 0.3.1 (containing a fix for a
bug preventing the saving of files).
This release contains a lot of exciting usability fixes and
enhancements. See the changelog below for a full list. We are getting
close to a stable release now!
Please refer to the installation docs. Deb and rpm packages are
available. Note that for the deb-packages, you need to set group
permissions on the configuration files first.
This is the changelog of 0.3:
* Partly support for CentOS/RHEL 7 (#55, #103)
* Note on security considerations in Readme to avoid misunderstandings
* Show versions of intelmq and intelmq manager on about page
* Update vis.js to current version
### Configuration
* interface for defaults.conf (#45)
* drag&drop (#105, #41)
* fix #96
* save buttons starts blinking after changes (#41)
* Allow redrawing of botnet on demand
* Save/load position of bots in/from /opt/intelmq/etc/manager/positions.conf
File needs to be writeable
* parameters from defaults are shown for new bots (#107)
* parameters are grouped by type: generic, runtime, defaults
* better feedback on errors with backend (#69, #99)
* pressing ESC in forms equals to pressing the cancel button
* Edit node window is now much bigger
* pressing enter in 'add key' window equals to pressing ok button
### Management
* Reload and restart have been added as actions on bots and the whole
botnet (#114)
* A click on the bot name opens the monitor page of the bot
### Monitor
* clearing queues is possible in general and specific view for all
queues (#54)
### Backend
* Fix regex checks on bot ids and log line number in controller, they
have not been effective
* fix overflow in extended message box (#49)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
I just released the bug fix release IntelMQ 1.0.1.
The existing bugs which have not been fixed in time have been moved to
the 1.0.2 milestone (10 bugs).
For upgrade instructions look at the documentation:
https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md
Updated packages have been built and are available in the repositories.
From the changelog:
### Documentation
- Feeds: use more https:// URLs
- minor fixes
### Bots
- bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for
rest.db.ripe.net
- bots/outputs/file/output.py: properly close the file handle on shutdown
### Core
- lib/bot: Bots will now log the used intelmq version at startup
### Tools
- intelmqctl: To check the status of a bot, the comandline of the
running process is compared to the actual executable of the bot.
Otherwise unrelated programs with the same PID are detected as running bot.
- intelmqctl: the "enable", "disable", "check", "clear" commands now
support the JSON output
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
forwarding here
> Begin forwarded message:
>
> From: Sebastian Wagner <wagner(a)cert.at>
> Subject: [Intelmq-dev] Published release candidate 1.0.1
> Date: 23 August 2017 at 16:56:23 GMT+2
> To: "intelmq-dev(a)lists.cert.at" <intelmq-dev(a)lists.cert.at>
>
> I just published the release candidate for the next bugfix release
> 1.0.1. You can expect the final release next week / end of august.
>
> Changelog:
>
> ### Documentation
> - Feeds: use more https:// URLs
> - minor fixes
>
> ### Bots
> - bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for
> rest.db.ripe.net
> - bots/outputs/file/output.py: properly close the file handle on shutdown
>
> ### Core
> - lib/bot: Bots will now log the used intelmq version at startup
>
> ### Tools
> - intelmqctl: To check the status of a bot, the comandline of the
> running process is compared to the actual executable of the bot.
> Otherwise unrelated programs with the same PID are detected as running bot.
> - intelmqctl: enable, disable, check, clear now support the JSON output
>
> --
> // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
> // CERT Austria - https://www.cert.at/
> // Eine Initiative der nic.at GmbH - https://www.nic.at/
> // Firmenbuchnummer 172568b, LG Salzburg
>
>
> _______________________________________________
> Intelmq-dev mailing list
> Intelmq-dev(a)lists.cert.at
> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
--
// L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg