Hi all,
Some of the recent changes of the build targets for deb/rpm packages:
New targets:
* Ubuntu 18.04
* Fedora 27
* openSUSE Leap 15.0
Removed targets:
* Ubuntu 17.04/17.10
* Fedora 25
* openSUSE 42.2
So this is the full list of supported OSs for native packages currently:
* CentOS 7
* RHEL 7
* Debian 8 and 9
* Fedora 26, 27, 28 and Rawhide
* openSUSE Leap 42.3, 15.0 and Tumbleweed
* Ubuntu 16.04, 17.10 and 18.04
The unstable repository additionally has:
* Debian Testing
* Fedora Rawhide
Docs are here:
https://github.com/certtools/intelmq/blob/maintenance/docs/INSTALL.md#nativ…
Please let me know your feedback any time!
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
I just released a new stable release 1.0.0 and maintenance release 1.0.1
of intelmq-manager. Thanks to Edvard Reijthar, the project is very active
again and many changes can be expected for the next versions too!
Version 1.0.0 fixes some bugs and adds an interface for `intelmqctl check`.
1.0.1 fixes the version number in the code (shown in the about page).
Installation instructions:
https://github.com/certtools/intelmq-manager/blob/1.0.1/docs/INSTALL.md
The version is compatible with intelmq >= 1.0.3
Combined changelog:
### Backend
* Set content type correctly for JSON data in configuration loading (#112)
* Fix version number.
### Pages
* All pages are now deliverd by php, reducing the amount of duplicate code drastically.
#### Landing page
* Added a new block for the new check page, changed the formatting a bit.
#### Configuration
* Fixed handling of special parameter `run_mode` (#150)
* Intelmqctl controller may be set via an env variable `INTELMQ_MANGER_CONTROLER_CMD`
#### Check
* Added, showing the output of `intelmqctl check` (#118).
### Documentation
* Note on header Content-Security-Policy (#113)
* Note on security considerations in Readme to avoid misunderstandings
* Remove compatibility warning from README
### Third-party libraries
* reverted update jQuery to 3.2.1
* reverted update metisMenu to 2.7.0
### Licenses
* Licenses of used and included software is now inventarized and properly declared in LICENSES/ (#134)
### Packaging
* fix packaging of positions.conf file for deb-packages (#133).
### Known issues
* Missing CSRF protection (#111).
* Missing copyright notices (#140).
* Graph jumps around on "Add edge" bug component (#148).
* new runtime parameters with _ not possible (#153).
* wrong error message for new bots with existing ID (#152).
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
I just released a new maintenance release 1.0.4 of intelmq. It only
fixes bugs in the 1.0.x series and may be the last version of the 1.0.x
series.
Installation instructions:
https://github.com/certtools/intelmq/blob/1.0.4/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.0.4/docs/UPGRADING.md
The changes are:
- make code style compatible to pycodestyle 2.4.0
- fixed permissions of some files (they were executable but shouldn't be)
### Core
- lib/harmonization:
* FQDN validation now handles None correctly (raised an Exception).
* Fixed several sanitize() methods, the generic sanitation method were
called by is_valid, not the sanitize methods (#1219).
### Harmonization
### Bots
* Use the new pypi website at https://pypi.org/ everywhere.
#### Parsers
- Shadowserver parser:
* The fields `url` and `http_url` now handle HTTP URL paths and HTTP
requests for all feeds (#1204).
* The conversion function `validate_fqdn` now handles empty strings
correctly.
* Feed 'drone (hadoop)':
* Correct validation of field `cc_dns`, will now only be added as
`destination.fqdn` if correct FQDN, otherwise ignored. Previously this
field could be saved in extra containing an IP address.
* Adding more mappings for added columns.
* A lot of newly added fields and fixed conversions.
* Add newly added columns of `Ssl-Scan` feed to parser
- Spamhaus CERT parser:
* fix parsing and classification for bot names 'openrelay', 'iotrdp',
'sshauth', 'telnetauth', 'iotcmd', 'iotuser', 'wpscanner', 'w_wplogin',
'iotscan'
see the NEWS file - Postgresql section - for all changes.
- CleanMX phishing parser: handle FQDNs in IP column (#1162).
#### Experts
- `bots.experts.ripencc_abuse_contact`: Add existing parameter `mode` to
BOTS file.
### Tools
- intelmqctl check: Fixed and extended message for 'run_mode' check.
- `intelmqctl start` botnet. When using `--type json`, no non-json
information about wrong bots are output because that would confuse eg.
intelmq-manager
### Tests
- lib/bot: No dumps will be written during tests (#934).
- lib/test: Expand regular expression on python version to match
pre-releases (debian testing).
### Packaging
* Static data is now included in source tarballs, development files are
excluded
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community
I have just release a new bugfix release of IntelMQ.
Installation instructions:
https://github.com/certtools/intelmq/blob/1.0.3/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.0.3/docs/UPGRADING.md
The released is published on PyPI, Github and the OpenBuildService (for
rpm/deb packages). If you installed intelmq with a package manager, the
new released will be installed automatically.
Full changelog:
### Contrib
* logrotate: use sudo for postrotate script
* cron-jobs: use the scripts in the bots' directories and link them
(#1056, #1142)
### Core
- `lib.harmonization`: Handle idna encoding error in FQDN sanitation
(#1175, #1176).
- `lib.bot`:
- Bots stop when redis gives the error "OOM command not allowed when
used memory > 'maxmemory'." (#1138).
- warnings of bots are catched by the logger (#1074, #1113).
- Fixed exitcodes 0 for graceful shutdowns .
- better handling of problems with pipeline and especially it's
initialization (#1178).
- All parsers using `ParserBot`'s methods now log the sum of
successfully parsed and failed lines at the end of each run (#1161).
### Harmonization
- Rule for harmonization keys is enforced (#1104, #1141).
- New allowed values for `classification.type`: `tor` & `leak` (see n6
parser below ).
### Bots
#### Collectors
- `bots.collectors.mail.collector_mail_attach`: Support attachment file
parsing for imbox versions newer than 0.9.5 (#1134).
- `bots.outputs.smtp.output`: Fix STARTTLS, threw an exception (#1152,
#1153).
#### Parsers
- All CSV parsers ignore NULL-bytes now, because the csv-library cannot
handle it (#967, #1114).
- `bots.experts.modify` default ruleset: changed conficker rule to catch
more spellings.
- `bots.parsers.shadowserver.parser`: Add Accessible Cisco Smart Install
(#1122).
- `bots.parsers.cleanmx.parser`: Handle new columns `first` and `last`,
rewritten for XML feed. See NEWS.md for upgrade instructions (#1131,
#1136, #1163).
- `bots.parsers.n6.parser`: Fix classification mappings. See NEWS file
for changes values (#738, #1127).
### Documentation
- `Release.md` add release procedure documentation
- `Bots.md`: fix example configuration for modify expert
### Tools
- intelmqctl now exits with exit codes > 0 when errors happened or the
operation was not successful. Also, the status operation exits with 1,
if bots are stopped, but enabled. (#977, #1143)
- `intelmctl check` checks for valid `run_mode` in runtime configuration
(#1140).
### Tests
- `tests.lib.test_pipeline`: Redis tests clear all queues before and
after tests (#1086).
- Repaired debian package build on travis (#1169).
- Warnings are not allowed by default, an allowed count can be specified
(#1129).
- `tests.bots.experts.cymru_whois/abusix`: Skipped on travis because of
ongoing problems.
### Packaging
* cron jobs: fix paths of executables
### Known issues
- `bots.collectors/outputs.xmpp` must be killed two times (#970).
- When running bots with `intelmqctl run [bot-id]` the log level is
always INFO (#1075).
- `intelmqctl run [bot-id] message send [msg]` does only support Events,
not Reports (#1077).
- `python3 setup.py sdist` does not include static files in the
resulting tarballs (#1146).
- `bots.parsers.cleanmx.parser`: The cleanMX feed may have FQDNs as IPs
in rare cases, such lines are dumped (#1162).
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hello everybody,
I have not much experience with intelmq nor python packages managments so while while implementing a couple of new bots, I have experienced the following issues:
1- According to the documentation the best way to proceed is to clone the repo and install intelmq via “pip3 install -e .”. The issue here is that it needs to be done from root, otherwise the install fails. However once it is installed the user intelmq cannot see the intelmq installation (pip3 list) since it is only installed for root.
2- If I Install from pip3 directly (not using a local repo) intelmq works just fine but then, new bot creation fails since according to the documentation new bots needs to be generated reinstalling intelmq. Trying to manually generate the binaries for the bots is not recommended according to previous emails (in the archive).
3- Using virtual env allowed me to install intelmq as recommended for developers (“pip3 install -e .”.) and new bots are generated appropriately . However, some issues appear with the pipeline connection and nowhere in the documentation it is suggested.
My question then is, what is the recommended and verified way of implementing new bots for intelmq? Starting from the installation of intelmq to testing the new bots.
My environment: VM with Ubuntu 16.04
Thanks in advance for your time!
Best regards,
Leandro Velasco
---
Rijnland 4c
1948 RL Beverwijk
Leandro.Velasco(a)dearbytes.nl<mailto:Leandro.Velasco@dearbytes.nl>
www.dearbytes.nl<http://www.dearbytes.nl>
Hi,
recently I came up into elasticsearch parsing exception.
Dump is attached below.
It only happens when it processes data from Blueliv Crimeserver and
Shadowserver-Open-XDMCP collectors.
Not so far ago my elasticsearch output bot didn't throw that exception.
Currently I'm using intelmq 1.0.2 and intelmq-manager 0.3.1, all
installed from .deb package and python client elasticsearch 6.0.0.
Anyone experienced the same?
Thanks for the efforts.
Regards,
--
Tomislav
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
since I'm new to Redis i would kindly ask for your assistance
to issue which is related to this error:
intelmq.lib.exceptions.PipelineError: pipeline failed -
ResponseError("OOM command not allowed when used memory > 'maxmemory'.",
)
Currently I have around 15GB memory allocated to intelmq virtual instanc
e.
In redis.conf in section "LIMITS" I have set
"maxmemory" to 6GB and
"maxmemory-policy volatile-lru"
I'm using scheduled run mode for collectors to run during the day at
different times with time spaces between them.
For example from 9-10h I have set "blocklist.de" collectors to fetch
data. After more then hour redis memory limit was reached and the
above error shows. I also tried with 10GB memory limit and the same
thing happened.
So I would kindly ask for some advise what to change to avoid this
situation?
Thanks in advance.
Kind regards,
- --
Tomislav
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iEYEARECAAYFAloxRG8ACgkQrREm8+n2Xc+uLwCggTkOK1E06gQGi4IGgjsO/Cc5
rTYAoPYo+PDhKPLzqD8YzFRwEjrLjHER
=vUYT
-----END PGP SIGNATURE-----
Dear community,
I just pushed the version 1.0.2 to pypi and the build servers.
Installation documentation:
https://github.com/certtools/intelmq/blob/1.0.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md
### Core
- `lib.message.add`: parameter force has finally been removed, should
have been gone in 1.0.0.rc1 already
### Bots
- `collectors.mail.collector_mail_url`: Fix bug which prevented marking
emails seen due to disconnects from server (#852).
- `parsers.spamhaus.parser_cert`: Handle/ignore 'AS?' in feed (#1111)
### Packaging
- The following changes have been in effect for the built packages
already since version 1.0.0
- Support building for more distributions, now supported: CentOS 7,
Debian 8 and 9, Fedora 25 and 26, RHEL 7, openSUSE Leap 42.2 and 42.3
and Tumbleweed, Ubuntu 14.04 and 16.04
- Use LSB-paths for created packages (/etc/intelmq/, /var/lib/intelmq/,
/run/intelmq/) (#470). Does does not affect installations with
setuptools/pip.
- Change the debian package format from native to quilt
- Fix problems in postint and postrm scripts
- Use systemd-tmpfile for creation of /run/intelmq/
### Documentation
- Add disclaimer on maxmind database in bot documentation and code and
the cron-job (#1110)
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi,
We need some consistent behavior for extracting files of downloaded
archives. For this, I'd like to hear some opinions from users. What do
you want to be able to configure, what should be done automatically? Do
you want it to be automatic and still have the possibility to override?
There are some possible settings:
* Do extraction at all
* What to extract? Some files vs everything. Can be combined with above
* archive type. Could be guessed from filename extension or mimetype.
The latter is as not trivial in python as I expected :/
Background:
The HTTP collector can currently extract files from zip-files on the
fly. There is no parameter for this, all files will be passed on as
separate reports.
The RT collector can extract zip on the fly if the parameter
`unzip_attachment` is true.
PR#1095[0] adds the ability to extract files for tar.gz archives
including a parameter `extract_files` to give a list of filenames to be
extracted. And all files will be extracted if the parameter is simply True.
Sebastian
[0]: https://github.com/certtools/intelmq/pull/1095
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg