Dear community
I have just release a new bugfix release of IntelMQ.
Installation instructions:
https://github.com/certtools/intelmq/blob/1.0.3/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.0.3/docs/UPGRADING.md
The released is published on PyPI, Github and the OpenBuildService (for
rpm/deb packages). If you installed intelmq with a package manager, the
new released will be installed automatically.
Full changelog:
### Contrib
* logrotate: use sudo for postrotate script
* cron-jobs: use the scripts in the bots' directories and link them
(#1056, #1142)
### Core
- `lib.harmonization`: Handle idna encoding error in FQDN sanitation
(#1175, #1176).
- `lib.bot`:
- Bots stop when redis gives the error "OOM command not allowed when
used memory > 'maxmemory'." (#1138).
- warnings of bots are catched by the logger (#1074, #1113).
- Fixed exitcodes 0 for graceful shutdowns .
- better handling of problems with pipeline and especially it's
initialization (#1178).
- All parsers using `ParserBot`'s methods now log the sum of
successfully parsed and failed lines at the end of each run (#1161).
### Harmonization
- Rule for harmonization keys is enforced (#1104, #1141).
- New allowed values for `classification.type`: `tor` & `leak` (see n6
parser below ).
### Bots
#### Collectors
- `bots.collectors.mail.collector_mail_attach`: Support attachment file
parsing for imbox versions newer than 0.9.5 (#1134).
- `bots.outputs.smtp.output`: Fix STARTTLS, threw an exception (#1152,
#1153).
#### Parsers
- All CSV parsers ignore NULL-bytes now, because the csv-library cannot
handle it (#967, #1114).
- `bots.experts.modify` default ruleset: changed conficker rule to catch
more spellings.
- `bots.parsers.shadowserver.parser`: Add Accessible Cisco Smart Install
(#1122).
- `bots.parsers.cleanmx.parser`: Handle new columns `first` and `last`,
rewritten for XML feed. See NEWS.md for upgrade instructions (#1131,
#1136, #1163).
- `bots.parsers.n6.parser`: Fix classification mappings. See NEWS file
for changes values (#738, #1127).
### Documentation
- `Release.md` add release procedure documentation
- `Bots.md`: fix example configuration for modify expert
### Tools
- intelmqctl now exits with exit codes > 0 when errors happened or the
operation was not successful. Also, the status operation exits with 1,
if bots are stopped, but enabled. (#977, #1143)
- `intelmctl check` checks for valid `run_mode` in runtime configuration
(#1140).
### Tests
- `tests.lib.test_pipeline`: Redis tests clear all queues before and
after tests (#1086).
- Repaired debian package build on travis (#1169).
- Warnings are not allowed by default, an allowed count can be specified
(#1129).
- `tests.bots.experts.cymru_whois/abusix`: Skipped on travis because of
ongoing problems.
### Packaging
* cron jobs: fix paths of executables
### Known issues
- `bots.collectors/outputs.xmpp` must be killed two times (#970).
- When running bots with `intelmqctl run [bot-id]` the log level is
always INFO (#1075).
- `intelmqctl run [bot-id] message send [msg]` does only support Events,
not Reports (#1077).
- `python3 setup.py sdist` does not include static files in the
resulting tarballs (#1146).
- `bots.parsers.cleanmx.parser`: The cleanMX feed may have FQDNs as IPs
in rare cases, such lines are dumped (#1162).
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear list,
in pull request #944 (netlab 360 enh [0]) by navtej an issue came up
which can't be solved trivially:
The feed Netlab 360 DGA[1] - which is already included in intelmq -
provides a validity time frame for each domain. Most of those (~90%) end
in 2030 while the start date is the current day at 00:00.
So both start and end time are artificial. And the source claims the
event is valid in the future, which is a very odd. And does it actually
make sense to forward this kind of information?
Also, we can't really handle this time information using the current
harmonization.
One idea would be to set time.source to time.observation if the
time.source is in the future. So time.source <= time.observation does
always apply.
What do you think?
Sebastian
[0]: https://github.com/certtools/intelmq/pull/944
[1]: http://data.netlab.360.com/feeds/dga/dga.txt - attention, quite
big! The domains at the beginning have a very near end date.
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi all,
Recently, I have installed IntelMQ in a CentOS 7.4 host (fully
patched) and I see some "errors" in official IntelMQ's rpm packages
installed from http://download.opensuse.org/repositories/home:/sebix:/intelmq/CentOS_7/.
a/ /etc/cron.d/intelmq-update-data. Content is:
# /etc/cron.d/intelmq-update-data: crontab fragment for intelmq
# This updates the data files used by some expert bots.
#
# m h dom mon dow command
# Update data for tor_nodes bot:
11 0 * * * intelmq /usr/bin/update-tor-nodes
/var/lib/intelmq/bots/tor_nodes/tor_nodes.dat
# Update data for maxmind_geoip bot:
17 0 * * * intelmq /usr/bin/update-geoip-data
/var/lib/intelmq/bots/maxmind_geoip/GeoLite2-City.mmdb
# Update data for asn_lookup bot:
23 0 * * * intelmq /usr/bin/update-asn-data
/var/lib/intelmq/bots/asn_lookup/ipasn.dat
# Update data for the RIPE DB abuse_c offline contact lookup
25 6 * * * intelmq /usr/bin/update-ripencc_abuse_contact_offline
/var/lib/intelmq/bots/ripencc_abuse_contact_offline/
Where are these scripts: update-tor-nodes, update-geoip-data,
update-asn-data and update-ripencc_abuse_contact_offline? They don't
exist in my system. But exists intelmq-update-asn-data,
intelmq-update-geoip-data and intelmq-update-tor-nodes (not ripe).
b/ /etc/logrotate.d/intelmq. Content is:
compress
delaycompress
copytruncate
create 640 intelmq intelmq
/var/log/intelmq/*.log {
su intelmq intelmq
daily
maxsize 10M
rotate 60
notifempty
sharedscripts
postrotate
/usr/bin/intelmqctl reload --quiet
endscript
}
/var/lib/intelmq/bots/file-output/*.txt {
su intelmq intelmq
daily
maxsize 10M
rotate 60
notifempty
sharedscripts
postrotate
/usr/bin/intelmqctl reload file-output --quiet
endscript
}
... but returns the following email error:
From root(a)cosintelmq.mydomain.com Mon Nov 13 08:29:04 2017
Return-Path: <root(a)cosintelmq.mydomain.com>
X-Original-To: root
Delivered-To: root(a)cosintelmq.mydomain.com
From: Anacron <root(a)cosintelmq.mydomain.com>
To: root(a)cosintelmq.mydomain.com
Content-Type: text/plain; charset="UTF-8"
Subject: Anacron job 'cron.daily' on cosintelmq.mydomain.com
Date: Mon, 13 Nov 2017 08:29:04 +0000 (UTC)
Status: R
/etc/cron.daily/logrotate:
intelmqctl: Running intelmqctl as root is highly discouraged!
usage: intelmqctl [-h] [-v] [--type {text,json}] [--quiet]
{list,check,clear,log,run,help,start,stop,restart,reload,status,enable,disable}
...
intelmqctl: error: unrecognized arguments: --quiet
error: error running shared postrotate script for '/var/log/intelmq/*.log '
Maybe is it more correct to do this:
- /usr/bin/intelmqctl reload --quiet
+ su -m intelmq -c ' /usr/bin/intelmqctl reload --quiet'
- /usr/bin/intelmqctl reload file-output --quiet
+ su -m intelmq -c '/usr/bin/intelmqctl reload file-output --quiet' ??
Thanks.
Hi All,
I am currently in the process of deciding whether ANZ should incorporate IntelMQ into its Threat Intelligence ingestion and sharing platform.
At the Deepsec conference Sebastian mentioned updating the harmonization to allow for fields with multiple values. Has this issue been progressed at all? We will require multiple values for some fields in our events, and I was considering adding this functionality (perhaps in a hacky way) to my own fork, but I would like an update on the progress on the work on the master before doing so.
Regards,
Alex Knight | ANZ | ISO | Cyber Security Engineering
Level 8, 55 Collins Street, Melbourne 3000
Phone: +61 386 545 888 | www.anz.com<http://www.anz.com/>
"This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."
Dear community,
I just pushed the version 1.0.2 to pypi and the build servers.
Installation documentation:
https://github.com/certtools/intelmq/blob/1.0.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md
### Core
- `lib.message.add`: parameter force has finally been removed, should
have been gone in 1.0.0.rc1 already
### Bots
- `collectors.mail.collector_mail_url`: Fix bug which prevented marking
emails seen due to disconnects from server (#852).
- `parsers.spamhaus.parser_cert`: Handle/ignore 'AS?' in feed (#1111)
### Packaging
- The following changes have been in effect for the built packages
already since version 1.0.0
- Support building for more distributions, now supported: CentOS 7,
Debian 8 and 9, Fedora 25 and 26, RHEL 7, openSUSE Leap 42.2 and 42.3
and Tumbleweed, Ubuntu 14.04 and 16.04
- Use LSB-paths for created packages (/etc/intelmq/, /var/lib/intelmq/,
/run/intelmq/) (#470). Does does not affect installations with
setuptools/pip.
- Change the debian package format from native to quilt
- Fix problems in postint and postrm scripts
- Use systemd-tmpfile for creation of /run/intelmq/
### Documentation
- Add disclaimer on maxmind database in bot documentation and code and
the cron-job (#1110)
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi,
recently I've discovered that there are a lot of security analysts actively
participating on Twitter. By participating I mean that they are posting
quite interesting data (@illegalFawn for example) and i thought that even
if the the amount of data being posted there is not that great it could
provide an interesting source of iocs, which could take traditional feeds a
lot of time to publish. For this a played a bit with the Twitter official
rest api and produced a demo which I would like to get your feedback on it
and what you think could be improved. The code can be found here:
https://codeshare.io/aVKXq9. The bot so far works like this: except for the
necessary parameters for twitter api it requires two lists of users, one
represents accounts which timeline will be processed (this is the feed-like
behaviour) the other list represents the users which mark the interesting
tweets (presumably "owners" of the bot) that should be downloaded the
"mark" here means like. This behaviour allows for automatic collection of
data from accounts like I've posted on the beginning, which post feed-like
information and a manual selection of interesting tweets from accounts
which post "various" posts. The bot gets tweets in bulk, that means that it
gets all the tweets and liked tweets and passes them on in concatenated
report. I've consulted this bot with Sebastian Wagner and he pointed out
some weaknesses of this way mainly data and feed classification. A better
approach is probably by creating a report for each individual which eases
the classification (which could be now done using hashtags if present). The
bot lacks a lot of comments and documentation so ask away if some features
are not clear. Again, I'd like to get your feedback and opinions on this
since I think it could be an interesting addition to intelmq ecosystem.
Sincerely,
Václav Brůžek
Hi,
All I wanted to have in the first place is an easy way to find out if an
event has any extra.* field. I.e. in python: 'extra' in event.
But this does not work, because there is no such field, only extra.foo.
So, what about changing the __contains__ method of the Message class? It
could check for sub-fields if necessary.
Using the (wrong) algorithm to check if some existing key starts with
(in this example) `extra` it turns out this also leads to the convenient
situation that you now can't add e.g. `extra.error.message` if
`extra.error` was previously defined.
And: the alienvault otx parser actually did this with extra.pulse
This is good, but also prevents adding `extra.errormessage` if
`extra.error` exists. The correct algorithm is to check if some field
starts with `extra.`. But using this assumption does not prevent
hierarchy conflicts.
IMO, we want both:
1) a simple way to check if any field of a group is (not) present
2) prevention to add keys which conflict with others in the hierarchy,
i.e. `extra.error.message` must not be added if `extra.error` is present.
We could adapt the contains-operator that e.g. a search for `extra.`
means: "Is there any field in the extra.* domain?". It's an invalid key
name anyway[1].
And for adding new fields: check if there is some value in a higher
hierarchy, if so, fail. Overwriting is not allowed in this case.
Any thoughts on this?
Sebastian
P.S.: In the long rung we can probably save the fields hierarchically
internally too, would avoid some of these shortcomings.
We also had problems with the hierarchy of malware.hash in the past:
https://github.com/certtools/intelmq/issues/732
[1]: actually it is allowed currently, but it shouldn't and does not
make sense, see https://github.com/certtools/intelmq/issues/1104
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear users and contributors,
Yesterday I release version 0.3 and today 0.3.1 (containing a fix for a
bug preventing the saving of files).
This release contains a lot of exciting usability fixes and
enhancements. See the changelog below for a full list. We are getting
close to a stable release now!
Please refer to the installation docs. Deb and rpm packages are
available. Note that for the deb-packages, you need to set group
permissions on the configuration files first.
This is the changelog of 0.3:
* Partly support for CentOS/RHEL 7 (#55, #103)
* Note on security considerations in Readme to avoid misunderstandings
* Show versions of intelmq and intelmq manager on about page
* Update vis.js to current version
### Configuration
* interface for defaults.conf (#45)
* drag&drop (#105, #41)
* fix #96
* save buttons starts blinking after changes (#41)
* Allow redrawing of botnet on demand
* Save/load position of bots in/from /opt/intelmq/etc/manager/positions.conf
File needs to be writeable
* parameters from defaults are shown for new bots (#107)
* parameters are grouped by type: generic, runtime, defaults
* better feedback on errors with backend (#69, #99)
* pressing ESC in forms equals to pressing the cancel button
* Edit node window is now much bigger
* pressing enter in 'add key' window equals to pressing ok button
### Management
* Reload and restart have been added as actions on bots and the whole
botnet (#114)
* A click on the bot name opens the monitor page of the bot
### Monitor
* clearing queues is possible in general and specific view for all
queues (#54)
### Backend
* Fix regex checks on bot ids and log line number in controller, they
have not been effective
* fix overflow in extended message box (#49)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
I just released the bug fix release IntelMQ 1.0.1.
The existing bugs which have not been fixed in time have been moved to
the 1.0.2 milestone (10 bugs).
For upgrade instructions look at the documentation:
https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md
Updated packages have been built and are available in the repositories.
From the changelog:
### Documentation
- Feeds: use more https:// URLs
- minor fixes
### Bots
- bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for
rest.db.ripe.net
- bots/outputs/file/output.py: properly close the file handle on shutdown
### Core
- lib/bot: Bots will now log the used intelmq version at startup
### Tools
- intelmqctl: To check the status of a bot, the comandline of the
running process is compared to the actual executable of the bot.
Otherwise unrelated programs with the same PID are detected as running bot.
- intelmqctl: the "enable", "disable", "check", "clear" commands now
support the JSON output
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
I just published the release candidate for the next bugfix release
1.0.1. You can expect the final release next week / end of august.
Changelog:
### Documentation
- Feeds: use more https:// URLs
- minor fixes
### Bots
- bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for
rest.db.ripe.net
- bots/outputs/file/output.py: properly close the file handle on shutdown
### Core
- lib/bot: Bots will now log the used intelmq version at startup
### Tools
- intelmqctl: To check the status of a bot, the comandline of the
running process is compared to the actual executable of the bot.
Otherwise unrelated programs with the same PID are detected as running bot.
- intelmqctl: enable, disable, check, clear now support the JSON output
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg