IntelMQ-dev

intelmq-dev@lists.cert.at

6 participants
232 discussions

handling of time frames
by Sebastian Wagner 14 Dec '17

14 Dec '17

Dear list, in pull request #944 (netlab 360 enh [0]) by navtej an issue came up which can't be solved trivially: The feed Netlab 360 DGA[1] - which is already included in intelmq - provides a validity time frame for each domain. Most of those (~90%) end in 2030 while the start date is the current day at 00:00. So both start and end time are artificial. And the source claims the event is valid in the future, which is a very odd. And does it actually make sense to forward this kind of information? Also, we can't really handle this time information using the current harmonization. One idea would be to set time.source to time.observation if the time.source is in the future. So time.source <= time.observation does always apply. What do you think? Sebastian [0]: https://github.com/certtools/intelmq/pull/944 [1]: http://data.netlab.360.com/feeds/dga/dga.txt - attention, quite big! The domains at the beginning have a very near end date. -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 3

Regarding rpm packages for RHEL/CentOS
by C. L. Martinez 14 Nov '17

14 Nov '17

Hi all, Recently, I have installed IntelMQ in a CentOS 7.4 host (fully patched) and I see some "errors" in official IntelMQ's rpm packages installed from http://download.opensuse.org/repositories/home:/sebix:/intelmq/CentOS_7/. a/ /etc/cron.d/intelmq-update-data. Content is: # /etc/cron.d/intelmq-update-data: crontab fragment for intelmq # This updates the data files used by some expert bots. # # m h dom mon dow command # Update data for tor_nodes bot: 11 0 * * * intelmq /usr/bin/update-tor-nodes /var/lib/intelmq/bots/tor_nodes/tor_nodes.dat # Update data for maxmind_geoip bot: 17 0 * * * intelmq /usr/bin/update-geoip-data /var/lib/intelmq/bots/maxmind_geoip/GeoLite2-City.mmdb # Update data for asn_lookup bot: 23 0 * * * intelmq /usr/bin/update-asn-data /var/lib/intelmq/bots/asn_lookup/ipasn.dat # Update data for the RIPE DB abuse_c offline contact lookup 25 6 * * * intelmq /usr/bin/update-ripencc_abuse_contact_offline /var/lib/intelmq/bots/ripencc_abuse_contact_offline/ Where are these scripts: update-tor-nodes, update-geoip-data, update-asn-data and update-ripencc_abuse_contact_offline? They don't exist in my system. But exists intelmq-update-asn-data, intelmq-update-geoip-data and intelmq-update-tor-nodes (not ripe). b/ /etc/logrotate.d/intelmq. Content is: compress delaycompress copytruncate create 640 intelmq intelmq /var/log/intelmq/*.log { su intelmq intelmq daily maxsize 10M rotate 60 notifempty sharedscripts postrotate /usr/bin/intelmqctl reload --quiet endscript } /var/lib/intelmq/bots/file-output/*.txt { su intelmq intelmq daily maxsize 10M rotate 60 notifempty sharedscripts postrotate /usr/bin/intelmqctl reload file-output --quiet endscript } ... but returns the following email error: From root(a)cosintelmq.mydomain.com Mon Nov 13 08:29:04 2017 Return-Path: <root(a)cosintelmq.mydomain.com> X-Original-To: root Delivered-To: root(a)cosintelmq.mydomain.com From: Anacron <root(a)cosintelmq.mydomain.com> To: root(a)cosintelmq.mydomain.com Content-Type: text/plain; charset="UTF-8" Subject: Anacron job 'cron.daily' on cosintelmq.mydomain.com Date: Mon, 13 Nov 2017 08:29:04 +0000 (UTC) Status: R /etc/cron.daily/logrotate: intelmqctl: Running intelmqctl as root is highly discouraged! usage: intelmqctl [-h] [-v] [--type {text,json}] [--quiet] {list,check,clear,log,run,help,start,stop,restart,reload,status,enable,disable} ... intelmqctl: error: unrecognized arguments: --quiet error: error running shared postrotate script for '/var/log/intelmq/*.log ' Maybe is it more correct to do this: - /usr/bin/intelmqctl reload --quiet + su -m intelmq -c ' /usr/bin/intelmqctl reload --quiet' - /usr/bin/intelmqctl reload file-output --quiet + su -m intelmq -c '/usr/bin/intelmqctl reload file-output --quiet' ?? Thanks.

2 2

Data Harmonization - Fields with multiple values
by Knight, Alexander 13 Nov '17

13 Nov '17

Hi All, I am currently in the process of deciding whether ANZ should incorporate IntelMQ into its Threat Intelligence ingestion and sharing platform. At the Deepsec conference Sebastian mentioned updating the harmonization to allow for fields with multiple values. Has this issue been progressed at all? We will require multiple values for some fields in our events, and I was considering adding this functionality (perhaps in a hacky way) to my own fork, but I would like an update on the progress on the work on the master before doing so. Regards, Alex Knight | ANZ | ISO | Cyber Security Engineering Level 8, 55 Collins Street, Melbourne 3000 Phone: +61 386 545 888 | www.anz.com<http://www.anz.com/> "This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."

4 6

Bugfix release 1.0.2
by Sebastian Wagner 09 Nov '17

09 Nov '17

Dear community, I just pushed the version 1.0.2 to pypi and the build servers. Installation documentation: https://github.com/certtools/intelmq/blob/1.0.0/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md ### Core - `lib.message.add`: parameter force has finally been removed, should have been gone in 1.0.0.rc1 already ### Bots - `collectors.mail.collector_mail_url`: Fix bug which prevented marking emails seen due to disconnects from server (#852). - `parsers.spamhaus.parser_cert`: Handle/ignore 'AS?' in feed (#1111) ### Packaging - The following changes have been in effect for the built packages already since version 1.0.0 - Support building for more distributions, now supported: CentOS 7, Debian 8 and 9, Fedora 25 and 26, RHEL 7, openSUSE Leap 42.2 and 42.3 and Tumbleweed, Ubuntu 14.04 and 16.04 - Use LSB-paths for created packages (/etc/intelmq/, /var/lib/intelmq/, /run/intelmq/) (#470). Does does not affect installations with setuptools/pip. - Change the debian package format from native to quilt - Fix problems in postint and postrm scripts - Use systemd-tmpfile for creation of /run/intelmq/ ### Documentation - Add disclaimer on maxmind database in bot documentation and code and the cron-job (#1110) Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Twitter bot a proposal
by Vaclav Bruzek 18 Oct '17

18 Oct '17

Hi, recently I've discovered that there are a lot of security analysts actively participating on Twitter. By participating I mean that they are posting quite interesting data (@illegalFawn for example) and i thought that even if the the amount of data being posted there is not that great it could provide an interesting source of iocs, which could take traditional feeds a lot of time to publish. For this a played a bit with the Twitter official rest api and produced a demo which I would like to get your feedback on it and what you think could be improved. The code can be found here: https://codeshare.io/aVKXq9. The bot so far works like this: except for the necessary parameters for twitter api it requires two lists of users, one represents accounts which timeline will be processed (this is the feed-like behaviour) the other list represents the users which mark the interesting tweets (presumably "owners" of the bot) that should be downloaded the "mark" here means like. This behaviour allows for automatic collection of data from accounts like I've posted on the beginning, which post feed-like information and a manual selection of interesting tweets from accounts which post "various" posts. The bot gets tweets in bulk, that means that it gets all the tweets and liked tweets and passes them on in concatenated report. I've consulted this bot with Sebastian Wagner and he pointed out some weaknesses of this way mainly data and feed classification. A better approach is probably by creating a report for each individual which eases the classification (which could be now done using hashtags if present). The bot lacks a lot of comments and documentation so ask away if some features are not clear. Again, I'd like to get your feedback and opinions on this since I think it could be an interesting addition to intelmq ecosystem. Sincerely, Václav Brůžek

2 1

contains-operator and hierarchies of messages
by Sebastian Wagner 10 Oct '17

10 Oct '17

Hi, All I wanted to have in the first place is an easy way to find out if an event has any extra.* field. I.e. in python: 'extra' in event. But this does not work, because there is no such field, only extra.foo. So, what about changing the __contains__ method of the Message class? It could check for sub-fields if necessary. Using the (wrong) algorithm to check if some existing key starts with (in this example) `extra` it turns out this also leads to the convenient situation that you now can't add e.g. `extra.error.message` if `extra.error` was previously defined. And: the alienvault otx parser actually did this with extra.pulse This is good, but also prevents adding `extra.errormessage` if `extra.error` exists. The correct algorithm is to check if some field starts with `extra.`. But using this assumption does not prevent hierarchy conflicts. IMO, we want both: 1) a simple way to check if any field of a group is (not) present 2) prevention to add keys which conflict with others in the hierarchy, i.e. `extra.error.message` must not be added if `extra.error` is present. We could adapt the contains-operator that e.g. a search for `extra.` means: "Is there any field in the extra.* domain?". It's an invalid key name anyway[1]. And for adding new fields: check if there is some value in a higher hierarchy, if so, fail. Overwriting is not allowed in this case. Any thoughts on this? Sebastian P.S.: In the long rung we can probably save the fields hierarchically internally too, would avoid some of these shortcomings. We also had problems with the hierarchy of malware.hash in the past: https://github.com/certtools/intelmq/issues/732 [1]: actually it is allowed currently, but it shouldn't and does not make sense, see https://github.com/certtools/intelmq/issues/1104 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq-manager release 0.3 and 0.3.1
by Sebastian Wagner 26 Sep '17

26 Sep '17

Dear users and contributors, Yesterday I release version 0.3 and today 0.3.1 (containing a fix for a bug preventing the saving of files). This release contains a lot of exciting usability fixes and enhancements. See the changelog below for a full list. We are getting close to a stable release now! Please refer to the installation docs. Deb and rpm packages are available. Note that for the deb-packages, you need to set group permissions on the configuration files first. This is the changelog of 0.3: * Partly support for CentOS/RHEL 7 (#55, #103) * Note on security considerations in Readme to avoid misunderstandings * Show versions of intelmq and intelmq manager on about page * Update vis.js to current version ### Configuration * interface for defaults.conf (#45) * drag&drop (#105, #41) * fix #96 * save buttons starts blinking after changes (#41) * Allow redrawing of botnet on demand * Save/load position of bots in/from /opt/intelmq/etc/manager/positions.conf File needs to be writeable * parameters from defaults are shown for new bots (#107) * parameters are grouped by type: generic, runtime, defaults * better feedback on errors with backend (#69, #99) * pressing ESC in forms equals to pressing the cancel button * Edit node window is now much bigger * pressing enter in 'add key' window equals to pressing ok button ### Management * Reload and restart have been added as actions on bots and the whole botnet (#114) * A click on the bot name opens the monitor page of the bot ### Monitor * clearing queues is possible in general and specific view for all queues (#54) ### Backend * Fix regex checks on bot ids and log line number in controller, they have not been effective * fix overflow in extended message box (#49) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

1.0.1 bugfix release
by Sebastian Wagner 30 Aug '17

30 Aug '17

Dear community, I just released the bug fix release IntelMQ 1.0.1. The existing bugs which have not been fixed in time have been moved to the 1.0.2 milestone (10 bugs). For upgrade instructions look at the documentation: https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md Updated packages have been built and are available in the repositories. From the changelog: ### Documentation - Feeds: use more https:// URLs - minor fixes ### Bots - bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for rest.db.ripe.net - bots/outputs/file/output.py: properly close the file handle on shutdown ### Core - lib/bot: Bots will now log the used intelmq version at startup ### Tools - intelmqctl: To check the status of a bot, the comandline of the running process is compared to the actual executable of the bot. Otherwise unrelated programs with the same PID are detected as running bot. - intelmqctl: the "enable", "disable", "check", "clear" commands now support the JSON output Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

Published release candidate 1.0.1
by Sebastian Wagner 23 Aug '17

23 Aug '17

I just published the release candidate for the next bugfix release 1.0.1. You can expect the final release next week / end of august. Changelog: ### Documentation - Feeds: use more https:// URLs - minor fixes ### Bots - bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for rest.db.ripe.net - bots/outputs/file/output.py: properly close the file handle on shutdown ### Core - lib/bot: Bots will now log the used intelmq version at startup ### Tools - intelmqctl: To check the status of a bot, the comandline of the running process is compared to the actual executable of the bot. Otherwise unrelated programs with the same PID are detected as running bot. - intelmqctl: enable, disable, check, clear now support the JSON output -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Harmonization configuration
by Vaclav Bruzek 21 Aug '17

21 Aug '17

Hi, I recently tried playing with my custom fields in harmonization.conf which would be needed for a potential bot. However I ran into some problems with that during testing, following is the displayed error: "AssertionError: Regex matched: 'ERROR' matches '(ERROR.*?){1}' Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 145, in start\n self.process() File "/untitled/test_bot.py", line 34, in process event.add("test.field", 50) File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/message.py", line 188, in add raise exceptions.InvalidKey(key) intelmq.lib.exceptions.InvalidKey: invalid key \'test.field\' Configuration in harmonization.conf is as follows for event and for report: "'test.field": {"description": "test value", "type": "Integer"}, I checked whether I am using the version in /opt/intelmq/etc/ and everything seemed to be ok. I couldn't figure out what is wrong because everything seems to be in order. I am using the 1.0.0 version of intelmq, on elementary OS 0.4.1. -- Sincerely, Václav Brůžek

2 1

← Newer
1
...
13
14
15
16
17
18
19
...
24
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev