IntelMQ-dev

intelmq-dev@lists.cert.at

251 discussions

Regarding rpm packages for RHEL/CentOS
by C. L. Martinez 14 Nov '17

14 Nov '17

Hi all, Recently, I have installed IntelMQ in a CentOS 7.4 host (fully patched) and I see some "errors" in official IntelMQ's rpm packages installed from http://download.opensuse.org/repositories/home:/sebix:/intelmq/CentOS_7/. a/ /etc/cron.d/intelmq-update-data. Content is: # /etc/cron.d/intelmq-update-data: crontab fragment for intelmq # This updates the data files used by some expert bots. # # m h dom mon dow command # Update data for tor_nodes bot: 11 0 * * * intelmq /usr/bin/update-tor-nodes /var/lib/intelmq/bots/tor_nodes/tor_nodes.dat # Update data for maxmind_geoip bot: 17 0 * * * intelmq /usr/bin/update-geoip-data /var/lib/intelmq/bots/maxmind_geoip/GeoLite2-City.mmdb # Update data for asn_lookup bot: 23 0 * * * intelmq /usr/bin/update-asn-data /var/lib/intelmq/bots/asn_lookup/ipasn.dat # Update data for the RIPE DB abuse_c offline contact lookup 25 6 * * * intelmq /usr/bin/update-ripencc_abuse_contact_offline /var/lib/intelmq/bots/ripencc_abuse_contact_offline/ Where are these scripts: update-tor-nodes, update-geoip-data, update-asn-data and update-ripencc_abuse_contact_offline? They don't exist in my system. But exists intelmq-update-asn-data, intelmq-update-geoip-data and intelmq-update-tor-nodes (not ripe). b/ /etc/logrotate.d/intelmq. Content is: compress delaycompress copytruncate create 640 intelmq intelmq /var/log/intelmq/*.log { su intelmq intelmq daily maxsize 10M rotate 60 notifempty sharedscripts postrotate /usr/bin/intelmqctl reload --quiet endscript } /var/lib/intelmq/bots/file-output/*.txt { su intelmq intelmq daily maxsize 10M rotate 60 notifempty sharedscripts postrotate /usr/bin/intelmqctl reload file-output --quiet endscript } ... but returns the following email error: From root(a)cosintelmq.mydomain.com Mon Nov 13 08:29:04 2017 Return-Path: <root(a)cosintelmq.mydomain.com> X-Original-To: root Delivered-To: root(a)cosintelmq.mydomain.com From: Anacron <root(a)cosintelmq.mydomain.com> To: root(a)cosintelmq.mydomain.com Content-Type: text/plain; charset="UTF-8" Subject: Anacron job 'cron.daily' on cosintelmq.mydomain.com Date: Mon, 13 Nov 2017 08:29:04 +0000 (UTC) Status: R /etc/cron.daily/logrotate: intelmqctl: Running intelmqctl as root is highly discouraged! usage: intelmqctl [-h] [-v] [--type {text,json}] [--quiet] {list,check,clear,log,run,help,start,stop,restart,reload,status,enable,disable} ... intelmqctl: error: unrecognized arguments: --quiet error: error running shared postrotate script for '/var/log/intelmq/*.log ' Maybe is it more correct to do this: - /usr/bin/intelmqctl reload --quiet + su -m intelmq -c ' /usr/bin/intelmqctl reload --quiet' - /usr/bin/intelmqctl reload file-output --quiet + su -m intelmq -c '/usr/bin/intelmqctl reload file-output --quiet' ?? Thanks.

2 2

Data Harmonization - Fields with multiple values
by Knight, Alexander 13 Nov '17

13 Nov '17

Hi All, I am currently in the process of deciding whether ANZ should incorporate IntelMQ into its Threat Intelligence ingestion and sharing platform. At the Deepsec conference Sebastian mentioned updating the harmonization to allow for fields with multiple values. Has this issue been progressed at all? We will require multiple values for some fields in our events, and I was considering adding this functionality (perhaps in a hacky way) to my own fork, but I would like an update on the progress on the work on the master before doing so. Regards, Alex Knight | ANZ | ISO | Cyber Security Engineering Level 8, 55 Collins Street, Melbourne 3000 Phone: +61 386 545 888 | www.anz.com<http://www.anz.com/> "This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."

4 6

Bugfix release 1.0.2
by Sebastian Wagner 09 Nov '17

09 Nov '17

Dear community, I just pushed the version 1.0.2 to pypi and the build servers. Installation documentation: https://github.com/certtools/intelmq/blob/1.0.0/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md ### Core - `lib.message.add`: parameter force has finally been removed, should have been gone in 1.0.0.rc1 already ### Bots - `collectors.mail.collector_mail_url`: Fix bug which prevented marking emails seen due to disconnects from server (#852). - `parsers.spamhaus.parser_cert`: Handle/ignore 'AS?' in feed (#1111) ### Packaging - The following changes have been in effect for the built packages already since version 1.0.0 - Support building for more distributions, now supported: CentOS 7, Debian 8 and 9, Fedora 25 and 26, RHEL 7, openSUSE Leap 42.2 and 42.3 and Tumbleweed, Ubuntu 14.04 and 16.04 - Use LSB-paths for created packages (/etc/intelmq/, /var/lib/intelmq/, /run/intelmq/) (#470). Does does not affect installations with setuptools/pip. - Change the debian package format from native to quilt - Fix problems in postint and postrm scripts - Use systemd-tmpfile for creation of /run/intelmq/ ### Documentation - Add disclaimer on maxmind database in bot documentation and code and the cron-job (#1110) Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Twitter bot a proposal
by Vaclav Bruzek 18 Oct '17

18 Oct '17

Hi, recently I've discovered that there are a lot of security analysts actively participating on Twitter. By participating I mean that they are posting quite interesting data (@illegalFawn for example) and i thought that even if the the amount of data being posted there is not that great it could provide an interesting source of iocs, which could take traditional feeds a lot of time to publish. For this a played a bit with the Twitter official rest api and produced a demo which I would like to get your feedback on it and what you think could be improved. The code can be found here: https://codeshare.io/aVKXq9. The bot so far works like this: except for the necessary parameters for twitter api it requires two lists of users, one represents accounts which timeline will be processed (this is the feed-like behaviour) the other list represents the users which mark the interesting tweets (presumably "owners" of the bot) that should be downloaded the "mark" here means like. This behaviour allows for automatic collection of data from accounts like I've posted on the beginning, which post feed-like information and a manual selection of interesting tweets from accounts which post "various" posts. The bot gets tweets in bulk, that means that it gets all the tweets and liked tweets and passes them on in concatenated report. I've consulted this bot with Sebastian Wagner and he pointed out some weaknesses of this way mainly data and feed classification. A better approach is probably by creating a report for each individual which eases the classification (which could be now done using hashtags if present). The bot lacks a lot of comments and documentation so ask away if some features are not clear. Again, I'd like to get your feedback and opinions on this since I think it could be an interesting addition to intelmq ecosystem. Sincerely, Václav Brůžek

2 1

contains-operator and hierarchies of messages
by Sebastian Wagner 10 Oct '17

10 Oct '17

Hi, All I wanted to have in the first place is an easy way to find out if an event has any extra.* field. I.e. in python: 'extra' in event. But this does not work, because there is no such field, only extra.foo. So, what about changing the __contains__ method of the Message class? It could check for sub-fields if necessary. Using the (wrong) algorithm to check if some existing key starts with (in this example) `extra` it turns out this also leads to the convenient situation that you now can't add e.g. `extra.error.message` if `extra.error` was previously defined. And: the alienvault otx parser actually did this with extra.pulse This is good, but also prevents adding `extra.errormessage` if `extra.error` exists. The correct algorithm is to check if some field starts with `extra.`. But using this assumption does not prevent hierarchy conflicts. IMO, we want both: 1) a simple way to check if any field of a group is (not) present 2) prevention to add keys which conflict with others in the hierarchy, i.e. `extra.error.message` must not be added if `extra.error` is present. We could adapt the contains-operator that e.g. a search for `extra.` means: "Is there any field in the extra.* domain?". It's an invalid key name anyway[1]. And for adding new fields: check if there is some value in a higher hierarchy, if so, fail. Overwriting is not allowed in this case. Any thoughts on this? Sebastian P.S.: In the long rung we can probably save the fields hierarchically internally too, would avoid some of these shortcomings. We also had problems with the hierarchy of malware.hash in the past: https://github.com/certtools/intelmq/issues/732 [1]: actually it is allowed currently, but it shouldn't and does not make sense, see https://github.com/certtools/intelmq/issues/1104 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq-manager release 0.3 and 0.3.1
by Sebastian Wagner 26 Sep '17

26 Sep '17

Dear users and contributors, Yesterday I release version 0.3 and today 0.3.1 (containing a fix for a bug preventing the saving of files). This release contains a lot of exciting usability fixes and enhancements. See the changelog below for a full list. We are getting close to a stable release now! Please refer to the installation docs. Deb and rpm packages are available. Note that for the deb-packages, you need to set group permissions on the configuration files first. This is the changelog of 0.3: * Partly support for CentOS/RHEL 7 (#55, #103) * Note on security considerations in Readme to avoid misunderstandings * Show versions of intelmq and intelmq manager on about page * Update vis.js to current version ### Configuration * interface for defaults.conf (#45) * drag&drop (#105, #41) * fix #96 * save buttons starts blinking after changes (#41) * Allow redrawing of botnet on demand * Save/load position of bots in/from /opt/intelmq/etc/manager/positions.conf File needs to be writeable * parameters from defaults are shown for new bots (#107) * parameters are grouped by type: generic, runtime, defaults * better feedback on errors with backend (#69, #99) * pressing ESC in forms equals to pressing the cancel button * Edit node window is now much bigger * pressing enter in 'add key' window equals to pressing ok button ### Management * Reload and restart have been added as actions on bots and the whole botnet (#114) * A click on the bot name opens the monitor page of the bot ### Monitor * clearing queues is possible in general and specific view for all queues (#54) ### Backend * Fix regex checks on bot ids and log line number in controller, they have not been effective * fix overflow in extended message box (#49) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

1.0.1 bugfix release
by Sebastian Wagner 30 Aug '17

30 Aug '17

Dear community, I just released the bug fix release IntelMQ 1.0.1. The existing bugs which have not been fixed in time have been moved to the 1.0.2 milestone (10 bugs). For upgrade instructions look at the documentation: https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md Updated packages have been built and are available in the repositories. From the changelog: ### Documentation - Feeds: use more https:// URLs - minor fixes ### Bots - bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for rest.db.ripe.net - bots/outputs/file/output.py: properly close the file handle on shutdown ### Core - lib/bot: Bots will now log the used intelmq version at startup ### Tools - intelmqctl: To check the status of a bot, the comandline of the running process is compared to the actual executable of the bot. Otherwise unrelated programs with the same PID are detected as running bot. - intelmqctl: the "enable", "disable", "check", "clear" commands now support the JSON output Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

Published release candidate 1.0.1
by Sebastian Wagner 23 Aug '17

23 Aug '17

I just published the release candidate for the next bugfix release 1.0.1. You can expect the final release next week / end of august. Changelog: ### Documentation - Feeds: use more https:// URLs - minor fixes ### Bots - bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for rest.db.ripe.net - bots/outputs/file/output.py: properly close the file handle on shutdown ### Core - lib/bot: Bots will now log the used intelmq version at startup ### Tools - intelmqctl: To check the status of a bot, the comandline of the running process is compared to the actual executable of the bot. Otherwise unrelated programs with the same PID are detected as running bot. - intelmqctl: enable, disable, check, clear now support the JSON output -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Harmonization configuration
by Vaclav Bruzek 21 Aug '17

21 Aug '17

Hi, I recently tried playing with my custom fields in harmonization.conf which would be needed for a potential bot. However I ran into some problems with that during testing, following is the displayed error: "AssertionError: Regex matched: 'ERROR' matches '(ERROR.*?){1}' Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 145, in start\n self.process() File "/untitled/test_bot.py", line 34, in process event.add("test.field", 50) File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/message.py", line 188, in add raise exceptions.InvalidKey(key) intelmq.lib.exceptions.InvalidKey: invalid key \'test.field\' Configuration in harmonization.conf is as follows for event and for report: "'test.field": {"description": "test value", "type": "Integer"}, I checked whether I am using the version in /opt/intelmq/etc/ and everything seemed to be ok. I couldn't figure out what is wrong because everything seems to be in order. I am using the 1.0.0 version of intelmq, on elementary OS 0.4.1. -- Sincerely, Václav Brůžek

2 1

usage of extra field made easier
by Sebastian Wagner 16 Aug '17

16 Aug '17

Dear contributors, using the extra.* fields was not so easy and could easily lead to errors with overwriting[1]. In PR#1046[2] I proposed a new behavior for this field. This introduces a new harmonization type 'JSONDict' which is the same as 'JSON' but requires the content to be a dictionary. The harmonization now allows "subfields" as they are used by the new 'JSONDict' type, including validity checks for the values. The message library now handles any read/write access to extra.* transparently, no workarounds are needed. That's the main feature. The field is properly exploded, see bug #791[3]. All bots currently present in certtools/develop are converted to use the new simpler behavior, including the tests. This change requires backwards compatibility: Bots can still use the old interfaces (reading and writing the whole field), that behavior should either be removed in 2.0 or be made consistent for all hierarchical fields. Old configurations (type of extra is JSON) still work, but raises a warning in the `intelmqctl check` tool. This behavior will be removed in 2.0 too. So compatibility with 1.0 is achieved, both for code and configuration. Do you see any issues with this approach? Sebastian [1]: For example here: https://github.com/certtools/intelmq/pull/1004#discussion_r121417044 [2]: https://github.com/certtools/intelmq/pull/1046 [3]: https://github.com/certtools/intelmq/issues/791 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

← Newer
1
...
15
16
17
18
19
20
21
...
26
Older →

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev