IntelMQ-dev

intelmq-dev@lists.cert.at

3 participants
238 discussions

Twitter bot a proposal
by Vaclav Bruzek 18 Oct '17

18 Oct '17

Hi, recently I've discovered that there are a lot of security analysts actively participating on Twitter. By participating I mean that they are posting quite interesting data (@illegalFawn for example) and i thought that even if the the amount of data being posted there is not that great it could provide an interesting source of iocs, which could take traditional feeds a lot of time to publish. For this a played a bit with the Twitter official rest api and produced a demo which I would like to get your feedback on it and what you think could be improved. The code can be found here: https://codeshare.io/aVKXq9. The bot so far works like this: except for the necessary parameters for twitter api it requires two lists of users, one represents accounts which timeline will be processed (this is the feed-like behaviour) the other list represents the users which mark the interesting tweets (presumably "owners" of the bot) that should be downloaded the "mark" here means like. This behaviour allows for automatic collection of data from accounts like I've posted on the beginning, which post feed-like information and a manual selection of interesting tweets from accounts which post "various" posts. The bot gets tweets in bulk, that means that it gets all the tweets and liked tweets and passes them on in concatenated report. I've consulted this bot with Sebastian Wagner and he pointed out some weaknesses of this way mainly data and feed classification. A better approach is probably by creating a report for each individual which eases the classification (which could be now done using hashtags if present). The bot lacks a lot of comments and documentation so ask away if some features are not clear. Again, I'd like to get your feedback and opinions on this since I think it could be an interesting addition to intelmq ecosystem. Sincerely, Václav Brůžek

2 1

contains-operator and hierarchies of messages
by Sebastian Wagner 10 Oct '17

10 Oct '17

Hi, All I wanted to have in the first place is an easy way to find out if an event has any extra.* field. I.e. in python: 'extra' in event. But this does not work, because there is no such field, only extra.foo. So, what about changing the __contains__ method of the Message class? It could check for sub-fields if necessary. Using the (wrong) algorithm to check if some existing key starts with (in this example) `extra` it turns out this also leads to the convenient situation that you now can't add e.g. `extra.error.message` if `extra.error` was previously defined. And: the alienvault otx parser actually did this with extra.pulse This is good, but also prevents adding `extra.errormessage` if `extra.error` exists. The correct algorithm is to check if some field starts with `extra.`. But using this assumption does not prevent hierarchy conflicts. IMO, we want both: 1) a simple way to check if any field of a group is (not) present 2) prevention to add keys which conflict with others in the hierarchy, i.e. `extra.error.message` must not be added if `extra.error` is present. We could adapt the contains-operator that e.g. a search for `extra.` means: "Is there any field in the extra.* domain?". It's an invalid key name anyway[1]. And for adding new fields: check if there is some value in a higher hierarchy, if so, fail. Overwriting is not allowed in this case. Any thoughts on this? Sebastian P.S.: In the long rung we can probably save the fields hierarchically internally too, would avoid some of these shortcomings. We also had problems with the hierarchy of malware.hash in the past: https://github.com/certtools/intelmq/issues/732 [1]: actually it is allowed currently, but it shouldn't and does not make sense, see https://github.com/certtools/intelmq/issues/1104 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq-manager release 0.3 and 0.3.1
by Sebastian Wagner 26 Sep '17

26 Sep '17

Dear users and contributors, Yesterday I release version 0.3 and today 0.3.1 (containing a fix for a bug preventing the saving of files). This release contains a lot of exciting usability fixes and enhancements. See the changelog below for a full list. We are getting close to a stable release now! Please refer to the installation docs. Deb and rpm packages are available. Note that for the deb-packages, you need to set group permissions on the configuration files first. This is the changelog of 0.3: * Partly support for CentOS/RHEL 7 (#55, #103) * Note on security considerations in Readme to avoid misunderstandings * Show versions of intelmq and intelmq manager on about page * Update vis.js to current version ### Configuration * interface for defaults.conf (#45) * drag&drop (#105, #41) * fix #96 * save buttons starts blinking after changes (#41) * Allow redrawing of botnet on demand * Save/load position of bots in/from /opt/intelmq/etc/manager/positions.conf File needs to be writeable * parameters from defaults are shown for new bots (#107) * parameters are grouped by type: generic, runtime, defaults * better feedback on errors with backend (#69, #99) * pressing ESC in forms equals to pressing the cancel button * Edit node window is now much bigger * pressing enter in 'add key' window equals to pressing ok button ### Management * Reload and restart have been added as actions on bots and the whole botnet (#114) * A click on the bot name opens the monitor page of the bot ### Monitor * clearing queues is possible in general and specific view for all queues (#54) ### Backend * Fix regex checks on bot ids and log line number in controller, they have not been effective * fix overflow in extended message box (#49) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

1.0.1 bugfix release
by Sebastian Wagner 30 Aug '17

30 Aug '17

Dear community, I just released the bug fix release IntelMQ 1.0.1. The existing bugs which have not been fixed in time have been moved to the 1.0.2 milestone (10 bugs). For upgrade instructions look at the documentation: https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md Updated packages have been built and are available in the repositories. From the changelog: ### Documentation - Feeds: use more https:// URLs - minor fixes ### Bots - bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for rest.db.ripe.net - bots/outputs/file/output.py: properly close the file handle on shutdown ### Core - lib/bot: Bots will now log the used intelmq version at startup ### Tools - intelmqctl: To check the status of a bot, the comandline of the running process is compared to the actual executable of the bot. Otherwise unrelated programs with the same PID are detected as running bot. - intelmqctl: the "enable", "disable", "check", "clear" commands now support the JSON output Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

Published release candidate 1.0.1
by Sebastian Wagner 23 Aug '17

23 Aug '17

I just published the release candidate for the next bugfix release 1.0.1. You can expect the final release next week / end of august. Changelog: ### Documentation - Feeds: use more https:// URLs - minor fixes ### Bots - bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for rest.db.ripe.net - bots/outputs/file/output.py: properly close the file handle on shutdown ### Core - lib/bot: Bots will now log the used intelmq version at startup ### Tools - intelmqctl: To check the status of a bot, the comandline of the running process is compared to the actual executable of the bot. Otherwise unrelated programs with the same PID are detected as running bot. - intelmqctl: enable, disable, check, clear now support the JSON output -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Harmonization configuration
by Vaclav Bruzek 21 Aug '17

21 Aug '17

Hi, I recently tried playing with my custom fields in harmonization.conf which would be needed for a potential bot. However I ran into some problems with that during testing, following is the displayed error: "AssertionError: Regex matched: 'ERROR' matches '(ERROR.*?){1}' Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 145, in start\n self.process() File "/untitled/test_bot.py", line 34, in process event.add("test.field", 50) File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/message.py", line 188, in add raise exceptions.InvalidKey(key) intelmq.lib.exceptions.InvalidKey: invalid key \'test.field\' Configuration in harmonization.conf is as follows for event and for report: "'test.field": {"description": "test value", "type": "Integer"}, I checked whether I am using the version in /opt/intelmq/etc/ and everything seemed to be ok. I couldn't figure out what is wrong because everything seems to be in order. I am using the 1.0.0 version of intelmq, on elementary OS 0.4.1. -- Sincerely, Václav Brůžek

2 1

usage of extra field made easier
by Sebastian Wagner 16 Aug '17

16 Aug '17

Dear contributors, using the extra.* fields was not so easy and could easily lead to errors with overwriting[1]. In PR#1046[2] I proposed a new behavior for this field. This introduces a new harmonization type 'JSONDict' which is the same as 'JSON' but requires the content to be a dictionary. The harmonization now allows "subfields" as they are used by the new 'JSONDict' type, including validity checks for the values. The message library now handles any read/write access to extra.* transparently, no workarounds are needed. That's the main feature. The field is properly exploded, see bug #791[3]. All bots currently present in certtools/develop are converted to use the new simpler behavior, including the tests. This change requires backwards compatibility: Bots can still use the old interfaces (reading and writing the whole field), that behavior should either be removed in 2.0 or be made consistent for all hierarchical fields. Old configurations (type of extra is JSON) still work, but raises a warning in the `intelmqctl check` tool. This behavior will be removed in 2.0 too. So compatibility with 1.0 is achieved, both for code and configuration. Do you see any issues with this approach? Sebastian [1]: For example here: https://github.com/certtools/intelmq/pull/1004#discussion_r121417044 [2]: https://github.com/certtools/intelmq/pull/1046 [3]: https://github.com/certtools/intelmq/issues/791 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Intelmq-dev newletter 201605
by L. Aaron Kaplan 07 Aug '17

07 Aug '17

= Intelmq-dev-news 05-2016 Issue 5/2016 == Topics == # Summary of IHAP meeting in April # Status update Intevation # Status update CERT.at # Status update misc == May 2016 == Dear Intelmq-dev mailing list readers, this is the second issue of intelmq developer news. We hope it's useful. TL;DR and important changes ----------------------------- The syntax of intelmqcli was changed to a new format: intelmqctl {start,stop} bot_id. This breaks compatibilty with existing scripts. If you put intelmqctl into some script, please adapt it. Also please be sure to check out the latest version of the intelmq-manager in case you use it. Lots of open issues. Progress with intelmqcli (to connect postgresql to the RT ticket system). / TL;DR === How to contribute to this newsletter? === -> contact Aaron, Dustin for future input === Summary of IHAP meeting in April === In April the IHAP Meeting took place in Vienna. * A Hacksession the night before the meeting was used by Raphael and Aaron in order to bridge MISP and IntelMQ. * Connections between Abusehelper and IntelMQ are on some CERTs wish list. XMPP is a good start. Unfortunately the XMPP Bot upstream was not fit for production. === Status report Intevation === * Still working on the KontaktDB, we appreciate the discussions that started on IHAP Meeting. We received a Pull Request from Cert.at and are currently reviewing it. * We have Scripts to import Data into the KontaktDB. Nevertheless there is some work left. * Demonstrated installation from packages on Ubuntu 14.04 on IHAP-Meeting. We propose to host the **signed** packages on our public apt-repositories. * Working on a tool similar to intelmqcli, intended to process events from the eventdb. Instead of using RT they are sent by e-mail. The tool has the working title "event-processor" and can be found here (https://github.com/Intevation/event-processor) * We did not start with support for IODEF or X-ARF yet. === Status report CERT.at developments === * we moved to python3 only. Intelmq dropped python2 support (https://github.com/certtools/intelmq/commit/2cbb42f1458a7e90539a443ec5e50ee…). This does not apply yet to the certat repo (github.com/certat/intelmq), which still supports python2.7 but only for the intelmqcli tool. * New active contributor: pedro m. reis! Welcome and thanks for working so hard on the Bitsight collector (https://github.com/certtools/intelmq/pull/493) * intelmqcli tool now supports a lot of new flags: https://github.com/certat/intelmq/issues/52 This was necessary for CERT.at since we use intelmqcli via cron job to connect to the (postgresql) eventDB , pull out all of the new data and use RT (ticket system) to send stuff out. Added flags --quiet --batch. Now intelmqcli sends via cronjob. These flags now allow CERT.at to run intelmq in full auto-mode! intelmqcli is started via cron and sends out all events to all ISPs. === Requests === * Intevation searches for testers for the packages. * We'd like to have some nice graphs in the intelmq-manager: events/sec , parse-failures/sec, etc. * implementation of whitelisting of events (filter out events based on whitelists). See https://github.com/certtools/intelmq/issues/426 * A good CSS design for the web page === Community === * RIPE abuse-c contacts can be done locally. RIPE might be able to export abuse-c infos publicly (fingers crossed). * more command line options for intelmqcli (see the https://github.com/certat/intelmq repo) * Aaron gave a presentation at the ENISA workshop "CSIRTs in Europe", 11th of May. Slides will be shared on the ENISA page. ==== intelmq.org ==== The website intelmq.org is now online, but we would like to have more content and a proper design. Do you want to contribute to intelmq, but you are not a programmer? This is your chance! Current ToDos: * Create Website Content: How-Tos / Installation Instructions, Success Stories ** How-Tos / Instructions: If you are using a special feature of IntelMQ, for instance an expert bot, try to find some time to write down a short article how you managed to get it to work and why you are using it. * Website Design == Wishlist == * **we need more test-cases!!!** * a specific config logic for ASNs: do this and that (for example sett ttl = 1 month) if event is in ASN xyz. Or "ignore" if event is in ASN xyz. This should support some kind of more-specific-less-specific inheritance, similarly to Apache directory settings. The most specific setting wins. The order could be: country code -> ASN -> netblock -> ip (/32). Open questions: what's more relevant if both domains and numbers (ASN, IPs, net blocks) exist in an event? * block based processing: for example block based team cymru lookups * parallelisation: We need to revisit this topic == Important Discussions == In case you missed something, here are the headlines of some discussion we consider interesting / important. === Mailing Lists === * [Intelmq-dev] Packaging Strategy for Bots with dependencies * [Intelmq-dev] Discussion on intelmq output / transformation architecture * [Intelmq-dev] Output format to syslog/splunk (PR#503) == Communication == Chat: irc #intelmq on freenode or webchat: [[https://webchat.freenode.net/?channels=intelmq]] Follow on twitter: @intelmqorg Weekly Conference Call every Tuesday: Dial in via the known conference bridge number. It is [[https://en.wikipedia.org/wiki/Telephone_number_mapping|ENUM]] enabled. Ask Aaron or Dustin for the number if you want to participate.

5 7

IntelMQ 1.0 released!
by L. Aaron Kaplan 07 Aug '17

07 Aug '17

Dear colleagues and CERT-ies, dear abuse handling teams, (sorry for big x-posting) we are happy to announce (finally!) the official 1.0 release of our IntelMQ tool. What is it? =========== IntelMQ [1] is a free open source tool initially developed by CERT.pt and CERT.at to automatically handle and process the many incident reports (mostly shadowserver and similar) that we receive. At CERT.at it processes many thousands of events per day. But first of all, let me thank all the contributors and the different teams involved in this collaborative open source effort! Starting with CERT.pt (Tomas Lima and Mauro), BSI, Intevation (Bernhard, Dustin), CZ.NIC, CESNET, CERT Australia, CERT.ee, the IHAP [2] group and many many others. You coded, helped, discussed and attended the regular IHAP meetings which allowed us to discuss your wishes and requirements. This all - combined with the testing and coding efforts that many of you contributed - finally gave us version 1.0. today. We counted at least 45 contributors. IntelMQ has been running quite stable at CERT.at for nearly a year now and we are processing the bulk of the incident reports with it. Of course, a 1.0 version always begs for some 1.0.1 bugfixes :) So therefore we would like to ask you to report any bugs or change requests on github's issue tracker [3]. Where can I get it? =================== Follow the instructions in https://github.com/certtools/intelmq/tree/master/docs Note that we also have (.deb, .rpm) packages for download. [10] Future plans ============= We now tagged the master branch "1.0.0". This will remain stable now. We also started with a new "develop" branch which will become the 1.1 and 2.0 releases in the future. You can read more about our branching strategy here [4] Development will continue towards 1.1 with a set of wishes and requests that we received. You can view them in the issue tracker. CSP integration =============== Some of you already know that IntelMQ is a tool included into the "Core Service Platform" (CSP) as part of the CSIRT network [5]. We are very proud to offer our open source solution to the CSP. Integration into your incident handling automation ================================================== If you want to integrate IntelMQ into your incident handling automation environment, please note that you might want to use further tools such as "mail-gen" [6] or "intelmqcli" [7] (residing in separate repositories) which connect your ticket system (OTRS or RT) with IntelMQ. In case you have questions, we have * an IRC channel on freenode.net (#intelmq) * a users mailing list [8] * a developers mailing list [9] Thanks again everyone who participated in this open source solution! & feel free to (re-)tweet #intelmq L. Aaron Kaplan and Sebastian Wagner, CERT.at [1] https://github.com/certtools/intelmq/ [2] incident handling automation project. [3] https://github.com/certtools/intelmq/issues [4] https://github.com/certtools/intelmq/blob/master/docs/Developers-Guide.md#r… [5] https://www.enisa.europa.eu/news/enisa-news/2nd-informal-meeting-of-csirt-n… [6] https://github.com/Intevation/intelmq-mailgen [7] https://github.com/certat/intelmq [8] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users [9] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev [10] https://software.opensuse.org//download.html?project=home%3Asebix%3Aintelmq… -- // L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

New distributions for packages
by Sebastian Wagner 07 Aug '17

07 Aug '17

Hi all, I have added a few new distributions as build targets: * Debian 9.0 * Fedora 26 * RHEL 7 * Ubuntu 17.04 The packages for those can be considered beta and are thus not yet documented in docs/INSTALL.md, this will follow for version 1.1. This is the list of distributions with existing support: * CentOS 7 * Debian 8.0 * Fedora 25 * openSUSE Leap 42.2 and 42.3 * Ubuntu 16.04 These two are also supported for testing purpose: * Fedora Rawhide * openSUSE Tumbleweed And if you wonder why the version in the packages is 1.0.0.rel instead of plain 1.0.0: Dealing with version number is hard with debian packages, especially as pypi's version handling is very different to debian's. I hope I do not need this workaround for the next releases, but they do not cause any problems. Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

← Newer
1
...
14
15
16
17
18
19
20
...
24
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev