IntelMQ-dev

intelmq-dev@lists.cert.at

251 discussions

Intelmq-dev newletter 201605
by L. Aaron Kaplan 07 Aug '17

07 Aug '17

= Intelmq-dev-news 05-2016 Issue 5/2016 == Topics == # Summary of IHAP meeting in April # Status update Intevation # Status update CERT.at # Status update misc == May 2016 == Dear Intelmq-dev mailing list readers, this is the second issue of intelmq developer news. We hope it's useful. TL;DR and important changes ----------------------------- The syntax of intelmqcli was changed to a new format: intelmqctl {start,stop} bot_id. This breaks compatibilty with existing scripts. If you put intelmqctl into some script, please adapt it. Also please be sure to check out the latest version of the intelmq-manager in case you use it. Lots of open issues. Progress with intelmqcli (to connect postgresql to the RT ticket system). / TL;DR === How to contribute to this newsletter? === -> contact Aaron, Dustin for future input === Summary of IHAP meeting in April === In April the IHAP Meeting took place in Vienna. * A Hacksession the night before the meeting was used by Raphael and Aaron in order to bridge MISP and IntelMQ. * Connections between Abusehelper and IntelMQ are on some CERTs wish list. XMPP is a good start. Unfortunately the XMPP Bot upstream was not fit for production. === Status report Intevation === * Still working on the KontaktDB, we appreciate the discussions that started on IHAP Meeting. We received a Pull Request from Cert.at and are currently reviewing it. * We have Scripts to import Data into the KontaktDB. Nevertheless there is some work left. * Demonstrated installation from packages on Ubuntu 14.04 on IHAP-Meeting. We propose to host the **signed** packages on our public apt-repositories. * Working on a tool similar to intelmqcli, intended to process events from the eventdb. Instead of using RT they are sent by e-mail. The tool has the working title "event-processor" and can be found here (https://github.com/Intevation/event-processor) * We did not start with support for IODEF or X-ARF yet. === Status report CERT.at developments === * we moved to python3 only. Intelmq dropped python2 support (https://github.com/certtools/intelmq/commit/2cbb42f1458a7e90539a443ec5e50ee…). This does not apply yet to the certat repo (github.com/certat/intelmq), which still supports python2.7 but only for the intelmqcli tool. * New active contributor: pedro m. reis! Welcome and thanks for working so hard on the Bitsight collector (https://github.com/certtools/intelmq/pull/493) * intelmqcli tool now supports a lot of new flags: https://github.com/certat/intelmq/issues/52 This was necessary for CERT.at since we use intelmqcli via cron job to connect to the (postgresql) eventDB , pull out all of the new data and use RT (ticket system) to send stuff out. Added flags --quiet --batch. Now intelmqcli sends via cronjob. These flags now allow CERT.at to run intelmq in full auto-mode! intelmqcli is started via cron and sends out all events to all ISPs. === Requests === * Intevation searches for testers for the packages. * We'd like to have some nice graphs in the intelmq-manager: events/sec , parse-failures/sec, etc. * implementation of whitelisting of events (filter out events based on whitelists). See https://github.com/certtools/intelmq/issues/426 * A good CSS design for the web page === Community === * RIPE abuse-c contacts can be done locally. RIPE might be able to export abuse-c infos publicly (fingers crossed). * more command line options for intelmqcli (see the https://github.com/certat/intelmq repo) * Aaron gave a presentation at the ENISA workshop "CSIRTs in Europe", 11th of May. Slides will be shared on the ENISA page. ==== intelmq.org ==== The website intelmq.org is now online, but we would like to have more content and a proper design. Do you want to contribute to intelmq, but you are not a programmer? This is your chance! Current ToDos: * Create Website Content: How-Tos / Installation Instructions, Success Stories ** How-Tos / Instructions: If you are using a special feature of IntelMQ, for instance an expert bot, try to find some time to write down a short article how you managed to get it to work and why you are using it. * Website Design == Wishlist == * **we need more test-cases!!!** * a specific config logic for ASNs: do this and that (for example sett ttl = 1 month) if event is in ASN xyz. Or "ignore" if event is in ASN xyz. This should support some kind of more-specific-less-specific inheritance, similarly to Apache directory settings. The most specific setting wins. The order could be: country code -> ASN -> netblock -> ip (/32). Open questions: what's more relevant if both domains and numbers (ASN, IPs, net blocks) exist in an event? * block based processing: for example block based team cymru lookups * parallelisation: We need to revisit this topic == Important Discussions == In case you missed something, here are the headlines of some discussion we consider interesting / important. === Mailing Lists === * [Intelmq-dev] Packaging Strategy for Bots with dependencies * [Intelmq-dev] Discussion on intelmq output / transformation architecture * [Intelmq-dev] Output format to syslog/splunk (PR#503) == Communication == Chat: irc #intelmq on freenode or webchat: [[https://webchat.freenode.net/?channels=intelmq]] Follow on twitter: @intelmqorg Weekly Conference Call every Tuesday: Dial in via the known conference bridge number. It is [[https://en.wikipedia.org/wiki/Telephone_number_mapping|ENUM]] enabled. Ask Aaron or Dustin for the number if you want to participate.

5 7

IntelMQ 1.0 released!
by L. Aaron Kaplan 07 Aug '17

07 Aug '17

Dear colleagues and CERT-ies, dear abuse handling teams, (sorry for big x-posting) we are happy to announce (finally!) the official 1.0 release of our IntelMQ tool. What is it? =========== IntelMQ [1] is a free open source tool initially developed by CERT.pt and CERT.at to automatically handle and process the many incident reports (mostly shadowserver and similar) that we receive. At CERT.at it processes many thousands of events per day. But first of all, let me thank all the contributors and the different teams involved in this collaborative open source effort! Starting with CERT.pt (Tomas Lima and Mauro), BSI, Intevation (Bernhard, Dustin), CZ.NIC, CESNET, CERT Australia, CERT.ee, the IHAP [2] group and many many others. You coded, helped, discussed and attended the regular IHAP meetings which allowed us to discuss your wishes and requirements. This all - combined with the testing and coding efforts that many of you contributed - finally gave us version 1.0. today. We counted at least 45 contributors. IntelMQ has been running quite stable at CERT.at for nearly a year now and we are processing the bulk of the incident reports with it. Of course, a 1.0 version always begs for some 1.0.1 bugfixes :) So therefore we would like to ask you to report any bugs or change requests on github's issue tracker [3]. Where can I get it? =================== Follow the instructions in https://github.com/certtools/intelmq/tree/master/docs Note that we also have (.deb, .rpm) packages for download. [10] Future plans ============= We now tagged the master branch "1.0.0". This will remain stable now. We also started with a new "develop" branch which will become the 1.1 and 2.0 releases in the future. You can read more about our branching strategy here [4] Development will continue towards 1.1 with a set of wishes and requests that we received. You can view them in the issue tracker. CSP integration =============== Some of you already know that IntelMQ is a tool included into the "Core Service Platform" (CSP) as part of the CSIRT network [5]. We are very proud to offer our open source solution to the CSP. Integration into your incident handling automation ================================================== If you want to integrate IntelMQ into your incident handling automation environment, please note that you might want to use further tools such as "mail-gen" [6] or "intelmqcli" [7] (residing in separate repositories) which connect your ticket system (OTRS or RT) with IntelMQ. In case you have questions, we have * an IRC channel on freenode.net (#intelmq) * a users mailing list [8] * a developers mailing list [9] Thanks again everyone who participated in this open source solution! & feel free to (re-)tweet #intelmq L. Aaron Kaplan and Sebastian Wagner, CERT.at [1] https://github.com/certtools/intelmq/ [2] incident handling automation project. [3] https://github.com/certtools/intelmq/issues [4] https://github.com/certtools/intelmq/blob/master/docs/Developers-Guide.md#r… [5] https://www.enisa.europa.eu/news/enisa-news/2nd-informal-meeting-of-csirt-n… [6] https://github.com/Intevation/intelmq-mailgen [7] https://github.com/certat/intelmq [8] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users [9] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev [10] https://software.opensuse.org//download.html?project=home%3Asebix%3Aintelmq… -- // L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

New distributions for packages
by Sebastian Wagner 07 Aug '17

07 Aug '17

Hi all, I have added a few new distributions as build targets: * Debian 9.0 * Fedora 26 * RHEL 7 * Ubuntu 17.04 The packages for those can be considered beta and are thus not yet documented in docs/INSTALL.md, this will follow for version 1.1. This is the list of distributions with existing support: * CentOS 7 * Debian 8.0 * Fedora 25 * openSUSE Leap 42.2 and 42.3 * Ubuntu 16.04 These two are also supported for testing purpose: * Fedora Rawhide * openSUSE Tumbleweed And if you wonder why the version in the packages is 1.0.0.rel instead of plain 1.0.0: Dealing with version number is hard with debian packages, especially as pypi's version handling is very different to debian's. I hope I do not need this workaround for the next releases, but they do not cause any problems. Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

GeoIP expert crashes with invalid database
by Thomas Hungenberg 31 Jul '17

31 Jul '17

I noticed the geoip expert bot crashed on our IntelMQ instance. In the logs I found: # tail /opt/intelmq/var/log/maxmind-geoip-expert.log maxminddb.errors.InvalidDatabaseError: Error opening database file (/opt/intelmq/var/lib/bots/maxmind_geoip/GeoLite2-City.mmdb). Is this a valid MaxMind DB file? 2017-07-28 09:23:00,674 - maxmind-geoip-expert - INFO - Bot stopped. It looks like the database file was only partially downloaded with the cron-based update last night. :-/ I was able to fix this by running /usr/bin/update-geoip-data To avoid this, I think the updater should verify the downloaded database before overwriting the file. Also, it looks like the script /usr/bin/update-geoip-data simply passes on any parameters to "mv" which is probably not a good idea: # /usr/bin/update-geoip-data --help Usage: mv [OPTION]... [-T] SOURCE DEST or: mv [OPTION]... SOURCE... DIRECTORY or: mv [OPTION]... -t DIRECTORY SOURCE... Rename SOURCE to DEST, or move SOURCE(s) to DIRECTORY. - Thomas CERT-Bund Incident Response & Malware Analysis Team

2 1

Last call for version 1.0
by Sebastian Wagner 25 Jul '17

25 Jul '17

Dear community, As the number changes introduced after RC1 is very low, we do not need a RC2. If you discovered any critical bugs and did not report them yet, please raise your voice in the next days. Bugfixes for one of the four open bugs in 1.0 are still welcome: https://github.com/certtools/intelmq/issues?q=is%3Aopen+is%3Aissue+mileston… If no new issues are found the first stable version 1.0 can thus be expected at the beginning of next week. Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Version 1.0.0.rc1 release
by Sebastian Wagner 05 Jul '17

05 Jul '17

Dear community, I just uploaded/released the Release candidate 1 for our first stable version 1.0.0. Depending on the amount of bugs reported / bug fixes arriving, we will have a second release candidate or none at all. The stable version 1.0 can be expected at end of July. No more features are accepted for 1.0, only bugfixes. I created a branch develop, which will become the next feature release 1.1. All currently open pull requests are now pointing to it. The master branch will have the latest stable release, and currently - until the first stable release is done - it's the release candidate branch. See also https://github.com/certtools/intelmq/blob/master/docs/Developers-Guide.md#r… and the following section Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Fwd: [Intelmq-users] openbl.org down -> bot removal
by L. Aaron Kaplan 27 Jun '17

27 Jun '17

Reminder... we now also have an intelmq-users(a)lists.cert.at mailing list. This is intended for users of the system. Feel free to join (even if you are on the -dev list) Best regards, a. > Begin forwarded message: > > From: Sebastian Wagner <wagner(a)cert.at> > Subject: [Intelmq-users] openbl.org down -> bot removal > Date: 26 June 2017 at 17:37:10 GMT+2 > To: intelmq-users(a)lists.cert.at > > Hi, > > OpenBL.org shut down their services at end of May. I removed the OpenBL > parser now. > > Sebastian > > https://github.com/certtools/intelmq/issues/1018 > https://github.com/certtools/intelmq/commit/d677112a02a016d097c224866e78524… > > -- > // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 > // CERT Austria - https://www.cert.at/ > // Eine Initiative der nic.at GmbH - https://www.nic.at/ > // Firmenbuchnummer 172568b, LG Salzburg > > > -- > Listen-Einstellungen: > https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users -- // L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Working on 1.0
by Sebastian Wagner 14 Jun '17

14 Jun '17

Dear community, We now have a pretty stable codebase and I think we should take this chance to finally release our first stable version 1.0 and implement upcoming bigger changes afterwards for 1.1 My plan for the next days would be: * to merge PR #974[1] which fixes our last feature request #743[2] for 1.0 * work on PR #853[3] * release a beta version Then we can work on the remaining bugs, and afterwards release the first RC which should then be thoroughly tested. This process also implies some changes to the workflow concerning e.g. the branches and packages. I will inform you later about these (basically it is described in #650[4]). Sebastian [1]: https://github.com/certtools/intelmq/pull/974 [2]: https://github.com/certtools/intelmq/issues/743 [3]: https://github.com/certtools/intelmq/pull/853 [4]: https://github.com/certtools/intelmq/issues/650 P.S.: If you have experience with CentOS, the intelmq-manager needs your help! -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

Destination host in malware feeds
by Thomas Hungenberg 14 Jun '17

14 Jun '17

Hi all, next to the destination IP and port, malware sinkhole feeds commonly also include information on the destination 'host'. This value is usually taken from the 'Host:' header of the HTTP request sent to the sinkhole by the infected client. The corresponding columns in the feeds are usually called 'dsthost', 'http_host' (e.g. Shadowserver HTTP-Sinkhole-Drone) or 'cc_dns' (Shadowserver Botnet-Drone). The name 'cc_dns' is probably misleading. In most cases, the value of the destination host field in the feeds is a FQDN - but depending on how the malware sets up the "Host:" header in the HTTP requests, it may also be FQDN:Port, an IP address or IP:Port (or just some crap). Some malware families perform a DNS lookup for the (DGA based) C2 domain first and then only use the IP address in the HTTP requests instead of the domain name. I noticed that the IntelMQ parsers for the Shadowserver feeds simply try to map the value of 'http_host' to 'destination.fqdn', e.g. https://github.com/certtools/intelmq/blob/master/intelmq/bots/parsers/shado… Line 295: ('destination.fqdn', 'http_host'), The scripts we currently use for generating notifications to network owners use the following logic (pseudo code here) to extract the destination host information: if (feed.dstip is a valid ip) { destination.ip = dstip } else { discard event # destination IP is required } dsthost = feed.http_host dsthost =~ s/:[0-9]+$// # remove potential port information (we already have this information in dstport) if (dsthost is a fqdn) { destination.fqdn = dsthost } else { do nothing # dsthost is either an IP (which we already have in destination.ip) or useless data } I'd like to suggest using a similar logic in the InteMQ parsers for the Shadowserver (an other malware sinkhole) feeds to not miss destination.fqdn information which comes in the form FQDN:Port - instead of writing the information to an extras field which is most probably not used by the output generators or even dropping the complete event. - Thomas CERT-Bund Incident Response & Malware Analysis Team

4 6

new intelmq-users mailinglist
by Sebastian Wagner 14 Jun '17

14 Jun '17

Dear contributors, As the intelmq project gains momentum more and more users are using the software. Also, not all users are developers anymore. To address the users need of support/assistance/help we created a new mailinglist intelmq-users. Please subscribe here: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users The archive is public. I also hope that with this step the Github Issues are only used for bugreports and not questions. best wishes, Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

← Newer
1
...
16
17
18
19
20
21
22
...
26
Older →

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev