IntelMQ-dev May 2016

intelmq-dev@lists.cert.at

5 participants
8 discussions

Intelmq-dev newletter 201605
by L. Aaron Kaplan 07 Aug '17

07 Aug '17

= Intelmq-dev-news 05-2016 Issue 5/2016 == Topics == # Summary of IHAP meeting in April # Status update Intevation # Status update CERT.at # Status update misc == May 2016 == Dear Intelmq-dev mailing list readers, this is the second issue of intelmq developer news. We hope it's useful. TL;DR and important changes ----------------------------- The syntax of intelmqcli was changed to a new format: intelmqctl {start,stop} bot_id. This breaks compatibilty with existing scripts. If you put intelmqctl into some script, please adapt it. Also please be sure to check out the latest version of the intelmq-manager in case you use it. Lots of open issues. Progress with intelmqcli (to connect postgresql to the RT ticket system). / TL;DR === How to contribute to this newsletter? === -> contact Aaron, Dustin for future input === Summary of IHAP meeting in April === In April the IHAP Meeting took place in Vienna. * A Hacksession the night before the meeting was used by Raphael and Aaron in order to bridge MISP and IntelMQ. * Connections between Abusehelper and IntelMQ are on some CERTs wish list. XMPP is a good start. Unfortunately the XMPP Bot upstream was not fit for production. === Status report Intevation === * Still working on the KontaktDB, we appreciate the discussions that started on IHAP Meeting. We received a Pull Request from Cert.at and are currently reviewing it. * We have Scripts to import Data into the KontaktDB. Nevertheless there is some work left. * Demonstrated installation from packages on Ubuntu 14.04 on IHAP-Meeting. We propose to host the **signed** packages on our public apt-repositories. * Working on a tool similar to intelmqcli, intended to process events from the eventdb. Instead of using RT they are sent by e-mail. The tool has the working title "event-processor" and can be found here (https://github.com/Intevation/event-processor) * We did not start with support for IODEF or X-ARF yet. === Status report CERT.at developments === * we moved to python3 only. Intelmq dropped python2 support (https://github.com/certtools/intelmq/commit/2cbb42f1458a7e90539a443ec5e50ee…). This does not apply yet to the certat repo (github.com/certat/intelmq), which still supports python2.7 but only for the intelmqcli tool. * New active contributor: pedro m. reis! Welcome and thanks for working so hard on the Bitsight collector (https://github.com/certtools/intelmq/pull/493) * intelmqcli tool now supports a lot of new flags: https://github.com/certat/intelmq/issues/52 This was necessary for CERT.at since we use intelmqcli via cron job to connect to the (postgresql) eventDB , pull out all of the new data and use RT (ticket system) to send stuff out. Added flags --quiet --batch. Now intelmqcli sends via cronjob. These flags now allow CERT.at to run intelmq in full auto-mode! intelmqcli is started via cron and sends out all events to all ISPs. === Requests === * Intevation searches for testers for the packages. * We'd like to have some nice graphs in the intelmq-manager: events/sec , parse-failures/sec, etc. * implementation of whitelisting of events (filter out events based on whitelists). See https://github.com/certtools/intelmq/issues/426 * A good CSS design for the web page === Community === * RIPE abuse-c contacts can be done locally. RIPE might be able to export abuse-c infos publicly (fingers crossed). * more command line options for intelmqcli (see the https://github.com/certat/intelmq repo) * Aaron gave a presentation at the ENISA workshop "CSIRTs in Europe", 11th of May. Slides will be shared on the ENISA page. ==== intelmq.org ==== The website intelmq.org is now online, but we would like to have more content and a proper design. Do you want to contribute to intelmq, but you are not a programmer? This is your chance! Current ToDos: * Create Website Content: How-Tos / Installation Instructions, Success Stories ** How-Tos / Instructions: If you are using a special feature of IntelMQ, for instance an expert bot, try to find some time to write down a short article how you managed to get it to work and why you are using it. * Website Design == Wishlist == * **we need more test-cases!!!** * a specific config logic for ASNs: do this and that (for example sett ttl = 1 month) if event is in ASN xyz. Or "ignore" if event is in ASN xyz. This should support some kind of more-specific-less-specific inheritance, similarly to Apache directory settings. The most specific setting wins. The order could be: country code -> ASN -> netblock -> ip (/32). Open questions: what's more relevant if both domains and numbers (ASN, IPs, net blocks) exist in an event? * block based processing: for example block based team cymru lookups * parallelisation: We need to revisit this topic == Important Discussions == In case you missed something, here are the headlines of some discussion we consider interesting / important. === Mailing Lists === * [Intelmq-dev] Packaging Strategy for Bots with dependencies * [Intelmq-dev] Discussion on intelmq output / transformation architecture * [Intelmq-dev] Output format to syslog/splunk (PR#503) == Communication == Chat: irc #intelmq on freenode or webchat: [[https://webchat.freenode.net/?channels=intelmq]] Follow on twitter: @intelmqorg Weekly Conference Call every Tuesday: Dial in via the known conference bridge number. It is [[https://en.wikipedia.org/wiki/Telephone_number_mapping|ENUM]] enabled. Ask Aaron or Dustin for the number if you want to participate.

5 7

Packaging Strategy for Bots with dependencies
by Dustin Demuth 07 Jul '16

07 Jul '16

Hey Folks, Whilst checking the dependencies, I've found out that a lot of Bots have their own requirements-file. From a packagers point of view, I think this is hard to maintain. I think, we should discuss if it is reasonable to split the software into a CORE package, were all packages have the same dependencies, and several BOTS packages with their own dependencies. This would create the additional need to rethink the BOTS json-file. It would be possible to have all JSON configs for the Bots in a bots.d directory and search for them in this directory. I know, that this might create additional complexity in some points. BR Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

8 15

Discussion on intelmq output / transformation architecture
by L. Aaron Kaplan 24 May '16

24 May '16

Hi folks, an interesting discussion started on https://github.com/certtools/intelmq/issues/487 Basically, pedro would like to send data to ArcSite in CEF format. I would like to propose that we enhance our architecture to include "transformer bots". Apart from the funny name which reminds me of http://transformers.hasbro.com/en-us/bots, this has a nice symmetry to the collector-> parser chain we have on the input side. So what are we talking about? Input side: ``` Collector ---> Parser ---> .... pipeline ``` Currently on the output side it's like this: ``` ... pipeline -> expert1 -> expert2 -> ... -> output bot 1 ``` But this basically leaves the work of transcoding/transforming the DHO format to the output bots. So far, this was enough. In the general case, it would be great to transform the DHO to some format. Let's assume CEF or IODEF. A generic output bot could then take that data and send it over whatever mechanism it knows to a destination. Example: ``` ... pipeline -> expert1 -> transform 2 IODEF -> mail bot or: ... pipeline -> expert1 -> transform 2 IODEF -> mail bot \ \-> sftp bot ``` The question in https://github.com/certtools/intelmq/issues/487 arose on where to store the output data. I'd suggest to create a new "raw_output" field in the DHO for this. A transformer bot shall write its format to the raw_output field (in base64) and an output bot shall take that data, decode the base64 andd send it via its own mechanism. What do you think? Would this work? Do you see any serious problem with this approach? Best, a.

3 2

xarf support (email sending and general mapping)
by Bernhard Reiter 24 May '16

24 May '16

Dear IntelMQ-Devs, for intelmq-mailgen I've constructed a first xarf writing support with an experimental mapping (see it at https://github.com/Intevation/intelmq-mailgen/issues/2) I haven't dived into it yet, but it seems like we need a practial mapping between xarf schemas (see https://github.com/certtools/intelmq/issues/522) So how do we construct such a mapping, so that in the end we can do xarf -> intelmq -> xarf? What intelmq field can be always expect or which only sometimes? I think you folks with more practical experience, having seen more data, are seminal for this mapping to become good. We (from Intevation) can implement the rules once we understand them well enough. :) Best, Bernhard -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

Output format to syslog/splunk (PR#503)
by Sebastian Wagner 24 May '16

24 May '16

Dear developers, contributors, users, etc. Pedro Reis (@pedromreis) opened a pull request for an UDP output bot, which can be used to send events to a syslog daemon (and then picked up by further processing software). The implementation has the following features: * Output formats are JSON or delimited by a configurable character * a optional header (at beginning of the line) can be set * `raw` field can be dropped I can see some potential problems with the 'delimited'-method here: * Strings can contain the delimiter itself, which breaks parsing. * Strings can contain arbitrary characters like \0 or \n which breaks everything Possible solutions could be: * ignore the problem as it's maybe not relevant * escape all problematic characters (solves problem with \n) * quote strings (solves problem with delimiters in strings) * strip non-printable characters * drop fields with non-printable characters * encode strings in base64 As you may have possible applications for this bot or you have experience with events in syslog, I would appreciate some feedback from you. Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

Re: [Intelmq-dev] Getting up and running CentOS
by Matthew Duncan 10 May '16

10 May '16

Thanks Sebastian, Running from master, I had to make a few modifications as CentOS is using different locations and usernames, but If I run the following: sudo -u intelmq /usr/bin/intelmqctl -t json I receive the following response null If I run the same command without the type specified, I receive the usage syntax help message usage: intelmqctl --bot [start|stop|restart|status] --id=cymru-expert intelmqctl --botnet [start|stop|restart|status] intelmqctl --list [bots|queues] ..... Other Notes: · CentOS uses “apache” in-place of www-data so I made the relevant modifications based on the install instructions. · I also has to ensure the $CONTROLLER variable as defined in the config.php matched the correct location (/usr/bin/intelmqctl instead of in local) From: Sebastian Wagner<mailto:wagner@cert.at> Sent: Tuesday, 10 May 2016 7:49 PM To: Matthew Duncan<mailto:matthew.duncan@mahdtech.com>; intelmq-dev(a)lists.cert.at<mailto:intelmq-dev@lists.cert.at> Subject: Re: [Intelmq-dev] Getting up and running CentOS Hi Matt, On 05/10/2016 11:43 AM, Matthew Duncan wrote: > I can successfully load the main page and “Configuration” page, > however if I click on either the “Management” or “Monitor” tabs, this > is what I get > > > > / / > > /Error loading botnet status: SyntaxError: JSON.parse: unexpected end > of data at line 1 column 1 of the JSON data/ > > / / > > > > Has anyone see this before or can provide some pointers where I have > gone wrong. > I assume you are running on current git master? The manager most probably can't access the cli-program. Possible reasons are insufficient permissions or misconfiguration or missing docs... To debug it, you can try to access the cli as user www-data (or whatever user your webserver is using): % sudo -u intelmq /usr/local/bin/intelmqctl Append `-t json` to see what the manager would see. Hope this helps, Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

Getting up and running CentOS
by Matthew Duncan 10 May '16

10 May '16

Hi there, You will have to excuse if this is really simple, I have just started with IntelMQ and have read the docs but can’t seem to find a fix. I am currently attempting to get IntelMQ and IntelMQ-Manager up and running on CentOS 7 and have stumbled into a road block. I can successfully start/stop and view bot configuration from the cli using intelmqctl however I am receiving an error using the Web UI. I can successfully load the main page and “Configuration” page, however if I click on either the “Management” or “Monitor” tabs, this is what I get Error loading botnet status: SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data Has anyone see this before or can provide some pointers where I have gone wrong. Thanks, Matt.

2 1

Possible License Conflict for ASN-Lookup Bot
by Dustin Demuth 02 May '16

02 May '16

Dear all, whilst packaging we've had a look at the licenses. Most things are ok, but there might be a problem with the asn_lookup expert. It makes use of pyasn (MIT-License) which builds on other products with different lisenses. One of them (mrtd) uses a 4-Clause BSD license which might be incompatible to AGPL. Do you know of alternatives to pyasn? Best Regards Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

4 5

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev May 2016