IntelMQ-dev

intelmq-dev@lists.cert.at

3 participants
238 discussions

Discussion on intelmq output / transformation architecture
by L. Aaron Kaplan 24 May '16

24 May '16

Hi folks, an interesting discussion started on https://github.com/certtools/intelmq/issues/487 Basically, pedro would like to send data to ArcSite in CEF format. I would like to propose that we enhance our architecture to include "transformer bots". Apart from the funny name which reminds me of http://transformers.hasbro.com/en-us/bots, this has a nice symmetry to the collector-> parser chain we have on the input side. So what are we talking about? Input side: ``` Collector ---> Parser ---> .... pipeline ``` Currently on the output side it's like this: ``` ... pipeline -> expert1 -> expert2 -> ... -> output bot 1 ``` But this basically leaves the work of transcoding/transforming the DHO format to the output bots. So far, this was enough. In the general case, it would be great to transform the DHO to some format. Let's assume CEF or IODEF. A generic output bot could then take that data and send it over whatever mechanism it knows to a destination. Example: ``` ... pipeline -> expert1 -> transform 2 IODEF -> mail bot or: ... pipeline -> expert1 -> transform 2 IODEF -> mail bot \ \-> sftp bot ``` The question in https://github.com/certtools/intelmq/issues/487 arose on where to store the output data. I'd suggest to create a new "raw_output" field in the DHO for this. A transformer bot shall write its format to the raw_output field (in base64) and an output bot shall take that data, decode the base64 andd send it via its own mechanism. What do you think? Would this work? Do you see any serious problem with this approach? Best, a.

3 2

xarf support (email sending and general mapping)
by Bernhard Reiter 24 May '16

24 May '16

Dear IntelMQ-Devs, for intelmq-mailgen I've constructed a first xarf writing support with an experimental mapping (see it at https://github.com/Intevation/intelmq-mailgen/issues/2) I haven't dived into it yet, but it seems like we need a practial mapping between xarf schemas (see https://github.com/certtools/intelmq/issues/522) So how do we construct such a mapping, so that in the end we can do xarf -> intelmq -> xarf? What intelmq field can be always expect or which only sometimes? I think you folks with more practical experience, having seen more data, are seminal for this mapping to become good. We (from Intevation) can implement the rules once we understand them well enough. :) Best, Bernhard -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

Output format to syslog/splunk (PR#503)
by Sebastian Wagner 24 May '16

24 May '16

Dear developers, contributors, users, etc. Pedro Reis (@pedromreis) opened a pull request for an UDP output bot, which can be used to send events to a syslog daemon (and then picked up by further processing software). The implementation has the following features: * Output formats are JSON or delimited by a configurable character * a optional header (at beginning of the line) can be set * `raw` field can be dropped I can see some potential problems with the 'delimited'-method here: * Strings can contain the delimiter itself, which breaks parsing. * Strings can contain arbitrary characters like \0 or \n which breaks everything Possible solutions could be: * ignore the problem as it's maybe not relevant * escape all problematic characters (solves problem with \n) * quote strings (solves problem with delimiters in strings) * strip non-printable characters * drop fields with non-printable characters * encode strings in base64 As you may have possible applications for this bot or you have experience with events in syslog, I would appreciate some feedback from you. Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

Re: [Intelmq-dev] Getting up and running CentOS
by Matthew Duncan 10 May '16

10 May '16

Thanks Sebastian, Running from master, I had to make a few modifications as CentOS is using different locations and usernames, but If I run the following: sudo -u intelmq /usr/bin/intelmqctl -t json I receive the following response null If I run the same command without the type specified, I receive the usage syntax help message usage: intelmqctl --bot [start|stop|restart|status] --id=cymru-expert intelmqctl --botnet [start|stop|restart|status] intelmqctl --list [bots|queues] ..... Other Notes: · CentOS uses “apache” in-place of www-data so I made the relevant modifications based on the install instructions. · I also has to ensure the $CONTROLLER variable as defined in the config.php matched the correct location (/usr/bin/intelmqctl instead of in local) From: Sebastian Wagner<mailto:wagner@cert.at> Sent: Tuesday, 10 May 2016 7:49 PM To: Matthew Duncan<mailto:matthew.duncan@mahdtech.com>; intelmq-dev(a)lists.cert.at<mailto:intelmq-dev@lists.cert.at> Subject: Re: [Intelmq-dev] Getting up and running CentOS Hi Matt, On 05/10/2016 11:43 AM, Matthew Duncan wrote: > I can successfully load the main page and “Configuration” page, > however if I click on either the “Management” or “Monitor” tabs, this > is what I get > > > > / / > > /Error loading botnet status: SyntaxError: JSON.parse: unexpected end > of data at line 1 column 1 of the JSON data/ > > / / > > > > Has anyone see this before or can provide some pointers where I have > gone wrong. > I assume you are running on current git master? The manager most probably can't access the cli-program. Possible reasons are insufficient permissions or misconfiguration or missing docs... To debug it, you can try to access the cli as user www-data (or whatever user your webserver is using): % sudo -u intelmq /usr/local/bin/intelmqctl Append `-t json` to see what the manager would see. Hope this helps, Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

Getting up and running CentOS
by Matthew Duncan 10 May '16

10 May '16

Hi there, You will have to excuse if this is really simple, I have just started with IntelMQ and have read the docs but can’t seem to find a fix. I am currently attempting to get IntelMQ and IntelMQ-Manager up and running on CentOS 7 and have stumbled into a road block. I can successfully start/stop and view bot configuration from the cli using intelmqctl however I am receiving an error using the Web UI. I can successfully load the main page and “Configuration” page, however if I click on either the “Management” or “Monitor” tabs, this is what I get Error loading botnet status: SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data Has anyone see this before or can provide some pointers where I have gone wrong. Thanks, Matt.

2 1

Possible License Conflict for ASN-Lookup Bot
by Dustin Demuth 02 May '16

02 May '16

Dear all, whilst packaging we've had a look at the licenses. Most things are ok, but there might be a problem with the asn_lookup expert. It makes use of pyasn (MIT-License) which builds on other products with different lisenses. One of them (mrtd) uses a 4-Clause BSD license which might be incompatible to AGPL. Do you know of alternatives to pyasn? Best Regards Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

4 5

Conf-Call 26.04.16
by Dustin Demuth 26 Apr '16

26 Apr '16

Hey Folks, This are the minutes of today's call. Participants: Sebastian, Aaron, Dustin Topics: _ Pull-Request ContactDB https://github.com/Intevation/intelmq/pull/3/commits/902896a96725ff288419a9… Aaron sent a PR for the Contact DB The Handles are URIs to the API of first.org or TI. * For TI the public service should be used as an example. * In Ripe an ORG-Handle exists Roles: New Table is supposed to model tech-c, abuse-c, etc. AS: ripe_aut_num refers to the the number in RIPE Map Organization to ASN instead of Contacts, bc. this mirrors the RIPE-Data more closely -> Problem: notification intervall per contact is not possible anymore. CONCEPT: Fill a Shadow-Database from ftp://ftp.ripe.net/ripe/dbase/ See tables in split directory _ Contact-DB Classification - This is similar to the incidents' classification in eventDB - Classification will need a little overhaul to reflect the taxonomy _ Newsletter * Sent Newsletter on Friday * Review on Thursday _ Merges from Sebastian * Redis Output Bot was successfully Merged * Several Bugfixes * New Bitsight collector _ Debian-Packages * Use HTTP Apt-Repository from intevation to provide the packages. _ XMPP - Bot * Intevation needs to do some more testing and a quick internal review. But the bot should already work. _ PyASN vs. Other Technologies * See upcoming Blog post _ Linux Standards * Logging: Syslog is currenntly in test * Configuration: runtime.d, bots.d * To think about: start intelmq with systemd in intelmq2 _ "Coding Desaster Contingency Plan" * Intevation and Cert.at will create backups of github repositories on a daily basis. BR Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

reminder: intelmq-dev conf call today
by L. Aaron Kaplan 26 Apr '16

26 Apr '16

Dear list members, this is a short reminder: last time we skipped the intelmq-dev conf call since we had the physical meeting in Vienna anyway. This week / this Tuesday we'd like to resume the regular calls. When? Today, 26th of April 2016, 16:00 UTC+2 Dial in-number: please ask me if you plan to attend. (not posting it since this is a public list with a public archive). Best, a.

1 0

Deb Packages for Ubuntu
by Dustin Demuth 25 Apr '16

25 Apr '16

Hey Folks, Sascha has been sucessful in creating *.deb Packages for intelmq and intelmq-manager. They work on Ubuntu 14.04. (with a little bit of tweaking) If you are interested in packages, have a look at: https://github.com/Intevation/intelmq/tree/deb-packaging and https://github.com/Intevation/intelmq-manager We needed to downgrade[1] a bunch of packages in intelmqs setup.py. ** Some testing will be required. ** Some packages are _not_ available in Ubuntu 14.04 but they can be installed from 15.10: - python3-redis_2.10.3-3ubuntu1 - python3-tz_2014.10dfsg1-0ubuntu2 - python3-termstyle_0.1.10-1 - python3-unicodecsv_0.13.0-2 The package intelmq-manager depends on intelmq. The dependency on sudo still exists. Packages do not follow the Debian packaging guidelines. Which must be improved. [1] https://github.com/Intevation/intelmq/commit/81b6af9ef0d08c4c6726e4c4feeaa8… BR Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

3 5

input for our intelmq newsletter April 2016
by L. Aaron Kaplan 20 Apr '16

20 Apr '16

Hi folks, Dustin and me are starting to collect input for our monthly intelmq newsletter. Please be so kind and send us your input. We write down as much as we know about of course, but I'd be helpful to still get your input upfront. Otherwise it might not be in the newletter. Best, a.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev