IntelMQ-dev July 2016

intelmq-dev@lists.cert.at

6 participants
9 discussions

Intelmq-dev newletter 201605
by L. Aaron Kaplan 07 Aug '17

07 Aug '17

= Intelmq-dev-news 05-2016 Issue 5/2016 == Topics == # Summary of IHAP meeting in April # Status update Intevation # Status update CERT.at # Status update misc == May 2016 == Dear Intelmq-dev mailing list readers, this is the second issue of intelmq developer news. We hope it's useful. TL;DR and important changes ----------------------------- The syntax of intelmqcli was changed to a new format: intelmqctl {start,stop} bot_id. This breaks compatibilty with existing scripts. If you put intelmqctl into some script, please adapt it. Also please be sure to check out the latest version of the intelmq-manager in case you use it. Lots of open issues. Progress with intelmqcli (to connect postgresql to the RT ticket system). / TL;DR === How to contribute to this newsletter? === -> contact Aaron, Dustin for future input === Summary of IHAP meeting in April === In April the IHAP Meeting took place in Vienna. * A Hacksession the night before the meeting was used by Raphael and Aaron in order to bridge MISP and IntelMQ. * Connections between Abusehelper and IntelMQ are on some CERTs wish list. XMPP is a good start. Unfortunately the XMPP Bot upstream was not fit for production. === Status report Intevation === * Still working on the KontaktDB, we appreciate the discussions that started on IHAP Meeting. We received a Pull Request from Cert.at and are currently reviewing it. * We have Scripts to import Data into the KontaktDB. Nevertheless there is some work left. * Demonstrated installation from packages on Ubuntu 14.04 on IHAP-Meeting. We propose to host the **signed** packages on our public apt-repositories. * Working on a tool similar to intelmqcli, intended to process events from the eventdb. Instead of using RT they are sent by e-mail. The tool has the working title "event-processor" and can be found here (https://github.com/Intevation/event-processor) * We did not start with support for IODEF or X-ARF yet. === Status report CERT.at developments === * we moved to python3 only. Intelmq dropped python2 support (https://github.com/certtools/intelmq/commit/2cbb42f1458a7e90539a443ec5e50ee…). This does not apply yet to the certat repo (github.com/certat/intelmq), which still supports python2.7 but only for the intelmqcli tool. * New active contributor: pedro m. reis! Welcome and thanks for working so hard on the Bitsight collector (https://github.com/certtools/intelmq/pull/493) * intelmqcli tool now supports a lot of new flags: https://github.com/certat/intelmq/issues/52 This was necessary for CERT.at since we use intelmqcli via cron job to connect to the (postgresql) eventDB , pull out all of the new data and use RT (ticket system) to send stuff out. Added flags --quiet --batch. Now intelmqcli sends via cronjob. These flags now allow CERT.at to run intelmq in full auto-mode! intelmqcli is started via cron and sends out all events to all ISPs. === Requests === * Intevation searches for testers for the packages. * We'd like to have some nice graphs in the intelmq-manager: events/sec , parse-failures/sec, etc. * implementation of whitelisting of events (filter out events based on whitelists). See https://github.com/certtools/intelmq/issues/426 * A good CSS design for the web page === Community === * RIPE abuse-c contacts can be done locally. RIPE might be able to export abuse-c infos publicly (fingers crossed). * more command line options for intelmqcli (see the https://github.com/certat/intelmq repo) * Aaron gave a presentation at the ENISA workshop "CSIRTs in Europe", 11th of May. Slides will be shared on the ENISA page. ==== intelmq.org ==== The website intelmq.org is now online, but we would like to have more content and a proper design. Do you want to contribute to intelmq, but you are not a programmer? This is your chance! Current ToDos: * Create Website Content: How-Tos / Installation Instructions, Success Stories ** How-Tos / Instructions: If you are using a special feature of IntelMQ, for instance an expert bot, try to find some time to write down a short article how you managed to get it to work and why you are using it. * Website Design == Wishlist == * **we need more test-cases!!!** * a specific config logic for ASNs: do this and that (for example sett ttl = 1 month) if event is in ASN xyz. Or "ignore" if event is in ASN xyz. This should support some kind of more-specific-less-specific inheritance, similarly to Apache directory settings. The most specific setting wins. The order could be: country code -> ASN -> netblock -> ip (/32). Open questions: what's more relevant if both domains and numbers (ASN, IPs, net blocks) exist in an event? * block based processing: for example block based team cymru lookups * parallelisation: We need to revisit this topic == Important Discussions == In case you missed something, here are the headlines of some discussion we consider interesting / important. === Mailing Lists === * [Intelmq-dev] Packaging Strategy for Bots with dependencies * [Intelmq-dev] Discussion on intelmq output / transformation architecture * [Intelmq-dev] Output format to syslog/splunk (PR#503) == Communication == Chat: irc #intelmq on freenode or webchat: [[https://webchat.freenode.net/?channels=intelmq]] Follow on twitter: @intelmqorg Weekly Conference Call every Tuesday: Dial in via the known conference bridge number. It is [[https://en.wikipedia.org/wiki/Telephone_number_mapping|ENUM]] enabled. Ask Aaron or Dustin for the number if you want to participate.

5 7

Reports larger than 500 MB in IntelMQ
by Dustin Demuth 29 Jul '16

29 Jul '16

Dear all, currently we are facing the problem, that IntelMQ is not capable of handling large reports ( > 500 MB) when using Redis as a Message Queuing System. First we thought this might get fixed in most recent redis versions (see: [1]), but apparently this is not the case. Are you aware of setups of IntelMQ which use a different backend, like ZMQ? Do you know of performance or other issues which occurred when using other Queuing Systems? [1] https://github.com/certtools/intelmq/issues/547 Best Regards Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

4 9

another naming update
by L. Aaron Kaplan 29 Jul '16

29 Jul '16

Hi *, if you are using the shadowserver parsers , especially the mDNS parser, please be aware that the feed name changed. "Open-m DNS" to "Open-mDNS" See: https://github.com/certtools/intelmq/issues/624 Please adapt your runtime.conf accordingly. Best, a.

1 0

Changes in harmonization implementation
by Sebastian Wagner 29 Jul '16

29 Jul '16

Dear contributors and beta-testers, We recently changed did some changes in the implementation of the harmonization. The type MalwareName has been removed, it was just a duplicate of String. This may break your existing installation if you do not update your harmonization.conf with upstream. It has been used as type for malware.name The type LowercaseString has been introduced, doing a conversion to lower case chars automatically. Applied to protocol names, emails, hashes and UUIDs etc. FQDNs are now lower case only too. Some restrictive regexes have been made more tolerant (e.g. malware names) misp_uuid is now misp.event_uuid misp.attribute_uuid introduced new iregex-type check available (case-insenstitive regex-check) allowed values for protocol.transport is a defined list Also: Currently, both ASCII and IDN-domains are allowed, this will be changed too (including automatic conversion): https://github.com/certtools/intelmq/issues/622 When pulling from our upstream-repository, make sure you also update your harmonization.conf Related commits: https://github.com/certtools/intelmq/commit/6624bdda04ca0595799150e97e719a5… https://github.com/certtools/intelmq/commit/640e1a771acb4ddb9cbd726e51ae903… https://github.com/certtools/intelmq/commit/fd81f0fd9c86ffa0ac3764adafc74e7… https://github.com/certtools/intelmq/commit/7dbed4bc8f9200d06dae2018850341a… https://github.com/certtools/intelmq/commit/6b8b759293069afc3d9f0f1cdf8e4ea… https://github.com/certtools/intelmq/commit/d7892bcb59ac54218c5c45630226d4a… https://github.com/certtools/intelmq/commit/6584ab72fa408732c02abc1c934637a… https://github.com/certtools/intelmq/commit/a742c034c80249e80d2f280fd0dc8a7… Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

tomorrow, Tuesday intelmq-dev conf call
by L. Aaron Kaplan 25 Jul '16

25 Jul '16

Hi, We'll be having our regular intelmq-dev call tomorrow at 9:00 a.m. CEST. This terribly early date was chosen so that the colleagues from Australia can join the call as well. If you want to participate in the call, please let me know, I will send you call in details. Best, a.

1 0

Re: [Intelmq-dev] today 16:00 UTC+2 conf call [SEC=UNCLASSIFIED]
by Clark, Andrew 11 Jul '16

11 Jul '16

UNCLASSIFIED Hi Aaron, Thanks for thinking of us down here in Australia, and sorry for the slow response - I was on leave last week. Early morning your time is going to suit us best - we can be a bit flexible in the evenings, it it's say 1630 or 1700 here (UTC+10) that is fine, or if that's too early for you we could try after 1900 and I can dial in from home. Andrew -- Andrew Clark | Senior Technical Advisor | CERT Australia Attorney-General's Department, Australian Government Phone: +61 2 6141 2538 Online: www.cert.gov.au For all CERT Australia operational matters, please call our hotline: 1300 172 499, or +61 26141 2999 or email: info(a)cert.gov.au -----Original Message----- From: Intelmq-dev [mailto:intelmq-dev-bounces@lists.cert.at] On Behalf Of L. Aaron Kaplan Sent: Tuesday, 5 July 2016 7:38 PM To: intelmq-dev(a)lists.cert.at Subject: [Intelmq-dev] today 16:00 UTC+2 conf call Hi *, today at 16:00 UTC+2 we'll have our conf call again (and resume it after some weeks of FIRST conference and post-FIRST catching up). Topics for this call: * generic syntax for transformations/generic mapping language * status update Intevation * status update CERT.at * ... your status update? * any other input for the monthly newsletter * biggest concerns/topics right now * discussion action items for next week * AoB Please feel free to suggest topics for this call by replying to this mail thread. Thanks, a. PS: I'd like to suggest to move this weekly conf call to an earlier time in the future since this would allow our Australian colleagues to join as well. When would be a good time for the Aussies? -- // CERT Austria // L. Aaron Kaplan <kaplan(a)cert.at> // T: +43 1 505 64 16 78 // http://www.cert.at // Eine Initiative der NIC.at Internet Verwaltungs- und Betriebs GmbH // http://www.nic.at/ - Firmenbuchnummer 172568b, LG Salzburg ---------------------------------------------------- If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

1 0

Re: [Intelmq-dev] Packaging Strategy for Bots with dependencies [SEC=UNCLASSIFIED]
by Clark, Andrew 11 Jul '16

11 Jul '16

UNCLASSIFIED Hi Bernhard, all, Sorry to come into this a bit late ... my comments are below: > Am Mittwoch, 20. April 2016 11:55:59 schrieb L. Aaron Kaplan: > > > Aren't we running into the same issue as with the BOTS file? If > > > > The BOTS file is simply a template of which bots exist in general. > > It is mainly used for the intelmq-manager to display which ones > > could get selected. And which parameters they have. > > In order to get this moving, I propose to tackle it in small steps. > > What about implementing a solution for the contents of BOTS first? > We keep the logic of etc/defaults.conf and etc/runtime.conf for this step. > So no hiearchical configuration (for now) and then we take it from there. If I understand correctly the IntelMQ 'runtime' configuration is currently spread across three configuration files (runtime.conf, pipeline.conf and startup.conf), and there are other configuration files as well (e.g. defaults.conf, harmonization.conf and system.conf). I think it's a great idea to consider how to improve on the current monolithic BOTS template configuration, but also wonder if some of the configuration that is currently in separate files could be merged. For example, could startup and runtime be merged? Maybe even pipeline too? > Thinking about how to solve the BOTS file first: > a) (proposed by Dusting) bots.d/ directory solution, > means adding and removing filenames. > b) adding an initialisation function to each module which > returns the necessary information > c) write a small packaging helper that adds/removes sections > from BOTS > > Some thoughts (being still new to the code base): > b) seems attractive to me, because it keeps information > close to where it is maintainted. Its main drawback probably is > that is the largest change. Are there other drawbacks? > c) has the drawback of requiring running code during (de)installation. > And it does not add much over a). I have a STRONG preference for an approach similar to 'a' above. In my opinion we should be aiming to decouple the configuration from the bot code itself as much as possible, and encouraging more generic bots (i.e., less bots that are more generic (and more configurable), and more configuration files. In general each bot should potentially accept many configurations. This approach should serve to limit the explosion in the number of bots (I think the parser bots are a good example of where we could reduce the number of bots focusing on 'what' they are parsing, rather than their source). Along the lines of option 'a' above I think the 'modules-available' and 'modules-enabled' approach used by the Apache web server could work for us too. All available configurations (templates, etc.) could be stored in a 'etc/bots-available' location, and the copied/edited into the 'etc/bots-enabled' location to enable them. Andrew -- Andrew Clark | Senior Technical Advisor | CERT Australia Attorney-General's Department, Australian Government Phone: +61 2 6141 2538 Online: www.cert.gov.au For all CERT Australia operational matters, please call our hotline: 1300 172 499, or +61 26141 2999 or email: info(a)cert.gov.au ---------------------------------------------------- If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

1 0

Packaging Strategy for Bots with dependencies
by Dustin Demuth 07 Jul '16

07 Jul '16

Hey Folks, Whilst checking the dependencies, I've found out that a lot of Bots have their own requirements-file. From a packagers point of view, I think this is hard to maintain. I think, we should discuss if it is reasonable to split the software into a CORE package, were all packages have the same dependencies, and several BOTS packages with their own dependencies. This would create the additional need to rethink the BOTS json-file. It would be possible to have all JSON configs for the Bots in a bots.d directory and search for them in this directory. I know, that this might create additional complexity in some points. BR Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

8 15

today 16:00 UTC+2 conf call
by L. Aaron Kaplan 05 Jul '16

05 Jul '16

Hi *, today at 16:00 UTC+2 we'll have our conf call again (and resume it after some weeks of FIRST conference and post-FIRST catching up). Topics for this call: * generic syntax for transformations/generic mapping language * status update Intevation * status update CERT.at * ... your status update? * any other input for the monthly newsletter * biggest concerns/topics right now * discussion action items for next week * AoB Please feel free to suggest topics for this call by replying to this mail thread. Thanks, a. PS: I'd like to suggest to move this weekly conf call to an earlier time in the future since this would allow our Australian colleagues to join as well. When would be a good time for the Aussies? -- // CERT Austria // L. Aaron Kaplan <kaplan(a)cert.at> // T: +43 1 505 64 16 78 // http://www.cert.at // Eine Initiative der NIC.at Internet Verwaltungs- und Betriebs GmbH // http://www.nic.at/ - Firmenbuchnummer 172568b, LG Salzburg

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev July 2016