IntelMQ-dev August 2016

intelmq-dev@lists.cert.at

5 participants
11 discussions

Intelmq-dev newletter 201605
by L. Aaron Kaplan 07 Aug '17

07 Aug '17

= Intelmq-dev-news 05-2016 Issue 5/2016 == Topics == # Summary of IHAP meeting in April # Status update Intevation # Status update CERT.at # Status update misc == May 2016 == Dear Intelmq-dev mailing list readers, this is the second issue of intelmq developer news. We hope it's useful. TL;DR and important changes ----------------------------- The syntax of intelmqcli was changed to a new format: intelmqctl {start,stop} bot_id. This breaks compatibilty with existing scripts. If you put intelmqctl into some script, please adapt it. Also please be sure to check out the latest version of the intelmq-manager in case you use it. Lots of open issues. Progress with intelmqcli (to connect postgresql to the RT ticket system). / TL;DR === How to contribute to this newsletter? === -> contact Aaron, Dustin for future input === Summary of IHAP meeting in April === In April the IHAP Meeting took place in Vienna. * A Hacksession the night before the meeting was used by Raphael and Aaron in order to bridge MISP and IntelMQ. * Connections between Abusehelper and IntelMQ are on some CERTs wish list. XMPP is a good start. Unfortunately the XMPP Bot upstream was not fit for production. === Status report Intevation === * Still working on the KontaktDB, we appreciate the discussions that started on IHAP Meeting. We received a Pull Request from Cert.at and are currently reviewing it. * We have Scripts to import Data into the KontaktDB. Nevertheless there is some work left. * Demonstrated installation from packages on Ubuntu 14.04 on IHAP-Meeting. We propose to host the **signed** packages on our public apt-repositories. * Working on a tool similar to intelmqcli, intended to process events from the eventdb. Instead of using RT they are sent by e-mail. The tool has the working title "event-processor" and can be found here (https://github.com/Intevation/event-processor) * We did not start with support for IODEF or X-ARF yet. === Status report CERT.at developments === * we moved to python3 only. Intelmq dropped python2 support (https://github.com/certtools/intelmq/commit/2cbb42f1458a7e90539a443ec5e50ee…). This does not apply yet to the certat repo (github.com/certat/intelmq), which still supports python2.7 but only for the intelmqcli tool. * New active contributor: pedro m. reis! Welcome and thanks for working so hard on the Bitsight collector (https://github.com/certtools/intelmq/pull/493) * intelmqcli tool now supports a lot of new flags: https://github.com/certat/intelmq/issues/52 This was necessary for CERT.at since we use intelmqcli via cron job to connect to the (postgresql) eventDB , pull out all of the new data and use RT (ticket system) to send stuff out. Added flags --quiet --batch. Now intelmqcli sends via cronjob. These flags now allow CERT.at to run intelmq in full auto-mode! intelmqcli is started via cron and sends out all events to all ISPs. === Requests === * Intevation searches for testers for the packages. * We'd like to have some nice graphs in the intelmq-manager: events/sec , parse-failures/sec, etc. * implementation of whitelisting of events (filter out events based on whitelists). See https://github.com/certtools/intelmq/issues/426 * A good CSS design for the web page === Community === * RIPE abuse-c contacts can be done locally. RIPE might be able to export abuse-c infos publicly (fingers crossed). * more command line options for intelmqcli (see the https://github.com/certat/intelmq repo) * Aaron gave a presentation at the ENISA workshop "CSIRTs in Europe", 11th of May. Slides will be shared on the ENISA page. ==== intelmq.org ==== The website intelmq.org is now online, but we would like to have more content and a proper design. Do you want to contribute to intelmq, but you are not a programmer? This is your chance! Current ToDos: * Create Website Content: How-Tos / Installation Instructions, Success Stories ** How-Tos / Instructions: If you are using a special feature of IntelMQ, for instance an expert bot, try to find some time to write down a short article how you managed to get it to work and why you are using it. * Website Design == Wishlist == * **we need more test-cases!!!** * a specific config logic for ASNs: do this and that (for example sett ttl = 1 month) if event is in ASN xyz. Or "ignore" if event is in ASN xyz. This should support some kind of more-specific-less-specific inheritance, similarly to Apache directory settings. The most specific setting wins. The order could be: country code -> ASN -> netblock -> ip (/32). Open questions: what's more relevant if both domains and numbers (ASN, IPs, net blocks) exist in an event? * block based processing: for example block based team cymru lookups * parallelisation: We need to revisit this topic == Important Discussions == In case you missed something, here are the headlines of some discussion we consider interesting / important. === Mailing Lists === * [Intelmq-dev] Packaging Strategy for Bots with dependencies * [Intelmq-dev] Discussion on intelmq output / transformation architecture * [Intelmq-dev] Output format to syslog/splunk (PR#503) == Communication == Chat: irc #intelmq on freenode or webchat: [[https://webchat.freenode.net/?channels=intelmq]] Follow on twitter: @intelmqorg Weekly Conference Call every Tuesday: Dial in via the known conference bridge number. It is [[https://en.wikipedia.org/wiki/Telephone_number_mapping|ENUM]] enabled. Ask Aaron or Dustin for the number if you want to participate.

5 7

modify bot: Do you use many custom rules?
by Bernhard Reiter 30 Aug '16

30 Aug '16

Hi IntelMQ-Users, there is a discussion to improve the modify bot at [1]. As it stand we will change the configuration syntax. So how many custom rules for this bot do you have in use? Let us know so we can estimate how much support for the old configuration syntax is necessary. Best Regards, Bernhard [1] https://github.com/certtools/intelmq/issues/647 -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

Re: [Intelmq-dev] Intelmq-dev Digest, Vol 6, Issue 7
by ben dosso 12 Aug '16

12 Aug '16

hello I'm still stuck on intelmq > From: intelmq-dev-request(a)lists.cert.at > Subject: Intelmq-dev Digest, Vol 6, Issue 7 > To: intelmq-dev(a)lists.cert.at > Date: Thu, 11 Aug 2016 12:00:01 +0200 > > Send Intelmq-dev mailing list submissions to > intelmq-dev(a)lists.cert.at > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev > or, via email, send a message with subject or body 'help' to > intelmq-dev-request(a)lists.cert.at > > You can reach the person managing the list at > intelmq-dev-owner(a)lists.cert.at > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Intelmq-dev digest..." > > > Today's Topics: > > 1. Re: 2 error (Bernhard Reiter) > 2. Re: 2 error (Bernhard Reiter) > 3. Re: 2 error (Bernhard Reiter) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 10 Aug 2016 15:39:42 +0200 > From: Bernhard Reiter <bernhard(a)intevation.de> > To: intelmq-dev(a)lists.cert.at > Cc: ben dosso <dbm93(a)live.fr> > Subject: Re: [Intelmq-dev] 2 error > Message-ID: <201608101539.42787.bernhard(a)intevation.de> > Content-Type: text/plain; charset="utf-8" > > Hi Ben, > > thanks for trying intelmq! > > Am Mittwoch, 10. August 2016 15:03:28 schrieb ben dosso: > > first error > > I installed Ubuntu on intelmq 16 application does not pass. > > Which version of intelmq did you install precisely? > Which documentation were you following? > (Or which steps did you take?) > > > I am sending > > you the error message with an attachment and when I am the way opt / > > intelmq intelmq the file can not be opened. I was told not to display the > > content and also you do not have the required permissions to view the > > contents of "intelmq". > > BTW: It would be nice if you could send the error message in a text format > (if you can copy and paste the text out of that window). > > It seems that the file in question is malformed (or does not have the > necessary contents), you should inspect the file. > (Which you will be able to do as root user.) > > Best Regards, > Bernhard > > -- > www.intevation.de/~bernhard +49 541 33 508 3-3 > Intevation GmbH, Osnabr?ck, DE; Amtsgericht Osnabr?ck, HRB 18998 > Gesch?ftsf?hrer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner >

2 1

Re: [Intelmq-dev] 2 error
by Bernhard Reiter 11 Aug '16

11 Aug '16

> > > https://github.com/certtools/intelmq/blob/master/docs/User-Guide.md > > What does > > ls -l /opt/intelmq/etc > > give you? > ubuntu 16 I can not open a file intelmq The "Install" step in the User-Guide.md should have created the directory or given you other error messages. If you can, maybe you would want to see if there is a GNU/Linux person you can find that gives you a hand. Best Regards, Bernhard ps.: Please keep the mailinglist in cc, there are many more people that may be able to help you with intelmq. -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

Re: [Intelmq-dev] 2 error
by Bernhard Reiter 10 Aug '16

10 Aug '16

Am Mittwoch, 10. August 2016 16:30:42 schrieb ben dosso: > I give you the links for the application. this is the tutorial I followed > to install the application on Ubuntu 16 and 14 I also ubuntu you copy > errors ubuntu 16 connections : > https://github.com/certtools/intelmq/blob/master/docs/User-Guide.mderror : > localhost indique:Failed to obtain JSON:http://localhost/php > load_configs.php ?file=bots with error: Parsererror SyntaxERROR:unexpected > token < in JSON at position 0 the error was made bold What does ls -l /opt/intelmq/etc give you? -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

2 error
by ben dosso 10 Aug '16

10 Aug '16

first error I installed Ubuntu on intelmq 16 application does not pass. I am sending you the error message with an attachment and when I am the way opt / intelmq intelmq the file can not be opened. I was told not to display the content and also you do not have the required permissions to view the contents of "intelmq". second error I installed intelmq on a virtual machine ubuntu 14 but this time I encounter this error

2 1

I have a problem about using Application intelmq
by ben dosso 10 Aug '16

10 Aug '16

1 0

Re: [Intelmq-dev] Intelmq-dev Digest, Vol 6, Issue 2
by ben dosso 10 Aug '16

10 Aug '16

I have a problem about using Application intelmq > From: intelmq-dev-request(a)lists.cert.at > Subject: Intelmq-dev Digest, Vol 6, Issue 2 > To: intelmq-dev(a)lists.cert.at > Date: Mon, 8 Aug 2016 12:00:02 +0200 > > Send Intelmq-dev mailing list submissions to > intelmq-dev(a)lists.cert.at > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev > or, via email, send a message with subject or body 'help' to > intelmq-dev-request(a)lists.cert.at > > You can reach the person managing the list at > intelmq-dev-owner(a)lists.cert.at > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Intelmq-dev digest..." > > > Today's Topics: > > 1. Re: Taxonomies & Sharing mechanism [SEC=UNCLASSIFIED] > (Clark, Andrew) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 8 Aug 2016 05:55:55 +0000 > From: "Clark, Andrew" <Andrew.Clark(a)cert.gov.au> > To: Otmar Lendl <lendl(a)cert.at>, "intelmq-dev(a)lists.cert.at" > <intelmq-dev(a)lists.cert.at> > Subject: Re: [Intelmq-dev] Taxonomies & Sharing mechanism > [SEC=UNCLASSIFIED] > Message-ID: > <454F4A633809FD40898A9D230EDFA93138B2E82B(a)ACTDC01MLD02V.agdnet.ag.gov.au> > > Content-Type: text/plain; charset="utf-8" > > UNCLASSIFIED > Hi Otmar, > > I have never really investigated what's out there in terms of taxonomies, to any great extent. > > We use MISP, and if you haven't seen it, take a look at how many taxonomies they've tried to accommodate: https://github.com/MISP/misp-taxonomies/ > > If I'm reading the correct things, I suspect we might be lucky because the CERT.pt taxonomy looks very similar to the eCSIRT taxonomy used by IntelMQ (and supported by MISP). > > The CERT.pt taxonomy (from this site: http://www.cncs.gov.pt/cert-pt-2/documents-2/) includes 18 "incident types" and 10 "incident classes". The ClassificationType class from IntelMQ supports 20 values, including the 18 from the CERT.pt taxonomy, plus "unknown" and "blocklist". Based on this, I don't think there is a good reason to change what IntelMQ uses now. > > Regarding STIX and Cybox (and TAXII), here at CERT Australia we are using them heavily. STIX includes a 'TTP' object which can be associated with Indicators. TTPs include 'behaviours' and while STIX supports the CAPEC (capec.mitre.org) taxonomy natively, it would be easy to extend to support arbitrary taxonomies. > > Hope you're enjoying your vacation! > > Andrew > > -----Original Message----- > From: Intelmq-dev [mailto:intelmq-dev-bounces@lists.cert.at] On Behalf Of Otmar Lendl > Sent: Saturday, 6 August 2016 2:07 AM > To: intelmq-dev(a)lists.cert.at > Subject: [Intelmq-dev] Taxonomies & Sharing mechanism > > > Folks, > > as I will attending the ENISA/EC3 workshop in The Hague this autumn, I got an invitation to a preparatory survey which asks questions about a consensus regarding taxonomies and information sharing formats to be used in CERT/CERT and CERT/LE information sharing. > > IntelMQ is based on eCSIRT II, which some working-group in the ENISA/EC3/EMPACT universe has declared to be obsolete. > > See this monster of a report: > https://www.enisa.europa.eu/publications/information-sharing-and-common-tax… > > Their new shiny pony is based on the work of CERT.pt, and they want to to use the meeting this year to finalize that decision. I have no clue how big the delta to eCSIRT II is. > > IMHO the IntelMQ community has to decide how to react. E.g. > > a) stay with eCSIRT II framework > b) adopt the new one > > and > > what stance to take on an inter-organisational sharing mechanism. > > So what do you all think? > > otmar (who will be on vacation the next weeks, don't expect me to reply > soon) > > ------------------ > > The survey asks: > > Do you believe that the Common Taxonomy for the national network of CSIRT/LEA (formerly known as CERT.PT Taxonomy) is suitable for CSIRT/LEA communication? > > Yes / No / Other > > Have you ever used one of the following? > > STIX / CybOX / Other sharing Mechanism > > What do you think could be a suitable sharing mechanism for the Common Taxonomy for the national network of CSIRT/LEA? > > STIX / CybOX / Other sharing Mechanism > > Extract from 'Report on Information Sharing and Common Taxonomies between CSIRTs and Law Enforcement Agencies' > > A clear distinction should be made between a taxonomy, a sharing mechanism and a sharing platform to avoid any possible confusion. While a taxonomy is a way of describing information through classification, a sharing mechanism structures the way the information is encoded. For example, a sharing mechanism might provide rules for names and positions of XML tags to allow a file to be treated automatically. Finally, a sharing platform is a tool allowing to share information. It is not mandatory to have such a platform ? files containing information structured according to a standard and classified according to a taxonomy could simply be sent by e-mail, for example. Nevertheless, the use of a sharing platform allows users to easily share information in a structured way. > > > -- > // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg > > > ---------------------------------------------------- > If you have received this transmission in error please > notify us immediately by return e-mail and delete all > copies. If this e-mail or any attachments have been sent > to you in error, that error does not constitute waiver > of any confidentiality, privilege or copyright in respect > of information in the e-mail or attachments. > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Intelmq-dev mailing list > Intelmq-dev(a)lists.cert.at > http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev > > > ------------------------------ > > End of Intelmq-dev Digest, Vol 6, Issue 2 > *****************************************

1 0

heads up - breaking changes to harmonization.conf
by L. Aaron Kaplan 09 Aug '16

09 Aug '16

Hi, Pull request #634 (https://github.com/certtools/intelmq/pull/634 ) will include some changes to the harmonisation.conf file. There was a previous discussion that some of the values (varchar(2000) in most cases) were arbitrary and sometimes even not long enough. Sebix changed a lot to the postgresql "text" type and in the course of doing so, cleaned up the regular expressions and definitions of fields in harmonisation.conf. Please have a thorough look at PR #634 before I merge it in. If there are any concerns, please let us know by the end of this week if possible. Thanks! a.

1 0

Re: [Intelmq-dev] Taxonomies & Sharing mechanism [SEC=UNCLASSIFIED]
by Clark, Andrew 08 Aug '16

08 Aug '16

UNCLASSIFIED Hi Otmar, I have never really investigated what's out there in terms of taxonomies, to any great extent. We use MISP, and if you haven't seen it, take a look at how many taxonomies they've tried to accommodate: https://github.com/MISP/misp-taxonomies/ If I'm reading the correct things, I suspect we might be lucky because the CERT.pt taxonomy looks very similar to the eCSIRT taxonomy used by IntelMQ (and supported by MISP). The CERT.pt taxonomy (from this site: http://www.cncs.gov.pt/cert-pt-2/documents-2/) includes 18 "incident types" and 10 "incident classes". The ClassificationType class from IntelMQ supports 20 values, including the 18 from the CERT.pt taxonomy, plus "unknown" and "blocklist". Based on this, I don't think there is a good reason to change what IntelMQ uses now. Regarding STIX and Cybox (and TAXII), here at CERT Australia we are using them heavily. STIX includes a 'TTP' object which can be associated with Indicators. TTPs include 'behaviours' and while STIX supports the CAPEC (capec.mitre.org) taxonomy natively, it would be easy to extend to support arbitrary taxonomies. Hope you're enjoying your vacation! Andrew -----Original Message----- From: Intelmq-dev [mailto:intelmq-dev-bounces@lists.cert.at] On Behalf Of Otmar Lendl Sent: Saturday, 6 August 2016 2:07 AM To: intelmq-dev(a)lists.cert.at Subject: [Intelmq-dev] Taxonomies & Sharing mechanism Folks, as I will attending the ENISA/EC3 workshop in The Hague this autumn, I got an invitation to a preparatory survey which asks questions about a consensus regarding taxonomies and information sharing formats to be used in CERT/CERT and CERT/LE information sharing. IntelMQ is based on eCSIRT II, which some working-group in the ENISA/EC3/EMPACT universe has declared to be obsolete. See this monster of a report: https://www.enisa.europa.eu/publications/information-sharing-and-common-tax… Their new shiny pony is based on the work of CERT.pt, and they want to to use the meeting this year to finalize that decision. I have no clue how big the delta to eCSIRT II is. IMHO the IntelMQ community has to decide how to react. E.g. a) stay with eCSIRT II framework b) adopt the new one and what stance to take on an inter-organisational sharing mechanism. So what do you all think? otmar (who will be on vacation the next weeks, don't expect me to reply soon) ------------------ The survey asks: Do you believe that the Common Taxonomy for the national network of CSIRT/LEA (formerly known as CERT.PT Taxonomy) is suitable for CSIRT/LEA communication? Yes / No / Other Have you ever used one of the following? STIX / CybOX / Other sharing Mechanism What do you think could be a suitable sharing mechanism for the Common Taxonomy for the national network of CSIRT/LEA? STIX / CybOX / Other sharing Mechanism Extract from 'Report on Information Sharing and Common Taxonomies between CSIRTs and Law Enforcement Agencies' A clear distinction should be made between a taxonomy, a sharing mechanism and a sharing platform to avoid any possible confusion. While a taxonomy is a way of describing information through classification, a sharing mechanism structures the way the information is encoded. For example, a sharing mechanism might provide rules for names and positions of XML tags to allow a file to be treated automatically. Finally, a sharing platform is a tool allowing to share information. It is not mandatory to have such a platform – files containing information structured according to a standard and classified according to a taxonomy could simply be sent by e-mail, for example. Nevertheless, the use of a sharing platform allows users to easily share information in a structured way. -- // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg ---------------------------------------------------- If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev August 2016