IntelMQ-dev

intelmq-dev@lists.cert.at

6 participants
232 discussions

heads up - breaking changes to harmonization.conf
by L. Aaron Kaplan 09 Aug '16

09 Aug '16

Hi, Pull request #634 (https://github.com/certtools/intelmq/pull/634 ) will include some changes to the harmonisation.conf file. There was a previous discussion that some of the values (varchar(2000) in most cases) were arbitrary and sometimes even not long enough. Sebix changed a lot to the postgresql "text" type and in the course of doing so, cleaned up the regular expressions and definitions of fields in harmonisation.conf. Please have a thorough look at PR #634 before I merge it in. If there are any concerns, please let us know by the end of this week if possible. Thanks! a.

1 0

Re: [Intelmq-dev] Taxonomies & Sharing mechanism [SEC=UNCLASSIFIED]
by Clark, Andrew 08 Aug '16

08 Aug '16

UNCLASSIFIED Hi Otmar, I have never really investigated what's out there in terms of taxonomies, to any great extent. We use MISP, and if you haven't seen it, take a look at how many taxonomies they've tried to accommodate: https://github.com/MISP/misp-taxonomies/ If I'm reading the correct things, I suspect we might be lucky because the CERT.pt taxonomy looks very similar to the eCSIRT taxonomy used by IntelMQ (and supported by MISP). The CERT.pt taxonomy (from this site: http://www.cncs.gov.pt/cert-pt-2/documents-2/) includes 18 "incident types" and 10 "incident classes". The ClassificationType class from IntelMQ supports 20 values, including the 18 from the CERT.pt taxonomy, plus "unknown" and "blocklist". Based on this, I don't think there is a good reason to change what IntelMQ uses now. Regarding STIX and Cybox (and TAXII), here at CERT Australia we are using them heavily. STIX includes a 'TTP' object which can be associated with Indicators. TTPs include 'behaviours' and while STIX supports the CAPEC (capec.mitre.org) taxonomy natively, it would be easy to extend to support arbitrary taxonomies. Hope you're enjoying your vacation! Andrew -----Original Message----- From: Intelmq-dev [mailto:intelmq-dev-bounces@lists.cert.at] On Behalf Of Otmar Lendl Sent: Saturday, 6 August 2016 2:07 AM To: intelmq-dev(a)lists.cert.at Subject: [Intelmq-dev] Taxonomies & Sharing mechanism Folks, as I will attending the ENISA/EC3 workshop in The Hague this autumn, I got an invitation to a preparatory survey which asks questions about a consensus regarding taxonomies and information sharing formats to be used in CERT/CERT and CERT/LE information sharing. IntelMQ is based on eCSIRT II, which some working-group in the ENISA/EC3/EMPACT universe has declared to be obsolete. See this monster of a report: https://www.enisa.europa.eu/publications/information-sharing-and-common-tax… Their new shiny pony is based on the work of CERT.pt, and they want to to use the meeting this year to finalize that decision. I have no clue how big the delta to eCSIRT II is. IMHO the IntelMQ community has to decide how to react. E.g. a) stay with eCSIRT II framework b) adopt the new one and what stance to take on an inter-organisational sharing mechanism. So what do you all think? otmar (who will be on vacation the next weeks, don't expect me to reply soon) ------------------ The survey asks: Do you believe that the Common Taxonomy for the national network of CSIRT/LEA (formerly known as CERT.PT Taxonomy) is suitable for CSIRT/LEA communication? Yes / No / Other Have you ever used one of the following? STIX / CybOX / Other sharing Mechanism What do you think could be a suitable sharing mechanism for the Common Taxonomy for the national network of CSIRT/LEA? STIX / CybOX / Other sharing Mechanism Extract from 'Report on Information Sharing and Common Taxonomies between CSIRTs and Law Enforcement Agencies' A clear distinction should be made between a taxonomy, a sharing mechanism and a sharing platform to avoid any possible confusion. While a taxonomy is a way of describing information through classification, a sharing mechanism structures the way the information is encoded. For example, a sharing mechanism might provide rules for names and positions of XML tags to allow a file to be treated automatically. Finally, a sharing platform is a tool allowing to share information. It is not mandatory to have such a platform – files containing information structured according to a standard and classified according to a taxonomy could simply be sent by e-mail, for example. Nevertheless, the use of a sharing platform allows users to easily share information in a structured way. -- // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg ---------------------------------------------------- If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

1 0

Taxonomies & Sharing mechanism
by Otmar Lendl 05 Aug '16

05 Aug '16

Folks, as I will attending the ENISA/EC3 workshop in The Hague this autumn, I got an invitation to a preparatory survey which asks questions about a consensus regarding taxonomies and information sharing formats to be used in CERT/CERT and CERT/LE information sharing. IntelMQ is based on eCSIRT II, which some working-group in the ENISA/EC3/EMPACT universe has declared to be obsolete. See this monster of a report: https://www.enisa.europa.eu/publications/information-sharing-and-common-tax… Their new shiny pony is based on the work of CERT.pt, and they want to to use the meeting this year to finalize that decision. I have no clue how big the delta to eCSIRT II is. IMHO the IntelMQ community has to decide how to react. E.g. a) stay with eCSIRT II framework b) adopt the new one and what stance to take on an inter-organisational sharing mechanism. So what do you all think? otmar (who will be on vacation the next weeks, don't expect me to reply soon) ------------------ The survey asks: Do you believe that the Common Taxonomy for the national network of CSIRT/LEA (formerly known as CERT.PT Taxonomy) is suitable for CSIRT/LEA communication? Yes / No / Other Have you ever used one of the following? STIX / CybOX / Other sharing Mechanism What do you think could be a suitable sharing mechanism for the Common Taxonomy for the national network of CSIRT/LEA? STIX / CybOX / Other sharing Mechanism Extract from 'Report on Information Sharing and Common Taxonomies between CSIRTs and Law Enforcement Agencies' A clear distinction should be made between a taxonomy, a sharing mechanism and a sharing platform to avoid any possible confusion. While a taxonomy is a way of describing information through classification, a sharing mechanism structures the way the information is encoded. For example, a sharing mechanism might provide rules for names and positions of XML tags to allow a file to be treated automatically. Finally, a sharing platform is a tool allowing to share information. It is not mandatory to have such a platform – files containing information structured according to a standard and classified according to a taxonomy could simply be sent by e-mail, for example. Nevertheless, the use of a sharing platform allows users to easily share information in a structured way. -- // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Reports larger than 500 MB in IntelMQ
by Dustin Demuth 29 Jul '16

29 Jul '16

Dear all, currently we are facing the problem, that IntelMQ is not capable of handling large reports ( > 500 MB) when using Redis as a Message Queuing System. First we thought this might get fixed in most recent redis versions (see: [1]), but apparently this is not the case. Are you aware of setups of IntelMQ which use a different backend, like ZMQ? Do you know of performance or other issues which occurred when using other Queuing Systems? [1] https://github.com/certtools/intelmq/issues/547 Best Regards Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

4 9

another naming update
by L. Aaron Kaplan 29 Jul '16

29 Jul '16

Hi *, if you are using the shadowserver parsers , especially the mDNS parser, please be aware that the feed name changed. "Open-m DNS" to "Open-mDNS" See: https://github.com/certtools/intelmq/issues/624 Please adapt your runtime.conf accordingly. Best, a.

1 0

Changes in harmonization implementation
by Sebastian Wagner 29 Jul '16

29 Jul '16

Dear contributors and beta-testers, We recently changed did some changes in the implementation of the harmonization. The type MalwareName has been removed, it was just a duplicate of String. This may break your existing installation if you do not update your harmonization.conf with upstream. It has been used as type for malware.name The type LowercaseString has been introduced, doing a conversion to lower case chars automatically. Applied to protocol names, emails, hashes and UUIDs etc. FQDNs are now lower case only too. Some restrictive regexes have been made more tolerant (e.g. malware names) misp_uuid is now misp.event_uuid misp.attribute_uuid introduced new iregex-type check available (case-insenstitive regex-check) allowed values for protocol.transport is a defined list Also: Currently, both ASCII and IDN-domains are allowed, this will be changed too (including automatic conversion): https://github.com/certtools/intelmq/issues/622 When pulling from our upstream-repository, make sure you also update your harmonization.conf Related commits: https://github.com/certtools/intelmq/commit/6624bdda04ca0595799150e97e719a5… https://github.com/certtools/intelmq/commit/640e1a771acb4ddb9cbd726e51ae903… https://github.com/certtools/intelmq/commit/fd81f0fd9c86ffa0ac3764adafc74e7… https://github.com/certtools/intelmq/commit/7dbed4bc8f9200d06dae2018850341a… https://github.com/certtools/intelmq/commit/6b8b759293069afc3d9f0f1cdf8e4ea… https://github.com/certtools/intelmq/commit/d7892bcb59ac54218c5c45630226d4a… https://github.com/certtools/intelmq/commit/6584ab72fa408732c02abc1c934637a… https://github.com/certtools/intelmq/commit/a742c034c80249e80d2f280fd0dc8a7… Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

tomorrow, Tuesday intelmq-dev conf call
by L. Aaron Kaplan 25 Jul '16

25 Jul '16

Hi, We'll be having our regular intelmq-dev call tomorrow at 9:00 a.m. CEST. This terribly early date was chosen so that the colleagues from Australia can join the call as well. If you want to participate in the call, please let me know, I will send you call in details. Best, a.

1 0

Re: [Intelmq-dev] today 16:00 UTC+2 conf call [SEC=UNCLASSIFIED]
by Clark, Andrew 11 Jul '16

11 Jul '16

UNCLASSIFIED Hi Aaron, Thanks for thinking of us down here in Australia, and sorry for the slow response - I was on leave last week. Early morning your time is going to suit us best - we can be a bit flexible in the evenings, it it's say 1630 or 1700 here (UTC+10) that is fine, or if that's too early for you we could try after 1900 and I can dial in from home. Andrew -- Andrew Clark | Senior Technical Advisor | CERT Australia Attorney-General's Department, Australian Government Phone: +61 2 6141 2538 Online: www.cert.gov.au For all CERT Australia operational matters, please call our hotline: 1300 172 499, or +61 26141 2999 or email: info(a)cert.gov.au -----Original Message----- From: Intelmq-dev [mailto:intelmq-dev-bounces@lists.cert.at] On Behalf Of L. Aaron Kaplan Sent: Tuesday, 5 July 2016 7:38 PM To: intelmq-dev(a)lists.cert.at Subject: [Intelmq-dev] today 16:00 UTC+2 conf call Hi *, today at 16:00 UTC+2 we'll have our conf call again (and resume it after some weeks of FIRST conference and post-FIRST catching up). Topics for this call: * generic syntax for transformations/generic mapping language * status update Intevation * status update CERT.at * ... your status update? * any other input for the monthly newsletter * biggest concerns/topics right now * discussion action items for next week * AoB Please feel free to suggest topics for this call by replying to this mail thread. Thanks, a. PS: I'd like to suggest to move this weekly conf call to an earlier time in the future since this would allow our Australian colleagues to join as well. When would be a good time for the Aussies? -- // CERT Austria // L. Aaron Kaplan <kaplan(a)cert.at> // T: +43 1 505 64 16 78 // http://www.cert.at // Eine Initiative der NIC.at Internet Verwaltungs- und Betriebs GmbH // http://www.nic.at/ - Firmenbuchnummer 172568b, LG Salzburg ---------------------------------------------------- If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

1 0

Re: [Intelmq-dev] Packaging Strategy for Bots with dependencies [SEC=UNCLASSIFIED]
by Clark, Andrew 11 Jul '16

11 Jul '16

UNCLASSIFIED Hi Bernhard, all, Sorry to come into this a bit late ... my comments are below: > Am Mittwoch, 20. April 2016 11:55:59 schrieb L. Aaron Kaplan: > > > Aren't we running into the same issue as with the BOTS file? If > > > > The BOTS file is simply a template of which bots exist in general. > > It is mainly used for the intelmq-manager to display which ones > > could get selected. And which parameters they have. > > In order to get this moving, I propose to tackle it in small steps. > > What about implementing a solution for the contents of BOTS first? > We keep the logic of etc/defaults.conf and etc/runtime.conf for this step. > So no hiearchical configuration (for now) and then we take it from there. If I understand correctly the IntelMQ 'runtime' configuration is currently spread across three configuration files (runtime.conf, pipeline.conf and startup.conf), and there are other configuration files as well (e.g. defaults.conf, harmonization.conf and system.conf). I think it's a great idea to consider how to improve on the current monolithic BOTS template configuration, but also wonder if some of the configuration that is currently in separate files could be merged. For example, could startup and runtime be merged? Maybe even pipeline too? > Thinking about how to solve the BOTS file first: > a) (proposed by Dusting) bots.d/ directory solution, > means adding and removing filenames. > b) adding an initialisation function to each module which > returns the necessary information > c) write a small packaging helper that adds/removes sections > from BOTS > > Some thoughts (being still new to the code base): > b) seems attractive to me, because it keeps information > close to where it is maintainted. Its main drawback probably is > that is the largest change. Are there other drawbacks? > c) has the drawback of requiring running code during (de)installation. > And it does not add much over a). I have a STRONG preference for an approach similar to 'a' above. In my opinion we should be aiming to decouple the configuration from the bot code itself as much as possible, and encouraging more generic bots (i.e., less bots that are more generic (and more configurable), and more configuration files. In general each bot should potentially accept many configurations. This approach should serve to limit the explosion in the number of bots (I think the parser bots are a good example of where we could reduce the number of bots focusing on 'what' they are parsing, rather than their source). Along the lines of option 'a' above I think the 'modules-available' and 'modules-enabled' approach used by the Apache web server could work for us too. All available configurations (templates, etc.) could be stored in a 'etc/bots-available' location, and the copied/edited into the 'etc/bots-enabled' location to enable them. Andrew -- Andrew Clark | Senior Technical Advisor | CERT Australia Attorney-General's Department, Australian Government Phone: +61 2 6141 2538 Online: www.cert.gov.au For all CERT Australia operational matters, please call our hotline: 1300 172 499, or +61 26141 2999 or email: info(a)cert.gov.au ---------------------------------------------------- If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

1 0

Packaging Strategy for Bots with dependencies
by Dustin Demuth 07 Jul '16

07 Jul '16

Hey Folks, Whilst checking the dependencies, I've found out that a lot of Bots have their own requirements-file. From a packagers point of view, I think this is hard to maintain. I think, we should discuss if it is reasonable to split the software into a CORE package, were all packages have the same dependencies, and several BOTS packages with their own dependencies. This would create the additional need to rethink the BOTS json-file. It would be possible to have all JSON configs for the Bots in a bots.d directory and search for them in this directory. I know, that this might create additional complexity in some points. BR Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

8 15

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev