- IntelMQ-dev - CERT.at

Aggregating events within IntelMQ
by Dustin Demuth 24 Oct '16

24 Oct '16

Dear IntelMQ-Devs, whilst analysing our current setup and possible requirements, we discovered that an aggregation of events within IntelMQ might be a reasonable thing to do. In [1] I sketched how this might work. I'd be glad if you could find the time to comment on this approach and participate in the discussion. [1] https://github.com/certtools/intelmq/issues/751 Best regards Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

3 2

dev: writing similiar tests and test_bot_name()
by Bernhard Reiter 19 Oct '16

19 Oct '16

Hi IntelMQ-Devs, when writing tests what is your development suggestion for writing similiar tests? My problem: intelmq/lib/test.py test_bot_name() checks that "Test class name must be Test{botclassname}." But sometimes it would make sense to have a second test class within one test file. For example in intelmq/tests/bots/experts/modify/test_expert.py I am reusing some of the data of this tile. On this occasion I have used a different class name and explicitely disabled the test_bot_name(). You can reach the code via this search: https://github.com/certtools/intelmq/search?utf8=%E2%9C%93&q=test_bot_name I can see some merit checking for a consistent naming. However making it easier to write more tests is also a good aim and it looks like a good pattern to write more than one test class within one file. What is your idea of solving this in the future? (I meanwhile add another test using the same trick.) Best Regards, Bernhard -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

FYI - priority assignment in issues
by L. Aaron Kaplan 12 Oct '16

12 Oct '16

Hi, Sebastian and me recently had a meeting at CERT.at - discussing the minimal feature set that we still want to have (from our perspective) for putting a "1.0" label on it. We came up with a list. I am now prioritising this list. I took a trick which I learned from another software project and assign priorities to the issues I create in github in the form of Fibonacci sequence numbers: 1,2,3,5,8,13. The benefit of this is that an intuitive value of 13 is really as valuable/high prio as the sum of the previous two priorities. It's rather a bit geeky, but it seems to work. Example: https://github.com/certtools/intelmq/issues/722 Anyway. Just letting you know (so that you don't wonder what this is). This is merely to be read as hint as what I see as high prio. Feel free to adapt the prios or discuss here. Best, a.

3 6

heads up: minor change in the shadowserver identifiers
by L. Aaron Kaplan 06 Oct '16

06 Oct '16

Hi *, as of the discussion in https://github.com/certtools/intelmq/issues/657, I renamed the classification.identifier of the shadowserver snmp to "opensnmp". This might have consequences in your eventDB if you store these events and group them by "classification.identifier". Please (SQL-) UPDATE your previous entries in the eventDB in that case. Thanks, a.

2 1

Future web server for intelmq apps?
by Bernhard Reiter 26 Sep '16

26 Sep '16

Hi friends of IntelMQ, because we are currently planning for the next months and additional webapps are on our lists (for use with the BSI) I've started thinking what the recommendation should be from intelqm. So far I am using the issue tracker to document the design process https://github.com/certtools/intelmq/issues/708 (Basic technology: serving web applications) On my planning list for intelmq web apps is: * move the intelmq-manager to it * add some statistic for the postgresql output bots running with certbund * add some certbund-contact database maintenance My current list is: 1. Bottle 2. Flask 3. CherryPy While we should be focussing on getting 1.0 out of the door, the planning on the web app side will probably continue. I am happy to hear your opinions here or in the issue. Best, Bernhard -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

heads up : one of the hopefully last changes before 1.0
by L. Aaron Kaplan 19 Sep '16

19 Sep '16

(.... never say "last change" ;-) I'd like to change the classification.taxonomy to become case insensitive. Because it makes sense. Any serious objections? Best, a.

3 3

reminder conf call tomorrow, 10:00 CET
by L. Aaron Kaplan 12 Sep '16

12 Sep '16

Hi *, Tuesday is our developer conf call day. We switched it to 10:00 a.m. CET in order to sort of accommodate the Australian colleagues. If you want to participate , please ping me and I'll send you the extension and tel#. Best, Aaron.

1 0

nested message representation
by Sebastian Wagner 12 Sep '16

12 Sep '16

Hi, IntelMQ uses dictionaries to represent messages (in python, json etc.). We use a flat and unnested structure, which is one of the first design goals made in the very beginning of IntelMQ AFAIK. E.g. we have field names like "source.ip" But there's also another possible representation, which is implemented in IntelMQ: nested structures. E.g.: flat: {"classification.type": "unknown", "source.asn": 456, "source.ip": "127.0.0.1"} nested: {"classification": {"type": "unknown"}, "source": {"ip": "127.0.0.1", "asn": 456}} The first is used everywhere except: The messages to_json and to_dict methods, which use the nested format by default. These methods are used in these output bots: file, xmpp, restapi, mongodb, intelmqmailer I think, that this is a wrong default. The default should be something which can be directly interpreted by IntelMQ: the flat structure. Proposal: make flat default and nested optional (for the function and the bots) Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

3 2

modify bot: Do you use many custom rules?
by Bernhard Reiter 30 Aug '16

30 Aug '16

Hi IntelMQ-Users, there is a discussion to improve the modify bot at [1]. As it stand we will change the configuration syntax. So how many custom rules for this bot do you have in use? Let us know so we can estimate how much support for the old configuration syntax is necessary. Best Regards, Bernhard [1] https://github.com/certtools/intelmq/issues/647 -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

Re: [Intelmq-dev] Intelmq-dev Digest, Vol 6, Issue 7
by ben dosso 12 Aug '16

12 Aug '16

hello I'm still stuck on intelmq > From: intelmq-dev-request(a)lists.cert.at > Subject: Intelmq-dev Digest, Vol 6, Issue 7 > To: intelmq-dev(a)lists.cert.at > Date: Thu, 11 Aug 2016 12:00:01 +0200 > > Send Intelmq-dev mailing list submissions to > intelmq-dev(a)lists.cert.at > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev > or, via email, send a message with subject or body 'help' to > intelmq-dev-request(a)lists.cert.at > > You can reach the person managing the list at > intelmq-dev-owner(a)lists.cert.at > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Intelmq-dev digest..." > > > Today's Topics: > > 1. Re: 2 error (Bernhard Reiter) > 2. Re: 2 error (Bernhard Reiter) > 3. Re: 2 error (Bernhard Reiter) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 10 Aug 2016 15:39:42 +0200 > From: Bernhard Reiter <bernhard(a)intevation.de> > To: intelmq-dev(a)lists.cert.at > Cc: ben dosso <dbm93(a)live.fr> > Subject: Re: [Intelmq-dev] 2 error > Message-ID: <201608101539.42787.bernhard(a)intevation.de> > Content-Type: text/plain; charset="utf-8" > > Hi Ben, > > thanks for trying intelmq! > > Am Mittwoch, 10. August 2016 15:03:28 schrieb ben dosso: > > first error > > I installed Ubuntu on intelmq 16 application does not pass. > > Which version of intelmq did you install precisely? > Which documentation were you following? > (Or which steps did you take?) > > > I am sending > > you the error message with an attachment and when I am the way opt / > > intelmq intelmq the file can not be opened. I was told not to display the > > content and also you do not have the required permissions to view the > > contents of "intelmq". > > BTW: It would be nice if you could send the error message in a text format > (if you can copy and paste the text out of that window). > > It seems that the file in question is malformed (or does not have the > necessary contents), you should inspect the file. > (Which you will be able to do as root user.) > > Best Regards, > Bernhard > > -- > www.intevation.de/~bernhard +49 541 33 508 3-3 > Intevation GmbH, Osnabr?ck, DE; Amtsgericht Osnabr?ck, HRB 18998 > Gesch?ftsf?hrer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner >

2 1

Re: [Intelmq-dev] 2 error
by Bernhard Reiter 11 Aug '16

11 Aug '16

> > > https://github.com/certtools/intelmq/blob/master/docs/User-Guide.md > > What does > > ls -l /opt/intelmq/etc > > give you? > ubuntu 16 I can not open a file intelmq The "Install" step in the User-Guide.md should have created the directory or given you other error messages. If you can, maybe you would want to see if there is a GNU/Linux person you can find that gives you a hand. Best Regards, Bernhard ps.: Please keep the mailinglist in cc, there are many more people that may be able to help you with intelmq. -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

Re: [Intelmq-dev] 2 error
by Bernhard Reiter 10 Aug '16

10 Aug '16

Am Mittwoch, 10. August 2016 16:30:42 schrieb ben dosso: > I give you the links for the application. this is the tutorial I followed > to install the application on Ubuntu 16 and 14 I also ubuntu you copy > errors ubuntu 16 connections : > https://github.com/certtools/intelmq/blob/master/docs/User-Guide.mderror : > localhost indique:Failed to obtain JSON:http://localhost/php > load_configs.php ?file=bots with error: Parsererror SyntaxERROR:unexpected > token < in JSON at position 0 the error was made bold What does ls -l /opt/intelmq/etc give you? -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

2 error
by ben dosso 10 Aug '16

10 Aug '16

first error I installed Ubuntu on intelmq 16 application does not pass. I am sending you the error message with an attachment and when I am the way opt / intelmq intelmq the file can not be opened. I was told not to display the content and also you do not have the required permissions to view the contents of "intelmq". second error I installed intelmq on a virtual machine ubuntu 14 but this time I encounter this error

2 1

I have a problem about using Application intelmq
by ben dosso 10 Aug '16

10 Aug '16

1 0

Re: [Intelmq-dev] Intelmq-dev Digest, Vol 6, Issue 2
by ben dosso 10 Aug '16

10 Aug '16

I have a problem about using Application intelmq > From: intelmq-dev-request(a)lists.cert.at > Subject: Intelmq-dev Digest, Vol 6, Issue 2 > To: intelmq-dev(a)lists.cert.at > Date: Mon, 8 Aug 2016 12:00:02 +0200 > > Send Intelmq-dev mailing list submissions to > intelmq-dev(a)lists.cert.at > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev > or, via email, send a message with subject or body 'help' to > intelmq-dev-request(a)lists.cert.at > > You can reach the person managing the list at > intelmq-dev-owner(a)lists.cert.at > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Intelmq-dev digest..." > > > Today's Topics: > > 1. Re: Taxonomies & Sharing mechanism [SEC=UNCLASSIFIED] > (Clark, Andrew) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 8 Aug 2016 05:55:55 +0000 > From: "Clark, Andrew" <Andrew.Clark(a)cert.gov.au> > To: Otmar Lendl <lendl(a)cert.at>, "intelmq-dev(a)lists.cert.at" > <intelmq-dev(a)lists.cert.at> > Subject: Re: [Intelmq-dev] Taxonomies & Sharing mechanism > [SEC=UNCLASSIFIED] > Message-ID: > <454F4A633809FD40898A9D230EDFA93138B2E82B(a)ACTDC01MLD02V.agdnet.ag.gov.au> > > Content-Type: text/plain; charset="utf-8" > > UNCLASSIFIED > Hi Otmar, > > I have never really investigated what's out there in terms of taxonomies, to any great extent. > > We use MISP, and if you haven't seen it, take a look at how many taxonomies they've tried to accommodate: https://github.com/MISP/misp-taxonomies/ > > If I'm reading the correct things, I suspect we might be lucky because the CERT.pt taxonomy looks very similar to the eCSIRT taxonomy used by IntelMQ (and supported by MISP). > > The CERT.pt taxonomy (from this site: http://www.cncs.gov.pt/cert-pt-2/documents-2/) includes 18 "incident types" and 10 "incident classes". The ClassificationType class from IntelMQ supports 20 values, including the 18 from the CERT.pt taxonomy, plus "unknown" and "blocklist". Based on this, I don't think there is a good reason to change what IntelMQ uses now. > > Regarding STIX and Cybox (and TAXII), here at CERT Australia we are using them heavily. STIX includes a 'TTP' object which can be associated with Indicators. TTPs include 'behaviours' and while STIX supports the CAPEC (capec.mitre.org) taxonomy natively, it would be easy to extend to support arbitrary taxonomies. > > Hope you're enjoying your vacation! > > Andrew > > -----Original Message----- > From: Intelmq-dev [mailto:intelmq-dev-bounces@lists.cert.at] On Behalf Of Otmar Lendl > Sent: Saturday, 6 August 2016 2:07 AM > To: intelmq-dev(a)lists.cert.at > Subject: [Intelmq-dev] Taxonomies & Sharing mechanism > > > Folks, > > as I will attending the ENISA/EC3 workshop in The Hague this autumn, I got an invitation to a preparatory survey which asks questions about a consensus regarding taxonomies and information sharing formats to be used in CERT/CERT and CERT/LE information sharing. > > IntelMQ is based on eCSIRT II, which some working-group in the ENISA/EC3/EMPACT universe has declared to be obsolete. > > See this monster of a report: > https://www.enisa.europa.eu/publications/information-sharing-and-common-tax… > > Their new shiny pony is based on the work of CERT.pt, and they want to to use the meeting this year to finalize that decision. I have no clue how big the delta to eCSIRT II is. > > IMHO the IntelMQ community has to decide how to react. E.g. > > a) stay with eCSIRT II framework > b) adopt the new one > > and > > what stance to take on an inter-organisational sharing mechanism. > > So what do you all think? > > otmar (who will be on vacation the next weeks, don't expect me to reply > soon) > > ------------------ > > The survey asks: > > Do you believe that the Common Taxonomy for the national network of CSIRT/LEA (formerly known as CERT.PT Taxonomy) is suitable for CSIRT/LEA communication? > > Yes / No / Other > > Have you ever used one of the following? > > STIX / CybOX / Other sharing Mechanism > > What do you think could be a suitable sharing mechanism for the Common Taxonomy for the national network of CSIRT/LEA? > > STIX / CybOX / Other sharing Mechanism > > Extract from 'Report on Information Sharing and Common Taxonomies between CSIRTs and Law Enforcement Agencies' > > A clear distinction should be made between a taxonomy, a sharing mechanism and a sharing platform to avoid any possible confusion. While a taxonomy is a way of describing information through classification, a sharing mechanism structures the way the information is encoded. For example, a sharing mechanism might provide rules for names and positions of XML tags to allow a file to be treated automatically. Finally, a sharing platform is a tool allowing to share information. It is not mandatory to have such a platform ? files containing information structured according to a standard and classified according to a taxonomy could simply be sent by e-mail, for example. Nevertheless, the use of a sharing platform allows users to easily share information in a structured way. > > > -- > // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg > > > ---------------------------------------------------- > If you have received this transmission in error please > notify us immediately by return e-mail and delete all > copies. If this e-mail or any attachments have been sent > to you in error, that error does not constitute waiver > of any confidentiality, privilege or copyright in respect > of information in the e-mail or attachments. > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Intelmq-dev mailing list > Intelmq-dev(a)lists.cert.at > http://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev > > > ------------------------------ > > End of Intelmq-dev Digest, Vol 6, Issue 2 > *****************************************

1 0

heads up - breaking changes to harmonization.conf
by L. Aaron Kaplan 09 Aug '16

09 Aug '16

Hi, Pull request #634 (https://github.com/certtools/intelmq/pull/634 ) will include some changes to the harmonisation.conf file. There was a previous discussion that some of the values (varchar(2000) in most cases) were arbitrary and sometimes even not long enough. Sebix changed a lot to the postgresql "text" type and in the course of doing so, cleaned up the regular expressions and definitions of fields in harmonisation.conf. Please have a thorough look at PR #634 before I merge it in. If there are any concerns, please let us know by the end of this week if possible. Thanks! a.

1 0

Re: [Intelmq-dev] Taxonomies & Sharing mechanism [SEC=UNCLASSIFIED]
by Clark, Andrew 08 Aug '16

08 Aug '16

UNCLASSIFIED Hi Otmar, I have never really investigated what's out there in terms of taxonomies, to any great extent. We use MISP, and if you haven't seen it, take a look at how many taxonomies they've tried to accommodate: https://github.com/MISP/misp-taxonomies/ If I'm reading the correct things, I suspect we might be lucky because the CERT.pt taxonomy looks very similar to the eCSIRT taxonomy used by IntelMQ (and supported by MISP). The CERT.pt taxonomy (from this site: http://www.cncs.gov.pt/cert-pt-2/documents-2/) includes 18 "incident types" and 10 "incident classes". The ClassificationType class from IntelMQ supports 20 values, including the 18 from the CERT.pt taxonomy, plus "unknown" and "blocklist". Based on this, I don't think there is a good reason to change what IntelMQ uses now. Regarding STIX and Cybox (and TAXII), here at CERT Australia we are using them heavily. STIX includes a 'TTP' object which can be associated with Indicators. TTPs include 'behaviours' and while STIX supports the CAPEC (capec.mitre.org) taxonomy natively, it would be easy to extend to support arbitrary taxonomies. Hope you're enjoying your vacation! Andrew -----Original Message----- From: Intelmq-dev [mailto:intelmq-dev-bounces@lists.cert.at] On Behalf Of Otmar Lendl Sent: Saturday, 6 August 2016 2:07 AM To: intelmq-dev(a)lists.cert.at Subject: [Intelmq-dev] Taxonomies & Sharing mechanism Folks, as I will attending the ENISA/EC3 workshop in The Hague this autumn, I got an invitation to a preparatory survey which asks questions about a consensus regarding taxonomies and information sharing formats to be used in CERT/CERT and CERT/LE information sharing. IntelMQ is based on eCSIRT II, which some working-group in the ENISA/EC3/EMPACT universe has declared to be obsolete. See this monster of a report: https://www.enisa.europa.eu/publications/information-sharing-and-common-tax… Their new shiny pony is based on the work of CERT.pt, and they want to to use the meeting this year to finalize that decision. I have no clue how big the delta to eCSIRT II is. IMHO the IntelMQ community has to decide how to react. E.g. a) stay with eCSIRT II framework b) adopt the new one and what stance to take on an inter-organisational sharing mechanism. So what do you all think? otmar (who will be on vacation the next weeks, don't expect me to reply soon) ------------------ The survey asks: Do you believe that the Common Taxonomy for the national network of CSIRT/LEA (formerly known as CERT.PT Taxonomy) is suitable for CSIRT/LEA communication? Yes / No / Other Have you ever used one of the following? STIX / CybOX / Other sharing Mechanism What do you think could be a suitable sharing mechanism for the Common Taxonomy for the national network of CSIRT/LEA? STIX / CybOX / Other sharing Mechanism Extract from 'Report on Information Sharing and Common Taxonomies between CSIRTs and Law Enforcement Agencies' A clear distinction should be made between a taxonomy, a sharing mechanism and a sharing platform to avoid any possible confusion. While a taxonomy is a way of describing information through classification, a sharing mechanism structures the way the information is encoded. For example, a sharing mechanism might provide rules for names and positions of XML tags to allow a file to be treated automatically. Finally, a sharing platform is a tool allowing to share information. It is not mandatory to have such a platform – files containing information structured according to a standard and classified according to a taxonomy could simply be sent by e-mail, for example. Nevertheless, the use of a sharing platform allows users to easily share information in a structured way. -- // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg ---------------------------------------------------- If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

1 0

Taxonomies & Sharing mechanism
by Otmar Lendl 05 Aug '16

05 Aug '16

Folks, as I will attending the ENISA/EC3 workshop in The Hague this autumn, I got an invitation to a preparatory survey which asks questions about a consensus regarding taxonomies and information sharing formats to be used in CERT/CERT and CERT/LE information sharing. IntelMQ is based on eCSIRT II, which some working-group in the ENISA/EC3/EMPACT universe has declared to be obsolete. See this monster of a report: https://www.enisa.europa.eu/publications/information-sharing-and-common-tax… Their new shiny pony is based on the work of CERT.pt, and they want to to use the meeting this year to finalize that decision. I have no clue how big the delta to eCSIRT II is. IMHO the IntelMQ community has to decide how to react. E.g. a) stay with eCSIRT II framework b) adopt the new one and what stance to take on an inter-organisational sharing mechanism. So what do you all think? otmar (who will be on vacation the next weeks, don't expect me to reply soon) ------------------ The survey asks: Do you believe that the Common Taxonomy for the national network of CSIRT/LEA (formerly known as CERT.PT Taxonomy) is suitable for CSIRT/LEA communication? Yes / No / Other Have you ever used one of the following? STIX / CybOX / Other sharing Mechanism What do you think could be a suitable sharing mechanism for the Common Taxonomy for the national network of CSIRT/LEA? STIX / CybOX / Other sharing Mechanism Extract from 'Report on Information Sharing and Common Taxonomies between CSIRTs and Law Enforcement Agencies' A clear distinction should be made between a taxonomy, a sharing mechanism and a sharing platform to avoid any possible confusion. While a taxonomy is a way of describing information through classification, a sharing mechanism structures the way the information is encoded. For example, a sharing mechanism might provide rules for names and positions of XML tags to allow a file to be treated automatically. Finally, a sharing platform is a tool allowing to share information. It is not mandatory to have such a platform – files containing information structured according to a standard and classified according to a taxonomy could simply be sent by e-mail, for example. Nevertheless, the use of a sharing platform allows users to easily share information in a structured way. -- // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Reports larger than 500 MB in IntelMQ
by Dustin Demuth 29 Jul '16

29 Jul '16

Dear all, currently we are facing the problem, that IntelMQ is not capable of handling large reports ( > 500 MB) when using Redis as a Message Queuing System. First we thought this might get fixed in most recent redis versions (see: [1]), but apparently this is not the case. Are you aware of setups of IntelMQ which use a different backend, like ZMQ? Do you know of performance or other issues which occurred when using other Queuing Systems? [1] https://github.com/certtools/intelmq/issues/547 Best Regards Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

4 9

another naming update
by L. Aaron Kaplan 29 Jul '16

29 Jul '16

Hi *, if you are using the shadowserver parsers , especially the mDNS parser, please be aware that the feed name changed. "Open-m DNS" to "Open-mDNS" See: https://github.com/certtools/intelmq/issues/624 Please adapt your runtime.conf accordingly. Best, a.

1 0

Changes in harmonization implementation
by Sebastian Wagner 29 Jul '16

29 Jul '16

Dear contributors and beta-testers, We recently changed did some changes in the implementation of the harmonization. The type MalwareName has been removed, it was just a duplicate of String. This may break your existing installation if you do not update your harmonization.conf with upstream. It has been used as type for malware.name The type LowercaseString has been introduced, doing a conversion to lower case chars automatically. Applied to protocol names, emails, hashes and UUIDs etc. FQDNs are now lower case only too. Some restrictive regexes have been made more tolerant (e.g. malware names) misp_uuid is now misp.event_uuid misp.attribute_uuid introduced new iregex-type check available (case-insenstitive regex-check) allowed values for protocol.transport is a defined list Also: Currently, both ASCII and IDN-domains are allowed, this will be changed too (including automatic conversion): https://github.com/certtools/intelmq/issues/622 When pulling from our upstream-repository, make sure you also update your harmonization.conf Related commits: https://github.com/certtools/intelmq/commit/6624bdda04ca0595799150e97e719a5… https://github.com/certtools/intelmq/commit/640e1a771acb4ddb9cbd726e51ae903… https://github.com/certtools/intelmq/commit/fd81f0fd9c86ffa0ac3764adafc74e7… https://github.com/certtools/intelmq/commit/7dbed4bc8f9200d06dae2018850341a… https://github.com/certtools/intelmq/commit/6b8b759293069afc3d9f0f1cdf8e4ea… https://github.com/certtools/intelmq/commit/d7892bcb59ac54218c5c45630226d4a… https://github.com/certtools/intelmq/commit/6584ab72fa408732c02abc1c934637a… https://github.com/certtools/intelmq/commit/a742c034c80249e80d2f280fd0dc8a7… Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

tomorrow, Tuesday intelmq-dev conf call
by L. Aaron Kaplan 25 Jul '16

25 Jul '16

Hi, We'll be having our regular intelmq-dev call tomorrow at 9:00 a.m. CEST. This terribly early date was chosen so that the colleagues from Australia can join the call as well. If you want to participate in the call, please let me know, I will send you call in details. Best, a.

1 0

Re: [Intelmq-dev] today 16:00 UTC+2 conf call [SEC=UNCLASSIFIED]
by Clark, Andrew 11 Jul '16

11 Jul '16

UNCLASSIFIED Hi Aaron, Thanks for thinking of us down here in Australia, and sorry for the slow response - I was on leave last week. Early morning your time is going to suit us best - we can be a bit flexible in the evenings, it it's say 1630 or 1700 here (UTC+10) that is fine, or if that's too early for you we could try after 1900 and I can dial in from home. Andrew -- Andrew Clark | Senior Technical Advisor | CERT Australia Attorney-General's Department, Australian Government Phone: +61 2 6141 2538 Online: www.cert.gov.au For all CERT Australia operational matters, please call our hotline: 1300 172 499, or +61 26141 2999 or email: info(a)cert.gov.au -----Original Message----- From: Intelmq-dev [mailto:intelmq-dev-bounces@lists.cert.at] On Behalf Of L. Aaron Kaplan Sent: Tuesday, 5 July 2016 7:38 PM To: intelmq-dev(a)lists.cert.at Subject: [Intelmq-dev] today 16:00 UTC+2 conf call Hi *, today at 16:00 UTC+2 we'll have our conf call again (and resume it after some weeks of FIRST conference and post-FIRST catching up). Topics for this call: * generic syntax for transformations/generic mapping language * status update Intevation * status update CERT.at * ... your status update? * any other input for the monthly newsletter * biggest concerns/topics right now * discussion action items for next week * AoB Please feel free to suggest topics for this call by replying to this mail thread. Thanks, a. PS: I'd like to suggest to move this weekly conf call to an earlier time in the future since this would allow our Australian colleagues to join as well. When would be a good time for the Aussies? -- // CERT Austria // L. Aaron Kaplan <kaplan(a)cert.at> // T: +43 1 505 64 16 78 // http://www.cert.at // Eine Initiative der NIC.at Internet Verwaltungs- und Betriebs GmbH // http://www.nic.at/ - Firmenbuchnummer 172568b, LG Salzburg ---------------------------------------------------- If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

1 0

Re: [Intelmq-dev] Packaging Strategy for Bots with dependencies [SEC=UNCLASSIFIED]
by Clark, Andrew 11 Jul '16

11 Jul '16

UNCLASSIFIED Hi Bernhard, all, Sorry to come into this a bit late ... my comments are below: > Am Mittwoch, 20. April 2016 11:55:59 schrieb L. Aaron Kaplan: > > > Aren't we running into the same issue as with the BOTS file? If > > > > The BOTS file is simply a template of which bots exist in general. > > It is mainly used for the intelmq-manager to display which ones > > could get selected. And which parameters they have. > > In order to get this moving, I propose to tackle it in small steps. > > What about implementing a solution for the contents of BOTS first? > We keep the logic of etc/defaults.conf and etc/runtime.conf for this step. > So no hiearchical configuration (for now) and then we take it from there. If I understand correctly the IntelMQ 'runtime' configuration is currently spread across three configuration files (runtime.conf, pipeline.conf and startup.conf), and there are other configuration files as well (e.g. defaults.conf, harmonization.conf and system.conf). I think it's a great idea to consider how to improve on the current monolithic BOTS template configuration, but also wonder if some of the configuration that is currently in separate files could be merged. For example, could startup and runtime be merged? Maybe even pipeline too? > Thinking about how to solve the BOTS file first: > a) (proposed by Dusting) bots.d/ directory solution, > means adding and removing filenames. > b) adding an initialisation function to each module which > returns the necessary information > c) write a small packaging helper that adds/removes sections > from BOTS > > Some thoughts (being still new to the code base): > b) seems attractive to me, because it keeps information > close to where it is maintainted. Its main drawback probably is > that is the largest change. Are there other drawbacks? > c) has the drawback of requiring running code during (de)installation. > And it does not add much over a). I have a STRONG preference for an approach similar to 'a' above. In my opinion we should be aiming to decouple the configuration from the bot code itself as much as possible, and encouraging more generic bots (i.e., less bots that are more generic (and more configurable), and more configuration files. In general each bot should potentially accept many configurations. This approach should serve to limit the explosion in the number of bots (I think the parser bots are a good example of where we could reduce the number of bots focusing on 'what' they are parsing, rather than their source). Along the lines of option 'a' above I think the 'modules-available' and 'modules-enabled' approach used by the Apache web server could work for us too. All available configurations (templates, etc.) could be stored in a 'etc/bots-available' location, and the copied/edited into the 'etc/bots-enabled' location to enable them. Andrew -- Andrew Clark | Senior Technical Advisor | CERT Australia Attorney-General's Department, Australian Government Phone: +61 2 6141 2538 Online: www.cert.gov.au For all CERT Australia operational matters, please call our hotline: 1300 172 499, or +61 26141 2999 or email: info(a)cert.gov.au ---------------------------------------------------- If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

1 0

Packaging Strategy for Bots with dependencies
by Dustin Demuth 07 Jul '16

07 Jul '16

Hey Folks, Whilst checking the dependencies, I've found out that a lot of Bots have their own requirements-file. From a packagers point of view, I think this is hard to maintain. I think, we should discuss if it is reasonable to split the software into a CORE package, were all packages have the same dependencies, and several BOTS packages with their own dependencies. This would create the additional need to rethink the BOTS json-file. It would be possible to have all JSON configs for the Bots in a bots.d directory and search for them in this directory. I know, that this might create additional complexity in some points. BR Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

8 15

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev