= Intelmq-dev-news 05-2016
Issue 5/2016
== Topics ==
# Summary of IHAP meeting in April
# Status update Intevation
# Status update CERT.at
# Status update misc
== May 2016 ==
Dear Intelmq-dev mailing list readers,
this is the second issue of intelmq developer news.
We hope it's useful.
TL;DR and important changes
-----------------------------
The syntax of intelmqcli was changed to a new format:
intelmqctl {start,stop} bot_id.
This breaks compatibilty with existing scripts.
If you put intelmqctl into some script, please adapt it. Also
please be sure to check out the latest version of the intelmq-manager
in case you use it.
Lots of open issues. Progress with intelmqcli (to connect postgresql to the RT ticket system).
/ TL;DR
=== How to contribute to this newsletter? ===
-> contact Aaron, Dustin for future input
=== Summary of IHAP meeting in April ===
In April the IHAP Meeting took place in Vienna.
* A Hacksession the night before the meeting was used by Raphael and Aaron in
order to bridge MISP and IntelMQ.
* Connections between Abusehelper and IntelMQ are on some CERTs wish list.
XMPP is a good start. Unfortunately the XMPP Bot upstream was not fit for
production.
=== Status report Intevation ===
* Still working on the KontaktDB, we appreciate the discussions that started
on IHAP Meeting.
We received a Pull Request from Cert.at and are currently reviewing it.
* We have Scripts to import Data into the KontaktDB.
Nevertheless there is some work left.
* Demonstrated installation from packages on Ubuntu 14.04 on IHAP-Meeting.
We propose to host the **signed** packages on our public apt-repositories.
* Working on a tool similar to intelmqcli, intended to process events from
the eventdb. Instead of using RT they are sent by e-mail.
The tool has the working title "event-processor" and can be found here
(https://github.com/Intevation/event-processor)
* We did not start with support for IODEF or X-ARF yet.
=== Status report CERT.at developments ===
* we moved to python3 only. Intelmq dropped python2 support
(https://github.com/certtools/intelmq/commit/2cbb42f1458a7e90539a443ec5e50ee…).
This does not apply yet to the certat repo (github.com/certat/intelmq), which
still supports python2.7 but only for the intelmqcli tool.
* New active contributor: pedro m. reis! Welcome and thanks for working so
hard on the Bitsight collector
(https://github.com/certtools/intelmq/pull/493)
* intelmqcli tool now supports a lot of new flags:
https://github.com/certat/intelmq/issues/52 This was necessary for CERT.at
since we use intelmqcli via cron job to connect to the (postgresql) eventDB ,
pull out all of the new data and use RT (ticket system) to send stuff out.
Added flags --quiet --batch. Now intelmqcli sends via cronjob.
These flags now allow CERT.at to run intelmq in full auto-mode! intelmqcli is
started via cron and sends out all events to all ISPs.
=== Requests ===
* Intevation searches for testers for the packages.
* We'd like to have some nice graphs in the intelmq-manager: events/sec , parse-failures/sec, etc.
* implementation of whitelisting of events (filter out events based on whitelists). See
https://github.com/certtools/intelmq/issues/426
* A good CSS design for the web page
=== Community ===
* RIPE abuse-c contacts can be done locally. RIPE might be able to export
abuse-c infos publicly (fingers crossed).
* more command line options for intelmqcli (see the
https://github.com/certat/intelmq repo)
* Aaron gave a presentation at the ENISA workshop "CSIRTs in Europe", 11th of May.
Slides will be shared on the ENISA page.
==== intelmq.org ====
The website intelmq.org is now online, but we would like to have more content and a proper
design.
Do you want to contribute to intelmq, but you are not a programmer? This is
your chance!
Current ToDos:
* Create Website Content: How-Tos / Installation Instructions, Success
Stories
** How-Tos / Instructions: If you are using a special feature of IntelMQ, for
instance an expert bot, try to find some time to write down a short article
how you managed to get it to work and why you are using it.
* Website Design
== Wishlist ==
* **we need more test-cases!!!**
* a specific config logic for ASNs: do this and that (for example sett ttl =
1 month) if event is in ASN xyz. Or "ignore" if event is in ASN xyz. This
should support some kind of more-specific-less-specific inheritance,
similarly to Apache directory settings. The most specific setting wins. The
order could be: country code -> ASN -> netblock -> ip (/32). Open questions:
what's more relevant if both domains and numbers (ASN, IPs, net blocks) exist
in an event?
* block based processing: for example block based team cymru lookups
* parallelisation: We need to revisit this topic
== Important Discussions ==
In case you missed something, here are the headlines of some discussion we
consider interesting / important.
=== Mailing Lists ===
* [Intelmq-dev] Packaging Strategy for Bots with dependencies
* [Intelmq-dev] Discussion on intelmq output / transformation architecture
* [Intelmq-dev] Output format to syslog/splunk (PR#503)
== Communication ==
Chat: irc #intelmq on freenode or webchat:
[[https://webchat.freenode.net/?channels=intelmq]]
Follow on twitter: @intelmqorg
Weekly Conference Call every Tuesday: Dial in via the known conference bridge number. It is
[[https://en.wikipedia.org/wiki/Telephone_number_mapping|ENUM]] enabled. Ask
Aaron or Dustin for the number if you want to participate.
Hey Folks,
Whilst checking the dependencies, I've found out that a lot of Bots have their
own requirements-file.
From a packagers point of view, I think this is hard to maintain.
I think, we should discuss if it is reasonable to split the software into a
CORE package, were all packages have the same dependencies, and several BOTS
packages with their own dependencies.
This would create the additional need to rethink the BOTS json-file.
It would be possible to have all JSON configs for the Bots in a bots.d
directory and search for them in this directory.
I know, that this might create additional complexity in some points.
BR
Dustin
--
dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF
Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998
Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
Dear all,
as a short announcement, we are currently starting to work on parsers for the
follwing shadowserver feeds.
Drone [Done]
Microsoft Sinkhole
Sinkhole HTTP Drone
DNS Open Resolvers
NTP Monitor [Done]
Open Portmapper
Open CharGen
Open Elasticsearch
Open IPMI
Open MDNS
Open Memcached [Done]
Open MongoDB
Open MS-SQL
Open NetBIOS
Open Redis
Open SNMP
Open SSDP
SSL FREAK
SSL POODLE [Done]
We expect them to be ready by the end of this week.
BR
Dustin
--
dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF
Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998
Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
UNCLASSIFIED
Hi guys,
With the help of a colleague I have prepared a MISP collector and parser for IntelMQ. It requires a tag to be added to the MISP events that need to be processed. This tag is removed from the MISP event by the collector once it has been processed (and a different tag is added to the MISP event to indicate that it has been processed). Anyway, without getting too bogged down in the details, I've put the code in a forked copy of the repo on my github page:
https://github.com/kralca/intelmq/commit/c3cdb0e
The deduplicator expert should be used to detect MISP event attributes that have been previously processed (for example following the addition of attributes to a MISP event).
I hope this is useful for the Hackathon on Sunday. Please let me know if you would prefer if I submit a pull request.
Cheers,
Andrew
--
Andrew Clark | Senior Technical Advisor | CERT Australia
Attorney-General's Department, Australian Government
Phone: +61 2 6141 2538
Online: www.cert.gov.au<http://www.cert.gov.au/>
For all CERT Australia operational matters, please call our
hotline: 1300 172 499, or +61 26141 2999 or
email: info(a)cert.gov.au<mailto:info@cert.gov.au>
----------------------------------------------------
If you have received this transmission in error please
notify us immediately by return e-mail and delete all
copies. If this e-mail or any attachments have been sent
to you in error, that error does not constitute waiver
of any confidentiality, privilege or copyright in respect
of information in the e-mail or attachments.
= Intelmq-dev-news 06-2016
Issue 6/2016
== Topics ==
# Summary
# Status update Intevation
# Status update CERT.at
# Status update misc
== Review of May 2016 ==
TL;DR and important changes
-----------------------------
* Lots of adaptations
* CERT Australia uses intelmq
* Hackathon on Sunday, 12th of June at the FIRST.org conference, Seoul. If you are attending FIRST, please do join!
/ TL;DR
=== How to contribute to this newsletter? ===
-> contact Aaron, Dustin for future input
=== Status report Intevation ===
* The schema of the contactDB will be revised: The idea is that the most specific template wins.
For instance: first check for a template for malware.name, then classification.type, then classification.taxonomy, else use default template or abort.
This is required as almost all feeds do not set the classification.identifier and the classification.type is not specific enough to pick a template.
* To enhance the process of identifying events, the parser needs to set the identifier.
Idea: Provide a list which is maintained in a central place which maps these identifiers. This mapping could be downloaded by IntelMQ and be used by the parsers.
* Gernot is now responsible for Packaging. We now use an APT repository for our releases.
** Idea: Bots can have their own packaging, this makes IntelMQ more modular.
* Intevation will submit a proposal for a mature branching strategy. We will bring this topic to the list.
* Intevation is also going to propose an idea for config files: Something like: etc/intelmq/bots-{available,enabled}/ directories as in Apache2. This might make life easier.
* Intelmq-mailgen script: A script supposed to send mails in different formats: X-ARF is one of them, but currently experimental.
** The script is tied to certbund-contactDB expert.
** Interesting concept: The mailgen-script enables to track which event was sent at what time to each customer (can add ticket # numbers).
* A Script which fills the contactDB with information from RIPE DB is in the queue
* taxonomy expert now supports the taxonomy "other"
* Created some Shadowserver parsers for drone, ntp monlist, open memchached, ssl poodle.
* 500 MB reports do not fit into redis messages. We expect an updated redis > 3.2 should work for these large messages. But this would require testing.
=== Status report CERT.at developments ===
* intelmq now can process SIGHUPs: this will reload the bot's configuration.
* related: new syntax for intelmqctl: check out ``intelmqctl reload``.
* cronjobs which intevation created are being used now at CERT.at.
* work on new parsers - new architecture (https://github.com/certtools/intelmq/pull/529)
* idea: new parser architecture is parsing based on individual lines. Now you can find individual lines which can't be parsed and just replay these.
* needs testing
* tor bot: depends on internet2.us. Fixed and made more robust by Aaron.
Coming:
* missing: monitoring, log check deployment: check if .dump files exist
* missing: intelmq-manager does not graph events/sec , etc yet. idea: use RRD
* packaging: sebix is looking at tools to create packages for Debian, RedHat, etc. all at once. Sebix is looking at OpenBuild System by opentools. We will upload the packages to the website.
* missing: branching concept for the release
* need to test shadowserver (https://github.com/certtools/intelmq/issues/524) and if it's okay, pull into master.
* dns-python has a new release. We use this everywhere. We should update it in the packages and code (let's wait a few weeks if some issues arises in the new version, but then we upgrade)
=== Wider community ===
* Koen wrote a octopress template for the website. Will try this out. Thanks very much!
* Discussions on MISP<-> Intelmq integration (https://github.com/certtools/intelmq/issues/537).
* We will have a hackathon on Sunday, 12th of June at the FIRST.org conference, Seoul. If you are attending FIRST, please do join!
=== Wish-list ===
* **we need more test-cases!!!** unit tests as well as integration tests.
* Intevation searches for testers for the packages.
* We'd like to have some nice graphs in the intelmq-manager: events/sec , parse-failures/sec, etc.
* implementation of whitelisting of events (filter out events based on whitelists). See
https://github.com/certtools/intelmq/issues/426
* A good CSS design for the web page
* Create more website Content: How-Tos / Installation Instructions, Success Stories
* How-Tos / Instructions: If you are using a special feature of IntelMQ, for
instance an expert bot, try to find some time to write down a short article
how you managed to get it to work and why you are using it.
* a specific config logic for ASNs: do this and that (for example sett ttl =
1 month) if event is in ASN xyz. Or "ignore" if event is in ASN xyz. This
should support some kind of more-specific-less-specific inheritance,
similarly to Apache directory settings. The most specific setting wins. The
order could be: country code -> ASN -> netblock -> ip (/32). Open questions:
what's more relevant if both domains and numbers (ASN, IPs, net blocks) exist
in an event?
* block based processing: for example block based team cymru lookups
* parallelisation: We need to revisit this topic
== Important Discussions ==
In case you missed something, here are the headlines of some discussion we
consider interesting / important.
=== Mailing Lists ===
https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
== Communication ==
Chat: irc #intelmq on freenode or webchat:
[[https://webchat.freenode.net/?channels=intelmq]]
Follow on twitter: @intelmqorg
Weekly Conference Call every Tuesday 16:00 UTC+2: Dial in via the known conference bridge number. It is
[[https://en.wikipedia.org/wiki/Telephone_number_mapping|ENUM]] enabled. Ask Aaron or Dustin for the number if you want to participate.