[Intelmq-dev] newsletter May 2016

[L. Aaron Kaplan]

= Intelmq-dev-news 06-2016

Issue 6/2016

== Topics == 

 # Summary
 # Status update Intevation
 # Status update CERT.at
 # Status update misc


== Review of May 2016 ==


TL;DR and important changes
-----------------------------

 * Lots of adaptations
 * CERT Australia uses intelmq
 * Hackathon on Sunday, 12th of June at the FIRST.org conference, Seoul. If you are attending FIRST, please do join!

/ TL;DR





=== How to contribute to this newsletter? ===

 -> contact Aaron, Dustin for future input


=== Status report Intevation === 

  * The schema of the contactDB will be revised: The idea is that the most specific template wins.
    For instance: first check for a template for malware.name, then classification.type, then classification.taxonomy, else use default template or abort.
    This is required as almost all feeds do not set the classification.identifier and the classification.type is not specific enough to pick a template.

  * To enhance the process of identifying events, the parser needs to set the identifier.
    Idea: Provide a list which is maintained in a central place which maps these identifiers. This mapping could be downloaded by IntelMQ and be used by the parsers.

  * Gernot is now responsible for Packaging. We now use an APT repository for our releases.
  ** Idea: Bots can have their own packaging, this makes IntelMQ more modular.

  * Intevation will submit a proposal for a mature branching strategy. We will bring this topic to the list.

  * Intevation is also going to propose an idea for config files: Something like: etc/intelmq/bots-{available,enabled}/ directories as in Apache2. This might make life easier.

  * Intelmq-mailgen script: A script supposed to send mails in different formats: X-ARF is one of them, but currently experimental.
    ** The script is tied to certbund-contactDB expert.
    ** Interesting concept: The mailgen-script enables to track which event was sent at what time to each customer (can add ticket # numbers).

  * A Script which fills the contactDB with information from RIPE DB is in the queue

  * taxonomy expert now supports the taxonomy "other"

  * Created some Shadowserver parsers for drone, ntp monlist, open memchached, ssl poodle.

  * 500 MB reports do not fit into redis messages. We expect an updated redis > 3.2 should work for these large messages. But this would require testing.


=== Status report CERT.at developments ===


  * intelmq now can process SIGHUPs: this will reload the bot's configuration.
  * related: new syntax for intelmqctl: check out ``intelmqctl reload``.
  * cronjobs which intevation created are being used now at CERT.at. 
  * work on new parsers - new architecture (https://github.com/certtools/intelmq/pull/529)
    * idea: new parser architecture is parsing based on individual lines. Now you can find individual lines which can't be parsed and just replay these.
    * needs testing
  * tor bot: depends on internet2.us. Fixed and made more robust by Aaron.

Coming:
  * missing: monitoring, log check deployment: check if .dump files exist 
  * missing: intelmq-manager does not graph events/sec , etc yet. idea: use RRD
  * packaging: sebix is looking at tools to create packages for Debian, RedHat, etc. all at once. Sebix is looking at OpenBuild System by opentools. We will upload the packages to the website.
  * missing: branching concept for the release
  * need to test shadowserver (https://github.com/certtools/intelmq/issues/524) and if it's okay, pull into master.

  * dns-python has a new release. We use this everywhere. We should update it in the packages and code (let's wait a few weeks if some issues arises in the new version, but then we upgrade)
  
=== Wider community ===

  * Koen wrote a octopress template for the website. Will try this out. Thanks very much!
  * Discussions on MISP<-> Intelmq integration (https://github.com/certtools/intelmq/issues/537). 
  * We will have a hackathon on Sunday, 12th of June at the FIRST.org conference, Seoul. If you are attending FIRST, please do join!


=== Wish-list ===


  * **we need more test-cases!!!** unit tests as well as integration tests.
  * Intevation searches for testers for the packages.
  * We'd like to have some nice graphs in the intelmq-manager: events/sec , parse-failures/sec, etc.
  * implementation of whitelisting of events (filter out events based on whitelists). See
    https://github.com/certtools/intelmq/issues/426
  * A good CSS design for the web page
  * Create more website Content: How-Tos / Installation Instructions,  Success Stories
  * How-Tos / Instructions: If you are using a special feature of IntelMQ, for 
instance an expert bot, try to find some time to write down a short article 
how you managed to get it to work and why you are using it.

  * a specific config logic for ASNs: do this and that (for example sett ttl = 
1 month) if event is in ASN xyz. Or "ignore" if event is in ASN xyz. This 
should support some kind of more-specific-less-specific inheritance, 
similarly to Apache directory settings. The most specific setting wins. The 
order could be: country code -> ASN -> netblock -> ip (/32). Open questions: 
what's more relevant if both domains and numbers (ASN, IPs, net blocks) exist 
in an event?
  * block based processing: for example block based team cymru lookups
  * parallelisation: We need to revisit this topic

== Important Discussions ==
 In case you missed something, here are the headlines of some discussion we 
consider interesting / important.
 
=== Mailing Lists ===

  https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev 

== Communication ==
Chat: irc #intelmq on freenode or webchat: 
[[https://webchat.freenode.net/?channels=intelmq]]

Follow on twitter: @intelmqorg

Weekly Conference Call every Tuesday 16:00 UTC+2: Dial in via the known conference bridge number. It is 
[[https://en.wikipedia.org/wiki/Telephone_number_mapping|ENUM]] enabled. Ask  Aaron or Dustin for the number if you want to participate.