= Intelmq-dev-news 05-2016
Issue 5/2016
== Topics ==
# Summary of IHAP meeting in April
# Status update Intevation
# Status update CERT.at
# Status update misc
== May 2016 ==
Dear Intelmq-dev mailing list readers,
this is the second issue of intelmq developer news.
We hope it's useful.
TL;DR and important changes
-----------------------------
The syntax of intelmqcli was changed to a new format:
intelmqctl {start,stop} bot_id.
This breaks compatibilty with existing scripts.
If you put intelmqctl into some script, please adapt it. Also
please be sure to check out the latest version of the intelmq-manager
in case you use it.
Lots of open issues. Progress with intelmqcli (to connect postgresql to the RT ticket system).
/ TL;DR
=== How to contribute to this newsletter? ===
-> contact Aaron, Dustin for future input
=== Summary of IHAP meeting in April ===
In April the IHAP Meeting took place in Vienna.
* A Hacksession the night before the meeting was used by Raphael and Aaron in
order to bridge MISP and IntelMQ.
* Connections between Abusehelper and IntelMQ are on some CERTs wish list.
XMPP is a good start. Unfortunately the XMPP Bot upstream was not fit for
production.
=== Status report Intevation ===
* Still working on the KontaktDB, we appreciate the discussions that started
on IHAP Meeting.
We received a Pull Request from Cert.at and are currently reviewing it.
* We have Scripts to import Data into the KontaktDB.
Nevertheless there is some work left.
* Demonstrated installation from packages on Ubuntu 14.04 on IHAP-Meeting.
We propose to host the **signed** packages on our public apt-repositories.
* Working on a tool similar to intelmqcli, intended to process events from
the eventdb. Instead of using RT they are sent by e-mail.
The tool has the working title "event-processor" and can be found here
(https://github.com/Intevation/event-processor)
* We did not start with support for IODEF or X-ARF yet.
=== Status report CERT.at developments ===
* we moved to python3 only. Intelmq dropped python2 support
(https://github.com/certtools/intelmq/commit/2cbb42f1458a7e90539a443ec5e50ee…).
This does not apply yet to the certat repo (github.com/certat/intelmq), which
still supports python2.7 but only for the intelmqcli tool.
* New active contributor: pedro m. reis! Welcome and thanks for working so
hard on the Bitsight collector
(https://github.com/certtools/intelmq/pull/493)
* intelmqcli tool now supports a lot of new flags:
https://github.com/certat/intelmq/issues/52 This was necessary for CERT.at
since we use intelmqcli via cron job to connect to the (postgresql) eventDB ,
pull out all of the new data and use RT (ticket system) to send stuff out.
Added flags --quiet --batch. Now intelmqcli sends via cronjob.
These flags now allow CERT.at to run intelmq in full auto-mode! intelmqcli is
started via cron and sends out all events to all ISPs.
=== Requests ===
* Intevation searches for testers for the packages.
* We'd like to have some nice graphs in the intelmq-manager: events/sec , parse-failures/sec, etc.
* implementation of whitelisting of events (filter out events based on whitelists). See
https://github.com/certtools/intelmq/issues/426
* A good CSS design for the web page
=== Community ===
* RIPE abuse-c contacts can be done locally. RIPE might be able to export
abuse-c infos publicly (fingers crossed).
* more command line options for intelmqcli (see the
https://github.com/certat/intelmq repo)
* Aaron gave a presentation at the ENISA workshop "CSIRTs in Europe", 11th of May.
Slides will be shared on the ENISA page.
==== intelmq.org ====
The website intelmq.org is now online, but we would like to have more content and a proper
design.
Do you want to contribute to intelmq, but you are not a programmer? This is
your chance!
Current ToDos:
* Create Website Content: How-Tos / Installation Instructions, Success
Stories
** How-Tos / Instructions: If you are using a special feature of IntelMQ, for
instance an expert bot, try to find some time to write down a short article
how you managed to get it to work and why you are using it.
* Website Design
== Wishlist ==
* **we need more test-cases!!!**
* a specific config logic for ASNs: do this and that (for example sett ttl =
1 month) if event is in ASN xyz. Or "ignore" if event is in ASN xyz. This
should support some kind of more-specific-less-specific inheritance,
similarly to Apache directory settings. The most specific setting wins. The
order could be: country code -> ASN -> netblock -> ip (/32). Open questions:
what's more relevant if both domains and numbers (ASN, IPs, net blocks) exist
in an event?
* block based processing: for example block based team cymru lookups
* parallelisation: We need to revisit this topic
== Important Discussions ==
In case you missed something, here are the headlines of some discussion we
consider interesting / important.
=== Mailing Lists ===
* [Intelmq-dev] Packaging Strategy for Bots with dependencies
* [Intelmq-dev] Discussion on intelmq output / transformation architecture
* [Intelmq-dev] Output format to syslog/splunk (PR#503)
== Communication ==
Chat: irc #intelmq on freenode or webchat:
[[https://webchat.freenode.net/?channels=intelmq]]
Follow on twitter: @intelmqorg
Weekly Conference Call every Tuesday: Dial in via the known conference bridge number. It is
[[https://en.wikipedia.org/wiki/Telephone_number_mapping|ENUM]] enabled. Ask
Aaron or Dustin for the number if you want to participate.
Dear contributors and users,
In the last year there have been so many changes to intelmq and now we
are well on the way towards a stable release.
We propose to make *Feature Freeze* *and* a *first release candidate in
October* so we can focus on heavy testing and bug fixing.
This is also the starting point for a better branching strategy:
https://github.com/certtools/intelmq/issues/650
Then, when all issues are fixed, we can do the release of version 1.0.
Also, we plan to publish packages for some distributions, .deb as well
as rpm. Currently there are still some open questions. But the
installation via pip will be no longer the recommended for production
environments.
To help, you can look at the open issues for 1.0:
https://github.com/certtools/intelmq/issues?q=is%3Aopen+is%3Aissue+mileston…
Tough I am assigned to most of them, you are encouraged to comment,
review or fix some of the tasks listed there!
The intelmq-manager needs some work to get partially compatible and
*much work* to get it "release-ready", basically a complete rewrite. See
https://github.com/certtools/intelmq-manager/issues/80
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201
// CERT Austria - http://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear IntelMQ-Devs,
whilst analysing our current setup and possible requirements, we discovered
that an aggregation of events within IntelMQ might be a reasonable thing to
do.
In [1] I sketched how this might work. I'd be glad if you could find the time
to comment on this approach and participate in the discussion.
[1] https://github.com/certtools/intelmq/issues/751
Best regards
Dustin
--
dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF
Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998
Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
Hi IntelMQ-Devs,
when writing tests what is your development suggestion for
writing similiar tests?
My problem:
intelmq/lib/test.py test_bot_name()
checks that "Test class name must be Test{botclassname}."
But sometimes it would make sense to have a second test class
within one test file. For example in
intelmq/tests/bots/experts/modify/test_expert.py
I am reusing some of the data of this tile.
On this occasion I have used a different class name and
explicitely disabled the test_bot_name().
You can reach the code via this search:
https://github.com/certtools/intelmq/search?utf8=%E2%9C%93&q=test_bot_name
I can see some merit checking for a consistent naming.
However making it easier to write more tests is also a good aim
and it looks like a good pattern to write more than one test class
within one file.
What is your idea of solving this in the future?
(I meanwhile add another test using the same trick.)
Best Regards,
Bernhard
--
www.intevation.de/~bernhard +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
Hi,
Sebastian and me recently had a meeting at CERT.at - discussing the minimal feature set that we still want to have (from our perspective) for putting a "1.0" label on it.
We came up with a list. I am now prioritising this list.
I took a trick which I learned from another software project and assign priorities to the issues I create in github in the form of Fibonacci sequence numbers: 1,2,3,5,8,13. The benefit of this is that an intuitive value of 13 is really as valuable/high prio as the sum of the previous two priorities.
It's rather a bit geeky, but it seems to work.
Example: https://github.com/certtools/intelmq/issues/722
Anyway. Just letting you know (so that you don't wonder what this is). This is merely to be read as hint as what I see as high prio.
Feel free to adapt the prios or discuss here.
Best,
a.
Hi *,
as of the discussion in https://github.com/certtools/intelmq/issues/657, I renamed the classification.identifier of the shadowserver snmp to "opensnmp".
This might have consequences in your eventDB if you store these events and group them by "classification.identifier".
Please (SQL-) UPDATE your previous entries in the eventDB in that case.
Thanks,
a.