IntelMQ-dev

intelmq-dev@lists.cert.at

3 participants
245 discussions

Fwd: [Intelmq-users] openbl.org down -> bot removal
by L. Aaron Kaplan 27 Jun '17

27 Jun '17

Reminder... we now also have an intelmq-users(a)lists.cert.at mailing list. This is intended for users of the system. Feel free to join (even if you are on the -dev list) Best regards, a. > Begin forwarded message: > > From: Sebastian Wagner <wagner(a)cert.at> > Subject: [Intelmq-users] openbl.org down -> bot removal > Date: 26 June 2017 at 17:37:10 GMT+2 > To: intelmq-users(a)lists.cert.at > > Hi, > > OpenBL.org shut down their services at end of May. I removed the OpenBL > parser now. > > Sebastian > > https://github.com/certtools/intelmq/issues/1018 > https://github.com/certtools/intelmq/commit/d677112a02a016d097c224866e78524… > > -- > // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 > // CERT Austria - https://www.cert.at/ > // Eine Initiative der nic.at GmbH - https://www.nic.at/ > // Firmenbuchnummer 172568b, LG Salzburg > > > -- > Listen-Einstellungen: > https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users -- // L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Working on 1.0
by Sebastian Wagner 14 Jun '17

14 Jun '17

Dear community, We now have a pretty stable codebase and I think we should take this chance to finally release our first stable version 1.0 and implement upcoming bigger changes afterwards for 1.1 My plan for the next days would be: * to merge PR #974[1] which fixes our last feature request #743[2] for 1.0 * work on PR #853[3] * release a beta version Then we can work on the remaining bugs, and afterwards release the first RC which should then be thoroughly tested. This process also implies some changes to the workflow concerning e.g. the branches and packages. I will inform you later about these (basically it is described in #650[4]). Sebastian [1]: https://github.com/certtools/intelmq/pull/974 [2]: https://github.com/certtools/intelmq/issues/743 [3]: https://github.com/certtools/intelmq/pull/853 [4]: https://github.com/certtools/intelmq/issues/650 P.S.: If you have experience with CentOS, the intelmq-manager needs your help! -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

Destination host in malware feeds
by Thomas Hungenberg 14 Jun '17

14 Jun '17

Hi all, next to the destination IP and port, malware sinkhole feeds commonly also include information on the destination 'host'. This value is usually taken from the 'Host:' header of the HTTP request sent to the sinkhole by the infected client. The corresponding columns in the feeds are usually called 'dsthost', 'http_host' (e.g. Shadowserver HTTP-Sinkhole-Drone) or 'cc_dns' (Shadowserver Botnet-Drone). The name 'cc_dns' is probably misleading. In most cases, the value of the destination host field in the feeds is a FQDN - but depending on how the malware sets up the "Host:" header in the HTTP requests, it may also be FQDN:Port, an IP address or IP:Port (or just some crap). Some malware families perform a DNS lookup for the (DGA based) C2 domain first and then only use the IP address in the HTTP requests instead of the domain name. I noticed that the IntelMQ parsers for the Shadowserver feeds simply try to map the value of 'http_host' to 'destination.fqdn', e.g. https://github.com/certtools/intelmq/blob/master/intelmq/bots/parsers/shado… Line 295: ('destination.fqdn', 'http_host'), The scripts we currently use for generating notifications to network owners use the following logic (pseudo code here) to extract the destination host information: if (feed.dstip is a valid ip) { destination.ip = dstip } else { discard event # destination IP is required } dsthost = feed.http_host dsthost =~ s/:[0-9]+$// # remove potential port information (we already have this information in dstport) if (dsthost is a fqdn) { destination.fqdn = dsthost } else { do nothing # dsthost is either an IP (which we already have in destination.ip) or useless data } I'd like to suggest using a similar logic in the InteMQ parsers for the Shadowserver (an other malware sinkhole) feeds to not miss destination.fqdn information which comes in the form FQDN:Port - instead of writing the information to an extras field which is most probably not used by the output generators or even dropping the complete event. - Thomas CERT-Bund Incident Response & Malware Analysis Team

4 6

new intelmq-users mailinglist
by Sebastian Wagner 14 Jun '17

14 Jun '17

Dear contributors, As the intelmq project gains momentum more and more users are using the software. Also, not all users are developers anymore. To address the users need of support/assistance/help we created a new mailinglist intelmq-users. Please subscribe here: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users The archive is public. I also hope that with this step the Github Issues are only used for bugreports and not questions. best wishes, Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Proposal (Request For Comments) - IntelMQ with Run Modes & Process Management
by Tomás Lima 05 May '17

05 May '17

Hi IntelMQ-ML, Before open the discussion, I think is important to write a few notes: 1) The topic that we are opening to discuss have been discussed between me, Aaron and Sebastian in order to understand the impact of the possible changes, the effort require, the complexity, the global perspective of how should be implemented, etc... Each of us has now an idea and perspective about it and is crucial to have the community involved from now on in order to agree on the way to proceed. 2) The proposal that will be shared here is my own perspective but NOT only my work because a lot of the structure and technical details was only possible with Sebastian and Aaron contributions ( Thank you Aaron and Sebastian ), even if in some specific details they might see that there is a space to do in other/better way. This thread will be a good place to listen everyone's perspective. :) 3) IMPORTANT: This proposal is just a proposal and does NOT mean that will be implemented in this way... it's only a base to discuss if it helps. About the Proposal: --------------------------- The proposal is available on the following link and tries to be clear for the readers although its possible to have some hide details (by mistake) that will raise some questions. Proposal: https://github.com/SYNchroACK/intelmq/blob/proposal/docs/proposal.md Reasons why we are starting this discussion is because two main things (I guess): 1) There is a need to configure bots to only execute in a specific time, therefore, it seems that there is a requirement to configure a bot in different run modes, in this proposal: scheduled and continuous. (see proposal for more details). 2) IntelMQ is now being used by multiple teams and it requires stability during execution time, etc... it seems that there is a need for integrations with tools like systemd. Please, if you have time, read the proposal and write to the mailing-list your thoughts spitted by "What you like" and "What you don't like". I hope that I didn't have forgot to mention something important... :) Thank you in advance, Regards

7 14

Setups with EMail output and Webfrontend
by Bernhard E. Reiter 24 Apr '17

24 Apr '17

Hi friends of IntelMQ, at https://github.com/Intevation/intelmq-mailgen-release you will find an IntelMQ setup for automated use in case a more complex contact database for sending out events via email (and other means) is necessary. Our working label for the combination is **intelmq-cb-mailgen**. The setup requires PostgreSQL. And we have added a web-frontend called "Fody", that is a single page webapplication offering an interface to maintain contact entries and do some queries. Screenshots Fody v0.4.2: https://github.com/Intevation/intelmq-fody/issues/49 Overall the software is in beta state and serves all use cases that we aim for, right now. As always with software, there are many ways how to improve the system from here and we'll do some of them in the next days. (You may know me and my colleagues from Intevation because of our contract work getting IntelMQ improved for CERT-Bund. Aaron and Sebastian from CERT.at are helping this contract with some consulting as well. This intelmq solution is a Free Software result of this contract.) We appreciate any feedback! Best Regards, Bernhard -- www.intevation.de/~bernhard (CEO) +49 541 33 508 3-3 Intevation GmbH, Osnabrück, Germany; Amtsgericht Osnabrück, HRB 18998 Owned and run by Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

Run modes + Systemd + Crontab
by Tomás Lima 20 Apr '17

20 Apr '17

Folks, We have been thinking about things to improve IntelMQ in order to have it really suitable for production... and now we want to raise the discussion about the following ideas or other related ideas that you might think of: - Run modes: stream (always running, e.g. shadowserver-parser bot) or scheduled (run once every day at 10:00AM, e.g. shadowserver-chargen-collector bot) - Systemd: take care of bots which are running permanently. Use systemd to manage bots in stream run mode that cannot stop and also, do a better job with managing which bots processes which are really running, not using the current implementation with pidfiles. - Crontab: manage bots that need to run once, e.g. at midnight every day. Use crontab to manage bots in scheduled run mode to just run once on a pre-defined scheduled time. Why these ideas came up? Mainly because of two things: 1. Run modes are needed due the different purposes of a bot which is collecting a real-time feed and a bot that is collecting shadowserver reports once in a day. 2. Systemd and Crontab are two well-know tools that we all know about it and IntelMQ seems to need them since we have seen some problems with managing bots processes (running or not) and report automatically failures by email, just to give some examples. Until now, we don't have any of these features implemented, we only have developed some "rubbish" code in order to do some Proof-of-concept but nothing serious. So, please guys, share your thoughts about these ideas or others related in order to define, as a community, a strategy for the next month. Thank you. Cheers!

3 7

How do you notify ISPs/network owners about accessible (open) devices?
by L. Aaron Kaplan 24 Mar '17

24 Mar '17

Hi everyone, I have a question. We are processing nearly all the shadowserver feeds now (VNC is still missing) and we stumbled across a problem that we can not 100% solve currently: how do you deal with 'accessible and only potentially vulnerable' devices? Let me elaborate. Usually we sent out notifications on vulnerable devices (ENISA taxonomy: "Vulnerable". Examples: open recursive DNS, open NTP, anything mis-useable for UDP amplification attacks, etc). However, at some point, accessible and (only potentially vulnerable) devices came into the game. I.e. a device running telnet (or something on the telnet port). Or VNC. The VNC server might be protected by a pwd. So, how to deal with that? An ISP might rightfully say that this telnet port is there intentionally and we should not complain? So, we now have two types that we are talking about: 1. Vulnerable and openly accessible ports 2. Potentially vulnerable (but not proven) and accessible ports Candidates for the second type would be: * VNC * telnet * RDP * (maybe) Redis * (maybe) ES * (maybe) memcached * (maybe) Mongo What's your stance on this? How do you deal with it? Note that we are sending out * a lot* as a national CERT and we would not like an ISP to be swamped by our mails if it does not have to be the case. Best, a. -- // L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

5 4

heads up: new syntax for the modify bot
by L. Aaron Kaplan 01 Mar '17

01 Mar '17

Hi everyone, just to give you a heads up: we had a pull request from autumn 2016 which got finally merged into master https://github.com/certtools/intelmq/pull/745 It's about a new format to the modify bot. New format: https://github.com/certtools/intelmq/blob/f02998b2cacb0dfe3ea6eb65f9d0b42bb… (scroll down to "modify") I believe the new format is way more readable. But if you use the modify bot, please be aware of these changes when you pull in the new master branch. Best, Aaron. (And thx sebix for the code) -- // L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - http://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

IntelMQ Data Harmonization (DHO) - malware.hash key (issue 732)
by Tomás Lima 09 Feb '17

09 Feb '17

Folks, In the current DHO there are 3 fields related to malware hash (' *malware.hash*', '*malware.hash.md5*' and '*malware.hash.sha1*') but one of them ('*malware.hash*') is not compliant with the current internal message structure (technical details can be found on the issue 732 <https://github.com/certtools/intelmq/issues/732#issuecomment-269602721>). Since it's a bug that needs to be fixed and affects the DHO, I would like to propose the only three approaches that I see (maybe there are more...) to solve this issue and would like to have your feedback to achieve an agreement. *Approaches**:* 1. Rename the key 'malware.hash' to something like 'malware.hash.other' for situations where we see a feed providing a different type of hash 2. Remove the key 'malware.hash' and keep with the other two ones 3. Remove the keys 'malware.hash.md5' and 'malware.hash.sha1' and only use the key 'malware.hash' for all types of hash. With this approach, if the feed provides a md5 and sha1 hashes in the same event, we will not be able to store both. The chosen approach is the first one. If you have chance, please take some minutes to give your feedback in order to understand if everyone is comfortable with that. Thank you in advance. Cheers! -- Tomás Lima* , * »-«* SYNchroACK *»-«

8 14

← Newer
1
...
16
17
18
19
20
21
22
...
25
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev