Hi,
We need some consistent behavior for extracting files of downloaded
archives. For this, I'd like to hear some opinions from users. What do
you want to be able to configure, what should be done automatically? Do
you want it to be automatic and still have the possibility to override?
There are some possible settings:
* Do extraction at all
* What to extract? Some files vs everything. Can be combined with above
* archive type. Could be guessed from filename extension or mimetype.
The latter is as not trivial in python as I expected :/
Background:
The HTTP collector can currently extract files from zip-files on the
fly. There is no parameter for this, all files will be passed on as
separate reports.
The RT collector can extract zip on the fly if the parameter
`unzip_attachment` is true.
PR#1095[0] adds the ability to extract files for tar.gz archives
including a parameter `extract_files` to give a list of filenames to be
extracted. And all files will be extracted if the parameter is simply True.
Sebastian
[0]: https://github.com/certtools/intelmq/pull/1095
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear users and contributors,
Yesterday I release version 0.3 and today 0.3.1 (containing a fix for a
bug preventing the saving of files).
This release contains a lot of exciting usability fixes and
enhancements. See the changelog below for a full list. We are getting
close to a stable release now!
Please refer to the installation docs. Deb and rpm packages are
available. Note that for the deb-packages, you need to set group
permissions on the configuration files first.
This is the changelog of 0.3:
* Partly support for CentOS/RHEL 7 (#55, #103)
* Note on security considerations in Readme to avoid misunderstandings
* Show versions of intelmq and intelmq manager on about page
* Update vis.js to current version
### Configuration
* interface for defaults.conf (#45)
* drag&drop (#105, #41)
* fix #96
* save buttons starts blinking after changes (#41)
* Allow redrawing of botnet on demand
* Save/load position of bots in/from /opt/intelmq/etc/manager/positions.conf
File needs to be writeable
* parameters from defaults are shown for new bots (#107)
* parameters are grouped by type: generic, runtime, defaults
* better feedback on errors with backend (#69, #99)
* pressing ESC in forms equals to pressing the cancel button
* Edit node window is now much bigger
* pressing enter in 'add key' window equals to pressing ok button
### Management
* Reload and restart have been added as actions on bots and the whole
botnet (#114)
* A click on the bot name opens the monitor page of the bot
### Monitor
* clearing queues is possible in general and specific view for all
queues (#54)
### Backend
* Fix regex checks on bot ids and log line number in controller, they
have not been effective
* fix overflow in extended message box (#49)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
I just released the bug fix release IntelMQ 1.0.1.
The existing bugs which have not been fixed in time have been moved to
the 1.0.2 milestone (10 bugs).
For upgrade instructions look at the documentation:
https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md
Updated packages have been built and are available in the repositories.
From the changelog:
### Documentation
- Feeds: use more https:// URLs
- minor fixes
### Bots
- bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for
rest.db.ripe.net
- bots/outputs/file/output.py: properly close the file handle on shutdown
### Core
- lib/bot: Bots will now log the used intelmq version at startup
### Tools
- intelmqctl: To check the status of a bot, the comandline of the
running process is compared to the actual executable of the bot.
Otherwise unrelated programs with the same PID are detected as running bot.
- intelmqctl: the "enable", "disable", "check", "clear" commands now
support the JSON output
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
forwarding here
> Begin forwarded message:
>
> From: Sebastian Wagner <wagner(a)cert.at>
> Subject: [Intelmq-dev] Published release candidate 1.0.1
> Date: 23 August 2017 at 16:56:23 GMT+2
> To: "intelmq-dev(a)lists.cert.at" <intelmq-dev(a)lists.cert.at>
>
> I just published the release candidate for the next bugfix release
> 1.0.1. You can expect the final release next week / end of august.
>
> Changelog:
>
> ### Documentation
> - Feeds: use more https:// URLs
> - minor fixes
>
> ### Bots
> - bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for
> rest.db.ripe.net
> - bots/outputs/file/output.py: properly close the file handle on shutdown
>
> ### Core
> - lib/bot: Bots will now log the used intelmq version at startup
>
> ### Tools
> - intelmqctl: To check the status of a bot, the comandline of the
> running process is compared to the actual executable of the bot.
> Otherwise unrelated programs with the same PID are detected as running bot.
> - intelmqctl: enable, disable, check, clear now support the JSON output
>
> --
> // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
> // CERT Austria - https://www.cert.at/
> // Eine Initiative der nic.at GmbH - https://www.nic.at/
> // Firmenbuchnummer 172568b, LG Salzburg
>
>
> _______________________________________________
> Intelmq-dev mailing list
> Intelmq-dev(a)lists.cert.at
> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
--
// L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear colleagues and CERT-ies, dear abuse handling teams,
(sorry for big x-posting)
we are happy to announce (finally!) the official 1.0 release of our IntelMQ tool.
What is it?
===========
IntelMQ [1] is a free open source tool initially developed by CERT.pt and CERT.at to automatically handle and process the many incident reports (mostly shadowserver and similar) that we receive.
At CERT.at it processes many thousands of events per day.
But first of all, let me thank all the contributors and the different teams involved in this collaborative open source effort! Starting with CERT.pt (Tomas Lima and Mauro), BSI, Intevation (Bernhard, Dustin), CZ.NIC, CESNET, CERT Australia, CERT.ee, the IHAP [2] group and many many others. You coded, helped, discussed and attended the regular IHAP meetings which allowed us to discuss your wishes and requirements. This all - combined with the testing and coding efforts that many of you contributed - finally gave us version 1.0. today.
We counted at least 45 contributors.
IntelMQ has been running quite stable at CERT.at for nearly a year now and we are processing the bulk of the incident reports with it.
Of course, a 1.0 version always begs for some 1.0.1 bugfixes :) So therefore we would like to ask you to report any bugs or change requests on github's issue tracker [3].
Where can I get it?
===================
Follow the instructions in https://github.com/certtools/intelmq/tree/master/docs
Note that we also have (.deb, .rpm) packages for download. [10]
Future plans
=============
We now tagged the master branch "1.0.0". This will remain stable now.
We also started with a new "develop" branch which will become the 1.1 and 2.0 releases in the future.
You can read more about our branching strategy here [4]
Development will continue towards 1.1 with a set of wishes and requests that we received. You can view them in the issue tracker.
CSP integration
===============
Some of you already know that IntelMQ is a tool included into the "Core Service Platform" (CSP) as part of the CSIRT network [5].
We are very proud to offer our open source solution to the CSP.
Integration into your incident handling automation
==================================================
If you want to integrate IntelMQ into your incident handling automation environment, please note that you might want to use further tools such as "mail-gen" [6] or "intelmqcli" [7] (residing in separate repositories) which connect your ticket system (OTRS or RT) with IntelMQ.
In case you have questions, we have
* an IRC channel on freenode.net (#intelmq)
* a users mailing list [8]
* a developers mailing list [9]
Thanks again everyone who participated in this open source solution!
& feel free to (re-)tweet #intelmq
L. Aaron Kaplan and Sebastian Wagner,
CERT.at
[1] https://github.com/certtools/intelmq/
[2] incident handling automation project.
[3] https://github.com/certtools/intelmq/issues
[4] https://github.com/certtools/intelmq/blob/master/docs/Developers-Guide.md#r…
[5] https://www.enisa.europa.eu/news/enisa-news/2nd-informal-meeting-of-csirt-n…
[6] https://github.com/Intevation/intelmq-mailgen
[7] https://github.com/certat/intelmq
[8] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
[9] https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
[10] https://software.opensuse.org//download.html?project=home%3Asebix%3Aintelmq…
--
// L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi,
On 08/01/2017 06:44 PM, Vaclav Bruzek wrote:
> #opt/intelmq/etc# intelmqctl start xxx-parser
> intelmqctl: Running intelmqctl as root is highly discouraged!
> intelmqctl: Starting xxx-parser...
> intelmqctl: lastline-parser is running.
> #/opt/intelmq/etc# intelmqctl stop xxx-parser
> intelmqctl: Running intelmqctl as root is highly discouraged!
> intelmqctl: xxx-parser was NOT RUNNING.
Please do not run intelmqctl as root, you may get a lot of wrong
permissions. Please check them all in /opt/intelmq/ (and all
sub-directories) before continuing.
Then check the logfiles and - if that does not reveal anything -
otherwise start the both with `intelmqctl run bot-id`. See also
https://github.com/certtools/intelmq/blob/develop/docs/FAQ.md#my-bots-died-…
and
https://github.com/certtools/intelmq/blob/develop/docs/intelmqctl.mdDistrib…
ID: elementary
> Description: elementary OS 0.4.1 Loki
something base on 16.04
> intelmq (1.0.0.dev8)
You missed the rc1
> Intelmqctl is located in /usr/local/bin/ next to all intelmq.bots.*
> files (now with my custom bot file definitions)
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi,
The setup.py reads intelmq/bots/BOTS file during installation and
installs all bots as script. intelmqctl and others are installed in the
same manner.
Which operating system are you using? Which version of intelmq are you
using? Please show the output of `pip3 list`. Where is intelmqctl
located? Which other intelmq.* files are there? What was the complete
(!) output of the mentioned commands?
Without more information I really can't help. I am not a fortuneteller.
Unfortunately.
Sebastian
On 08/01/2017 10:43 AM, Vaclav Bruzek wrote:
> Hi,
> I've trid several methods of installtion:
> - pip3 install . (with and withou -e option)
> - python3 setup.py install
>
> with simmilar degree of success that is not having the deffinitions in
> /usr/local/bin/
>
> intelmqctl wtih bot id sais following:
> intelmqctl: Starting xxxx-parser...
> intelmqctl: xxxx-parser failed to START because the file cannot be found.
>
>
> On 17 July 2017 at 12:26, Sebastian Wagner <wagner(a)cert.at
> <mailto:wagner@cert.at>> wrote:
>
> Hi,
>
> On 07/14/2017 04:05 PM, Vaclav Bruzek wrote:
>> I had the definition in BOTS and configs, the files were there as
>> specified by the path. It was suggested to me that that the
>> definitons in /usr/local/bin/"bot path" were missing which indeed
>> was true but creating the respective files solved some issues and
>> created some more as it happens.
> You created the files /usr/local/bin/intelmq.bots... manually?
> Better let them created by installing intelmq with pip. After
> adding new bots, you need to run the installation method again. It
> creates the executables in the bin-directory for you.
> See also
> https://github.com/certtools/intelmq/blob/master/docs/Developers-Guide.md#u…
> <https://github.com/certtools/intelmq/blob/master/docs/Developers-Guide.md#u…>
>
>> The intelmqctl now recognizes the bots and runs them, but when I
>> try to stop the bot it displays that the bot wasn't running.
> What does `intelmqctl run bot-id` show?
>
> Sebastian
>
> --
> // Sebastian Wagner <wagner(a)cert.at> <mailto:wagner@cert.at> - T: +43 1 5056416 7201
> // CERT Austria - https://www.cert.at/
> // Eine Initiative der nic.at <http://nic.at> GmbH - https://www.nic.at/
> // Firmenbuchnummer 172568b, LG Salzburg
>
>
>
>
> --
> S pozdravem,
> Václav Brůžek
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
I've recently opened an issue (
https://github.com/certtools/intelmq/issues/1038) regarding my custom bots
which the intelmq was unable to find and run. I had the definition in BOTS
and configs, the files were there as specified by the path. It was
suggested to me that that the definitons in /usr/local/bin/"bot path" were
missing which indeed was true but creating the respective files solved some
issues and created some more as it happens. The intelmqctl now recognizes
the bots and runs them, but when I try to stop the bot it displays that the
bot wasn't running.
--
S pozdravem,
Václav Brůžek