IntelMQ-dev

intelmq-dev@lists.cert.at

1 participants
244 discussions

Help for fixing configuration of intelmq-manager 3.0.1-1
by Mika Silander 03 Nov '21

03 Nov '21

Hi, Seems I've gotten stuck on the same problem than before. However, this time around the order of installation was (on a Ubuntu 20.04 LTS/Apache 2): apt-get install intelmq apt-get install intelmq-api apt-get install intelmq-manager and the versions were: intelmq 3.0.2-1 intelmq-api 3.0.1-1 intelmq-manager 3.0.1-1 I'm able to access the intelmq-manager start page, https://ourimq.domain/intelmq-manager/index.html but when opening the Configuration tab https://ourimq.domain/intelmq-manager/configs.html, expected frames appear but no bots are shown in the frame (normally) allowing editing of the bot network. Instead an endless loop starts with the error shown further below. All hints for troubleshooting & debugging are welcome. Br, Mika P.S: No SELinux lurking in the shadows on the install host. --- 16:42:30 115× Error loading bot queues information: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> <hr> <address>Apache/2.4.41 (Ubuntu) Server at ourimq.domain Port 443</address> </body></html> Not Found 16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/positions . 16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/runtime . 16:40:37 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/bots

2 6

Potential attribute name clashes in bot parameters in intelmq 3.1.x ?
by Mika Silander 02 Nov '21

02 Nov '21

Hi, At long last I updated my develop branch from the 2.3. days to 3.1.0 and noticed my tests fail due to the fact that bots don't have anymore a "parameters" attribute. At a closer look it seems all parameters from runtime.yaml are turned into direct attributes of self (correct?), e.g. if one defines a conf parameter "myparam", this shows up in the bot as self.myparam and not accessible as getattr(self.parameters, "myparam") as it did before. After experimenting with my own bot tests I ended up in a situation where there's a potential attribute name clash. Assume a bot is tested like self.input_message = some_event_here self.run_bot() self.assertSomethingHere() In the above test, the bot gets initiated with a big number of default attributes, e.g. accuracy, group, enabled, logger, run_mode etc etc. Assume then that as a developer I want to use parameters with matching names for the needs of my own bot like self.input_message = some_event_here self.run_bot(parameters={ 'group': 'Plumber', 'run_mode': 'disruptive' }) self.assertSomethingHere() I can set those parameters and they happily override the defaults and I imagine this can mean trouble ahead. Is this intended behaviour and if yes, is there a way to prevent it? With the earlier self.parameters construct this did not happen and one could have e.g. a "group" attribute for the needs of intelmq's internal operation (self.group) separate from a bot developer's "group" attribute (that ended up under self.parameters) - no clashes despite equal names. Thus, to avoid the above in intelmq 3.0.x this means every bot developer needs to check her own bot won't use any of the 'reserved' attributes before defining their own ones, right? I might and I definitively hope I'm wrong here but please let me know if this is the case. Br, Mika

4 8

Systemd startup scripts
by Mika Silander 02 Nov '21

02 Nov '21

Hi, Afaik the bots can currently be managed with the command intelmqctl only. Has anyone implemented systemd startup scripts for intelmq? Br, Mika

3 5

Building packages for intelmq 3.0.2 stumbles on two tests
by Mika Silander 29 Oct '21

29 Oct '21

Hi, Tried building the intelmq debian packages on a Ubuntu 18 LTS, python 3.6 with dpkg-buildpackage -us -uc -b -d Branch was maintenance, tag 3.0.2 and the option -d above was added because Cerberus 1.3.2 was installed with pip3 and not with apt from python3-cerberus. Seems two tests make the build fail, see below. A known problem? Any hints for troubleshooting this? Airing this here before starting to debug on my own ... Cheers, Mika ====================================================================== FAIL: test_multiple_bots (intelmq.tests.bots.collectors.tcp.test_collector.TestTCPCollectorBot) Let's simulate multiple IntelMQ instances are pushing the events at once! ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/mika/intelmq/intelmq/tests/bots/collectors/tcp/test_collector.py", line 175, in test_multiple_bots self.run_bot(iterations=client_count + 1) File "/home/mika/intelmq/intelmq/lib/test.py", line 361, in run_bot self.assertLessEqual(len(re.findall(' - ERROR - ', self.loglines_buffer)), allowed_error_count) AssertionError: 2 not less than or equal to 0 ====================================================================== FAIL: test_random_input (intelmq.tests.bots.collectors.tcp.test_collector.TestTCPCollectorBot) Check how we handle a random input, coming from an unknown source. We should put all the data to report['raw']. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/mika/intelmq/intelmq/tests/bots/collectors/tcp/test_collector.py", line 106, in test_random_input self.run_bot() File "/home/mika/intelmq/intelmq/lib/test.py", line 363, in run_bot self.assertLessEqual(len(re.findall(' - WARNING - ', self.loglines_buffer)), allowed_warning_count) AssertionError: 1 not less than or equal to 0 ---------------------------------------------------------------------- Ran 1404 tests in 125.121s FAILED (failures=2, skipped=274) Test failed: <unittest.runner.TextTestResult run=1404 errors=0 failures=2> error: Test failed: <unittest.runner.TextTestResult run=1404 errors=0 failures=2> E: pybuild pybuild:336: test: plugin distutils failed with: exit code=1: python3.6 setup.py test dh_auto_test: pybuild --test -i python{version} -p 3.6 returned exit code 13 debian/rules:23: recipe for target 'build' failed make: *** [build] Error 25 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

2 4

IEP04: The choice of the UUID-format
by Sebastian Wagner 16 Sep '21

16 Sep '21

Dear allies, The discussion around the IEP04 proposal, adding meta-information to IntelMQ messages, has stalled over the last months - first because of the time-intensive IntelMQ 3.0 release preparations and then because of the vacation season. Here is the current proposal: https://github.com/certtools/ieps/tree/main/004#readme Aaron, Sebastian Waldbauer and myself worked on it over the summer and also identified two open issues to be discussed: 1. The exact format of the meta-information and how to name and structure the fields. AIL made the first move and now uses a format similar to the previously proposed Variant "A". The IEP04 document contains the current proposal which is in line with the AIL format: https://github.com/certtools/ieps/tree/main/004#user-content-variant-ail If there are no other proposals, this will most probably the way to go. 2. The format of the UUID format which we want to uniquely identify IntelMQ events. We don't necessarily need to use the UUIDv4 format which represents pure randomness, but also other options which include the time and are even /time-sortable/. Sebastian Waldbauer analysed a couple of options and summarised his results in this document: https://github.com/certtools/ieps/blob/main/004/UUID.md Please let us know your opinion on the different UUID options. cheers Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

4 10

IntelMQ 3.0.2 release fixing performance issues
by Sebastian Wagner 10 Sep '21

10 Sep '21

Dear community, Last week I said in the announcement for 3.0.1, that we didn't plan any further bugfix release before 3.1.0. I'm now proving myself wrong. Narzhan reported general performance issues and we investigated the problem on our own instances. We found two problems which different effects. It could be the case that you didn't even notice any performance drop, as both bugs affected only certain bots or situations. - When any collector creates a new report, it initializes an object of type Report. Since five years there was a bug, that this always loaded the harmonization.conf, instead of loading it just once at the start (which is - correctly - the case for new events). This was never a problem, as the Python's built-in json module was fast enough, but ruamel.yaml is slower. If a collector didn't massively create reports, that bug was not obvious, especially under testing conditions. And the bug was only relevant, if stream data is processed (for example with the Shodan stream collector). - Two bots, the STOMP collector and the API collector, don't have a proper process method, because they are using threading. A thread retrieves data asynchronously and sends the data. Normally, the process-method is running in an endless-loop, with sleeping rate_limit seconds between. If rate_limit is 0 or not defined, this would result in high load, therefore these bots defined the __collector_empty_process attribute which should have prohibited the endless loop. But due to class-restructuring, the attribute which would have been to be set was _Bot__collector_empty_process, and the flag was not effective. That did not only cause a loop, but as bot statistics are written to redis after every process-call, that caused continuous connections to redis. We've now renamed the flag to _collector_empty_process and fixed the two bots. The flag __is_multithreadable was affected analogous, but that should not have cause performance issues in particular. Additionally, the bot statistics can now be disabled by setting |statistics_host| to null (previously this caused an exception).| | The new version is on GitHub and PyPI, very soon in the deb/rpm repositories. The Docker image will follow on Monday or earlier. You can read the full changelog here: https://github.com/certtools/intelmq/releases/tag/3.0.2 have a nice weekend Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ & API & Manager release 3.0.1 and upgrade note
by Sebastian Wagner 02 Sep '21

02 Sep '21

Dear community, Over the past two months, IntelMQ contributors had no summer pause, but did the final finish for IntelMQ 3.0. A special thanks goes to Mikk Margus Möll (CERT.ee) who has put tremendous efforts in the IntelMQ Manager tackling structural and usability issues, mainly in the JavaScript-components! The deb/rpm repositories did not receive the 3.0.0 release at beginning of July to get more experience with the major changes before doing automatic upgrades, but now they deliver the brand-new 3.0.1 version. Please note, that the automatic upgrade procedures may still not be fully smooth. Just now, we have noticed, that the packages contain a small flaw, which harms the upgrade experience: The packages ship a default configuration (the file is now called `runtime.yaml`), but only if the file does not exist before - for new installations. But now in this special case, we renamed the configuration from `runtime.conf` to `runtime.yaml` and therefore, the new - default shipped - configuration takes precedence. I hope the following commands and hints will be of help to you. # remove the runtime configuration shipped by the package (can be called /etc/intelmq/runtime.*) and rename your original one to /etc/intelmq/runtime.yaml # the previously used runtime.conf can be used as drop-in to runtime.yaml (YAML is backwards-compatible with JSON) sudo -u intelmq intelmqctl upgrade-config -f -u v300_pipeline_file_removal sudo -u intelmq intelmqctl upgrade-config -f -u v300_defaults_file_removal sudo -u intelmq intelmqctl upgrade-config -f -u v301_deprecations The last three steps are important to merge the defaults and pipeline configuration into the new combined configuration file Please do not hesitate to ask. The deb-packages are also already available for the newly released Debian 11 Bullseye. We are not planning a bugfix release until the 3.1.0 release, so that one will be the next version to be released. Here's a short summary of what happened during the summer: - various fixes related to the IEP001 implementation (IEP001 was the change configuration format and merge of files, rewrite oft the internal parameter-handling) - removal of the malwaredomains feed and parser, because it does not exist anymore - Various fixes in the Shadowserver Parser and support for new reports: Vulnerable SMTP Server, Microsoft Sinkhole Events Report & Microsoft Sinkhole HTTP Events Report, Honeypot HTTP Scan - SMTP Output: Added Content-Disposition-header to the attachment, fixing the display in MS Outlook clients (as reported and dicussed on the Mailinglist). - Heavy refactoring of IntelMQ-Manager's JavaScript parts to fix errors and usability issues. If you are interested in developing on IntelMQ and you don't know where to start, have a look at the dev guide an the issues labeled "good first issue": https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3A… We are especially welcoming contributions to the documentation! You can read the full changelogs here: - https://github.com/certtools/intelmq/releases/tag/3.0.1 - https://github.com/certtools/intelmq-api/releases/tag/3.0.1 - https://github.com/certtools/intelmq-manager/releases/tag/3.0.1 https://cert.at/en/blog/2021/9/intelmq-301-release https://twitter.com/CERT_at/status/1433475188381806594 btw: There's new contact management portal called "tuency" for administrating abuse contacts available, which can be used in conjunction with IntelMQ. Read more about its features here: https://cert.at/en/blog/2021/9/tuency-constituency-portal-for-iocs-and-certs https://gitlab.com/intevation/tuency/tuency -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ & API & Manager Release 3.0
by Sebastian Wagner 05 Aug '21

05 Aug '21

Dear community, The time has come and IntelMQ 3.0 is final! We, as community, made loads of changes, smaller and bigger ones, and I really think that IntelMQ became more user-friendly, developer-friendly and feature-rich at the same time! There are some major changes in this release, especially the in the field of the configuration and Internal Data format (previously: "harmonization"). For the configuration-part, the upgrade part should be automatic with `intelmqctl upgrade-config` as usual. For the Data format, carefully look at your bot configurations (filters, sieve, etc.) to update them. Adaptions in systems connected to IntelMQ, especially also databases might be necessary as well. The NEWS.md file give a summary of what has changed: https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-… We don't recommend to upgrade existing production instance of IntelMQ yet. We of course did testing, including the end-to-end tests, and have detailed release notes. But for critical systems, a delayed upgrade makes sense ;) Therefore the stable deb/rpm repositories don't contain the 3.0 release yet! Even though an upgrade of production systems is not yet recommended, extensive usage and testing of the new releases are very much welcome and required to get the necessary feedback for the next (maintenance) releases. The releases are available via git, PyPI, Docker and the *unstable* deb/rpm repositories. Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html IntelMQ API documentation: https://intelmq.readthedocs.io/en/maintenance/user/intelmq-api.html IntelMQ Manager documentation: https://intelmq.readthedocs.io/en/maintenance/user/intelmq-manager.html NEWS/release notes of IntelMQ (Core): https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-… Full Changelog of IntelMQ (Core): https://github.com/certtools/intelmq/blob/maintenance/CHANGELOG.md#300-2021… On a high level, these are the major changes compared to version 2.3.x (2.3.3 was released 2021-05-31): In the core and Docker: * Configuration rewrite including parameter loading and handling (IEP01), plus the required adoption of the API and Manager, by Birger Schacht (CERT.at). * Classification sync with RSIT, by Sebastian Wagner (CERT.at). * Removal of the BOTS file, by Sebastian Waldbauer (CERT.at). * Creation and maintenance of the Docker images by Sebastian Waldbauer (CERT.at). * Creation of Docker-instructions for development setups by Einar Lanfranco and Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar). New and majorly enhanced bots: * Added |intelmq.bots.collectors.fireeye|: A bot that collects indicators from Fireeye MAS appliances (PR#1745 by Christopher Schappelwein). * |intelmq.bots.collectors.api.collector_api|: Added UNIX socket capability (PR#1987 by Mikk Margus Möll, fixes #1986). * Added |intelmq.bots.parsers.fireeye|: A bot that parses hashes and URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein). * Added |intelmq.bots.experts.http.expert_status|: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR#1789 by Birger Schacht, fixes #1047 partly). * Added |intelmq.bots.experts.http.expert_content|: A bot that fetches an HTTP resource and checks if it contains a specific string (PR#1811 by Birger Schacht). * Added |intelmq.bots.experts.lookyloo.expert|: A bot that sends requests to a lookyloo instance & adds |screenshot_url| to the event (PR#1844 by Sebastian Waldbauer, fixes #1048). * Added |intelmq.bots.experts.rdap.expert|: A bot that checks the RDAP protocol for an abuse contact for a given domain (PR#1881 by Sebastian Waldbauer and Sebastian Wagner). * |intelmq.bots.experts.sieve.expert|: Major refactoring and lot's of new functionality New operators for working with various types (lists, sets, booleans, float, int), generic rule negation and nesting (PR#1895 by Mikk Margus Möll). * Added |intelmq.bots.experts.uwhoisd|: A bot that fetches the whois entry from a uwhois-instance (PR#1918 by Raphaël Vinot). * Added |intelmq.bots.experts.aggregate|: A bot that aggregate events based upon given fields & a timespan. (PR#1959 by Sebastian Waldbauer) * Added |intelmq.bots.experts.tuency|: A bot that queries the IntelMQ API of a tuency instance (PR#1857 by Sebastian Wagner, fixes #1856). * Added |intelmq.bots.outputs.templated_smtp| (PR#1901 by Karl-Johan Karlsson). On the documentation front, these are the most important changes * License and copyright information was added to all the bots (by Birger Schacht). * Added documentation on the EventDB (PR#1955 by Birger Schacht, PR#1985 by Sebastian Wagner). * Added TimescaleDB for time-series documentation (PR#1990 by Sebastian Waldbauer). * n6 interoperability documentation: Adding more graphs and illustrations (PR#1991 by Sebastian Wagner). * Added documentation on abuse-contact look-ups (PR#2021 by Sebastian Waldbauer and Sebastian Wagner). And not to forget all the smaller changes and additions. Thanks to (in random order) Raphaël Vinot (circl.lu) Bernhard Reiter (intevation.de) Sebastian Wagner (CERT.AT) Filip Pokorný (CSIRT.CZ) Guillaume GRANJON de LÉPINEY (CERT XLM excellium-services.com) Mikk Margus Möll (CERT.ee) Alex Kaplan Thomas Hungenberg (CERT-BUND.DE) Einar Lanfranco (CERT-UNLP cert.unlp.edu.ar) Christopher Schappelwein (milCERT, BMLV.gv.at) Marcos Gonzalez (CSIRT-RD cncs.gob.do/csirt-rd/) Marius Karotkis (NRDCS.LT) Sebastian Waldbauer (CERT.AT) Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar) Karl-Johan Karlsson (Linköping University LIU.SE) Birger Schacht (CERT.AT) ... and all the contributors of previous releases and as well to all reporters, supporters, etc! best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

How does one avoid reprocessing mail attachments already processed?
by Mika Silander 04 Aug '21

04 Aug '21

Hi all, We are using the mail collector bot to read email attachments over IMAPS. It works fine but rereads all the messages in the message folder even if it had processed them already. Is there a way to configure the mail bot to remove the messages once it has processed them? I didn't find a hint concerning this in the bot's documentation nor the sources. What approaches do people use to avoid reprocessing messages? Br, Mika

2 3

Path/permission problem with intelmq-manager (v2.3.1-2) ?
by Mika Silander 02 Aug '21

02 Aug '21

Hi all, Returning to an earlier problem that surfaced after having updated intelmq to version 2.3.3-1: intelmq-manager starts, but when opening the configuration tab, the network of bots doesn't get rendered. Looking at the output of the browser's developer console one can see successful retrieval of https://ourhost/configs.html that in turn initiates the successful retrieval of a number of CSS, PNG and JS files. Next we have five failures (404 Not found): https://ourhost/intelmq/v1/api/config?file=bots https://ourhost/intelmq/v1/api/config?file=defaults https://ourhost/intelmq/v1/api/config?file=runtime https://ourhost/intelmq/v1/api/config?file=pipeline https://ourhost/intelmq/v1/api/config?file=positions Then a few more successful retrievals of PNGs and a GIF. After this we end up in an infinite loop that tries unsuccessfully (404 Not found) to fetch https://ourhost/intelmq/v1/api/queues-and-status once every second. The error the browser shows repeatedly is "Error loading bot queues information:". What is common to all failures above is that they seem to be due to jquery-actions. Any ideas as to where the problem could be? I suspect some path and/or permission problem but that's the first guess. Br, Mika For reference, these are the package versions on our intelmq Ubuntu 20 LTS host: intelmq 2.3.3-1 intelmq-api 2.3.1-1 intelmq-manager 2.3.1-2

2 3

← Newer
1
...
4
5
6
7
8
9
10
...
25
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev