IntelMQ-dev

intelmq-dev@lists.cert.at

3 participants
238 discussions

Alerting on repeated connection failures in a bot
by Mika Silander 25 Mar '22

25 Mar '22

Hi all, Calling out to the list again before trying to reinvent a wheel. We have a collector that tries to connect to the source of security events, but the source service is (temporarily) inaccessible. IntelMQ restarts the bot, an error message and the connection failure exception gets output to the log, and after a while, the bot gets restarted. For us it is acceptable there's a few failures in a configurable time frame, but if the situation prevails, we'd like to be alerted and if possible, prevent the bot from restarting. There are bot configuration parameters to stop a bot in case the processing of events fails repeatedly (error_procedure=stop, error_max_retries=suitable_number as in https://intelmq.readthedocs.io/en/maintenance/user/configuration-management…), but after some experimenting it appears they are not applicable here since we never reach the stage of processing events. Any ideas how to address this issue? We'd prefer not to touch the bot's source code. Br, Mika P.S: Setup is intelmq v3.0.2.

2 1

Fwd: Invitation to an IntelMQ in depth training on February 23th, 14:30h CET
by Otmar Lendl 03 Feb '22

03 Feb '22

FYI, and sorry if you receive this message multiple times. otmar -------- Forwarded Message -------- Subject: Invitation to an IntelMQ in depth training on February 23th, 14:30h CET Date: Thu, 3 Feb 2022 16:10:57 +0100 From: Otmar Lendl <lendl(a)cert.at> Dear members of the CSIRTs Network, CERT.at will host a presentation on IntelMQ as part of our training activities connected to the MeliCERTes 2 project. This presentation (and hopefully lively discussion ) will be on technical details: - IntelMQ Development, - Insights, - Core Knowledge The target audience are IntelMQ-devs & admins on an advanced level. Date/Time: 2021-02-23 14:30 - 16:00 UTC+1 (CET) Location: Online via Zoom: > https://us06web.zoom.us/j/84028475065?pwd=VmR3aUVrdGNFYk1EVTcxOEUvVHNFZz09 > Meeting-ID: 840 2847 5065 > Kenncode: 522946 Presenter: Sebastian Waldbauer Feel free to share this invitation to interested people. No registration is required. Regards, Otmar Lendl -- // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

1 0

Contribute by Reviews and Maintenance
by Sebix 17 Nov '21

17 Nov '21

Dear community, In my last mail on this list I already hided the message that I left CERT.at a short while ago and I was already asked what that means for the IntelMQ project. First of all: Don't panic! IntelMQ is and has always been a community project, with contributors from multiple teams, countries and continents[0]. In every release announcement I say thank you to the community for all the contributions and I really mean it. IntelMQ would never be that feature-rich without your involvement. All of us are part of this project, and joining our forces makes us strong. IntelMQ made various big steps in the last 7 years. It all started with Tomás Lima and Aaron Kaplan. The architecture was so robust, that it didn't change since then. I joined one year later and since then I almost memorized every line of code. In the last years I lost that ability as more and more code was coming for contributors, who are also maintaining code. And that's the essence: With IntelMQ growing, it's important to share the maintenance work. The IntelMQ project is looking for your contributions, including: - fixing open bugs[0] - reviewing open pull requests[1] - discussing and deciding when to make releases - testing before we do releases - documentation - and much more ;) IntelMQ is ready for the next evolutional leap forward and you are part of it! best regards Sebastian [0]: Maybe there will be a Moon- or Mars CERT in the not-so-distant future? ;) [1]: https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3A… [2]: https://github.com/certtools/intelmq/pulls

1 0

Redis log file gets wrong owner & group(?)
by Mika Silander 15 Nov '21

15 Nov '21

Hi all, Occasionally we see the /var/log/redis/redis-server.log file getting intelmq as its owner and group. This makes redis output to the log file fail. Once the owner is reset to redis and group to adm (on Ubuntu 20.04 LTS) and running systemctl restart redis, redis works fine. I've tried to debug the reason for this change in ownership in logrotate confs, intelmqctl sources etc but so far no luck. Hints as to the reason or how to troubleshoot are again welcome. Br, Mika P.S: We haven't configured redis to listen to unix sockets in /etc/redis/redis.conf so we assume https://intelmq.readthedocs.io/en/maintenance/user/FAQ.html#id4 does not apply to this case.

2 2

Help for fixing configuration of intelmq-manager 3.0.1-1
by Mika Silander 03 Nov '21

03 Nov '21

Hi, Seems I've gotten stuck on the same problem than before. However, this time around the order of installation was (on a Ubuntu 20.04 LTS/Apache 2): apt-get install intelmq apt-get install intelmq-api apt-get install intelmq-manager and the versions were: intelmq 3.0.2-1 intelmq-api 3.0.1-1 intelmq-manager 3.0.1-1 I'm able to access the intelmq-manager start page, https://ourimq.domain/intelmq-manager/index.html but when opening the Configuration tab https://ourimq.domain/intelmq-manager/configs.html, expected frames appear but no bots are shown in the frame (normally) allowing editing of the bot network. Instead an endless loop starts with the error shown further below. All hints for troubleshooting & debugging are welcome. Br, Mika P.S: No SELinux lurking in the shadows on the install host. --- 16:42:30 115× Error loading bot queues information: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> <hr> <address>Apache/2.4.41 (Ubuntu) Server at ourimq.domain Port 443</address> </body></html> Not Found 16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/positions . 16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/runtime . 16:40:37 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/bots

2 6

Potential attribute name clashes in bot parameters in intelmq 3.1.x ?
by Mika Silander 02 Nov '21

02 Nov '21

Hi, At long last I updated my develop branch from the 2.3. days to 3.1.0 and noticed my tests fail due to the fact that bots don't have anymore a "parameters" attribute. At a closer look it seems all parameters from runtime.yaml are turned into direct attributes of self (correct?), e.g. if one defines a conf parameter "myparam", this shows up in the bot as self.myparam and not accessible as getattr(self.parameters, "myparam") as it did before. After experimenting with my own bot tests I ended up in a situation where there's a potential attribute name clash. Assume a bot is tested like self.input_message = some_event_here self.run_bot() self.assertSomethingHere() In the above test, the bot gets initiated with a big number of default attributes, e.g. accuracy, group, enabled, logger, run_mode etc etc. Assume then that as a developer I want to use parameters with matching names for the needs of my own bot like self.input_message = some_event_here self.run_bot(parameters={ 'group': 'Plumber', 'run_mode': 'disruptive' }) self.assertSomethingHere() I can set those parameters and they happily override the defaults and I imagine this can mean trouble ahead. Is this intended behaviour and if yes, is there a way to prevent it? With the earlier self.parameters construct this did not happen and one could have e.g. a "group" attribute for the needs of intelmq's internal operation (self.group) separate from a bot developer's "group" attribute (that ended up under self.parameters) - no clashes despite equal names. Thus, to avoid the above in intelmq 3.0.x this means every bot developer needs to check her own bot won't use any of the 'reserved' attributes before defining their own ones, right? I might and I definitively hope I'm wrong here but please let me know if this is the case. Br, Mika

4 8

Systemd startup scripts
by Mika Silander 02 Nov '21

02 Nov '21

Hi, Afaik the bots can currently be managed with the command intelmqctl only. Has anyone implemented systemd startup scripts for intelmq? Br, Mika

3 5

Building packages for intelmq 3.0.2 stumbles on two tests
by Mika Silander 29 Oct '21

29 Oct '21

Hi, Tried building the intelmq debian packages on a Ubuntu 18 LTS, python 3.6 with dpkg-buildpackage -us -uc -b -d Branch was maintenance, tag 3.0.2 and the option -d above was added because Cerberus 1.3.2 was installed with pip3 and not with apt from python3-cerberus. Seems two tests make the build fail, see below. A known problem? Any hints for troubleshooting this? Airing this here before starting to debug on my own ... Cheers, Mika ====================================================================== FAIL: test_multiple_bots (intelmq.tests.bots.collectors.tcp.test_collector.TestTCPCollectorBot) Let's simulate multiple IntelMQ instances are pushing the events at once! ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/mika/intelmq/intelmq/tests/bots/collectors/tcp/test_collector.py", line 175, in test_multiple_bots self.run_bot(iterations=client_count + 1) File "/home/mika/intelmq/intelmq/lib/test.py", line 361, in run_bot self.assertLessEqual(len(re.findall(' - ERROR - ', self.loglines_buffer)), allowed_error_count) AssertionError: 2 not less than or equal to 0 ====================================================================== FAIL: test_random_input (intelmq.tests.bots.collectors.tcp.test_collector.TestTCPCollectorBot) Check how we handle a random input, coming from an unknown source. We should put all the data to report['raw']. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/mika/intelmq/intelmq/tests/bots/collectors/tcp/test_collector.py", line 106, in test_random_input self.run_bot() File "/home/mika/intelmq/intelmq/lib/test.py", line 363, in run_bot self.assertLessEqual(len(re.findall(' - WARNING - ', self.loglines_buffer)), allowed_warning_count) AssertionError: 1 not less than or equal to 0 ---------------------------------------------------------------------- Ran 1404 tests in 125.121s FAILED (failures=2, skipped=274) Test failed: <unittest.runner.TextTestResult run=1404 errors=0 failures=2> error: Test failed: <unittest.runner.TextTestResult run=1404 errors=0 failures=2> E: pybuild pybuild:336: test: plugin distutils failed with: exit code=1: python3.6 setup.py test dh_auto_test: pybuild --test -i python{version} -p 3.6 returned exit code 13 debian/rules:23: recipe for target 'build' failed make: *** [build] Error 25 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

2 4

IEP04: The choice of the UUID-format
by Sebastian Wagner 16 Sep '21

16 Sep '21

Dear allies, The discussion around the IEP04 proposal, adding meta-information to IntelMQ messages, has stalled over the last months - first because of the time-intensive IntelMQ 3.0 release preparations and then because of the vacation season. Here is the current proposal: https://github.com/certtools/ieps/tree/main/004#readme Aaron, Sebastian Waldbauer and myself worked on it over the summer and also identified two open issues to be discussed: 1. The exact format of the meta-information and how to name and structure the fields. AIL made the first move and now uses a format similar to the previously proposed Variant "A". The IEP04 document contains the current proposal which is in line with the AIL format: https://github.com/certtools/ieps/tree/main/004#user-content-variant-ail If there are no other proposals, this will most probably the way to go. 2. The format of the UUID format which we want to uniquely identify IntelMQ events. We don't necessarily need to use the UUIDv4 format which represents pure randomness, but also other options which include the time and are even /time-sortable/. Sebastian Waldbauer analysed a couple of options and summarised his results in this document: https://github.com/certtools/ieps/blob/main/004/UUID.md Please let us know your opinion on the different UUID options. cheers Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

4 10

IntelMQ 3.0.2 release fixing performance issues
by Sebastian Wagner 10 Sep '21

10 Sep '21

Dear community, Last week I said in the announcement for 3.0.1, that we didn't plan any further bugfix release before 3.1.0. I'm now proving myself wrong. Narzhan reported general performance issues and we investigated the problem on our own instances. We found two problems which different effects. It could be the case that you didn't even notice any performance drop, as both bugs affected only certain bots or situations. - When any collector creates a new report, it initializes an object of type Report. Since five years there was a bug, that this always loaded the harmonization.conf, instead of loading it just once at the start (which is - correctly - the case for new events). This was never a problem, as the Python's built-in json module was fast enough, but ruamel.yaml is slower. If a collector didn't massively create reports, that bug was not obvious, especially under testing conditions. And the bug was only relevant, if stream data is processed (for example with the Shodan stream collector). - Two bots, the STOMP collector and the API collector, don't have a proper process method, because they are using threading. A thread retrieves data asynchronously and sends the data. Normally, the process-method is running in an endless-loop, with sleeping rate_limit seconds between. If rate_limit is 0 or not defined, this would result in high load, therefore these bots defined the __collector_empty_process attribute which should have prohibited the endless loop. But due to class-restructuring, the attribute which would have been to be set was _Bot__collector_empty_process, and the flag was not effective. That did not only cause a loop, but as bot statistics are written to redis after every process-call, that caused continuous connections to redis. We've now renamed the flag to _collector_empty_process and fixed the two bots. The flag __is_multithreadable was affected analogous, but that should not have cause performance issues in particular. Additionally, the bot statistics can now be disabled by setting |statistics_host| to null (previously this caused an exception).| | The new version is on GitHub and PyPI, very soon in the deb/rpm repositories. The Docker image will follow on Monday or earlier. You can read the full changelog here: https://github.com/certtools/intelmq/releases/tag/3.0.2 have a nice weekend Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

← Newer
1
2
3
4
5
6
7
8
9
...
24
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev