IntelMQ-dev

intelmq-dev@lists.cert.at

2 participants
253 discussions

heads-up: major code change in intelmq: shadowserver bots code base changed
by L. Aaron Kaplan 04 Aug '22

04 Aug '22

Dear intelmq users and -developers, yesterday I merged a major change to intelmq in the the shadowserver code base[1]. The big news is that from now on, Jason from shadowserver will maintain the intelmq shadowserver code base in intelmq (i.e., /intelmq/bots/{collectors,parsers}/shadowserver/*) directly, which is IMHO really good news for us. Why is this good news? First of all, this allows shadowserver to quickly add code to IntelMQ whenever they release a new scan-report. And they currently do roughly once per week. Sometimes more often! Secondly, the new shadowserver code uses the Shadowserver API [3] and JSON schema, which makes things easier. Finally, intelmq will be much more up to date and complete with respect to shadowserver's feeds. Please pay particular attention to the changes as outlined in the NEWS.md file here [2] Currently, this change is only in the "develop" branch [4]. What do we need from you? ========================= Testing feedback! If you have some time to try out the new shadowserver feeds, we would appreciate you trying out the "develop" branch with your shadowserver API key and tell us / this list / github's issue tracker [5] what worked and what did not work. After all, this is a large code change, and as always, there might be things which break unexpectedly. I would like to have this code well tested before we put it into the upcoming release 3.1. A *BIG* thank you goes out to shadowserver for offering the man-power, the help and all of the time they already put into improving the code base. And in my personal experience, shadowserver feeds are amongst the very best, highest quality data sources on vulnerable devices and IoCs in your network that you could get. So, in case you don't process their feeds yet, sign up here: https://www.shadowserver.org/what-we-do/network-reporting/get-reports/ (and it's even for free [6]). Thank you for your attention and best regards, Aaron Kaplan. [1] https://github.com/certtools/intelmq/pull/2227 [2] https://github.com/certtools/intelmq/pull/2227/files#diff-51920e95310ebfbc1… [3] https://www.shadowserver.org/what-we-do/network-reporting/api-documentation/ [4] https://github.com/certtools/intelmq/branches [5] https://github.com/certtools/intelmq/issues [6] though the urgently need sponsors for running their massive operations: https://www.shadowserver.org/sponsor/

1 0

Expected behaviour of intelmqctl reload ?
by Mika Silander 23 Jun '22

23 Jun '22

Hi all, Tracking down a rather elusive bug led me to suspect that the command intelmqctl reload your_bot_name_here does not necessarily rerun the bot's init method. At least I see an exception related to undefined variables (NoneType) when I reload the bot in question and let it chew a new event. Is this expected behaviour? Still, skimming through the code, it seems to me that the __init__ method in lib/bot.py gets invoked and should invoke the bot's init method as well. Any hints/ideas? Br, Mika

2 4

intelmq-manager with HTTP basic authentication
by Mika Silander 13 Jun '22

13 Jun '22

Hi all, Calling to the list again : I'm having a second round of arm-wrestling with intelmq-manager trying to put it behind the traditional HTTP basic authentication in apache2. So far I'm losing miserably and the only thing that appears to be working is the default intelmq-api authentication. Anyone having a check-list on the steps needed to switch between the two? Br, Mika P.S: Our setup on a Ubuntu 20.04 LTS is intelmq 3.0.2 intelmq-api 3.0.1.beta1 intelmq-manager 3.1.0.alpha2

2 5

update on IEP04
by L. Aaron Kaplan 03 Jun '22

03 Jun '22

Dear IntelMQ community, in the last call on May 25th 2022, we converged on IEP04 (https://github.com/certtools/ieps/). Summary is that we will stay as close as possible to the AIL-exchange format which is fulfilling all our requirements for IntelMQ-to-IntelMQ communication. Also the UUID discussion was resolved. As part of the JTAN-hackathon at CIRCL, the AIL-exchange format was formalised as a JSON-Schema (https://github.com/certtools/ieps/blob/main/004/schema/schema.json) which will also allow us in IntelMQ to use it and to even verify data coming in to an IntelMQ instance. So, now that it is formalised, we'll see how to write collectors and output processors/bots for IntelMQ to IntelMQ communications. Thanks for the patience for IEP04 and I am happy that we have a solution now. Best, Aaron Kaplan.

1 0

Update on my involvement in IntelMQ
by Sebix 18 May '22

18 May '22

Dear IntelMQ community, Last autumn I left my previous employer CERT.at. Now I want to give you a short update regarding my future involvement with IntelMQ. First, let me state I'm operating in different roles here. There is voluntary non-profit work ("sebix"[0]) and my work as part my employment. I recently joined Intevation[1] part-time. This is a Free Software company located in Germany. They are already engaged in IntelMQ since 2016 (1½ years after IntelMQ's beginning) and are e.g. contracted by CERT-BUND[2] for IntelMQ-related work. As part of that contract I'm able to contribute a small part of my work time to the community support and maintenance of IntelMQ. If you are interested in contracting with us, please get in touch. In addition, I have founded a non-profit association[0] for non-commercial initiatives and cooperation. If you are interested in collaborating with us, please get in contact with me. IntelMQ is still in need of mid- and long-term contribution and financing. One approach is led by Aaron, who works to form an umbrella organisational support structure for IntelMQ. Intevation and myself aim to complement this effort. cheers Sebastian [0]: https://sebix.at/ <https://intevation.de/> [1]: https://intevation.de/ [2]: https://www.cert-bund.de/

1 0

Alerting on repeated connection failures in a bot
by Mika Silander 25 Mar '22

25 Mar '22

Hi all, Calling out to the list again before trying to reinvent a wheel. We have a collector that tries to connect to the source of security events, but the source service is (temporarily) inaccessible. IntelMQ restarts the bot, an error message and the connection failure exception gets output to the log, and after a while, the bot gets restarted. For us it is acceptable there's a few failures in a configurable time frame, but if the situation prevails, we'd like to be alerted and if possible, prevent the bot from restarting. There are bot configuration parameters to stop a bot in case the processing of events fails repeatedly (error_procedure=stop, error_max_retries=suitable_number as in https://intelmq.readthedocs.io/en/maintenance/user/configuration-management…), but after some experimenting it appears they are not applicable here since we never reach the stage of processing events. Any ideas how to address this issue? We'd prefer not to touch the bot's source code. Br, Mika P.S: Setup is intelmq v3.0.2.

2 1

Fwd: Invitation to an IntelMQ in depth training on February 23th, 14:30h CET
by Otmar Lendl 03 Feb '22

03 Feb '22

FYI, and sorry if you receive this message multiple times. otmar -------- Forwarded Message -------- Subject: Invitation to an IntelMQ in depth training on February 23th, 14:30h CET Date: Thu, 3 Feb 2022 16:10:57 +0100 From: Otmar Lendl <lendl(a)cert.at> Dear members of the CSIRTs Network, CERT.at will host a presentation on IntelMQ as part of our training activities connected to the MeliCERTes 2 project. This presentation (and hopefully lively discussion ) will be on technical details: - IntelMQ Development, - Insights, - Core Knowledge The target audience are IntelMQ-devs & admins on an advanced level. Date/Time: 2021-02-23 14:30 - 16:00 UTC+1 (CET) Location: Online via Zoom: > https://us06web.zoom.us/j/84028475065?pwd=VmR3aUVrdGNFYk1EVTcxOEUvVHNFZz09 > Meeting-ID: 840 2847 5065 > Kenncode: 522946 Presenter: Sebastian Waldbauer Feel free to share this invitation to interested people. No registration is required. Regards, Otmar Lendl -- // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

1 0

Contribute by Reviews and Maintenance
by Sebix 17 Nov '21

17 Nov '21

Dear community, In my last mail on this list I already hided the message that I left CERT.at a short while ago and I was already asked what that means for the IntelMQ project. First of all: Don't panic! IntelMQ is and has always been a community project, with contributors from multiple teams, countries and continents[0]. In every release announcement I say thank you to the community for all the contributions and I really mean it. IntelMQ would never be that feature-rich without your involvement. All of us are part of this project, and joining our forces makes us strong. IntelMQ made various big steps in the last 7 years. It all started with Tomás Lima and Aaron Kaplan. The architecture was so robust, that it didn't change since then. I joined one year later and since then I almost memorized every line of code. In the last years I lost that ability as more and more code was coming for contributors, who are also maintaining code. And that's the essence: With IntelMQ growing, it's important to share the maintenance work. The IntelMQ project is looking for your contributions, including: - fixing open bugs[0] - reviewing open pull requests[1] - discussing and deciding when to make releases - testing before we do releases - documentation - and much more ;) IntelMQ is ready for the next evolutional leap forward and you are part of it! best regards Sebastian [0]: Maybe there will be a Moon- or Mars CERT in the not-so-distant future? ;) [1]: https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3A… [2]: https://github.com/certtools/intelmq/pulls

1 0

Redis log file gets wrong owner & group(?)
by Mika Silander 15 Nov '21

15 Nov '21

Hi all, Occasionally we see the /var/log/redis/redis-server.log file getting intelmq as its owner and group. This makes redis output to the log file fail. Once the owner is reset to redis and group to adm (on Ubuntu 20.04 LTS) and running systemctl restart redis, redis works fine. I've tried to debug the reason for this change in ownership in logrotate confs, intelmqctl sources etc but so far no luck. Hints as to the reason or how to troubleshoot are again welcome. Br, Mika P.S: We haven't configured redis to listen to unix sockets in /etc/redis/redis.conf so we assume https://intelmq.readthedocs.io/en/maintenance/user/FAQ.html#id4 does not apply to this case.

2 2

Help for fixing configuration of intelmq-manager 3.0.1-1
by Mika Silander 03 Nov '21

03 Nov '21

Hi, Seems I've gotten stuck on the same problem than before. However, this time around the order of installation was (on a Ubuntu 20.04 LTS/Apache 2): apt-get install intelmq apt-get install intelmq-api apt-get install intelmq-manager and the versions were: intelmq 3.0.2-1 intelmq-api 3.0.1-1 intelmq-manager 3.0.1-1 I'm able to access the intelmq-manager start page, https://ourimq.domain/intelmq-manager/index.html but when opening the Configuration tab https://ourimq.domain/intelmq-manager/configs.html, expected frames appear but no bots are shown in the frame (normally) allowing editing of the bot network. Instead an endless loop starts with the error shown further below. All hints for troubleshooting & debugging are welcome. Br, Mika P.S: No SELinux lurking in the shadows on the install host. --- 16:42:30 115× Error loading bot queues information: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> <hr> <address>Apache/2.4.41 (Ubuntu) Server at ourimq.domain Port 443</address> </body></html> Not Found 16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/positions . 16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/runtime . 16:40:37 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/bots

2 6

← Newer
1
...
4
5
6
7
8
9
10
...
26
Older →

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev