- IntelMQ-dev - CERT.at

Unexpected log entry ordering in bots' logs
by Mika Silander 31 Oct '22

31 Oct '22

Hi all, I've been implementing a set of monitoring scripts for our intelmq instance. The scripts analyse the log files the bots output. One basic assumption of these scripts is that log entries are ordered according to time: the log entry on line N in the log file is older than the one on line N+1 and so forth. One of our scripts alerts if this assumption does not hold. This morning it reported the following unexpected ordering in the log of mail attachment bot (logrotated once, the file was /var/log/intelmq/mail-attachment-collector.log.1) : ... 2022-10-30 03:58:49,387 - mail-attachment-collector - INFO - Idling for 60.0s (1m) now. 2022-10-30 03:59:49,537 - mail-attachment-collector - INFO - Idling for 60.0s (1m) now. 2022-10-30 03:00:49,681 - mail-attachment-collector - INFO - Idling for 60.0s (1m) now. 2022-10-30 03:01:49,829 - mail-attachment-collector - INFO - Idling for 60.0s (1m) now. ... Any ideas as to why this happens? I already suspected logrotate and the bot net reload functionality, but logrotate is run for the bot net at 06:25 so this seems to be unrelated. I also guess this does not have to do with the mail bot's implementation but one could rather see this happen with other bots as well. A file handle flushing issue? Br, Mika

3 3

Feed handling bug in shadowserver parser bot?
by Mika Silander 05 Sep '22

05 Sep '22

Hi, We've been running intelmq in a production-like setup for some time now and occasionally we see the Shadowserver Parser bot behave in a way that looks odd. We push reports to it via the mail attachment collector bot. At times, the parser bot ends up treating all reports as being of the type Vulnerable ISAKMP. We don't know what triggers this behaviour but so far we've seen this when the parser parses Sandbox URL and Open TFTP reports, both filling the log (debug level) with notifications of "missing keys" or "optional keys not found" because the parser assumes the feed to be Vulnerable ISAKMP. And before this occurs, the parser has been running for a good while. We've picked up the the problematic Sandbox URL and Open TFTP reports above and made unit test cases of these - all tests are correctly parsed, the feed name is deduced correctly and the tests are successful, so the parser handles these correctly if these are the first reports parsed. Therefore, our assumption is that this problem occurs only when the parser has been running for a longer time and some state information about the feed type does not get cleared between parsing incoming reports. Anyone else experiencing similar problems? This is a tricky one to debug so I decided to ask on the list first. Br, Mika P.S: Our setup is intelmq 3.0.2 on an Ubuntu 20.04 LTS

3 7

Documentation on bot statistics collection
by Mika Silander 09 Aug '22

09 Aug '22

Hi, A few issues still remain to be solved before going to production and one of them is how to collect statistics on events. It looks to me that lib/bot.py contains code for collecting statistics and pushing cached data to a Redis instance, but I'd like to know if there's documentation describing the statistics collection in general and recommendations on its use? Any best practices or similar docs? I see there is the EventDB implementation but that in turn seems to be limited to statistics data being pushed out from intelmq via an SQL output bot. In our case we'd also need to keep a count on events being fed into intelmq. We'll probably also need some intermediate collection points within our bot net to keep track of the number of anomalous etc events. Therefore, leveraging somehow the constructs within lib/bot.py seems a better choice for us. As always, pointers are welcome and please correct me if I've understood something wrong above. Br, Mika

2 2

IntelMQ & API & Manager 3.1.0 Release candidate: Please test!
by Sebix 08 Aug '22

08 Aug '22

Dear community, about one year after the 3.0 release, 3.1 is just around the corner. Containing more than 200 commits in IntelMQ, this versions features numerous fixes and improvements from various contributors. The IntelMQ API did not change, the Manager did also receive two small bug fixes. Thanks to elsif2 the Shadowserver parser received massive updates (see Aaron's mail last week) and it needs testing. Therefore, we decided to do create a release candidate before the final "stable" 3.1.0 version. If you can spend some time on testing the new release, please do so and give feedback. Bear in mind that IntelMQ is a community project and has no sponsors, hence please consider contributing. The details you need to know before upgrading: https://github.com/certtools/intelmq/blob/3.1.0-rc1/NEWS.md Full list of changes: https://github.com/certtools/intelmq/blob/3.1.0-rc1/CHANGELOG.md The RC1 version is not available in the stable deb/rpm repository, only in the special "unstable" repository: https://intelmq.readthedocs.io/en/latest/dev/guide.html#testing-pre-releases https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… You can add the unstable repo alongside the stable repository. On PyPI the RC1 releases are marked as unstable and as such will only be installed if pip is called with intelmq==3.1.0rc1.post1 https://pypi.org/project/intelmq/3.1.0rc1.post1/ https://pypi.org/project/intelmq-api/3.1.0/ https://pypi.org/project/intelmq-manager/3.1.0rc1/ If you installed IntelMQ from git, just update the "develop" branch. The git tag is "3.1.0-rc1" best regards Sebastian -- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://sebix.at/ ZVR 1510673578

1 0

heads-up: major code change in intelmq: shadowserver bots code base changed
by L. Aaron Kaplan 04 Aug '22

04 Aug '22

Dear intelmq users and -developers, yesterday I merged a major change to intelmq in the the shadowserver code base[1]. The big news is that from now on, Jason from shadowserver will maintain the intelmq shadowserver code base in intelmq (i.e., /intelmq/bots/{collectors,parsers}/shadowserver/*) directly, which is IMHO really good news for us. Why is this good news? First of all, this allows shadowserver to quickly add code to IntelMQ whenever they release a new scan-report. And they currently do roughly once per week. Sometimes more often! Secondly, the new shadowserver code uses the Shadowserver API [3] and JSON schema, which makes things easier. Finally, intelmq will be much more up to date and complete with respect to shadowserver's feeds. Please pay particular attention to the changes as outlined in the NEWS.md file here [2] Currently, this change is only in the "develop" branch [4]. What do we need from you? ========================= Testing feedback! If you have some time to try out the new shadowserver feeds, we would appreciate you trying out the "develop" branch with your shadowserver API key and tell us / this list / github's issue tracker [5] what worked and what did not work. After all, this is a large code change, and as always, there might be things which break unexpectedly. I would like to have this code well tested before we put it into the upcoming release 3.1. A *BIG* thank you goes out to shadowserver for offering the man-power, the help and all of the time they already put into improving the code base. And in my personal experience, shadowserver feeds are amongst the very best, highest quality data sources on vulnerable devices and IoCs in your network that you could get. So, in case you don't process their feeds yet, sign up here: https://www.shadowserver.org/what-we-do/network-reporting/get-reports/ (and it's even for free [6]). Thank you for your attention and best regards, Aaron Kaplan. [1] https://github.com/certtools/intelmq/pull/2227 [2] https://github.com/certtools/intelmq/pull/2227/files#diff-51920e95310ebfbc1… [3] https://www.shadowserver.org/what-we-do/network-reporting/api-documentation/ [4] https://github.com/certtools/intelmq/branches [5] https://github.com/certtools/intelmq/issues [6] though the urgently need sponsors for running their massive operations: https://www.shadowserver.org/sponsor/

1 0

Expected behaviour of intelmqctl reload ?
by Mika Silander 23 Jun '22

23 Jun '22

Hi all, Tracking down a rather elusive bug led me to suspect that the command intelmqctl reload your_bot_name_here does not necessarily rerun the bot's init method. At least I see an exception related to undefined variables (NoneType) when I reload the bot in question and let it chew a new event. Is this expected behaviour? Still, skimming through the code, it seems to me that the __init__ method in lib/bot.py gets invoked and should invoke the bot's init method as well. Any hints/ideas? Br, Mika

2 4

intelmq-manager with HTTP basic authentication
by Mika Silander 13 Jun '22

13 Jun '22

Hi all, Calling to the list again : I'm having a second round of arm-wrestling with intelmq-manager trying to put it behind the traditional HTTP basic authentication in apache2. So far I'm losing miserably and the only thing that appears to be working is the default intelmq-api authentication. Anyone having a check-list on the steps needed to switch between the two? Br, Mika P.S: Our setup on a Ubuntu 20.04 LTS is intelmq 3.0.2 intelmq-api 3.0.1.beta1 intelmq-manager 3.1.0.alpha2

2 5

update on IEP04
by L. Aaron Kaplan 03 Jun '22

03 Jun '22

Dear IntelMQ community, in the last call on May 25th 2022, we converged on IEP04 (https://github.com/certtools/ieps/). Summary is that we will stay as close as possible to the AIL-exchange format which is fulfilling all our requirements for IntelMQ-to-IntelMQ communication. Also the UUID discussion was resolved. As part of the JTAN-hackathon at CIRCL, the AIL-exchange format was formalised as a JSON-Schema (https://github.com/certtools/ieps/blob/main/004/schema/schema.json) which will also allow us in IntelMQ to use it and to even verify data coming in to an IntelMQ instance. So, now that it is formalised, we'll see how to write collectors and output processors/bots for IntelMQ to IntelMQ communications. Thanks for the patience for IEP04 and I am happy that we have a solution now. Best, Aaron Kaplan.

1 0

Update on my involvement in IntelMQ
by Sebix 18 May '22

18 May '22

Dear IntelMQ community, Last autumn I left my previous employer CERT.at. Now I want to give you a short update regarding my future involvement with IntelMQ. First, let me state I'm operating in different roles here. There is voluntary non-profit work ("sebix"[0]) and my work as part my employment. I recently joined Intevation[1] part-time. This is a Free Software company located in Germany. They are already engaged in IntelMQ since 2016 (1½ years after IntelMQ's beginning) and are e.g. contracted by CERT-BUND[2] for IntelMQ-related work. As part of that contract I'm able to contribute a small part of my work time to the community support and maintenance of IntelMQ. If you are interested in contracting with us, please get in touch. In addition, I have founded a non-profit association[0] for non-commercial initiatives and cooperation. If you are interested in collaborating with us, please get in contact with me. IntelMQ is still in need of mid- and long-term contribution and financing. One approach is led by Aaron, who works to form an umbrella organisational support structure for IntelMQ. Intevation and myself aim to complement this effort. cheers Sebastian [0]: https://sebix.at/ <https://intevation.de/> [1]: https://intevation.de/ [2]: https://www.cert-bund.de/

1 0

Alerting on repeated connection failures in a bot
by Mika Silander 25 Mar '22

25 Mar '22

Hi all, Calling out to the list again before trying to reinvent a wheel. We have a collector that tries to connect to the source of security events, but the source service is (temporarily) inaccessible. IntelMQ restarts the bot, an error message and the connection failure exception gets output to the log, and after a while, the bot gets restarted. For us it is acceptable there's a few failures in a configurable time frame, but if the situation prevails, we'd like to be alerted and if possible, prevent the bot from restarting. There are bot configuration parameters to stop a bot in case the processing of events fails repeatedly (error_procedure=stop, error_max_retries=suitable_number as in https://intelmq.readthedocs.io/en/maintenance/user/configuration-management…), but after some experimenting it appears they are not applicable here since we never reach the stage of processing events. Any ideas how to address this issue? We'd prefer not to touch the bot's source code. Br, Mika P.S: Setup is intelmq v3.0.2.

2 1

Fwd: Invitation to an IntelMQ in depth training on February 23th, 14:30h CET
by Otmar Lendl 03 Feb '22

03 Feb '22

FYI, and sorry if you receive this message multiple times. otmar -------- Forwarded Message -------- Subject: Invitation to an IntelMQ in depth training on February 23th, 14:30h CET Date: Thu, 3 Feb 2022 16:10:57 +0100 From: Otmar Lendl <lendl(a)cert.at> Dear members of the CSIRTs Network, CERT.at will host a presentation on IntelMQ as part of our training activities connected to the MeliCERTes 2 project. This presentation (and hopefully lively discussion ) will be on technical details: - IntelMQ Development, - Insights, - Core Knowledge The target audience are IntelMQ-devs & admins on an advanced level. Date/Time: 2021-02-23 14:30 - 16:00 UTC+1 (CET) Location: Online via Zoom: > https://us06web.zoom.us/j/84028475065?pwd=VmR3aUVrdGNFYk1EVTcxOEUvVHNFZz09 > Meeting-ID: 840 2847 5065 > Kenncode: 522946 Presenter: Sebastian Waldbauer Feel free to share this invitation to interested people. No registration is required. Regards, Otmar Lendl -- // Otmar Lendl <lendl(a)cert.at> - T: +43 1 5056416 711 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

1 0

Contribute by Reviews and Maintenance
by Sebix 17 Nov '21

17 Nov '21

Dear community, In my last mail on this list I already hided the message that I left CERT.at a short while ago and I was already asked what that means for the IntelMQ project. First of all: Don't panic! IntelMQ is and has always been a community project, with contributors from multiple teams, countries and continents[0]. In every release announcement I say thank you to the community for all the contributions and I really mean it. IntelMQ would never be that feature-rich without your involvement. All of us are part of this project, and joining our forces makes us strong. IntelMQ made various big steps in the last 7 years. It all started with Tomás Lima and Aaron Kaplan. The architecture was so robust, that it didn't change since then. I joined one year later and since then I almost memorized every line of code. In the last years I lost that ability as more and more code was coming for contributors, who are also maintaining code. And that's the essence: With IntelMQ growing, it's important to share the maintenance work. The IntelMQ project is looking for your contributions, including: - fixing open bugs[0] - reviewing open pull requests[1] - discussing and deciding when to make releases - testing before we do releases - documentation - and much more ;) IntelMQ is ready for the next evolutional leap forward and you are part of it! best regards Sebastian [0]: Maybe there will be a Moon- or Mars CERT in the not-so-distant future? ;) [1]: https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3A… [2]: https://github.com/certtools/intelmq/pulls

1 0

Redis log file gets wrong owner & group(?)
by Mika Silander 15 Nov '21

15 Nov '21

Hi all, Occasionally we see the /var/log/redis/redis-server.log file getting intelmq as its owner and group. This makes redis output to the log file fail. Once the owner is reset to redis and group to adm (on Ubuntu 20.04 LTS) and running systemctl restart redis, redis works fine. I've tried to debug the reason for this change in ownership in logrotate confs, intelmqctl sources etc but so far no luck. Hints as to the reason or how to troubleshoot are again welcome. Br, Mika P.S: We haven't configured redis to listen to unix sockets in /etc/redis/redis.conf so we assume https://intelmq.readthedocs.io/en/maintenance/user/FAQ.html#id4 does not apply to this case.

2 2

Help for fixing configuration of intelmq-manager 3.0.1-1
by Mika Silander 03 Nov '21

03 Nov '21

Hi, Seems I've gotten stuck on the same problem than before. However, this time around the order of installation was (on a Ubuntu 20.04 LTS/Apache 2): apt-get install intelmq apt-get install intelmq-api apt-get install intelmq-manager and the versions were: intelmq 3.0.2-1 intelmq-api 3.0.1-1 intelmq-manager 3.0.1-1 I'm able to access the intelmq-manager start page, https://ourimq.domain/intelmq-manager/index.html but when opening the Configuration tab https://ourimq.domain/intelmq-manager/configs.html, expected frames appear but no bots are shown in the frame (normally) allowing editing of the bot network. Instead an endless loop starts with the error shown further below. All hints for troubleshooting & debugging are welcome. Br, Mika P.S: No SELinux lurking in the shadows on the install host. --- 16:42:30 115× Error loading bot queues information: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> <hr> <address>Apache/2.4.41 (Ubuntu) Server at ourimq.domain Port 443</address> </body></html> Not Found 16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/positions . 16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/runtime . 16:40:37 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/bots

2 6

Potential attribute name clashes in bot parameters in intelmq 3.1.x ?
by Mika Silander 02 Nov '21

02 Nov '21

Hi, At long last I updated my develop branch from the 2.3. days to 3.1.0 and noticed my tests fail due to the fact that bots don't have anymore a "parameters" attribute. At a closer look it seems all parameters from runtime.yaml are turned into direct attributes of self (correct?), e.g. if one defines a conf parameter "myparam", this shows up in the bot as self.myparam and not accessible as getattr(self.parameters, "myparam") as it did before. After experimenting with my own bot tests I ended up in a situation where there's a potential attribute name clash. Assume a bot is tested like self.input_message = some_event_here self.run_bot() self.assertSomethingHere() In the above test, the bot gets initiated with a big number of default attributes, e.g. accuracy, group, enabled, logger, run_mode etc etc. Assume then that as a developer I want to use parameters with matching names for the needs of my own bot like self.input_message = some_event_here self.run_bot(parameters={ 'group': 'Plumber', 'run_mode': 'disruptive' }) self.assertSomethingHere() I can set those parameters and they happily override the defaults and I imagine this can mean trouble ahead. Is this intended behaviour and if yes, is there a way to prevent it? With the earlier self.parameters construct this did not happen and one could have e.g. a "group" attribute for the needs of intelmq's internal operation (self.group) separate from a bot developer's "group" attribute (that ended up under self.parameters) - no clashes despite equal names. Thus, to avoid the above in intelmq 3.0.x this means every bot developer needs to check her own bot won't use any of the 'reserved' attributes before defining their own ones, right? I might and I definitively hope I'm wrong here but please let me know if this is the case. Br, Mika

4 8

Systemd startup scripts
by Mika Silander 02 Nov '21

02 Nov '21

Hi, Afaik the bots can currently be managed with the command intelmqctl only. Has anyone implemented systemd startup scripts for intelmq? Br, Mika

3 5

Building packages for intelmq 3.0.2 stumbles on two tests
by Mika Silander 29 Oct '21

29 Oct '21

Hi, Tried building the intelmq debian packages on a Ubuntu 18 LTS, python 3.6 with dpkg-buildpackage -us -uc -b -d Branch was maintenance, tag 3.0.2 and the option -d above was added because Cerberus 1.3.2 was installed with pip3 and not with apt from python3-cerberus. Seems two tests make the build fail, see below. A known problem? Any hints for troubleshooting this? Airing this here before starting to debug on my own ... Cheers, Mika ====================================================================== FAIL: test_multiple_bots (intelmq.tests.bots.collectors.tcp.test_collector.TestTCPCollectorBot) Let's simulate multiple IntelMQ instances are pushing the events at once! ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/mika/intelmq/intelmq/tests/bots/collectors/tcp/test_collector.py", line 175, in test_multiple_bots self.run_bot(iterations=client_count + 1) File "/home/mika/intelmq/intelmq/lib/test.py", line 361, in run_bot self.assertLessEqual(len(re.findall(' - ERROR - ', self.loglines_buffer)), allowed_error_count) AssertionError: 2 not less than or equal to 0 ====================================================================== FAIL: test_random_input (intelmq.tests.bots.collectors.tcp.test_collector.TestTCPCollectorBot) Check how we handle a random input, coming from an unknown source. We should put all the data to report['raw']. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/mika/intelmq/intelmq/tests/bots/collectors/tcp/test_collector.py", line 106, in test_random_input self.run_bot() File "/home/mika/intelmq/intelmq/lib/test.py", line 363, in run_bot self.assertLessEqual(len(re.findall(' - WARNING - ', self.loglines_buffer)), allowed_warning_count) AssertionError: 1 not less than or equal to 0 ---------------------------------------------------------------------- Ran 1404 tests in 125.121s FAILED (failures=2, skipped=274) Test failed: <unittest.runner.TextTestResult run=1404 errors=0 failures=2> error: Test failed: <unittest.runner.TextTestResult run=1404 errors=0 failures=2> E: pybuild pybuild:336: test: plugin distutils failed with: exit code=1: python3.6 setup.py test dh_auto_test: pybuild --test -i python{version} -p 3.6 returned exit code 13 debian/rules:23: recipe for target 'build' failed make: *** [build] Error 25 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

2 4

IEP04: The choice of the UUID-format
by Sebastian Wagner 16 Sep '21

16 Sep '21

Dear allies, The discussion around the IEP04 proposal, adding meta-information to IntelMQ messages, has stalled over the last months - first because of the time-intensive IntelMQ 3.0 release preparations and then because of the vacation season. Here is the current proposal: https://github.com/certtools/ieps/tree/main/004#readme Aaron, Sebastian Waldbauer and myself worked on it over the summer and also identified two open issues to be discussed: 1. The exact format of the meta-information and how to name and structure the fields. AIL made the first move and now uses a format similar to the previously proposed Variant "A". The IEP04 document contains the current proposal which is in line with the AIL format: https://github.com/certtools/ieps/tree/main/004#user-content-variant-ail If there are no other proposals, this will most probably the way to go. 2. The format of the UUID format which we want to uniquely identify IntelMQ events. We don't necessarily need to use the UUIDv4 format which represents pure randomness, but also other options which include the time and are even /time-sortable/. Sebastian Waldbauer analysed a couple of options and summarised his results in this document: https://github.com/certtools/ieps/blob/main/004/UUID.md Please let us know your opinion on the different UUID options. cheers Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

4 10

IntelMQ 3.0.2 release fixing performance issues
by Sebastian Wagner 10 Sep '21

10 Sep '21

Dear community, Last week I said in the announcement for 3.0.1, that we didn't plan any further bugfix release before 3.1.0. I'm now proving myself wrong. Narzhan reported general performance issues and we investigated the problem on our own instances. We found two problems which different effects. It could be the case that you didn't even notice any performance drop, as both bugs affected only certain bots or situations. - When any collector creates a new report, it initializes an object of type Report. Since five years there was a bug, that this always loaded the harmonization.conf, instead of loading it just once at the start (which is - correctly - the case for new events). This was never a problem, as the Python's built-in json module was fast enough, but ruamel.yaml is slower. If a collector didn't massively create reports, that bug was not obvious, especially under testing conditions. And the bug was only relevant, if stream data is processed (for example with the Shodan stream collector). - Two bots, the STOMP collector and the API collector, don't have a proper process method, because they are using threading. A thread retrieves data asynchronously and sends the data. Normally, the process-method is running in an endless-loop, with sleeping rate_limit seconds between. If rate_limit is 0 or not defined, this would result in high load, therefore these bots defined the __collector_empty_process attribute which should have prohibited the endless loop. But due to class-restructuring, the attribute which would have been to be set was _Bot__collector_empty_process, and the flag was not effective. That did not only cause a loop, but as bot statistics are written to redis after every process-call, that caused continuous connections to redis. We've now renamed the flag to _collector_empty_process and fixed the two bots. The flag __is_multithreadable was affected analogous, but that should not have cause performance issues in particular. Additionally, the bot statistics can now be disabled by setting |statistics_host| to null (previously this caused an exception).| | The new version is on GitHub and PyPI, very soon in the deb/rpm repositories. The Docker image will follow on Monday or earlier. You can read the full changelog here: https://github.com/certtools/intelmq/releases/tag/3.0.2 have a nice weekend Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ & API & Manager release 3.0.1 and upgrade note
by Sebastian Wagner 02 Sep '21

02 Sep '21

Dear community, Over the past two months, IntelMQ contributors had no summer pause, but did the final finish for IntelMQ 3.0. A special thanks goes to Mikk Margus Möll (CERT.ee) who has put tremendous efforts in the IntelMQ Manager tackling structural and usability issues, mainly in the JavaScript-components! The deb/rpm repositories did not receive the 3.0.0 release at beginning of July to get more experience with the major changes before doing automatic upgrades, but now they deliver the brand-new 3.0.1 version. Please note, that the automatic upgrade procedures may still not be fully smooth. Just now, we have noticed, that the packages contain a small flaw, which harms the upgrade experience: The packages ship a default configuration (the file is now called `runtime.yaml`), but only if the file does not exist before - for new installations. But now in this special case, we renamed the configuration from `runtime.conf` to `runtime.yaml` and therefore, the new - default shipped - configuration takes precedence. I hope the following commands and hints will be of help to you. # remove the runtime configuration shipped by the package (can be called /etc/intelmq/runtime.*) and rename your original one to /etc/intelmq/runtime.yaml # the previously used runtime.conf can be used as drop-in to runtime.yaml (YAML is backwards-compatible with JSON) sudo -u intelmq intelmqctl upgrade-config -f -u v300_pipeline_file_removal sudo -u intelmq intelmqctl upgrade-config -f -u v300_defaults_file_removal sudo -u intelmq intelmqctl upgrade-config -f -u v301_deprecations The last three steps are important to merge the defaults and pipeline configuration into the new combined configuration file Please do not hesitate to ask. The deb-packages are also already available for the newly released Debian 11 Bullseye. We are not planning a bugfix release until the 3.1.0 release, so that one will be the next version to be released. Here's a short summary of what happened during the summer: - various fixes related to the IEP001 implementation (IEP001 was the change configuration format and merge of files, rewrite oft the internal parameter-handling) - removal of the malwaredomains feed and parser, because it does not exist anymore - Various fixes in the Shadowserver Parser and support for new reports: Vulnerable SMTP Server, Microsoft Sinkhole Events Report & Microsoft Sinkhole HTTP Events Report, Honeypot HTTP Scan - SMTP Output: Added Content-Disposition-header to the attachment, fixing the display in MS Outlook clients (as reported and dicussed on the Mailinglist). - Heavy refactoring of IntelMQ-Manager's JavaScript parts to fix errors and usability issues. If you are interested in developing on IntelMQ and you don't know where to start, have a look at the dev guide an the issues labeled "good first issue": https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3A… We are especially welcoming contributions to the documentation! You can read the full changelogs here: - https://github.com/certtools/intelmq/releases/tag/3.0.1 - https://github.com/certtools/intelmq-api/releases/tag/3.0.1 - https://github.com/certtools/intelmq-manager/releases/tag/3.0.1 https://cert.at/en/blog/2021/9/intelmq-301-release https://twitter.com/CERT_at/status/1433475188381806594 btw: There's new contact management portal called "tuency" for administrating abuse contacts available, which can be used in conjunction with IntelMQ. Read more about its features here: https://cert.at/en/blog/2021/9/tuency-constituency-portal-for-iocs-and-certs https://gitlab.com/intevation/tuency/tuency -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ & API & Manager Release 3.0
by Sebastian Wagner 05 Aug '21

05 Aug '21

Dear community, The time has come and IntelMQ 3.0 is final! We, as community, made loads of changes, smaller and bigger ones, and I really think that IntelMQ became more user-friendly, developer-friendly and feature-rich at the same time! There are some major changes in this release, especially the in the field of the configuration and Internal Data format (previously: "harmonization"). For the configuration-part, the upgrade part should be automatic with `intelmqctl upgrade-config` as usual. For the Data format, carefully look at your bot configurations (filters, sieve, etc.) to update them. Adaptions in systems connected to IntelMQ, especially also databases might be necessary as well. The NEWS.md file give a summary of what has changed: https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-… We don't recommend to upgrade existing production instance of IntelMQ yet. We of course did testing, including the end-to-end tests, and have detailed release notes. But for critical systems, a delayed upgrade makes sense ;) Therefore the stable deb/rpm repositories don't contain the 3.0 release yet! Even though an upgrade of production systems is not yet recommended, extensive usage and testing of the new releases are very much welcome and required to get the necessary feedback for the next (maintenance) releases. The releases are available via git, PyPI, Docker and the *unstable* deb/rpm repositories. Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html IntelMQ API documentation: https://intelmq.readthedocs.io/en/maintenance/user/intelmq-api.html IntelMQ Manager documentation: https://intelmq.readthedocs.io/en/maintenance/user/intelmq-manager.html NEWS/release notes of IntelMQ (Core): https://github.com/certtools/intelmq/blob/maintenance/NEWS.md#user-content-… Full Changelog of IntelMQ (Core): https://github.com/certtools/intelmq/blob/maintenance/CHANGELOG.md#300-2021… On a high level, these are the major changes compared to version 2.3.x (2.3.3 was released 2021-05-31): In the core and Docker: * Configuration rewrite including parameter loading and handling (IEP01), plus the required adoption of the API and Manager, by Birger Schacht (CERT.at). * Classification sync with RSIT, by Sebastian Wagner (CERT.at). * Removal of the BOTS file, by Sebastian Waldbauer (CERT.at). * Creation and maintenance of the Docker images by Sebastian Waldbauer (CERT.at). * Creation of Docker-instructions for development setups by Einar Lanfranco and Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar). New and majorly enhanced bots: * Added |intelmq.bots.collectors.fireeye|: A bot that collects indicators from Fireeye MAS appliances (PR#1745 by Christopher Schappelwein). * |intelmq.bots.collectors.api.collector_api|: Added UNIX socket capability (PR#1987 by Mikk Margus Möll, fixes #1986). * Added |intelmq.bots.parsers.fireeye|: A bot that parses hashes and URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein). * Added |intelmq.bots.experts.http.expert_status|: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR#1789 by Birger Schacht, fixes #1047 partly). * Added |intelmq.bots.experts.http.expert_content|: A bot that fetches an HTTP resource and checks if it contains a specific string (PR#1811 by Birger Schacht). * Added |intelmq.bots.experts.lookyloo.expert|: A bot that sends requests to a lookyloo instance & adds |screenshot_url| to the event (PR#1844 by Sebastian Waldbauer, fixes #1048). * Added |intelmq.bots.experts.rdap.expert|: A bot that checks the RDAP protocol for an abuse contact for a given domain (PR#1881 by Sebastian Waldbauer and Sebastian Wagner). * |intelmq.bots.experts.sieve.expert|: Major refactoring and lot's of new functionality New operators for working with various types (lists, sets, booleans, float, int), generic rule negation and nesting (PR#1895 by Mikk Margus Möll). * Added |intelmq.bots.experts.uwhoisd|: A bot that fetches the whois entry from a uwhois-instance (PR#1918 by Raphaël Vinot). * Added |intelmq.bots.experts.aggregate|: A bot that aggregate events based upon given fields & a timespan. (PR#1959 by Sebastian Waldbauer) * Added |intelmq.bots.experts.tuency|: A bot that queries the IntelMQ API of a tuency instance (PR#1857 by Sebastian Wagner, fixes #1856). * Added |intelmq.bots.outputs.templated_smtp| (PR#1901 by Karl-Johan Karlsson). On the documentation front, these are the most important changes * License and copyright information was added to all the bots (by Birger Schacht). * Added documentation on the EventDB (PR#1955 by Birger Schacht, PR#1985 by Sebastian Wagner). * Added TimescaleDB for time-series documentation (PR#1990 by Sebastian Waldbauer). * n6 interoperability documentation: Adding more graphs and illustrations (PR#1991 by Sebastian Wagner). * Added documentation on abuse-contact look-ups (PR#2021 by Sebastian Waldbauer and Sebastian Wagner). And not to forget all the smaller changes and additions. Thanks to (in random order) Raphaël Vinot (circl.lu) Bernhard Reiter (intevation.de) Sebastian Wagner (CERT.AT) Filip Pokorný (CSIRT.CZ) Guillaume GRANJON de LÉPINEY (CERT XLM excellium-services.com) Mikk Margus Möll (CERT.ee) Alex Kaplan Thomas Hungenberg (CERT-BUND.DE) Einar Lanfranco (CERT-UNLP cert.unlp.edu.ar) Christopher Schappelwein (milCERT, BMLV.gv.at) Marcos Gonzalez (CSIRT-RD cncs.gob.do/csirt-rd/) Marius Karotkis (NRDCS.LT) Sebastian Waldbauer (CERT.AT) Jeremias Pretto (CERT-UNLP cert.unlp.edu.ar) Karl-Johan Karlsson (Linköping University LIU.SE) Birger Schacht (CERT.AT) ... and all the contributors of previous releases and as well to all reporters, supporters, etc! best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

How does one avoid reprocessing mail attachments already processed?
by Mika Silander 04 Aug '21

04 Aug '21

Hi all, We are using the mail collector bot to read email attachments over IMAPS. It works fine but rereads all the messages in the message folder even if it had processed them already. Is there a way to configure the mail bot to remove the messages once it has processed them? I didn't find a hint concerning this in the bot's documentation nor the sources. What approaches do people use to avoid reprocessing messages? Br, Mika

2 3

Path/permission problem with intelmq-manager (v2.3.1-2) ?
by Mika Silander 02 Aug '21

02 Aug '21

Hi all, Returning to an earlier problem that surfaced after having updated intelmq to version 2.3.3-1: intelmq-manager starts, but when opening the configuration tab, the network of bots doesn't get rendered. Looking at the output of the browser's developer console one can see successful retrieval of https://ourhost/configs.html that in turn initiates the successful retrieval of a number of CSS, PNG and JS files. Next we have five failures (404 Not found): https://ourhost/intelmq/v1/api/config?file=bots https://ourhost/intelmq/v1/api/config?file=defaults https://ourhost/intelmq/v1/api/config?file=runtime https://ourhost/intelmq/v1/api/config?file=pipeline https://ourhost/intelmq/v1/api/config?file=positions Then a few more successful retrievals of PNGs and a GIF. After this we end up in an infinite loop that tries unsuccessfully (404 Not found) to fetch https://ourhost/intelmq/v1/api/queues-and-status once every second. The error the browser shows repeatedly is "Error loading bot queues information:". What is common to all failures above is that they seem to be due to jquery-actions. Any ideas as to where the problem could be? I suspect some path and/or permission problem but that's the first guess. Br, Mika For reference, these are the package versions on our intelmq Ubuntu 20 LTS host: intelmq 2.3.3-1 intelmq-api 2.3.1-1 intelmq-manager 2.3.1-2

2 3

Preventing lost events when stopping intelmq
by Mika Silander 30 Jul '21

30 Jul '21

Hi, Returning to a similar issue but from a different angle: for maintenance I'd like to be able to cleanly shutdown the server running intelmq. Is there a way to guarantee that none of the bots is in a processing state (i.e. processing an event in the process method) before server shutdown? Can "intelmqctl stop" for example stop the bot chain in such a way that none of the bots is in the midst of processing an event? If not, what would be the best approach for achieving this? Br, Mika

2 5

Re: [IntelMQ-dev] [IntelMQ-users] Unable to send CSV file using Intelmq
by Bernhard Reiter 13 Jul '21

13 Jul '21

Hi Kossi, Am Montag 12 Juli 2021 18:25:05 schrieb Kossi DOH via IntelMQ-users: > I will work on it with and see what it gives. If I understand well this > issue is related to Microsoft Outlook. Sebastian also created the following issue, which you can add information, help and subscribe to, if you like: https://github.com/certtools/intelmq/issues/2018 Regards, Bernhard -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev