- IntelMQ-users

regarding IEP03
by L. Aaron Kaplan 22 Apr '21

22 Apr '21

Hi everyone, I wanted to send back a better write-up of my stance on IEP03 ("multiple values" in the IntelMQ internal data format). Alas, I was quite busy and I am sprinting to push out some code for our deadline at work. However, let me summarise it: I think the IEP03 is very well written, thank you a lot for this! Thinking this through was important and I think Sebastian Waldbauer did a great job. Reading it, I realised that my initial proposal of having multiple values is really breaking the KISS principle of IntelMQ in a bad way. Worse than I had thought. So, I am thinking of retracting the proposal. However, .... https://github.com/certtools/ieps/tree/main/003#alternatives has a good core in it. If we have multiple values, instead of doing the n x m complexity explosion, we link different events (JSON rows) together via UUIDs this gives us what we need: * UUIDs help with deduplication! That's important when linking IntelMQ instances! * lower complexity / keep the KISS principle * consumers can ignore the UUID-linking if it's not relevant for them (f.ex enrichment processes/bots) * we can still represent linked events. I would like to add one little but important thing for the UUID linking idea: add a "link-type". Examples for link-types: * parent-child event * grouping types (all of these events belong to the same report) etc. With this triplet information , we are close to RDF (left-side, type, right-side) and thus we can (future-proof) represent any type of relation. A list of valid types needs to be documented in the IDF format page of course. So, I think with that, we can go ahead. Thanks, a. PS: and sorry that my feedback came a bit late, as said - code sprints.

1 0

CentOS 8 packages fixed
by Sebastian Wagner 21 Apr '21

21 Apr '21

Hi, The centos8 rpm package building failed in the last weeks because of upstream changes in the dependencies (naming). It took me a while to fix all the errors but it works again since approximately one week. If you encounter any errors, please reply back or create an issue. Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

pipeline failed
by Soni, Drupad 19 Apr '21

19 Apr '21

Hi Sebastian/Bernhard, After a while Intelmq worked smoothly now it is showing ES bot will start in 15 secs and pipeline is failed! Showing error below. Pipeline failed. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/redis/connection.py", line 484, in connect sock = self._connect() File "/usr/lib/python3/dist-packages/redis/connection.py", line 541, in _connect raise err File "/usr/lib/python3/dist-packages/redis/connection.py", line 529, in _connect sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/redis/client.py", line 667, in execute_command connection.send_command(*args) File "/usr/lib/python3/dist-packages/redis/connection.py", line 610, in send_command self.send_packed_command(self.pack_command(*args)) File "/usr/lib/python3/dist-packages/redis/connection.py", line 585, in send_packed_command self.connect() File "/usr/lib/python3/dist-packages/redis/connection.py", line 489, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/redis/connection.py", line 484, in connect sock = self._connect() File "/usr/lib/python3/dist-packages/redis/connection.py", line 541, in _connect raise err File "/usr/lib/python3/dist-packages/redis/connection.py", line 529, in _connect sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/intelmq/lib/pipeline.py", line 258, in _receive retval = self.pipe.lindex(self.internal_queue, -1) # returns None if no value File "/usr/lib/python3/dist-packages/redis/client.py", line 1311, in lindex return self.execute_command('LINDEX', name, index) File "/usr/lib/python3/dist-packages/redis/client.py", line 673, in execute_command connection.send_command(*args) File "/usr/lib/python3/dist-packages/redis/connection.py", line 610, in send_command self.send_packed_command(self.pack_command(*args)) File "/usr/lib/python3/dist-packages/redis/connection.py", line 585, in send_packed_command self.connect() File "/usr/lib/python3/dist-packages/redis/connection.py", line 489, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 272, in start self.process() File "/usr/lib/python3/dist-packages/intelmq/bots/outputs/elasticsearch/output.py", line 104, in process event = self.receive_message() File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 592, in receive_message message = self.__source_pipeline.receive() File "/usr/lib/python3/dist-packages/intelmq/lib/pipeline.py", line 124, in receive retval = self._receive() File "/usr/lib/python3/dist-packages/intelmq/lib/pipeline.py", line 267, in _receive raise exceptions.PipelineError(exc) intelmq.lib.exceptions.PipelineError: pipeline failed - ConnectionError('Error 111 connecting to 127.0.0.1:6379. Connection refused.',) Regards, Drupad Soni KPMG - Cyber Security Embassy Golf Links Business Park, Pebble Beach, 'B' Block, 1st & 2nd Floor, Off Intermediate Ring Road Mobile : +91 8140283894 Know more about our Cyber Security Services https://home.kpmg.com/in/en/home/services/advisory/risk-consulting/it-advis… ********************************************************************** KPMG (in India) allows reasonable personal use of the e-mail system. Views and opinions expressed in these communications do not necessarily represent those of KPMG (in India). ******************************************************************************************************* DISCLAIMER The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading "Received in error," send to postmaster1(a)kpmg.com, then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing KPMG client engagement letter. Opinions, conclusions and other information in this e-mail and any attachments that do not relate to the official business of the firm are neither given nor endorsed by it. KPMG cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. KPMG, an Indian partnership and a member firm of KPMG International Cooperative ("KPMG International"), a Swiss entity that serves as a coordinating entity for a network of independent firms operating under the KPMG name. KPMG International Cooperative (“KPMG International”) provides no services to clients. Each member firm of KPMG International Cooperative (“KPMG International”) is a legally distinct and separate entity and each describes itself as such. “Notwithstanding anything inconsistent contained in the meeting invite to which this acceptance pertains, this acceptance is restricted solely to confirming my availability for the proposed call and should not be construed in any manner as acceptance of any other terms or conditions. Specifically, nothing contained herein may be construed as an acceptance (or deemed acceptance) of any request or notification for recording of the call, which can be done only if it is based on my explicit and written consent and subject to the terms and conditions on which such consent has been granted” *******************************************************************************************************

2 1

Intelmq | ELK
by Drupad Soni 15 Apr '21

15 Apr '21

Hi Sebastian/Bernhard, I am unsure why es bot fails with error below, I have installed pip3 install elasticsearch==7.12 I have checked every single dependency error then too why does it fail. Bot initialization failed. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 164, in __init__ self.init() File "/usr/lib/python3/dist-packages/intelmq/bots/outputs/elasticsearch/output.py", line 58, in init raise MissingDependencyError('elasticsearch', version='>=5.0.0,<6.0.0') intelmq.lib.exceptions.MissingDependencyError: Could not load dependency 'elasticsearch', please install it with apt/yum/dnf/zypper (possibly named python3-elasticsearch) or pip3. Please note that this bot requires elasticsearch version >=5.0.0,<6.0.0! Requesting to guide me on this. Regards, Drupad Soni. Security Researcher. +918140283894. drupad8140(a)gmail.com Twitter: @drupad8140

2 3

IntelMQ Data format (IEP03 & IEP04) hackathon 2021-04-20 10h UTC (12h CEST)
by Sebastian Wagner 15 Apr '21

15 Apr '21

Dear community, The selected date for the IEP03 & IEP04 (data format) hackathon is: Thursday 22th April, 10h-12h UTC / 12h-14h CEST Location: https://jitsi.cert.ee/GreenGalaxiesDanceSeldom Unfortunately, there was no suitable date for all people who entered their interest in the poll. The main goal of the hackathon is to discuss the IEP03 and IEP04 proposal on changes to the data format and get a decision how to proceed. Here are the direct links to the propsals: IEP03 Multiple Values: https://lists.cert.at/pipermail/intelmq-dev/2021-March/000425.html IEP04 Meta Information and Data Exchange: https://lists.cert.at/pipermail/intelmq-dev/2021-March/000426.html see you next week, Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

Re: [IntelMQ-users] Error | elasticsearch bot doesn't work
by Sebastian Wagner 12 Apr '21

12 Apr '21

Hi, To get more insight why you redis server is not willing to start, we need more logs. `journalctl -x -e -u redis-server.service` may help. I again recommend you get some help from someone with linux system administration or consultancy, that will make life much easier for you and you can reach your goal faster. Sebastian On 4/12/21 11:59 AM, Drupad Soni wrote: > Hi Sebastian, > > No it is not working, Tried starting but it failed. > > sudo service redis status > ● redis-server.service - Advanced key-value store > Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; > vendor preset: enabled) > Active: failed (Result: exit-code) since Mon 2021-04-12 08:11:55 > UTC; 1h 45min ago > Docs: http://redis.io/documentation, > man:redis-server(1) > Process: 1399 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf > (code=exited, status=1/FAILURE) > > Apr 12 08:11:55 intelmqelkmisp systemd[1]: redis-server.service: > Service hold-off time over, scheduling restart. > Apr 12 08:11:55 intelmqelkmisp systemd[1]: redis-server.service: > Scheduled restart job, restart counter is at 5. > Apr 12 08:11:55 intelmqelkmisp systemd[1]: Stopped Advanced key-value > store. > Apr 12 08:11:55 intelmqelkmisp systemd[1]: redis-server.service: Start > request repeated too quickly. > Apr 12 08:11:55 intelmqelkmisp systemd[1]: redis-server.service: > Failed with result 'exit-code'. > Apr 12 08:11:55 intelmqelkmisp systemd[1]: Failed to start Advanced > key-value store. > Regards, > Drupad Soni. > Security Researcher. > +918140283894. > drupad8140(a)gmail.com <mailto:drupad8140@gmail.com> > Twitter: @drupad8140 > > > On Mon, Apr 12, 2021 at 3:26 PM Sebastian Wagner <wagner(a)cert.at > <mailto:wagner@cert.at>> wrote: > > Hi, > > Is your redis service running? > > On 4/12/21 11:51 AM, Drupad Soni wrote: >> Hi Sebastian/Bernhard, >> >> /I am receiving error as below/ >> / >> / >> 3:19 PM 45× Error loading bot queues information: Traceback (most >> recent call last): >> File "/usr/lib/python3/dist-packages/redis/connection.py", line >> 484, in connect >> sock = self._connect() >> File "/usr/lib/python3/dist-packages/redis/connection.py", line >> 541, in _connect >> raise err >> File "/usr/lib/python3/dist-packages/redis/connection.py", line >> 529, in _connect >> sock.connect(socket_address) >> ConnectionRefusedError: [Errno 111] Connection refused >> >> During handling of the above exception, another exception occurred: >> >> Traceback (most recent call last): >> File "/usr/lib/python3/dist-packages/redis/client.py", line 667, >> in execute_command >> connection.send_command(*args) >> File "/usr/lib/python3/dist-packages/redis/connection.py", line >> 610, in send_command >> self.send_packed_command(self.pack_command(*args)) >> File "/usr/lib/python3/dist-packages/redis/connection.py", line >> 585, in send_packed_command >> self.connect() >> File "/usr/lib/python3/dist-packages/redis/connection.py", line >> 489, in connect >> raise ConnectionError(self._error_message(e)) >> redis.exceptions.ConnectionError: Error 111 connecting to >> 127.0.0.1:6379 <http://127.0.0.1:6379>. Connection refused. >> >> During handling of the above exception, another exception occurred: >> >> Traceback (most recent call last): >> File "/usr/lib/python3/dist-packages/redis/connection.py", line >> 484, in connect >> sock = self._connect() >> File "/usr/lib/python3/dist-packages/redis/connection.py", line >> 541, in _connect >> raise err >> File "/usr/lib/python3/dist-packages/redis/connection.py", line >> 529, in _connect >> sock.connect(socket_address) >> ConnectionRefusedError: [Errno 111] Connection refused >> >> During handling of the above exception, another exception occurred: >> >> Traceback (most recent call last): >> File "/usr/lib/python3/dist-packages/intelmq/lib/pipeline.py", >> line 286, in count_queued_messages >> queue_dict[queue] = self.pipe.llen(queue) >> File "/usr/lib/python3/dist-pack.../ >> / >> Regards, >> Drupad Soni. >> Security Researcher. >> +918140283894. >> drupad8140(a)gmail.com <mailto:drupad8140@gmail.com> >> Twitter: @drupad8140 >> > -- > // Sebastian Wagner <wagner(a)cert.at> <mailto:wagner@cert.at> - T: +43 1 5056416 7201 > // CERT Austria - https://www.cert.at/ > // Eine Initiative der nic.at <http://nic.at> GmbH - https://www.nic.at/ > // Firmenbuchnummer 172568b, LG Salzburg > -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Error | elasticsearch bot doesn't work
by Drupad Soni 12 Apr '21

12 Apr '21

Hi Sebastian/Bernhard, *I am receiving error as below* 3:19 PM 45× Error loading bot queues information: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/redis/connection.py", line 484, in connect sock = self._connect() File "/usr/lib/python3/dist-packages/redis/connection.py", line 541, in _connect raise err File "/usr/lib/python3/dist-packages/redis/connection.py", line 529, in _connect sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/redis/client.py", line 667, in execute_command connection.send_command(*args) File "/usr/lib/python3/dist-packages/redis/connection.py", line 610, in send_command self.send_packed_command(self.pack_command(*args)) File "/usr/lib/python3/dist-packages/redis/connection.py", line 585, in send_packed_command self.connect() File "/usr/lib/python3/dist-packages/redis/connection.py", line 489, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/redis/connection.py", line 484, in connect sock = self._connect() File "/usr/lib/python3/dist-packages/redis/connection.py", line 541, in _connect raise err File "/usr/lib/python3/dist-packages/redis/connection.py", line 529, in _connect sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/intelmq/lib/pipeline.py", line 286, in count_queued_messages queue_dict[queue] = self.pipe.llen(queue) File "/usr/lib/python3/dist-pack... Regards, Drupad Soni. Security Researcher. +918140283894. drupad8140(a)gmail.com Twitter: @drupad8140

2 1

Event graph/ correlation graph
by Drupad Soni 11 Apr '21

11 Apr '21

Hi Bernhard, Is there way to create correlation or event graph on the basis of feeds? Using any other tool or in ELK/MISP? If there is a way to this please let me know. Regards, Drupad Soni Security Researcher +918140283894 drupad8140(a)gmail.com Twitter: @drupad8140

1 1

IntelMQ Data format Hackathon (IEP03 & IEP04): scheduling
by Sebastian Wagner 09 Apr '21

09 Apr '21

Dear community, Thank you for your input and interest on the proposed changes for IntelMQ's internal data format IEP03 and IEP04. To get the discussion forward and hopefully to get some conclusions, we propose to do a small *hackathon*. We think that this would be a great opportunity as well to bring together the IntelMQ community. If you are interested, please add your availability here until Tuesday 20th April: https://www.termino.gv.at/meet/en/p/219061bf603a2dbb91efe8af61ee4718-64494 The poll offers several 2h-slots in the week 19th-23rd April, we plan to select a date on Wednesday 21th April and announce it here, together with a location. kind regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

Need help on issue with installation of intelmq
by Kossi DOH 08 Apr '21

08 Apr '21

Hello teams Please is anyone able to help me solve this issue? I finished the installation of intelMQ and intelMQ manager. But when I open the manager I have the following error as shown below: [cid:image001.png@01D72C6F.4FC39A10] Attached is the log from the web server. Cordialement / Best Regards, Kossi DOH Analyste Cyber Securite CYBER DEFENSE AFRICA S.A.S. Mobile: +228 70 54 93 26 www.cda.tg<http://www.cda.tg> This information is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Unauthorised use of this information by a person or entity other than the intended recipient is prohibited by law. If you received this by mistake, please immediately contact the sender by email or phone and delete this information from any computer. Thank you.

3 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users