April 2021 - IntelMQ-users

mispfeedoutput bot feed
by Soni, Drupad 30 Apr '21

30 Apr '21

Hi Sebastian/Bernhard, Mispfeed-output bot failed. Error is as below, Bot initialization failed. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 164, in __init__ self.init() File "/usr/lib/python3/dist-packages/intelmq/bots/outputs/misp/output_feed.py", line 65, in init self.current_event.load_file(self.current_file) File "/usr/local/lib/python3.6/dist-packages/pymisp/mispevent.py", line 1598, in load_file self.load(f, … [View More]validate, metadata_only) File "/usr/local/lib/python3.6/dist-packages/pymisp/mispevent.py", line 1606, in load json_event = json.loads(json_event) File "/usr/lib/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Regards, Drupad Soni KPMG - Cyber Security Embassy Golf Links Business Park, Pebble Beach, 'B' Block, 1st & 2nd Floor, Off Intermediate Ring Road Mobile : +91 8140283894 Know more about our Cyber Security Services https://home.kpmg.com/in/en/home/services/advisory/risk-consulting/it-advis… ********************************************************************** KPMG (in India) allows reasonable personal use of the e-mail system. Views and opinions expressed in these communications do not necessarily represent those of KPMG (in India). ******************************************************************************************************* DISCLAIMER The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading "Received in error," send to postmaster1(a)kpmg.com, then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing KPMG client engagement letter. Opinions, conclusions and other information in this e-mail and any attachments that do not relate to the official business of the firm are neither given nor endorsed by it. KPMG cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. KPMG, an Indian partnership and a member firm of KPMG International Cooperative ("KPMG International"), a Swiss entity that serves as a coordinating entity for a network of independent firms operating under the KPMG name. KPMG International Cooperative (“KPMG International”) provides no services to clients. Each member firm of KPMG International Cooperative (“KPMG International”) is a legally distinct and separate entity and each describes itself as such. “Notwithstanding anything inconsistent contained in the meeting invite to which this acceptance pertains, this acceptance is restricted solely to confirming my availability for the proposed call and should not be construed in any manner as acceptance of any other terms or conditions. Specifically, nothing contained herein may be construed as an acceptance (or deemed acceptance) of any request or notification for recording of the call, which can be done only if it is based on my explicit and written consent and subject to the terms and conditions on which such consent has been granted” ******************************************************************************************************* [View Less]

2 5

Bot Unknown
by hatibu chande 29 Apr '21

29 Apr '21

Hello Team, I recently installed inetlmq and configured shadowsever API bot collector with shadowserverAPI parser, Cymru-Whois-Expert and File-Output but i got this error when running. Starting Botnet... Starting Cymru-Whois-Expert... Starting File-Output... Starting Shadowserver-JSON-Parser... Starting Shadowserver-Reports-API-Collector... Status of Bot Cymru-Whois-Expert is unknown: 'Unhandled error checking the process 18850 with commandline [].'. Cymru-Whois-Expert unknown Status of Bot … [View More]

2 1

Re: [IntelMQ-users] [IntelMQ -users] AttributeError: 'Parameters' object has no attribute 'logging_level'
by Sebastian Wagner 28 Apr '21

28 Apr '21

Hi, That's interesting, I wonder how it emptied itself. You can use this one here to fill it with the standard values: https://raw.githubusercontent.com/certtools/intelmq/2.3.2/intelmq/etc/defau… In IntelMQ 3.0, that file does not exist anymore and it's not necessary to specify all the "default default" values in a file at all :) Sebastian On 4/28/21 3:01 PM, hatibu chande wrote: > Hi, > > No logging_level field.., It looks like this {} only. > > Hatibu. > > … [View More]On Wed, Apr 28, 2021 at 3:55 PM Sebastian Wagner <wagner(a)cert.at > <mailto:wagner@cert.at>> wrote: > > Ok, how does you defaults.conf look like? in particular, is there > a logging_level field? > > On 4/28/21 2:53 PM, hatibu chande wrote: >> Hello Sebastian, >> >> I upgraded to version 2.3.2 but still got the same error. >> >> File "/usr/bin/intelmqctl", line 11, in <module> >> load_entry_point('intelmq==2.3.2', 'console_scripts', >> 'intelmqctl')() >> File >> "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line >> 1899, in main >> x = IntelMQController(interactive=True) >> File >> "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line >> 706, in __init__ >> self.logging_level = self.parameters.logging_level.upper() >> AttributeError: 'Parameters' object has no attribute 'logging_level' >> >> Regards, >> Hatibu. >> >> On Wed, Apr 28, 2021 at 3:19 PM Sebastian Wagner <wagner(a)cert.at >> <mailto:wagner@cert.at>> wrote: >> >> Dear hatibu, >> >> It looks like you are using an outdated version of IntelMQ >> (2.3.1). Please upgrade to the latest version 2.3.2 first, >> otherwise we might be chasing bugs which are already fixed. >> >> Sebastian >> >> P.S: you also posted to intelmq-dev, to which you are not >> subscribed and the message was thus held back. I rejected >> that one as posting to one list is enough. >> >> On 4/28/21 2:11 PM, hatibu chande wrote: >>> Hello Team, >>> >>> I recently installed IntelMQ and got this error when running >>> intelmqctl status command. Please can any one help me. >>> >>> File "/usr/bin/intelmqctl", line 11, in <module> >>> load_entry_point('intelmq==2.3.1', 'console_scripts', >>> 'intelmqctl')() >>> File >>> "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", >>> line 1899, in main >>> x = IntelMQController(interactive=True) >>> File >>> "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", >>> line 706, in __init__ >>> self.logging_level = self.parameters.logging_level.upper() >>> AttributeError: 'Parameters' object has no attribute >>> 'logging_level' >>> >>> >>> Regards, >>> >>> Hatibu. >>> >> -- >> // Sebastian Wagner <wagner(a)cert.at> <mailto:wagner@cert.at> - T: +43 676 898 298 7201 >> // CERT Austria - https://www.cert.at/ >> // Eine Initiative der nic.at <http://nic.at> GmbH - https://www.nic.at/ >> // Firmenbuchnummer 172568b, LG Salzburg >> >> > -- > // Sebastian Wagner <wagner(a)cert.at> <mailto:wagner@cert.at> - T: +43 676 898 298 7201 > // CERT Austria - https://www.cert.at/ > // Eine Initiative der nic.at <http://nic.at> GmbH - https://www.nic.at/ > // Firmenbuchnummer 172568b, LG Salzburg > -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg [View Less]

1 0

[IntelMQ -users] AttributeError: 'Parameters' object has no attribute 'logging_level'
by hatibu chande 28 Apr '21

28 Apr '21

Hello Team, I recently installed IntelMQ and got this error when running intelmqctl status command. Please can any one help me. File "/usr/bin/intelmqctl", line 11, in <module> load_entry_point('intelmq==2.3.1', 'console_scripts', 'intelmqctl')() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1899, in main x = IntelMQController(interactive=True) File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 706, in __init__ self.… [View More]

2 3

IntelMQ 2.3.2
by Sebastian Wagner 27 Apr '21

27 Apr '21

Dear community, April is nearing it's end and it's time to release a bunch of bugfixes. Please find below the list of changes. Thanks to all contributors for the issues reported and pull requests! The new version is already available on GitHub, PyPI, the deb+rpm repositories and DockerHub. Installation documentation: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade documentation: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html ### Core - `intelmq.… [View More]lib.harmonization`: - `TLP` type: accept value "yellow" for TLP level AMBER. ### Bots #### Collectors - `intelmq.bots.collectors.shadowserver.collector_reports_api`: - Handle timeouts by logging the error and continuing to next report (PR#1852 by Marius Karotkis and Sebastian Wagner, fixes #1823). #### Parsers - `intelmq.bots.parsers.shadowserver.config`: - Parse and harmonize field `end_time` as date in Feeds "Drone-Brute-Force" and "Amplification-DDoS-Victim" (PR#1833 by Mikk Margus Möll). - Add conversion function `convert_date_utc` which assumes UTC and sanitizes the data to datetime (by Sebastian Wagner, fixes #1848). - `intelmq.bots.parsers.shadowserver.parser_json`: - Use the overwrite parameter for optionally overwriting the "feed.name" field (by Sebastian Wagner). - `intelmq.bots.parsers.microsoft.parser_ctip`: - Handle fields `timestamp`, `timestamp_utc`, `source_ip`, `source_port`, `destination_ip`, `destination_port`, `computer_name`, `bot_id`, `asn`, `geo` in `Payload` of CTIP Azure format (PR#1841, PR#1851 and PR#1879 by Sebastian Wagner). - `intelmq.bots.parsers.shodan.parser`: - Added support for unique keys and verified vulns (PR#1835 by Mikk Margus Möll). - `intelmq.bots.parsers.cymru.parser_cap_program`: - Fix parsing in whitespace edge case in comments (PR#1870 by Alex Kaplan, fixes #1862). #### Experts - `intelmq.bots.experts.modify`: - Add a new rule to the example configuration to change the type of malicious-code events to `c2server` if the malware name indicates c2 (PR#1854 by Sebastian Wagner). - `intelmq.bots.experts.gethostbyname.expert`: - Fix handling of parameter `gaierrors_to_ignore` with value `None` (PR#1890 by Sebastian Wagner, fixes #1886). #### Outputs - `intelmq.bots.outputs.elasticsearch`: Fix log message on required elasticsearch library message (by Sebastian Wagner). ### Documentation - `dev/data-harmonization`: Fix taxonomy name "information gathering" should be "information-gathering" (by Sebastian Wagner). ### Tests - `intelmq.tests.bots.parsers.microsoft.test_parser_ctip_azure`: - Add test case for TLP level "YELLOW". ### Known issues - ParserBot: erroneous raw line recovery in error handling (#1850). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg [View Less]

1 0

IEP04: IntelMQ Data Format - Meta-Information
by Sebastian Waldbauer 22 Apr '21

22 Apr '21

Dear IntelMQ Developers and Users, nowadays security incidents are more important than 10 years ago. As IntelMQ can be used as core element for automated security incident handling, we would like to provide a way to share information with other intelmq instances. This proposal is also an alternative to IEP03 insofar as solving the "multiple values" is possible by using UUIDs so "link" related events in a backwards-compatible manner. If you're interested, please let us know, so we could … [View More]organize a hackathon for further discussions about the specification of the meta-information. Previously this idea was discussed in [0] and [1]. [0] https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectu… [1] https://github.com/certtools/intelmq/issues/1521 # IEP04: Internal Data Format: Meta Information and Data Exchange To ease data exchange between two or more IntelMQ instances, adding some meta-information to the events can make this sharing easier in certain regards. "Linking" events could be based on the same theory as `git` using it - with parent hashes ( we would call it UUID ). ### TL;DR Communication between one or more IntelMQ instances & exchange data with a backwards-compatible format. P2P or centralized architecture is a big topic, which has to be discussed after the format is being set. ### Why is metadata important? Short and simple. To avoid race conditions & being able to discard/drop already processed events from other instances. ### Meta information Metadata is used to transfer some general data, which is not likely related to the event itself. It's more or less just an information to keep events clear & sortable. A message could look like: { "meta": { "version": 1, # protocol version, so we are allowed to fallback to old versions too "uuid": { current: "cert_at:aaaa-bbbb-cccc-dddd" # format to be decided parent: "cert_at:xxxx-yyyy-zzzz-ffff" # format to be discussed, if not set -> current is the parent uuid }, "type": "event", "format": "intelmq", # i. e. this field could contain "n6" or "idea", so the receiving component can decode on demand. }, "payload": { # normal intelmq data "source.ip": "127.0.0.1", "source.fqdn": "example.com", "raw": base64-blob } } Tell us your opinion about adding non-standardized meta-information fields ( i. e. RTIR ticket number, origin, other local contact informationen ... and so on ) #### The UUID For the UUID there are multiple options: 1. Generate a random 128 bit UUID 2. A list of entities, which dealt with this event already. For example if an event was passed on from cert-at to cert-ee, the field could look like `!cert-at!cert-ee`. A message sending loop can be detected if the own name is already in this field upon reception. 3. Using CyCat: `publisher-short-name:project-short-name:UUID`. For example: `cert-at:intelmq:72ddb00c-2d0a-4eea-b7ac-ae122b8e6c3b`, or `cert-pl:n6:f60c9fb9-81f9-4e0b-8a44-ea41326a15b3`. Some more research and discussion is required before the implementation of this option. Have a look at https://www.cycat.org/services/concept/ for more details. 4. A hash: A benefit using a hash is that we're able to recalculate them on every intelmq instance. ### Exporting events to other systems In IntelMQ 2.x the events only comprise of the "payload" and no meta information. For local storages like file output or databases, the meta information may not be relevant in some use-cases. So it needs to be possible to export events *without* meta information, which is also the backwards-compatible behaviour. The "type" field exists in the current format as "__type" in the flat payload structure. In the output bots there's currently a boolean parameter `message_with_type` to include the field `__type` in the "export". For optionally exporting meta-information like uuid or format, a similar logic could be used. ### How can data exchange work? This now depends on how IntelMQ instances can communicate, either Peer-to-peer or via a central data hub. Both of them do have pro's and con's. #### P2P ( Peer 2 Peer ) Decentralized network + Less downtimes: A downtime of one instance, does not affect the whole network + Better privacy: data is not shared to an unrelated instance + More secure: data can optionally be encrypted (key-exchange between instances?) + Decentralized and local maintenance ~ Network latency depends on server locations - Networking issues may occur How would data exchange looks like between two instances: 1) Instance A has events which should be relayed to Instance B & C, because they're not sure who the actually receiver should be 2) Instance A ensures all messages have a UUID 3) Instance A sends the data to Instance B & Instance C 4) Instance B checks the data & they're sure that the data should be for Instance C 5) Instance C receives data from Instance A & Instance B 6) Instance C checks the UUID, which is the same & drops the package from Instance B #### (Central) Data hub + Less maintenance: Is maintained by the hub administrator + Central data storage (reports can optionally be cached to be downloaded later) ~ Central data analysis (e.g. statistics) is possible ~ Network latency depends on server locations - point of failure: if network problems occur, no exchange is possible As already seen above, data exchange here would be less complicated. The sending may look like: 1) Instance A has events which should be relayed to Instance B (e.g. different country) 2) Instance A ensures all messages have a UUID 3) Instance A sends these messages to the data hub The reception side can look like: 1) Instance B connects to central instance 2) Instance B queries and downloads all available messages 3) Upon reception, all messages are de-duplicated based on the UUID: a) If the UUID is already known, discard the message b) If the UUID has not been seen before, continue with processing To sum up, both exchange variants are useful. More research is needed, i. e. a mixed infrastructure with centralized parts but can be decentralized too. However, this shall not be neither the purpose nor the aim of this IEP. -- // Sebastian Waldbauer <waldbauer(a)cert.at> - T: +43 1 5056416 7202 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg [View Less]

6 8

IEP03: IntelMQ Data Format - Multiple Values
by Sebastian Waldbauer 22 Apr '21

22 Apr '21

Dear IntelMQ Developers and Users, an evaluation of current challenges with the internal data format led to the idea of allowing multiple values for one field in IntelMQ 3.0 (scheduled for June 2021)[0]. The idea is described below, including various advantages and disadvantages. We appreciate your input, opinion and analysis of further implications on this idea. We plan to evaluate the feedback that emerged in two weeks. [0] https://github.com/certtools/intelmq/blob/version-3.0-ideas/… [View More]

3 4

regarding IEP03
by L. Aaron Kaplan 22 Apr '21

22 Apr '21

Hi everyone, I wanted to send back a better write-up of my stance on IEP03 ("multiple values" in the IntelMQ internal data format). Alas, I was quite busy and I am sprinting to push out some code for our deadline at work. However, let me summarise it: I think the IEP03 is very well written, thank you a lot for this! Thinking this through was important and I think Sebastian Waldbauer did a great job. Reading it, I realised that my initial proposal of having multiple values is really breaking … [View More]

1 0

CentOS 8 packages fixed
by Sebastian Wagner 21 Apr '21

21 Apr '21

Hi, The centos8 rpm package building failed in the last weeks because of upstream changes in the dependencies (naming). It took me a while to fix all the errors but it works again since approximately one week. If you encounter any errors, please reply back or create an issue. Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

pipeline failed
by Soni, Drupad 19 Apr '21

19 Apr '21

Hi Sebastian/Bernhard, After a while Intelmq worked smoothly now it is showing ES bot will start in 15 secs and pipeline is failed! Showing error below. Pipeline failed. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/redis/connection.py", line 484, in connect sock = self._connect() File "/usr/lib/python3/dist-packages/redis/connection.py", line 541, in _connect raise err File "/usr/lib/python3/dist-packages/redis/connection.py", line 529, in _connect … [View More]sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/redis/client.py", line 667, in execute_command connection.send_command(*args) File "/usr/lib/python3/dist-packages/redis/connection.py", line 610, in send_command self.send_packed_command(self.pack_command(*args)) File "/usr/lib/python3/dist-packages/redis/connection.py", line 585, in send_packed_command self.connect() File "/usr/lib/python3/dist-packages/redis/connection.py", line 489, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/redis/connection.py", line 484, in connect sock = self._connect() File "/usr/lib/python3/dist-packages/redis/connection.py", line 541, in _connect raise err File "/usr/lib/python3/dist-packages/redis/connection.py", line 529, in _connect sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/intelmq/lib/pipeline.py", line 258, in _receive retval = self.pipe.lindex(self.internal_queue, -1) # returns None if no value File "/usr/lib/python3/dist-packages/redis/client.py", line 1311, in lindex return self.execute_command('LINDEX', name, index) File "/usr/lib/python3/dist-packages/redis/client.py", line 673, in execute_command connection.send_command(*args) File "/usr/lib/python3/dist-packages/redis/connection.py", line 610, in send_command self.send_packed_command(self.pack_command(*args)) File "/usr/lib/python3/dist-packages/redis/connection.py", line 585, in send_packed_command self.connect() File "/usr/lib/python3/dist-packages/redis/connection.py", line 489, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 272, in start self.process() File "/usr/lib/python3/dist-packages/intelmq/bots/outputs/elasticsearch/output.py", line 104, in process event = self.receive_message() File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 592, in receive_message message = self.__source_pipeline.receive() File "/usr/lib/python3/dist-packages/intelmq/lib/pipeline.py", line 124, in receive retval = self._receive() File "/usr/lib/python3/dist-packages/intelmq/lib/pipeline.py", line 267, in _receive raise exceptions.PipelineError(exc) intelmq.lib.exceptions.PipelineError: pipeline failed - ConnectionError('Error 111 connecting to 127.0.0.1:6379. Connection refused.',) Regards, Drupad Soni KPMG - Cyber Security Embassy Golf Links Business Park, Pebble Beach, 'B' Block, 1st & 2nd Floor, Off Intermediate Ring Road Mobile : +91 8140283894 Know more about our Cyber Security Services https://home.kpmg.com/in/en/home/services/advisory/risk-consulting/it-advis… ********************************************************************** KPMG (in India) allows reasonable personal use of the e-mail system. Views and opinions expressed in these communications do not necessarily represent those of KPMG (in India). ******************************************************************************************************* DISCLAIMER The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading "Received in error," send to postmaster1(a)kpmg.com, then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing KPMG client engagement letter. Opinions, conclusions and other information in this e-mail and any attachments that do not relate to the official business of the firm are neither given nor endorsed by it. KPMG cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. KPMG, an Indian partnership and a member firm of KPMG International Cooperative ("KPMG International"), a Swiss entity that serves as a coordinating entity for a network of independent firms operating under the KPMG name. KPMG International Cooperative (“KPMG International”) provides no services to clients. Each member firm of KPMG International Cooperative (“KPMG International”) is a legally distinct and separate entity and each describes itself as such. “Notwithstanding anything inconsistent contained in the meeting invite to which this acceptance pertains, this acceptance is restricted solely to confirming my availability for the proposed call and should not be construed in any manner as acceptance of any other terms or conditions. Specifically, nothing contained herein may be construed as an acceptance (or deemed acceptance) of any request or notification for recording of the call, which can be done only if it is based on my explicit and written consent and subject to the terms and conditions on which such consent has been granted” ******************************************************************************************************* [View Less]

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users April 2021