[Intelmq-users] elasticsearch parsing exception

[Tomislav Protega]

Hi,

recently I came up into elasticsearch parsing exception.
Dump is attached below.

It only happens when it processes data from Blueliv Crimeserver and
Shadowserver-Open-XDMCP collectors.

Not so far ago my elasticsearch output bot didn't throw that exception.

Currently I'm using intelmq 1.0.2 and intelmq-manager 0.3.1, all
installed from .deb package and python client elasticsearch 6.0.0.

Anyone experienced the same?

Thanks for the efforts.

Regards,

-- 
Tomislav
-------------- next part --------------
"message": "{\"time.observation\": \"2017-12-14T20:20:09+00:00\", \"source.allocated\": \"2005-02-16T00:00:00+00:00\", \"source.geolocation.cc\": \"HR\", \"source.geolocation.region\": \"SPLITAKO-DALMATINSKA\", \"classification.type\": \"vulnerable service\", \"source.port\": 177, \"__type\": \"Event\", \"protocol.transport\": \"udp\", \"source.network\": \"85.114.32.0/19\", \"classification.taxonomy\": \"Vulnerable\", \"source.registry\": \"RIPE\", \"extra\": \"{\\\"opcode\\\": \\\"Willing\\\", \\\"reported_hostname\\\": \\\"m15meridijan\\\", \\\"size\\\": \\\"52\\\", \\\"status\\\": \\\"Linux 2.6.32-300.10.1.el5uek\\\", \\\"tag\\\": \\\"xdmcp\\\"}\", \"source.asn\": 34594, \"classification.identifier\": \"openxdmcp\", \"feed.provider\": \"ShadowServer\", \"protocol.application\": \"xdmcp\", \"source.as_name\": \"OT-AS, HR\", \"source.geolocation.city\": \"SPLIT\", \"feed.name\": \"Open-XDMCP\", \"time.source\": \"2017-12-06T01:34:42+00:00\", \"source.ip\": \"85.114.48.198\", \"raw\": \"InRpbWVzdGFtcCIsImlwIiwicHJvdG9jb2wiLCJwb3J0IiwiaG9zdG5hbWUiLCJ0YWciLCJhc24iLCJnZW8iLCJyZWdpb24iLCJjaXR5IiwibmFpY3MiLCJzaWMiLCJvcGNvZGUiLCJyZXBvcnRlZF9ob3N0bmFtZSIsInN0YXR1cyIsInNpemUiCiIyMDE3LTEyLTA2IDAxOjM0OjQyIiwiODUuMTE0LjQ4LjE5OCIsInVkcCIsIjE3NyIsIiIsInhkbWNwIiwiMzQ1OTQiLCJIUiIsIlNQTElUQUtPLURBTE1BVElOU0tBIiwiU1BMSVQiLCIwIiwiMCIsIldpbGxpbmciLCJtMTVtZXJpZGlqYW4iLCJMaW51eCAyLjYuMzItMzAwLjEwLjEuZWw1dWVrIiwiNTIiCg==\", \"feed.accuracy\": 100.0}",
        "source_queue": "elasticsearch-output-ALL-queue",
        "traceback": [
            "Traceback (most recent call last):\n",
            "  File \"/usr/lib/python3/dist-packages/intelmq/lib/bot.py\", line 144, in start\n    self.process()\n",
            "  File \"/usr/lib/python3/dist-packages/intelmq/bots/outputs/elasticsearch/output.py\", line 83, in process\n    body=event_dict)\n",
            "  File \"/usr/local/lib/python3.5/dist-packages/elasticsearch/client/utils.py\", line 76, in _wrapped\n    return func(*args, params=params, **kwargs)\n",
            "  File \"/usr/local/lib/python3.5/dist-packages/elasticsearch/client/__init__.py\", line 300, in index\n    _make_path(index, doc_type, id), params=params, body=body)\n",
            "  File \"/usr/local/lib/python3.5/dist-packages/elasticsearch/transport.py\", line 314, in perform_request\n    status, headers, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)\n",
            "  File \"/usr/local/lib/python3.5/dist-packages/elasticsearch/connection/http_urllib3.py\", line 161, in perform_request\n    self._raise_error(response.status, raw_data)\n",
            "  File \"/usr/local/lib/python3.5/dist-packages/elasticsearch/connection/base.py\", line 125, in _raise_error\n    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)\n",
            "elasticsearch.exceptions.RequestError: TransportError(400, 'mapper_parsing_exception', 'failed to parse [extra_status]')\n"
        ]
    },

============================================================================

        "message": "{\"time.source\": \"2017-10-10T13:44:50+00:00\", \"source.geolocation.latitude\": 45.1667, \"source.reverse_dns\": \"lin-p13.infonet.hr\", \"source.registry\": \"RIPE\", \"source.allocated\": \"2012-09-10T00:00:00+00:00\", \"__type\": \"Event\", \"source.ip\": \"91.234.46.213\", \"classification.type\": \"phishing\", \"source.as_name\": \"SEDMIODJEL-AS, HR\", \"source.geolocation.longitude\": 15.5, \"extra\": \"{\\\"confidence\\\": 2, \\\"status\\\": \\\"ONLINE\\\", \\\"time_first_seen\\\": \\\"2017-05-29T20:48:28+0200\\\", \\\"time_updated\\\": \\\"2017-10-10T13:44:51+0200\\\"}\", \"time.observation\": \"2017-12-30T02:30:11+00:00\", \"feed.name\": \"Blueliv Crimeserver\", \"feed.provider\": \"Blueliv\", \"source.url\": \"http://friscic-kastel.com/wp-includes/pomo/ssh.php/\", \"source.network\": \"91.234.46.0/24\", \"source.geolocation.cc\": \"HR\", \"feed.accuracy\": 100.0, \"raw\": \"eyJjb25maWRlbmNlIjogMiwgImNvdW50cnkiOiAiSFIiLCAiZmlyc3RTZWVuQXQiOiAiMjAxNy0wNS0yOVQyMDo0ODoyOCswMjAwIiwgImlwIjogIjkxLjIzNC40Ni4yMTMiLCAibGFzdFNlZW5BdCI6ICIyMDE3LTEwLTEwVDEzOjQ0OjUwKzAyMDAiLCAibGF0aXR1ZGUiOiA0NS4xNjY3LCAibG9uZ2l0dWRlIjogMTUuNSwgInN0YXR1cyI6ICJPTkxJTkUiLCAidHlwZSI6ICJQSElTSElORyIsICJ1cGRhdGVkQXQiOiAiMjAxNy0xMC0xMFQxMzo0NDo1MSswMjAwIiwgInVybCI6ICJodHRwOi8vZnJpc2NpYy1rYXN0ZWwuY29tL3dwLWluY2x1ZGVzL3BvbW8vc3NoLnBocC8ifQ==\", \"source.asn\": 198785}",
        "source_queue": "Elasticsearch-Output-ALL-queue",
        "traceback": [
            "Traceback (most recent call last):\n",
            "  File \"/usr/lib/python3/dist-packages/intelmq/lib/bot.py\", line 144, in start\n    self.process()\n",
            "  File \"/usr/lib/python3/dist-packages/intelmq/bots/outputs/elasticsearch/output.py\", line 83, in process\n    body=event_dict)\n",
            "  File \"/usr/local/lib/python3.5/dist-packages/elasticsearch/client/utils.py\", line 76, in _wrapped\n    return func(*args, params=params, **kwargs)\n",
            "  File \"/usr/local/lib/python3.5/dist-packages/elasticsearch/client/__init__.py\", line 300, in index\n    _make_path(index, doc_type, id), params=params, body=body)\n",
            "  File \"/usr/local/lib/python3.5/dist-packages/elasticsearch/transport.py\", line 314, in perform_request\n    status, headers, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)\n",
            "  File \"/usr/local/lib/python3.5/dist-packages/elasticsearch/connection/http_urllib3.py\", line 161, in perform_request\n    self._raise_error(response.status, raw_data)\n",
            "  File \"/usr/local/lib/python3.5/dist-packages/elasticsearch/connection/base.py\", line 125, in _raise_error\n    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)\n",
            "elasticsearch.exceptions.RequestError: TransportError(400, 'mapper_parsing_exception', 'failed to parse [extra_status]')\n"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20180102/6970ba19/attachment.sig>