[Intelmq-users] MAIL ATTACHMENT FETCHER FAILS TO READ/PARSE CSV FILE.

[patric sungura]

 Sebastian,Only python  fix.Find below  error at different time .
Current I receive csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?
@13:14 GMT    2020-03-19 16:14:27,444 - ShadowServer-Parser - ERROR - Bot has found a problem.Traceback (most recent call last):  File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start    self.process()  File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process    for line in self.parse(report):  File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict    self.csv_fieldnames = csv_reader.fieldnames  File "/usr/lib/python3.5/csv.py", line 96, in fieldnames    self._fieldnames = next(self.reader)_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?2020-03-19 16:14:27,445 - ShadowServer-Parser - INFO - Dumping message to dump file.

     "traceback": [            "Traceback (most recent call last):\n",            "  File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 267, in start\n    self.process()\n",            "  File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 941, in process\n    for line in self.parse(report):\n",            "  File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 884, in parse_csv_dict\n    self.csv_fieldnames = csv_reader.fieldnames\n",            "  File \"/usr/lib/python3.5/csv.py\", line 96, in fieldnames\n    self._fieldnames = next(self.reader)\n",            "_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?\n"        ]    }

@11:02 GMT2020-03-19 14:02:16,409 - ShadowServer-Parser - ERROR - Failed to parse line.Traceback (most recent call last):  File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 951, in process    events = list(filter(bool, value))  File "/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py", line 111, in parse_line    ' format or misconfiguration.'.format(shadowkey, self.feedname))ValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.2020-03-19 14:02:16,410 - ShadowServer-Parser - INFO - Dumping message to dump file.2020-03-19 14:02:16,514 - ShadowServer-Parser - INFO - Dumping message to dump file.2020-03-19 14:02:16,612 - ShadowServer-Parser - INFO - Sent 0 events and found 2 problem(s).

>From Dump;       "source_queue": "ShadowServer-Parser-queue",        "traceback": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 951, in process\n    events = list(filter(bool, value))\n  File \"/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py\", line 111, in parse_line\n    ' format or misconfiguration.'.format(shadowkey, self.feedname))\nValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.\n"    }Your assistance is highly requested.
Thank you,Patrick
    On Thursday, March 19, 2020, 04:33:45 PM GMT+3, Sebastian Wagner <wagner at cert.at> wrote:  
 
  Hi, 
  On 3/19/20 12:27 PM, patric sungura wrote:
  
Find output below ;   sha256sum /usr/lib/python3.5/csv.py 6d96e56ec22d1603ffed0f8d20f5b5090fb6ae4c7417126d09afc45447b34a64  /usr/lib/python3.5/csv.py   
 
Great, so we now fixed the Python problem :)
 
 On 3/19/20 2:29 PM, patric sungura wrote:
  
 
    2020-03-19 16:14:27,444 - ShadowServer-Parser - ERROR - Bot has found a problem.   Traceback (most recent call last):   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start     self.process()   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process     for line in self.parse(report):   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict     self.csv_fieldnames = csv_reader.fieldnames   File "/usr/lib/python3.5/csv.py", line 96, in fieldnames     self._fieldnames = next(self.reader) _csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?    In the previous mail the error was different - so what changed? 
         2020-03-19 14:02:16,409 - ShadowServer-Parser - ERROR - Failed to parse line. Traceback (most recent call last):   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 951, in process     events = list(filter(bool, value))   File"/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py", line 111, in parse_line     ' format or misconfiguration.'.format(shadowkey, self.feedname)) ValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.         
 Sebastian
 -- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg -- 
Listen-Einstellungen:
 https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20200319/932821b5/attachment.html>