Hi Team,Now I get this error on parser after the BOT read the email;
2020-03-19 16:14:27,444 - ShadowServer-Parser - ERROR - Bot has found a problem.Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start self.process() File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process for line in self.parse(report): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict self.csv_fieldnames = csv_reader.fieldnames File "/usr/lib/python3.5/csv.py", line 96, in fieldnames self._fieldnames = next(self.reader)_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?2020-03-19 16:14:27,445 - ShadowServer-Parser - INFO - Dumping message to dump file.
"traceback": [ "Traceback (most recent call last):\n", " File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start\n self.process()\n", " File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process\n for line in self.parse(report):\n", " File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict\n self.csv_fieldnames = csv_reader.fieldnames\n", " File "/usr/lib/python3.5/csv.py", line 96, in fieldnames\n self._fieldnames = next(self.reader)\n", "_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?\n" ] }
kr,Patrick On Thursday, March 19, 2020, 02:27:56 PM GMT+3, patric sungura patsung87@yahoo.com wrote:
Hi Sebastian,Find output below ;sha256sum /usr/lib/python3.5/csv.py6d96e56ec22d1603ffed0f8d20f5b5090fb6ae4c7417126d09afc45447b34a64 /usr/lib/python3.5/csv.py
Below is error I get after I parse email; 2020-03-19 14:02:16,409 - ShadowServer-Parser - ERROR - Failed to parse line.Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 951, in process events = list(filter(bool, value)) File "/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py", line 111, in parse_line ' format or misconfiguration.'.format(shadowkey, self.feedname))ValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.2020-03-19 14:02:16,410 - ShadowServer-Parser - INFO - Dumping message to dump file.2020-03-19 14:02:16,514 - ShadowServer-Parser - INFO - Dumping message to dump file.2020-03-19 14:02:16,612 - ShadowServer-Parser - INFO - Sent 0 events and found 2 problem(s).
From Dump; "source_queue": "ShadowServer-Parser-queue", "traceback": "Traceback (most recent call last):\n File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 951, in process\n events = list(filter(bool, value))\n File "/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py", line 111, in parse_line\n ' format or misconfiguration.'.format(shadowkey, self.feedname))\nValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.\n" }
Thank you,Patrick On Thursday, March 19, 2020, 01:48:02 PM GMT+3, Sebastian Wagner wagner@cert.at wrote:
Hi,
On a fresh, virgin and up-to-date xenial installation I get the same results as Bernhard, which differ from yours. I suggest to re-install the affected python3 package on your system (or, maybe just the whole python3-stack): sudo apt-get install --reinstall libpython3.5-stdlib
And then show the output of the commands again.
Sebsatian On 3/19/20 10:48 AM, patric sungura wrote:
Dear team, Find below inputs as requested; ##ls -l /usr/lib/python3.5/csv.py -rw-r--r-- 1 root root 16138 Mar 19 11:28 /usr/lib/python3.5/csv.py ##sha256sum /usr/lib/python3.5/csv.py fbed6ea919f70b4fd29608ab9620785304fc50a4b90cd2dd5d2363a0b81d9e81 /usr/lib/python3.5/csv.py Find line aroud line#96 79 class DictReader: 80 def __init__(self, f, fieldnames=None, restkey=None, restval=None, 81 dialect="excel", *args, **kwds): 82 self._fieldnames = fieldnames # list of keys for the dict 83 self.restkey = restkey # key to catch long rows 84 self.restval = restval # default value for short rows 85 self.reader = reader(f, dialect, *args, **kwds) 86 self.dialect = dialect 87 self.line_num = 0 88 89 def __iter__(self): 90 return self 91 92 @property 93 def fieldnames(self): 94 if self._fieldnames is None: 95 try: 96 self._fieldnames = next(self.reader) 97 except StopIteration: 98 pass 99 self.line_num = self.reader.line_num 100 return self._fieldnames 101 102 @fieldnames.setter 103 def fieldnames(self, value): 104 self._fieldnames = value I have two shadow server Parser and now have recorded different errors; ShadowServer-Parser-2 2020-03-19 12:31:07,318 - ShadowServer-Parser-2 - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start self.process() File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process for line in self.parse(report): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict self.csv_fieldnames = csv_reader.fieldnames File "/usr/lib/python3.5/csv.py", line 96, in fieldnames self._fieldnames = next(self.reader) NameError: name 'csv' is not defined 2020-03-19 12:31:07,319 - ShadowServer-Parser-2 - INFO - Dumping message to dump file. ShadowServer-Parser 2020-03-19 12:37:58,573 - ShadowServer-Parser - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start self.process() File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process for line in self.parse(report): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict self.csv_fieldnames = csv_reader.fieldnames File "/usr/lib/python3.5/csv.py", line 96, in fieldnames self._fieldnames = next(self.reader) _csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode? 2020-03-19 12:37:58,574 - ShadowServer-Parser - INFO - Dumping message to dump file
Thank you very much, Patrick On Thursday, March 19, 2020, 11:56:49 AM GMT+3, Bernhard Reiter bernhard@intevation.de wrote:
Am Mittwoch, 18. März 2020, 17:02:13 CET schrieben Sie:
libpython3.5-stdlib:amd64 3.5.2-2ubuntu0~16.04.9 amd64
This is the standard version, but still the backtrace looks strange, especially the last bit: File "/usr/lib/python3.5/csv.py", line 96, in fieldnames self._fieldnames = next(self.reader, dialect=csv.excel_tab) NameError: name 'csv' is not defined
The second part is strange as the original sources [1] don't have the second parameter. So we should double check. Can you open the file `/usr/lib/python3.5/csv.py` and check line 96?
Please send the whole function around the lines here. (Can you also post your mail in text-only format? This is something I can and many others can read much faster, and thus can answer you faster. :) Thanks! )
Can you also post the length, permissions and sha256 sum of the file? Here for comparison the data for my file: ls -l /usr/lib/python3.5/csv.py -rw-r--r-- 1 root root 16128 Oct 8 16:38 /usr/lib/python3.5/csv.py sha256sum /usr/lib/python3.5/csv.py 6d96e56ec22d1603ffed0f8d20f5b5090fb6ae4c7417126d09afc45447b34a64 /usr/lib/ python3.5/csv.py
Best Regards, Bernhard [1] https://github.com/python/cpython/blob/master/Lib/csv.py
-- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner -- Listen-Einstellungen: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
-- // Sebastian Wagner wagner@cert.at - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg -- Listen-Einstellungen: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users