[Intelmq-users] MAIL ATTACHMENT FETCHER FAILS TO READ/PARSE CSV FILE.
patric sungura
patsung87 at yahoo.com
Thu Mar 19 14:29:40 CET 2020
Hi Team,Now I get this error on parser after the BOT read the email;
2020-03-19 16:14:27,444 - ShadowServer-Parser - ERROR - Bot has found a problem.Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start self.process() File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process for line in self.parse(report): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict self.csv_fieldnames = csv_reader.fieldnames File "/usr/lib/python3.5/csv.py", line 96, in fieldnames self._fieldnames = next(self.reader)_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?2020-03-19 16:14:27,445 - ShadowServer-Parser - INFO - Dumping message to dump file.
"traceback": [ "Traceback (most recent call last):\n", " File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 267, in start\n self.process()\n", " File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 941, in process\n for line in self.parse(report):\n", " File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 884, in parse_csv_dict\n self.csv_fieldnames = csv_reader.fieldnames\n", " File \"/usr/lib/python3.5/csv.py\", line 96, in fieldnames\n self._fieldnames = next(self.reader)\n", "_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?\n" ] }
kr,Patrick
On Thursday, March 19, 2020, 02:27:56 PM GMT+3, patric sungura <patsung87 at yahoo.com> wrote:
Hi Sebastian,Find output below ;sha256sum /usr/lib/python3.5/csv.py6d96e56ec22d1603ffed0f8d20f5b5090fb6ae4c7417126d09afc45447b34a64 /usr/lib/python3.5/csv.py
Below is error I get after I parse email;
2020-03-19 14:02:16,409 - ShadowServer-Parser - ERROR - Failed to parse line.Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 951, in process events = list(filter(bool, value)) File "/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py", line 111, in parse_line ' format or misconfiguration.'.format(shadowkey, self.feedname))ValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.2020-03-19 14:02:16,410 - ShadowServer-Parser - INFO - Dumping message to dump file.2020-03-19 14:02:16,514 - ShadowServer-Parser - INFO - Dumping message to dump file.2020-03-19 14:02:16,612 - ShadowServer-Parser - INFO - Sent 0 events and found 2 problem(s).
>From Dump; "source_queue": "ShadowServer-Parser-queue", "traceback": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 951, in process\n events = list(filter(bool, value))\n File \"/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py\", line 111, in parse_line\n ' format or misconfiguration.'.format(shadowkey, self.feedname))\nValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.\n" }
Thank you,Patrick
On Thursday, March 19, 2020, 01:48:02 PM GMT+3, Sebastian Wagner <wagner at cert.at> wrote:
Hi,
On a fresh, virgin and up-to-date xenial installation I get the same results as Bernhard, which differ from yours. I suggest to re-install the affected python3 package on your system (or, maybe just the whole python3-stack):
sudo apt-get install --reinstall libpython3.5-stdlib
And then show the output of the commands again.
Sebsatian
On 3/19/20 10:48 AM, patric sungura wrote:
Dear team, Find below inputs as requested;
##ls -l /usr/lib/python3.5/csv.py -rw-r--r-- 1 root root 16138 Mar 19 11:28 /usr/lib/python3.5/csv.py
##sha256sum /usr/lib/python3.5/csv.py fbed6ea919f70b4fd29608ab9620785304fc50a4b90cd2dd5d2363a0b81d9e81 /usr/lib/python3.5/csv.py
Find line aroud line#96
79 class DictReader: 80 def __init__(self, f, fieldnames=None, restkey=None, restval=None, 81 dialect="excel", *args, **kwds): 82 self._fieldnames = fieldnames # list of keys for the dict 83 self.restkey = restkey # key to catch long rows 84 self.restval = restval # default value for short rows 85 self.reader = reader(f, dialect, *args, **kwds) 86 self.dialect = dialect 87 self.line_num = 0 88 89 def __iter__(self): 90 return self 91 92 @property 93 def fieldnames(self): 94 if self._fieldnames is None: 95 try: 96 self._fieldnames = next(self.reader) 97 except StopIteration: 98 pass 99 self.line_num = self.reader.line_num 100 return self._fieldnames 101 102 @fieldnames.setter 103 def fieldnames(self, value): 104 self._fieldnames = value
I have two shadow server Parser and now have recorded different errors;
ShadowServer-Parser-2
2020-03-19 12:31:07,318 - ShadowServer-Parser-2 - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start self.process() File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process for line in self.parse(report): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict self.csv_fieldnames = csv_reader.fieldnames File "/usr/lib/python3.5/csv.py", line 96, in fieldnames self._fieldnames = next(self.reader) NameError: name 'csv' is not defined 2020-03-19 12:31:07,319 - ShadowServer-Parser-2 - INFO - Dumping message to dump file.
ShadowServer-Parser
2020-03-19 12:37:58,573 - ShadowServer-Parser - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start self.process() File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process for line in self.parse(report): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict self.csv_fieldnames = csv_reader.fieldnames File "/usr/lib/python3.5/csv.py", line 96, in fieldnames self._fieldnames = next(self.reader) _csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode? 2020-03-19 12:37:58,574 - ShadowServer-Parser - INFO - Dumping message to dump file
Thank you very much, Patrick On Thursday, March 19, 2020, 11:56:49 AM GMT+3, Bernhard Reiter <bernhard at intevation.de> wrote:
Am Mittwoch, 18. März 2020, 17:02:13 CET schrieben Sie:
> libpython3.5-stdlib:amd64 3.5.2-2ubuntu0~16.04.9 amd64
This is the standard version, but still the backtrace looks strange,
especially the last bit:
File "/usr/lib/python3.5/csv.py", line 96, in fieldnames
self._fieldnames = next(self.reader, dialect=csv.excel_tab)
NameError: name 'csv' is not defined
The second part is strange as the original sources [1] don't have the second
parameter. So we should double check.
Can you open the file `/usr/lib/python3.5/csv.py` and check line 96?
Please send the whole function around the lines here.
(Can you also post your mail in text-only format? This is something I can and
many others can read much faster, and thus can answer you faster. :) Thanks! )
Can you also post the length, permissions and sha256 sum of the file?
Here for comparison the data for my file:
ls -l /usr/lib/python3.5/csv.py
-rw-r--r-- 1 root root 16128 Oct 8 16:38 /usr/lib/python3.5/csv.py
sha256sum /usr/lib/python3.5/csv.py
6d96e56ec22d1603ffed0f8d20f5b5090fb6ae4c7417126d09afc45447b34a64 /usr/lib/
python3.5/csv.py
Best Regards,
Bernhard
[1] https://github.com/python/cpython/blob/master/Lib/csv.py
--
www.intevation.de/~bernhard +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner --
Listen-Einstellungen:
https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
--
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg --
Listen-Einstellungen:
https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20200319/41e1936b/attachment.html>
More information about the Intelmq-users
mailing list