[Intelmq-users] MAIL ATTACHMENT FETCHER FAILS TO READ/PARSE CSV FILE.

Thu Mar 19 11:47:48 CET 2020

Hi,

On a fresh, virgin and up-to-date xenial installation I get the same
results as Bernhard, which differ from yours. I suggest to re-install
the affected python3 package on your system (or, maybe just the whole
python3-stack):

|sudo apt-get install --reinstall libpython3.5-stdlib And then show the
output of the commands again. Sebsatian |

On 3/19/20 10:48 AM, patric sungura wrote:
> Dear team,
> Find below inputs as requested;
>
> *##ls -l /usr/lib/python3.5/csv.py*
> -rw-r--r-- 1 root root 16138 Mar 19 11:28 /usr/lib/python3.5/csv.py
>
> *##sha256sum /usr/lib/python3.5/csv.py*
> fbed6ea919f70b4fd29608ab9620785304fc50a4b90cd2dd5d2363a0b81d9e81 
> /usr/lib/python3.5/csv.py
>
> *Find line aroud line#96*
>
>  79 class DictReader:
>  80     def __init__(self, f, fieldnames=None, restkey=None, restval=None,
>  81                  dialect="excel", *args, **kwds):
>  82         self._fieldnames = fieldnames   # list of keys for the dict
>  83         self.restkey = restkey          # key to catch long rows
>  84         self.restval = restval          # default value for short rows
>  85         self.reader = reader(f, dialect, *args, **kwds)
>  86         self.dialect = dialect
>  87         self.line_num = 0
>  88 
>  89     def __iter__(self):
>  90         return self
>  91 
>  92     @property
>  93     def fieldnames(self):
>  94         if self._fieldnames is None:
>  95             try:
> * 96                 self._fieldnames = next(self.reader)*
>  97             except StopIteration:
>  98                 pass
>  99         self.line_num = self.reader.line_num
> 100         return self._fieldnames
> 101 
> 102     @fieldnames.setter
> 103     def fieldnames(self, value):
> 104         self._fieldnames = value 
>
> I have two shadow server Parser and now have recorded different errors;
>
> *ShadowServer-Parser-2 *
> 2020-03-19 12:31:07,318 - ShadowServer-Parser-2 - ERROR - Bot has
> found a problem.
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py",
> line 267, in start
>     self.process()
>   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py",
> line 941, in process
>     for line in self.parse(report):
>   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py",
> line 884, in parse_csv_dict
>     self.csv_fieldnames = csv_reader.fieldnames
>   File "/usr/lib/python3.5/csv.py", line 96, in fieldnames
>     self._fieldnames = next(self.reader)
> *NameError: name 'csv' is not defined*
> 2020-03-19 12:31:07,319 - ShadowServer-Parser-2 - INFO - Dumping
> message to dump file.
>
> *ShadowServer-Parser*
> 2020-03-19 12:37:58,573 - ShadowServer-Parser - ERROR - Bot has found
> a problem.
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py",
> line 267, in start
>     self.process()
>   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py",
> line 941, in process
>     for line in self.parse(report):
>   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py",
> line 884, in parse_csv_dict
>     self.csv_fieldnames = csv_reader.fieldnames
>   File "/usr/lib/python3.5/csv.py", line 96, in fieldnames
>     self._fieldnames = next(self.reader)
> _*csv.Error: new-line character seen in unquoted field - do you need
> to open the file in universal-newline mode?*
> 2020-03-19 12:37:58,574 - ShadowServer-Parser - INFO - Dumping message
> to dump file
>
>
>
> Thank you very much,
> Patrick
> On Thursday, March 19, 2020, 11:56:49 AM GMT+3, Bernhard Reiter
> <bernhard at intevation.de> wrote:
>
>
> Am Mittwoch, 18. März 2020, 17:02:13 CET schrieben Sie:
> >  libpython3.5-stdlib:amd64 3.5.2-2ubuntu0~16.04.9 amd64
>
> This is the standard version, but still the backtrace looks strange,
> especially the last bit:
>   File "/usr/lib/python3.5/csv.py", line 96, in fieldnames
>   self._fieldnames = next(self.reader, dialect=csv.excel_tab)
>   NameError: name 'csv' is not defined
>
> The second part is strange as the original sources [1] don't have the
> second
> parameter. So we should double check.
> Can you open the file `/usr/lib/python3.5/csv.py` and check line 96?
>
> Please send the whole function around the lines here.
> (Can you also post your mail in text-only format? This is something I
> can and
> many others can read much faster, and thus can answer you faster. :)
> Thanks! )
>
> Can you also post the length, permissions and sha256 sum of the file?
> Here for comparison the data for my file:
> ls -l /usr/lib/python3.5/csv.py
> -rw-r--r-- 1 root root 16128 Oct  8 16:38 /usr/lib/python3.5/csv.py
> sha256sum /usr/lib/python3.5/csv.py
> 6d96e56ec22d1603ffed0f8d20f5b5090fb6ae4c7417126d09afc45447b34a64 
> /usr/lib/
> python3.5/csv.py
>
> Best Regards,
> Bernhard
> [1] https://github.com/python/cpython/blob/master/Lib/csv.py
>
>
> -- 
> www.intevation.de/~bernhard  +49 541 33 508 3-3
> Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
> Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
> -- 
> Listen-Einstellungen:
> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
>
-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20200319/2b4e6704/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20200319/2b4e6704/attachment.sig>