[Intelmq-users] MAIL ATTACHMENT FETCHER FAILS TO READ/PARSE CSV FILE.

patric sungura patsung87 at yahoo.com
Thu Mar 19 12:27:54 CET 2020


 
Hi Sebastian,Find output below ;sha256sum /usr/lib/python3.5/csv.py6d96e56ec22d1603ffed0f8d20f5b5090fb6ae4c7417126d09afc45447b34a64  /usr/lib/python3.5/csv.py

Below is error I get after I parse email;
2020-03-19 14:02:16,409 - ShadowServer-Parser - ERROR - Failed to parse line.Traceback (most recent call last):  File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 951, in process    events = list(filter(bool, value))  File "/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py", line 111, in parse_line    ' format or misconfiguration.'.format(shadowkey, self.feedname))ValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.2020-03-19 14:02:16,410 - ShadowServer-Parser - INFO - Dumping message to dump file.2020-03-19 14:02:16,514 - ShadowServer-Parser - INFO - Dumping message to dump file.2020-03-19 14:02:16,612 - ShadowServer-Parser - INFO - Sent 0 events and found 2 problem(s).

>From Dump;       "source_queue": "ShadowServer-Parser-queue",        "traceback": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 951, in process\n    events = list(filter(bool, value))\n  File \"/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py\", line 111, in parse_line\n    ' format or misconfiguration.'.format(shadowkey, self.feedname))\nValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.\n"    }
Thank you,Patrick
    On Thursday, March 19, 2020, 01:48:02 PM GMT+3, Sebastian Wagner <wagner at cert.at> wrote:  
 
  
Hi,
 
On a fresh, virgin and up-to-date xenial installation I get the same results as Bernhard, which differ from yours. I suggest to re-install the affected python3 package on your system (or, maybe just the whole python3-stack):
 sudo apt-get install --reinstall libpython3.5-stdlib

And then show the output of the commands again.

Sebsatian
 On 3/19/20 10:48 AM, patric sungura wrote:
  
 
 Dear team, Find below inputs as requested; 
    ##ls -l /usr/lib/python3.5/csv.py -rw-r--r-- 1 root root 16138 Mar 19 11:28 /usr/lib/python3.5/csv.py 
  ##sha256sum /usr/lib/python3.5/csv.py fbed6ea919f70b4fd29608ab9620785304fc50a4b90cd2dd5d2363a0b81d9e81  /usr/lib/python3.5/csv.py 
  Find line aroud line#96 
   79 class DictReader:  80     def __init__(self, f, fieldnames=None, restkey=None, restval=None,  81                  dialect="excel", *args, **kwds):  82         self._fieldnames = fieldnames   # list of keys for the dict  83         self.restkey = restkey          # key to catch long rows  84         self.restval = restval          # default value for short rows  85         self.reader = reader(f, dialect, *args, **kwds)  86         self.dialect = dialect  87         self.line_num = 0  88   89     def __iter__(self):  90         return self  91   92     @property  93     def fieldnames(self):  94         if self._fieldnames is None:  95             try:  96                 self._fieldnames = next(self.reader)  97             except StopIteration:  98                 pass  99         self.line_num = self.reader.line_num 100         return self._fieldnames 101  102     @fieldnames.setter 103     def fieldnames(self, value): 104         self._fieldnames = value   
  I have two shadow server Parser and now have recorded different errors; 
  ShadowServer-Parser-2 
    2020-03-19 12:31:07,318 - ShadowServer-Parser-2 - ERROR - Bot has found a problem. Traceback (most recent call last):   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start     self.process()   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process     for line in self.parse(report):   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict     self.csv_fieldnames = csv_reader.fieldnames   File "/usr/lib/python3.5/csv.py", line 96, in fieldnames     self._fieldnames = next(self.reader) NameError: name 'csv' is not defined 2020-03-19 12:31:07,319 - ShadowServer-Parser-2 - INFO - Dumping message to dump file. 
  ShadowServer-Parser
  2020-03-19 12:37:58,573 - ShadowServer-Parser - ERROR - Bot has found a problem. Traceback (most recent call last):   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start     self.process()   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process     for line in self.parse(report):   File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict     self.csv_fieldnames = csv_reader.fieldnames   File "/usr/lib/python3.5/csv.py", line 96, in fieldnames     self._fieldnames = next(self.reader) _csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode? 2020-03-19 12:37:58,574 - ShadowServer-Parser - INFO - Dumping message to dump file  
  
  
  Thank you very much, Patrick     On Thursday, March 19, 2020, 11:56:49 AM GMT+3, Bernhard Reiter <bernhard at intevation.de> wrote:  
  
   Am Mittwoch, 18. März 2020, 17:02:13 CET schrieben Sie:
 >  libpython3.5-stdlib:amd64 3.5.2-2ubuntu0~16.04.9 amd64
 
 This is the standard version, but still the backtrace looks strange,
 especially the last bit:
   File "/usr/lib/python3.5/csv.py", line 96, in fieldnames
   self._fieldnames = next(self.reader, dialect=csv.excel_tab)
   NameError: name 'csv' is not defined
 
 The second part is strange as the original sources [1] don't have the second
 parameter. So we should double check.
 Can you open the file `/usr/lib/python3.5/csv.py` and check line 96?
 
 Please send the whole function around the lines here.
 (Can you also post your mail in text-only format? This is something I can and 
 many others can read much faster, and thus can answer you faster. :) Thanks! )
 
 Can you also post the length, permissions and sha256 sum of the file?
 Here for comparison the data for my file:
 ls -l /usr/lib/python3.5/csv.py
 -rw-r--r-- 1 root root 16128 Oct  8 16:38 /usr/lib/python3.5/csv.py
 sha256sum /usr/lib/python3.5/csv.py
6d96e56ec22d1603ffed0f8d20f5b5090fb6ae4c7417126d09afc45447b34a64  /usr/lib/
 python3.5/csv.py
 
 Best Regards,
 Bernhard
 [1] https://github.com/python/cpython/blob/master/Lib/csv.py 
 
 -- 
 www.intevation.de/~bernhard  +49 541 33 508 3-3
 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner  -- 
 Listen-Einstellungen:
 https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
     
  -- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg -- 
Listen-Einstellungen:
 https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20200319/458a5f87/attachment.html>


More information about the Intelmq-users mailing list