Hi Team,

Now I get this error on parser after the BOT read the email;

2020-03-19 16:14:27,444 - ShadowServer-Parser - ERROR - Bot has found a problem.

Traceback (most recent call last):

File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start

self.process()

File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process

for line in self.parse(report):

File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict

self.csv_fieldnames = csv_reader.fieldnames

File "/usr/lib/python3.5/csv.py", line 96, in fieldnames

self._fieldnames = next(self.reader)

_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

2020-03-19 16:14:27,445 - ShadowServer-Parser - INFO - Dumping message to dump file.

"traceback": [

"Traceback (most recent call last):\n",

" File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 267, in start\n self.process()\n",

" File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 941, in process\n for line in self.parse(report):\n",

" File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 884, in parse_csv_dict\n self.csv_fieldnames = csv_reader.fieldnames\n",

" File \"/usr/lib/python3.5/csv.py\", line 96, in fieldnames\n self._fieldnames = next(self.reader)\n",

"_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?\n"

]

}

kr,

Patrick

On Thursday, March 19, 2020, 02:27:56 PM GMT+3, patric sungura <patsung87@yahoo.com> wrote:

Hi Sebastian,

Find output below ;

sha256sum /usr/lib/python3.5/csv.py

6d96e56ec22d1603ffed0f8d20f5b5090fb6ae4c7417126d09afc45447b34a64 /usr/lib/python3.5/csv.py

Below is error I get after I parse email;

2020-03-19 14:02:16,409 - ShadowServer-Parser - ERROR - Failed to parse line.

Traceback (most recent call last):

File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 951, in process

events = list(filter(bool, value))

File "/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py", line 111, in parse_line

' format or misconfiguration.'.format(shadowkey, self.feedname))

ValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.

2020-03-19 14:02:16,410 - ShadowServer-Parser - INFO - Dumping message to dump file.

2020-03-19 14:02:16,514 - ShadowServer-Parser - INFO - Dumping message to dump file.

2020-03-19 14:02:16,612 - ShadowServer-Parser - INFO - Sent 0 events and found 2 problem(s).

From Dump;

"source_queue": "ShadowServer-Parser-queue",

"traceback": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 951, in process\n events = list(filter(bool, value))\n File \"/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py\", line 111, in parse_line\n ' format or misconfiguration.'.format(shadowkey, self.feedname))\nValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.\n"

}

Thank you,

Patrick

On Thursday, March 19, 2020, 01:48:02 PM GMT+3, Sebastian Wagner <wagner@cert.at> wrote:

Hi,

On a fresh, virgin and up-to-date xenial installation I get the same results as Bernhard, which differ from yours. I suggest to re-install the affected python3 package on your system (or, maybe just the whole python3-stack):

sudo apt-get install --reinstall libpython3.5-stdlib

And then show the output of the commands again.

Sebsatian

On 3/19/20 10:48 AM, patric sungura wrote:

Dear team,

Find below inputs as requested;

##ls -l /usr/lib/python3.5/csv.py

-rw-r--r-- 1 root root 16138 Mar 19 11:28 /usr/lib/python3.5/csv.py

##sha256sum /usr/lib/python3.5/csv.py

fbed6ea919f70b4fd29608ab9620785304fc50a4b90cd2dd5d2363a0b81d9e81 /usr/lib/python3.5/csv.py

Find line aroud line#96

79 class DictReader:

80 def __init__(self, f, fieldnames=None, restkey=None, restval=None,

81 dialect="excel", *args, **kwds):

82 self._fieldnames = fieldnames # list of keys for the dict

83 self.restkey = restkey # key to catch long rows

84 self.restval = restval # default value for short rows

85 self.reader = reader(f, dialect, *args, **kwds)

86 self.dialect = dialect

87 self.line_num = 0

89 def __iter__(self):

90 return self

92 @property

93 def fieldnames(self):

94 if self._fieldnames is None:

95 try:

96 self._fieldnames = next(self.reader)

97 except StopIteration:

98 pass

99 self.line_num = self.reader.line_num

100 return self._fieldnames

101

102 @fieldnames.setter

103 def fieldnames(self, value):

104 self._fieldnames = value

I have two shadow server Parser and now have recorded different errors;

ShadowServer-Parser-2

2020-03-19 12:31:07,318 - ShadowServer-Parser-2 - ERROR - Bot has found a problem.

Traceback (most recent call last):

File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start

self.process()

File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process

for line in self.parse(report):

File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict

self.csv_fieldnames = csv_reader.fieldnames

File "/usr/lib/python3.5/csv.py", line 96, in fieldnames

self._fieldnames = next(self.reader)

NameError: name 'csv' is not defined

2020-03-19 12:31:07,319 - ShadowServer-Parser-2 - INFO - Dumping message to dump file.

ShadowServer-Parser

2020-03-19 12:37:58,573 - ShadowServer-Parser - ERROR - Bot has found a problem.

Traceback (most recent call last):

File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start

self.process()

File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process

for line in self.parse(report):

File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict

self.csv_fieldnames = csv_reader.fieldnames

File "/usr/lib/python3.5/csv.py", line 96, in fieldnames

self._fieldnames = next(self.reader)

_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

2020-03-19 12:37:58,574 - ShadowServer-Parser - INFO - Dumping message to dump file

Thank you very much,

Patrick

On Thursday, March 19, 2020, 11:56:49 AM GMT+3, Bernhard Reiter <bernhard@intevation.de> wrote:

Am Mittwoch, 18. März 2020, 17:02:13 CET schrieben Sie:
> libpython3.5-stdlib:amd64 3.5.2-2ubuntu0~16.04.9 amd64

This is the standard version, but still the backtrace looks strange,
especially the last bit:
File "/usr/lib/python3.5/csv.py", line 96, in fieldnames
self._fieldnames = next(self.reader, dialect=csv.excel_tab)
NameError: name 'csv' is not defined

The second part is strange as the original sources [1] don't have the second
parameter. So we should double check.
Can you open the file `/usr/lib/python3.5/csv.py` and check line 96?

Please send the whole function around the lines here.
(Can you also post your mail in text-only format? This is something I can and
many others can read much faster, and thus can answer you faster. :) Thanks! )

Can you also post the length, permissions and sha256 sum of the file?
Here for comparison the data for my file:
ls -l /usr/lib/python3.5/csv.py
-rw-r--r-- 1 root root 16128 Oct 8 16:38 /usr/lib/python3.5/csv.py
sha256sum /usr/lib/python3.5/csv.py
6d96e56ec22d1603ffed0f8d20f5b5090fb6ae4c7417126d09afc45447b34a64 /usr/lib/
python3.5/csv.py

Best Regards,
Bernhard
[1] https://github.com/python/cpython/blob/master/Lib/csv.py

--
www.intevation.de/~bernhard +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

--
Listen-Einstellungen:
https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users

-- 
// Sebastian Wagner <wagner@cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

--
Listen-Einstellungen:
https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users