<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,</p>
<p>From the provided logs I can see that the message has the
following fields:<br>
</p>
<p> * extra.email_from<br>
* extra.email_message_id<br>
* extra.email_subject<br>
* feed.accuracy<br>
* feed.name<br>
* feed.provider<br>
* raw, contains a zip file<br>
* time.observation</p>
<p>So we can follow from this: Wile the mails are correctly fetched
and the attachments are correctly identified, the attachments are
not extracted and are still in ZIP file format. It should be
text/csv.</p>
<p>So I tried to reproduced this in a local setup and it turns out
that the handling of the (deprecated) parameter `attach_unzip` is
currently broken. And this Warning in your logs is directly
related to it:</p>
<p>shadowserver-mail-Collector: The parameter 'attach_unzip' is
deprecated and will be removed in version 4.0. Use 'extract_files'
instead.</p>
<p>The affected code is the part handling the value of that
deprecated parameter:<br>
</p>
<p>--- lib.py.old 2020-02-20 12:20:19.356103494 +0100<br>
+++ lib.py 2020-02-20 12:20:26.360150384 +0100<br>
@@ -18,7 +18,7 @@<br>
raise ValueError('Could not import imbox. Please
install it.')<br>
<br>
if getattr(self.parameters, 'attach_unzip', None) and not
self.extract_files:<br>
- self.parameters.extract_files = True<br>
+ self.extract_files = True<br>
self.logger.warning("The parameter 'attach_unzip' is
deprecated and will "<br>
"be removed in version 4.0. Use
'extract_files' instead.")<br>
</p>
<p>I will fix the bug in the IntelMQ code today, but for you I
recommend to set the parameter `extract_files` to `true` (just a
rename).</p>
<p>For the output of intelmqctl check: You can follow it's output
(executing `intelmqctl upgrade-config`, and then once again)</p>
<p>best regards<br>
Sebastian<br>
</p>
<div class="moz-cite-prefix">On 2/20/20 11:42 AM, info wrote:<br>
</div>
<blockquote type="cite"
cite="mid:016f01d5e7da$70adc1e0$520945a0$@ug-cert.ug">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle27
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.s1
{mso-style-name:s1;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Yes I did
restart the bots and also loaded new emails for the bots to
process.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Below is the
output after issuing the command intelmqctl check<o:p></o:p></span></p>
<p class="p1" style="margin:0cm;margin-bottom:.0001pt"><span
class="s1"><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black">Reading
configuration files.</span></span><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0cm;margin-bottom:.0001pt;orphans:
2;text-align:start;widows: 2;-webkit-text-stroke-width:
0px;text-decoration-style: initial;text-decoration-color:
initial;word-spacing:0px"><span class="s1"><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black">Checking
defaults configuration.</span></span><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0cm;margin-bottom:.0001pt;orphans:
2;text-align:start;widows: 2;-webkit-text-stroke-width:
0px;text-decoration-style: initial;text-decoration-color:
initial;word-spacing:0px"><span class="s1"><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black">Checking
runtime configuration.</span></span><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0cm;margin-bottom:.0001pt;orphans:
2;text-align:start;widows: 2;-webkit-text-stroke-width:
0px;text-decoration-style: initial;text-decoration-color:
initial;word-spacing:0px"><span class="s1"><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black">Checking
runtime and pipeline configuration.</span></span><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0cm;margin-bottom:.0001pt;orphans:
2;text-align:start;widows: 2;-webkit-text-stroke-width:
0px;text-decoration-style: initial;text-decoration-color:
initial;word-spacing:0px"><span class="s1"><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black">Checking
harmonization configuration.</span></span><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black"><o:p></o:p></span></p>
<p class="p1" style="margin:0cm;margin-bottom:.0001pt;orphans:
2;text-align:start;widows: 2;-webkit-text-stroke-width:
0px;text-decoration-style: initial;text-decoration-color:
initial;word-spacing:0px"><span class="s1"><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black">Checking
for bots.</span></span><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black"><o:p></o:p></span></p>
<p class="p2" style="margin:0cm;margin-bottom:.0001pt;orphans:
2;text-align:start;widows: 2;-webkit-text-stroke-width:
0px;text-decoration-style: initial;text-decoration-color:
initial;word-spacing:0px"><span class="s1"><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:#CA3323">No
state file found. Please call 'intelmqctl upgrade-config'.</span></span><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:#CA3323"><o:p></o:p></span></p>
<p class="p1" style="margin:0cm;margin-bottom:.0001pt;orphans:
2;text-align:start;widows: 2;-webkit-text-stroke-width:
0px;text-decoration-style: initial;text-decoration-color:
initial;word-spacing:0px"><span class="s1"><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black">No
issues found.<o:p></o:p></span></span></p>
<p class="p1" style="margin:0cm;margin-bottom:.0001pt"><span
class="s1"><span
style="font-size:8.5pt;font-family:"Menlo",serif;color:black"><o:p> </o:p></span></span></p>
<p class="p1" style="margin:0cm;margin-bottom:.0001pt"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="p1" style="margin:0cm;margin-bottom:.0001pt"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Bwogi
Emmanuel<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Sebastian Wagner
[<a class="moz-txt-link-freetext" href="mailto:wagner@cert.at">mailto:wagner@cert.at</a>] <br>
<b>Sent:</b> Thursday, 20 February 2020 13:30<br>
<b>To:</b> info <a class="moz-txt-link-rfc2396E" href="mailto:info@ug-cert.ug"><info@ug-cert.ug></a>;
<a class="moz-txt-link-abbreviated" href="mailto:intelmq-users@lists.cert.at">intelmq-users@lists.cert.at</a><br>
<b>Cc:</b> 'UCC CERT' <a class="moz-txt-link-rfc2396E" href="mailto:cert@ucc.co.ug"><cert@ucc.co.ug></a><br>
<b>Subject:</b> Re: [Intelmq-users] IntelMQ<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Hi,<o:p></o:p></p>
<div>
<p class="MsoNormal">On 2/20/20 11:22 AM, info wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif;color:#1F497D">We have corrected the
parameter as you advised in the previous email however we
are still getting the same error with the shadowserver
parser. Have attached the error in a notepad file.</span><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p></o:p></span></p>
</blockquote>
<p>Did you reload or restart the bot afterwards? Did the
collector re-fetch the mails and did the parser process these
new messages?<o:p></o:p></p>
<p>Sebastian<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>// Sebastian Wagner <a href="mailto:wagner@cert.at" moz-do-not-send="true"><wagner@cert.at></a> - T: +43 1 5056416 7201<o:p></o:p></pre>
<pre>// CERT Austria - <a href="https://www.cert.at/" moz-do-not-send="true">https://www.cert.at/</a><o:p></o:p></pre>
<pre>// Eine Initiative der nic.at GmbH - <a href="https://www.nic.at/" moz-do-not-send="true">https://www.nic.at/</a><o:p></o:p></pre>
<pre>// Firmenbuchnummer 172568b, LG Salzburg<o:p></o:p></pre>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
// Sebastian Wagner <a class="moz-txt-link-rfc2396E" href="mailto:wagner@cert.at"><wagner@cert.at></a> - T: +43 1 5056416 7201
// CERT Austria - <a class="moz-txt-link-freetext" href="https://www.cert.at/">https://www.cert.at/</a>
// Eine Initiative der nic.at GmbH - <a class="moz-txt-link-freetext" href="https://www.nic.at/">https://www.nic.at/</a>
// Firmenbuchnummer 172568b, LG Salzburg</pre>
</body>
</html>