Hi,

From the provided logs I can see that the message has the following fields:

 * extra.email_from
 * extra.email_message_id
 * extra.email_subject
 * feed.accuracy
 * feed.name
 * feed.provider
 * raw, contains a zip file
 * time.observation

So we can follow from this: Wile the mails are correctly fetched and the attachments are correctly identified, the attachments are not extracted and are still in ZIP file format. It should be text/csv.

So I tried to reproduced this in a local setup and it turns out that the handling of the (deprecated) parameter `attach_unzip` is currently broken. And this Warning in your logs is directly related to it:

shadowserver-mail-Collector: The parameter 'attach_unzip' is deprecated and will be removed in version 4.0. Use 'extract_files' instead.

The affected code is the part handling the value of that deprecated parameter:

--- lib.py.old  2020-02-20 12:20:19.356103494 +0100
+++ lib.py      2020-02-20 12:20:26.360150384 +0100
@@ -18,7 +18,7 @@
             raise ValueError('Could not import imbox. Please install it.')
 
         if getattr(self.parameters, 'attach_unzip', None) and not self.extract_files:
-            self.parameters.extract_files = True
+            self.extract_files = True
             self.logger.warning("The parameter 'attach_unzip' is deprecated and will "
                                 "be removed in version 4.0. Use 'extract_files' instead.")

I will fix the bug in the IntelMQ code today, but for you I recommend to set the parameter `extract_files` to `true` (just a rename).

For the output of intelmqctl check: You can follow it's output (executing `intelmqctl upgrade-config`, and then once again)

best regards
Sebastian

Hi,

 

Yes I did restart the bots and also  loaded new emails for the bots to process.

Below is the output after issuing the command intelmqctl check

Reading configuration files.

Checking defaults configuration.

Checking runtime configuration.

Checking runtime and pipeline configuration.

Checking harmonization configuration.

Checking for bots.

No state file found. Please call 'intelmqctl upgrade-config'.

No issues found.​

 

Regards,

Bwogi Emmanuel

 

 

From: Sebastian Wagner [mailto:wagner@cert.at]
Sent: Thursday, 20 February 2020 13:30
To: info <info@ug-cert.ug>; intelmq-users@lists.cert.at
Cc: 'UCC CERT' <cert@ucc.co.ug>
Subject: Re: [Intelmq-users] IntelMQ

 

Hi,

On 2/20/20 11:22 AM, info wrote:

We have corrected the parameter as you advised in the previous email however we are still getting the same error with the shadowserver parser. Have attached the error in a notepad file.

Did you reload or restart the bot afterwards? Did the collector re-fetch the mails and did the parser process these new messages?

Sebastian

-- 

// Sebastian Wagner <wagner@cert.at> - T: +43 1 5056416 7201

// CERT Austria - https://www.cert.at/

// Eine Initiative der nic.at GmbH - https://www.nic.at/

// Firmenbuchnummer 172568b, LG Salzburg

-- 
// Sebastian Wagner <wagner@cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg