[Intelmq-users] IntelMQ

[Sebastian Wagner]

Hi,

From the provided logs I can see that the message has the following fields:

 * extra.email_from
 * extra.email_message_id
 * extra.email_subject
 * feed.accuracy
 * feed.name
 * feed.provider
 * raw, contains a zip file
 * time.observation

So we can follow from this: Wile the mails are correctly fetched and the
attachments are correctly identified, the attachments are not extracted
and are still in ZIP file format. It should be text/csv.

So I tried to reproduced this in a local setup and it turns out that the
handling of the (deprecated) parameter `attach_unzip` is currently
broken. And this Warning in your logs is directly related to it:

shadowserver-mail-Collector: The parameter 'attach_unzip' is deprecated
and will be removed in version 4.0. Use 'extract_files' instead.

The affected code is the part handling the value of that deprecated
parameter:

--- lib.py.old  2020-02-20 12:20:19.356103494 +0100
+++ lib.py      2020-02-20 12:20:26.360150384 +0100
@@ -18,7 +18,7 @@
             raise ValueError('Could not import imbox. Please install it.')
 
         if getattr(self.parameters, 'attach_unzip', None) and not
self.extract_files:
-            self.parameters.extract_files = True
+            self.extract_files = True
             self.logger.warning("The parameter 'attach_unzip' is
deprecated and will "
                                 "be removed in version 4.0. Use
'extract_files' instead.")

I will fix the bug in the IntelMQ code today, but for you I recommend to
set the parameter `extract_files` to `true` (just a rename).

For the output of intelmqctl check: You can follow it's output
(executing `intelmqctl upgrade-config`, and then once again)

best regards
Sebastian

On 2/20/20 11:42 AM, info wrote:
>
> Hi,
>
>  
>
> Yes I did restart the bots and also  loaded new emails for the bots to
> process.
>
> Below is the output after issuing the command intelmqctl check
>
> Reading configuration files.
>
> Checking defaults configuration.
>
> Checking runtime configuration.
>
> Checking runtime and pipeline configuration.
>
> Checking harmonization configuration.
>
> Checking for bots.
>
> No state file found. Please call 'intelmqctl upgrade-config'.
>
> No issues found.​
>
>  
>
> Regards,
>
> Bwogi Emmanuel
>
>  
>
>  
>
> *From:* Sebastian Wagner [mailto:wagner at cert.at]
> *Sent:* Thursday, 20 February 2020 13:30
> *To:* info <info at ug-cert.ug>; intelmq-users at lists.cert.at
> *Cc:* 'UCC CERT' <cert at ucc.co.ug>
> *Subject:* Re: [Intelmq-users] IntelMQ
>
>  
>
> Hi,
>
> On 2/20/20 11:22 AM, info wrote:
>
>     We have corrected the parameter as you advised in the previous
>     email however we are still getting the same error with the
>     shadowserver parser. Have attached the error in a notepad file.
>
> Did you reload or restart the bot afterwards? Did the collector
> re-fetch the mails and did the parser process these new messages?
>
> Sebastian
>
> -- 
> // Sebastian Wagner <wagner at cert.at> <mailto:wagner at cert.at> - T: +43 1 5056416 7201
> // CERT Austria - https://www.cert.at/
> // Eine Initiative der nic.at GmbH - https://www.nic.at/
> // Firmenbuchnummer 172568b, LG Salzburg

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20200220/82eeed9d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-users/attachments/20200220/82eeed9d/attachment.sig>