Dear Aaron, See below the pipeline.conf

{ "Mail-Attachment-Fetcher-Collector": { "destination-queues": [ "ShadowServer-Parser-queue" ] }, "Mail-URL-Fetcher-Collector": { "destination-queues": [ "ShadowServer-Parser-queue" ] }, "ShadowServer-Parser": { "source-queue": "ShadowServer-Parser-queue", "destination-queues": [ "deduplicator-expert-queue" ] }, "cymru-whois-expert": { "source-queue": "cymru-whois-expert-queue", "destination-queues": [ "file-output-queue" ] }, "deduplicator-expert": { "source-queue": "deduplicator-expert-queue", "destination-queues": [ "taxonomy-expert-queue" ] }, "feodo-tracker-browse-collector": { "destination-queues": [ "feodo-tracker-browse-parser-queue" ] }, "feodo-tracker-browse-parser": { "source-queue": "feodo-tracker-browse-parser-queue", "destination-queues": [ "deduplicator-expert-queue" ] }, "file-output": { "source-queue": "file-output-queue" }, "gethostbyname-1-expert": { "source-queue": "gethostbyname-1-expert-queue", "destination-queues": [

Thannks

Vincent M

-----Original Message----- From: L. Aaron Kaplan [mailto:kaplan@cert.at] Sent: Tuesday, February 18, 2020 8:11 PM To: UCC-CERT info@ug-cert.ug Cc: intelmq-users@lists.cert.at; UCC CERT cert@ucc.co.ug Subject: Re: [Intelmq-users] IntelMQ

Dear UCC-CERT, dear Vincent,

thanks :)

So, could you please also post the pipeline.conf file? I have the gut feeling that either the parser is not running (you can see this in the manager) or that it's not connected to the collector.

All the best, Aaron.

On 18.02.2020, at 18:03, UCC-CERT info@ug-cert.ug wrote:

Dear Experts, We currently have a mail box which contains only shadow server feeds

attachment files in a zipped form. The IntelMQ is able to read the emails but cannot extract and forward them to the shadow server parser.

We need your assistance .

See details below

Configuration From Runtime.conf

---

"Mail-Attachment-Fetcher-Collector": { "parameters": { "extract_files": "True", "attach_regex": "[A-Za-z:0-9\.\_ \[\]\-]", "folder": "INBOX", "mail_host": "imap.xxxx.xxx", "mail_password": "xxxxxxxxxx", "mail_ssl": true, "mail_user": "johndoe", "name": "Via IMAP", "provider": "ShadowServer", "rate_limit": 86400, "subject_regex": "[A-Za-z:0-9 \[\]\-]" }, "name": "Mail Attachment Fetcher", "group": "Collector", "module": "intelmq.bots.collectors.mail.collector_mail_attach", "description": "Monitor IMAP mailboxes and retrieve mail

attachments",

    "enabled": true,
    "run_mode": "continuous"

Below are the logs tail -n 1000 Mail-Attachment-Fetcher-Collector.log 2020-02-18 18:31:12,672 - Mail-Attachment-Fetcher-Collector - INFO - Email

report read.

2020-02-18 18:31:19,310 - Mail-Attachment-Fetcher-Collector - INFO - Email

report read.

2020-02-18 18:31:25,574 - Mail-Attachment-Fetcher-Collector - INFO - Email

report read.

2020-02-18 18:31:31,816 - Mail-Attachment-Fetcher-Collector - INFO - Email

report read.

Should you need any further information, please do not hesitate to contact

me.

Thanks

Regards,

Vincent M UG-CERT

-- Listen-Einstellungen: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users

-- // L. Aaron Kaplan kaplan@cert.at - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

Hi Sebastian,

We have corrected the parameter as you advised in the previous email however we are still getting the same error with the shadowserver parser. Have attached the error in a notepad file. Attached too is an image of our current architecture kindly advise if from the shadowserver parser we are supposed to link to the deduplicator-expert or connect to the Elasticsearch output directly.

Regards,

Bwogi Emmanuel

From: Intelmq-users [mailto:intelmq-users-bounces@lists.cert.at] On Behalf Of UCC-CERT Sent: Thursday, 20 February 2020 12:41 To: 'Sebastian Wagner' wagner@cert.at; intelmq-users@lists.cert.at Cc: 'UCC CERT' cert@ucc.co.ug Subject: Re: [Intelmq-users] IntelMQ

Thanks Sebastian,

We have edited the script accordingly and now testing. We shall get back to you on the progress

BR,

Vincent M

From: Sebastian Wagner [mailto:wagner@cert.at] Sent: Thursday, February 20, 2020 11:27 AM To: UCC-CERT <info@ug-cert.ug mailto:info@ug-cert.ug >; intelmq-users@lists.cert.at mailto:intelmq-users@lists.cert.at Cc: UCC CERT <cert@ucc.co.ug mailto:cert@ucc.co.ug > Subject: Re: [Intelmq-users] IntelMQ

Hi,

This line

"extract_files": "True",

should be:

        "extract_files": true,

best regards, Sebastian

Hi,

Yes I did restart the bots and also loaded new emails for the bots to process.

Below is the output after issuing the command intelmqctl check

Reading configuration files.

Checking defaults configuration.

Checking runtime configuration.

Checking runtime and pipeline configuration.

Checking harmonization configuration.

Checking for bots.

No state file found. Please call 'intelmqctl upgrade-config'.

No issues found.​

Regards,

Bwogi Emmanuel

From: Sebastian Wagner [mailto:wagner@cert.at] Sent: Thursday, 20 February 2020 13:30 To: info info@ug-cert.ug; intelmq-users@lists.cert.at Cc: 'UCC CERT' cert@ucc.co.ug Subject: Re: [Intelmq-users] IntelMQ

Hi,

On 2/20/20 11:22 AM, info wrote:

We have corrected the parameter as you advised in the previous email however we are still getting the same error with the shadowserver parser. Have attached the error in a notepad file.

Did you reload or restart the bot afterwards? Did the collector re-fetch the mails and did the parser process these new messages?

Sebastian

On 20/02/2020 12.34, Sebastian Wagner wrote:

I will fix the bug in the IntelMQ code today, but for you I recommend to set the parameter `extract_files` to `true` (just a rename).

Now fixed in the maintenance and develop branches and to be included in the next releases.

Related commits:

https://github.com/certtools/intelmq/commit/bf48f08c780d336527c9b85396774bc3... https://github.com/certtools/intelmq/commit/9f1fbe007f72e350ee9e6def1a35fd63... https://github.com/certtools/intelmq/commit/6d88c7e0d3ed849032b334a9989dd07f...

Sebastian