[IntelMQ-dev] How does one avoid reprocessing mail attachments already processed?

[Sebastian Wagner]

On 8/4/21 9:11 AM, Mika Silander wrote:
>  I went through the mail attachment collector bot's logs again and
> noticed that I had misinterpreted the info, apologies for that. Based
> on the log it looks like every time the bot is restarted, it reads and
> forwards all the messages available in the mail box. Once that's done,
> it does not reprocess the messages anymore, only the new incoming
> ones, so no bug there. Thus, I should invent a way to remove the
> processed messages permanently between restarts.

There's already a suggestion to move mails to a different IMAP folder
after processing: https://github.com/certtools/intelmq/issues/1632

Pull requests welcome :)


>
> Br, Mika
>
> ------------------------------------------------------------------------
> *From: *"Sebastian Wagner" <wagner at cert.at>
> *To: *"intelmq-dev" <intelmq-dev at lists.cert.at>
> *Sent: *Wednesday, 4 August, 2021 09:43:27
> *Subject: *Re: [IntelMQ-dev] How does one avoid reprocessing mail
> attachments already processed?
>
> Hi,
>
> On 8/4/21 8:26 AM, Mika Silander wrote:
>
>      We are using the mail collector bot to read email attachments over IMAPS. It works fine but rereads all the messages in the message folder even if it had processed them already. Is there a way to configure the mail bot to remove the messages once it has processed them? I didn't find a hint concerning this in the bot's documentation nor the sources. What approaches do people use to avoid reprocessing messages?
>
> The bot should only look at /unread/ mails, plus the other criteria
> like subjects etc. After the successful processing, the mails are
> marked as read to avoid re-processing. If that does not work for you,
> that's a bug.
>
> best regards
> Sebastian
>
> Reading just unread mails:
> https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc419703f461/intelmq/bots/collectors/mail/_lib.py#L58
> Marking mail as seen after processing a mail:
> https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc419703f461/intelmq/bots/collectors/mail/_lib.py#L74
>
> -- 
> // Sebastian Wagner <wagner at cert.at> - T: +43 676 898 298 7201
> // CERT Austria - https://www.cert.at/
> // Eine Initiative der nic.at GmbH - https://www.nic.at/
> // Firmenbuchnummer 172568b, LG Salzburg
>
> _______________________________________________
> IntelMQ-dev mailing list
> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
> https://intelmq.readthedocs.io/

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210804/29d9d8bf/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210804/29d9d8bf/attachment-0001.sig>