Hi all,
We are using the mail collector bot to read email attachments over IMAPS. It works fine but rereads all the messages in the message folder even if it had processed them already. Is there a way to configure the mail bot to remove the messages once it has processed them? I didn't find a hint concerning this in the bot's documentation nor the sources. What approaches do people use to avoid reprocessing messages?
Br, Mika
Hi,
On 8/4/21 8:26 AM, Mika Silander wrote:
We are using the mail collector bot to read email attachments over IMAPS. It works fine but rereads all the messages in the message folder even if it had processed them already. Is there a way to configure the mail bot to remove the messages once it has processed them? I didn't find a hint concerning this in the bot's documentation nor the sources. What approaches do people use to avoid reprocessing messages?
The bot should only look at /unread/ mails, plus the other criteria like subjects etc. After the successful processing, the mails are marked as read to avoid re-processing. If that does not work for you, that's a bug.
best regards Sebastian
Reading just unread mails: https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc4197... Marking mail as seen after processing a mail: https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc4197...
Hi Sebastian, all,
I went through the mail attachment collector bot's logs again and noticed that I had misinterpreted the info, apologies for that. Based on the log it looks like every time the bot is restarted, it reads and forwards all the messages available in the mail box. Once that's done, it does not reprocess the messages anymore, only the new incoming ones, so no bug there. Thus, I should invent a way to remove the processed messages permanently between restarts.
Br, Mika
From: "Sebastian Wagner" wagner@cert.at To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Wednesday, 4 August, 2021 09:43:27 Subject: Re: [IntelMQ-dev] How does one avoid reprocessing mail attachments already processed?
Hi, On 8/4/21 8:26 AM, Mika Silander wrote:
We are using the mail collector bot to read email attachments over IMAPS. It works fine but rereads all the messages in the message folder even if it had processed them already. Is there a way to configure the mail bot to remove the messages once it has processed them? I didn't find a hint concerning this in the bot's documentation nor the sources. What approaches do people use to avoid reprocessing messages?
The bot should only look at unread mails, plus the other criteria like subjects etc. After the successful processing, the mails are marked as read to avoid re-processing. If that does not work for you, that's a bug.
best regards Sebastian
Reading just unread mails: [ https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc4197... | https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc4197... ] Marking mail as seen after processing a mail: [ https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc4197... | https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc4197... ]
BQ_BEGIN
BQ_END
On 8/4/21 9:11 AM, Mika Silander wrote:
I went through the mail attachment collector bot's logs again and noticed that I had misinterpreted the info, apologies for that. Based on the log it looks like every time the bot is restarted, it reads and forwards all the messages available in the mail box. Once that's done, it does not reprocess the messages anymore, only the new incoming ones, so no bug there. Thus, I should invent a way to remove the processed messages permanently between restarts.
There's already a suggestion to move mails to a different IMAP folder after processing: https://github.com/certtools/intelmq/issues/1632
Pull requests welcome :)
Br, Mika
*From: *"Sebastian Wagner" wagner@cert.at *To: *"intelmq-dev" intelmq-dev@lists.cert.at *Sent: *Wednesday, 4 August, 2021 09:43:27 *Subject: *Re: [IntelMQ-dev] How does one avoid reprocessing mail attachments already processed?
Hi,
On 8/4/21 8:26 AM, Mika Silander wrote:
We are using the mail collector bot to read email attachments over IMAPS. It works fine but rereads all the messages in the message folder even if it had processed them already. Is there a way to configure the mail bot to remove the messages once it has processed them? I didn't find a hint concerning this in the bot's documentation nor the sources. What approaches do people use to avoid reprocessing messages?The bot should only look at /unread/ mails, plus the other criteria like subjects etc. After the successful processing, the mails are marked as read to avoid re-processing. If that does not work for you, that's a bug.
best regards Sebastian
Reading just unread mails: https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc4197... Marking mail as seen after processing a mail: https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc4197...
-- // Sebastian Wagner wagner@cert.at - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/