[IntelMQ-dev] How does one avoid reprocessing mail attachments already processed?

[Mika Silander]

Hi Sebastian, all, 

I went through the mail attachment collector bot's logs again and noticed that I had misinterpreted the info, apologies for that. Based on the log it looks like every time the bot is restarted, it reads and forwards all the messages available in the mail box. Once that's done, it does not reprocess the messages anymore, only the new incoming ones, so no bug there. Thus, I should invent a way to remove the processed messages permanently between restarts. 

Br, Mika 


From: "Sebastian Wagner" <wagner at cert.at> 
To: "intelmq-dev" <intelmq-dev at lists.cert.at> 
Sent: Wednesday, 4 August, 2021 09:43:27 
Subject: Re: [IntelMQ-dev] How does one avoid reprocessing mail attachments already processed? 



Hi, 
On 8/4/21 8:26 AM, Mika Silander wrote: 



We are using the mail collector bot to read email attachments over IMAPS. It works fine but rereads all the messages in the message folder even if it had processed them already. Is there a way to configure the mail bot to remove the messages once it has processed them? I didn't find a hint concerning this in the bot's documentation nor the sources. What approaches do people use to avoid reprocessing messages? 




The bot should only look at unread mails, plus the other criteria like subjects etc. After the successful processing, the mails are marked as read to avoid re-processing. If that does not work for you, that's a bug. 

best regards 
Sebastian 


Reading just unread mails: [ https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc419703f461/intelmq/bots/collectors/mail/_lib.py#L58 | https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc419703f461/intelmq/bots/collectors/mail/_lib.py#L58 ] 
Marking mail as seen after processing a mail: [ https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc419703f461/intelmq/bots/collectors/mail/_lib.py#L74 | https://github.com/certtools/intelmq/blob/72cee0ee5eeede17700604b138f2bc419703f461/intelmq/bots/collectors/mail/_lib.py#L74 ] 

BQ_BEGIN

BQ_END

-- 
// Sebastian Wagner [ mailto:wagner at cert.at | <wagner at cert.at> ] - T: +43 676 898 298 7201
// CERT Austria - [ https://www.cert.at/ | https://www.cert.at/ ] // Eine Initiative der nic.at GmbH - [ https://www.nic.at/ | https://www.nic.at/ ] // Firmenbuchnummer 172568b, LG Salzburg 

_______________________________________________ 
IntelMQ-dev mailing list 
https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev 
https://intelmq.readthedocs.io/ 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210804/c6e027c2/attachment.htm>