<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: #000000"><div>Hi again,</div><div><br data-mce-bogus="1"></div><div> Dang ... mistyping and the unfinished message went off. A new try:</div><div><br data-mce-bogus="1"></div><div> I managed to isolate and solve the below problem yesterday and share some details here so that you won't fall in the same trap. The root cause is I didn't fully understand how bot reloading works. Here's some pseudo-code explaining the problem in my expert bot implementation. First the initial implementation:<br></div><div><br style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">class MyExpertBot(Bot):</div><br style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> def init(self):</div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> self.dataconstruct = None</div><br style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> def process(self):</div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> if self.dataconstruct is None or is_too_old(self.dataconstruct):</div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> # initiate/update the construct here</div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> event = self.receive_message()</div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> # Access the data within self.dataconstruct here to process the event</div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> ...</div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><br data-mce-bogus="1"></div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> Then the fixed process method:</div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><br data-mce-bogus="1"></div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; font-size: 13.3333px; background-color: rgb(255, 255, 255);"> def process(self):</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; font-size: 13.3333px; background-color: rgb(255, 255, 255);"> <span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">event = self.receive_message()</span><div style="clear: both;" data-mce-style="clear: both;"> if self.dataconstruct is None or is_too_old(self.dataconstruct):<br></div></div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; font-size: 13.3333px; background-color: rgb(255, 255, 255);"> # initiate/update the construct here</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; font-size: 13.3333px; background-color: rgb(255, 255, 255);"> # Access the data within self.dataconstruct here to process the event</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 13.3333px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;" style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; font-size: 13.3333px; background-color: rgb(255, 255, 255);"> ...</div></div></div><div><br></div><div> Both versions work fine as long as you don't reload the bot. Reloading of the first version will however, reset self.dataconstruct to None while we are waiting for a new event. Once we receive the event, accesses to self.dataconstruct will of course fail. The latter version in turn, survives the reload since self.dataconstruct gets reinitiated _after_ the event is received. It took some time to realise the reload is specified in /etc/logrotate.d/intelmq (silly me).</div><div><br data-mce-bogus="1"></div><div> Finally, the usual request: correct me if my interpretation above is wrong.</div><div><br data-mce-bogus="1"></div><div>Best regards, Mika</div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>From: </b>"Mika Silander" <mika.silander@csc.fi><br><b>To: </b>"intelmq-dev" <intelmq-dev@lists.cert.at><br><b>Sent: </b>Thursday, 23 June, 2022 07:53:52<br><b>Subject: </b>Re: [IntelMQ-dev] Expected behaviour of intelmqctl reload ?<br></div><div><br></div><div data-marker="__QUOTED_TEXT__"><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:10pt;color:#000000"><div>Hi all,</div><br><div> I managed to isolate and solve the below problem yesterday and share some details here so that you won't fall in the same trap. The root cause is I didn't fully understand how bot reloading works. Here's some pseudo-code explaining the problem in my expert bot implementation. First the initial implementation:</div><br><div>class MyExpertBot(Bot):</div><br><div> def init(self):</div><div> self.dataconstruct = None</div><br><div> def process(self):</div><div> if self.dataconstruct is None:</div><br><br><hr id="zwchr"><div><b>From: </b>"Mika Silander" <mika.silander@csc.fi><br><b>To: </b>"intelmq-dev" <intelmq-dev@lists.cert.at><br><b>Sent: </b>Wednesday, 22 June, 2022 15:59:11<br><b>Subject: </b>Re: [IntelMQ-dev] Expected behaviour of intelmqctl reload ?<br></div><br><div><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:10pt;color:#000000"><div>Hi Sebastian,</div><br><div> Thanks for a prompt response. And yes, I was not very generous concerning details, so here's a hopefully better description:</div><br><div>I have an expert bot of my own that I start normally and then reload it with intelmqctl reload. When I now push an event to it for processing, a message appears in the log saying </div><br><div>"TypeError: 'NoneType' object is not subscriptable"</div><br><div> This is due to a variable (dict) becoming undefined in the reload. During a normal start this variable gets however, properly initiated once the first event comes in for</div><div>processing. The reason for its initiation in the process method arises from the need to regularly update it from a file that changes occasionally. In other words, my bot's init method sets this variable</div><div> to None and if it is None when reaching the bot's process method, it gets initialised there before any accesses to it.<span style="color:#000000;font-family:'arial' , 'helvetica' , sans-serif;font-size:13.3333px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:#ffffff;display:inline !important;float:none"> Later accesses/updates to this variable are successful unless I reload the bot.</span></div><br><div> In a reload, my bot's log starts with:</div><br><div><div><span style="font-size:13.3333px">INFO - Received SIGHUP, initializin</span><span style="font-size:13.3333px">g again later.</span></div><div><span style="font-size:13.3333px">INFO - Handling SIGHUP, initializing again now.</span></div><div><span style="font-size:13.3333px"><br></span></div><div><span style="font-size:13.3333px"> which leads me to assume that any possible delay needed ahead of a safe reload has passed and the reload can in fact take place as you describe below. There are indications that the Bot::__init__ (from intelmq/lib/bot.py) is run as</span></div><div><span style="font-size:13.3333px">the lines:</span></div><div><span style="font-size:13.3333px"><br></span></div><div><span style="font-size:13.3333px">INFO - Bot is starting.</span><br></div><div><span style="font-size:13.3333px">INFO - Bot initialization completed.</span><br></div><div><span style="font-size:13.3333px"><br></span></div><div><span style="font-size:13.3333px"> also show up in the bot's log. No log message from my bot's own init method gets written in between the above two, which leads me to assume that my bot's init method is not run, right? But shouldn't it be?</span></div><div><span style="font-size:13.3333px"><br></span></div><div><span style="font-size:13.3333px">Thanks again, Mika</span></div></div><br><hr id="zwchr"><div><b>From: </b>"Sebix" <sebix@sebix.at><br><b>To: </b>"Mika Silander" <mika.silander@csc.fi>, "intelmq-dev" <intelmq-dev@lists.cert.at><br><b>Sent: </b>Wednesday, 22 June, 2022 15:15:20<br><b>Subject: </b>Re: [IntelMQ-dev] Expected behaviour of intelmqctl reload ?<br></div><br><div><p>Hi Mika,<br>
</p>
<div class="moz-cite-prefix">On 6/22/22 1:58 PM, Mika Silander
wrote:<br>
</div>
<blockquote>
<pre class="moz-quote-pre"> Tracking down a rather elusive bug led me to suspect that the command
intelmqctl reload your_bot_name_here
does not necessarily rerun the bot's init method.</pre>
</blockquote>
<p>It does, but not directly.</p>
<p>reloading relies on sending a signal to the process. But signals
interrupt system calls, which is normally not an issue, except the
process is currently doing something on the network, more
specifically with the pipeline (sending or receiving data). Let's
assume that the bot sends an event to redis, get's interrupted.
What to do now? Run init, and then? we can't just resume the
previously running process() call. Therefore, the behaviour is not
completely straighforward, but not magic.<br>
The signal is caught (<span class="pl-en"><span class="pl-token">__handle_sighup_signal)</span></span>
and python is commanded to resume system calls instead of
interrupted (<span class="pl-s1"><span class="pl-token">signal</span></span>.<span class="pl-en"><span class="pl-token">siginterrupt</span></span>(<span class="pl-s1"><span class="pl-token">signal</span></span>.<span class="pl-v"><span class="pl-token">SIGHUP</span></span>,
<span class="pl-c1">False</span>) in __init__)<br>
Once there is time to handle the reload safely (before/after
process, during sleeps, when waiting for messages), the bot is
actually reloaded and __init__ called (<span class="pl-en"><span class="pl-token active">__handle_sighup).</span></span></p>
<blockquote>
<pre class="moz-quote-pre">At least I see an exception related to undefined variables (NoneType)</pre>
</blockquote>
In the code of the core or in the bot's code? If it's the bot's
code: What bot is it? In any case, the stacktrace helps in
explaining the issue, because unfortunately crystal ball broke last
night when my neighbors' dragon chased my little dinosaur....<br>
<blockquote>
<pre class="moz-quote-pre">when I reload the bot in question and let it chew a new event. Is this expected behaviour? Still, skimming through the code, it seems to me that the __init__ method in lib/bot.py gets invoked and should invoke the bot's init method as well.</pre>
</blockquote>
<p>Yes, it shoud/does:</p>
<p><a href="https://github.com/certtools/intelmq/blob/79cae294a74ab77f09b1f371a19f04ad65abab12/intelmq/lib/bot.py#L216" rel="nofollow noopener noreferrer nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://github.com/certtools/intelmq/blob/79cae294a74ab77f09b1f371a19f04ad65abab12/intelmq/lib/bot.py#L216</a>
</p>
<blockquote>
</blockquote>
<pre class="moz-signature">--
Institute for Common Good Technology
<a href="http://commongoodtechnology.org/" rel="nofollow noopener noreferrer nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">http://commongoodtechnology.org/</a>
ZVR 1510673578</pre><br></div></div><br>_______________________________________________<br>IntelMQ-dev mailing list<br>https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev<br>https://intelmq.readthedocs.io/</div></div><br></div></div></body></html>