Hi,
At long last I updated my develop branch from the 2.3. days to 3.1.0 and noticed my tests fail due to the fact that bots don't have anymore a "parameters" attribute. At a closer look it seems all parameters from runtime.yaml are turned into direct attributes of self (correct?), e.g. if one defines a conf parameter "myparam", this shows up in the bot as self.myparam and not accessible as getattr(self.parameters, "myparam") as it did before.
After experimenting with my own bot tests I ended up in a situation where there's a potential attribute name clash. Assume a bot is tested like
self.input_message = some_event_here self.run_bot() self.assertSomethingHere()
In the above test, the bot gets initiated with a big number of default attributes, e.g. accuracy, group, enabled, logger, run_mode etc etc. Assume then that as a developer I want to use parameters with matching names for the needs of my own bot like
self.input_message = some_event_here self.run_bot(parameters={ 'group': 'Plumber', 'run_mode': 'disruptive' }) self.assertSomethingHere()
I can set those parameters and they happily override the defaults and I imagine this can mean trouble ahead. Is this intended behaviour and if yes, is there a way to prevent it? With the earlier self.parameters construct this did not happen and one could have e.g. a "group" attribute for the needs of intelmq's internal operation (self.group) separate from a bot developer's "group" attribute (that ended up under self.parameters) - no clashes despite equal names.
Thus, to avoid the above in intelmq 3.0.x this means every bot developer needs to check her own bot won't use any of the 'reserved' attributes before defining their own ones, right?
I might and I definitively hope I'm wrong here but please let me know if this is the case.
Br, Mika
Hey Mika,
Sorry for my late answer. Is the apache 2 correctly configured? Because IntelMQ tries to access the api endpoints, which doesnt respond with any data as it seems.
Have you checked https://intelmq.readthedocs.io/en/latest/user/intelmq-api.html already?
In addition please check the apache configuration file, if its configured correctly :)
Cheers,
Sebastian
On 10/8/21 1:17 PM, Mika Silander wrote:
Hi,
At long last I updated my develop branch from the 2.3. days to 3.1.0 and noticed my tests fail due to the fact that bots don't have anymore a "parameters" attribute. At a closer look it seems all parameters from runtime.yaml are turned into direct attributes of self (correct?), e.g. if one defines a conf parameter "myparam", this shows up in the bot as self.myparam and not accessible as getattr(self.parameters, "myparam") as it did before.
After experimenting with my own bot tests I ended up in a situation where there's a potential attribute name clash. Assume a bot is tested like
self.input_message = some_event_here self.run_bot() self.assertSomethingHere()
In the above test, the bot gets initiated with a big number of default attributes, e.g. accuracy, group, enabled, logger, run_mode etc etc. Assume then that as a developer I want to use parameters with matching names for the needs of my own bot like
self.input_message = some_event_here self.run_bot(parameters={ 'group': 'Plumber', 'run_mode': 'disruptive' }) self.assertSomethingHere()
I can set those parameters and they happily override the defaults and I imagine this can mean trouble ahead. Is this intended behaviour and if yes, is there a way to prevent it? With the earlier self.parameters construct this did not happen and one could have e.g. a "group" attribute for the needs of intelmq's internal operation (self.group) separate from a bot developer's "group" attribute (that ended up under self.parameters) - no clashes despite equal names.
Thus, to avoid the above in intelmq 3.0.x this means every bot developer needs to check her own bot won't use any of the 'reserved' attributes before defining their own ones, right?
I might and I definitively hope I'm wrong here but please let me know if this is the case.
Br, Mika
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Hey again,
The first answer was meant for "Help for fixing configuration of intelmq-manager 3.0.1-1", misclicked..
On 10/20/21 12:16 PM, Sebastian Waldbauer via IntelMQ-dev wrote:
Hey Mika,
Sorry for my late answer. Is the apache 2 correctly configured? Because IntelMQ tries to access the api endpoints, which doesnt respond with any data as it seems.
Have you checked https://intelmq.readthedocs.io/en/latest/user/intelmq-api.html already?
In addition please check the apache configuration file, if its configured correctly :)
Cheers,
Sebastian
On 10/8/21 1:17 PM, Mika Silander wrote:
Hi,
At long last I updated my develop branch from the 2.3. days to 3.1.0 and noticed my tests fail due to the fact that bots don't have anymore a "parameters" attribute. At a closer look it seems all parameters from runtime.yaml are turned into direct attributes of self (correct?), e.g. if one defines a conf parameter "myparam", this shows up in the bot as self.myparam and not accessible as getattr(self.parameters, "myparam") as it did before.
Yes, parameters are now attributes in the bots itself, which was a change as part of the IEP01 (https://lists.cert.at/pipermail/intelmq-users/2020-December/000185.html) proposed by Birger Schacht.
After experimenting with my own bot tests I ended up in a situation where there's a potential attribute name clash. Assume a bot is tested like
self.input_message = some_event_here self.run_bot() self.assertSomethingHere()
In the above test, the bot gets initiated with a big number of default attributes, e.g. accuracy, group, enabled, logger, run_mode etc etc. Assume then that as a developer I want to use parameters with matching names for the needs of my own bot like
self.input_message = some_event_here self.run_bot(parameters={ 'group': 'Plumber', 'run_mode': 'disruptive' }) self.assertSomethingHere()
I can set those parameters and they happily override the defaults and I imagine this can mean trouble ahead. Is this intended behaviour and if yes, is there a way to prevent it? With the earlier self.parameters construct this did not happen and one could have e.g. a "group" attribute for the needs of intelmq's internal operation (self.group) separate from a bot developer's "group" attribute (that ended up under self.parameters) - no clashes despite equal names.
Thus, to avoid the above in intelmq 3.0.x this means every bot developer needs to check her own bot won't use any of the 'reserved' attributes before defining their own ones, right?
Well, everything important should be changed in the lib/bot.py with double underscores. Everything else is changable in the bot itself + your own attributes.
I might and I definitively hope I'm wrong here but please let me know if this is the case.
Br, Mika
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Hi Sebastian,
Np, I've been busy with fixing problems in our own components after the upgrade to intelmq 3.0.2 :-). Thanks for pointing out the intelmq-api documentation has been updated, it has slipped my radar, I'll try with that and see what happens.
Btw, for contributions, what is the easiest way? A public repo on github and then pull requests? Are there licensing issues/incompatibilities? E.g. my employer wants all our published code to be released under MIT (Expat).
Br, Mika
----- Original Message ----- From: "intelmq-dev" intelmq-dev@lists.cert.at To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Wednesday, 20 October, 2021 13:46:36 Subject: Re: [IntelMQ-dev] Potential attribute name clashes in bot parameters in intelmq 3.1.x ?
Hey again,
The first answer was meant for "Help for fixing configuration of intelmq-manager 3.0.1-1", misclicked..
On 10/20/21 12:16 PM, Sebastian Waldbauer via IntelMQ-dev wrote:
Hey Mika,
Sorry for my late answer. Is the apache 2 correctly configured? Because IntelMQ tries to access the api endpoints, which doesnt respond with any data as it seems.
Have you checked https://intelmq.readthedocs.io/en/latest/user/intelmq-api.html already?
In addition please check the apache configuration file, if its configured correctly :)
Cheers,
Sebastian
On 10/8/21 1:17 PM, Mika Silander wrote:
Hi,
At long last I updated my develop branch from the 2.3. days to 3.1.0 and noticed my tests fail due to the fact that bots don't have anymore a "parameters" attribute. At a closer look it seems all parameters from runtime.yaml are turned into direct attributes of self (correct?), e.g. if one defines a conf parameter "myparam", this shows up in the bot as self.myparam and not accessible as getattr(self.parameters, "myparam") as it did before.
Yes, parameters are now attributes in the bots itself, which was a change as part of the IEP01 (https://lists.cert.at/pipermail/intelmq-users/2020-December/000185.html) proposed by Birger Schacht.
After experimenting with my own bot tests I ended up in a situation where there's a potential attribute name clash. Assume a bot is tested like
self.input_message = some_event_here self.run_bot() self.assertSomethingHere()
In the above test, the bot gets initiated with a big number of default attributes, e.g. accuracy, group, enabled, logger, run_mode etc etc. Assume then that as a developer I want to use parameters with matching names for the needs of my own bot like
self.input_message = some_event_here self.run_bot(parameters={ 'group': 'Plumber', 'run_mode': 'disruptive' }) self.assertSomethingHere()
I can set those parameters and they happily override the defaults and I imagine this can mean trouble ahead. Is this intended behaviour and if yes, is there a way to prevent it? With the earlier self.parameters construct this did not happen and one could have e.g. a "group" attribute for the needs of intelmq's internal operation (self.group) separate from a bot developer's "group" attribute (that ended up under self.parameters) - no clashes despite equal names.
Thus, to avoid the above in intelmq 3.0.x this means every bot developer needs to check her own bot won't use any of the 'reserved' attributes before defining their own ones, right?
Well, everything important should be changed in the lib/bot.py with double underscores. Everything else is changable in the bot itself + your own attributes.
I might and I definitively hope I'm wrong here but please let me know if this is the case.
Br, Mika
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Hey Mika,
On 10/20/21 1:13 PM, Mika Silander wrote:
Hi Sebastian,
Np, I've been busy with fixing problems in our own components after the upgrade to intelmq 3.0.2 :-). Thanks for pointing out the intelmq-api documentation has been updated, it has slipped my radar, I'll try with that and see what happens.
Sure, if anything is unclear or other problems occur, please ask :)
Btw, for contributions, what is the easiest way? A public repo on github and then pull requests? Are there licensing issues/incompatibilities? E.g. my employer wants all our published code to be released under MIT (Expat).
Currently intelmq is released under the AGPL. There might be some problems within the proprietary rights of use and exploitation, but I would be awesome if you could ask a lawyer to check on this. But to answer the question about the easiest way to contribute - Yes, you can fork intelmq & pull request your changes ofc. :)
Br, Mika
Cheers,
Sebastian
Hi Mika
Am Mittwoch 20 Oktober 2021 13:13:01 schrieb Mika Silander:
Btw, for contributions, what is the easiest way? A public repo on github and then pull requests?
for contributing to IntelMQ itself, yes, using github, issues and pull-requests probably works best.
(For independent code parts maybe more Free Software friendly hosting could also be considered, like gitlab or heptapod. E.g we've used https://gitlab.com/intevation/tuency/tuency )
Are there licensing issues/incompatibilities? E.g. my employer wants all our published code to be released under MIT (Expat).
Anything that is compatible with "AGPL-3.0-or-later" is fine. SPDX-License-Identifier: MIT is fine in this regard.
My additional recommendation is to use https://reuse.software/spec/ 3.0 (the current one) to indicate the license and who is the rights holder.
Best Regards, Bernhard
Dear Mika,
Thanks for your interest in contributing. After a multi-week absence of mine I'll try to catch up on all the events that happened, including the mailing list an contributions on GitHub.
I can only second what Bernhard said. AFAIK you can contribute new code licensed as MIT, as it is compatible with AGPL 3.0 and can be re-distributed as the latter, as long as you don't choose any weird MIT-variant ;) If you modify existing code, you have no other choice than accepting the license of course. But you are obliged to publish any modifications anyway if you do that, that's the soul of the AGPL.
On 10/21/21 12:17 PM, Bernhard Reiter wrote:
Are there licensing issues/incompatibilities? E.g. my employer wants all our published code to be released under MIT (Expat).
Anything that is compatible with "AGPL-3.0-or-later" is fine. SPDX-License-Identifier: MIT is fine in this regard.
My additional recommendation is to use https://reuse.software/spec/ 3.0 (the current one) to indicate the license and who is the rights holder.
Actually, all contributions must comply to the standard which is automatically checked by a GitHub Action that we have in place.
Sebastian
Hi,
On 10/8/21 1:17 PM, Mika Silander wrote:
At long last I updated my develop branch from the 2.3. days to 3.1.0 and noticed my tests fail due to the fact that bots don't have anymore a "parameters" attribute. At a closer look it seems all parameters from runtime.yaml are turned into direct attributes of self (correct?), e.g. if one defines a conf parameter "myparam", this shows up in the bot as self.myparam and not accessible as getattr(self.parameters, "myparam") as it did before.
After experimenting with my own bot tests I ended up in a situation where there's a potential attribute name clash. Assume a bot is tested like
self.input_message = some_event_here self.run_bot() self.assertSomethingHere()
In the above test, the bot gets initiated with a big number of default attributes, e.g. accuracy, group, enabled, logger, run_mode etc etc. Assume then that as a developer I want to use parameters with matching names for the needs of my own bot like
self.input_message = some_event_here self.run_bot(parameters={ 'group': 'Plumber', 'run_mode': 'disruptive' }) self.assertSomethingHere()
I can set those parameters and they happily override the defaults and I imagine this can mean trouble ahead. Is this intended behaviour and if yes, is there a way to prevent it? With the earlier self.parameters construct this did not happen and one could have e.g. a "group" attribute for the needs of intelmq's internal operation (self.group) separate from a bot developer's "group" attribute (that ended up under self.parameters) - no clashes despite equal names.
Thus, to avoid the above in intelmq 3.0.x this means every bot developer needs to check her own bot won't use any of the 'reserved' attributes before defining their own ones, right?
"group" is one of the values which is reserved/used my the Bot class itself: https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07...
Sorry for causing the confusion. We can only take care of the bots that are part of IntelMQ itself and make sure they are compatible, not for the bots unknown to us. Still, I hope that the simplifications and advantages of the new approach outweigh your disappointment.
Sebastian
Hi Sebastian, all,
Thanks for the clarification. Yes, I was admittedly surprised by this turn in development. It simplifies things yes, but especially to newcomers this might provide a confusing surprise. Pondering on solutions to this I thought one could isolate these intelmq infra-reserved configuration parameters and bot specific parameters into two separate sets so as to maintain name space hygiene. It means of course we'd also need to modify the structure and parsing of runtime.yaml to accomodate these two sets and maybe implement slightly old-fashioned getters and setters for manipulating these parameters but the end result would imho be clearer and no risk for clashes at a small additional cost of coding.
Br, Mika
----- Original Message ----- From: "Sebastian Wagner" wagner@cert.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Tuesday, 2 November, 2021 15:11:36 Subject: Re: [IntelMQ-dev] Potential attribute name clashes in bot parameters in intelmq 3.1.x ?
Hi,
On 10/8/21 1:17 PM, Mika Silander wrote:
At long last I updated my develop branch from the 2.3. days to 3.1.0 and noticed my tests fail due to the fact that bots don't have anymore a "parameters" attribute. At a closer look it seems all parameters from runtime.yaml are turned into direct attributes of self (correct?), e.g. if one defines a conf parameter "myparam", this shows up in the bot as self.myparam and not accessible as getattr(self.parameters, "myparam") as it did before.
After experimenting with my own bot tests I ended up in a situation where there's a potential attribute name clash. Assume a bot is tested like
self.input_message = some_event_here self.run_bot() self.assertSomethingHere()
In the above test, the bot gets initiated with a big number of default attributes, e.g. accuracy, group, enabled, logger, run_mode etc etc. Assume then that as a developer I want to use parameters with matching names for the needs of my own bot like
self.input_message = some_event_here self.run_bot(parameters={ 'group': 'Plumber', 'run_mode': 'disruptive' }) self.assertSomethingHere()
I can set those parameters and they happily override the defaults and I imagine this can mean trouble ahead. Is this intended behaviour and if yes, is there a way to prevent it? With the earlier self.parameters construct this did not happen and one could have e.g. a "group" attribute for the needs of intelmq's internal operation (self.group) separate from a bot developer's "group" attribute (that ended up under self.parameters) - no clashes despite equal names.
Thus, to avoid the above in intelmq 3.0.x this means every bot developer needs to check her own bot won't use any of the 'reserved' attributes before defining their own ones, right?
"group" is one of the values which is reserved/used my the Bot class itself: https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07...
Sorry for causing the confusion. We can only take care of the bots that are part of IntelMQ itself and make sure they are compatible, not for the bots unknown to us. Still, I hope that the simplifications and advantages of the new approach outweigh your disappointment.
Sebastian