From mika.silander at csc.fi Tue Nov 2 13:38:08 2021 From: mika.silander at csc.fi (Mika Silander) Date: Tue, 2 Nov 2021 14:38:08 +0200 (EET) Subject: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 In-Reply-To: <689431396.5543717.1633615403644.JavaMail.zimbra@csc.fi> References: <689431396.5543717.1633615403644.JavaMail.zimbra@csc.fi> Message-ID: <1869326763.5783047.1635856688385.JavaMail.zimbra@csc.fi> Hi all, Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab https://ourimq/intelmq/v1/api/bots is accessed which in turn invokes (under the hood) sudo -u www-data -u intelmq intelmqctl --type json list bots This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives: Traceback (most recent call last): File "/usr/bin/intelmqctl", line 11, in load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main return x.run() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run print(json.dumps(results)) File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib/python3.8/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.8/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type Pattern is not JSON serializable I assume this is the reason why the Configuration tab in my instance is dysfunctional. Any hints? Br, Mika ----- Original Message ----- From: "Mika Silander" To: "intelmq-dev" Sent: Thursday, 7 October, 2021 17:03:23 Subject: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 Hi, Seems I've gotten stuck on the same problem than before. However, this time around the order of installation was (on a Ubuntu 20.04 LTS/Apache 2): apt-get install intelmq apt-get install intelmq-api apt-get install intelmq-manager and the versions were: intelmq 3.0.2-1 intelmq-api 3.0.1-1 intelmq-manager 3.0.1-1 I'm able to access the intelmq-manager start page, https://ourimq.domain/intelmq-manager/index.html but when opening the Configuration tab https://ourimq.domain/intelmq-manager/configs.html, expected frames appear but no bots are shown in the frame (normally) allowing editing of the bot network. Instead an endless loop starts with the error shown further below. All hints for troubleshooting & debugging are welcome. Br, Mika P.S: No SELinux lurking in the shadows on the install host. --- 16:42:30 115× Error loading bot queues information: 404 Not Found

Not Found

The requested URL was not found on this server.

Apache/2.4.41 (Ubuntu) Server at ourimq.domain Port 443

Not Found 16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/positions . 16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/runtime . 16:40:37 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/bots _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/ From wagner at cert.at Tue Nov 2 14:05:46 2021 From: wagner at cert.at (Sebastian Wagner) Date: Tue, 2 Nov 2021 14:05:46 +0100 Subject: [IntelMQ-dev] Systemd startup scripts In-Reply-To: <215793762.3338673.1635422824563.JavaMail.zimbra@csc.fi> References: <1837096700.2669195.1635345833225.JavaMail.zimbra@csc.fi> <91980450.2862899.1635399424858.JavaMail.zimbra@csc.fi> <215793762.3338673.1635422824563.JavaMail.zimbra@csc.fi> Message-ID: <6e25ce79-b29f-bb48-9bda-63194ad35e00@cert.at> Hi, The systemd service file *template* itself works with 3.x, we have this in use in one of our set-ups. Sebastian On 10/28/21 2:07 PM, Mika Silander wrote: > Hi Navtej, > > No problem, no reason for you to be sorry. This is the way things evolve in open source. I guess me or some fellow of mine will work on this later unless someone else provides us a "free lunch" before that ;-) > > Br, Mika > > ----- Original Message ----- > From: "Navtej Singh" > To: "intelmq-dev" > Sent: Thursday, 28 October, 2021 12:00:58 > Subject: Re: [IntelMQ-dev] Systemd startup scripts > > I am sorry, I could not update those scripts. If you can make the > changes required, please do PR. > > On Thu, Oct 28, 2021 at 11:07 AM Mika Silander wrote: >> Hi Navtej, >> >> Thank you very much for the pointer. I took a look at the modules and to me it seems the scripts work for intelmq 2.3.x but perhaps not for intelmq 3.0.x. At least there were subtle indications this might be the case, e.g. the scripts assume pipeline.conf is available but afaik pipeline.conf has been phased out from intelmq 3.0.x, runtime.conf has been turned into runtime.yaml and covers the queue definitions that were earlier found in pipeline.conf. Maybe a revised version of the scripts is called for for intelmq 3.0.x (?) >> >> Br, Mika >> >> ----- Original Message ----- >> From: "Navtej Singh" >> To: "Mika Silander" >> Cc: "intelmq-dev" >> Sent: Thursday, 28 October, 2021 06:44:03 >> Subject: Re: [IntelMQ-dev] Systemd startup scripts >> >> Please take a look at >> https://github.com/certtools/intelmq/tree/develop/contrib/systemd . >> >> >> On Wed, Oct 27, 2021 at 8:16 PM Mika Silander wrote: >>> Hi, >>> >>> Afaik the bots can currently be managed with the command intelmqctl only. Has anyone implemented systemd startup scripts for intelmq? >>> >>> Br, Mika >>> _______________________________________________ >>> IntelMQ-dev mailing list >>> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev >>> https://intelmq.readthedocs.io/ > _______________________________________________ > IntelMQ-dev mailing list > https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev > https://intelmq.readthedocs.io/ > _______________________________________________ > IntelMQ-dev mailing list > https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev > https://intelmq.readthedocs.io/ -- // Sebastian Wagner - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From wagner at cert.at Tue Nov 2 14:11:36 2021 From: wagner at cert.at (Sebastian Wagner) Date: Tue, 2 Nov 2021 14:11:36 +0100 Subject: [IntelMQ-dev] Potential attribute name clashes in bot parameters in intelmq 3.1.x ? In-Reply-To: <626527344.5946678.1633691855581.JavaMail.zimbra@csc.fi> References: <626527344.5946678.1633691855581.JavaMail.zimbra@csc.fi> Message-ID: <14353116-30f8-5241-86ce-c83c6cea80af@cert.at> Hi, On 10/8/21 1:17 PM, Mika Silander wrote: > At long last I updated my develop branch from the 2.3. days to 3.1.0 and noticed my tests fail due to the fact that bots don't have anymore a "parameters" attribute. At a closer look it seems all parameters from runtime.yaml are turned into direct attributes of self (correct?), e.g. if one defines a conf parameter "myparam", this shows up in the bot as self.myparam and not accessible as getattr(self.parameters, "myparam") as it did before. > > After experimenting with my own bot tests I ended up in a situation where there's a potential attribute name clash. Assume a bot is tested like > > self.input_message = some_event_here > self.run_bot() > self.assertSomethingHere() > > In the above test, the bot gets initiated with a big number of default attributes, e.g. accuracy, group, enabled, logger, run_mode etc etc. Assume then that as a developer I want to use parameters with matching names for the needs of my own bot like > > self.input_message = some_event_here > self.run_bot(parameters={ > 'group': 'Plumber', > 'run_mode': 'disruptive' > }) > self.assertSomethingHere() > > I can set those parameters and they happily override the defaults and I imagine this can mean trouble ahead. Is this intended behaviour and if yes, is there a way to prevent it? With the earlier self.parameters construct this did not happen and one could have e.g. a "group" attribute for the needs of intelmq's internal operation (self.group) separate from a bot developer's "group" attribute (that ended up under self.parameters) - no clashes despite equal names. > > Thus, to avoid the above in intelmq 3.0.x this means every bot developer needs to check her own bot won't use any of the 'reserved' attributes before defining their own ones, right? "group" is one of the values which is reserved/used my the Bot class itself: https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/lib/bot.py#L62 Sorry for causing the confusion. We can only take care of the bots that are part of IntelMQ itself and make sure they are compatible, not for the bots unknown to us. Still, I hope that the simplifications and advantages of the new approach outweigh your disappointment. Sebastian -- // Sebastian Wagner - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From wagner at cert.at Tue Nov 2 14:15:52 2021 From: wagner at cert.at (Sebastian Wagner) Date: Tue, 2 Nov 2021 14:15:52 +0100 Subject: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 In-Reply-To: <1869326763.5783047.1635856688385.JavaMail.zimbra@csc.fi> References: <689431396.5543717.1633615403644.JavaMail.zimbra@csc.fi> <1869326763.5783047.1635856688385.JavaMail.zimbra@csc.fi> Message-ID: <4a6aa034-4cb8-440f-32b0-9eca42cf029e@cert.at> Hi, On 11/2/21 1:38 PM, Mika Silander wrote: > Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues > with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab https://ourimq/intelmq/v1/api/bots is accessed which in turn invokes (under the hood) > > sudo -u www-data -u intelmq intelmqctl --type json list bots > > This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives: > > Traceback (most recent call last): > File "/usr/bin/intelmqctl", line 11, in > load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() > File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main > return x.run() > File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run > print(json.dumps(results)) > File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps > return _default_encoder.encode(obj) > File "/usr/lib/python3.8/json/encoder.py", line 199, in encode > chunks = self.iterencode(o, _one_shot=True) > File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode > return _iterencode(o, 0) > File "/usr/lib/python3.8/json/encoder.py", line 179, in default > raise TypeError(f'Object of type {o.__class__.__name__} ' > TypeError: Object of type Pattern is not JSON serializable > > > I assume this is the reason why the Configuration tab in my instance is dysfunctional. Yes. > Any hints? Do you have a custom bot which has a parameter that is of type re.Pattern[0]? json.dumps fails serialzing this value. The solution is to use a string as parameter type, then you are even able to configure it :) You can compile the parameter's value at initialization. Sebastian [0]: https://docs.python.org/3/library/re.html#regular-expression-objects -- // Sebastian Wagner - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From mika.silander at csc.fi Tue Nov 2 14:30:19 2021 From: mika.silander at csc.fi (Mika Silander) Date: Tue, 2 Nov 2021 15:30:19 +0200 (EET) Subject: [IntelMQ-dev] Potential attribute name clashes in bot parameters in intelmq 3.1.x ? In-Reply-To: <14353116-30f8-5241-86ce-c83c6cea80af@cert.at> References: <626527344.5946678.1633691855581.JavaMail.zimbra@csc.fi> <14353116-30f8-5241-86ce-c83c6cea80af@cert.at> Message-ID: <284146325.5832757.1635859819421.JavaMail.zimbra@csc.fi> Hi Sebastian, all, Thanks for the clarification. Yes, I was admittedly surprised by this turn in development. It simplifies things yes, but especially to newcomers this might provide a confusing surprise. Pondering on solutions to this I thought one could isolate these intelmq infra-reserved configuration parameters and bot specific parameters into two separate sets so as to maintain name space hygiene. It means of course we'd also need to modify the structure and parsing of runtime.yaml to accomodate these two sets and maybe implement slightly old-fashioned getters and setters for manipulating these parameters but the end result would imho be clearer and no risk for clashes at a small additional cost of coding. Br, Mika ----- Original Message ----- From: "Sebastian Wagner" To: "Mika Silander" , "intelmq-dev" Sent: Tuesday, 2 November, 2021 15:11:36 Subject: Re: [IntelMQ-dev] Potential attribute name clashes in bot parameters in intelmq 3.1.x ? Hi, On 10/8/21 1:17 PM, Mika Silander wrote: > At long last I updated my develop branch from the 2.3. days to 3.1.0 and noticed my tests fail due to the fact that bots don't have anymore a "parameters" attribute. At a closer look it seems all parameters from runtime.yaml are turned into direct attributes of self (correct?), e.g. if one defines a conf parameter "myparam", this shows up in the bot as self.myparam and not accessible as getattr(self.parameters, "myparam") as it did before. > > After experimenting with my own bot tests I ended up in a situation where there's a potential attribute name clash. Assume a bot is tested like > > self.input_message = some_event_here > self.run_bot() > self.assertSomethingHere() > > In the above test, the bot gets initiated with a big number of default attributes, e.g. accuracy, group, enabled, logger, run_mode etc etc. Assume then that as a developer I want to use parameters with matching names for the needs of my own bot like > > self.input_message = some_event_here > self.run_bot(parameters={ > 'group': 'Plumber', > 'run_mode': 'disruptive' > }) > self.assertSomethingHere() > > I can set those parameters and they happily override the defaults and I imagine this can mean trouble ahead. Is this intended behaviour and if yes, is there a way to prevent it? With the earlier self.parameters construct this did not happen and one could have e.g. a "group" attribute for the needs of intelmq's internal operation (self.group) separate from a bot developer's "group" attribute (that ended up under self.parameters) - no clashes despite equal names. > > Thus, to avoid the above in intelmq 3.0.x this means every bot developer needs to check her own bot won't use any of the 'reserved' attributes before defining their own ones, right? "group" is one of the values which is reserved/used my the Bot class itself: https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/lib/bot.py#L62 Sorry for causing the confusion. We can only take care of the bots that are part of IntelMQ itself and make sure they are compatible, not for the bots unknown to us. Still, I hope that the simplifications and advantages of the new approach outweigh your disappointment. Sebastian -- // Sebastian Wagner - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg From mika.silander at csc.fi Wed Nov 3 13:50:34 2021 From: mika.silander at csc.fi (Mika Silander) Date: Wed, 3 Nov 2021 14:50:34 +0200 (EET) Subject: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 In-Reply-To: <4a6aa034-4cb8-440f-32b0-9eca42cf029e@cert.at> References: <689431396.5543717.1633615403644.JavaMail.zimbra@csc.fi> <1869326763.5783047.1635856688385.JavaMail.zimbra@csc.fi> <4a6aa034-4cb8-440f-32b0-9eca42cf029e@cert.at> Message-ID: <55715227.6390617.1635943834643.JavaMail.zimbra@csc.fi> Hi Sebastian, all, Yes, I had a regexp in one of my bots although I don't understand why it should be prohibited to set these in a bot's parameter. It doesn't sound good to be limited to what can be serialized to JSON - most advanced Python features get excluded based on this criterion. Anyway, after moving the regexp into the init method, the URL https://ourimq/intelmq/v1/api/bots gets downloaded correctly. The only thing remaining is https://ourimq/intelmq/v1/api/positions accessed from within the Configuration tab: Apache returns the response happily with HTTP 200 status, the JSON corresponding to /etc/intelmq/manager/positions.conf is returned but an error still appears in the browser tab: "Failed to load config file properly" For the positions URL the Javascript console of the browser states two TypeError(s): edge_map[path] is not iterable and app.network is null Also, an attempt to load https://ourimq/intelmq_manager/js/var.js shows up but this file doesn't exist under /usr/share/intelmq_manager/html/js. A left-over from earlier versions? Hints welcome as before. Br, Mika ----- Original Message ----- From: "Sebastian Wagner" To: "Mika Silander" , "intelmq-dev" Sent: Tuesday, 2 November, 2021 15:15:52 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 Hi, On 11/2/21 1:38 PM, Mika Silander wrote: > Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues > with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab https://ourimq/intelmq/v1/api/bots is accessed which in turn invokes (under the hood) > > sudo -u www-data -u intelmq intelmqctl --type json list bots > > This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives: > > Traceback (most recent call last): > File "/usr/bin/intelmqctl", line 11, in > load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() > File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main > return x.run() > File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run > print(json.dumps(results)) > File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps > return _default_encoder.encode(obj) > File "/usr/lib/python3.8/json/encoder.py", line 199, in encode > chunks = self.iterencode(o, _one_shot=True) > File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode > return _iterencode(o, 0) > File "/usr/lib/python3.8/json/encoder.py", line 179, in default > raise TypeError(f'Object of type {o.__class__.__name__} ' > TypeError: Object of type Pattern is not JSON serializable > > > I assume this is the reason why the Configuration tab in my instance is dysfunctional. Yes. > Any hints? Do you have a custom bot which has a parameter that is of type re.Pattern[0]? json.dumps fails serialzing this value. The solution is to use a string as parameter type, then you are even able to configure it :) You can compile the parameter's value at initialization. Sebastian [0]: https://docs.python.org/3/library/re.html#regular-expression-objects -- // Sebastian Wagner - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg From wagner at cert.at Wed Nov 3 14:05:36 2021 From: wagner at cert.at (Sebastian Wagner) Date: Wed, 3 Nov 2021 14:05:36 +0100 Subject: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 In-Reply-To: <55715227.6390617.1635943834643.JavaMail.zimbra@csc.fi> References: <689431396.5543717.1633615403644.JavaMail.zimbra@csc.fi> <1869326763.5783047.1635856688385.JavaMail.zimbra@csc.fi> <4a6aa034-4cb8-440f-32b0-9eca42cf029e@cert.at> <55715227.6390617.1635943834643.JavaMail.zimbra@csc.fi> Message-ID: <5cb0e1af-34f1-c224-123d-e9ff502b91be@cert.at> Hi, I think there's a misunderstanding between parameters - which can be set by users - and bots' internals. On 11/3/21 1:50 PM, Mika Silander wrote: > Yes, I had a regexp in one of my bots although I don't understand why it should be prohibited to set these in a bot's parameter. It doesn't sound good to be limited to what can be serialized to JSON The requirement is that parameter's values can be /set/, in the runtime configuration or in the manager etc. It's impossible to express the instance of the re.Pattern class in runtime.yaml (and JSON). > - most advanced Python features get excluded based on this criterion. IntelMQ's configuration is intentionally not Python code, but Text files. It's perfectly fine to compile regular expressions. If the expression comes from the user, i.e. it's a configuration parameter, you can do the compilation in init: https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/collectors/microsoft/collector_interflow.py#L79 If it's not a parameter, but a constant value, the expression can be compiled earlier: https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/parsers/cymru/parser_cap_program.py#L47 > gets downloaded correctly. The only thing remaining is https://ourimq/intelmq/v1/api/positions accessed from within the Configuration tab: > > Apache returns the response happily with HTTP 200 status, the JSON corresponding to /etc/intelmq/manager/positions.conf is returned but an error still > appears in the browser tab: > > "Failed to load config file properly" > > For the positions URL the Javascript console of the browser states two TypeError(s): > > edge_map[path] is not iterable > > and > > app.network is null Sounds like a bug to me. I'd happy if someone with better knowledge of the manager's JS code can have a look. > Also, an attempt to load https://ourimq/intelmq_manager/js/var.js shows up but this file doesn't exist under /usr/share/intelmq_manager/html/js. A left-over from earlier versions? No, that's intentional: https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration best regards Sebastian > > Hints welcome as before. > > Br, Mika > > > ----- Original Message ----- > From: "Sebastian Wagner" > To: "Mika Silander" , "intelmq-dev" > Sent: Tuesday, 2 November, 2021 15:15:52 > Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 > > Hi, > > On 11/2/21 1:38 PM, Mika Silander wrote: >> Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues >> with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab https://ourimq/intelmq/v1/api/bots is accessed which in turn invokes (under the hood) >> >> sudo -u www-data -u intelmq intelmqctl --type json list bots >> >> This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives: >> >> Traceback (most recent call last): >> File "/usr/bin/intelmqctl", line 11, in >> load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() >> File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main >> return x.run() >> File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run >> print(json.dumps(results)) >> File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps >> return _default_encoder.encode(obj) >> File "/usr/lib/python3.8/json/encoder.py", line 199, in encode >> chunks = self.iterencode(o, _one_shot=True) >> File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode >> return _iterencode(o, 0) >> File "/usr/lib/python3.8/json/encoder.py", line 179, in default >> raise TypeError(f'Object of type {o.__class__.__name__} ' >> TypeError: Object of type Pattern is not JSON serializable >> >> >> I assume this is the reason why the Configuration tab in my instance is dysfunctional. > Yes. >> Any hints? > Do you have a custom bot which has a parameter that is of type > re.Pattern[0]? json.dumps fails serialzing this value. The solution is > to use a string as parameter type, then you are even able to configure > it :) You can compile the parameter's value at initialization. > > Sebastian > > [0]: https://docs.python.org/3/library/re.html#regular-expression-objects > -- // Sebastian Wagner - T: +43 676 898 298 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From mika.silander at csc.fi Wed Nov 3 14:37:38 2021 From: mika.silander at csc.fi (Mika Silander) Date: Wed, 3 Nov 2021 15:37:38 +0200 (EET) Subject: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 In-Reply-To: <5cb0e1af-34f1-c224-123d-e9ff502b91be@cert.at> References: <689431396.5543717.1633615403644.JavaMail.zimbra@csc.fi> <1869326763.5783047.1635856688385.JavaMail.zimbra@csc.fi> <4a6aa034-4cb8-440f-32b0-9eca42cf029e@cert.at> <55715227.6390617.1635943834643.JavaMail.zimbra@csc.fi> <5cb0e1af-34f1-c224-123d-e9ff502b91be@cert.at> Message-ID: <466891770.6422841.1635946658851.JavaMail.zimbra@csc.fi> Hi, I do understand the separation of textual definition of a bot's parameters vs. the internal representation. In my problem case the regexp was in a class variable. If you refer to that variable using self.varnamehere it will show up among the bot's other instance parameters. So in my case this setting got inadvertently included among the ones that are iterated over when generating the JSON file for intelmq-manager. More on the latter: the docs on [ https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration | https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration ] speak about a "vars.js" file, however, according to the console "var.js" is searched for. Once this file is defined and contains the right ROOT variable setting, we are left with one TypeError: edge_map[path] is not iterable in the javascript console. Unfortunately, I am not sufficiently familiar with Javascript either to sort out what generates this error. Br, Mika From: "Sebastian Wagner" To: "Mika Silander" , "intelmq-dev" Sent: Wednesday, 3 November, 2021 15:05:36 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 Hi, I think there's a misunderstanding between parameters - which can be set by users - and bots' internals. On 11/3/21 1:50 PM, Mika Silander wrote: Yes, I had a regexp in one of my bots although I don't understand why it should be prohibited to set these in a bot's parameter. It doesn't sound good to be limited to what can be serialized to JSON The requirement is that parameter's values can be set , in the runtime configuration or in the manager etc. It's impossible to express the instance of the re.Pattern class in runtime.yaml (and JSON). BQ_BEGIN - most advanced Python features get excluded based on this criterion. BQ_END IntelMQ's configuration is intentionally not Python code, but Text files. It's perfectly fine to compile regular expressions. If the expression comes from the user, i.e. it's a configuration parameter, you can do the compilation in init: [ https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/collectors/microsoft/collector_interflow.py#L79 | https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/collectors/microsoft/collector_interflow.py#L79 ] If it's not a parameter, but a constant value, the expression can be compiled earlier: [ https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/parsers/cymru/parser_cap_program.py#L47 | https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/parsers/cymru/parser_cap_program.py#L47 ] BQ_BEGIN gets downloaded correctly. The only thing remaining is [ https://ourimq/intelmq/v1/api/positions | https://ourimq/intelmq/v1/api/positions ] accessed from within the Configuration tab: Apache returns the response happily with HTTP 200 status, the JSON corresponding to /etc/intelmq/manager/positions.conf is returned but an error still appears in the browser tab: "Failed to load config file properly" For the positions URL the Javascript console of the browser states two TypeError(s): edge_map[path] is not iterable and app.network is null BQ_END Sounds like a bug to me. I'd happy if someone with better knowledge of the manager's JS code can have a look. BQ_BEGIN Also, an attempt to load [ https://ourimq/intelmq_manager/js/var.js | https://ourimq/intelmq_manager/js/var.js ] shows up but this file doesn't exist under /usr/share/intelmq_manager/html/js. A left-over from earlier versions? BQ_END No, that's intentional: [ https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration | https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration ] best regards Sebastian BQ_BEGIN Hints welcome as before. Br, Mika ----- Original Message ----- From: "Sebastian Wagner" [ mailto:wagner at cert.at | ] To: "Mika Silander" [ mailto:mika.silander at csc.fi | ] , "intelmq-dev" [ mailto:intelmq-dev at lists.cert.at | ] Sent: Tuesday, 2 November, 2021 15:15:52 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 Hi, On 11/2/21 1:38 PM, Mika Silander wrote: BQ_BEGIN Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab [ https://ourimq/intelmq/v1/api/bots | https://ourimq/intelmq/v1/api/bots ] is accessed which in turn invokes (under the hood) sudo -u www-data -u intelmq intelmqctl --type json list bots This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives: Traceback (most recent call last): File "/usr/bin/intelmqctl", line 11, in load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main return x.run() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run print(json.dumps(results)) File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib/python3.8/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.8/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type Pattern is not JSON serializable I assume this is the reason why the Configuration tab in my instance is dysfunctional. BQ_END Yes. BQ_BEGIN Any hints? BQ_END Do you have a custom bot which has a parameter that is of type re.Pattern[0]? json.dumps fails serialzing this value. The solution is to use a string as parameter type, then you are even able to configure it :) You can compile the parameter's value at initialization. Sebastian [0]: [ https://docs.python.org/3/library/re.html#regular-expression-objects | https://docs.python.org/3/library/re.html#regular-expression-objects ] BQ_END -- // Sebastian Wagner [ mailto:wagner at cert.at | ] - T: +43 676 898 298 7201 // CERT Austria - [ https://www.cert.at/ | https://www.cert.at/ ] // Eine Initiative der nic.at GmbH - [ https://www.nic.at/ | https://www.nic.at/ ] // Firmenbuchnummer 172568b, LG Salzburg -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika.silander at csc.fi Wed Nov 3 15:42:01 2021 From: mika.silander at csc.fi (Mika Silander) Date: Wed, 3 Nov 2021 16:42:01 +0200 (EET) Subject: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 In-Reply-To: <466891770.6422841.1635946658851.JavaMail.zimbra@csc.fi> References: <689431396.5543717.1633615403644.JavaMail.zimbra@csc.fi> <1869326763.5783047.1635856688385.JavaMail.zimbra@csc.fi> <4a6aa034-4cb8-440f-32b0-9eca42cf029e@cert.at> <55715227.6390617.1635943834643.JavaMail.zimbra@csc.fi> <5cb0e1af-34f1-c224-123d-e9ff502b91be@cert.at> <466891770.6422841.1635946658851.JavaMail.zimbra@csc.fi> Message-ID: <1917692519.6465243.1635950521552.JavaMail.zimbra@csc.fi> Hi again, Seems the culprit was found and it has nothing to do with javascript. Within the runtime.yaml we have queue definitions like parameters: destination_queues: _default: [some-example-queue] If one removes the colon and space between "_default" and the queue list, [ https://ourimq/intelmq/v1/api/positions | https://ourimq/intelmq/v1/api/positions ] coughs up the "Failed to load config file properly" error. We had this typo in one of our bot definitions and intelmq didn't complain about this typo in the configuration at any moment. Log level has been debug all the time afaik but no hints were written to /var/log/intelmq/intelmqctl.log for example. Thus, my vote for next feature implementation goes to stricter checks on the contents of runtime.yaml. Br, Mika From: "Mika Silander" To: "intelmq-dev" Sent: Wednesday, 3 November, 2021 15:37:38 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 Hi, I do understand the separation of textual definition of a bot's parameters vs. the internal representation. In my problem case the regexp was in a class variable. If you refer to that variable using self.varnamehere it will show up among the bot's other instance parameters. So in my case this setting got inadvertently included among the ones that are iterated over when generating the JSON file for intelmq-manager. More on the latter: the docs on [ https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration | https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration ] speak about a "vars.js" file, however, according to the console "var.js" is searched for. Once this file is defined and contains the right ROOT variable setting, we are left with one TypeError: edge_map[path] is not iterable in the javascript console. Unfortunately, I am not sufficiently familiar with Javascript either to sort out what generates this error. Br, Mika From: "Sebastian Wagner" To: "Mika Silander" , "intelmq-dev" Sent: Wednesday, 3 November, 2021 15:05:36 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 Hi, I think there's a misunderstanding between parameters - which can be set by users - and bots' internals. On 11/3/21 1:50 PM, Mika Silander wrote: Yes, I had a regexp in one of my bots although I don't understand why it should be prohibited to set these in a bot's parameter. It doesn't sound good to be limited to what can be serialized to JSON The requirement is that parameter's values can be set , in the runtime configuration or in the manager etc. It's impossible to express the instance of the re.Pattern class in runtime.yaml (and JSON). BQ_BEGIN - most advanced Python features get excluded based on this criterion. BQ_END IntelMQ's configuration is intentionally not Python code, but Text files. It's perfectly fine to compile regular expressions. If the expression comes from the user, i.e. it's a configuration parameter, you can do the compilation in init: [ https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/collectors/microsoft/collector_interflow.py#L79 | https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/collectors/microsoft/collector_interflow.py#L79 ] If it's not a parameter, but a constant value, the expression can be compiled earlier: [ https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/parsers/cymru/parser_cap_program.py#L47 | https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/parsers/cymru/parser_cap_program.py#L47 ] BQ_BEGIN gets downloaded correctly. The only thing remaining is [ https://ourimq/intelmq/v1/api/positions | https://ourimq/intelmq/v1/api/positions ] accessed from within the Configuration tab: Apache returns the response happily with HTTP 200 status, the JSON corresponding to /etc/intelmq/manager/positions.conf is returned but an error still appears in the browser tab: "Failed to load config file properly" For the positions URL the Javascript console of the browser states two TypeError(s): edge_map[path] is not iterable and app.network is null BQ_END Sounds like a bug to me. I'd happy if someone with better knowledge of the manager's JS code can have a look. BQ_BEGIN Also, an attempt to load [ https://ourimq/intelmq_manager/js/var.js | https://ourimq/intelmq_manager/js/var.js ] shows up but this file doesn't exist under /usr/share/intelmq_manager/html/js. A left-over from earlier versions? BQ_END No, that's intentional: [ https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration | https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration ] best regards Sebastian BQ_BEGIN Hints welcome as before. Br, Mika ----- Original Message ----- From: "Sebastian Wagner" [ mailto:wagner at cert.at | ] To: "Mika Silander" [ mailto:mika.silander at csc.fi | ] , "intelmq-dev" [ mailto:intelmq-dev at lists.cert.at | ] Sent: Tuesday, 2 November, 2021 15:15:52 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1 Hi, On 11/2/21 1:38 PM, Mika Silander wrote: BQ_BEGIN Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab [ https://ourimq/intelmq/v1/api/bots | https://ourimq/intelmq/v1/api/bots ] is accessed which in turn invokes (under the hood) sudo -u www-data -u intelmq intelmqctl --type json list bots This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives: Traceback (most recent call last): File "/usr/bin/intelmqctl", line 11, in load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main return x.run() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run print(json.dumps(results)) File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib/python3.8/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.8/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type Pattern is not JSON serializable I assume this is the reason why the Configuration tab in my instance is dysfunctional. BQ_END Yes. BQ_BEGIN Any hints? BQ_END Do you have a custom bot which has a parameter that is of type re.Pattern[0]? json.dumps fails serialzing this value. The solution is to use a string as parameter type, then you are even able to configure it :) You can compile the parameter's value at initialization. Sebastian [0]: [ https://docs.python.org/3/library/re.html#regular-expression-objects | https://docs.python.org/3/library/re.html#regular-expression-objects ] BQ_END -- // Sebastian Wagner [ mailto:wagner at cert.at | ] - T: +43 676 898 298 7201 // CERT Austria - [ https://www.cert.at/ | https://www.cert.at/ ] // Eine Initiative der nic.at GmbH - [ https://www.nic.at/ | https://www.nic.at/ ] // Firmenbuchnummer 172568b, LG Salzburg _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mika.silander at csc.fi Wed Nov 10 12:14:53 2021 From: mika.silander at csc.fi (Mika Silander) Date: Wed, 10 Nov 2021 13:14:53 +0200 (EET) Subject: [IntelMQ-dev] Redis log file gets wrong owner & group(?) Message-ID: <1702294405.2159555.1636542893012.JavaMail.zimbra@csc.fi> Hi all, Occasionally we see the /var/log/redis/redis-server.log file getting intelmq as its owner and group. This makes redis output to the log file fail. Once the owner is reset to redis and group to adm (on Ubuntu 20.04 LTS) and running systemctl restart redis, redis works fine. I've tried to debug the reason for this change in ownership in logrotate confs, intelmqctl sources etc but so far no luck. Hints as to the reason or how to troubleshoot are again welcome. Br, Mika P.S: We haven't configured redis to listen to unix sockets in /etc/redis/redis.conf so we assume https://intelmq.readthedocs.io/en/maintenance/user/FAQ.html#id4 does not apply to this case. From sebix at sebix.at Sun Nov 14 20:34:55 2021 From: sebix at sebix.at (Sebix) Date: Sun, 14 Nov 2021 20:34:55 +0100 Subject: [IntelMQ-dev] Redis log file gets wrong owner & group(?) In-Reply-To: <1702294405.2159555.1636542893012.JavaMail.zimbra@csc.fi> References: <1702294405.2159555.1636542893012.JavaMail.zimbra@csc.fi> Message-ID: Dear Mika, On 11/10/21 12:14 PM, Mika Silander wrote: > Hi all, > > Occasionally we see the /var/log/redis/redis-server.log file getting intelmq as its owner and group. Everytime when logrotate kicks in? > This makes redis output to the log file fail. Once the owner is reset to redis and group to adm (on Ubuntu 20.04 LTS) and running systemctl restart redis, redis works fine. I've tried to debug the reason for this change in ownership in logrotate confs, intelmqctl sources etc but so far no luck. Hints as to the reason or how to troubleshoot are again welcome. I discovered, debugged and fixed this issue a few weeks ago when I was still working at CERT.at: https://github.com/certtools/intelmq/commit/5b3c68b571b04ae816f3e8314a2d97b78dae76aa The problem is that the option `create 644 intelmq intelmq` in intelmq's logrotate config does not only apply to the intelmq files, but to all files managed by logrotate globally. Not only redis is affected, but lots more. You can find all affected files with sudo find /var/log/ -user intelmq ! -path \*intelmq\* I hope that helps Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From mika.silander at csc.fi Mon Nov 15 08:41:38 2021 From: mika.silander at csc.fi (Mika Silander) Date: Mon, 15 Nov 2021 09:41:38 +0200 (EET) Subject: [IntelMQ-dev] Redis log file gets wrong owner & group(?) In-Reply-To: References: <1702294405.2159555.1636542893012.JavaMail.zimbra@csc.fi> Message-ID: <120011932.3913234.1636962098353.JavaMail.zimbra@csc.fi> Hi Sebastian, Thanks for the confirmation of a bug and its fix. I had already configured auditd to narrow down this problem but now it's not needed anymore. Br, Mika ----- Original Message ----- From: "Sebix" To: "Mika Silander" , "intelmq-dev" Sent: Sunday, 14 November, 2021 21:34:55 Subject: Re: [IntelMQ-dev] Redis log file gets wrong owner & group(?) Dear Mika, On 11/10/21 12:14 PM, Mika Silander wrote: > Hi all, > > Occasionally we see the /var/log/redis/redis-server.log file getting intelmq as its owner and group. Everytime when logrotate kicks in? > This makes redis output to the log file fail. Once the owner is reset to redis and group to adm (on Ubuntu 20.04 LTS) and running systemctl restart redis, redis works fine. I've tried to debug the reason for this change in ownership in logrotate confs, intelmqctl sources etc but so far no luck. Hints as to the reason or how to troubleshoot are again welcome. I discovered, debugged and fixed this issue a few weeks ago when I was still working at CERT.at: https://github.com/certtools/intelmq/commit/5b3c68b571b04ae816f3e8314a2d97b78dae76aa The problem is that the option `create 644 intelmq intelmq` in intelmq's logrotate config does not only apply to the intelmq files, but to all files managed by logrotate globally. Not only redis is affected, but lots more. You can find all affected files with sudo find /var/log/ -user intelmq ! -path \*intelmq\* I hope that helps Sebastian From sebix at sebix.at Wed Nov 17 20:23:58 2021 From: sebix at sebix.at (Sebix) Date: Wed, 17 Nov 2021 20:23:58 +0100 Subject: [IntelMQ-dev] Contribute by Reviews and Maintenance Message-ID: Dear community, In my last mail on this list I already hided the message that I left CERT.at a short while ago and I was already asked what that means for the IntelMQ project. First of all: Don't panic! IntelMQ is and has always been a community project, with contributors from multiple teams, countries and continents[0]. In every release announcement I say thank you to the community for all the contributions and I really mean it. IntelMQ would never be that feature-rich without your involvement. All of us are part of this project, and joining our forces makes us strong. IntelMQ made various big steps in the last 7 years. It all started with Tomás Lima and Aaron Kaplan. The architecture was so robust, that it didn't change since then. I joined one year later and since then I almost memorized every line of code. In the last years I lost that ability as more and more code was coming for contributors, who are also maintaining code. And that's the essence: With IntelMQ growing, it's important to share the maintenance work. The IntelMQ project is looking for your contributions, including: - fixing open bugs[0] - reviewing open pull requests[1] - discussing and deciding when to make releases - testing before we do releases - documentation - and much more ;) IntelMQ is ready for the next evolutional leap forward and you are part of it! best regards Sebastian [0]: Maybe there will be a Moon- or Mars CERT in the not-so-distant future? ;) [1]: https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3Abug [2]: https://github.com/certtools/intelmq/pulls -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: