Hi,
Seems I've gotten stuck on the same problem than before. However, this time around the order of installation was (on a Ubuntu 20.04 LTS/Apache 2):
apt-get install intelmq apt-get install intelmq-api apt-get install intelmq-manager
and the versions were:
intelmq 3.0.2-1 intelmq-api 3.0.1-1 intelmq-manager 3.0.1-1
I'm able to access the intelmq-manager start page, https://ourimq.domain/intelmq-manager/index.html but when opening the Configuration tab https://ourimq.domain/intelmq-manager/configs.html, expected frames appear but no bots are shown in the frame (normally) allowing editing of the bot network. Instead an endless loop starts with the error shown further below.
All hints for troubleshooting & debugging are welcome.
Br, Mika
P.S: No SELinux lurking in the shadows on the install host.
---
16:42:30 115× Error loading bot queues information: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> <hr> <address>Apache/2.4.41 (Ubuntu) Server at ourimq.domain Port 443</address> </body></html> Not Found
16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/positions .
16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/runtime .
16:40:37 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/bots
Hi all,
Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab https://ourimq/intelmq/v1/api/bots is accessed which in turn invokes (under the hood)
sudo -u www-data -u intelmq intelmqctl --type json list bots
This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives:
Traceback (most recent call last): File "/usr/bin/intelmqctl", line 11, in <module> load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main return x.run() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run print(json.dumps(results)) File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib/python3.8/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.8/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type Pattern is not JSON serializable
I assume this is the reason why the Configuration tab in my instance is dysfunctional. Any hints?
Br, Mika
----- Original Message ----- From: "Mika Silander" mika.silander@csc.fi To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Thursday, 7 October, 2021 17:03:23 Subject: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1
Hi,
Seems I've gotten stuck on the same problem than before. However, this time around the order of installation was (on a Ubuntu 20.04 LTS/Apache 2):
apt-get install intelmq apt-get install intelmq-api apt-get install intelmq-manager
and the versions were:
intelmq 3.0.2-1 intelmq-api 3.0.1-1 intelmq-manager 3.0.1-1
I'm able to access the intelmq-manager start page, https://ourimq.domain/intelmq-manager/index.html but when opening the Configuration tab https://ourimq.domain/intelmq-manager/configs.html, expected frames appear but no bots are shown in the frame (normally) allowing editing of the bot network. Instead an endless loop starts with the error shown further below.
All hints for troubleshooting & debugging are welcome.
Br, Mika
P.S: No SELinux lurking in the shadows on the install host.
---
16:42:30 115× Error loading bot queues information: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> <hr> <address>Apache/2.4.41 (Ubuntu) Server at ourimq.domain Port 443</address> </body></html> Not Found
16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/positions .
16:40:38 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/runtime .
16:40:37 Get an error error, Not Found when trying to obtain config file properly https://ourimq.domain/intelmq/v1/api/bots _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Hi,
On 11/2/21 1:38 PM, Mika Silander wrote:
Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab https://ourimq/intelmq/v1/api/bots is accessed which in turn invokes (under the hood)
sudo -u www-data -u intelmq intelmqctl --type json list bots
This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives:
Traceback (most recent call last): File "/usr/bin/intelmqctl", line 11, in <module> load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main return x.run() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run print(json.dumps(results)) File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib/python3.8/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.8/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type Pattern is not JSON serializable
I assume this is the reason why the Configuration tab in my instance is dysfunctional.
Yes.
Any hints?
Do you have a custom bot which has a parameter that is of type re.Pattern[0]? json.dumps fails serialzing this value. The solution is to use a string as parameter type, then you are even able to configure it :) You can compile the parameter's value at initialization.
Sebastian
[0]: https://docs.python.org/3/library/re.html#regular-expression-objects
Hi Sebastian, all,
Yes, I had a regexp in one of my bots although I don't understand why it should be prohibited to set these in a bot's parameter. It doesn't sound good to be limited to what can be serialized to JSON - most advanced Python features get excluded based on this criterion. Anyway, after moving the regexp into the init method, the URL
https://ourimq/intelmq/v1/api/bots
gets downloaded correctly. The only thing remaining is https://ourimq/intelmq/v1/api/positions accessed from within the Configuration tab:
Apache returns the response happily with HTTP 200 status, the JSON corresponding to /etc/intelmq/manager/positions.conf is returned but an error still appears in the browser tab:
"Failed to load config file properly"
For the positions URL the Javascript console of the browser states two TypeError(s):
edge_map[path] is not iterable
and
app.network is null
Also, an attempt to load https://ourimq/intelmq_manager/js/var.js shows up but this file doesn't exist under /usr/share/intelmq_manager/html/js. A left-over from earlier versions?
Hints welcome as before.
Br, Mika
----- Original Message ----- From: "Sebastian Wagner" wagner@cert.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Tuesday, 2 November, 2021 15:15:52 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1
Hi,
On 11/2/21 1:38 PM, Mika Silander wrote:
Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab https://ourimq/intelmq/v1/api/bots is accessed which in turn invokes (under the hood)
sudo -u www-data -u intelmq intelmqctl --type json list bots
This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives:
Traceback (most recent call last): File "/usr/bin/intelmqctl", line 11, in <module> load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main return x.run() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run print(json.dumps(results)) File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib/python3.8/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.8/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type Pattern is not JSON serializable
I assume this is the reason why the Configuration tab in my instance is dysfunctional.
Yes.
Any hints?
Do you have a custom bot which has a parameter that is of type re.Pattern[0]? json.dumps fails serialzing this value. The solution is to use a string as parameter type, then you are even able to configure it :) You can compile the parameter's value at initialization.
Sebastian
[0]: https://docs.python.org/3/library/re.html#regular-expression-objects
Hi,
I think there's a misunderstanding between parameters - which can be set by users - and bots' internals.
On 11/3/21 1:50 PM, Mika Silander wrote:
Yes, I had a regexp in one of my bots although I don't understand why it should be prohibited to set these in a bot's parameter. It doesn't sound good to be limited to what can be serialized to JSON
The requirement is that parameter's values can be /set/, in the runtime configuration or in the manager etc. It's impossible to express the instance of the re.Pattern class in runtime.yaml (and JSON).
- most advanced Python features get excluded based on this criterion.
IntelMQ's configuration is intentionally not Python code, but Text files.
It's perfectly fine to compile regular expressions.
If the expression comes from the user, i.e. it's a configuration parameter, you can do the compilation in init: https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07...
If it's not a parameter, but a constant value, the expression can be compiled earlier: https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07...
gets downloaded correctly. The only thing remaining is https://ourimq/intelmq/v1/api/positions accessed from within the Configuration tab:
Apache returns the response happily with HTTP 200 status, the JSON corresponding to /etc/intelmq/manager/positions.conf is returned but an error still appears in the browser tab:
"Failed to load config file properly"
For the positions URL the Javascript console of the browser states two TypeError(s):
edge_map[path] is not iterable
and
app.network is null
Sounds like a bug to me. I'd happy if someone with better knowledge of the manager's JS code can have a look.
Also, an attempt to load https://ourimq/intelmq_manager/js/var.js shows up but this file doesn't exist under /usr/share/intelmq_manager/html/js. A left-over from earlier versions?
No, that's intentional: https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configura...
best regards Sebastian
Hints welcome as before.
Br, Mika
----- Original Message ----- From: "Sebastian Wagner" wagner@cert.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Tuesday, 2 November, 2021 15:15:52 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1
Hi,
On 11/2/21 1:38 PM, Mika Silander wrote:
Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab https://ourimq/intelmq/v1/api/bots is accessed which in turn invokes (under the hood)
sudo -u www-data -u intelmq intelmqctl --type json list bots
This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives:
Traceback (most recent call last): File "/usr/bin/intelmqctl", line 11, in <module> load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main return x.run() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run print(json.dumps(results)) File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib/python3.8/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.8/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type Pattern is not JSON serializable
I assume this is the reason why the Configuration tab in my instance is dysfunctional.
Yes.
Any hints?
Do you have a custom bot which has a parameter that is of type re.Pattern[0]? json.dumps fails serialzing this value. The solution is to use a string as parameter type, then you are even able to configure it :) You can compile the parameter's value at initialization.
Sebastian
Hi,
I do understand the separation of textual definition of a bot's parameters vs. the internal representation. In my problem case the regexp was in a class variable. If you refer to that variable using self.varnamehere it will show up among the bot's other instance parameters. So in my case this setting got inadvertently included among the ones that are iterated over when generating the JSON file for intelmq-manager.
More on the latter: the docs on [ https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configura... | https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configura... ] speak about a "vars.js" file, however, according to the console "var.js" is searched for. Once this file is defined and contains the right ROOT variable setting, we are left with one TypeError:
edge_map[path] is not iterable
in the javascript console. Unfortunately, I am not sufficiently familiar with Javascript either to sort out what generates this error.
Br, Mika
From: "Sebastian Wagner" wagner@cert.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Wednesday, 3 November, 2021 15:05:36 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1
Hi,
I think there's a misunderstanding between parameters - which can be set by users - and bots' internals.
On 11/3/21 1:50 PM, Mika Silander wrote:
Yes, I had a regexp in one of my bots although I don't understand why it should be prohibited to set these in a bot's parameter. It doesn't sound good to be limited to what can be serialized to JSON
The requirement is that parameter's values can be set , in the runtime configuration or in the manager etc. It's impossible to express the instance of the re.Pattern class in runtime.yaml (and JSON).
BQ_BEGIN
- most advanced Python features get excluded based on this criterion.
BQ_END
IntelMQ's configuration is intentionally not Python code, but Text files.
It's perfectly fine to compile regular expressions.
If the expression comes from the user, i.e. it's a configuration parameter, you can do the compilation in init: [ https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07... | https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07... ]
If it's not a parameter, but a constant value, the expression can be compiled earlier: [ https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07... | https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07... ]
BQ_BEGIN
gets downloaded correctly. The only thing remaining is [ https://ourimq/intelmq/v1/api/positions | https://ourimq/intelmq/v1/api/positions ] accessed from within the Configuration tab:
Apache returns the response happily with HTTP 200 status, the JSON corresponding to /etc/intelmq/manager/positions.conf is returned but an error still appears in the browser tab:
"Failed to load config file properly"
For the positions URL the Javascript console of the browser states two TypeError(s):
edge_map[path] is not iterable
and
app.network is null
BQ_END Sounds like a bug to me. I'd happy if someone with better knowledge of the manager's JS code can have a look.
BQ_BEGIN
Also, an attempt to load [ https://ourimq/intelmq_manager/js/var.js | https://ourimq/intelmq_manager/js/var.js ] shows up but this file doesn't exist under /usr/share/intelmq_manager/html/js. A left-over from earlier versions?
BQ_END
No, that's intentional: [ https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configura... | https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configura... ]
best regards Sebastian
BQ_BEGIN
Hints welcome as before.
Br, Mika
----- Original Message ----- From: "Sebastian Wagner" [ mailto:wagner@cert.at | wagner@cert.at ] To: "Mika Silander" [ mailto:mika.silander@csc.fi | mika.silander@csc.fi ] , "intelmq-dev" [ mailto:intelmq-dev@lists.cert.at | intelmq-dev@lists.cert.at ] Sent: Tuesday, 2 November, 2021 15:15:52 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1
Hi,
On 11/2/21 1:38 PM, Mika Silander wrote:
BQ_BEGIN
Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab [ https://ourimq/intelmq/v1/api/bots | https://ourimq/intelmq/v1/api/bots ] is accessed which in turn invokes (under the hood)
sudo -u www-data -u intelmq intelmqctl --type json list bots
This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives:
Traceback (most recent call last): File "/usr/bin/intelmqctl", line 11, in <module> load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main return x.run() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run print(json.dumps(results)) File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib/python3.8/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.8/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type Pattern is not JSON serializable
I assume this is the reason why the Configuration tab in my instance is dysfunctional.
BQ_END
Yes.
BQ_BEGIN
Any hints?
BQ_END
Do you have a custom bot which has a parameter that is of type re.Pattern[0]? json.dumps fails serialzing this value. The solution is to use a string as parameter type, then you are even able to configure it :) You can compile the parameter's value at initialization.
Sebastian
[0]: [ https://docs.python.org/3/library/re.html#regular-expression-objects | https://docs.python.org/3/library/re.html#regular-expression-objects ]
BQ_END
Hi again,
Seems the culprit was found and it has nothing to do with javascript. Within the runtime.yaml we have queue definitions like
parameters: destination_queues: _default: [some-example-queue]
If one removes the colon and space between "_default" and the queue list, [ https://ourimq/intelmq/v1/api/positions | https://ourimq/intelmq/v1/api/positions ] coughs up the
"Failed to load config file properly"
error. We had this typo in one of our bot definitions and intelmq didn't complain about this typo in the configuration at any moment. Log level has been debug all the time afaik but no hints were written to /var/log/intelmq/intelmqctl.log for example. Thus, my vote for next feature implementation goes to stricter checks on the contents of runtime.yaml.
Br, Mika
From: "Mika Silander" mika.silander@csc.fi To: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Wednesday, 3 November, 2021 15:37:38 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1
Hi,
I do understand the separation of textual definition of a bot's parameters vs. the internal representation. In my problem case the regexp was in a class variable. If you refer to that variable using self.varnamehere it will show up among the bot's other instance parameters. So in my case this setting got inadvertently included among the ones that are iterated over when generating the JSON file for intelmq-manager.
More on the latter: the docs on [ https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configura... | https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configura... ] speak about a "vars.js" file, however, according to the console "var.js" is searched for. Once this file is defined and contains the right ROOT variable setting, we are left with one TypeError:
edge_map[path] is not iterable
in the javascript console. Unfortunately, I am not sufficiently familiar with Javascript either to sort out what generates this error.
Br, Mika
From: "Sebastian Wagner" wagner@cert.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Wednesday, 3 November, 2021 15:05:36 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1
Hi,
I think there's a misunderstanding between parameters - which can be set by users - and bots' internals.
On 11/3/21 1:50 PM, Mika Silander wrote:
Yes, I had a regexp in one of my bots although I don't understand why it should be prohibited to set these in a bot's parameter. It doesn't sound good to be limited to what can be serialized to JSON
The requirement is that parameter's values can be set , in the runtime configuration or in the manager etc. It's impossible to express the instance of the re.Pattern class in runtime.yaml (and JSON).
BQ_BEGIN
- most advanced Python features get excluded based on this criterion.
BQ_END
IntelMQ's configuration is intentionally not Python code, but Text files.
It's perfectly fine to compile regular expressions.
If the expression comes from the user, i.e. it's a configuration parameter, you can do the compilation in init: [ https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07... | https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07... ]
If it's not a parameter, but a constant value, the expression can be compiled earlier: [ https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07... | https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07... ]
BQ_BEGIN
gets downloaded correctly. The only thing remaining is [ https://ourimq/intelmq/v1/api/positions | https://ourimq/intelmq/v1/api/positions ] accessed from within the Configuration tab:
Apache returns the response happily with HTTP 200 status, the JSON corresponding to /etc/intelmq/manager/positions.conf is returned but an error still appears in the browser tab:
"Failed to load config file properly"
For the positions URL the Javascript console of the browser states two TypeError(s):
edge_map[path] is not iterable
and
app.network is null
BQ_END Sounds like a bug to me. I'd happy if someone with better knowledge of the manager's JS code can have a look.
BQ_BEGIN
Also, an attempt to load [ https://ourimq/intelmq_manager/js/var.js | https://ourimq/intelmq_manager/js/var.js ] shows up but this file doesn't exist under /usr/share/intelmq_manager/html/js. A left-over from earlier versions?
BQ_END
No, that's intentional: [ https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configura... | https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configura... ]
best regards Sebastian
BQ_BEGIN
Hints welcome as before.
Br, Mika
----- Original Message ----- From: "Sebastian Wagner" [ mailto:wagner@cert.at | wagner@cert.at ] To: "Mika Silander" [ mailto:mika.silander@csc.fi | mika.silander@csc.fi ] , "intelmq-dev" [ mailto:intelmq-dev@lists.cert.at | intelmq-dev@lists.cert.at ] Sent: Tuesday, 2 November, 2021 15:15:52 Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1
Hi,
On 11/2/21 1:38 PM, Mika Silander wrote:
BQ_BEGIN
Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab [ https://ourimq/intelmq/v1/api/bots | https://ourimq/intelmq/v1/api/bots ] is accessed which in turn invokes (under the hood)
sudo -u www-data -u intelmq intelmqctl --type json list bots
This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives:
Traceback (most recent call last): File "/usr/bin/intelmqctl", line 11, in <module> load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main return x.run() File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run print(json.dumps(results)) File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib/python3.8/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.8/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type Pattern is not JSON serializable
I assume this is the reason why the Configuration tab in my instance is dysfunctional.
BQ_END
Yes.
BQ_BEGIN
Any hints?
BQ_END
Do you have a custom bot which has a parameter that is of type re.Pattern[0]? json.dumps fails serialzing this value. The solution is to use a string as parameter type, then you are even able to configure it :) You can compile the parameter's value at initialization.
Sebastian
[0]: [ https://docs.python.org/3/library/re.html#regular-expression-objects | https://docs.python.org/3/library/re.html#regular-expression-objects ]
BQ_END