[IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1

[Sebastian Wagner]

Hi,

I think there's a misunderstanding between parameters - which can be set
by users - and bots' internals.

On 11/3/21 1:50 PM, Mika Silander wrote:
>  Yes, I had a regexp in one of my bots although I don't understand why it should be prohibited to set these in a bot's parameter. It doesn't sound good to be limited to what can be serialized to JSON
The requirement is that parameter's values can be /set/, in the runtime
configuration or in the manager etc. It's impossible to express the
instance of the re.Pattern class in runtime.yaml (and JSON).
> - most advanced Python features get excluded based on this criterion.

IntelMQ's configuration is intentionally not Python code, but Text files.

It's perfectly fine to compile regular expressions.

If the expression comes from the user, i.e. it's a configuration
parameter, you can do the compilation in init:
https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/collectors/microsoft/collector_interflow.py#L79

If it's not a parameter, but a constant value, the expression can be
compiled earlier:
https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/parsers/cymru/parser_cap_program.py#L47

>  gets downloaded correctly. The only thing remaining is https://ourimq/intelmq/v1/api/positions accessed from within the Configuration tab:
>
>  Apache returns the response happily with HTTP 200 status, the JSON corresponding to /etc/intelmq/manager/positions.conf is returned but an error still
> appears in the browser tab:
>
> "Failed to load config file properly"
>
>  For the positions URL the Javascript console of the browser states two TypeError(s):
>
> edge_map[path] is not iterable
>
> and
>
> app.network is null
Sounds like a bug to me. I'd happy if someone with better knowledge of
the manager's JS code can have a look.
>  Also, an attempt to load https://ourimq/intelmq_manager/js/var.js shows up but this file doesn't exist under /usr/share/intelmq_manager/html/js. A left-over from earlier versions?

No, that's intentional:
https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration

best regards
Sebastian

>
>  Hints welcome as before.
>
> Br, Mika
>
>
> ----- Original Message -----
> From: "Sebastian Wagner" <wagner at cert.at>
> To: "Mika Silander" <mika.silander at csc.fi>, "intelmq-dev" <intelmq-dev at lists.cert.at>
> Sent: Tuesday, 2 November, 2021 15:15:52
> Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1
>
> Hi,
>
> On 11/2/21 1:38 PM, Mika Silander wrote:
>>  Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues
>> with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab https://ourimq/intelmq/v1/api/bots is accessed which in turn invokes (under the hood)
>>
>> sudo -u www-data -u intelmq intelmqctl --type json list bots
>>
>>  This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives:
>>
>> Traceback (most recent call last):
>>   File "/usr/bin/intelmqctl", line 11, in <module>
>>     load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')()
>>   File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main
>>     return x.run()
>>   File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run
>>     print(json.dumps(results))
>>   File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps
>>     return _default_encoder.encode(obj)
>>   File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
>>     chunks = self.iterencode(o, _one_shot=True)
>>   File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
>>     return _iterencode(o, 0)
>>   File "/usr/lib/python3.8/json/encoder.py", line 179, in default
>>     raise TypeError(f'Object of type {o.__class__.__name__} '
>> TypeError: Object of type Pattern is not JSON serializable
>>
>>
>>  I assume this is the reason why the Configuration tab in my instance is dysfunctional.
> Yes.
>> Any hints?
> Do you have a custom bot which has a parameter that is of type
> re.Pattern[0]? json.dumps fails serialzing this value. The solution is
> to use a string as parameter type, then you are even able to configure
> it :) You can compile the parameter's value at initialization.
>
> Sebastian
>
> [0]: https://docs.python.org/3/library/re.html#regular-expression-objects
>
-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 676 898 298 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20211103/6c25c5c4/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20211103/6c25c5c4/attachment.sig>