<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Hi,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">I think there's a misunderstanding
between parameters - which can be set by users - and bots'
internals.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 11/3/21 1:50 PM, Mika Silander
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:55715227.6390617.1635943834643.JavaMail.zimbra@csc.fi">
<pre class="moz-quote-pre" wrap=""> Yes, I had a regexp in one of my bots although I don't understand why it should be prohibited to set these in a bot's parameter. It doesn't sound good to be limited to what can be serialized to JSON</pre>
</blockquote>
The requirement is that parameter's values can be <i>set</i>, in
the runtime configuration or in the manager etc. It's impossible to
express the instance of the re.Pattern class in runtime.yaml (and
JSON).<br>
<blockquote type="cite"
cite="mid:55715227.6390617.1635943834643.JavaMail.zimbra@csc.fi">
<pre class="moz-quote-pre" wrap="">- most advanced Python features get excluded based on this criterion.</pre>
</blockquote>
<p>IntelMQ's configuration is intentionally not Python code, but
Text files.</p>
<p>It's perfectly fine to compile regular expressions.</p>
<p>If the expression comes from the user, i.e. it's a configuration
parameter, you can do the compilation in init:
<a class="moz-txt-link-freetext" href="https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/collectors/microsoft/collector_interflow.py#L79">https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/collectors/microsoft/collector_interflow.py#L79</a><br>
</p>
<p>If it's not a parameter, but a constant value, the expression can
be compiled earlier:
<a class="moz-txt-link-freetext" href="https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/parsers/cymru/parser_cap_program.py#L47">https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/parsers/cymru/parser_cap_program.py#L47</a><br>
</p>
<blockquote type="cite"
cite="mid:55715227.6390617.1635943834643.JavaMail.zimbra@csc.fi">
<pre class="moz-quote-pre" wrap=""> gets downloaded correctly. The only thing remaining is <a class="moz-txt-link-freetext" href="https://ourimq/intelmq/v1/api/positions">https://ourimq/intelmq/v1/api/positions</a> accessed from within the Configuration tab:
Apache returns the response happily with HTTP 200 status, the JSON corresponding to /etc/intelmq/manager/positions.conf is returned but an error still
appears in the browser tab:
"Failed to load config file properly"
For the positions URL the Javascript console of the browser states two TypeError(s):
edge_map[path] is not iterable
and
app.network is null</pre>
</blockquote>
Sounds like a bug to me. I'd happy if someone with better knowledge
of the manager's JS code can have a look.<br>
<blockquote type="cite"
cite="mid:55715227.6390617.1635943834643.JavaMail.zimbra@csc.fi">
<pre class="moz-quote-pre" wrap=""> Also, an attempt to load <a class="moz-txt-link-freetext" href="https://ourimq/intelmq_manager/js/var.js">https://ourimq/intelmq_manager/js/var.js</a> shows up but this file doesn't exist under /usr/share/intelmq_manager/html/js. A left-over from earlier versions?</pre>
</blockquote>
<p>No, that's intentional:
<a class="moz-txt-link-freetext" href="https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration">https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration</a></p>
<p>best regards<br>
Sebastian<br>
</p>
<blockquote type="cite"
cite="mid:55715227.6390617.1635943834643.JavaMail.zimbra@csc.fi">
<pre class="moz-quote-pre" wrap="">
Hints welcome as before.
Br, Mika
----- Original Message -----
From: "Sebastian Wagner" <a class="moz-txt-link-rfc2396E" href="mailto:wagner@cert.at"><wagner@cert.at></a>
To: "Mika Silander" <a class="moz-txt-link-rfc2396E" href="mailto:mika.silander@csc.fi"><mika.silander@csc.fi></a>, "intelmq-dev" <a class="moz-txt-link-rfc2396E" href="mailto:intelmq-dev@lists.cert.at"><intelmq-dev@lists.cert.at></a>
Sent: Tuesday, 2 November, 2021 15:15:52
Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1
Hi,
On 11/2/21 1:38 PM, Mika Silander wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap=""> Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues
with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab <a class="moz-txt-link-freetext" href="https://ourimq/intelmq/v1/api/bots">https://ourimq/intelmq/v1/api/bots</a> is accessed which in turn invokes (under the hood)
sudo -u www-data -u intelmq intelmqctl --type json list bots
This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives:
Traceback (most recent call last):
File "/usr/bin/intelmqctl", line 11, in <module>
load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')()
File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main
return x.run()
File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run
print(json.dumps(results))
File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Pattern is not JSON serializable
I assume this is the reason why the Configuration tab in my instance is dysfunctional.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">Yes.
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Any hints?
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
Do you have a custom bot which has a parameter that is of type
re.Pattern[0]? json.dumps fails serialzing this value. The solution is
to use a string as parameter type, then you are even able to configure
it :) You can compile the parameter's value at initialization.
Sebastian
[0]: <a class="moz-txt-link-freetext" href="https://docs.python.org/3/library/re.html#regular-expression-objects">https://docs.python.org/3/library/re.html#regular-expression-objects</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
// Sebastian Wagner <a class="moz-txt-link-rfc2396E" href="mailto:wagner@cert.at"><wagner@cert.at></a> - T: +43 676 898 298 7201
// CERT Austria - <a class="moz-txt-link-freetext" href="https://www.cert.at/">https://www.cert.at/</a>
// Eine Initiative der nic.at GmbH - <a class="moz-txt-link-freetext" href="https://www.nic.at/">https://www.nic.at/</a>
// Firmenbuchnummer 172568b, LG Salzburg</pre>
</body>
</html>