<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: #000000"><div>Hi again,</div><div><br data-mce-bogus="1"></div><div> Seems the culprit was found and it has nothing to do with javascript. Within the runtime.yaml we have queue definitions like</div><div><br data-mce-bogus="1"></div><div>parameters:</div><div><div><span style="font-size: 13.3333px;"> destination_queues:</span></div><div><span style="font-size: 13.3333px;"> _default: [some-example-queue]</span></div><div><span style="font-size: 13.3333px;"><br data-mce-bogus="1"></span></div><div><span style="font-size: 13.3333px;"> If one removes the colon and space between "_default" and the queue list, <a href="https://ourimq/intelmq/v1/api/positions">https://ourimq/intelmq/v1/api/positions</a> coughs up the</span></div><div><span style="font-size: 13.3333px;"><br data-mce-bogus="1"></span></div><div><span style="font-size: 13.3333px;">"Failed to load config file properly"</span></div><div><span style="font-size: 13.3333px;"><br data-mce-bogus="1"></span></div><div><span style="font-size: 13.3333px;"> error. We had this typo in one of our bot definitions and intelmq didn't complain about this typo in the configuration at any moment. Log level has been debug all the time afaik but no hints were written to /var/log/intelmq/intelmqctl.log for example. Thus, my vote for next feature implementation goes to stricter checks on the contents of runtime.yaml.</span></div><div><span style="font-size: 13.3333px;"><br data-mce-bogus="1"></span></div><div><span style="font-size: 13.3333px;">Br, Mika</span></div></div><div><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>From: </b>"Mika Silander" <mika.silander@csc.fi><br><b>To: </b>"intelmq-dev" <intelmq-dev@lists.cert.at><br><b>Sent: </b>Wednesday, 3 November, 2021 15:37:38<br><b>Subject: </b>Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1<br></div><div><br></div><div data-marker="__QUOTED_TEXT__"><div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:10pt;color:#000000"><div>Hi,</div><br><div> I do understand the separation of textual definition of a bot's parameters vs. the internal representation. In my problem case the regexp was in a class variable. If you refer to that variable using self.varnamehere it will show up among the bot's other instance parameters. So in my case this setting got inadvertently included among the ones that are iterated over when generating the JSON file for intelmq-manager.</div><br><div> More on the latter: the docs on <a href="https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration" rel="nofollow noopener noreferrer nofollow noopener noreferrer" style="background:#bfe6ff;font-family:'arial' , 'helvetica' , sans-serif;font-size:13.3333px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration</a> speak about a "vars.js" file, however, according to the console "var.js" is searched for. Once this file is defined and contains the right ROOT variable setting, we are left with one TypeError:</div><br><div>edge_map[path] is not iterable</div><br><div>in the javascript console. Unfortunately, I am not sufficiently familiar with Javascript either to sort out what generates this error.</div><br><div>Br, Mika</div><hr id="zwchr"><div><b>From: </b>"Sebastian Wagner" <wagner@cert.at><br><b>To: </b>"Mika Silander" <mika.silander@csc.fi>, "intelmq-dev" <intelmq-dev@lists.cert.at><br><b>Sent: </b>Wednesday, 3 November, 2021 15:05:36<br><b>Subject: </b>Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1<br></div><br><div><div class="moz-cite-prefix">Hi,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">I think there's a misunderstanding
between parameters - which can be set by users - and bots'
internals.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 11/3/21 1:50 PM, Mika Silander
wrote:<br>
</div>
<blockquote>
<pre class="moz-quote-pre"> Yes, I had a regexp in one of my bots although I don't understand why it should be prohibited to set these in a bot's parameter. It doesn't sound good to be limited to what can be serialized to JSON</pre>
</blockquote>
The requirement is that parameter's values can be <i>set</i>, in
the runtime configuration or in the manager etc. It's impossible to
express the instance of the re.Pattern class in runtime.yaml (and
JSON).<br>
<blockquote>
<pre class="moz-quote-pre">- most advanced Python features get excluded based on this criterion.</pre>
</blockquote>
<p>IntelMQ's configuration is intentionally not Python code, but
Text files.</p>
<p>It's perfectly fine to compile regular expressions.</p>
<p>If the expression comes from the user, i.e. it's a configuration
parameter, you can do the compilation in init:
<a href="https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/collectors/microsoft/collector_interflow.py#L79" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/collectors/microsoft/collector_interflow.py#L79</a><br>
</p>
<p>If it's not a parameter, but a constant value, the expression can
be compiled earlier:
<a href="https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/parsers/cymru/parser_cap_program.py#L47" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://github.com/certtools/intelmq/blob/7ebb8e16d821c372a44b077dd18a151c07f75807/intelmq/bots/parsers/cymru/parser_cap_program.py#L47</a><br>
</p>
<blockquote>
<pre class="moz-quote-pre"> gets downloaded correctly. The only thing remaining is <a href="https://ourimq/intelmq/v1/api/positions" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://ourimq/intelmq/v1/api/positions</a> accessed from within the Configuration tab:
Apache returns the response happily with HTTP 200 status, the JSON corresponding to /etc/intelmq/manager/positions.conf is returned but an error still
appears in the browser tab:
"Failed to load config file properly"
For the positions URL the Javascript console of the browser states two TypeError(s):
edge_map[path] is not iterable
and
app.network is null</pre>
</blockquote>
Sounds like a bug to me. I'd happy if someone with better knowledge
of the manager's JS code can have a look.<br>
<blockquote>
<pre class="moz-quote-pre"> Also, an attempt to load <a href="https://ourimq/intelmq_manager/js/var.js" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://ourimq/intelmq_manager/js/var.js</a> shows up but this file doesn't exist under /usr/share/intelmq_manager/html/js. A left-over from earlier versions?</pre>
</blockquote>
<p>No, that's intentional:
<a href="https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://intelmq.readthedocs.io/en/latest/user/intelmq-manager.html#configuration</a></p>
<p>best regards<br>
Sebastian<br>
</p>
<blockquote>
<pre class="moz-quote-pre"> Hints welcome as before.
Br, Mika
----- Original Message -----
From: "Sebastian Wagner" <a href="mailto:wagner@cert.at" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank"><wagner@cert.at></a>
To: "Mika Silander" <a href="mailto:mika.silander@csc.fi" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank"><mika.silander@csc.fi></a>, "intelmq-dev" <a href="mailto:intelmq-dev@lists.cert.at" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank"><intelmq-dev@lists.cert.at></a>
Sent: Tuesday, 2 November, 2021 15:15:52
Subject: Re: [IntelMQ-dev] Help for fixing configuration of intelmq-manager 3.0.1-1
Hi,
On 11/2/21 1:38 PM, Mika Silander wrote:
</pre>
<blockquote>
<pre class="moz-quote-pre"> Getting back to this anew. I've revised and rerevised all imaginable intelmq-manager configurations and problems remain. After debugging it looks like I still have issues
with intelmq-manager's Configuration tab, all other tabs are ok. From within the Configuration tab <a href="https://ourimq/intelmq/v1/api/bots" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://ourimq/intelmq/v1/api/bots</a> is accessed which in turn invokes (under the hood)
sudo -u www-data -u intelmq intelmqctl --type json list bots
This, I gather, is for generating the list of bots available on the right-hand side of the Configuration tab. Running this command from the command line gives:
Traceback (most recent call last):
File "/usr/bin/intelmqctl", line 11, in <module>
load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')()
File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main
return x.run()
File "/usr/lib/python3/dist-packages/intelmq/bin/intelmqctl.py", line 1051, in run
print(json.dumps(results))
File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Pattern is not JSON serializable
I assume this is the reason why the Configuration tab in my instance is dysfunctional.
</pre>
</blockquote>
<pre class="moz-quote-pre">Yes.
</pre>
<blockquote>
<pre class="moz-quote-pre">Any hints?
</pre>
</blockquote>
<pre class="moz-quote-pre">Do you have a custom bot which has a parameter that is of type
re.Pattern[0]? json.dumps fails serialzing this value. The solution is
to use a string as parameter type, then you are even able to configure
it :) You can compile the parameter's value at initialization.
Sebastian
[0]: <a href="https://docs.python.org/3/library/re.html#regular-expression-objects" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://docs.python.org/3/library/re.html#regular-expression-objects</a>
</pre>
</blockquote>
<pre class="moz-signature">--
// Sebastian Wagner <a href="mailto:wagner@cert.at" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank"><wagner@cert.at></a> - T: +43 676 898 298 7201
// CERT Austria - <a href="https://www.cert.at/" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://www.cert.at/</a>
// Eine Initiative der nic.at GmbH - <a href="https://www.nic.at/" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://www.nic.at/</a>
// Firmenbuchnummer 172568b, LG Salzburg</pre><br></div></div><br>_______________________________________________<br>IntelMQ-dev mailing list<br>https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev<br>https://intelmq.readthedocs.io/<br></div></div></body></html>