<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; color: #000000"><div>Hi Sebastian,<br></div><div><br></div><div> That explains although I think the precise snippet of source code I missed is inside the try-expect clause, in between the two unzip calls:<br></div><div><br data-mce-bogus="1"></div><div><div style="background-color: #ffffff; color: #080808;" data-mce-style="background-color: #ffffff; color: #080808;"><pre style="font-family: 'JetBrains Mono',monospace; font-size: 9.8pt;" data-mce-style="font-family: 'JetBrains Mono',monospace; font-size: 9.8pt;"><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 0); font-size: 10pt;" data-mce-style="font-family: arial, helvetica, sans-serif; color: #000000; font-size: 10pt;">except ValueError:</span><br><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 0); font-size: 10pt;" data-mce-style="font-family: arial, helvetica, sans-serif; color: #000000; font-size: 10pt;"> raw_reports.append((None, resp.text))<br><br> This is the part of the code that gets executed in the case the collected input data is not zipped. I missed that and then assumed the input data is always packed in some way (zip and friends).<br><br>Thanks again, Mika<br></span></pre></div></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>From: </b>"Sebix" <sebix@sebix.at><br><b>To: </b>"Mika Silander" <mika.silander@csc.fi>, "intelmq-dev" <intelmq-dev@lists.cert.at><br><b>Sent: </b>Monday, 16 December, 2024 13:50:21<br><b>Subject: </b>Re: [IntelMQ-dev] A suitable collector for the Abusech Feodo Tracker feed<br></div><div><br></div><div data-marker="__QUOTED_TEXT__"><p>Dear Mika</p>
<p>You can find the documentation for the HTTP collector (aka
Generic URL Fetcher) here:<br>
<a href="https://docs.intelmq.org/latest/user/bots/#generic-url-fetcher" target="_blank" rel="nofollow noopener noreferrer">https://docs.intelmq.org/latest/user/bots/#generic-url-fetcher</a><br>
and especially the paragraph on the parameter extract_files.</p>
<p>> Not 100% sure but it looks to me that collector_http.py for
example, expects the incoming data to be in zip format since in
the sources one can see unzipping being done. Correct?</p>
<p>No, the collector does expect zipped data unless the parameter
extract_files is in use.<br>
What you may have mis-interpreted in the source code is the
automatic unzipping if the input data is a valid zip-archive.<br>
btw: extract_files works with zip, gzip (gz), tar and tar.gz
archives.<br>
</p>
<p>> Therefore, my question was, what would be the recommended
collector to be used to push reports to the Abusech Feodo Tracker
parser?</p>
<p>The correct parser to use the Abusech Feodo Tracker feed, is the
HTTP collector/Generic URL Fetcher.<br>
See also our feed documentation:
<a href="https://docs.intelmq.org/latest/user/feeds/#feodo-tracker" target="_blank" rel="nofollow noopener noreferrer">https://docs.intelmq.org/latest/user/feeds/#feodo-tracker</a><br data-mce-bogus="1"></p>
<p>Hope that helps</p>
<p>Sebastian<br>
</p>
<pre class="moz-signature">Institute for Common Good Technology
gemeinnütziger Kulturverein - nonprofit cultural society
<a href="https://commongoodtechnology.org/" target="_blank" rel="nofollow noopener noreferrer">https://commongoodtechnology.org/</a>
ZVR 1510673578</pre>
<div class="moz-cite-prefix">On 12/16/24 12:37 PM, Mika Silander
wrote:<br>
</div>
<blockquote>
<div style="font-family:'arial' , 'helvetica' , sans-serif;font-size:10pt;color:#000000">
<div>Hi Sebastian, all,<br>
</div>
<div><br>
</div>
<div> Seems I rushed when sending out a message to the list
(once again, I shouldn't have). Yes, I checked the feed's
current contents after clicking "send" and, as you said, there
were no events.<br>
</div>
<div>What comes to my comment on http collectors manipulating
data, a better wording would have been "the http collectors
make assumptions on the structure of the incoming data". Not
100% sure but it looks to me that collector_http.py for
example, expects the incoming data to be in zip format since
in the sources one can see unzipping being done. Correct?<br>
</div>
<div><br>
</div>
<div> I hadn't tried to fetch the ipblocklist.json with any
collector yet since I thought the collector_http.py would not
be suitable due to the unzipping. Therefore, my question was,
what would be the recommended collector to be used to push
reports to the Abusech Feodo Tracker parser? I expect the
parser to be fine as long as its incoming reports are plain
JSON(?)<br>
</div>
<div><br>
</div>
<div> If you hear something concerning the Feodo tracker feed,
please let me know. Meanwhile, I'll look for other candidate
sources for vuln info.<br>
</div>
<div><br>
</div>
<div>Br, Mika<br>
</div>
<div><br>
</div>
<hr id="zwchr">
<div><b>From: </b>"Sebix"
<a href="mailto:sebix@sebix.at" target="_blank" rel="nofollow noopener noreferrer"><sebix@sebix.at></a><br>
<b>To: </b>"Mika Silander" <a href="mailto:mika.silander@csc.fi" target="_blank" rel="nofollow noopener noreferrer"><mika.silander@csc.fi></a>,
"intelmq-dev" <a href="mailto:intelmq-dev@lists.cert.at" target="_blank" rel="nofollow noopener noreferrer"><intelmq-dev@lists.cert.at></a><br>
<b>Sent: </b>Friday, 13 December, 2024 19:48:23<br>
<b>Subject: </b>Re: [IntelMQ-dev] A suitable collector for
the Abusech Feodo Tracker feed<br>
</div>
<div><br>
</div>
<div>
<div class="moz-cite-prefix">On 12/13/24 6:38 PM, Sebix wrote:<br>
</div>
<blockquote>On 12/13/24 1:29 PM, Mika Silander via IntelMQ-dev
wrote: <br>
<blockquote> I'm attempting to find a suitable collector
for retrieving the Abusech Feodo Tracker feed (<a href="https://feodotracker.abuse.ch/downloads/ipblocklist.json" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://feodotracker.abuse.ch/downloads/ipblocklist.json</a>).
Afaiks, the ready-made Abusech Feodo Tracker parser
expects reports in plain JSON but the available http
collectors are manipulating the retrieved information in
one way or the other before passing it on to the parser. <br>
</blockquote>
<br>
Not sure what you mean with the http collector data
manipulation, but to me it appears that the feodotracker is
either dysfunctional or dead. Not one of the data feed files
contains actual data. <br>
</blockquote>
<p>Never mind, the other feeds are empty because there's
simply no data.<font size="5"> 😇️</font><br>
</p>
<p>Parsing the mentioned<br>
<a href="https://feodotracker.abuse.ch/downloads/ipblocklist.json" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://feodotracker.abuse.ch/downloads/ipblocklist.json</a><br>
works fine with<br>
intelmq.bots.parsers.abusech.parser_feodotracker<br>
as documented in <a href="https://docs.intelmq.org/latest/user/feeds/#feodo-tracker" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://docs.intelmq.org/latest/user/feeds/#feodo-tracker</a><br>
</p>
<p>Could you please describe what erroneous behavior you see?</p>
<p>best regards<br>
Sebastian<br>
</p>
<pre class="moz-signature">--
Institute for Common Good Technology
gemeinnütziger Kulturverein - nonprofit cultural society
<a href="https://commongoodtechnology.org/" rel="nofollow noopener noreferrer nofollow noopener noreferrer" target="_blank">https://commongoodtechnology.org/</a>
ZVR 1510673578</pre>
<br>
</div>
</div>
</blockquote><br></div></div></body></html>