Dear Mika

You can find the documentation for the HTTP collector (aka Generic URL Fetcher) here:
https://docs.intelmq.org/latest/user/bots/#generic-url-fetcher
and especially the paragraph on the parameter extract_files.

> Not 100% sure but it looks to me that collector_http.py for example, expects the incoming data to be in zip format since in the sources one can see unzipping being done. Correct?

No, the collector does expect zipped data unless the parameter extract_files is in use.
What you may have mis-interpreted in the source code is the automatic unzipping if the input data is a valid zip-archive.
btw: extract_files works with zip, gzip (gz), tar and tar.gz archives.

> Therefore, my question was, what would be the recommended collector to be used to push reports to the Abusech Feodo Tracker parser?

The correct parser to use the Abusech Feodo Tracker feed, is the HTTP collector/Generic URL Fetcher.
See also our feed documentation: https://docs.intelmq.org/latest/user/feeds/#feodo-tracker

Hope that helps

Sebastian

Institute for Common Good Technology
gemeinnütziger Kulturverein - nonprofit cultural society
https://commongoodtechnology.org/
ZVR 1510673578
On 12/16/24 12:37 PM, Mika Silander wrote:
Hi Sebastian, all,

 Seems I rushed when sending out a message to the list (once again, I shouldn't have). Yes, I checked the feed's current contents after clicking "send" and, as you said, there were no events.
What comes to my comment on http collectors manipulating data, a better wording would have been "the http collectors make assumptions on the structure of the incoming data". Not 100% sure but it looks to me that collector_http.py for example, expects the incoming data to be in zip format since in the sources one can see unzipping being done. Correct?

 I hadn't tried to fetch the ipblocklist.json with any collector yet since I thought the collector_http.py would not be suitable due to the unzipping. Therefore, my question was, what would be the recommended collector to be used to push reports to the Abusech Feodo Tracker parser? I expect the parser to be fine as long as its incoming reports are plain JSON(?)

 If you hear something concerning the Feodo tracker feed, please let me know. Meanwhile, I'll look for other candidate sources for vuln info.

Br, Mika

From: "Sebix" <sebix@sebix.at>
To: "Mika Silander" <mika.silander@csc.fi>, "intelmq-dev" <intelmq-dev@lists.cert.at>
Sent: Friday, 13 December, 2024 19:48:23
Subject: Re: [IntelMQ-dev] A suitable collector for the Abusech Feodo Tracker feed

On 12/13/24 6:38 PM, Sebix wrote:
On 12/13/24 1:29 PM, Mika Silander via IntelMQ-dev wrote:
  I'm attempting to find a suitable collector for retrieving the Abusech Feodo Tracker feed (https://feodotracker.abuse.ch/downloads/ipblocklist.json). Afaiks, the ready-made Abusech Feodo Tracker parser expects reports in plain JSON but the available http collectors are manipulating the retrieved information in one way or the other before passing it on to the parser.

Not sure what you mean with the http collector data manipulation, but to me it appears that the feodotracker is either dysfunctional or dead. Not one of the data feed files contains actual data.

Never mind, the other feeds are empty because there's simply no data. 😇️

Parsing the mentioned
https://feodotracker.abuse.ch/downloads/ipblocklist.json
works fine with
intelmq.bots.parsers.abusech.parser_feodotracker
as documented in https://docs.intelmq.org/latest/user/feeds/#feodo-tracker

Could you please describe what erroneous behavior you see?

best regards
Sebastian

-- 
Institute for Common Good Technology
gemeinnütziger Kulturverein - nonprofit cultural society
https://commongoodtechnology.org/
ZVR 1510673578