[IntelMQ-dev] A suitable collector for the Abusech Feodo Tracker feed

[Mika Silander]

Hi Sebastian, 

That explains although I think the precise snippet of source code I missed is inside the try-expect clause, in between the two unzip calls: 

except ValueError: 
raw_reports.append((None, resp.text)) 

This is the part of the code that gets executed in the case the collected input data is not zipped. I missed that and then assumed the input data is always packed in some way (zip and friends). 

Thanks again, Mika 

From: "Sebix" <sebix at sebix.at> 
To: "Mika Silander" <mika.silander at csc.fi>, "intelmq-dev" <intelmq-dev at lists.cert.at> 
Sent: Monday, 16 December, 2024 13:50:21 
Subject: Re: [IntelMQ-dev] A suitable collector for the Abusech Feodo Tracker feed 



Dear Mika 

You can find the documentation for the HTTP collector (aka Generic URL Fetcher) here: 
[ https://docs.intelmq.org/latest/user/bots/#generic-url-fetcher | https://docs.intelmq.org/latest/user/bots/#generic-url-fetcher ] 
and especially the paragraph on the parameter extract_files. 

> Not 100% sure but it looks to me that collector_http.py for example, expects the incoming data to be in zip format since in the sources one can see unzipping being done. Correct? 

No, the collector does expect zipped data unless the parameter extract_files is in use. 
What you may have mis-interpreted in the source code is the automatic unzipping if the input data is a valid zip-archive. 
btw: extract_files works with zip, gzip (gz), tar and tar.gz archives. 


> Therefore, my question was, what would be the recommended collector to be used to push reports to the Abusech Feodo Tracker parser? 

The correct parser to use the Abusech Feodo Tracker feed, is the HTTP collector/Generic URL Fetcher. 
See also our feed documentation: [ https://docs.intelmq.org/latest/user/feeds/#feodo-tracker | https://docs.intelmq.org/latest/user/feeds/#feodo-tracker ] 


Hope that helps 

Sebastian 
Institute for Common Good Technology
gemeinnütziger Kulturverein - nonprofit cultural society [ https://commongoodtechnology.org/ | https://commongoodtechnology.org/ ] ZVR 1510673578 
On 12/16/24 12:37 PM, Mika Silander wrote: 



Hi Sebastian, all, 

Seems I rushed when sending out a message to the list (once again, I shouldn't have). Yes, I checked the feed's current contents after clicking "send" and, as you said, there were no events. 
What comes to my comment on http collectors manipulating data, a better wording would have been "the http collectors make assumptions on the structure of the incoming data". Not 100% sure but it looks to me that collector_http.py for example, expects the incoming data to be in zip format since in the sources one can see unzipping being done. Correct? 

I hadn't tried to fetch the ipblocklist.json with any collector yet since I thought the collector_http.py would not be suitable due to the unzipping. Therefore, my question was, what would be the recommended collector to be used to push reports to the Abusech Feodo Tracker parser? I expect the parser to be fine as long as its incoming reports are plain JSON(?) 

If you hear something concerning the Feodo tracker feed, please let me know. Meanwhile, I'll look for other candidate sources for vuln info. 

Br, Mika 


From: "Sebix" [ mailto:sebix at sebix.at | <sebix at sebix.at> ] 
To: "Mika Silander" [ mailto:mika.silander at csc.fi | <mika.silander at csc.fi> ] , "intelmq-dev" [ mailto:intelmq-dev at lists.cert.at | <intelmq-dev at lists.cert.at> ] 
Sent: Friday, 13 December, 2024 19:48:23 
Subject: Re: [IntelMQ-dev] A suitable collector for the Abusech Feodo Tracker feed 

On 12/13/24 6:38 PM, Sebix wrote: 

BQ_BEGIN
On 12/13/24 1:29 PM, Mika Silander via IntelMQ-dev wrote: 

BQ_BEGIN
I'm attempting to find a suitable collector for retrieving the Abusech Feodo Tracker feed ( [ https://feodotracker.abuse.ch/downloads/ipblocklist.json | https://feodotracker.abuse.ch/downloads/ipblocklist.json ] ). Afaiks, the ready-made Abusech Feodo Tracker parser expects reports in plain JSON but the available http collectors are manipulating the retrieved information in one way or the other before passing it on to the parser. 



Not sure what you mean with the http collector data manipulation, but to me it appears that the feodotracker is either dysfunctional or dead. Not one of the data feed files contains actual data. 

BQ_END


Never mind, the other feeds are empty because there's simply no data. 😇️ 


Parsing the mentioned 
[ https://feodotracker.abuse.ch/downloads/ipblocklist.json | https://feodotracker.abuse.ch/downloads/ipblocklist.json ] 
works fine with 
intelmq.bots.parsers.abusech.parser_feodotracker 
as documented in [ https://docs.intelmq.org/latest/user/feeds/#feodo-tracker | https://docs.intelmq.org/latest/user/feeds/#feodo-tracker ] 


Could you please describe what erroneous behavior you see? 

best regards 
Sebastian 
-- 
Institute for Common Good Technology
gemeinnütziger Kulturverein - nonprofit cultural society [ https://commongoodtechnology.org/ | https://commongoodtechnology.org/ ] ZVR 1510673578 


BQ_END

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20241216/19f3d1c5/attachment-0001.htm>