Good day,
I wanted to set up a test in intelmq, with the following pipe: Fetch Mail Attachement -> Parse Shadow Server csv -> output to file.
The mail and file output bots are running fine, however I am getting the following error with the shadowserver parser bot: 2018-06-05 10:54:51,880 - ShadowServer-Parser - INFO - ShadowserverParserBot initialized with id ShadowServer-Parser and intelmq 1.0.3 and python 3.5.2 (default, Nov 23 2017, 16:37:01) as process 7223. 2018-06-05 10:54:51,881 - ShadowServer-Parser - INFO - Bot is starting. 2018-06-05 10:54:51,888 - ShadowServer-Parser - ERROR - No feedname provided or feedname not in conf. 2018-06-05 10:54:51,889 - ShadowServer-Parser - INFO - Bot stopped.
The feedname parameter in my runtime.conf is definitely populated. I have tried defining different feed names, based on the report name from the shadowserver website, but still getting the same error. Any idea what I am missing?
I've attached my runtime.conf and pipeline.conf for reference. Unfortunately I cannot share the shadowserver email as it contains device scans in my ASN.
Regards, Peter
Hi Peter,
On 2018-06-05 11:11, Peter Chan wrote:
2018-06-05 10:54:51,880 - ShadowServer-Parser - INFO - ShadowserverParserBot initialized with id ShadowServer-Parser and intelmq 1.0.3 and python 3.5.2 (default, Nov 23 2017, 16:37:01) as process 7223.
Version 1.0.3 is outdated btw.
I have tried defining different feed names, based on the report name from the shadowserver website, but still getting the same error.
Shadowserver itself does not use consistent names unfortunately. Here is the documentation for the shadowserver parser in intelmq version 1.0.3: https://github.com/certtools/intelmq/blob/1.0.3/intelmq/bots/parsers/shadows... The feedname you are using in your configuration ("ASN Summary Report") is not in this list. Have a look at the list and see which feed name your report corresponds to.
Sebastian
On 05.06.2018 11:26, Sebastian Wagner wrote:
The feedname you are using in your configuration ("ASN Summary Report") is not in this list. Have a look at the list and see which feed name your report corresponds to.
A parser for the ASN Summary Report does not yet exist. Are those reports still provided by Shadowserver? Haven't seen them for a longer time.
- Thomas
CERT-Bund Incident Response & Malware Analysis Team
Thanks all for the feedback.
I've updated intelmq and using the feed name from the list and seems to be working - at least not getting errors.
@Thomas, I will have to confirm whether the ASN feeds are still available now, that used to be a thing.
Just another question regarding the feedname. Is it possible to define multiple feeds in the feedname parameter like:
"feedname": " Ssl-Scan ", "Ssl-Freak-Scan", " Open-SNMP",
On Tue, 5 Jun 2018 at 11:54, Thomas Hungenberg th@cert-bund.de wrote:
On 05.06.2018 11:26, Sebastian Wagner wrote:
The feedname you are using in your configuration ("ASN Summary Report") is not in this list. Have a look at the list and see which feed name your report corresponds to.
A parser for the ASN Summary Report does not yet exist. Are those reports still provided by Shadowserver? Haven't seen them for a longer time.
- ThomasCERT-Bund Incident Response & Malware Analysis Team
Hi,
On 2018-06-05 13:46, Peter Chan wrote:
Just another question regarding the feedname. Is it possible to define multiple feeds in the feedname parameter like:
"feedname": " |Ssl-Scan| ", "|Ssl-Freak-Scan|", " |Open-SNMP|",
No, as the parser needs to know what feed to expect to parse it correctly. That would not be possible with multiple feed names given. You need to create multiple collectors/parsers.
It might sound strange that the parser can't detect the feed, but as feeds look very similar and the csv headers change regularly, this would result in wrongly parsed feeds otherwise.
Sebastian
Hi Sebastian and team,
Another question regarding the feedname.
I currently have a directory of mixed feeds (Spam-URL, Open-mDNS, Open-LDAP etc.) that needs to be processed individually. Can the file-collector bot take regular expression in one of its parameters (feedname or postfix?) to filter which csv to queue to the appropriate shadowserver parser?
My current workaround is to move the different feeds into their own directory and then point the file-collector to their respective location - will automate this process pending answer :)
Regards, Peter
On Tue, 5 Jun 2018 at 13:51, Sebastian Wagner wagner@cert.at wrote:
Hi,
On 2018-06-05 13:46, Peter Chan wrote:
Just another question regarding the feedname. Is it possible to define multiple feeds in the feedname parameter like:
"feedname": " Ssl-Scan ", "Ssl-Freak-Scan", " Open-SNMP",
No, as the parser needs to know what feed to expect to parse it correctly. That would not be possible with multiple feed names given. You need to create multiple collectors/parsers.
It might sound strange that the parser can't detect the feed, but as feeds look very similar and the csv headers change regularly, this would result in wrongly parsed feeds otherwise.
Sebastian
-- // Sebastian Wagner wagner@cert.at wagner@cert.at - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
Hi,
On 2018-06-21 14:17, Peter Chan wrote:
I currently have a directory of mixed feeds (Spam-URL, Open-mDNS, Open-LDAP etc.) that needs to be processed individually. Can the file-collector bot take regular expression in one of its parameters (feedname or postfix?) to filter which csv to queue to the appropriate shadowserver parser?
The file selection/search uses globbing currently: [path]/*[postfix] I think this is most intuitive.
I guess you mean that a regular expression search on the filename results in the name of the path the bot sends the data to. That's an interesting use case for the newly introduced named destination pipelines :)
Pull requests are welcome.
Sebastian
Hi,
On 2018-06-21 14:17, Peter Chan wrote:
I currently have a directory of mixed feeds (Spam-URL, Open-mDNS, Open-LDAP etc.) that needs to be processed individually. Can the file-collector bot take regular expression in one of its parameters (feedname or postfix?) to filter which csv to queue to the appropriate shadowserver parser?
Not currently, but that is possible to implement. First, the file collector would need regex-support. Then a match group could be used as "parameter" for the sending step. Second, the file collector would need to support queue paths, using the match group of the file name to format the path.
Paths are a great new feature available in the to-be-released version 1.1.0 and in the development branch. See the docs here: https://github.com/certtools/intelmq/blob/maintenance/docs/User-Guide.md#pip...
Sebastian