I think the URL parsing is fixed by Thomas' PR https://github.com/certtools/intelmq/pull/1243 That was part of the last releases already
On 2018-01-07 00:20, Tomislav Protega wrote:
I took a look at the other reports where there is domain under 'http_host', but the main problem is that parser is joining wrong fields from shadowserver report.
It joins 'hostname' with 'url' parameters which it shouldn't do, because under hostname is actually dns ptr record (source_reverse.dns). So it should join 'http_host'(source.fqdn) + 'url' to get the real source.url.
Regards,
-- Tomislav
On 07.01.2018 00:02, Tomislav Protega wrote:
Hi,
I ran into this error: Shadowserver-Compromised-Website-Parser - ERROR - Could not convert shadowkey: 'http_host', value: '' via conversion function 'validate_fqdn'. More detailed log is attached.
This happens when "http_host" field in the shadowserver origin report contains IP instead of domain which is not something unusual.
At the end IntelMQ does produce the output data, but there's no 'source.url' field which should contain merged 'http_host' and 'url' parameters from the origin report.
Regards,