Hi Thomas, Mika and other readers
As Mika already assumed correctly, these fields are to be used by the Parser bot optionally.
https://docs.intelmq.org/latest/user/bots/#generic-mail-url-fetcher says "The resulting reports contain the following special fields:" (This explanation can be made clearer, as you are pointing out)
Parsers can use this information to process the data and, for example, decide the type of feed. For example, the Shadowserver parser uses extra.file_name. After parsing, this information is no longer needed, as it constitutes only additional information to a Report, not the resulting event. The mentioned fields will not automatically passed on to Events.
Short demo:
import intelmq.lib.message as msg
# create a report with such a field:
rep = msg.Report() rep.add('extra.file_name', 'test')
True
rep
{'time.observation': '2024-07-08T11:33:53+00:00', 'extra.file_name': 'test'} # now create an Event from the Report
ev = msg.Event(rep) ev
{'time.observation': '2024-07-08T11:33:53+00:00'}
Hope this helps Sebastian
Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578
On 7/8/24 11:16 AM, Thomas Hungenberg via IntelMQ-users wrote:
Hello,
according to https://github.com/certtools/intelmq/blob/develop/docs/user/bots.md events collected using a "Generic Mail URL Fetcher" should include this information:
feed.url extra.email_date extra.email_subject extra.email_from extra.email_message_id extra.file_name
In our database, the events DO include feed.url but DO NOT include any of the extra fields. Events collected using a "Generic Mail Attachment Fetcher" are missing the extra fields as well.
I wonder if this is a bug or caused by some configuration issue with our setup.
- Thomas