- IntelMQ-users

Updating datastores
by joanna＠scate.tech 18 Sep '18

18 Sep '18

Hi, Is there a way of updating outputs such as databases when IOCs expire? Don't want to spend time re-inventing the wheel. Thanks.

4 5

IntelMQ 1.1.0 released
by Sebastian Wagner 05 Sep '18

05 Sep '18

Dear community, I just released the feature release 1.1.0 for the 1.x series the first stable release after the initial 1.0.0. This version is already in use in several setups without known issues and introduces many new features and bots. Installation instructions: https://github.com/certtools/intelmq/blob/1.1.0/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.0/docs/UPGRADING.md The deb/rpm packages will be available in some hours in the stable repository (see linked instructions). The full changelog is very long: - Support for Python 3.3 has been dropped in IntelMQ and some dependencies of it. Python 3.3 reached its end of life and Python 3.4 or newer is a hard requirement now. - The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml A tool to convert from yaml to md has been added. ### Tools - `intelmq_gen_feeds_docs` addded to bin directory, allows generating the Feeds.md documentation file from feeds.yaml - `intelmq_gen_docs` merges both `intelmq_gen_feeds_docs` and `intelmq_gen_harm_docs` in one file and automatically updates the documentation files. #### intelmqctl - `intelmqctl start` prints the bot's last error messages if the bot failed to start (#1021). - `intelmqctl start` message "is running" is printed every time. (Until now, it wasn't said when a bot was just starting.) - `intelmqctl start/stop/restart/reload/status` now has a "--group" flag which allows you to specify the group of the bots that should be influenced by the command. - `intelmqctl check` checks for defaults.conf completeness if the shipped file from the package can be found. - `intelmqctl check` shows errors for non-importable bots. - `intelmqctl list bots -q` only prints the IDs of enabled bots. - `intelmqctl list queues-and-status` prints both queues and bots statuses (so that it can be used in eg. intelmq-manager). - `intelmqctl run` parameter for showing a sent message. - `intelmqctl run` if message is sent to a non-default path, it is printed out. - `intelmqctl restart` bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226). - `intelmqctl check`: New parameter `--no-connections` to prevent the command from making connections e.g. to the redis pipeline.s - `intelmqctl list queues`: don't display named paths amongst standard queues. - The process status test failed if the PATH did not include the bot executables and the `which` command failed. Then the proccess's command line could not be compared correctly. The fix warns of this and adds a new status 'unknown' (#1297). ### Contrib - tool `feeds-config-generator` to automatically generate the collector and parser runtime and pipeline configurations. - `malware_name_mapping`: Download and convert tool for malware family name mapping has been added. - Added a systemd script which creates systemd units for bots (#953). - `contrib/cron-jobs/update-asn-data`, `contrib/cron-jobs/update-geoip-data`, `contrib/cron-jobs/update-tor-nodes`: Errors produce proper output. ### Core - lib/bot - use SIGTERM instead of SIGINT to stop bots (#981). - Bots can specify a static method `check(parameters)` which can perform individual checks specific to the bot. These functions will be called by `intelmqctl check` if the bot is configured with the given parameters - top level bot parameters (description, group, module, name) are exposed as members of the class. - The parameter `feed` for collectors is deprecated for 2.0 and has been replaced by the more consistent `name` (#1144). - bug: allow path parameter for CollectorBot class. - Handle errors better when the logger could not be initialized. - `ParserBot`: - For the csv parsing methods, `ParserBot.csv_params` is now used for all these methods. - `ParserBot.parse_csv_dict` now saves the field names in `ParserBot.csv_fieldnames`. - `ParserBot.parse_csv_dict` now saves the raw current line in `ParserBot.current_line`. - `ParserBot.recover_line_csv_dict` now uses the raw current line. - lib/message: - Subitems in fields of type `JSONDict` (see below) can be accessed directly. E.g. you can do: event['extra.foo'] = 'bar' event['extra.foo'] # gives 'bar' It is still possible to set and get the field as whole, however this may be removed or changed in the future: event['extra'] = '{"foo": "bar"}' event['extra'] # gives '{"foo": "bar"}' "Old" bots and configurations compatible with 1.0.x do still work. Also, the extra field is now properly exploded when exporting events, analogous to all other fields. The `in` operator works now for both - the old and the new - behavior. - `Message.add`: The parameter `overwrite` accepts now three different values: `True`, `False` and `None` (new). True: An existing value will be overwritten False: An existing value will not be overwritten (previously an exception has been raised when the value was given). None (default): If the value exists an `KeyExists` exception is thrown (previously the same as False). This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function. - The message class has now the possibility to return a default value for non-exisiting fields, see `Message.set_default_value`. - Message.get behaves the same like `Message.__getitem__` (#1305). - Add `RewindableFileHandle` to utils making handling of CSV files more easy (optionally) - lib/pipeline: * you may now define more than one destination queues path the bot should pass the message to, see [Pipelines](https://github.com/certtools/intelmq/blob/develop/docs/User-Guid… (#1088, #1190). * the special path `"_on_error"` can be used to pass messages to different queues in case of processing errors (#1133). - `lib/harmonization`: Accept `AS` prefix for ASN values (automatically stripped). ### Bots - Removed print statements from various bots. - Replaced various occurences of `self.logger.error()` + `self.stop()` with `raise ValueError`. #### Collectors - `bots.collectors.mail`: - New parameters; `sent_from`: filter messages by sender, `sent_to`: filter messages by recipient - More debug logs - `bots.collectors.n6.collector_stomp`: renamed to `bots.collectors.stomp.collector` (#716) - bots.collectors.rt: - New parameter `search_requestor` to search for field Requestor. - Empty strings and `null` as value for search parameters are ignored. - Empty parameters `attachment_regex` and `url_regex` handled. - `bots.collectors.http.collector_http`: Ability to optionally use the current time in parameter `http_url`, added parameter `http_url_formatting`. - `bots.collectors.stomp.collector`: Heartbeat timeout is now logged with log level info instead of warning. - added `intelmq.bots.collectors.twitter.collector_twitter` - added `intelmq.bots.collectors.tcp.collector` that can be bound to another IntelMQ instance by a TCP output - `bots.collectors.microsoft.collector_interflow`: added for MS interflow API - Automatic ungzipping for .gz files. - added `intelmq.bots.collectors.calidog.collector_certstream` for collecting certstream data (#1120). - added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (#1096). - Add proxy support. - Fix handling of parameter `countries`. #### Parsers - `bots.parsers.shadowserver`: - changed feednames . Please refer to it's README for the exact changes. - If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration. Previously errors like these were only logged and ignored otherwise. - add support for the feeds - `Accessible-Hadoop` (#1231) - `Accessible ADB` (#1285) - Remove deprecated parameter `override`, use `overwrite` instead (#1071). - The `raw` values now are exactly the input with quotes unchanged, the ParserBot methods are now used directly (#1011). - The Generic CSV Parser `bots.parsers.generic.parser_csv`: - It is possible to filter the data before processing them using the new parameters `filter_type` and `filter_text`. - It is possible to specify multiple columns using `|` character in parameter `columns`. - The parameter `time_format` now supports `'epoch_millis'` for seconds since the Epoch, milliseconds are supported but not used. - renamed `bots.parsers.cymru_full_bogons.parser` to `bots.parsers.cymru.parser_full_bogons`, compatibility shim will be removed in version 2.0 - added `bots.parsers.cymru.parser_cap_program` - added `intelmq.bots.parsers.zoneh.parser` for ZoneH feeds - added `intelmq.bots.parsers.sucuri.parser` - added `intelmq.bots.parsers.malwareurl.parser` - added `intelmq.bots.parsers.threatminer.parser` - added `intelmq.bots.parsers.webinspektor.parser` - added `intelmq.bots.parsers.twitter.parser` - added `intelmq.bots.parsers.microsoft.parser_ctip` * ignore the invalid IP '0.0.0.0' for the destination * fix the raw/dumped messages, did not contain the paling list previously. * use the new harmonization field `tlp` instead of `extra.tlp`. - `bots.parsers.alienvault.parser_otx`: Save TLP data in the new harmonization field `tlp`. - added `intelmq.bots.parsers.openphish.parser_commercial` - added `intelmq.bots.parsers.microsoft.parser_bingmurls` - added `intelmq.bots.parsers.calidog.parser_certstream` for parsing certstream data (#1120). - added `intelmq.bots.parsers.shodan.parser` for parsing shodan data (#1096). - change the classification type from 'botnet drone' to infected system' in various parses. - `intelmq.bots.parsers.spamhaus.parser_cert`: Added support for all known bot types. #### Experts - Added sieve expert for filtering and modifying events (#1083) * capable of distributing the event to appropriate named queues - `bots.experts.modify` * default rulesets: all malware name mappings have been migrated to the [Malware Name Mapping repository](https://github.com/certtools/malware_name_mapping) ruleset. See the new added contrib tool for download and conversion. * new parameter `case_sensitive` (default: True) - Added wait expert for sleeping - Added domain suffix expert to extract the TLD/Suffix from a domain name. - `bots.experts.maxmind_geoip`: New (optional) parameter `overwrite`, by default false. The current default was to overwrite! - `intelmq.bots.experts.ripencc_abuse_contact`: * Extend deprecated parameter compatibility `query_ripe_stat` until 2.0 because of a logic bug in the compatibility code, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (#1071, #1291). * Handle HTTP status code 404 for DB AS queries. * Add caching capability. - `intelmq/bots/experts/asn_lookup/update-asn-data`: Errors produce proper output on stdout/stderr. - `intelmq/bots/experts/maxmind_geoip/update-geoip-data`: Errors produce proper output on stdout/stderr. - `intelmq/bots/experts/tor_nodes/update-tor-nodes`: Errors produce proper output on stdout/stderr. #### Outputs - `bots.outputs.file`: - String formatting can be used for file names with new parameter `format_filename`. - New parameter `single_key` to only save one field. - New parameter `encoding_errors_mode` with default value `'strict'` to handle encoding errors for the files written. ### Harmonization - Renamed `JSON` to `JSONDict` and added a new type `JSON`. `JSONDict` saves data internally as JSON, but acts like a dictionary. `JSON` accepts any valid JSON. - fixed regex for `protocol.transport` it previously allowed more values than it should have. - New ASN type. Like integer but checks the range. - added `destination.urlpath` and `source.urlpath` to harmonization. - New field `tlp` for tlp level specification. - New TLP type. Allows all four tlp levels, removes 'TLP:' prefix and converts to upper case. - Added new `classification.type` 'vulnerable client' - Added `(destination|source).domain_suffix` to hold the TLD/domain suffix. - New allowed value for `classification.type`: `infected system` for taxonomy `malicious code` (#1197). ### Requirements - Requests is no longer listed as dependency of the core. For depending bots the requirement is noted in their REQUIREMENTS.txt file. ### Documentation - Use Markdown for README again, as pypi now supports it. - Developers Guide: Add instructions for pre-release testing. ### Packaging - Add logcheck configuration to the packages. - Fix packaging of bash completion script. ### Tests - Travis now correctly stops if a requirement could not be installed (#1257). - New tests for validating `etc/feeds.yaml` and `bots/BOTS` using cerberus and schemes are added (#1166). - New test for checking if `docs/Feeds.md` is up to date with `etc/feeds.yaml`. ### Known bugs - contrib: feeds-config-generator does not add feed name as parameter (#1314). - bot debugger requires configured source pipeline (#1307). - shadowserver parser: drone feed has spam events (#1271). - debug log level on python 3.7 not applied (#1269). - `bots.experts.sieve` does not support textX (#1246). - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 1.0.6 released
by Sebastian Wagner 05 Sep '18

05 Sep '18

Dear community, at the end of last weak I finalized the last planned maintenance release of the 1.0.x series, version 1.0.6. I do not intend to do any further maintenance releases for the 1.0 version, however I am open for discussion here. The branch maintenance-1.0 has all the commits of the 1.0.x series. Installation instructions: https://github.com/certtools/intelmq/blob/1.0.6/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.6/docs/UPGRADING.md The changelog: ### Bots #### Collectors - `bots.collectors.rt.collector_rt`: Log ticket id for downloaded reports. #### Parsers - `bots.parsers.shadowserver`: - if required fields do not exist in data, an exception is raised, so the line will be dumped and not further processed. - fix a bug in the parsing of column `cipher_suite` in ssl poodle reports (#1288). #### Experts - Reverse DNS Expert: ignore all invalid results and use first valid one (#1264). - `intelmq/bots/experts/tor_nodes/update-tor-nodes`: Use check.torproject.org as source as internet2.us is down (#1289). #### Outputs - `bots.output.amqptopic`: - The default exchange must not be declared (#1295). - Unencodable characters are prepended by backslashes by default. Otherwise Unicode characters can't be encoded and sent (#1296). - Gracefully close AMQP connection on shutdown of bot. ### Documentation - Bots: document redis cache parameters. - Installation documentation: Ubuntu needs universe repositories. ### Packaging - Dropped support for Ubuntu 17.10, it reached its End of Life as of 2018-07-19. ### Tests - Drop tests for Python 3.3 for the mode with all requirements, as some optional dependencies do not support Python 3.3 anymore. - `lib.test`: Add parameter `compare_raw` (default: `True`) to `assertMessageEqual`, to optionally skip the comparison of the raw field. - Add tests for RT collector. - Add tests for Shadowserver Parser: - SSL Poodle Reports. - Helper functions. ### Tools - `intelmqctl list` now sorts the output of bots and queues (#1262). - `intelmqctl`: Correctly handle the corner cases with collectors and outputs for getting/sending messages in the bot debugger (#1263). - `intelmqdump`: fix ordering of dumps in a file in runtime. All operations are applied to a sorted list (#1280). ### Contrib - `cron-jobs/update-tor-nodes`: Use check.torproject.org as source as internet2.us is down (#1289). Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

1.1.0 release candidate
by Sebastian Wagner 07 Aug '18

07 Aug '18

Hi, I just released a release candidate version for 1.1.0. If there are no big issues coming up in the next days, there won't be another release candidate and a final release in one or two weeks, depending on the feedback. Everybody is encouraged to give feedback and test this pre release. 1.0.x installations are compatible with the 1.1.0 code. Some things have been deprecated, have a look at warnings and `intelmqctl check`. The deb/rpm packages in the unstable repository will be available tomorrow or next week. This is the full current changelog: Installation instructions: https://github.com/certtools/intelmq/blob/1.1.0rc1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.0rc1/docs/UPGRADING.md - Support for Python 3.3 has been dropped, it reached its end of life. - The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml A tool to convert from yaml to md has been added. # Tools - `intelmq_gen_feeds_docs` add to bin directory, allows generating the Feeds.md documentation file from feeds.yaml - `intelmq_gen_docs` merges both `intelmq_gen_feeds_docs` and `intelmq_gen_harm_docs` in one file and automatically updates the documentation files. ## intelmqctl - `intelmqctl start` prints the bot's last error messages if the bot failed to start (#1021). - `intelmqctl start` message "is running" is printed every time. (Until now, it wasn't said when a bot was just starting.) - `intelmqctl start/stop/restart/reload/status` now have a "--group" flag which allows you to specify the group of the bots that should be influenced by the command. - `intelmqctl check` checks for defaults.conf completeness if the shipped file from the package can be found. - `intelmqctl check` shows errors for non-importable bots. - `intelmqctl list bots -q` only prints the IDs of enabled bots. - `intelmqctl list queues-and-status` prints both queues and bots statuses (so that it can be used in eg. intelmq-manager). - `intelmqctl run` parameter for showing a sent message. - `intelmqctl run` if message is sent to a non-default path, it is printed out. - `intelmqctl restart` bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226). - `intelmqctl check`: New parameter `--no-connections` to prevent the command from making connections e.g. to the redis pipeline.s # Contrib - tool `feeds-config-generator` to automatically generate the collector and parser runtime and pipeline configurations. - `malware_name_mapping`: Download and convert tool for malware family name mapping has been added. - Added a systemd script which creates systemd units for bots (#953). # Core - lib/bot - use SIGTERM instead of SIGINT to stop bots (#981). - Bots can specify a static method `check(parameters)` which can perform individual checks specific to the bot. These functions will be called by `intelmqctl check` if the bot is configured with the given parameters - top level bot parameters (description, group, module, name) are exposed as members of the class. - The parameter `feed` for collectors is deprecated for 2.0 and has been replaced by the more consistent `name` (#1144). - lib/message: - Subitems in fields of type `JSONDict` (see below) can be accessed directly. E.g. you can do: event['extra.foo'] = 'bar' event['extra.foo'] # gives 'bar' It is still possible to set and get the field as whole, however this may be removed or changed in the future: event['extra'] = '{"foo": "bar"}' event['extra'] # gives '{"foo": "bar"}' "Old" bots and configurations compatible with 1.0.x do still work. Also, the extra field is now properly exploded when exporting events, analogous to all other fields. - `Message.add`: The parameter `overwrite` accepts now three different values: `True`, `False` and `None` (new). True: An existing value will be overwritten False: An existing value will not be overwritten (previously an exception has been raised when the value was given). None (default): If the value exists an `KeyExists` exception is thrown (previously the same as False). This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function. - The message class has now the possibility to return a default value for non-exisiting fields, see `Message.set_default_value`. - Add `RewindableFileHandle` to utils making handling of CSV files more easy (optionally) - lib/pipeline: * you may now define more than one destination queues path the bot should pass the message to, see [Pipelines](https://github.com/certtools/intelmq/blob/develop/docs/User-Guid… (#1088, #1190). * the special path `"_on_error"` can be used to pass messages to different queues in case of processing errors (#1133). - `lib/harmonization`: Accept `AS` prefix for ASN values (automatically stripped). # Bots ## Collectors - `bots.collectors.mail`: - New parameters; `sent_from`: filter messages by sender, `sent_to`: filter messages by recipient - More debug logs - `bots.collectors.n6.collector_stomp`: renamed to `bots.collectors.stomp.collector` (#716) - bots.collectors.rt: - New parameter `search_requestor` to search for field Requestor. - Empty strings and `null` as value for search parameters are ignored. - Empty parameters `attachment_regex` and `url_regex` handled. - `bots.collectors.http.collector_http`: Ability to optionally use the current time in parameter `http_url`, added parameter `http_url_formatting`. - `bots.collectors.stomp.collector`: Heartbeat timeout is now logged with log level info instead of warning. - added `intelmq.bots.collectors.twitter.collector_twitter` - added `intelmq.bots.collectors.tcp.collector` that can be bound to another IntelMQ instance by a TCP output - `bots.collectors.microsoft.collector_interflow`: added for MS interflow API - Automatic ungzipping for .gz files. - added `intelmq.bots.collectors.calidog.collector_certstream` for collecting certstream data (#1120). - added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (#1096). ## Parsers - `bots.parsers.shadowserver`: - changed feednames . Please refer to it's README for the exact changes. - If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration. Previously errors like these were only logged and ignored otherwise. - add support for the feed `Accessible-Hadoop` - The Generic CSV Parser `bots.parsers.generic.parser_csv`: - It is possible to filter the data before processing them using the new parameters `filter_type` and `filter_text`. - It is possible to specify multiple columns using `|` character in parameter `columns`. - The parameter `time_format` now supports `'epoch_millis'` for seconds since the Epoch, milliseconds are supported but not used. - renamed `bots.parsers.cymru_full_bogons.parser` to `bots.parsers.cymru.parser_full_bogons`, compatibility shim will be removed in version 2.0 - added `bots.parsers.cymru.parser_cap_program` - added `intelmq.bots.parsers.zoneh.parser` for ZoneH feeds - added `intelmq.bots.parsers.sucuri.parser` - added `intelmq.bots.parsers.malwareurl.parser` - added `intelmq.bots.parsers.threatminer.parser` - added `intelmq.bots.parsers.webinspektor.parser` - added `intelmq.bots.parsers.twitter.parser` - added `intelmq.bots.parsers.microsoft.parser_ctip` * ignore the invalid IP '0.0.0.0' for the destination * fix the raw/dumped messages, did not contain the paling list previously. * use the new harmonization field `tlp` instead of `extra.tlp`. - `bots.parsers.alienvault.parser_otx`: Save TLP data in the new harmonization field `tlp`. - added `intelmq.bots.parsers.openphish.parser_commercial` - added `intelmq.bots.parsers.microsoft.parser_bingmurls` - added `intelmq.bots.parsers.calidog.parser_certstream` for parsing certstream data (#1120). - added `intelmq.bots.parsers.shodan.parser` for parsing shodan data (#1096). ## Experts - Added sieve expert for filtering and modifying events (#1083) * capable of distributing the event to appropriate named queues - `bots.experts.modify` * default rulesets: all malware name mappings have been migrated to the [Malware Name Mapping repository](https://github.com/certtools/malware_name_mapping) ruleset. See the new added contrib tool for download and conversion. * new parameter `case_sensitive` (default: True) - Added wait expert for sleeping - Added domain suffix expert to extract the TLD/Suffix from a domain name. - `bots.experts.maxmind_geoip`: New (optional) parameter `overwrite`, by default false. The current default was to overwrite! ## Outputs - `bots.outputs.file`: - String formatting can be used for file names. - New parameter `single_key` to only save one field. # Harmonization - Renamed `JSON` to `JSONDict` and added a new type `JSON`. `JSONDict` saves data internally as JSON, but acts like a dictionary. `JSON` accepts any valid JSON. - fixed regex for `protocol.transport` it previously allowed more values than it should have. - New ASN type. Like integer but checks the range. - added `destination.urlpath` and `source.urlpath` to harmonization. - New field `tlp` for tlp level specification. - New TLP type. Allows all four tlp levels, removes 'TLP:' prefix and converts to upper case. - Added new `classification.type` 'vulnerable client' - Added `(destination|source).domain_suffix` to hold the TLD/domain suffix. - New allowed value for `classification.type`: `infected system` for taxonomy `malicious code` (#1197). # Requirements - Requests is no longer listed as dependency of the core. For depending bots the requirement is noted in their REQUIREMENTS.txt file. # Documentation - Use Markdown for README again, as pypi now supports it. # Tests - Travis now correctly stops if a requirement could not be installed (#1257). - New tests for validating `etc/feeds.yaml` and `bots/BOTS` using cerberus and schemes are added (#1166). - New test for checking if `docs/Feeds.md` is up to date with `etc/feeds.yaml`. # Known bugs - `bots.experts.sieve` does not support textX (#1246). - performance degradation for extra fields (#1117). - Postgres output: support condensed JSONDicts (#1107). - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

ShadowServer Parser bot feedname
by Peter Chan 02 Aug '18

02 Aug '18

Good day, I wanted to set up a test in intelmq, with the following pipe: Fetch Mail Attachement -> Parse Shadow Server csv -> output to file. The mail and file output bots are running fine, however I am getting the following error with the shadowserver parser bot: 2018-06-05 10:54:51,880 - ShadowServer-Parser - INFO - ShadowserverParserBot initialized with id ShadowServer-Parser and intelmq 1.0.3 and python 3.5.2 (default, Nov 23 2017, 16:37:01) as process 7223. 2018-06-05 10:54:51,881 - ShadowServer-Parser - INFO - Bot is starting. 2018-06-05 10:54:51,888 - ShadowServer-Parser - ERROR - No feedname provided or feedname not in conf. 2018-06-05 10:54:51,889 - ShadowServer-Parser - INFO - Bot stopped. The feedname parameter in my runtime.conf is definitely populated. I have tried defining different feed names, based on the report name from the shadowserver website, but still getting the same error. Any idea what I am missing? I've attached my runtime.conf and pipeline.conf for reference. Unfortunately I cannot share the shadowserver email as it contains device scans in my ASN. Regards, Peter

3 7

shadowserver parser
by Ole Kristoffer Dybvik Apeland 06 Jul '18

06 Jul '18

Hi We had an issue with shadowserver data today where those feeds that only provide a download link would yield a php script. I have spoken with someone connected to the project and they are aware of the problem. Is it possible to let the shadowserver parser drop the messages in these instances? In it's current form only a warning is given (line 92 in bots/parsers/shadowserver/parser.py) even though required_fields are missing in the data. best regards Ole Kristoffer Nkom EkomCERT

2 1

Shadowserver compromised website parser -ERROR
by Tomislav Protega 02 Jul '18

02 Jul '18

Hi, I ran into this error: Shadowserver-Compromised-Website-Parser - ERROR - Could not convert shadowkey: 'http_host', value: '' via conversion function 'validate_fqdn'. More detailed log is attached. This happens when "http_host" field in the shadowserver origin report contains IP instead of domain which is not something unusual. At the end IntelMQ does produce the output data, but there's no 'source.url' field which should contain merged 'http_host' and 'url' parameters from the origin report. Regards, -- Tomislav

2 2

reverse_dns-expert exception InvalidValue
by Tomislav Protega 02 Jul '18

02 Jul '18

Hi, here's another exception I ran into. If Reverse DNS expert bot runs into IP address which doesn't have DNS PTR record in fqdn form, then it gives exception. In attachment are logs. Example: There are cases when DNS gives two records for PTR. $ nslookup 5.157.80.221 Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: 221.80.157.5.in-addr.arpa name = 5.157.80.221. 221.80.157.5.in-addr.arpa name = aliancys.peopleinc.nl. -------------------------------------------------------- $ nslookup 121.201.38.118 Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: 118.38.201.121.in-addr.arpa name = 121.201.38.118. Regards, -- Tomislav

2 1

Question on Intelmq
by LIMITHA THOMAS 28 Jun '18

28 Jun '18

Hi , Can I integrate Intelmq with Sumologic to feed in data ? If yes , how do I do it? Thanks and regards, Limitha

2 1

intelmq 1.0.5 released
by Sebastian Wagner 22 Jun '18

22 Jun '18

Dear community, I just released the next maintenance release 1.0.5 for the 1.0.x series and it could be the last one. I will also release a RC for 1.1.0 very soon. The pre-build deb and rpm packages will be available soon (in some hours). Installation instructions: https://github.com/certtools/intelmq/blob/1.0.5/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.5/docs/UPGRADING.md This is the changelog: # Core - `lib/message`: `Report()` can now create a Report instance from Event instances (#1225). - `lib/bot`: * The first word in the log line `Processed ... messages since last logging.` is now adaptible and set to `Forwarded` in the existing filtering bots (#1237). * Kills oneself again after proper shutdown if the bot is XMPP collector or output (#970). Previously these two bots needed two stop commands to get actually stopped. - `lib/utils`: log: set the name of the `py.warnings` logger to the bot name (#1184). # Bots ## Collectors - `bots.collectors.mail.collector_mail_url`: handle empty downloaded reports (#988). - `bots.collectos.file.collector_file`: handle empty files (#1244). ## Parsers - Shadowserver parser: * SSL FREAK: Remove optional column `device_serial` and add several new ones. * Fixed HTTP URL parsing for multiple feeds (#1243). - Spamhaus CERT parser: * add support for `smtpauth`, `l_spamlink`, `pop`, `imap`, `rdp`, `smb`, `iotscan`, `proxyget`, `iotmicrosoftds`, `automatedtest`, `ioturl`, `iotmirai`, `iotcmd`, `iotlogin` and `iotuser` (#1254). * fix `extra.destination.local_port` -> `extra.source.local_port`. ## Experts - `bots.experts.filter`: Pre-compile regex at bot initialization. # Tests - Ensure that the bots did process all messages (#291). # Tools - `intelmqctl`: * `intelmqctl run` has a new parameter `-l` `--loglevel` to overwrite the log level for the run (#1075). * `intelmqctl run [bot-id] mesage send` can now send report messages (#1077). - `intelmqdump`: * has now command completion for bot names, actions and queue names in interacive console. * automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (#1225). * is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (#1256). * truncated messages are no longer used/saved to the file after being shown (#1255). * now again denies recovery of dumps if the corresponding bot is running. The check was broken (#1258). * now sorts the dump by the time of the dump. Previously, the list was in random order (#1020). # Known issues no known issues Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users