- IntelMQ-dev - CERT.at

intelmq-manager release 1.0.2
by Sebastian Wagner 25 Sep '18

25 Sep '18

Dear community, Just released the second and most probably last bugfix release for the intelmq-manager 1.0 series. Installation instructions: https://github.com/certtools/intelmq-manager/blob/1.0.2/docs/INSTALL.md The built deb/rpm packages are already available. The full changelog: # Backend - Emit a more specific error message for cases like #160. - The timeout for calls to `intelmqctl` has been raised to 20s (#164). # Pages ## Configuration - Underscore is now allowed for new parameter names (#153). ## Monitor * Fix link to monitor page (#157). # Documentation - Add a FAQ and add a section about the docker issue (#160). - Add instructions for Debian 9, Ubuntu 18.04, and openSUSE Leap 42.3, 15.0 (#168). # Packaging - Include a positions file matching the default configuration of intelmq (#171). # Known issues * Missing CSRF protection (#111). * Missing copyright notices (#140). * Graph jumps around on "Add edge" bug component (#148). * new runtime parameters with _ not possible (#153). * wrong error message for new bots with existing ID (#152). * Queue size for deleted queue remains displayed (#158) best regards, Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 1.1.0 released
by Sebastian Wagner 05 Sep '18

05 Sep '18

Dear community, I just released the feature release 1.1.0 for the 1.x series the first stable release after the initial 1.0.0. This version is already in use in several setups without known issues and introduces many new features and bots. Installation instructions: https://github.com/certtools/intelmq/blob/1.1.0/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.0/docs/UPGRADING.md The deb/rpm packages will be available in some hours in the stable repository (see linked instructions). The full changelog is very long: - Support for Python 3.3 has been dropped in IntelMQ and some dependencies of it. Python 3.3 reached its end of life and Python 3.4 or newer is a hard requirement now. - The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml A tool to convert from yaml to md has been added. ### Tools - `intelmq_gen_feeds_docs` addded to bin directory, allows generating the Feeds.md documentation file from feeds.yaml - `intelmq_gen_docs` merges both `intelmq_gen_feeds_docs` and `intelmq_gen_harm_docs` in one file and automatically updates the documentation files. #### intelmqctl - `intelmqctl start` prints the bot's last error messages if the bot failed to start (#1021). - `intelmqctl start` message "is running" is printed every time. (Until now, it wasn't said when a bot was just starting.) - `intelmqctl start/stop/restart/reload/status` now has a "--group" flag which allows you to specify the group of the bots that should be influenced by the command. - `intelmqctl check` checks for defaults.conf completeness if the shipped file from the package can be found. - `intelmqctl check` shows errors for non-importable bots. - `intelmqctl list bots -q` only prints the IDs of enabled bots. - `intelmqctl list queues-and-status` prints both queues and bots statuses (so that it can be used in eg. intelmq-manager). - `intelmqctl run` parameter for showing a sent message. - `intelmqctl run` if message is sent to a non-default path, it is printed out. - `intelmqctl restart` bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226). - `intelmqctl check`: New parameter `--no-connections` to prevent the command from making connections e.g. to the redis pipeline.s - `intelmqctl list queues`: don't display named paths amongst standard queues. - The process status test failed if the PATH did not include the bot executables and the `which` command failed. Then the proccess's command line could not be compared correctly. The fix warns of this and adds a new status 'unknown' (#1297). ### Contrib - tool `feeds-config-generator` to automatically generate the collector and parser runtime and pipeline configurations. - `malware_name_mapping`: Download and convert tool for malware family name mapping has been added. - Added a systemd script which creates systemd units for bots (#953). - `contrib/cron-jobs/update-asn-data`, `contrib/cron-jobs/update-geoip-data`, `contrib/cron-jobs/update-tor-nodes`: Errors produce proper output. ### Core - lib/bot - use SIGTERM instead of SIGINT to stop bots (#981). - Bots can specify a static method `check(parameters)` which can perform individual checks specific to the bot. These functions will be called by `intelmqctl check` if the bot is configured with the given parameters - top level bot parameters (description, group, module, name) are exposed as members of the class. - The parameter `feed` for collectors is deprecated for 2.0 and has been replaced by the more consistent `name` (#1144). - bug: allow path parameter for CollectorBot class. - Handle errors better when the logger could not be initialized. - `ParserBot`: - For the csv parsing methods, `ParserBot.csv_params` is now used for all these methods. - `ParserBot.parse_csv_dict` now saves the field names in `ParserBot.csv_fieldnames`. - `ParserBot.parse_csv_dict` now saves the raw current line in `ParserBot.current_line`. - `ParserBot.recover_line_csv_dict` now uses the raw current line. - lib/message: - Subitems in fields of type `JSONDict` (see below) can be accessed directly. E.g. you can do: event['extra.foo'] = 'bar' event['extra.foo'] # gives 'bar' It is still possible to set and get the field as whole, however this may be removed or changed in the future: event['extra'] = '{"foo": "bar"}' event['extra'] # gives '{"foo": "bar"}' "Old" bots and configurations compatible with 1.0.x do still work. Also, the extra field is now properly exploded when exporting events, analogous to all other fields. The `in` operator works now for both - the old and the new - behavior. - `Message.add`: The parameter `overwrite` accepts now three different values: `True`, `False` and `None` (new). True: An existing value will be overwritten False: An existing value will not be overwritten (previously an exception has been raised when the value was given). None (default): If the value exists an `KeyExists` exception is thrown (previously the same as False). This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function. - The message class has now the possibility to return a default value for non-exisiting fields, see `Message.set_default_value`. - Message.get behaves the same like `Message.__getitem__` (#1305). - Add `RewindableFileHandle` to utils making handling of CSV files more easy (optionally) - lib/pipeline: * you may now define more than one destination queues path the bot should pass the message to, see [Pipelines](https://github.com/certtools/intelmq/blob/develop/docs/User-Guid… (#1088, #1190). * the special path `"_on_error"` can be used to pass messages to different queues in case of processing errors (#1133). - `lib/harmonization`: Accept `AS` prefix for ASN values (automatically stripped). ### Bots - Removed print statements from various bots. - Replaced various occurences of `self.logger.error()` + `self.stop()` with `raise ValueError`. #### Collectors - `bots.collectors.mail`: - New parameters; `sent_from`: filter messages by sender, `sent_to`: filter messages by recipient - More debug logs - `bots.collectors.n6.collector_stomp`: renamed to `bots.collectors.stomp.collector` (#716) - bots.collectors.rt: - New parameter `search_requestor` to search for field Requestor. - Empty strings and `null` as value for search parameters are ignored. - Empty parameters `attachment_regex` and `url_regex` handled. - `bots.collectors.http.collector_http`: Ability to optionally use the current time in parameter `http_url`, added parameter `http_url_formatting`. - `bots.collectors.stomp.collector`: Heartbeat timeout is now logged with log level info instead of warning. - added `intelmq.bots.collectors.twitter.collector_twitter` - added `intelmq.bots.collectors.tcp.collector` that can be bound to another IntelMQ instance by a TCP output - `bots.collectors.microsoft.collector_interflow`: added for MS interflow API - Automatic ungzipping for .gz files. - added `intelmq.bots.collectors.calidog.collector_certstream` for collecting certstream data (#1120). - added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (#1096). - Add proxy support. - Fix handling of parameter `countries`. #### Parsers - `bots.parsers.shadowserver`: - changed feednames . Please refer to it's README for the exact changes. - If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration. Previously errors like these were only logged and ignored otherwise. - add support for the feeds - `Accessible-Hadoop` (#1231) - `Accessible ADB` (#1285) - Remove deprecated parameter `override`, use `overwrite` instead (#1071). - The `raw` values now are exactly the input with quotes unchanged, the ParserBot methods are now used directly (#1011). - The Generic CSV Parser `bots.parsers.generic.parser_csv`: - It is possible to filter the data before processing them using the new parameters `filter_type` and `filter_text`. - It is possible to specify multiple columns using `|` character in parameter `columns`. - The parameter `time_format` now supports `'epoch_millis'` for seconds since the Epoch, milliseconds are supported but not used. - renamed `bots.parsers.cymru_full_bogons.parser` to `bots.parsers.cymru.parser_full_bogons`, compatibility shim will be removed in version 2.0 - added `bots.parsers.cymru.parser_cap_program` - added `intelmq.bots.parsers.zoneh.parser` for ZoneH feeds - added `intelmq.bots.parsers.sucuri.parser` - added `intelmq.bots.parsers.malwareurl.parser` - added `intelmq.bots.parsers.threatminer.parser` - added `intelmq.bots.parsers.webinspektor.parser` - added `intelmq.bots.parsers.twitter.parser` - added `intelmq.bots.parsers.microsoft.parser_ctip` * ignore the invalid IP '0.0.0.0' for the destination * fix the raw/dumped messages, did not contain the paling list previously. * use the new harmonization field `tlp` instead of `extra.tlp`. - `bots.parsers.alienvault.parser_otx`: Save TLP data in the new harmonization field `tlp`. - added `intelmq.bots.parsers.openphish.parser_commercial` - added `intelmq.bots.parsers.microsoft.parser_bingmurls` - added `intelmq.bots.parsers.calidog.parser_certstream` for parsing certstream data (#1120). - added `intelmq.bots.parsers.shodan.parser` for parsing shodan data (#1096). - change the classification type from 'botnet drone' to infected system' in various parses. - `intelmq.bots.parsers.spamhaus.parser_cert`: Added support for all known bot types. #### Experts - Added sieve expert for filtering and modifying events (#1083) * capable of distributing the event to appropriate named queues - `bots.experts.modify` * default rulesets: all malware name mappings have been migrated to the [Malware Name Mapping repository](https://github.com/certtools/malware_name_mapping) ruleset. See the new added contrib tool for download and conversion. * new parameter `case_sensitive` (default: True) - Added wait expert for sleeping - Added domain suffix expert to extract the TLD/Suffix from a domain name. - `bots.experts.maxmind_geoip`: New (optional) parameter `overwrite`, by default false. The current default was to overwrite! - `intelmq.bots.experts.ripencc_abuse_contact`: * Extend deprecated parameter compatibility `query_ripe_stat` until 2.0 because of a logic bug in the compatibility code, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (#1071, #1291). * Handle HTTP status code 404 for DB AS queries. * Add caching capability. - `intelmq/bots/experts/asn_lookup/update-asn-data`: Errors produce proper output on stdout/stderr. - `intelmq/bots/experts/maxmind_geoip/update-geoip-data`: Errors produce proper output on stdout/stderr. - `intelmq/bots/experts/tor_nodes/update-tor-nodes`: Errors produce proper output on stdout/stderr. #### Outputs - `bots.outputs.file`: - String formatting can be used for file names with new parameter `format_filename`. - New parameter `single_key` to only save one field. - New parameter `encoding_errors_mode` with default value `'strict'` to handle encoding errors for the files written. ### Harmonization - Renamed `JSON` to `JSONDict` and added a new type `JSON`. `JSONDict` saves data internally as JSON, but acts like a dictionary. `JSON` accepts any valid JSON. - fixed regex for `protocol.transport` it previously allowed more values than it should have. - New ASN type. Like integer but checks the range. - added `destination.urlpath` and `source.urlpath` to harmonization. - New field `tlp` for tlp level specification. - New TLP type. Allows all four tlp levels, removes 'TLP:' prefix and converts to upper case. - Added new `classification.type` 'vulnerable client' - Added `(destination|source).domain_suffix` to hold the TLD/domain suffix. - New allowed value for `classification.type`: `infected system` for taxonomy `malicious code` (#1197). ### Requirements - Requests is no longer listed as dependency of the core. For depending bots the requirement is noted in their REQUIREMENTS.txt file. ### Documentation - Use Markdown for README again, as pypi now supports it. - Developers Guide: Add instructions for pre-release testing. ### Packaging - Add logcheck configuration to the packages. - Fix packaging of bash completion script. ### Tests - Travis now correctly stops if a requirement could not be installed (#1257). - New tests for validating `etc/feeds.yaml` and `bots/BOTS` using cerberus and schemes are added (#1166). - New test for checking if `docs/Feeds.md` is up to date with `etc/feeds.yaml`. ### Known bugs - contrib: feeds-config-generator does not add feed name as parameter (#1314). - bot debugger requires configured source pipeline (#1307). - shadowserver parser: drone feed has spam events (#1271). - debug log level on python 3.7 not applied (#1269). - `bots.experts.sieve` does not support textX (#1246). - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 1.0.6 released
by Sebastian Wagner 05 Sep '18

05 Sep '18

Dear community, at the end of last weak I finalized the last planned maintenance release of the 1.0.x series, version 1.0.6. I do not intend to do any further maintenance releases for the 1.0 version, however I am open for discussion here. The branch maintenance-1.0 has all the commits of the 1.0.x series. Installation instructions: https://github.com/certtools/intelmq/blob/1.0.6/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.6/docs/UPGRADING.md The changelog: ### Bots #### Collectors - `bots.collectors.rt.collector_rt`: Log ticket id for downloaded reports. #### Parsers - `bots.parsers.shadowserver`: - if required fields do not exist in data, an exception is raised, so the line will be dumped and not further processed. - fix a bug in the parsing of column `cipher_suite` in ssl poodle reports (#1288). #### Experts - Reverse DNS Expert: ignore all invalid results and use first valid one (#1264). - `intelmq/bots/experts/tor_nodes/update-tor-nodes`: Use check.torproject.org as source as internet2.us is down (#1289). #### Outputs - `bots.output.amqptopic`: - The default exchange must not be declared (#1295). - Unencodable characters are prepended by backslashes by default. Otherwise Unicode characters can't be encoded and sent (#1296). - Gracefully close AMQP connection on shutdown of bot. ### Documentation - Bots: document redis cache parameters. - Installation documentation: Ubuntu needs universe repositories. ### Packaging - Dropped support for Ubuntu 17.10, it reached its End of Life as of 2018-07-19. ### Tests - Drop tests for Python 3.3 for the mode with all requirements, as some optional dependencies do not support Python 3.3 anymore. - `lib.test`: Add parameter `compare_raw` (default: `True`) to `assertMessageEqual`, to optionally skip the comparison of the raw field. - Add tests for RT collector. - Add tests for Shadowserver Parser: - SSL Poodle Reports. - Helper functions. ### Tools - `intelmqctl list` now sorts the output of bots and queues (#1262). - `intelmqctl`: Correctly handle the corner cases with collectors and outputs for getting/sending messages in the bot debugger (#1263). - `intelmqdump`: fix ordering of dumps in a file in runtime. All operations are applied to a sorted list (#1280). ### Contrib - `cron-jobs/update-tor-nodes`: Use check.torproject.org as source as internet2.us is down (#1289). Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

1.1.0 release candidate
by Sebastian Wagner 07 Aug '18

07 Aug '18

Hi, I just released a release candidate version for 1.1.0. If there are no big issues coming up in the next days, there won't be another release candidate and a final release in one or two weeks, depending on the feedback. Everybody is encouraged to give feedback and test this pre release. 1.0.x installations are compatible with the 1.1.0 code. Some things have been deprecated, have a look at warnings and `intelmqctl check`. The deb/rpm packages in the unstable repository will be available tomorrow or next week. This is the full current changelog: Installation instructions: https://github.com/certtools/intelmq/blob/1.1.0rc1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.0rc1/docs/UPGRADING.md - Support for Python 3.3 has been dropped, it reached its end of life. - The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml A tool to convert from yaml to md has been added. # Tools - `intelmq_gen_feeds_docs` add to bin directory, allows generating the Feeds.md documentation file from feeds.yaml - `intelmq_gen_docs` merges both `intelmq_gen_feeds_docs` and `intelmq_gen_harm_docs` in one file and automatically updates the documentation files. ## intelmqctl - `intelmqctl start` prints the bot's last error messages if the bot failed to start (#1021). - `intelmqctl start` message "is running" is printed every time. (Until now, it wasn't said when a bot was just starting.) - `intelmqctl start/stop/restart/reload/status` now have a "--group" flag which allows you to specify the group of the bots that should be influenced by the command. - `intelmqctl check` checks for defaults.conf completeness if the shipped file from the package can be found. - `intelmqctl check` shows errors for non-importable bots. - `intelmqctl list bots -q` only prints the IDs of enabled bots. - `intelmqctl list queues-and-status` prints both queues and bots statuses (so that it can be used in eg. intelmq-manager). - `intelmqctl run` parameter for showing a sent message. - `intelmqctl run` if message is sent to a non-default path, it is printed out. - `intelmqctl restart` bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226). - `intelmqctl check`: New parameter `--no-connections` to prevent the command from making connections e.g. to the redis pipeline.s # Contrib - tool `feeds-config-generator` to automatically generate the collector and parser runtime and pipeline configurations. - `malware_name_mapping`: Download and convert tool for malware family name mapping has been added. - Added a systemd script which creates systemd units for bots (#953). # Core - lib/bot - use SIGTERM instead of SIGINT to stop bots (#981). - Bots can specify a static method `check(parameters)` which can perform individual checks specific to the bot. These functions will be called by `intelmqctl check` if the bot is configured with the given parameters - top level bot parameters (description, group, module, name) are exposed as members of the class. - The parameter `feed` for collectors is deprecated for 2.0 and has been replaced by the more consistent `name` (#1144). - lib/message: - Subitems in fields of type `JSONDict` (see below) can be accessed directly. E.g. you can do: event['extra.foo'] = 'bar' event['extra.foo'] # gives 'bar' It is still possible to set and get the field as whole, however this may be removed or changed in the future: event['extra'] = '{"foo": "bar"}' event['extra'] # gives '{"foo": "bar"}' "Old" bots and configurations compatible with 1.0.x do still work. Also, the extra field is now properly exploded when exporting events, analogous to all other fields. - `Message.add`: The parameter `overwrite` accepts now three different values: `True`, `False` and `None` (new). True: An existing value will be overwritten False: An existing value will not be overwritten (previously an exception has been raised when the value was given). None (default): If the value exists an `KeyExists` exception is thrown (previously the same as False). This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function. - The message class has now the possibility to return a default value for non-exisiting fields, see `Message.set_default_value`. - Add `RewindableFileHandle` to utils making handling of CSV files more easy (optionally) - lib/pipeline: * you may now define more than one destination queues path the bot should pass the message to, see [Pipelines](https://github.com/certtools/intelmq/blob/develop/docs/User-Guid… (#1088, #1190). * the special path `"_on_error"` can be used to pass messages to different queues in case of processing errors (#1133). - `lib/harmonization`: Accept `AS` prefix for ASN values (automatically stripped). # Bots ## Collectors - `bots.collectors.mail`: - New parameters; `sent_from`: filter messages by sender, `sent_to`: filter messages by recipient - More debug logs - `bots.collectors.n6.collector_stomp`: renamed to `bots.collectors.stomp.collector` (#716) - bots.collectors.rt: - New parameter `search_requestor` to search for field Requestor. - Empty strings and `null` as value for search parameters are ignored. - Empty parameters `attachment_regex` and `url_regex` handled. - `bots.collectors.http.collector_http`: Ability to optionally use the current time in parameter `http_url`, added parameter `http_url_formatting`. - `bots.collectors.stomp.collector`: Heartbeat timeout is now logged with log level info instead of warning. - added `intelmq.bots.collectors.twitter.collector_twitter` - added `intelmq.bots.collectors.tcp.collector` that can be bound to another IntelMQ instance by a TCP output - `bots.collectors.microsoft.collector_interflow`: added for MS interflow API - Automatic ungzipping for .gz files. - added `intelmq.bots.collectors.calidog.collector_certstream` for collecting certstream data (#1120). - added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (#1096). ## Parsers - `bots.parsers.shadowserver`: - changed feednames . Please refer to it's README for the exact changes. - If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration. Previously errors like these were only logged and ignored otherwise. - add support for the feed `Accessible-Hadoop` - The Generic CSV Parser `bots.parsers.generic.parser_csv`: - It is possible to filter the data before processing them using the new parameters `filter_type` and `filter_text`. - It is possible to specify multiple columns using `|` character in parameter `columns`. - The parameter `time_format` now supports `'epoch_millis'` for seconds since the Epoch, milliseconds are supported but not used. - renamed `bots.parsers.cymru_full_bogons.parser` to `bots.parsers.cymru.parser_full_bogons`, compatibility shim will be removed in version 2.0 - added `bots.parsers.cymru.parser_cap_program` - added `intelmq.bots.parsers.zoneh.parser` for ZoneH feeds - added `intelmq.bots.parsers.sucuri.parser` - added `intelmq.bots.parsers.malwareurl.parser` - added `intelmq.bots.parsers.threatminer.parser` - added `intelmq.bots.parsers.webinspektor.parser` - added `intelmq.bots.parsers.twitter.parser` - added `intelmq.bots.parsers.microsoft.parser_ctip` * ignore the invalid IP '0.0.0.0' for the destination * fix the raw/dumped messages, did not contain the paling list previously. * use the new harmonization field `tlp` instead of `extra.tlp`. - `bots.parsers.alienvault.parser_otx`: Save TLP data in the new harmonization field `tlp`. - added `intelmq.bots.parsers.openphish.parser_commercial` - added `intelmq.bots.parsers.microsoft.parser_bingmurls` - added `intelmq.bots.parsers.calidog.parser_certstream` for parsing certstream data (#1120). - added `intelmq.bots.parsers.shodan.parser` for parsing shodan data (#1096). ## Experts - Added sieve expert for filtering and modifying events (#1083) * capable of distributing the event to appropriate named queues - `bots.experts.modify` * default rulesets: all malware name mappings have been migrated to the [Malware Name Mapping repository](https://github.com/certtools/malware_name_mapping) ruleset. See the new added contrib tool for download and conversion. * new parameter `case_sensitive` (default: True) - Added wait expert for sleeping - Added domain suffix expert to extract the TLD/Suffix from a domain name. - `bots.experts.maxmind_geoip`: New (optional) parameter `overwrite`, by default false. The current default was to overwrite! ## Outputs - `bots.outputs.file`: - String formatting can be used for file names. - New parameter `single_key` to only save one field. # Harmonization - Renamed `JSON` to `JSONDict` and added a new type `JSON`. `JSONDict` saves data internally as JSON, but acts like a dictionary. `JSON` accepts any valid JSON. - fixed regex for `protocol.transport` it previously allowed more values than it should have. - New ASN type. Like integer but checks the range. - added `destination.urlpath` and `source.urlpath` to harmonization. - New field `tlp` for tlp level specification. - New TLP type. Allows all four tlp levels, removes 'TLP:' prefix and converts to upper case. - Added new `classification.type` 'vulnerable client' - Added `(destination|source).domain_suffix` to hold the TLD/domain suffix. - New allowed value for `classification.type`: `infected system` for taxonomy `malicious code` (#1197). # Requirements - Requests is no longer listed as dependency of the core. For depending bots the requirement is noted in their REQUIREMENTS.txt file. # Documentation - Use Markdown for README again, as pypi now supports it. # Tests - Travis now correctly stops if a requirement could not be installed (#1257). - New tests for validating `etc/feeds.yaml` and `bots/BOTS` using cerberus and schemes are added (#1166). - New test for checking if `docs/Feeds.md` is up to date with `etc/feeds.yaml`. # Known bugs - `bots.experts.sieve` does not support textX (#1246). - performance degradation for extra fields (#1117). - Postgres output: support condensed JSONDicts (#1107). - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

rsync collector
by Ondřej Schindler 07 Aug '18

07 Aug '18

Hi, i write these email, because i have to simply question, please. I want to add rsync collector to intelmq. Should i modify file collector (it would be max 5 lines of code) or should i create new collector? Best regards Ondřej Schindler

2 2

OSError: [Errno 16] Resource busy: '/opt/intelmq/var/log' by update the intelmq
by Salehi Ghamsari, Majid 01 Aug '18

01 Aug '18

Hi, i have strange problem to compile my intelmq dev enviroment. I have stopped intelmq and try to compile, but i get OSError: [Errno 16] Resource busy: '/opt/intelmq/var/log' Has anyone ever had that? Regards Majid git clone https://github.com/certtools/intelmq.git /opt/dev_intelmq cd opt/dev_intelmq/ /opt/dev_intelmq # pip3 install -e . Obtaining file:///opt/dev_intelmq Requirement already satisfied: dnspython>=1.11.1 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: psutil>=1.2.1 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: python-dateutil>=2.5 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: python-termstyle>=0.1.10 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: pytz>=2012c in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: redis>=2.10 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: six>=1.5 in /usr/lib/python3.5/site-packages (from python-dateutil>=2.5->intelmq==1.2.0a1) Requirement already satisfied: setuptools in /usr/lib/python3.5/site-packages (from python-termstyle>=0.1.10->intelmq==1.2.0a1) Installing collected packages: intelmq Found existing installation: intelmq 1.0.4 Uninstalling intelmq-1.0.4: Exception: Traceback (most recent call last): File "/usr/lib/python3.5/shutil.py", line 538, in move os.rename(src, real_dst) OSError: [Errno 16] Resource busy: '/opt/intelmq/var/log' -> '/tmp/pip-6vus5dcm-uninstall/opt/intelmq/var/log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/usr/lib/python3.5/site-packages/pip/commands/install.py", line 342, in run prefix=options.prefix_path, File "/usr/lib/python3.5/site-packages/pip/req/req_set.py", line 778, in install requirement.uninstall(auto_confirm=True) File "/usr/lib/python3.5/site-packages/pip/req/req_install.py", line 754, in uninstall paths_to_remove.remove(auto_confirm) File "/usr/lib/python3.5/site-packages/pip/req/req_uninstall.py", line 115, in remove renames(path, new_path) File "/usr/lib/python3.5/site-packages/pip/utils/__init__.py", line 267, in renames shutil.move(old, new) File "/usr/lib/python3.5/shutil.py", line 550, in move rmtree(src) File "/usr/lib/python3.5/shutil.py", line 478, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib/python3.5/shutil.py", line 476, in rmtree os.rmdir(path) OSError: [Errno 16] Resource busy: '/opt/intelmq/var/log' You are using pip version 9.0.1, however version 10.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. /opt/dev_intelmq #

2 2

using the intelmq twitter bot
by Salehi Ghamsari, Majid 17 Jul '18

17 Jul '18

Hi there, I try to get the CVEs from this twitter page https://twitter.com/CVEnew but i get an exception that “that page does not exist” >target_timelines: screen_names of twitter accounts to be followed What should I put for target_timelines parameter? I do not know what I am missing. In the documentation we have to put for each bot parameters some examples input for better understanding the parameters. https://github.com/certtools/intelmq/blob/develop/docs/Bots.md#twitter Thank you for your help Majid Salehi Ghamsari Digital Public Services (DPS) Fraunhofer Institute for Open Communication Systems Kaiserin-Augusta-Allee 31, 10589 Berlin, Germany Email: majid.salehi.ghamsari(a)fokus.fraunhofer.de<mailto:majid.salehi.ghamsari@fokus.fraunhofer.de> Office: +49 (0) 30 3463 7118 Skype: salehimajid WEB: fokus.fraunhofer.de "cve-twitter-collector": { "parameters": { "feed": "Twitter", "target_timelines": "https://twitter.com/CVEnew ", "tweet_count": 10, "timelimit": 60000, "include_rts": false, "exclude_replies": true, "follow_urls": "", "consumer_key": "myconsumer_key ", "consumer_secret": "myconsumer_secret ", "access_token_key": "myaccess_token_key ", "access_token_secret": "myaccess_token_secret " }, "name": "Twitter", "group": "Collector", "module": "intelmq.bots.collectors.twitter.collector_twitter", "description": "Collects tweets from target_timelines. Maximum number of tweet_count from each timeline and up to timelimit back in time.", "enabled": true, "run_mode": "continuous" } Traceback (most recent call last): File "/opt/dev_intelmq/intelmq/lib/bot.py", line 167, in start self.process() File "/opt/dev_intelmq/intelmq/bots/collectors/twitter/collector_twitter.py", line 94, in process exclude_replies=self.exclude_replies) File "/usr/local/lib/python3.6/dist-packages/twitter/api.py", line 810, in GetUserTimeline data = self._ParseAndCheckTwitter(resp.content.decode('utf-8')) File "/usr/local/lib/python3.6/dist-packages/twitter/api.py", line 4890, in _ParseAndCheckTwitter self._CheckForTwitterError(data) File "/usr/local/lib/python3.6/dist-packages/twitter/api.py", line 4910, in _CheckForTwitterError raise TwitterError(data['errors']) twitter.error.TwitterError: [{'code': 34, 'message': 'Sorry, that page does not exist.'}]

2 2

changing bot runtime parameter
by Salehi Ghamsari, Majid 11 Jul '18

11 Jul '18

Hello there, I'm trying to develop a simple pipeline with three bots. test-file-collector---> test-message-parser---> test-tcp-output I would like to put or change the IP runtime parameter of the TCP output bot "test-tcp-output" depends on Message that I get from file-collector in the runtime. What is the best method to do this in intelmq? Thank you for your help. Majid

3 4

testing tcpcollecor bot 1.1.0rc1
by Salehi Ghamsari, Majid 05 Jul '18

05 Jul '18

Hi, i try to test the tcpcollector bot, on sender site I send this message { "feed.accuracy":100.0, "feed.name":"FileCollector", "feed.url":"file://localhost/opt/intelmq/var/lib/bots/file-input/event.json", "time.observation":"2018-07-05T08:25:51+00:00", "extra.info":"TC_IMQ_Event_001", "extra.threat_level_id":"1", "raw":"eyJFdmVudCI6IHsiZGlzdHJpYnV0aW9uIjogIjEiLCAiaW5mbyI6ICJUQ19JTVFfRXZlbnRfMDAxIiwgInNoYXJpbmdfZ3JvdXBfaWQiOiAiMCIsICJ0aHJlYXRfbGV2ZWxfaWQiOiAiMSJ9fQ==" } On the other site the TcpColltector bot failed with this message. Bot has found a problem. Traceback (most recent call last): File "/opt/dev_intelmq/intelmq/lib/bot.py", line 167, in start self.process() File "/opt/dev_intelmq/intelmq/bots/collectors/tcp/collector.py", line 47, in process self.send_message(report) File "/opt/dev_intelmq/intelmq/lib/bot.py", line 816, in send_message super(CollectorBot, self).send_message(*messages, path=path) File "/opt/dev_intelmq/intelmq/lib/bot.py", line 797, in __add_report_fields report.add("feed.name", self.parameters.name) AttributeError: 'Parameters' object has no attribute 'name' What do I missing? Regards Majid

2 3

intelmq 1.0.5 released
by Sebastian Wagner 22 Jun '18

22 Jun '18

Dear community, I just released the next maintenance release 1.0.5 for the 1.0.x series and it could be the last one. I will also release a RC for 1.1.0 very soon. The pre-build deb and rpm packages will be available soon (in some hours). Installation instructions: https://github.com/certtools/intelmq/blob/1.0.5/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.5/docs/UPGRADING.md This is the changelog: # Core - `lib/message`: `Report()` can now create a Report instance from Event instances (#1225). - `lib/bot`: * The first word in the log line `Processed ... messages since last logging.` is now adaptible and set to `Forwarded` in the existing filtering bots (#1237). * Kills oneself again after proper shutdown if the bot is XMPP collector or output (#970). Previously these two bots needed two stop commands to get actually stopped. - `lib/utils`: log: set the name of the `py.warnings` logger to the bot name (#1184). # Bots ## Collectors - `bots.collectors.mail.collector_mail_url`: handle empty downloaded reports (#988). - `bots.collectos.file.collector_file`: handle empty files (#1244). ## Parsers - Shadowserver parser: * SSL FREAK: Remove optional column `device_serial` and add several new ones. * Fixed HTTP URL parsing for multiple feeds (#1243). - Spamhaus CERT parser: * add support for `smtpauth`, `l_spamlink`, `pop`, `imap`, `rdp`, `smb`, `iotscan`, `proxyget`, `iotmicrosoftds`, `automatedtest`, `ioturl`, `iotmirai`, `iotcmd`, `iotlogin` and `iotuser` (#1254). * fix `extra.destination.local_port` -> `extra.source.local_port`. ## Experts - `bots.experts.filter`: Pre-compile regex at bot initialization. # Tests - Ensure that the bots did process all messages (#291). # Tools - `intelmqctl`: * `intelmqctl run` has a new parameter `-l` `--loglevel` to overwrite the log level for the run (#1075). * `intelmqctl run [bot-id] mesage send` can now send report messages (#1077). - `intelmqdump`: * has now command completion for bot names, actions and queue names in interacive console. * automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (#1225). * is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (#1256). * truncated messages are no longer used/saved to the file after being shown (#1255). * now again denies recovery of dumps if the corresponding bot is running. The check was broken (#1258). * now sorts the dump by the time of the dump. Previously, the list was in random order (#1020). # Known issues no known issues Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

New OS versions as build targets for native packages
by Sebastian Wagner 07 May '18

07 May '18

Hi all, Some of the recent changes of the build targets for deb/rpm packages: New targets: * Ubuntu 18.04 * Fedora 27 * openSUSE Leap 15.0 Removed targets: * Ubuntu 17.04/17.10 * Fedora 25 * openSUSE 42.2 So this is the full list of supported OSs for native packages currently: * CentOS 7 * RHEL 7 * Debian 8 and 9 * Fedora 26, 27, 28 and Rawhide * openSUSE Leap 42.3, 15.0 and Tumbleweed * Ubuntu 16.04, 17.10 and 18.04 The unstable repository additionally has: * Debian Testing * Fedora Rawhide Docs are here: https://github.com/certtools/intelmq/blob/maintenance/docs/INSTALL.md#nativ… Please let me know your feedback any time! Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq-manager release 1.0.0 & 1.0.1
by Sebastian Wagner 23 Apr '18

23 Apr '18

Dear community, I just released a new stable release 1.0.0 and maintenance release 1.0.1 of intelmq-manager. Thanks to Edvard Reijthar, the project is very active again and many changes can be expected for the next versions too! Version 1.0.0 fixes some bugs and adds an interface for `intelmqctl check`. 1.0.1 fixes the version number in the code (shown in the about page). Installation instructions: https://github.com/certtools/intelmq-manager/blob/1.0.1/docs/INSTALL.md The version is compatible with intelmq >= 1.0.3 Combined changelog: ### Backend * Set content type correctly for JSON data in configuration loading (#112) * Fix version number. ### Pages * All pages are now deliverd by php, reducing the amount of duplicate code drastically. #### Landing page * Added a new block for the new check page, changed the formatting a bit. #### Configuration * Fixed handling of special parameter `run_mode` (#150) * Intelmqctl controller may be set via an env variable `INTELMQ_MANGER_CONTROLER_CMD` #### Check * Added, showing the output of `intelmqctl check` (#118). ### Documentation * Note on header Content-Security-Policy (#113) * Note on security considerations in Readme to avoid misunderstandings * Remove compatibility warning from README ### Third-party libraries * reverted update jQuery to 3.2.1 * reverted update metisMenu to 2.7.0 ### Licenses * Licenses of used and included software is now inventarized and properly declared in LICENSES/ (#134) ### Packaging * fix packaging of positions.conf file for deb-packages (#133). ### Known issues * Missing CSRF protection (#111). * Missing copyright notices (#140). * Graph jumps around on "Add edge" bug component (#148). * new runtime parameters with _ not possible (#153). * wrong error message for new bots with existing ID (#152). Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq release 1.0.4
by Sebastian Wagner 20 Apr '18

20 Apr '18

Dear community, I just released a new maintenance release 1.0.4 of intelmq. It only fixes bugs in the 1.0.x series and may be the last version of the 1.0.x series. Installation instructions: https://github.com/certtools/intelmq/blob/1.0.4/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.4/docs/UPGRADING.md The changes are: - make code style compatible to pycodestyle 2.4.0 - fixed permissions of some files (they were executable but shouldn't be) ### Core - lib/harmonization: * FQDN validation now handles None correctly (raised an Exception). * Fixed several sanitize() methods, the generic sanitation method were called by is_valid, not the sanitize methods (#1219). ### Harmonization ### Bots * Use the new pypi website at https://pypi.org/ everywhere. #### Parsers - Shadowserver parser: * The fields `url` and `http_url` now handle HTTP URL paths and HTTP requests for all feeds (#1204). * The conversion function `validate_fqdn` now handles empty strings correctly. * Feed 'drone (hadoop)': * Correct validation of field `cc_dns`, will now only be added as `destination.fqdn` if correct FQDN, otherwise ignored. Previously this field could be saved in extra containing an IP address. * Adding more mappings for added columns. * A lot of newly added fields and fixed conversions. * Add newly added columns of `Ssl-Scan` feed to parser - Spamhaus CERT parser: * fix parsing and classification for bot names 'openrelay', 'iotrdp', 'sshauth', 'telnetauth', 'iotcmd', 'iotuser', 'wpscanner', 'w_wplogin', 'iotscan' see the NEWS file - Postgresql section - for all changes. - CleanMX phishing parser: handle FQDNs in IP column (#1162). #### Experts - `bots.experts.ripencc_abuse_contact`: Add existing parameter `mode` to BOTS file. ### Tools - intelmqctl check: Fixed and extended message for 'run_mode' check. - `intelmqctl start` botnet. When using `--type json`, no non-json information about wrong bots are output because that would confuse eg. intelmq-manager ### Tests - lib/bot: No dumps will be written during tests (#934). - lib/test: Expand regular expression on python version to match pre-releases (debian testing). ### Packaging * Static data is now included in source tarballs, development files are excluded Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Classification of malware events
by Thomas Hungenberg 15 Mar '18

15 Mar '18

The current classification scheme for malware events in shadowserver/parser/config.py is: 'constant_fields': { 'classification.taxonomy': 'malicious code', 'classification.type': 'botnet drone', 'classification.identifier': 'botnet', }, The modify expert (if used) overwrites the classification.identifier with a malware name (either a "harmonized" name or the value of malware.name as default). Last year, we discussed dropping the term "botnet (drone)" and replace it by "infected system" (as not all malware infected systems are necessarily part of a botnet). The config.py in branch develop currently looks like: 'classification.taxonomy': 'malicious code', 'classification.type': 'botnet drone', 'classification.identifier': 'infected system', However, my intention was to set the *type* to 'infected system' and not the *identifier* (which will be overwritten by the modify expert). So I'd like to propose to change the classification scheme as follows: 'classification.taxonomy': 'malicious code', 'classification.type': 'infected system', 'classification.identifier': 'malware', # default name, will be overwritten by modify expert So the final classification of an event will look like: 'classification.taxonomy': 'malicious code', 'classification.type': 'infected system', 'classification.identifier': 'ramnit', Thoughts? Objections? - Thomas CERT-Bund Incident Response & Malware Analysis Team

3 9

IntelMQ 1.0.3 released
by Sebastian Wagner 05 Feb '18

05 Feb '18

Dear community I have just release a new bugfix release of IntelMQ. Installation instructions: https://github.com/certtools/intelmq/blob/1.0.3/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.3/docs/UPGRADING.md The released is published on PyPI, Github and the OpenBuildService (for rpm/deb packages). If you installed intelmq with a package manager, the new released will be installed automatically. Full changelog: ### Contrib * logrotate: use sudo for postrotate script * cron-jobs: use the scripts in the bots' directories and link them (#1056, #1142) ### Core - `lib.harmonization`: Handle idna encoding error in FQDN sanitation (#1175, #1176). - `lib.bot`: - Bots stop when redis gives the error "OOM command not allowed when used memory > 'maxmemory'." (#1138). - warnings of bots are catched by the logger (#1074, #1113). - Fixed exitcodes 0 for graceful shutdowns . - better handling of problems with pipeline and especially it's initialization (#1178). - All parsers using `ParserBot`'s methods now log the sum of successfully parsed and failed lines at the end of each run (#1161). ### Harmonization - Rule for harmonization keys is enforced (#1104, #1141). - New allowed values for `classification.type`: `tor` & `leak` (see n6 parser below ). ### Bots #### Collectors - `bots.collectors.mail.collector_mail_attach`: Support attachment file parsing for imbox versions newer than 0.9.5 (#1134). - `bots.outputs.smtp.output`: Fix STARTTLS, threw an exception (#1152, #1153). #### Parsers - All CSV parsers ignore NULL-bytes now, because the csv-library cannot handle it (#967, #1114). - `bots.experts.modify` default ruleset: changed conficker rule to catch more spellings. - `bots.parsers.shadowserver.parser`: Add Accessible Cisco Smart Install (#1122). - `bots.parsers.cleanmx.parser`: Handle new columns `first` and `last`, rewritten for XML feed. See NEWS.md for upgrade instructions (#1131, #1136, #1163). - `bots.parsers.n6.parser`: Fix classification mappings. See NEWS file for changes values (#738, #1127). ### Documentation - `Release.md` add release procedure documentation - `Bots.md`: fix example configuration for modify expert ### Tools - intelmqctl now exits with exit codes > 0 when errors happened or the operation was not successful. Also, the status operation exits with 1, if bots are stopped, but enabled. (#977, #1143) - `intelmctl check` checks for valid `run_mode` in runtime configuration (#1140). ### Tests - `tests.lib.test_pipeline`: Redis tests clear all queues before and after tests (#1086). - Repaired debian package build on travis (#1169). - Warnings are not allowed by default, an allowed count can be specified (#1129). - `tests.bots.experts.cymru_whois/abusix`: Skipped on travis because of ongoing problems. ### Packaging * cron jobs: fix paths of executables ### Known issues - `bots.collectors/outputs.xmpp` must be killed two times (#970). - When running bots with `intelmqctl run [bot-id]` the log level is always INFO (#1075). - `intelmqctl run [bot-id] message send [msg]` does only support Events, not Reports (#1077). - `python3 setup.py sdist` does not include static files in the resulting tarballs (#1146). - `bots.parsers.cleanmx.parser`: The cleanMX feed may have FQDNs as IPs in rare cases, such lines are dumped (#1162). Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

handling of time frames
by Sebastian Wagner 14 Dec '17

14 Dec '17

Dear list, in pull request #944 (netlab 360 enh [0]) by navtej an issue came up which can't be solved trivially: The feed Netlab 360 DGA[1] - which is already included in intelmq - provides a validity time frame for each domain. Most of those (~90%) end in 2030 while the start date is the current day at 00:00. So both start and end time are artificial. And the source claims the event is valid in the future, which is a very odd. And does it actually make sense to forward this kind of information? Also, we can't really handle this time information using the current harmonization. One idea would be to set time.source to time.observation if the time.source is in the future. So time.source <= time.observation does always apply. What do you think? Sebastian [0]: https://github.com/certtools/intelmq/pull/944 [1]: http://data.netlab.360.com/feeds/dga/dga.txt - attention, quite big! The domains at the beginning have a very near end date. -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 3

Regarding rpm packages for RHEL/CentOS
by C. L. Martinez 14 Nov '17

14 Nov '17

Hi all, Recently, I have installed IntelMQ in a CentOS 7.4 host (fully patched) and I see some "errors" in official IntelMQ's rpm packages installed from http://download.opensuse.org/repositories/home:/sebix:/intelmq/CentOS_7/. a/ /etc/cron.d/intelmq-update-data. Content is: # /etc/cron.d/intelmq-update-data: crontab fragment for intelmq # This updates the data files used by some expert bots. # # m h dom mon dow command # Update data for tor_nodes bot: 11 0 * * * intelmq /usr/bin/update-tor-nodes /var/lib/intelmq/bots/tor_nodes/tor_nodes.dat # Update data for maxmind_geoip bot: 17 0 * * * intelmq /usr/bin/update-geoip-data /var/lib/intelmq/bots/maxmind_geoip/GeoLite2-City.mmdb # Update data for asn_lookup bot: 23 0 * * * intelmq /usr/bin/update-asn-data /var/lib/intelmq/bots/asn_lookup/ipasn.dat # Update data for the RIPE DB abuse_c offline contact lookup 25 6 * * * intelmq /usr/bin/update-ripencc_abuse_contact_offline /var/lib/intelmq/bots/ripencc_abuse_contact_offline/ Where are these scripts: update-tor-nodes, update-geoip-data, update-asn-data and update-ripencc_abuse_contact_offline? They don't exist in my system. But exists intelmq-update-asn-data, intelmq-update-geoip-data and intelmq-update-tor-nodes (not ripe). b/ /etc/logrotate.d/intelmq. Content is: compress delaycompress copytruncate create 640 intelmq intelmq /var/log/intelmq/*.log { su intelmq intelmq daily maxsize 10M rotate 60 notifempty sharedscripts postrotate /usr/bin/intelmqctl reload --quiet endscript } /var/lib/intelmq/bots/file-output/*.txt { su intelmq intelmq daily maxsize 10M rotate 60 notifempty sharedscripts postrotate /usr/bin/intelmqctl reload file-output --quiet endscript } ... but returns the following email error: From root(a)cosintelmq.mydomain.com Mon Nov 13 08:29:04 2017 Return-Path: <root(a)cosintelmq.mydomain.com> X-Original-To: root Delivered-To: root(a)cosintelmq.mydomain.com From: Anacron <root(a)cosintelmq.mydomain.com> To: root(a)cosintelmq.mydomain.com Content-Type: text/plain; charset="UTF-8" Subject: Anacron job 'cron.daily' on cosintelmq.mydomain.com Date: Mon, 13 Nov 2017 08:29:04 +0000 (UTC) Status: R /etc/cron.daily/logrotate: intelmqctl: Running intelmqctl as root is highly discouraged! usage: intelmqctl [-h] [-v] [--type {text,json}] [--quiet] {list,check,clear,log,run,help,start,stop,restart,reload,status,enable,disable} ... intelmqctl: error: unrecognized arguments: --quiet error: error running shared postrotate script for '/var/log/intelmq/*.log ' Maybe is it more correct to do this: - /usr/bin/intelmqctl reload --quiet + su -m intelmq -c ' /usr/bin/intelmqctl reload --quiet' - /usr/bin/intelmqctl reload file-output --quiet + su -m intelmq -c '/usr/bin/intelmqctl reload file-output --quiet' ?? Thanks.

2 2

Data Harmonization - Fields with multiple values
by Knight, Alexander 13 Nov '17

13 Nov '17

Hi All, I am currently in the process of deciding whether ANZ should incorporate IntelMQ into its Threat Intelligence ingestion and sharing platform. At the Deepsec conference Sebastian mentioned updating the harmonization to allow for fields with multiple values. Has this issue been progressed at all? We will require multiple values for some fields in our events, and I was considering adding this functionality (perhaps in a hacky way) to my own fork, but I would like an update on the progress on the work on the master before doing so. Regards, Alex Knight | ANZ | ISO | Cyber Security Engineering Level 8, 55 Collins Street, Melbourne 3000 Phone: +61 386 545 888 | www.anz.com<http://www.anz.com/> "This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."

4 6

Bugfix release 1.0.2
by Sebastian Wagner 09 Nov '17

09 Nov '17

Dear community, I just pushed the version 1.0.2 to pypi and the build servers. Installation documentation: https://github.com/certtools/intelmq/blob/1.0.0/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md ### Core - `lib.message.add`: parameter force has finally been removed, should have been gone in 1.0.0.rc1 already ### Bots - `collectors.mail.collector_mail_url`: Fix bug which prevented marking emails seen due to disconnects from server (#852). - `parsers.spamhaus.parser_cert`: Handle/ignore 'AS?' in feed (#1111) ### Packaging - The following changes have been in effect for the built packages already since version 1.0.0 - Support building for more distributions, now supported: CentOS 7, Debian 8 and 9, Fedora 25 and 26, RHEL 7, openSUSE Leap 42.2 and 42.3 and Tumbleweed, Ubuntu 14.04 and 16.04 - Use LSB-paths for created packages (/etc/intelmq/, /var/lib/intelmq/, /run/intelmq/) (#470). Does does not affect installations with setuptools/pip. - Change the debian package format from native to quilt - Fix problems in postint and postrm scripts - Use systemd-tmpfile for creation of /run/intelmq/ ### Documentation - Add disclaimer on maxmind database in bot documentation and code and the cron-job (#1110) Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Twitter bot a proposal
by Vaclav Bruzek 18 Oct '17

18 Oct '17

Hi, recently I've discovered that there are a lot of security analysts actively participating on Twitter. By participating I mean that they are posting quite interesting data (@illegalFawn for example) and i thought that even if the the amount of data being posted there is not that great it could provide an interesting source of iocs, which could take traditional feeds a lot of time to publish. For this a played a bit with the Twitter official rest api and produced a demo which I would like to get your feedback on it and what you think could be improved. The code can be found here: https://codeshare.io/aVKXq9. The bot so far works like this: except for the necessary parameters for twitter api it requires two lists of users, one represents accounts which timeline will be processed (this is the feed-like behaviour) the other list represents the users which mark the interesting tweets (presumably "owners" of the bot) that should be downloaded the "mark" here means like. This behaviour allows for automatic collection of data from accounts like I've posted on the beginning, which post feed-like information and a manual selection of interesting tweets from accounts which post "various" posts. The bot gets tweets in bulk, that means that it gets all the tweets and liked tweets and passes them on in concatenated report. I've consulted this bot with Sebastian Wagner and he pointed out some weaknesses of this way mainly data and feed classification. A better approach is probably by creating a report for each individual which eases the classification (which could be now done using hashtags if present). The bot lacks a lot of comments and documentation so ask away if some features are not clear. Again, I'd like to get your feedback and opinions on this since I think it could be an interesting addition to intelmq ecosystem. Sincerely, Václav Brůžek

2 1

contains-operator and hierarchies of messages
by Sebastian Wagner 10 Oct '17

10 Oct '17

Hi, All I wanted to have in the first place is an easy way to find out if an event has any extra.* field. I.e. in python: 'extra' in event. But this does not work, because there is no such field, only extra.foo. So, what about changing the __contains__ method of the Message class? It could check for sub-fields if necessary. Using the (wrong) algorithm to check if some existing key starts with (in this example) `extra` it turns out this also leads to the convenient situation that you now can't add e.g. `extra.error.message` if `extra.error` was previously defined. And: the alienvault otx parser actually did this with extra.pulse This is good, but also prevents adding `extra.errormessage` if `extra.error` exists. The correct algorithm is to check if some field starts with `extra.`. But using this assumption does not prevent hierarchy conflicts. IMO, we want both: 1) a simple way to check if any field of a group is (not) present 2) prevention to add keys which conflict with others in the hierarchy, i.e. `extra.error.message` must not be added if `extra.error` is present. We could adapt the contains-operator that e.g. a search for `extra.` means: "Is there any field in the extra.* domain?". It's an invalid key name anyway[1]. And for adding new fields: check if there is some value in a higher hierarchy, if so, fail. Overwriting is not allowed in this case. Any thoughts on this? Sebastian P.S.: In the long rung we can probably save the fields hierarchically internally too, would avoid some of these shortcomings. We also had problems with the hierarchy of malware.hash in the past: https://github.com/certtools/intelmq/issues/732 [1]: actually it is allowed currently, but it shouldn't and does not make sense, see https://github.com/certtools/intelmq/issues/1104 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq-manager release 0.3 and 0.3.1
by Sebastian Wagner 26 Sep '17

26 Sep '17

Dear users and contributors, Yesterday I release version 0.3 and today 0.3.1 (containing a fix for a bug preventing the saving of files). This release contains a lot of exciting usability fixes and enhancements. See the changelog below for a full list. We are getting close to a stable release now! Please refer to the installation docs. Deb and rpm packages are available. Note that for the deb-packages, you need to set group permissions on the configuration files first. This is the changelog of 0.3: * Partly support for CentOS/RHEL 7 (#55, #103) * Note on security considerations in Readme to avoid misunderstandings * Show versions of intelmq and intelmq manager on about page * Update vis.js to current version ### Configuration * interface for defaults.conf (#45) * drag&drop (#105, #41) * fix #96 * save buttons starts blinking after changes (#41) * Allow redrawing of botnet on demand * Save/load position of bots in/from /opt/intelmq/etc/manager/positions.conf File needs to be writeable * parameters from defaults are shown for new bots (#107) * parameters are grouped by type: generic, runtime, defaults * better feedback on errors with backend (#69, #99) * pressing ESC in forms equals to pressing the cancel button * Edit node window is now much bigger * pressing enter in 'add key' window equals to pressing ok button ### Management * Reload and restart have been added as actions on bots and the whole botnet (#114) * A click on the bot name opens the monitor page of the bot ### Monitor * clearing queues is possible in general and specific view for all queues (#54) ### Backend * Fix regex checks on bot ids and log line number in controller, they have not been effective * fix overflow in extended message box (#49) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

1.0.1 bugfix release
by Sebastian Wagner 30 Aug '17

30 Aug '17

Dear community, I just released the bug fix release IntelMQ 1.0.1. The existing bugs which have not been fixed in time have been moved to the 1.0.2 milestone (10 bugs). For upgrade instructions look at the documentation: https://github.com/certtools/intelmq/blob/develop/docs/UPGRADING.md Updated packages have been built and are available in the repositories. From the changelog: ### Documentation - Feeds: use more https:// URLs - minor fixes ### Bots - bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for rest.db.ripe.net - bots/outputs/file/output.py: properly close the file handle on shutdown ### Core - lib/bot: Bots will now log the used intelmq version at startup ### Tools - intelmqctl: To check the status of a bot, the comandline of the running process is compared to the actual executable of the bot. Otherwise unrelated programs with the same PID are detected as running bot. - intelmqctl: the "enable", "disable", "check", "clear" commands now support the JSON output Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

Published release candidate 1.0.1
by Sebastian Wagner 23 Aug '17

23 Aug '17

I just published the release candidate for the next bugfix release 1.0.1. You can expect the final release next week / end of august. Changelog: ### Documentation - Feeds: use more https:// URLs - minor fixes ### Bots - bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for rest.db.ripe.net - bots/outputs/file/output.py: properly close the file handle on shutdown ### Core - lib/bot: Bots will now log the used intelmq version at startup ### Tools - intelmqctl: To check the status of a bot, the comandline of the running process is compared to the actual executable of the bot. Otherwise unrelated programs with the same PID are detected as running bot. - intelmqctl: enable, disable, check, clear now support the JSON output -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Harmonization configuration
by Vaclav Bruzek 21 Aug '17

21 Aug '17

Hi, I recently tried playing with my custom fields in harmonization.conf which would be needed for a potential bot. However I ran into some problems with that during testing, following is the displayed error: "AssertionError: Regex matched: 'ERROR' matches '(ERROR.*?){1}' Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 145, in start\n self.process() File "/untitled/test_bot.py", line 34, in process event.add("test.field", 50) File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/message.py", line 188, in add raise exceptions.InvalidKey(key) intelmq.lib.exceptions.InvalidKey: invalid key \'test.field\' Configuration in harmonization.conf is as follows for event and for report: "'test.field": {"description": "test value", "type": "Integer"}, I checked whether I am using the version in /opt/intelmq/etc/ and everything seemed to be ok. I couldn't figure out what is wrong because everything seems to be in order. I am using the 1.0.0 version of intelmq, on elementary OS 0.4.1. -- Sincerely, Václav Brůžek

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev