IntelMQ-dev July 2018

intelmq-dev@lists.cert.at

3 participants
5 discussions

1.1.0 release candidate
by Sebastian Wagner 07 Aug '18

07 Aug '18

Hi, I just released a release candidate version for 1.1.0. If there are no big issues coming up in the next days, there won't be another release candidate and a final release in one or two weeks, depending on the feedback. Everybody is encouraged to give feedback and test this pre release. 1.0.x installations are compatible with the 1.1.0 code. Some things have been deprecated, have a look at warnings and `intelmqctl check`. The deb/rpm packages in the unstable repository will be available tomorrow or next week. This is the full current changelog: Installation instructions: https://github.com/certtools/intelmq/blob/1.1.0rc1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.0rc1/docs/UPGRADING.md - Support for Python 3.3 has been dropped, it reached its end of life. - The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml A tool to convert from yaml to md has been added. # Tools - `intelmq_gen_feeds_docs` add to bin directory, allows generating the Feeds.md documentation file from feeds.yaml - `intelmq_gen_docs` merges both `intelmq_gen_feeds_docs` and `intelmq_gen_harm_docs` in one file and automatically updates the documentation files. ## intelmqctl - `intelmqctl start` prints the bot's last error messages if the bot failed to start (#1021). - `intelmqctl start` message "is running" is printed every time. (Until now, it wasn't said when a bot was just starting.) - `intelmqctl start/stop/restart/reload/status` now have a "--group" flag which allows you to specify the group of the bots that should be influenced by the command. - `intelmqctl check` checks for defaults.conf completeness if the shipped file from the package can be found. - `intelmqctl check` shows errors for non-importable bots. - `intelmqctl list bots -q` only prints the IDs of enabled bots. - `intelmqctl list queues-and-status` prints both queues and bots statuses (so that it can be used in eg. intelmq-manager). - `intelmqctl run` parameter for showing a sent message. - `intelmqctl run` if message is sent to a non-default path, it is printed out. - `intelmqctl restart` bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226). - `intelmqctl check`: New parameter `--no-connections` to prevent the command from making connections e.g. to the redis pipeline.s # Contrib - tool `feeds-config-generator` to automatically generate the collector and parser runtime and pipeline configurations. - `malware_name_mapping`: Download and convert tool for malware family name mapping has been added. - Added a systemd script which creates systemd units for bots (#953). # Core - lib/bot - use SIGTERM instead of SIGINT to stop bots (#981). - Bots can specify a static method `check(parameters)` which can perform individual checks specific to the bot. These functions will be called by `intelmqctl check` if the bot is configured with the given parameters - top level bot parameters (description, group, module, name) are exposed as members of the class. - The parameter `feed` for collectors is deprecated for 2.0 and has been replaced by the more consistent `name` (#1144). - lib/message: - Subitems in fields of type `JSONDict` (see below) can be accessed directly. E.g. you can do: event['extra.foo'] = 'bar' event['extra.foo'] # gives 'bar' It is still possible to set and get the field as whole, however this may be removed or changed in the future: event['extra'] = '{"foo": "bar"}' event['extra'] # gives '{"foo": "bar"}' "Old" bots and configurations compatible with 1.0.x do still work. Also, the extra field is now properly exploded when exporting events, analogous to all other fields. - `Message.add`: The parameter `overwrite` accepts now three different values: `True`, `False` and `None` (new). True: An existing value will be overwritten False: An existing value will not be overwritten (previously an exception has been raised when the value was given). None (default): If the value exists an `KeyExists` exception is thrown (previously the same as False). This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function. - The message class has now the possibility to return a default value for non-exisiting fields, see `Message.set_default_value`. - Add `RewindableFileHandle` to utils making handling of CSV files more easy (optionally) - lib/pipeline: * you may now define more than one destination queues path the bot should pass the message to, see [Pipelines](https://github.com/certtools/intelmq/blob/develop/docs/User-Guid… (#1088, #1190). * the special path `"_on_error"` can be used to pass messages to different queues in case of processing errors (#1133). - `lib/harmonization`: Accept `AS` prefix for ASN values (automatically stripped). # Bots ## Collectors - `bots.collectors.mail`: - New parameters; `sent_from`: filter messages by sender, `sent_to`: filter messages by recipient - More debug logs - `bots.collectors.n6.collector_stomp`: renamed to `bots.collectors.stomp.collector` (#716) - bots.collectors.rt: - New parameter `search_requestor` to search for field Requestor. - Empty strings and `null` as value for search parameters are ignored. - Empty parameters `attachment_regex` and `url_regex` handled. - `bots.collectors.http.collector_http`: Ability to optionally use the current time in parameter `http_url`, added parameter `http_url_formatting`. - `bots.collectors.stomp.collector`: Heartbeat timeout is now logged with log level info instead of warning. - added `intelmq.bots.collectors.twitter.collector_twitter` - added `intelmq.bots.collectors.tcp.collector` that can be bound to another IntelMQ instance by a TCP output - `bots.collectors.microsoft.collector_interflow`: added for MS interflow API - Automatic ungzipping for .gz files. - added `intelmq.bots.collectors.calidog.collector_certstream` for collecting certstream data (#1120). - added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (#1096). ## Parsers - `bots.parsers.shadowserver`: - changed feednames . Please refer to it's README for the exact changes. - If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration. Previously errors like these were only logged and ignored otherwise. - add support for the feed `Accessible-Hadoop` - The Generic CSV Parser `bots.parsers.generic.parser_csv`: - It is possible to filter the data before processing them using the new parameters `filter_type` and `filter_text`. - It is possible to specify multiple columns using `|` character in parameter `columns`. - The parameter `time_format` now supports `'epoch_millis'` for seconds since the Epoch, milliseconds are supported but not used. - renamed `bots.parsers.cymru_full_bogons.parser` to `bots.parsers.cymru.parser_full_bogons`, compatibility shim will be removed in version 2.0 - added `bots.parsers.cymru.parser_cap_program` - added `intelmq.bots.parsers.zoneh.parser` for ZoneH feeds - added `intelmq.bots.parsers.sucuri.parser` - added `intelmq.bots.parsers.malwareurl.parser` - added `intelmq.bots.parsers.threatminer.parser` - added `intelmq.bots.parsers.webinspektor.parser` - added `intelmq.bots.parsers.twitter.parser` - added `intelmq.bots.parsers.microsoft.parser_ctip` * ignore the invalid IP '0.0.0.0' for the destination * fix the raw/dumped messages, did not contain the paling list previously. * use the new harmonization field `tlp` instead of `extra.tlp`. - `bots.parsers.alienvault.parser_otx`: Save TLP data in the new harmonization field `tlp`. - added `intelmq.bots.parsers.openphish.parser_commercial` - added `intelmq.bots.parsers.microsoft.parser_bingmurls` - added `intelmq.bots.parsers.calidog.parser_certstream` for parsing certstream data (#1120). - added `intelmq.bots.parsers.shodan.parser` for parsing shodan data (#1096). ## Experts - Added sieve expert for filtering and modifying events (#1083) * capable of distributing the event to appropriate named queues - `bots.experts.modify` * default rulesets: all malware name mappings have been migrated to the [Malware Name Mapping repository](https://github.com/certtools/malware_name_mapping) ruleset. See the new added contrib tool for download and conversion. * new parameter `case_sensitive` (default: True) - Added wait expert for sleeping - Added domain suffix expert to extract the TLD/Suffix from a domain name. - `bots.experts.maxmind_geoip`: New (optional) parameter `overwrite`, by default false. The current default was to overwrite! ## Outputs - `bots.outputs.file`: - String formatting can be used for file names. - New parameter `single_key` to only save one field. # Harmonization - Renamed `JSON` to `JSONDict` and added a new type `JSON`. `JSONDict` saves data internally as JSON, but acts like a dictionary. `JSON` accepts any valid JSON. - fixed regex for `protocol.transport` it previously allowed more values than it should have. - New ASN type. Like integer but checks the range. - added `destination.urlpath` and `source.urlpath` to harmonization. - New field `tlp` for tlp level specification. - New TLP type. Allows all four tlp levels, removes 'TLP:' prefix and converts to upper case. - Added new `classification.type` 'vulnerable client' - Added `(destination|source).domain_suffix` to hold the TLD/domain suffix. - New allowed value for `classification.type`: `infected system` for taxonomy `malicious code` (#1197). # Requirements - Requests is no longer listed as dependency of the core. For depending bots the requirement is noted in their REQUIREMENTS.txt file. # Documentation - Use Markdown for README again, as pypi now supports it. # Tests - Travis now correctly stops if a requirement could not be installed (#1257). - New tests for validating `etc/feeds.yaml` and `bots/BOTS` using cerberus and schemes are added (#1166). - New test for checking if `docs/Feeds.md` is up to date with `etc/feeds.yaml`. # Known bugs - `bots.experts.sieve` does not support textX (#1246). - performance degradation for extra fields (#1117). - Postgres output: support condensed JSONDicts (#1107). - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

OSError: [Errno 16] Resource busy: '/opt/intelmq/var/log' by update the intelmq
by Salehi Ghamsari, Majid 01 Aug '18

01 Aug '18

Hi, i have strange problem to compile my intelmq dev enviroment. I have stopped intelmq and try to compile, but i get OSError: [Errno 16] Resource busy: '/opt/intelmq/var/log' Has anyone ever had that? Regards Majid git clone https://github.com/certtools/intelmq.git /opt/dev_intelmq cd opt/dev_intelmq/ /opt/dev_intelmq # pip3 install -e . Obtaining file:///opt/dev_intelmq Requirement already satisfied: dnspython>=1.11.1 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: psutil>=1.2.1 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: python-dateutil>=2.5 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: python-termstyle>=0.1.10 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: pytz>=2012c in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: redis>=2.10 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: six>=1.5 in /usr/lib/python3.5/site-packages (from python-dateutil>=2.5->intelmq==1.2.0a1) Requirement already satisfied: setuptools in /usr/lib/python3.5/site-packages (from python-termstyle>=0.1.10->intelmq==1.2.0a1) Installing collected packages: intelmq Found existing installation: intelmq 1.0.4 Uninstalling intelmq-1.0.4: Exception: Traceback (most recent call last): File "/usr/lib/python3.5/shutil.py", line 538, in move os.rename(src, real_dst) OSError: [Errno 16] Resource busy: '/opt/intelmq/var/log' -> '/tmp/pip-6vus5dcm-uninstall/opt/intelmq/var/log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/usr/lib/python3.5/site-packages/pip/commands/install.py", line 342, in run prefix=options.prefix_path, File "/usr/lib/python3.5/site-packages/pip/req/req_set.py", line 778, in install requirement.uninstall(auto_confirm=True) File "/usr/lib/python3.5/site-packages/pip/req/req_install.py", line 754, in uninstall paths_to_remove.remove(auto_confirm) File "/usr/lib/python3.5/site-packages/pip/req/req_uninstall.py", line 115, in remove renames(path, new_path) File "/usr/lib/python3.5/site-packages/pip/utils/__init__.py", line 267, in renames shutil.move(old, new) File "/usr/lib/python3.5/shutil.py", line 550, in move rmtree(src) File "/usr/lib/python3.5/shutil.py", line 478, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib/python3.5/shutil.py", line 476, in rmtree os.rmdir(path) OSError: [Errno 16] Resource busy: '/opt/intelmq/var/log' You are using pip version 9.0.1, however version 10.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. /opt/dev_intelmq #

2 2

using the intelmq twitter bot
by Salehi Ghamsari, Majid 17 Jul '18

17 Jul '18

Hi there, I try to get the CVEs from this twitter page https://twitter.com/CVEnew but i get an exception that “that page does not exist” >target_timelines: screen_names of twitter accounts to be followed What should I put for target_timelines parameter? I do not know what I am missing. In the documentation we have to put for each bot parameters some examples input for better understanding the parameters. https://github.com/certtools/intelmq/blob/develop/docs/Bots.md#twitter Thank you for your help Majid Salehi Ghamsari Digital Public Services (DPS) Fraunhofer Institute for Open Communication Systems Kaiserin-Augusta-Allee 31, 10589 Berlin, Germany Email: majid.salehi.ghamsari(a)fokus.fraunhofer.de<mailto:majid.salehi.ghamsari@fokus.fraunhofer.de> Office: +49 (0) 30 3463 7118 Skype: salehimajid WEB: fokus.fraunhofer.de "cve-twitter-collector": { "parameters": { "feed": "Twitter", "target_timelines": "https://twitter.com/CVEnew ", "tweet_count": 10, "timelimit": 60000, "include_rts": false, "exclude_replies": true, "follow_urls": "", "consumer_key": "myconsumer_key ", "consumer_secret": "myconsumer_secret ", "access_token_key": "myaccess_token_key ", "access_token_secret": "myaccess_token_secret " }, "name": "Twitter", "group": "Collector", "module": "intelmq.bots.collectors.twitter.collector_twitter", "description": "Collects tweets from target_timelines. Maximum number of tweet_count from each timeline and up to timelimit back in time.", "enabled": true, "run_mode": "continuous" } Traceback (most recent call last): File "/opt/dev_intelmq/intelmq/lib/bot.py", line 167, in start self.process() File "/opt/dev_intelmq/intelmq/bots/collectors/twitter/collector_twitter.py", line 94, in process exclude_replies=self.exclude_replies) File "/usr/local/lib/python3.6/dist-packages/twitter/api.py", line 810, in GetUserTimeline data = self._ParseAndCheckTwitter(resp.content.decode('utf-8')) File "/usr/local/lib/python3.6/dist-packages/twitter/api.py", line 4890, in _ParseAndCheckTwitter self._CheckForTwitterError(data) File "/usr/local/lib/python3.6/dist-packages/twitter/api.py", line 4910, in _CheckForTwitterError raise TwitterError(data['errors']) twitter.error.TwitterError: [{'code': 34, 'message': 'Sorry, that page does not exist.'}]

2 2

changing bot runtime parameter
by Salehi Ghamsari, Majid 11 Jul '18

11 Jul '18

Hello there, I'm trying to develop a simple pipeline with three bots. test-file-collector---> test-message-parser---> test-tcp-output I would like to put or change the IP runtime parameter of the TCP output bot "test-tcp-output" depends on Message that I get from file-collector in the runtime. What is the best method to do this in intelmq? Thank you for your help. Majid

3 4

testing tcpcollecor bot 1.1.0rc1
by Salehi Ghamsari, Majid 05 Jul '18

05 Jul '18

Hi, i try to test the tcpcollector bot, on sender site I send this message { "feed.accuracy":100.0, "feed.name":"FileCollector", "feed.url":"file://localhost/opt/intelmq/var/lib/bots/file-input/event.json", "time.observation":"2018-07-05T08:25:51+00:00", "extra.info":"TC_IMQ_Event_001", "extra.threat_level_id":"1", "raw":"eyJFdmVudCI6IHsiZGlzdHJpYnV0aW9uIjogIjEiLCAiaW5mbyI6ICJUQ19JTVFfRXZlbnRfMDAxIiwgInNoYXJpbmdfZ3JvdXBfaWQiOiAiMCIsICJ0aHJlYXRfbGV2ZWxfaWQiOiAiMSJ9fQ==" } On the other site the TcpColltector bot failed with this message. Bot has found a problem. Traceback (most recent call last): File "/opt/dev_intelmq/intelmq/lib/bot.py", line 167, in start self.process() File "/opt/dev_intelmq/intelmq/bots/collectors/tcp/collector.py", line 47, in process self.send_message(report) File "/opt/dev_intelmq/intelmq/lib/bot.py", line 816, in send_message super(CollectorBot, self).send_message(*messages, path=path) File "/opt/dev_intelmq/intelmq/lib/bot.py", line 797, in __add_report_fields report.add("feed.name", self.parameters.name) AttributeError: 'Parameters' object has no attribute 'name' What do I missing? Regards Majid

2 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev July 2018