IntelMQ-users August 2019

intelmq-users@lists.cert.at

2 participants
3 discussions

Permissions error
by Vaclav Bruzek 28 Aug '19

28 Aug '19

Hi mailing list, I've recently upgraded intelmq from version 1.1.2 to 2.0.1. My intelmq runs in Docker. I have a slight issue with the botnet startup on the new version (the previous version works ok). Intelmq fails to start because of the exception present at the bottom of the mail. Upon further investigation, when the folder was manually created, other issues started to pop up. All of them were due to a permission error on either the files in /opt/intelmq/etc/ or some else that intelmq tried to access during startup. Did something change in the setup requirements that I've overlooked? 28/08/2019 09:48:272019-08-28 07:48:27,796 DEBG 'intelmq' stdout output: 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 657, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 334, in log 28/08/2019 09:48:27 handler = FileHandler("%s/%s.log" % (log_path, name)) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1032, in __init__ 28/08/2019 09:48:27 StreamHandler.__init__(self, self._open()) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1061, in _open 28/08/2019 09:48:27 return open(self.baseFilename, self.mode, encoding=self.encoding) 28/08/2019 09:48:27FileNotFoundError: [Errno 2] No such file or directory: '/opt/intelmq/var/log/intelmqctl.log' 28/08/2019 09:48:27 28/08/2019 09:48:27During handling of the above exception, another exception occurred: 28/08/2019 09:48:27 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/bin/intelmqctl", line 11, in <module> 28/08/2019 09:48:27 load_entry_point('intelmq==2.0.1', 'console_scripts', 'intelmqctl')() 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 1710, in main 28/08/2019 09:48:27 x = IntelMQController(interactive=True) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 661, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 345, in log 28/08/2019 09:48:27 raise ValueError("Invalid configuration, neither log_path is given nor syslog is used.") 28/08/2019 09:48:27ValueError: Invalid configuration, neither log_path is given nor syslog is used. Sincerely, Václav Brůžek

2 4

IntelMQ 2.0.1 Bugfix release
by Sebastian Wagner 23 Aug '19

23 Aug '19

Dear community, I just released version 2.0.1 of IntelMQ, the first bugfix release for the 2.x series, published in May. This is also the first 2.0 release which will hit the deb/rpm stable repositories. If you are using them, you'll receive the update in the next few hours. The version further introduces a new intelmqctl command: intelmqctl upgrade-conf. This command takes care of changed configurations, or newly added parameters. Call it after every upgrade, as documented in the Upgrade documentation linked below. The tools updates the configuration if necessary and creates backup files on it's own. If it can't fix the issues automatically, it will show you an error message. A side effect is, that intelmq now saves a "state" of itself in /var/lib/intelmq/state.json or /opt/intelmq/var/lib/state.json depending on your setup. Install documentation: https://github.com/certtools/intelmq/blob/2.0.1/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.0.1/docs/UPGRADING.md As always: read the NEWS file, upgrade according to the documentation and have fun! If you get any errors, please report them here or in the bug tracker. The full changelog: ### Core - `intelmq.lib.harmonization`: - `IPAddress`: Remove Scope/Zone IDs for IPv6 addresses in sanitation. - All types: Handle `None` for validation and sanitation gracefully. - `intelmq.lib.bot`: - fix parameters of ParserBot and CollectorBot constructors, allowing `intelmqctl run` with these bots again (#1414). - Also run `rate_limit` after retry counter reset (#1431). - `__version_info__`: - is now available in the top level module. - uses integer values now instead of strings for numerical version parts - Also provide (empty) `ROOT_DIR` for non-pip installations. - `intelmq.lib.upgrades`: New library file `upgrades` with upgrade functions. - `intelmq.lib.utils`: - New function `setup_list_logging` for intelmqctl check an possibly others. - Fix return values (#1423). - New function `version_smaller` for version comparisons. - New function `lazy_int` for version conversions. - `parse_logline`: Handle thread IDs. - `log` takes a new argument `logging_level_stream` for the logging level of the console handler. - New constant `LOG_FORMAT_SIMPLE`, used by intelmqctl. - New function `write_configuration` to write dicts to files in the correct json formatting. - New function `create_request_session_from_bot`. - `intelmq.lib.pipeline`: - AMQP: - Actually use `source/destination_pipeline_amqp_virtual_host` parameter. - Support for SSL with `source/destination_pipeline_ssl` parameter. - pipeline base class: add missing dummy methods. - Add missing return types. - Redis: Evaluate return parameter of queue/key deletion. - Variable `STATE_FILE_PATH` added. ### Development - `intelmq.bin.intelmq_gen_docs`: For yaml use `safe_load` instead of unsafe `load`. ### Harmonization - IPAddress type: Remove Scope/Zone IDs for IPv6 addresses in sanitation. - TLP: Sanitation handles now more cases: case-insensitive prefixes and arbitrary whitespace between the prefix and the value (#1420). ### Bots #### Collectors - `intelmq.bots.collectors.http.collector_http`: Use `utils.create_request_session_from_bot`. - `intelmq.bots.collectors.http.collector_http_stream`: Use `utils.create_request_session_from_bot` and thus fix some retries on connection timeouts. - `intelmq.bots.collectors.mail.collector_mail_url`: Use `utils.create_request_session_from_bot`. - `intelmq.bots.collectors.microsoft.collector_interflow`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. - `intelmq.bots.collectors.rt.collector_rt`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. - `intelmq.bots.collectors.twitter.collector_twitter`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts for non-twitter connections. #### Parsers - `intelmq.bots.parsers.n6.parser_n6stomp`: use `malware-generic` instead of `generic-n6-drone` for unknown infected system events. - `intelmq.bots.parsers.abusech.parser_ip`: Support LastOnline column in feodo feed (#1400) and use it for `time.source` if available. - Use lower case malware names as default, should not make a difference in practice. - Fix handling of CSV header for feodotracker (#1417, #1418). - `intelmq.bots.parsers.netlab_360.parser`: Detect feeds with `https://` too. #### Experts - `intelmq.bots.experts.generic_db_lookup`: Recommend psycopg2-binary package. - `intelmq.bots.experts.modify.expert`: - Compile regular expressions (all string rules) at initialization, improves the speed. - Warn about old configuration style deprecation. - `intelmq.bots.experts.do_portal.expert`: - Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts (#1432). - Treat "502 Bad Gateway" as timeout which can be retried. - `intelmq.bots.experts.ripe.expert`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. - `intelmq.bots.experts.url2fqdn.expert`: Support for IP addresses in hostnames (#1416). - `intelmq.bots.experts.national_cert_contact_certat.expert`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. #### Outputs - `intelmq.bots.outputs.postgresql`: Recommend psycopg2-binary package. - `intelmq.bots.outputs.amqptopic`: - Shutdown: Close connection only if connection exists. - Add support for pika > 1. Pika changed the way it indicates (Non-)Acknowledgments of sent messages. - Gracefully handle unroutable messages and give advice. - Support for connections without authentication. - Replace deprecated parameter `type` with `exchange_type` for `exchange_declare`, supporting pika >= 0.11 (#1425). - New parameters `message_hierarchical_output`, `message_with_type`, `message_jsondict_as_string`. - New parameter `use_ssl` for SSL connections. - New parameter `single_key` for sending single fields instead of the full event. - `intelmq.bots.outputs.mongodb.output`: Support for pymongo >= 3.0.0 (#1063, PR#1421). - `intelmq.bots.outputs.file`: `time.*` field serialization: support for microseconds. - `intelmq.bots.outputs.mongodb.output`: Support for authentication in pymongo >= 3.5 (#1062). - `intelmq.bots.outputs.restapi.output`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. ### Documentation - Add certbund-contact to the ecosystem document. - Rename the IDEA expert to "IDEA Converter". - Add the new configuration upgrade function to the docs. - User Guide: - Clarify on Uninstallation ### Packaging - Do not execute the tcp collector tests during debian and ubuntu builds as they fail there. ### Tests - `intelmq.lib.test`: Disable statistics for test runs of bots. - `contrib.malware_name_mapping`: Added tests. - Travis: Also run tests of contrib. ### Tools - `intelmqsetup`: Only change directory ownerships if necessary. - `intelmqctl`: - Provide new command `upgrade-conf` to uprade configuration to a newer version. - Makes backups of configurations files on its own. - Also checks for previously skipped or new functions of older versions and catches up. - Provides logging level on class layer. - Fix `-q` flag for `intelmqctl list queues` by renaming its alternative name to `--non-zero` to avoid a name collision with the global `--quiet` parameter. - For console output the string `intelmqctl: ` at the beginning of each line is no longer present. - `check`: Support for the state file added. Checks if it exists and all upgrade functions have been executed successfully. - Waits for up to 2 seconds when stopping a bot (#1434). - Exits early on restart when stopping a bot did not work (#1434). - `intelmqctl run process -m` debugging: Mock acknowledge method if incoming message is mocked too, otherwise a different message is acknowledged. - Queue listing for AMQP: Support non-default monitoring URLs, see User-Guide. ### Contrib * logcheck rules: Adapt ignore rule to cover the instance IDs of bot names. * malware name mapping: - Ignore lines in mapping starting with '#'. - Optionally include malpedia data. - Fix command line parsing for not arguments (#1427). - bash-completion: Support for `intelmqctl upgrade-config` added. ### Known issues - http stream collector: retry on regular connection problems? (#1435) - tests: capture logging with context manager (#1342) - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952) - n6 parser: mapping is modified within each run (#905) - reverse DNS: Only first record is used (#877) - Corrupt dump files when interrupted during writing (#870) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Max retries issue
by Vaclav Bruzek 01 Aug '19

01 Aug '19

Hi mailinglist, I've went over intelmq logs today and that some bots keep retrying indefinitely when they encounter a failure. However each of those bots have *error_max_retries* set to 1 (and *error_retry_delay *to 60). Which in my mind should limit the bot to just try one more time and than give up and sleep (if sleep is set up) however this is not happening. Am I doing something wrong or does this functionality work differently? My version of intelmq is 1.1.1 in Docker on Ubuntu 18.04 as base. Following is an example from logs os collector collecting from phishtank: 2019-03-01 20:07:51,747 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:07:52,139 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:07:52,140 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:07:52,140 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:07:52,298 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:07:52,299 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:07:52,299 - phishtank-collector - INFO - Bot will continue in 60 seconds. 2019-03-01 20:08:52,359 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:08:52,654 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:08:52,656 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:08:52,656 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:08:52,831 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:08:52,832 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:08:52,832 - phishtank-collector - INFO - Bot will continue in 60 seconds. 2019-03-01 20:09:52,891 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:09:53,138 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:09:53,139 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:09:53,139 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:09:53,315 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:09:53,317 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:09:53,318 - phishtank-collector - INFO - Bot will continue in 60 seconds. Sincerely, Václav Brůžek

2 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users August 2019