Hi mailing list, I've recently upgraded intelmq from version 1.1.2 to 2.0.1. My intelmq runs in Docker. I have a slight issue with the botnet startup on the new version (the previous version works ok). Intelmq fails to start because of the exception present at the bottom of the mail. Upon further investigation, when the folder was manually created, other issues started to pop up. All of them were due to a permission error on either the files in /opt/intelmq/etc/ or some else that intelmq tried to access during startup. Did something change in the setup requirements that I've overlooked?
28/08/2019 09:48:272019-08-28 07:48:27,796 DEBG 'intelmq' stdout output: 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 657, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 334, in log 28/08/2019 09:48:27 handler = FileHandler("%s/%s.log" % (log_path, name)) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1032, in __init__ 28/08/2019 09:48:27 StreamHandler.__init__(self, self._open()) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1061, in _open 28/08/2019 09:48:27 return open(self.baseFilename, self.mode, encoding=self.encoding) 28/08/2019 09:48:27FileNotFoundError: [Errno 2] No such file or directory: '/opt/intelmq/var/log/intelmqctl.log' 28/08/2019 09:48:27 28/08/2019 09:48:27During handling of the above exception, another exception occurred: 28/08/2019 09:48:27 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/bin/intelmqctl", line 11, in <module> 28/08/2019 09:48:27 load_entry_point('intelmq==2.0.1', 'console_scripts', 'intelmqctl')() 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 1710, in main 28/08/2019 09:48:27 x = IntelMQController(interactive=True) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 661, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 345, in log 28/08/2019 09:48:27 raise ValueError("Invalid configuration, neither log_path is given nor syslog is used.") 28/08/2019 09:48:27ValueError: Invalid configuration, neither log_path is given nor syslog is used.
Sincerely, Václav Brůžek
Hi Václav,
Is /opt/intelmq/var/log/intelmqctl.log owned by root? If yes: chown intelmq:intelmq /opt/intelmq/var/log/intelmqctl.log
Sebastian
On 28/08/2019 11.14, Vaclav Bruzek wrote:
Hi mailing list, I've recently upgraded intelmq from version 1.1.2 to 2.0.1. My intelmq runs in Docker. I have a slight issue with the botnet startup on the new version (the previous version works ok). Intelmq fails to start because of the exception present at the bottom of the mail. Upon further investigation, when the folder was manually created, other issues started to pop up. All of them were due to a permission error on either the files in /opt/intelmq/etc/ or some else that intelmq tried to access during startup. Did something change in the setup requirements that I've overlooked?
28/08/2019 09:48:272019-08-28 07:48:27,796 DEBG 'intelmq' stdout output: 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 657, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 334, in log 28/08/2019 09:48:27 handler = FileHandler("%s/%s.log" % (log_path, name)) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1032, in __init__ 28/08/2019 09:48:27 StreamHandler.__init__(self, self._open()) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1061, in _open 28/08/2019 09:48:27 return open(self.baseFilename, self.mode, encoding=self.encoding) 28/08/2019 09:48:27FileNotFoundError: [Errno 2] No such file or directory: '/opt/intelmq/var/log/intelmqctl.log' 28/08/2019 09:48:27 28/08/2019 09:48:27During handling of the above exception, another exception occurred: 28/08/2019 09:48:27 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/bin/intelmqctl", line 11, in
<module> 28/08/2019 09:48:27 load_entry_point('intelmq==2.0.1', 'console_scripts', 'intelmqctl')() 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 1710, in main 28/08/2019 09:48:27 x = IntelMQController(interactive=True) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 661, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 345, in log 28/08/2019 09:48:27 raise ValueError("Invalid configuration, neither log_path is given nor syslog is used.") 28/08/2019 09:48:27ValueError: Invalid configuration, neither log_path is given nor syslog is used. Sincerely, Václav Brůžek
Hi Sebastian, the whole directory /opt/intelmq/var/ is not created only /etc/ which was created by me during image build. I've tried to fix the problem by creating the required folders but then the the permissions on files in etc were wrong, then intelmq was unable to load pid files and so on. That was the moment I decided to ask the mailing list. My guess is that something is being done differently now. Even weirder is that the whole container (and intelmq) runs under root so there should be no errors regarding permissions.
Sincerely, Václav Brůžek
On Wed, 28 Aug 2019 at 11:57, Sebastian Wagner wagner@cert.at wrote:
Hi Václav,
Is /opt/intelmq/var/log/intelmqctl.log owned by root? If yes: chown intelmq:intelmq /opt/intelmq/var/log/intelmqctl.log
Sebastian On 28/08/2019 11.14, Vaclav Bruzek wrote:
Hi mailing list, I've recently upgraded intelmq from version 1.1.2 to 2.0.1. My intelmq runs in Docker. I have a slight issue with the botnet startup on the new version (the previous version works ok). Intelmq fails to start because of the exception present at the bottom of the mail. Upon further investigation, when the folder was manually created, other issues started to pop up. All of them were due to a permission error on either the files in /opt/intelmq/etc/ or some else that intelmq tried to access during startup. Did something change in the setup requirements that I've overlooked?
28/08/2019 09:48:272019-08-28 07:48:27,796 DEBG 'intelmq' stdout output: 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 657, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 334, in log 28/08/2019 09:48:27 handler = FileHandler("%s/%s.log" % (log_path, name)) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1032, in __init__ 28/08/2019 09:48:27 StreamHandler.__init__(self, self._open()) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1061, in _open 28/08/2019 09:48:27 return open(self.baseFilename, self.mode, encoding=self.encoding) 28/08/2019 09:48:27FileNotFoundError: [Errno 2] No such file or directory: '/opt/intelmq/var/log/intelmqctl.log' 28/08/2019 09:48:27 28/08/2019 09:48:27During handling of the above exception, another exception occurred: 28/08/2019 09:48:27 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/bin/intelmqctl", line 11, in <module> 28/08/2019 09:48:27 load_entry_point('intelmq==2.0.1', 'console_scripts', 'intelmqctl')() 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 1710, in main 28/08/2019 09:48:27 x = IntelMQController(interactive=True) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 661, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 345, in log 28/08/2019 09:48:27 raise ValueError("Invalid configuration, neither log_path is given nor syslog is used.") 28/08/2019 09:48:27ValueError: Invalid configuration, neither log_path is given nor syslog is used.
Sincerely, Václav Brůžek
-- // Sebastian Wagner wagner@cert.at wagner@cert.at - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
Hi,
When you do a manual installation, you also need to create the directory structure. For this purpose there's a new executable `intelmqsetup` doing all these steps. See also the installation documentation: https://github.com/certtools/intelmq/blob/maintenance/docs/INSTALL.md#pypi
All bots and intelmq actually check if they run privileged as root and degrade to "intelmq" for security reasons. That's definitely a bug if that's not working for you as you write that they are all running as root.
Sebastian
On 28/08/2019 13.00, Vaclav Bruzek wrote:
Hi Sebastian, the whole directory /opt/intelmq/var/ is not created only /etc/ which was created by me during image build. I've tried to fix the problem by creating the required folders but then the the permissions on files in etc were wrong, then intelmq was unable to load pid files and so on. That was the moment I decided to ask the mailing list. My guess is that something is being done differently now. Even weirder is that the whole container (and intelmq) runs under root so there should be no errors regarding permissions.
Sincerely, Václav Brůžek
On Wed, 28 Aug 2019 at 11:57, Sebastian Wagner <wagner@cert.at mailto:wagner@cert.at> wrote:
Hi Václav, Is /opt/intelmq/var/log/intelmqctl.log owned by root? If yes: chown intelmq:intelmq /opt/intelmq/var/log/intelmqctl.log Sebastian On 28/08/2019 11.14, Vaclav Bruzek wrote:Hi mailing list, I've recently upgraded intelmq from version 1.1.2 to 2.0.1. My intelmq runs in Docker. I have a slight issue with the botnet startup on the new version (the previous version works ok). Intelmq fails to start because of the exception present at the bottom of the mail. Upon further investigation, when the folder was manually created, other issues started to pop up. All of them were due to a permission error on either the files in /opt/intelmq/etc/ or some else that intelmq tried to access during startup. Did something change in the setup requirements that I've overlooked? 28/08/2019 09:48:272019-08-28 07:48:27,796 DEBG 'intelmq' stdout output: 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 657, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 334, in log 28/08/2019 09:48:27 handler = FileHandler("%s/%s.log" % (log_path, name)) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1032, in __init__ 28/08/2019 09:48:27 StreamHandler.__init__(self, self._open()) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1061, in _open 28/08/2019 09:48:27 return open(self.baseFilename, self.mode, encoding=self.encoding) 28/08/2019 09:48:27FileNotFoundError: [Errno 2] No such file or directory: '/opt/intelmq/var/log/intelmqctl.log' 28/08/2019 09:48:27 28/08/2019 09:48:27During handling of the above exception, another exception occurred: 28/08/2019 09:48:27 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/bin/intelmqctl", line 11, in <module> 28/08/2019 09:48:27 load_entry_point('intelmq==2.0.1', 'console_scripts', 'intelmqctl')() 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 1710, in main 28/08/2019 09:48:27 x = IntelMQController(interactive=True) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 661, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 345, in log 28/08/2019 09:48:27 raise ValueError("Invalid configuration, neither log_path is given nor syslog is used.") 28/08/2019 09:48:27ValueError: Invalid configuration, neither log_path is given nor syslog is used. Sincerely, Václav Brůžek-- // Sebastian Wagner <wagner@cert.at> <mailto:wagner@cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at <http://nic.at> GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
Hi, thanks for the suggestion however after build with intelmqsetup the problem persists. Following is an exception and the files in etc and the access privileges.
Traceback (most recent call last): File "/usr/local/bin/intelmqctl", line 11, in <module> load_entry_point('intelmq==2.0.1', 'console_scripts', 'intelmqctl')() File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 1710, in main x = IntelMQController(interactive=True) File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 749, in __init__ self.load_defaults_configuration() File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 929, in load_defaults_configuration config = utils.load_configuration(DEFAULTS_CONF_FILE) File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 194, in load_configuration with open(configuration_filepath, 'r') as fpconfig: PermissionError: [Errno 13] Permission denied: '/opt/intelmq/etc/defaults.conf'
drwxr-xr-x 3 intelmq root 4096 Aug 28 12:04 . drwxr-xr-x 9 intelmq root 4096 Aug 28 12:04 .. -rwxrwx--- 1 root root 6846 Aug 22 08:00 BOTS -rwxrwx--- 1 root root 1136 Aug 28 12:04 defaults.conf drwxr-xr-x 2 root root 4096 Aug 28 12:03 examples -rwxrwx--- 1 root root 21650 Aug 22 08:00 harmonization.conf -rwxrwx--- 1 root root 24517 Aug 22 08:00 pipeline.conf -rwxrwx--- 1 root root 38675 Aug 28 12:04 runtime.conf
Sincerely, Václav Brůžek
On Wed, 28 Aug 2019 at 13:34, Sebastian Wagner wagner@cert.at wrote:
Hi,
When you do a manual installation, you also need to create the directory structure. For this purpose there's a new executable `intelmqsetup` doing all these steps. See also the installation documentation: https://github.com/certtools/intelmq/blob/maintenance/docs/INSTALL.md#pypi
All bots and intelmq actually check if they run privileged as root and degrade to "intelmq" for security reasons. That's definitely a bug if that's not working for you as you write that they are all running as root.
Sebastian On 28/08/2019 13.00, Vaclav Bruzek wrote:
Hi Sebastian, the whole directory /opt/intelmq/var/ is not created only /etc/ which was created by me during image build. I've tried to fix the problem by creating the required folders but then the the permissions on files in etc were wrong, then intelmq was unable to load pid files and so on. That was the moment I decided to ask the mailing list. My guess is that something is being done differently now. Even weirder is that the whole container (and intelmq) runs under root so there should be no errors regarding permissions.
Sincerely, Václav Brůžek
On Wed, 28 Aug 2019 at 11:57, Sebastian Wagner wagner@cert.at wrote:
Hi Václav,
Is /opt/intelmq/var/log/intelmqctl.log owned by root? If yes: chown intelmq:intelmq /opt/intelmq/var/log/intelmqctl.log
Sebastian On 28/08/2019 11.14, Vaclav Bruzek wrote:
Hi mailing list, I've recently upgraded intelmq from version 1.1.2 to 2.0.1. My intelmq runs in Docker. I have a slight issue with the botnet startup on the new version (the previous version works ok). Intelmq fails to start because of the exception present at the bottom of the mail. Upon further investigation, when the folder was manually created, other issues started to pop up. All of them were due to a permission error on either the files in /opt/intelmq/etc/ or some else that intelmq tried to access during startup. Did something change in the setup requirements that I've overlooked?
28/08/2019 09:48:272019-08-28 07:48:27,796 DEBG 'intelmq' stdout output: 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 657, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 334, in log 28/08/2019 09:48:27 handler = FileHandler("%s/%s.log" % (log_path, name)) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1032, in __init__ 28/08/2019 09:48:27 StreamHandler.__init__(self, self._open()) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1061, in _open 28/08/2019 09:48:27 return open(self.baseFilename, self.mode, encoding=self.encoding) 28/08/2019 09:48:27FileNotFoundError: [Errno 2] No such file or directory: '/opt/intelmq/var/log/intelmqctl.log' 28/08/2019 09:48:27 28/08/2019 09:48:27During handling of the above exception, another exception occurred: 28/08/2019 09:48:27 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/bin/intelmqctl", line 11, in
<module> 28/08/2019 09:48:27 load_entry_point('intelmq==2.0.1', 'console_scripts', 'intelmqctl')() 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 1710, in main 28/08/2019 09:48:27 x = IntelMQController(interactive=True) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 661, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 345, in log 28/08/2019 09:48:27 raise ValueError("Invalid configuration, neither log_path is given nor syslog is used.") 28/08/2019 09:48:27ValueError: Invalid configuration, neither log_path is given nor syslog is used.
Sincerely, Václav Brůžek
-- // Sebastian Wagner wagner@cert.at wagner@cert.at - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg
--
// Sebastian Wagner wagner@cert.at wagner@cert.at - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg