Hi Jonathan and list members,
Thank you very much for your instructions, Jonathan! Please kindly let me ask more. I am very new to intelmq, and this is my first run :-).
In order to check "what is going on" inside your IntelMQ botnet, you could use the following commands: "sudo -u <intelmq_user_account> intelmqctl status" -> this one checks which bots are running, which are stopped and which are disabled.
$ intelmqctl status Bot cymru-whois-expert is running. Bot deduplicator-expert is running. Bot feodo-tracker-browse-collector is running. Bot feodo-tracker-browse-parser is stopped. Bot file-output is running. Bot gethostbyname-1-expert is running. Bot gethostbyname-2-expert is running. Bot malc0de-parser is running. Bot malc0de-windows-format-collector is running. Bot spamhaus-drop-collector is running. Bot spamhaus-drop-parser is running. Bot taxonomy-expert is running. Bot url2fqdn-expert is running.
"sudo -u <intelmq_user_account> intelmqctl list queues" -> this one displays the current amount of messages stored in the internal or external bots queues. (use "-q" at the end if you want to hide queues with 0 messages)
$ intelmqctl list queues cymru-whois-expert-queue - 0 cymru-whois-expert-queue-internal - 0 deduplicator-expert-queue - 0 deduplicator-expert-queue-internal - 0 feodo-tracker-browse-parser-queue - 1 feodo-tracker-browse-parser-queue-internal - 0 file-output-queue - 0 file-output-queue-internal - 0 gethostbyname-1-expert-queue - 0 gethostbyname-1-expert-queue-internal - 0 gethostbyname-2-expert-queue - 0 gethostbyname-2-expert-queue-internal - 0 malc0de-parser-queue - 0 malc0de-parser-queue-internal - 0 spamhaus-drop-parser-queue - 0 spamhaus-drop-parser-queue-internal - 0 taxonomy-expert-queue - 0 taxonomy-expert-queue-internal - 0 url2fqdn-expert-queue - 0 url2fqdn-expert-queue-internal - 0
"cat /var/log/intelmq/<bot_name>.log" will display the bot output (by default only info and error messages are shown, debug message are hidden -> am I right?)
Yes, some INFO messages are shown in the log files, so that I reckon they are anyway working.
Intelmq's setup.py might miss the dependency for beautiflsoap4 in REQUIRES, according to the ERROR message in feodo-tracker-browse-parser-queue.log (quoted below). (See also https://github.com/certtools/intelmq/blob/develop/setup.py)
| 2021-03-11 16:30:47,166 - feodo-tracker-browse-parser - INFO - Bot is starting. | 2021-03-11 16:30:47,168 - feodo-tracker-browse-parser - ERROR - Bot initialization failed. | Traceback (most recent call last): | File "/usr/local/lib/python3.7/site-packages/intelmq/lib/bot.py", line 164, in __init__ | self.init() | File "/usr/local/lib/python3.7/site-packages/intelmq/bots/parsers/html_table/parser.py", line 37, in init | raise MissingDependencyError("beautifulsoup4") | intelmq.lib.exceptions.MissingDependencyError: Could not load dependency 'be | autifulsoup4', please install it with apt/yum/dnf/zypper (possibly named python3-beautifulsoup4) or pip3. | 2021-03-11 16:30:47,171 - feodo-tracker-browse-parser - INFO - Bot stopped.
Finally, you can check the output of the botnet (your DB, a MISP instance, whatever you have) to make sure that what your bots have collected has been processed properly.
I am lost here. Can someone tell me which manual page I should refer, please? I'd use PostgreSQL fot the data store.
Thank you in advance!
Best Regards,