Dear intelmq-users list,
I've just installed intelmq 2.3.0 via pypi and run it via `intelmqctl start`, and I can see several python processes are running with intelmq user in top command.
Can I see some data that is collected by intelmq bots at this stage? If yes, where should I find it?
Now I reckon intelmq-manager is the answer to the above question, and I am trying to figure out how to use intelmq-manager web interface. (pypi installation has done)
Any suggestions will be appreciated. Thank you!
Best Regards.
Greetings Moto,
In order to check "what is going on" inside your IntelMQ botnet, you could use the following commands: "sudo -u <intelmq_user_account> intelmqctl status" -> this one checks which bots are running, which are stopped and which are disabled. "sudo -u <intelmq_user_account> intelmqctl list queues" -> this one displays the current amount of messages stored in the internal or external bots queues. (use "-q" at the end if you want to hide queues with 0 messages) "cat /var/log/intelmq/<bot_name>.log" will display the bot output (by default only info and error messages are shown, debug message are hidden -> am I right?) Finally, you can check the output of the botnet (your DB, a MISP instance, whatever you have) to make sure that what your bots have collected has been processed properly.
You could also manually run your bots with "sudo -u <intelmq_user_account> intelmqctl run <bot_name> -l DEBUG" so you can check what the bot is doing in real time.
Best regards, Jonathan
-- Jonathan SCOUPREMAN | jscoupreman@excellium-services.lu | PGP Key ID: 0xAD971C07 CERT-XLM | cert@excellium-services.com | PGP Key ID: 0xD74E5AC0 CERT-XLM Incident Handler @ excellium-services.com Excellium Services S.A. | 5 rue Goell L-5326 Contern Mobile: +352 691 982 790 Emergency: +352 262 039 64 708 | emergency@excellium-services.com | PGP Key ID: 0x42662EFE
-----Original Message----- From: IntelMQ-users intelmq-users-bounces@lists.cert.at On Behalf Of moto kawasaki Sent: jeudi 11 mars 2021 09:26 To: intelmq-users@lists.cert.at Subject: [IntelMQ-users] where can I see data gathered by intelmq?
Dear intelmq-users list,
I've just installed intelmq 2.3.0 via pypi and run it via `intelmqctl start`, and I can see several python processes are running with intelmq user in top command.
Can I see some data that is collected by intelmq bots at this stage? If yes, where should I find it?
Now I reckon intelmq-manager is the answer to the above question, and I am trying to figure out how to use intelmq-manager web interface. (pypi installation has done)
Any suggestions will be appreciated. Thank you!
Best Regards.
-- moto kawasaki moto@kawasaki3.org +81-90-2464-8454
-- List settings: https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.cert... IntelMQ Documentation: https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fintelmq.re... This email is confidential and may contain legally privileged information. If you are not the intended recipient, you should not copy, distribute, disclose or use the information it contains, please e-mail the sender immediately and delete this message from your system. Note: e-mails are susceptible to corruption, interception and unauthorised amendment; we do not accept liability for any such changes, or for their consequences. You should be aware that we may monitor your e-mails and their content. Excellium Services SA.
Hi Jonathan and list members,
Thank you very much for your instructions, Jonathan! Please kindly let me ask more. I am very new to intelmq, and this is my first run :-).
In order to check "what is going on" inside your IntelMQ botnet, you could use the following commands: "sudo -u <intelmq_user_account> intelmqctl status" -> this one checks which bots are running, which are stopped and which are disabled.
$ intelmqctl status Bot cymru-whois-expert is running. Bot deduplicator-expert is running. Bot feodo-tracker-browse-collector is running. Bot feodo-tracker-browse-parser is stopped. Bot file-output is running. Bot gethostbyname-1-expert is running. Bot gethostbyname-2-expert is running. Bot malc0de-parser is running. Bot malc0de-windows-format-collector is running. Bot spamhaus-drop-collector is running. Bot spamhaus-drop-parser is running. Bot taxonomy-expert is running. Bot url2fqdn-expert is running.
"sudo -u <intelmq_user_account> intelmqctl list queues" -> this one displays the current amount of messages stored in the internal or external bots queues. (use "-q" at the end if you want to hide queues with 0 messages)
$ intelmqctl list queues cymru-whois-expert-queue - 0 cymru-whois-expert-queue-internal - 0 deduplicator-expert-queue - 0 deduplicator-expert-queue-internal - 0 feodo-tracker-browse-parser-queue - 1 feodo-tracker-browse-parser-queue-internal - 0 file-output-queue - 0 file-output-queue-internal - 0 gethostbyname-1-expert-queue - 0 gethostbyname-1-expert-queue-internal - 0 gethostbyname-2-expert-queue - 0 gethostbyname-2-expert-queue-internal - 0 malc0de-parser-queue - 0 malc0de-parser-queue-internal - 0 spamhaus-drop-parser-queue - 0 spamhaus-drop-parser-queue-internal - 0 taxonomy-expert-queue - 0 taxonomy-expert-queue-internal - 0 url2fqdn-expert-queue - 0 url2fqdn-expert-queue-internal - 0
"cat /var/log/intelmq/<bot_name>.log" will display the bot output (by default only info and error messages are shown, debug message are hidden -> am I right?)
Yes, some INFO messages are shown in the log files, so that I reckon they are anyway working.
Intelmq's setup.py might miss the dependency for beautiflsoap4 in REQUIRES, according to the ERROR message in feodo-tracker-browse-parser-queue.log (quoted below). (See also https://github.com/certtools/intelmq/blob/develop/setup.py)
| 2021-03-11 16:30:47,166 - feodo-tracker-browse-parser - INFO - Bot is starting. | 2021-03-11 16:30:47,168 - feodo-tracker-browse-parser - ERROR - Bot initialization failed. | Traceback (most recent call last): | File "/usr/local/lib/python3.7/site-packages/intelmq/lib/bot.py", line 164, in __init__ | self.init() | File "/usr/local/lib/python3.7/site-packages/intelmq/bots/parsers/html_table/parser.py", line 37, in init | raise MissingDependencyError("beautifulsoup4") | intelmq.lib.exceptions.MissingDependencyError: Could not load dependency 'be | autifulsoup4', please install it with apt/yum/dnf/zypper (possibly named python3-beautifulsoup4) or pip3. | 2021-03-11 16:30:47,171 - feodo-tracker-browse-parser - INFO - Bot stopped.
Finally, you can check the output of the botnet (your DB, a MISP instance, whatever you have) to make sure that what your bots have collected has been processed properly.
I am lost here. Can someone tell me which manual page I should refer, please? I'd use PostgreSQL fot the data store.
Thank you in advance!
Best Regards,
Dear Moto,
(..)
Finally, you can check the output of the botnet (your DB, a MISP instance, whatever you have) to make sure that what your bots have collected has been processed properly.
I am lost here. Can someone tell me which manual page I should refer, please? I'd use PostgreSQL fot the data store.
You can use the instructions here: https://intelmq.readthedocs.io/en/latest/user/bots.html#sql
Please do let us know if it works for you or if the documentation is lacking something.
Take care and my best greetings to JP, Aaron.
Hi Aaron,
Thanks, I'll try and feed back! Also please be kindly informed this instance runs on FreeBSD12 B-)
Thanks!
On 11.03.2021, at 11:19, moto kawasaki moto@kawasaki3.org wrote:
Hi Aaron,
Thanks, I'll try and feed back! Also please be kindly informed this instance runs on FreeBSD12 B-)
\o/ whohooo!!
Shall we make a FreeBSD port file for it?
Best, Aaron.
Thanks!
-- moto kawasaki moto@kawasaki3.org +81-90-2464-8454
on Thu, 11 Mar 2021 11:14:00 +0100, "L. Aaron Kaplan" aaron@lo-res.org wrote:
Dear Moto,
(..)
Finally, you can check the output of the botnet (your DB, a MISP instance, whatever you have) to make sure that what your bots have collected has been processed properly.
I am lost here. Can someone tell me which manual page I should refer, please? I'd use PostgreSQL fot the data store.
You can use the instructions here: https://intelmq.readthedocs.io/en/latest/user/bots.html#sql
Please do let us know if it works for you or if the documentation is lacking something.
Take care and my best greetings to JP, Aaron.
-- List settings: https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users IntelMQ Documentation: https://intelmq.readthedocs.io/
\o/ whohooo!!
Shall we make a FreeBSD port file for it?
Yes, Sir! I am surely eager to try that!
Thanks