[IntelMQ-users] where can I see data gathered by intelmq?

[moto kawasaki]

Hi Jonathan and list members,

Thank you very much for your instructions, Jonathan!
Please kindly let me ask more. I am very new to intelmq, and this is
my first run :-).

> In order to check "what is going on" inside your IntelMQ botnet, you
> could use the following commands:
> "sudo -u <intelmq_user_account> intelmqctl status" -> this one
> checks which bots are running, which are stopped and which are
> disabled.

$ intelmqctl status
Bot cymru-whois-expert is running.
Bot deduplicator-expert is running.
Bot feodo-tracker-browse-collector is running.
Bot feodo-tracker-browse-parser is stopped.
Bot file-output is running.
Bot gethostbyname-1-expert is running.
Bot gethostbyname-2-expert is running.
Bot malc0de-parser is running.
Bot malc0de-windows-format-collector is running.
Bot spamhaus-drop-collector is running.
Bot spamhaus-drop-parser is running.
Bot taxonomy-expert is running.
Bot url2fqdn-expert is running.

> "sudo -u <intelmq_user_account> intelmqctl list queues" -> this one
> displays the current amount of messages stored in the internal or
> external bots queues. (use "-q" at the end if you want to hide
> queues with 0 messages)

$ intelmqctl list queues
cymru-whois-expert-queue - 0
cymru-whois-expert-queue-internal - 0
deduplicator-expert-queue - 0
deduplicator-expert-queue-internal - 0
feodo-tracker-browse-parser-queue - 1
feodo-tracker-browse-parser-queue-internal - 0
file-output-queue - 0
file-output-queue-internal - 0
gethostbyname-1-expert-queue - 0
gethostbyname-1-expert-queue-internal - 0
gethostbyname-2-expert-queue - 0
gethostbyname-2-expert-queue-internal - 0
malc0de-parser-queue - 0
malc0de-parser-queue-internal - 0
spamhaus-drop-parser-queue - 0
spamhaus-drop-parser-queue-internal - 0
taxonomy-expert-queue - 0
taxonomy-expert-queue-internal - 0
url2fqdn-expert-queue - 0
url2fqdn-expert-queue-internal - 0

> "cat /var/log/intelmq/<bot_name>.log" will display the bot output
> (by default only info and error messages are shown, debug message
> are hidden -> am I right?)

Yes, some INFO messages are shown in the log files, so that I reckon
they are anyway working.

Intelmq's setup.py might miss the dependency for beautiflsoap4 in
REQUIRES, according to the ERROR message in
feodo-tracker-browse-parser-queue.log (quoted below).
(See also https://github.com/certtools/intelmq/blob/develop/setup.py)

  | 2021-03-11 16:30:47,166 - feodo-tracker-browse-parser - INFO - Bot is starting.
  | 2021-03-11 16:30:47,168 - feodo-tracker-browse-parser - ERROR - Bot initialization failed.
  | Traceback (most recent call last):
  |   File "/usr/local/lib/python3.7/site-packages/intelmq/lib/bot.py", line 164, in __init__
  |     self.init()
  |   File "/usr/local/lib/python3.7/site-packages/intelmq/bots/parsers/html_table/parser.py", line 37, in init
  |     raise MissingDependencyError("beautifulsoup4")
  | intelmq.lib.exceptions.MissingDependencyError: Could not load dependency 'be  | autifulsoup4', please install it with apt/yum/dnf/zypper (possibly named python3-beautifulsoup4) or pip3.
  | 2021-03-11 16:30:47,171 - feodo-tracker-browse-parser - INFO - Bot stopped.

> Finally, you can check the output of the botnet (your DB, a MISP
> instance, whatever you have) to make sure that what your bots have
> collected has been processed properly.

I am lost here.
Can someone tell me which manual page I should refer, please?
I'd use PostgreSQL fot the data store.

Thank you in advance!




Best Regards,



-- 
moto kawasaki <moto at kawasaki3.org> +81-90-2464-8454




> You could also manually run your bots with "sudo -u
> <intelmq_user_account> intelmqctl run <bot_name> -l DEBUG" so you
> can check what the bot is doing in real time.

> 
> Best regards,
> Jonathan
> 
> --
> Jonathan SCOUPREMAN | jscoupreman at excellium-services.lu | PGP Key ID: 0xAD971C07
> CERT-XLM | cert at excellium-services.com | PGP Key ID: 0xD74E5AC0
> CERT-XLM Incident Handler @ excellium-services.com
> Excellium Services S.A. | 5 rue Goell L-5326 Contern
> Mobile: +352 691 982 790
> Emergency: +352 262 039 64 708 | emergency at excellium-services.com | PGP Key ID: 0x42662EFE
> 
> -----Original Message-----
   <snip>