- IntelMQ-users

IntelMQ Issue
by Thakur, Richu 12 Mar '21

12 Mar '21

Hi Team, We are facing issue in accessing and using IntelMQ setup and the manager. We have integrated the setup with MongoDB and Elastic search though as we have checked we are unable to access anything right now. Also, we are wondering if the issue is taking place since IntelMQ is in maintenance. Please suggest us a wayout. Thanks & Regards, Richu Thakur KPMG - Cyber Security Pebble Beach, Embassy Golf Links, Bengaluru - 560071 Mobile : +91 7009151798 | Email : richuthakur(a)kpmg.com<mailto:richuthakur@kpmg.com> ********************************************************************** KPMG (in India) allows reasonable personal use of the e-mail system. Views and opinions expressed in these communications do not necessarily represent those of KPMG (in India). ******************************************************************************************************* DISCLAIMER The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading "Received in error," send to postmaster1(a)kpmg.com, then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing KPMG client engagement letter. Opinions, conclusions and other information in this e-mail and any attachments that do not relate to the official business of the firm are neither given nor endorsed by it. KPMG cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. KPMG, an Indian partnership and a member firm of KPMG International Cooperative ("KPMG International"), a Swiss entity that serves as a coordinating entity for a network of independent firms operating under the KPMG name. KPMG International Cooperative (“KPMG International”) provides no services to clients. Each member firm of KPMG International Cooperative (“KPMG International”) is a legally distinct and separate entity and each describes itself as such. “Notwithstanding anything inconsistent contained in the meeting invite to which this acceptance pertains, this acceptance is restricted solely to confirming my availability for the proposed call and should not be construed in any manner as acceptance of any other terms or conditions. Specifically, nothing contained herein may be construed as an acceptance (or deemed acceptance) of any request or notification for recording of the call, which can be done only if it is based on my explicit and written consent and subject to the terms and conditions on which such consent has been granted” *******************************************************************************************************

4 10

where can I see data gathered by intelmq?
by moto kawasaki 12 Mar '21

12 Mar '21

Dear intelmq-users list, I've just installed intelmq 2.3.0 via pypi and run it via `intelmqctl start`, and I can see several python processes are running with intelmq user in top command. Can I see some data that is collected by intelmq bots at this stage? If yes, where should I find it? Now I reckon intelmq-manager is the answer to the above question, and I am trying to figure out how to use intelmq-manager web interface. (pypi installation has done) Any suggestions will be appreciated. Thank you! Best Regards. -- moto kawasaki <moto(a)kawasaki3.org> +81-90-2464-8454

3 6

Classification of malware itself in IntelMQ
by Sebastian Wagner 08 Mar '21

08 Mar '21

Dear IntelMQ community, sorry for cross-posting, but I think this topic should be discussed in a wider group. IntelMQ always followed the Reference Security Incident Taxonomy (short: RSIT)[0] and its predecessor for its 'classification.taxonomy/type' fields. The Classification column in the RSIT corresponds to our "classification.taxonomy" field, and the RSIT's second column (currently called Incident examples) corresponds to our "classification.type" field. "classification.identifier" is an optional third level free-text field to give more specific context.[1] Due to historical reasons and changes on both sides - IntelMQ as well as the RSIT -, IntelMQ's classification scheme deviated a bit from the RSIT over time. I'm working on aligning them again for 3.0, which works straightforward in most cases. But for one case, I need your input. The predecessor of the RSIT (the eCSIRT.net taxonomy)[2] used the malicious code taxonomy differently: To classify malware itself into categories, like virus, worm, trojan, etc. The RSIT never did that, as classifying malware is never unambiguous and there are plenty of existing classification scheme out there, which do this already. Also, the focus of the RSIT is different, as it classifies the incidents/events, not malware samples. And for this reason, IntelMQ had (until < 3.0.0) the classification.type "malware" in IntelMQ. Most of the usages were wrong anyway, and should have been infected-device, malware-distribution or something else anyway. There is only one usage in IntelMQ, which can not be changed. And that one is really about malware itself (or: the hashes of samples) as used in the GitHub Feed parser[3] and the FireEye Parser[4]. But the issue is more generic, as we need to decide anyway, how we want to deal with such malware-IoCs. A malware (hash) does not fit into the RSIT. It's neither an Infected System, a C2 Server, Malware Distribution nor Malware Configuration. It's just a malware (hash). I see four options: 1) Deviate from the RSIT and just use 'classification.taxonomy' = 'Malicious Code' and 'classification.type' = 'malware' 2) Deviate slightly less from the RSIT and use 'classification.taxonomy' = 'other' and 'classification.type' = 'malware' 3) Adhere strictly to the RSIT and use 'classification.taxonomy' = 'other' and 'classification.type' = 'other' and "classification.identifier" = 'malware' 4) IntelMQ does not support this use case In cases 1) and 2) "classification.identifier" could be used to specify what the event is about, e.g. "hash", or the malware family. I'm currently in favor of option 2), as we can keep the meaning of "Malicious Code" in sync with the RSIT and still support the use-case sufficiently. But my opinion could change during the discussion :) Do you see any more options than I listed above? What do you favor? best regards Sebastian [0]: https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/… [1]: https://intelmq.readthedocs.io/en/latest/dev/data-harmonization.html#classi… [2]: https://www.trusted-introducer.org/Incident-Classification-Taxonomy.pdf [3]: https://github.com/certtools/intelmq/blob/f7507ca2643fe8ddb3817c9be1209504e… [4]: https://github.com/certtools/intelmq/pull/1745 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

3 4

IntelMQ & API & Manager Version 2.3.0!
by Sebastian Wagner 04 Mar '21

04 Mar '21

Dear community, Another important intermediate step on the way to IntelMQ 3.0 is completed - IntelMQ 2.3.0 is really final as of today. Many thanks to all the contributors and supporters around the world - the major changes would never be possible without you! One thing you will immediately notice its a completely new component: the IntelMQ API. It originates from the IntelMQ Manager, but is a complete rewrite of it's backend in Python (finally!) financed by SUNET and realised by Intevation. We have then split the Backend off into a separate API. This means, that to run the Manager, you need the API as well. The installation instructions: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade instructions: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html All packages have been published to pypi, the deb/rpm-repositories and dockerhub. You can read a summary of the most important changes here: https://cert.at/en/blog/2021/3/intelmq-230-api-docker-shadowserver-reports-… The new or heavily changed bots are: * CZ.nic HAAS and PROKI Parsers, by Filip Pokorný and Edvard Rejthar (CSIRT.CZ) * ESET Collector and Parser, by Mikk Margus Möll (CERT.EE) * Kafka Collector, by Birger Schacht (CERT.at) * Key-Value Parser, by Karl-Johan Karlsson (Linköping University) * Request Tracker Output, by Marius Urkis (NRDCS.LT) * Shadowserver Reports API and JSON Parser, by Birger Schacht (CERT.at) * Splunk Saved Search Expert, by Karl-Johan Karlsson (Linköping University) * Threshold Expert, by Karl-Johan Karlsson (Linköping University) * Shadowserver CSV & JSON Parser: Support for the feeds MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS and fixes for existing feed mappings, by Sebastian Waldbauer and Sebastian Wagner (CERT.at) * HTTP collector: PGP signature check functionality, by sinus-x * Several Experts (1, 2, 3, 4): Integrated local database update mechanisms, by Filip Pokorný (CSIRT.CZ) Please find below the full changelog. best regards Sebastian IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the minimum supported Python version is 3.6. ### Configuration ### Core - `intelmq.lib.bot`: - `ParserBot.recover_line_json_stream`: Make `line` parameter optional, as it is not needed for this method (by Sebastian Wagner). - `Bot.argparser`: Added class method `_create_argparser` (returns `argparse.ArgumentParser`) for easy command line arguments parsing (PR#1586 by Filip Pokorný). - Runtime configuration does not necessarily need a parameter entry for each block. Previously at least an empty block was required (PR#1604 by Filip Pokorný). - Allow setting the pipeline host and the Redis cache host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer). - Better logging message for SIGHUP handling if the handling of the signal is not delayed (by Sebastian Wagner). - `intelmq.lib.upgrades`: - Add upgrade function for removal of *HPHosts Hosts file* feed and `intelmq.bots.parsers.hphosts` parser (#1559, by Sebastian Wagner). - `intelmq.lib.exceptions`: - `PipelineError`: Remove unused code to format exceptions (by Sebastian Wagner). - `intelmq.lib.utils`: - `create_request_session_from_bot`: - Changed bot argument to optional, uses defaults.conf as fallback, renamed to `create_request_session`. Name `create_request_session_from_bot` will be removed in version 3.0.0 (PR#1524 by Filip Pokorný). - Fixed setting of `http_verify_cert` from defaults configuration (PR#1758 by Birger Schacht). - `log`: Use `RotatingFileHandler` for allow log file rotation without external tools (PR#1637 by Vasek Bruzek). - `intelmq.lib.harmonization`: - The `IPAddress` type sanitation now accepts integer IP addresses and converts them to the string representation (by Sebastian Wagner). - `DateTime.parse_utc_isoformat`: Add parameter `return_datetime` to return `datetime` object instead of string ISO format (by Sebastian Wagner). - `DateTime.convert`: Fix `utc_isoformat` format, it pointed to a string and not a function, causing an exception when used (by Sebastian Wagner). - `DateTime.from_timestamp`: Ensure that time zone information (`+00:00`) is always present (by Sebastian Wagner). - `DateTime.__parse` now handles OverflowError exceptions from the dateutil library, happens for large numbers, e.g. telehpone numbers (by Sebastian Wagner). - `intelmq.lib.upgrades`: - Added upgrade function for CSV parser parameter misspelling (by Sebastian Wagner). - Check for existence of collector and parser for the obsolete Malware Domain List feed and raise warning if found (#1762, PR#1771 by Birger Schacht). ### Development - `intelmq.bin.intelmq_gen_docs`: - Add bot name to the resulting feed documentation (PR#1617 by Birger Schacht). - Merged into `docs/autogen.py` (PR#1622 by Birger Schacht). ### Bots #### Collectors - `intelmq.bots.collectors.eset.collector`: Added (PR#1554 by Mikk Margus Möll). - `intelmq.bots.collectors.http.collector_http`: - Added PGP signature check functionality (PR#1602 by sinus-x). - If status code is not 2xx, the request's and response's headers and body are logged in debug logging level (#1615, by Sebastian Wagner). - `intelmq.bots.collectors.kafka.collector`: Added (PR#1654 by Birger Schacht, closes #1634). - `intelmq.bots.collectors.xmpp.collector`: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht). - `intelmq.bots.collectors.shadowserver.collector_api`: - Added (#1683, PR#1700 by Birger Schacht). - Change file names in the report to `.json` instead of the original and wrong `.csv` (PR#1769 by Sebastian Wagner). - `intelmq.bots.collectors.mail`: Add content of the email's `Date` header as `extra.email_date` to the report in all email collectors (PR#1749 by aleksejsv and Sebastian Wagner). - `intelmq.bots.collectors.http.collector_http_stream`: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner). - `intelmq.bots.collectors.shodan.collector_stream`: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner). - `intelmq.bots.collectors.twitter.collector_twitter`: - Proper input validation in URLs using urllib. CWE-20, found by GitHub's CodeQL (PR#1754 by Sebastian Wagner). - Limit replacement ("pastebin.com", "pastebin.com/raw") to a maximum of one (PR#1754 by Sebastian Wagner). #### Parsers - `intelmq.bots.parsers.eset.parser`: Added (PR#1554 by Mikk Margus Möll). - Ignore invalid "NXDOMAIN" IP addresses (PR#1573 by Mikk Margus Möll). - `intelmq.bots.parsers.hphosts`: Removed, feed is unavailable (#1559, by Sebastian Wagner). - `intelmq.bots.parsers.cznic.parser_haas`: Added (PR#1560 by Filip Pokorný and Edvard Rejthar). - `intelmq.bots.parsers.cznic.parser_proki`: Added (PR#1599 by sinus-x). - `intelmq.bots.parsers.key_value.parser`: Added (PR#1607 by Karl-Johan Karlsson). - `intelmq.bots.parsers.generic.parser_csv`: Added new parameter `compose_fields` (by Sebastian Wagner). - `intelmq.bots.parsers.shadowserver.parser_json`: Added (PR#1700 by Birger Schacht). - `intelmq.bots.parsers.shadowserver.config`: - Fixed mapping for Block list feed to accept network ranges in CIDR notation (#1720, PR#1728 by Sebastian Waldbauer). - Added mapping for new feed MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS (#1716, #1726, #1733, PR#1732, PR#1735, PR#1736 by Sebastian Waldbauer). - Ignore value `0` for `source.asn` and `destination.asn` in all mappings to avoid parsing errors (PR#1769 by Sebastian Wagner). - `intelmq.bots.parsers.abusech.parser_ip`: Adapt to changes in the Feodo Tracker Botnet C2 IP Blocklist feed (PR#1741 by Thomas Bellus). - `intelmq.bots.parsers.malwaredomainlist`: Removed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht). #### Experts - `intelmq.bots.experts.rfc1918.expert`: - Add support for ASNs (PR#1557 by Mladen Markovic). - Speed improvements. - More output in debug logging mode (by Sebastian Wagner). - Checks parameter length on initialization and in check method (by Sebastian Wagner). - `intelmq.bots.experts.gethostbyname.expert`: - Added parameter `fallback_to_url` and set to True (PR#1586 by Edvard Rejthar). - Added parameter `gaierrors_to_ignore` to optionally ignore other `gethostbyname` errors (#1553). - Added parameter `overwrite` to optionally overwrite existing IP addresses (by Sebastian Wagner). - `intelmq.bots.experts.asn_lookup.expert`: - Added `--update-database` option (PR#1524 by Filip Pokorný). - The script `update-asn-data` is now deprecated and will be removed in version 3.0. - `intelmq.bots.experts.maxmind_geoip.expert`: - Added `--update-database` option (PR#1524 by Filip Pokorný). - Added `license_key` parameter (PR#1524 by Filip Pokorný). - The script `update-geoip-data` is now deprecated and will be removed in version 3.0. - `intelmq.bots.experts.tor_nodes.expert`: - Added `--update-database` option (PR#1524 by Filip Pokorný). - The script `update-tor-nodes` is now deprecated and will be removed in version 3.0. - `intelmq.bots.experts.recordedfuture_iprisk.expert`: - Added `--update-database` option (PR#1524 by Filip Pokorný). - Added `api_token` parameter (PR#1524 by Filip Pokorný). - The script `update-rfiprisk-data` is now deprecated and will be removed in version 3.0. - Added `intelmq.bots.experts.threshold` (PR#1608 by Karl-Johan Karlsson). - Added `intelmq.bots.experts.splunk_saved_search.expert` (PR#1666 by Karl-Johan Karlsson). - `intelmq.bots.experts.sieve.expert`: - Added possibility to give multiple queue names for the `path` directive (#1462, by Sebastian Wagner). - Added possibility to run actions without filtering expression (#1706, PR#1708 by Sebastian Waldbauer). - Added datetime math operations (#1680, PR#1696 by Sebastian Waldbauer). - `intelmq.bots.experts.maxmind_geoip.expert`: - Fixed handing over of `overwrite` parameter to `event.add` (PR#1743 by Birger Schacht). #### Outputs - `intelmq.bots.outputs.rt`: Added Request Tracker output bot (PR#1589 by Marius Urkis). - `intelmq.bots.outputs.xmpp.output`: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht). - `intelmq.bots.outputs.smtp.output`: Fix sending to multiple recipients when recipients are defined by event-data (#1759, PR#1760 by Sebastian Waldbauer and Sebastian Wagner). ### Documentation - Feeds: - Add ESET URL and Domain feeds (by Sebastian Wagner). - Remove unavailable *HPHosts Hosts file* feed (#1559 by Sebastian Wagner). - Added CZ.NIC HaaS feed (PR#1560 by Filip Pokorný and Edvard Rejthar). - Added CZ.NIC Proki feed (PR#1599 by sinus-x). - Updated Abuse.ch URLhaus feed (PT#1572 by Filip Pokorný). - Added CERT-BUND CB-Report Malware infections feed (PR#1598 by sinus-x and Sebastian Wagner). - Updated Turris Greylist feed with PGP verification information (by Sebastian Wagner). - Fixed parsing of the `public` field in the generated feeds documentation (PR#1641 by Birger Schacht). - Change the `rate_limit` parameter of some feeds from 2 days (129600 seconds) to one day (86400 seconds). - Update the cAPTure Ponmocup Domains feed documentation (PR#1574 by Filip Pokorný and Sebastian Wagner). - Added Shadowserver Reports API (by Sebastian Wagner). - Change the `rate_limit` parameter for many feeds from 2 days to the default one day (by Sebastian Wagner). - Removed Malware Domain List feed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht). - Bots: - Enhanced documentation of RFC1918 Expert (PR#1557 by Mladen Markovic and Sebastian Wagner). - Enhanced documentation of SQL Output (PR#1620 by Edvard Rejthar). - Updated documentation for MaxMind GeoIP, ASN Lookup, TOR Nodes and Recorded Future experts to reflect new `--update-database` option (PR#1524 by Filip Pokorný). - Added documentation for Shadowserver API collector and parser (PR#1700 by Birger Schacht and Sebastian Wagner). - Add n6 integration documentation (by Sebastian Wagner). - Moved 'Orphaned Queues' section from the FAQ to the intelmqctl documentation (by Sebastian Wagner). - Generate documentation using Sphinx (PR#1622 by Birger Schacht). - The documentation is now available at https://intelmq.readthedocs.io/en/latest/ - Refactor documentation and fix broken syntax (#1639, PRs #1638 #1640 #1642 by Birger Schacht). - Integrate intelmq-manager and intelmq-api user documentation to provide unified documentation place (PR#1714 & PR#1714 by Birger Schacht). ### Packaging - Fix paths in the packaged logcheck rules (by Sebastian Wagner). - Build the sphinx documentation on package build (PR#1701 by Birger Schacht). - Ignore non-zero exit-codes for the `intelmqctl check` call in postinst (#1748, by Sebastian Wagner). ### Tests - Added tests for `intelmq.lib.exceptions.PipelineError` (by Sebastian Wagner). - `intelmq.tests.bots.collectors.http_collector.test_collector`: Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner). - `intelmq.tests.bots.outputs.restapi.test_output`: - Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner). - Add a test for checking the response status code (by Sebastian Wagner). - `intelmq.tests.bots.collectors.mail.test_collector_url`: Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner). - `intelmq.tests.bots.experts.ripe.test_expert`: Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner). - The test flag (environment variable) `INTELMQ_TEST_LOCAL_WEB` is no longer used (by Sebastian Wagner). - Added tests for `intelmq.harmonization.DateTime.parse_utc_isoformat` and `convert_fuzzy` (by Sebastian Wagner). - Move from Travis to GitHub Actions (PR#1707 by Birger Schacht). - `intelmq.lib.test`: - `test_static_bot_check_method` checks the bot's static `check(parameters)` method for any exceptions, and a valid formatted return value (#1505, by Sebastian Wagner). - `setUpClass`: Skip tests if cache was requests with `use_cache` member, but Redis is deactivated with the environment variable `INTELMQ_SKIP_REDIS` (by Sebastian Wagner). - `intelmq.tests.bots.experts.cymru_whois.test_expert`: - Switch from `example.com` to `ns2.univie.ac.at` for hopefully more stable responses (#1730, PR#1731 by Sebastian Waldbauer). - Do not test for exact expected values in the 6to4 network test, as the values are changing regularly (by Sebastian Wagner). - `intelmq.tests.bots.parsers.abusech`: Remove tests cases of discontinued feeds (PR#1741 by Thomas Bellus). - Activate GitHub's CodeQL Code Analyzing tool as GitHub Action (by Sebastian Wagner). ### Tools - `intelmqdump`: - Check if given queue is configured upon recovery (#1433, PR#1587 by Mladen Markovic). - `intelmqctl`: - `intelmq list queues`: `--sum`, `--count`, `-s` flag for showing total count of messages (#1408, PR#1581 by Mladen Markovic). - `intelmq check`: Added a possibility to ignore queues from the orphaned queues check (by Sebastian Wagner). - Allow setting the pipeline host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer). ### Contrib - EventDB: - Add SQL script for keeping track of the oldest inserted/update "time.source" information (by Sebastian Wagner). - Cron Jobs: The script `intelmq-update-data` has been renamed to `intelmq-update-database` (by Filip Pokorný). - Dropped utterly outdated contrib modules (by Sebastian Wagner): - ansible - vagrant - vagrant-ansible - logrotate: - Do not use the deprecated "copytruncate" option as intelmq re-opens the log anyways (by Sebastian Wagner). - Set file permissions to `0644` (by Sebastian Wagner). ### Known issues - Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952). - Corrupt dump files when interrupted during writing (#870). - CSV line recovery forces Windows line endings (#1597). - intelmqdump: Honor logging_path variable (#1605). - Timeout error in mail URL fetcher (#1621). - AMQP pipeline: get_queues needs to check vhost of response (#1746). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Error in parser module?
by Henrik Jensen - TeleDCIS 01 Mar '21

01 Mar '21

Hi everybody, I'm still new to IntelMQ. Could anyone tell me what to do about the following error (highlighted) in the Malware-Domain-List-Parser: Failed to parse line. Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/intelmq/lib/bot.py", line 978, in process events = list(filter(bool, value)) File "/usr/local/lib/python3.7/dist-packages/intelmq/bots/parsers/malwaredomainlist/parser.py", line 21, in parse_line event.add("time.source", row[0].replace('_', ' ') + " UTC") File "/usr/local/lib/python3.7/dist-packages/intelmq/lib/message.py", line 249, in add raise exceptions.InvalidValue(key, old_value) intelmq.lib.exceptions.InvalidValue: invalid value ' UTC' Bedste hilsner/regards Henrik Jensen TeleDCIS Tlf: +45 35 88 82 84 Mobil: +45 93 51 00 03 Mail: hj(a)teledcis.dk<mailto:hj@teledcis.dk> www.teledcis.dk

4 5

More specific CIDRs from a different country from RIPE?
by Bernhard Reiter 12 Feb '21

12 Feb '21

Hello IntelMQ-Users, if you are responsible to only deal with reports for a country and base your decisions on the RIPE database, how do you deal with more specific CIDRs that are from a different country, but within a CIDR that belongs to yours? See more details of the problem as seen from the ripe importer the intelmq-cb-mailgen solution uses: https://github.com/Intevation/intelmq-certbund-contact/issues/13 (Feel free to answer here or in the issue or personally.) Thanks in advance, Bernhard -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

3 2

Parameters for file-collectors
by Henrik Jensen - TeleDCIS 27 Jan '21

27 Jan '21

Hallo there, I'm new to IntelMQ. But nevertheless I'm wondering how you are getting IntelMQ to use files uploaded in a directory. My only knowledge of IntelMQ Is using the GUI I'm not a scripting guy at all. Could someone in this community please help with setting the right parameters for the "file-collector2 bot in the GUI of IntelMQ? In advance many thanks. Bedste hilsner/regards Henrik Jensen TeleDCIS Tlf: +45 35 88 82 84 Mobil: +45 93 51 00 03 Mail: hj(a)teledcis.dk<mailto:hj@teledcis.dk> www.teledcis.dk

2 3

Output issue
by Henrik Jensen - TeleDCIS 27 Jan '21

27 Jan '21

Hi Everybody, Could anyone of you lovely persons explain to me, what the following warning means and how to circumvent it? Bedste hilsner/regards Henrik Jensen TeleDCIS Tlf: +45 35 88 82 84 Mobil: +45 93 51 00 03 Mail: hj(a)teledcis.dk<mailto:hj@teledcis.dk> www.teledcis.dk

1 0

IntelMQ and Python 3.5
by Birger Schacht 22 Jan '21

22 Jan '21

Dear community, TLDR: We are thinking about dropping support for Python 3.5 with the release of IntelMQ 2.3. At the moment IntelMQ supports Python back to and including Python 3.5, which was released in 2015. The last version of Python 3.5 was 3.5.10 which was released this September and with that Python 3.5 has reached end-of-life. The two Distributions that ship Python 3.5 only are Debian 9 and Ubuntu 16.04. Debian 9 (Stretch) was superseded by Debian 10 (Buster) this summer, but does still get LTS support until June 2022. Support for Ubuntu 16.04 LTS (Xenial Xerus) will end in April 2021. Therefore dropping support for Python 3.5 would mean that we would also have to drop support for Debian 9 and Ubuntu 16.04. The upside would not only be cleaner code through improvements in Python 3.6 but we could also update the packaging code. Furthermore, there might be IntelMQ dependencies that drop support for Python 3.5 now that it is EOL which would limit the functionality of IntelMQ. So the basic question is, are there still (m)any setups out there that require IntelMQ to run with Python 3.5? cheers, Birger -- // Birger Schacht <schacht(a)cert.at> // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

4 3

IEP01: IntelMQ Configuration Handling
by Birger Schacht 18 Jan '21

18 Jan '21

Dear IntelMQ developers and users, below are a couple of ideas how to (hopefully) make configuration of IntelMQ easier. Feel free to give feedback, voice concerns or simply ask if there is something unclear. We plan to evaluate the feedback that emerged in two weeks (after the christmas holidays). # IntelMQ Configuration Handling (IntelMQ Enhancement Proposal 01) ## Format ### JSON At the moment, the configuration format of IntelMQ is JSON[^1]. It is parsed using the Python json library, which is part of the Python Standard Library. The downside of JSON is, that is is hard to read and and write for humans and it cannot contain comments. [^1]: https://docs.python.org/3/library/index.html ### YAML There is a proposal[^2] to use YAML as the default configuration format. YAML provides way better readability for humans and YAML supports single line comments. There are two Python YAML libraries out there, the one being PyYAML[^3] and the other being ruamel.yaml[^4]. The former is a project by the YAML project itself. The latter is a fork of the former and had much more activity over the years and better support of the standard. It seems that pyyaml caught up in the last few years. We don't need any edge cases, so both libraries would be good for configuration files. According to this issue[^5] pyyaml does not support “editing YAML whilst maintaining comments”, which might be a deal breaker, but this issue is from 2016, this might have changed. On the other hand, IntelMQ does not edit configuration at the moment. pyyaml and ruamel.yaml are available as package in all relevant Linux distributions. [^2]: https://github.com/gethvi/intelmq/blob/ideas/docs/Ideas.md#changing-configu… [^3]: https://pyyaml.org/ [^4]: https://yaml.readthedocs.io/en/latest/ [^5]: https://github.com/yaml/pyyaml/issues/46 ### INI The Python Standard Library also ships configparser[^6], which is a “configuration language which provides a structure similar to what’s found in Microsoft Windows INI files”. The files can contain comments, it comes with a [DEFAULT] section, which can be used for default values and the configuration files can contain variables. One downside is that all the configurations are Strings, which means we would have to do parsing ourself. [^6]: https://docs.python.org/3/library/configparser.html ### toml Tom's Obvious, Minimal Language is another contender for the role of IntelMQs configuration file format. It looks similar to the INI file format, but comes with various data types. It also allows comments. There is a Python library[^7] that seems to be very active. toml is also used as the format for the proposed pyproject.toml file and by the rust community for their package configuration files. toml's syntax for dictionaries is hard to read/write, harder than with JSON. [^7]: https://pypi.org/project/toml/ ### Further information * The summary on file formats on the PEP518 proposition: https://www.python.org/dev/peps/pep-0518/#other-file-formats * At the moment we are leaning towards YAML. Regarding the library, we would choose ruamel.yaml, because it seems to have a more active upstream and it can retain comments when it modifies a yaml file. ## Storage This part is about the question where do we store the configuration?. The ideas document[^8] on GitHub already proposes to remove the pipeline.conf and specifying the destination pipelines in the individual bot configuration part. The declaration of the source queue can be dropped then as well, as it follows a rule anyway. In addition to that, to make the setup of IntelMQ easier, the defaults.conf should be dropped. Default values should be set in the Bot classes respectively in the IntelMQ process managers, but there is no need for a separate file. Another question is, if every bot should have their own configuration file. Some users wish to be able to start a bot without having to rely on IntelMQ, but at the moment, the bot gets the configuration from IntelMQ's runtime.conf. If we want to support the request to be able to pass individual configurations to bots, we could allow users to pass a separate configuration file to the bot (i.e. using `-c /path/to/config.$ext`). If that file is not set or does not contain the bots id, it is ignored and IntelMQ's runtime.conf is used as usual. If it does exists, the global runtime.conf is still parsed (if it exists - it should also be possible to run a bot without a runtime.conf) but only the values that are not set in the individual configuration file are considered. This individual configuration file would also allow a bot to be run in a docker environment without having to set any environment variables. This would make configuration handling probably easier, because then configuration settings could be stored in a file (and managed by a configuration management system) and the configuration file could contain comments. Proposal: * IntelMQ gets one global configuration file for all the bots and the pipeline.conf will be removed * This global configuration file is `${PREFIX}/etc/intelmq/intelmq.$ext`. If it does not exists or does not define any bots, IntelMQ should exit gracefully. The file extension depends on the chosen format. * The global configuration file contains an array of bot configurations with bot-ids as keys. * Every bot reads the global configuration file and extracts their own settings (as usual). * Every bot handles 0 to n `-c /path/to/configurationfile.$ext` flags, which are treated the same way as the global configuration file. The further ahead the configuration file in the commandline, the stronger the content (this allows us to have multiple non-global configuration files (i.e. for multiple groups)) Example: ``` > botcommand bot-id -c /etc/bots/botname.$ext -c /etc/bots/groups/group_foo.$ext ``` * Every bot also consults the environment and the values that are set their overwrite the values in any configuration file * There are also configuration files which list settings that are not bot specific, i.e. via a reserved key default (successor of the defaults.conf file) or group:id, those are also handled like other configuration files, but the bot does not compare its name to the key of the configuration. All the evaluated configuration formats provide the possibility to arrange the configuration parameters in hierarchies. To make the configuration files more readable, IntelMQ should make use of this hierarchy instead of denoting the different hierarchy levels with underscores. So instead of writing `http_proxy` the http parameter would have a childparameter proxy. For backwards compatibility and cases where the underscore does not imply hierarchy, the underscore notation will still work. In addition, IntelMQ should also make use of environment variables - those are still denoted using an underscore as delimiter and are prepended with `INTELMQ`: `INTELMQ_HTTP_PROXY`. [^8]: https://github.com/gethvi/intelmq/blob/ideas/docs/Ideas.md ### Caveats There are configuration settings, that do not really concern the bot- for example the type of process manager, that should be used to run the bot. In an ideal setup, the bot should be totally indifferent as to if it runs in a Docker container, on bare metal, in a SystemD unit file or with SupervisorD. This decision should only concern the tool managing all the bots (intelmqctl or in the future intelmq-api (which at the moment uses intelmqctl)). Another example is the enabled setting. At the moment, those are part of the individual bot configuration, but it might make sense to move them to a management.conf configuration file which is only for managing the individual bots, but not for configuring their parameters (this file would then also (for every bot) have a field that lists the configuration files the bot should consider when reading its configuration). On the other hand, this might make the configuration more complex again, now that we are trying to merge pipeline.conf and runtime.conf. We could also decide to make those configuration settings be part of the global configuration file, given that the individual bots should anyway simply ignore settings they do not know how to handle. ### Overriding by command line parameters If needed, a user can override specific bot settings using the -p switch (i.e. `-p redis_cache=example.com`). This should be easy to implement, in the best case scenario this is only one line of additional code in the Bot class. ### Examples A global configuration file with multiple bots /etc/intelmq/intelmq.yml ``` - shodan1: module: intelmq.bots.collectors.shodan.collector - mylittlebot23: module: intelmq.bots.expert.asn_lookup.expert http: proxy: http://myproxy.tld:80 - fop1: module: intelmq.bots.outputs.file output: filename: /dev/null ``` We can run a bot with intelmq-bot shodan1 which is the same as `intelmq-bot shodan1 -c /etc/intelmq/intelmq.yml` Another configuration file with multiple bots /root/intelmq-bots-managed-by-root: ``` - shodan2: module: intelmq.bots.collectors.shodan.collector - fop1: module: intelmq.bots.outputs.file output: filename: /var/log/fop1.log ``` We can run a bot with `intelmq-bot shodan2 -c /root/intelmq-bots-managed-by-root`; We can run a bot using `intelmq-bot fop1 -c /root/intelmq-bots-managed-by-root` which would then send output to `/var/log/fop1.log`. A configuration for a group in /etc/intelmq/collector-group.yml ``` - group:collectors http: proxy: http://thirdparty.proxy.tld:9000 ``` We can run a bot with intelmq-bot `mylittlebot23 -c /etc/intelmq/collector-group.yml` which uses the third-party proxy. ## Internal handling Every bot class defines their own settings as class variables. Every class variable has to be typed. Every class variable should be set to a reasonable default, otherwise None. The init of the (abstract) Bot class should load all the relevant configuration files and then overwrite the settings. If a setting is still None and the value of the setting is vital for the functionality of the bot, the bot should stop and emit a meaningful error message. For the most common types of settings, there should be Python objects to check the values. Value checking should only be done after all the configurations are merged. -- // Birger Schacht <schacht(a)cert.at> // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

4 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users