IntelMQ-dev

intelmq-dev@lists.cert.at

3 participants
238 discussions

gitlab docker
by Marius Karotkis 09 Mar '21

09 Mar '21

Hello, maybe someone can help. I try run test from gitlab with gitlab runner on docker. But I get errors on stage build: ERROR: Cannot install intelmq 2.3.0.1 (from /builds/deployment/intelmq/dist/intelmq-2.3.0.6374.tar.gz) and intelmq 2.3.0.6374 (from /builds/deployment/intelmq/dist/intelmq-2.3.0.6374-py2.py3-none-any.whl) because these package versions have conflicting dependencies. The conflict is caused by: The user requested intelmq 2.3.0.6374 (from /builds/deployment/intelmq/dist/intelmq-2.3.0.6374-py2.py3-none-any.whl) The user requested intelmq 2.3.0.1 (from /builds/deployment/intelmq/dist/intelmq-2.3.0.6374.tar.gz)

2 1

Re: [IntelMQ-dev] [IntelMQ-users] Classification of malware itself in IntelMQ
by Sebastian Wagner 08 Mar '21

08 Mar '21

Hi, Thanks Filip for sharing your opinion. As there were no further comments on this topic, I'll implement option 2. Sebastian On 2/24/21 10:32 AM, Filip Pokorny wrote: > Hi Sebastian, > >> I'm currently in favor of option 2), as we can keep the meaning of "Malicious Code" in sync with the RSIT and still support the use-case sufficiently. But my opinion could change during the discussion :) > I agree. This combined with "malware.name", "malware.version" and > "malware.hash" seems good enough to describe malware for the IntelMQ > use-case. > > Best Regards, > Filip > > > On 2/22/21 11:24 AM, Sebastian Wagner wrote: >> Dear IntelMQ community, >> >> sorry for cross-posting, but I think this topic should be discussed in a >> wider group. >> >> IntelMQ always followed the Reference Security Incident Taxonomy (short: >> RSIT)[0] and its predecessor for its 'classification.taxonomy/type' >> fields. The Classification column in the RSIT corresponds to our >> "classification.taxonomy" field, and the RSIT's second column (currently >> called Incident examples) corresponds to our "classification.type" >> field. "classification.identifier" is an optional third level free-text >> field to give more specific context.[1] >> >> Due to historical reasons and changes on both sides - IntelMQ as well as >> the RSIT -, IntelMQ's classification scheme deviated a bit from the RSIT >> over time. I'm working on aligning them again for 3.0, which works >> straightforward in most cases. But for one case, I need your input. >> >> The predecessor of the RSIT (the eCSIRT.net taxonomy)[2] used the >> malicious code taxonomy differently: To classify malware itself into >> categories, like virus, worm, trojan, etc. The RSIT never did that, as >> classifying malware is never unambiguous and there are plenty of >> existing classification scheme out there, which do this already. Also, >> the focus of the RSIT is different, as it classifies the >> incidents/events, not malware samples. >> >> And for this reason, IntelMQ had (until < 3.0.0) the classification.type >> "malware" in IntelMQ. Most of the usages were wrong anyway, and should >> have been infected-device, malware-distribution or something else >> anyway. There is only one usage in IntelMQ, which can not be changed. >> And that one is really about malware itself (or: the hashes of samples) >> as used in the GitHub Feed parser[3] and the FireEye Parser[4]. But the >> issue is more generic, as we need to decide anyway, how we want to deal >> with such malware-IoCs. >> >> A malware (hash) does not fit into the RSIT. It's neither an Infected >> System, a C2 Server, Malware Distribution nor Malware Configuration. >> It's just a malware (hash). I see four options: >> >> 1) Deviate from the RSIT and just use 'classification.taxonomy' = >> 'Malicious Code' and 'classification.type' = 'malware' >> 2) Deviate slightly less from the RSIT and use 'classification.taxonomy' >> = 'other' and 'classification.type' = 'malware' >> 3) Adhere strictly to the RSIT and use 'classification.taxonomy' = >> 'other' and 'classification.type' = 'other' and >> "classification.identifier" = 'malware' >> 4) IntelMQ does not support this use case >> >> In cases 1) and 2) "classification.identifier" could be used to specify >> what the event is about, e.g. "hash", or the malware family. >> >> I'm currently in favor of option 2), as we can keep the meaning of >> "Malicious Code" in sync with the RSIT and still support the use-case >> sufficiently. But my opinion could change during the discussion :) >> >> Do you see any more options than I listed above? What do you favor? >> >> best regards >> Sebastian >> >> [0]: >> https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/… >> [1]: >> https://intelmq.readthedocs.io/en/latest/dev/data-harmonization.html#classi… >> [2]: https://www.trusted-introducer.org/Incident-Classification-Taxonomy.pdf >> [3]: >> https://github.com/certtools/intelmq/blob/f7507ca2643fe8ddb3817c9be1209504e… >> [4]: https://github.com/certtools/intelmq/pull/1745 >> >> >> -- >> // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 >> // CERT Austria - https://www.cert.at/ >> // Eine Initiative der nic.at GmbH - https://www.nic.at/ >> // Firmenbuchnummer 172568b, LG Salzburg >> >> >> _______________________________________________ >> IntelMQ-dev mailing list >> IntelMQ-dev(a)lists.cert.at >> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev >> -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ & API & Manager Version 2.3.0!
by Sebastian Wagner 04 Mar '21

04 Mar '21

Dear community, Another important intermediate step on the way to IntelMQ 3.0 is completed - IntelMQ 2.3.0 is really final as of today. Many thanks to all the contributors and supporters around the world - the major changes would never be possible without you! One thing you will immediately notice its a completely new component: the IntelMQ API. It originates from the IntelMQ Manager, but is a complete rewrite of it's backend in Python (finally!) financed by SUNET and realised by Intevation. We have then split the Backend off into a separate API. This means, that to run the Manager, you need the API as well. The installation instructions: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade instructions: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html All packages have been published to pypi, the deb/rpm-repositories and dockerhub. You can read a summary of the most important changes here: https://cert.at/en/blog/2021/3/intelmq-230-api-docker-shadowserver-reports-… The new or heavily changed bots are: * CZ.nic HAAS and PROKI Parsers, by Filip Pokorný and Edvard Rejthar (CSIRT.CZ) * ESET Collector and Parser, by Mikk Margus Möll (CERT.EE) * Kafka Collector, by Birger Schacht (CERT.at) * Key-Value Parser, by Karl-Johan Karlsson (Linköping University) * Request Tracker Output, by Marius Urkis (NRDCS.LT) * Shadowserver Reports API and JSON Parser, by Birger Schacht (CERT.at) * Splunk Saved Search Expert, by Karl-Johan Karlsson (Linköping University) * Threshold Expert, by Karl-Johan Karlsson (Linköping University) * Shadowserver CSV & JSON Parser: Support for the feeds MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS and fixes for existing feed mappings, by Sebastian Waldbauer and Sebastian Wagner (CERT.at) * HTTP collector: PGP signature check functionality, by sinus-x * Several Experts (1, 2, 3, 4): Integrated local database update mechanisms, by Filip Pokorný (CSIRT.CZ) Please find below the full changelog. best regards Sebastian IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the minimum supported Python version is 3.6. ### Configuration ### Core - `intelmq.lib.bot`: - `ParserBot.recover_line_json_stream`: Make `line` parameter optional, as it is not needed for this method (by Sebastian Wagner). - `Bot.argparser`: Added class method `_create_argparser` (returns `argparse.ArgumentParser`) for easy command line arguments parsing (PR#1586 by Filip Pokorný). - Runtime configuration does not necessarily need a parameter entry for each block. Previously at least an empty block was required (PR#1604 by Filip Pokorný). - Allow setting the pipeline host and the Redis cache host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer). - Better logging message for SIGHUP handling if the handling of the signal is not delayed (by Sebastian Wagner). - `intelmq.lib.upgrades`: - Add upgrade function for removal of *HPHosts Hosts file* feed and `intelmq.bots.parsers.hphosts` parser (#1559, by Sebastian Wagner). - `intelmq.lib.exceptions`: - `PipelineError`: Remove unused code to format exceptions (by Sebastian Wagner). - `intelmq.lib.utils`: - `create_request_session_from_bot`: - Changed bot argument to optional, uses defaults.conf as fallback, renamed to `create_request_session`. Name `create_request_session_from_bot` will be removed in version 3.0.0 (PR#1524 by Filip Pokorný). - Fixed setting of `http_verify_cert` from defaults configuration (PR#1758 by Birger Schacht). - `log`: Use `RotatingFileHandler` for allow log file rotation without external tools (PR#1637 by Vasek Bruzek). - `intelmq.lib.harmonization`: - The `IPAddress` type sanitation now accepts integer IP addresses and converts them to the string representation (by Sebastian Wagner). - `DateTime.parse_utc_isoformat`: Add parameter `return_datetime` to return `datetime` object instead of string ISO format (by Sebastian Wagner). - `DateTime.convert`: Fix `utc_isoformat` format, it pointed to a string and not a function, causing an exception when used (by Sebastian Wagner). - `DateTime.from_timestamp`: Ensure that time zone information (`+00:00`) is always present (by Sebastian Wagner). - `DateTime.__parse` now handles OverflowError exceptions from the dateutil library, happens for large numbers, e.g. telehpone numbers (by Sebastian Wagner). - `intelmq.lib.upgrades`: - Added upgrade function for CSV parser parameter misspelling (by Sebastian Wagner). - Check for existence of collector and parser for the obsolete Malware Domain List feed and raise warning if found (#1762, PR#1771 by Birger Schacht). ### Development - `intelmq.bin.intelmq_gen_docs`: - Add bot name to the resulting feed documentation (PR#1617 by Birger Schacht). - Merged into `docs/autogen.py` (PR#1622 by Birger Schacht). ### Bots #### Collectors - `intelmq.bots.collectors.eset.collector`: Added (PR#1554 by Mikk Margus Möll). - `intelmq.bots.collectors.http.collector_http`: - Added PGP signature check functionality (PR#1602 by sinus-x). - If status code is not 2xx, the request's and response's headers and body are logged in debug logging level (#1615, by Sebastian Wagner). - `intelmq.bots.collectors.kafka.collector`: Added (PR#1654 by Birger Schacht, closes #1634). - `intelmq.bots.collectors.xmpp.collector`: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht). - `intelmq.bots.collectors.shadowserver.collector_api`: - Added (#1683, PR#1700 by Birger Schacht). - Change file names in the report to `.json` instead of the original and wrong `.csv` (PR#1769 by Sebastian Wagner). - `intelmq.bots.collectors.mail`: Add content of the email's `Date` header as `extra.email_date` to the report in all email collectors (PR#1749 by aleksejsv and Sebastian Wagner). - `intelmq.bots.collectors.http.collector_http_stream`: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner). - `intelmq.bots.collectors.shodan.collector_stream`: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner). - `intelmq.bots.collectors.twitter.collector_twitter`: - Proper input validation in URLs using urllib. CWE-20, found by GitHub's CodeQL (PR#1754 by Sebastian Wagner). - Limit replacement ("pastebin.com", "pastebin.com/raw") to a maximum of one (PR#1754 by Sebastian Wagner). #### Parsers - `intelmq.bots.parsers.eset.parser`: Added (PR#1554 by Mikk Margus Möll). - Ignore invalid "NXDOMAIN" IP addresses (PR#1573 by Mikk Margus Möll). - `intelmq.bots.parsers.hphosts`: Removed, feed is unavailable (#1559, by Sebastian Wagner). - `intelmq.bots.parsers.cznic.parser_haas`: Added (PR#1560 by Filip Pokorný and Edvard Rejthar). - `intelmq.bots.parsers.cznic.parser_proki`: Added (PR#1599 by sinus-x). - `intelmq.bots.parsers.key_value.parser`: Added (PR#1607 by Karl-Johan Karlsson). - `intelmq.bots.parsers.generic.parser_csv`: Added new parameter `compose_fields` (by Sebastian Wagner). - `intelmq.bots.parsers.shadowserver.parser_json`: Added (PR#1700 by Birger Schacht). - `intelmq.bots.parsers.shadowserver.config`: - Fixed mapping for Block list feed to accept network ranges in CIDR notation (#1720, PR#1728 by Sebastian Waldbauer). - Added mapping for new feed MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS (#1716, #1726, #1733, PR#1732, PR#1735, PR#1736 by Sebastian Waldbauer). - Ignore value `0` for `source.asn` and `destination.asn` in all mappings to avoid parsing errors (PR#1769 by Sebastian Wagner). - `intelmq.bots.parsers.abusech.parser_ip`: Adapt to changes in the Feodo Tracker Botnet C2 IP Blocklist feed (PR#1741 by Thomas Bellus). - `intelmq.bots.parsers.malwaredomainlist`: Removed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht). #### Experts - `intelmq.bots.experts.rfc1918.expert`: - Add support for ASNs (PR#1557 by Mladen Markovic). - Speed improvements. - More output in debug logging mode (by Sebastian Wagner). - Checks parameter length on initialization and in check method (by Sebastian Wagner). - `intelmq.bots.experts.gethostbyname.expert`: - Added parameter `fallback_to_url` and set to True (PR#1586 by Edvard Rejthar). - Added parameter `gaierrors_to_ignore` to optionally ignore other `gethostbyname` errors (#1553). - Added parameter `overwrite` to optionally overwrite existing IP addresses (by Sebastian Wagner). - `intelmq.bots.experts.asn_lookup.expert`: - Added `--update-database` option (PR#1524 by Filip Pokorný). - The script `update-asn-data` is now deprecated and will be removed in version 3.0. - `intelmq.bots.experts.maxmind_geoip.expert`: - Added `--update-database` option (PR#1524 by Filip Pokorný). - Added `license_key` parameter (PR#1524 by Filip Pokorný). - The script `update-geoip-data` is now deprecated and will be removed in version 3.0. - `intelmq.bots.experts.tor_nodes.expert`: - Added `--update-database` option (PR#1524 by Filip Pokorný). - The script `update-tor-nodes` is now deprecated and will be removed in version 3.0. - `intelmq.bots.experts.recordedfuture_iprisk.expert`: - Added `--update-database` option (PR#1524 by Filip Pokorný). - Added `api_token` parameter (PR#1524 by Filip Pokorný). - The script `update-rfiprisk-data` is now deprecated and will be removed in version 3.0. - Added `intelmq.bots.experts.threshold` (PR#1608 by Karl-Johan Karlsson). - Added `intelmq.bots.experts.splunk_saved_search.expert` (PR#1666 by Karl-Johan Karlsson). - `intelmq.bots.experts.sieve.expert`: - Added possibility to give multiple queue names for the `path` directive (#1462, by Sebastian Wagner). - Added possibility to run actions without filtering expression (#1706, PR#1708 by Sebastian Waldbauer). - Added datetime math operations (#1680, PR#1696 by Sebastian Waldbauer). - `intelmq.bots.experts.maxmind_geoip.expert`: - Fixed handing over of `overwrite` parameter to `event.add` (PR#1743 by Birger Schacht). #### Outputs - `intelmq.bots.outputs.rt`: Added Request Tracker output bot (PR#1589 by Marius Urkis). - `intelmq.bots.outputs.xmpp.output`: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht). - `intelmq.bots.outputs.smtp.output`: Fix sending to multiple recipients when recipients are defined by event-data (#1759, PR#1760 by Sebastian Waldbauer and Sebastian Wagner). ### Documentation - Feeds: - Add ESET URL and Domain feeds (by Sebastian Wagner). - Remove unavailable *HPHosts Hosts file* feed (#1559 by Sebastian Wagner). - Added CZ.NIC HaaS feed (PR#1560 by Filip Pokorný and Edvard Rejthar). - Added CZ.NIC Proki feed (PR#1599 by sinus-x). - Updated Abuse.ch URLhaus feed (PT#1572 by Filip Pokorný). - Added CERT-BUND CB-Report Malware infections feed (PR#1598 by sinus-x and Sebastian Wagner). - Updated Turris Greylist feed with PGP verification information (by Sebastian Wagner). - Fixed parsing of the `public` field in the generated feeds documentation (PR#1641 by Birger Schacht). - Change the `rate_limit` parameter of some feeds from 2 days (129600 seconds) to one day (86400 seconds). - Update the cAPTure Ponmocup Domains feed documentation (PR#1574 by Filip Pokorný and Sebastian Wagner). - Added Shadowserver Reports API (by Sebastian Wagner). - Change the `rate_limit` parameter for many feeds from 2 days to the default one day (by Sebastian Wagner). - Removed Malware Domain List feed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht). - Bots: - Enhanced documentation of RFC1918 Expert (PR#1557 by Mladen Markovic and Sebastian Wagner). - Enhanced documentation of SQL Output (PR#1620 by Edvard Rejthar). - Updated documentation for MaxMind GeoIP, ASN Lookup, TOR Nodes and Recorded Future experts to reflect new `--update-database` option (PR#1524 by Filip Pokorný). - Added documentation for Shadowserver API collector and parser (PR#1700 by Birger Schacht and Sebastian Wagner). - Add n6 integration documentation (by Sebastian Wagner). - Moved 'Orphaned Queues' section from the FAQ to the intelmqctl documentation (by Sebastian Wagner). - Generate documentation using Sphinx (PR#1622 by Birger Schacht). - The documentation is now available at https://intelmq.readthedocs.io/en/latest/ - Refactor documentation and fix broken syntax (#1639, PRs #1638 #1640 #1642 by Birger Schacht). - Integrate intelmq-manager and intelmq-api user documentation to provide unified documentation place (PR#1714 & PR#1714 by Birger Schacht). ### Packaging - Fix paths in the packaged logcheck rules (by Sebastian Wagner). - Build the sphinx documentation on package build (PR#1701 by Birger Schacht). - Ignore non-zero exit-codes for the `intelmqctl check` call in postinst (#1748, by Sebastian Wagner). ### Tests - Added tests for `intelmq.lib.exceptions.PipelineError` (by Sebastian Wagner). - `intelmq.tests.bots.collectors.http_collector.test_collector`: Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner). - `intelmq.tests.bots.outputs.restapi.test_output`: - Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner). - Add a test for checking the response status code (by Sebastian Wagner). - `intelmq.tests.bots.collectors.mail.test_collector_url`: Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner). - `intelmq.tests.bots.experts.ripe.test_expert`: Use `requests_mock` to mock all requests and do not require a local webserver (by Sebastian Wagner). - The test flag (environment variable) `INTELMQ_TEST_LOCAL_WEB` is no longer used (by Sebastian Wagner). - Added tests for `intelmq.harmonization.DateTime.parse_utc_isoformat` and `convert_fuzzy` (by Sebastian Wagner). - Move from Travis to GitHub Actions (PR#1707 by Birger Schacht). - `intelmq.lib.test`: - `test_static_bot_check_method` checks the bot's static `check(parameters)` method for any exceptions, and a valid formatted return value (#1505, by Sebastian Wagner). - `setUpClass`: Skip tests if cache was requests with `use_cache` member, but Redis is deactivated with the environment variable `INTELMQ_SKIP_REDIS` (by Sebastian Wagner). - `intelmq.tests.bots.experts.cymru_whois.test_expert`: - Switch from `example.com` to `ns2.univie.ac.at` for hopefully more stable responses (#1730, PR#1731 by Sebastian Waldbauer). - Do not test for exact expected values in the 6to4 network test, as the values are changing regularly (by Sebastian Wagner). - `intelmq.tests.bots.parsers.abusech`: Remove tests cases of discontinued feeds (PR#1741 by Thomas Bellus). - Activate GitHub's CodeQL Code Analyzing tool as GitHub Action (by Sebastian Wagner). ### Tools - `intelmqdump`: - Check if given queue is configured upon recovery (#1433, PR#1587 by Mladen Markovic). - `intelmqctl`: - `intelmq list queues`: `--sum`, `--count`, `-s` flag for showing total count of messages (#1408, PR#1581 by Mladen Markovic). - `intelmq check`: Added a possibility to ignore queues from the orphaned queues check (by Sebastian Wagner). - Allow setting the pipeline host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer). ### Contrib - EventDB: - Add SQL script for keeping track of the oldest inserted/update "time.source" information (by Sebastian Wagner). - Cron Jobs: The script `intelmq-update-data` has been renamed to `intelmq-update-database` (by Filip Pokorný). - Dropped utterly outdated contrib modules (by Sebastian Wagner): - ansible - vagrant - vagrant-ansible - logrotate: - Do not use the deprecated "copytruncate" option as intelmq re-opens the log anyways (by Sebastian Wagner). - Set file permissions to `0644` (by Sebastian Wagner). ### Known issues - Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952). - Corrupt dump files when interrupted during writing (#870). - CSV line recovery forces Windows line endings (#1597). - intelmqdump: Honor logging_path variable (#1605). - Timeout error in mail URL fetcher (#1621). - AMQP pipeline: get_queues needs to check vhost of response (#1746). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IEP02: Introduce IntelMQ Mixins
by Birger Schacht 03 Mar '21

03 Mar '21

Dear IntelMQ developers, below I am writing about an idea that might make contributing to IntelMQ easier- it is a small change, but could have implications for various bots: # Introduce IntelMQ Mixins (IntelMQ Enhancement Proposal 02) I would like to lower the bar for adding additional bots by introducing mixins. Mixins are a concept coming from OOP theory and it is closely related to multiple inheritance. I think it could be a way to avoid a lot of duplicate code by outsourcing various functionalities to specific classes. ## Current situation At the moment there are two types of bots that could make use of mixins. The first are bots that make HTTP requests. Currently we make that easier for bots by providing two methods, `create_request_session` in intelmq/lib/utils.py and `set_request_parameters` as part of the bot module (intelmq/lib/bot.py) itself. When the bot wants to use a request session, it first uses `set_request_parameters` and then uses the session object returned by `create_request_session` to make the request. I think its a bit confusing to have methods that are related in two different modules and I think the Bot class itself should not contain any code that is not used by all the bots. The second type of bots that could make use of mixins are bots that use a cache. There are a lot of those that all use the `intelmq.lib.cache` module which provides a Cache object to work with. ## Improvement using mixins Using a Python class we could move everything related to HTTP requests to a class called HttpMixin. A bot that wants to use functionality from this Mixin just would have to inherit from this class, i.e.: ``` class MySuperBot(CollectorBot, HttpMixin): ``` The MySuperBot class would then have all the relevant attributes to use session objects and it would also have a private `__session` attribute to work with. Another upside of this approach is, that it is possible to generate a list of all the attributes a Bot has write access to (i.e. http_proxy, http_username, ssl_client_cert...) Similarly we could introduce a `CacheMixin` that allow access to private `__cache` attribute. If MySuperBot does not only want to make HTTP requests, but also have some caching mechanisms, it could simply inherit from both classes: ``` class MySuperBot(CollectorBot, HttpMixin, CacheMixin): ``` I propose to create a new module `intelmq.lib.mixins` that should be the home for the mixins. I have created a POC that implements the HttpMixin and uses it in the HTTPCollectorBot: https://github.com/certtools/intelmq/tree/schacht/http-mixin Mixins are also used in Django to add functionality to their class based Views, for more details see https://docs.djangoproject.com/en/3.1/topics/class-based-views/mixins/ A longer article about inheritance, MRO, mixins and Python that also talks about the implementation in Django can be found on https://www.thedigitalcatonline.com/blog/2020/03/27/mixin-classes-in-python/ cheers, Birger -- // Birger Schacht <schacht(a)cert.at> // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Installing new bots from devel host to production host
by Mika Silander 02 Mar '21

02 Mar '21

Hi, Bothering you again with a probably simple question. I've created half a dozen of bots on a development machine and now I'd like to create an installation system, Makefile or whatever to install those bots + configurations to the production host. It appears to me that the section "Update" (https://intelmq.readthedocs.io/en/latest/dev/guide.html#id12) contains the recommended steps to follow and assumes that one has the development environment (below /opt/dev_intelmq) available on the same host where one intends to run the production set up (under /opt/intelmq), correct? So, what do you recommend? Should I copy over the entire /opt/dev_intelmq to the production host and then "Update" there according to the above instructions or are there other ways to just copy/install the relevant bots+configurations from below /opt/dev_intelmq to the production host? Best regards and thanks again, Mika

3 6

Default crontab values for local lookup-databases update
by Sebastian Wagner 26 Feb '21

26 Feb '21

Dear devs, Thanks to Filip (CZ.NIC) IntelMQ comes now with an update mechanism for local lookup databases like TOR exit nodes, IP address to ASN ("ASN Lookup") and Maxmind GeoIP (IP address geolocation)[0]. Also, IntelMQ ships with update scripts for cron which are included in the deb/rpm packages as well. Currently the update scripts are scheduled as follows[1]: * TOR nodes: once per day. The database is very small. * Maxmind GeoIP: Once per week. Changes are scarce. * ASN Lookup: Every two hours. Big database, but the data is vital for subsequent routing of incidents. I'd like to hear your opinion if the default values are ok to ship with 2.3.0, especially for the last one. best regards, Sebastian [0]: https://github.com/certtools/intelmq/pull/1524 [1]: https://github.com/certtools/intelmq/blob/24f2355d0c549021a713c938d1d69a521… -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 2

Classification of malware itself in IntelMQ
by Sebastian Wagner 22 Feb '21

22 Feb '21

Dear IntelMQ community, sorry for cross-posting, but I think this topic should be discussed in a wider group. IntelMQ always followed the Reference Security Incident Taxonomy (short: RSIT)[0] and its predecessor for its 'classification.taxonomy/type' fields. The Classification column in the RSIT corresponds to our "classification.taxonomy" field, and the RSIT's second column (currently called Incident examples) corresponds to our "classification.type" field. "classification.identifier" is an optional third level free-text field to give more specific context.[1] Due to historical reasons and changes on both sides - IntelMQ as well as the RSIT -, IntelMQ's classification scheme deviated a bit from the RSIT over time. I'm working on aligning them again for 3.0, which works straightforward in most cases. But for one case, I need your input. The predecessor of the RSIT (the eCSIRT.net taxonomy)[2] used the malicious code taxonomy differently: To classify malware itself into categories, like virus, worm, trojan, etc. The RSIT never did that, as classifying malware is never unambiguous and there are plenty of existing classification scheme out there, which do this already. Also, the focus of the RSIT is different, as it classifies the incidents/events, not malware samples. And for this reason, IntelMQ had (until < 3.0.0) the classification.type "malware" in IntelMQ. Most of the usages were wrong anyway, and should have been infected-device, malware-distribution or something else anyway. There is only one usage in IntelMQ, which can not be changed. And that one is really about malware itself (or: the hashes of samples) as used in the GitHub Feed parser[3] and the FireEye Parser[4]. But the issue is more generic, as we need to decide anyway, how we want to deal with such malware-IoCs. A malware (hash) does not fit into the RSIT. It's neither an Infected System, a C2 Server, Malware Distribution nor Malware Configuration. It's just a malware (hash). I see four options: 1) Deviate from the RSIT and just use 'classification.taxonomy' = 'Malicious Code' and 'classification.type' = 'malware' 2) Deviate slightly less from the RSIT and use 'classification.taxonomy' = 'other' and 'classification.type' = 'malware' 3) Adhere strictly to the RSIT and use 'classification.taxonomy' = 'other' and 'classification.type' = 'other' and "classification.identifier" = 'malware' 4) IntelMQ does not support this use case In cases 1) and 2) "classification.identifier" could be used to specify what the event is about, e.g. "hash", or the malware family. I'm currently in favor of option 2), as we can keep the meaning of "Malicious Code" in sync with the RSIT and still support the use-case sufficiently. But my opinion could change during the discussion :) Do you see any more options than I listed above? What do you favor? best regards Sebastian [0]: https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/… [1]: https://intelmq.readthedocs.io/en/latest/dev/data-harmonization.html#classi… [2]: https://www.trusted-introducer.org/Incident-Classification-Taxonomy.pdf [3]: https://github.com/certtools/intelmq/blob/f7507ca2643fe8ddb3817c9be1209504e… [4]: https://github.com/certtools/intelmq/pull/1745 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

The IntelMQ 2.3.0 Release candidate waits to be tested
by Sebastian Wagner 22 Feb '21

22 Feb '21

Dear devs, We are in the final sprint of releasing the feature release version 2.3.0 and we are actively looking for testers! Ideally, we'd like to finish the 2.3.0 version at end of February. As Python 3.6 is required, support for Debian 9 and Ubuntu 16.04 had to be dropped. The list of supported operating systems is now: * CentOS 7 and 8 (with EPEL activated) * Debian 10 * Fedora 32, 33, Rawhide * openSUSE Leap 15.1, 15.2, Tumbleweed * Ubuntu: 18.04, 20.04, 20.10 (with universe activated) * Docker Engine: 18.x or higher The other major change is the split of the IntelMQ Manager into a separate API as backend (Thanks Birger) and the switch to Python for the latter (Thanks SUNET and Intevation). And thanks for all the numerous contributors who contributed greatly to the majorly extended feature set, by providing code, bug reports and other inputs. For all the other changes, please have a look at the changelog and the news file in the repository. I will give a more detailed overview of the changes in the final 2.3.0 announcement. https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md https://github.com/certtools/intelmq/blob/develop/NEWS.md To install the release candidate version from git, use the "maintenance" branch or 2.3.0.rc1 Tag (or if you are looking for more adventures, the "develop" branch,which we will use for the development of 3.0 now). For installation via pip, use `pip install --pre intelmq`, the version is called 2.3.0rc1 there. And for the packages, use the unstable repository: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… For the new beta docker image (Thanks Sebastian Waldbauer), install certat/intelmq-full: https://hub.docker.com/r/certat/intelmq-full Also summarized here: https://intelmq.readthedocs.io/en/latest/dev/guide.html#testing-pre-releases best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Advice for setting up tests for bot chains
by Mika Silander 19 Feb '21

19 Feb '21

Hi, While writing tests to individual bots seems quite straightforward, what would be the recommended way for writing (unit?) tests for a chain of bots? Ideally, I'd like to feed in individual test events to the first bot and only check what is returned by the last bot (event or whatever the expected outcome should be). I suppose there are many bad and clumsy ways of doing this so that's why I air this question on the dev list. Thanks for all help in advance, Mika

4 6

Bot behaviour in case of unrecoverable errors
by Mika Silander 16 Feb '21

16 Feb '21

Hi, Trying to assess what safeguards are sufficient: what happens when a bot has some internal failure and it "dies"? Will intelmq restart the bot automatically or will it be up to the admin of intelmq to manually restart it? And if automatic restarts is the norm, how could one stop the bot from processing new incoming messages if say, X consecutive failures like these have happened within the time frame of the last 5 minutes? By writing some log entries at bot startup and then making the bot itself analyze the log at every restart? I'm trying to make sure a burst of erroneous/malformed events are not accidentally forwarded by a malfunctioning or partially functioning bot. Cheers, Mika

2 6

← Newer
1
...
7
8
9
10
11
12
13
...
24
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev