- IntelMQ-users

MISP feed output bot usage
by Marius Urkis 04 Sep '20

04 Sep '20

Hello IntelMQ users, Trying to figure out how to use MISP feed output bot, could someone advise. MISP creates new event once per period (per hour, or per day), and that makes MISP doing correlation between these events created previously. And actually that results correlation table grows exponentially. Am I doing something wrong on IntelMQ side or MISP? At IntelMQ I configure bot to make one event per day (actually containing ~1500 events in resulting json file). At the MISP side I have MISP feed format feed. Best regards -- Marius Urkis

3 2

IntelMQ & IntelMQ Manager releases 2.2.1
by Sebastian Wagner 30 Jul '20

30 Jul '20

Dear community, Today we have again a twin release 2.2.1 for both IntelMQ as well as IntelMQ Manager. This IntelMQ Manager version requires IntelMQ >= 2.2.1. There are currently issues with the signature in the package repositories for Debian/Ubuntu. I hope to get them resolved soon. IntelMQ Installation documentation: https://github.com/certtools/intelmq/blob/2.2.1/docs/INSTALL.md IntelMQ Upgrade documentation: https://github.com/certtools/intelmq/blob/2.2.1/docs/UPGRADING.md IntelMQ Manager Installation instructions: https://github.com/certtools/intelmq-manager/blob/2.2.1/docs/INSTALL.md *The changelog for IntelMQ Manager:* ### Backend - Fix loading paths from `intelmqctl` executable (PR #205 by Einar Felipe Lanfranco). ### Documentation - User Guide: - Add section on configuration paths. - Add section on named queues / paths. - Readme: - Update screenshots (#201, PR#207 by Mladen Markovic). ### Known issues * Graph jumps around on "Add edge" (#148). * wrong error message for new bots with existing ID (#152). * Monitor page: Automatic log refresh reset log page to first one (#190). *The News for IntelMQ:* ### Requirements #### MaxMind GeoIP Expert Bot The current python library versions of geoip (version 4) and maxminddb (version 2) no longer support Python 3.5. Keep older versions of these libraries if you are using this Python version. ### Configuration #### Abuse.ch URLHaus The current documented value for the `column` parameter was: ```json ['time.source', 'source.url', 'status', 'extra.urlhaus.threat_type', 'source.fqdn', 'source.ip', 'source.asn', 'source.geolocation.cc'] ``` Better is: ```json ['time.source', 'source.url', 'status', 'classification.type|__IGNORE__', 'source.fqdn|__IGNORE__', 'source.ip', 'source.asn', 'source.geolocation.cc'] ``` *And the changelog for IntelMQ:* ### Core - `intelmq.lib.upgrades`: - Add upgrade function for changed configuration of the feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný). - Add upgrade function for removal of *HPHosts Hosts file* feed and `intelmq.bots.parsers.hphosts` parser (#1559). - `intelmq.lib.harmonization`: - For IP Addresses, explicitly reject IPv6 addresses with scope ID (due to changed behavior in Python 3.9, #1550). ### Development - Ignore line length (E501) in code-style checks altogether. ### Bots #### Collectors - `intelmq.bots.collectors.misp`: Fix access to actual MISP object (PR#1548 by Tomas Bellus @tomas321) - `intelmq.bots.collectors.stomp`: Remove empty `client.pem` file. #### Parsers - `intelmq.bots.parsers.shadowserver.config`: - Add support for Accessible-CoAP feed (PR #1555 by Thomas Hungenberg). - Add support for Accessible-ARD feed (PR #1584 by Tomas Bellus @tomas321). - `intelmq.bots.parser.anubisnetworks.parser`: Ignore "TestSinkholingLoss" events, these are not intended to be sent out at all. - `intelmq.bots.parsers.generic.parser_csv`: Allow values of type dictionary for parameter `type_translation`. - `intelmq.bots.parsers.hphosts`: Removed, feed is unavailable (#1559). - `intelmq.bots.parsers.cymru.parser_cap_program`: Add support for comment "username" for "scanner" category. - `intelmq.bots.parsers.malwareurl.parser`: Check for valid FQDN and IP address in URL and IP address columns (PR#1585 by Marius Urkis). #### Experts - `intelmq.bots.experts.maxmind_geoip`: On Python < 3.6, require maxminddb < 2, as that version does no longer support Python 3.5. #### Outputs - `intelmq.bot.outputs.udp`: Fix error handling on sending, had a bug itself. ### Documentation - Feeds: - Update documentation of feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný). - Bots: - Overhaul of all bots' description fields (#1570). - User-Guide: - Overhaul pipeline configuration section and explain named queues better (#1577). ### Tests - `intelmq.tests.bots.experts.cymru`: Adapt `test_empty_result`, remove `test_unicode_as_name` and `test_country_question_mark` (#1576). ### Tools - `intelmq.bin.intelmq_gen_docs`: Format parameters of types lists with double quotes around values to produce conform JSON, ready to copy and paste the value into the IntelMQ Manager's bot parameter form. - `intelmq.bin.intelmqctl`: - `debug`: In JSON mode, use dictionaries instead of lists. - `debug`: Add `PATH` to the paths shown. - `check`: Show `$PATH` environment variable if executable cannot be found. ### Contrib - `malware_name_mapping`: Change MISP Threat Actors URL to new URL (branch master -> main) in download script. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Corrupt dump files when interrupted during writing (#870). - Bash completion scripts search in wrong directory in packages (#1561). - Cymru Expert: Wrong Cache-Key Calculation (#1592). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

2.2.0 Packages are now available in the repositories
by Sebastian Wagner 24 Jun '20

24 Jun '20

Hi, The packages for the 2.2.0 versions of IntelMQ and IntelMQ Manager are now available in the repositories, you can use the package management of your operating system to update your IntelMQ instance. Any feedback on the packages is as always appreciated. I also updated and cleaned up the supported operating systems / version by removing those which are End of Life and added a few new versions. I am still struggling with CentOS 8. best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ Manager 2.2.0
by Sebastian Wagner 24 Jun '20

24 Jun '20

Dear community, To get the versions of intelmq and intelmq-manager in sync, we of course need a 2.2.0 for intelmq-manager too :) The number of changes is very short this time. However, a migration of the backend from PHP to Python, coded by Intevation and funded by SUNET, is in progress :) The deb/rpm packages are currently building, I will send a separate mail to intelmq-users once the 2.2.0 of intelmq and intelmq-manager packages have finally hit the repositories. This IntelMQ Manager version requires IntelMQ >= 2.2.0. Installation instructions: https://github.com/certtools/intelmq-manager/blob/2.2.0/docs/INSTALL.md Full changelog: ### Backend - `config`: Get file paths from `intelmctl debug --get-paths` if possible and fall back to hard-coded paths otherwise. Thereby environment variables influencing the paths are respected (#193). ### Pages #### About - Show output of `intelmqctl debug`. ### Documentation - Update release from intelmq's release documentation. - Update Installation documentation: Fix & update dependencies and supported operating systems. ### Packaging - Update default `positions.conf` to the default runtime/pipeline configuration of intelmq >= 2.1.1. ### Known issues * Missing CSRF protection (#111). * Graph jumps around on "Add edge" (#148). * wrong error message for new bots with existing ID (#152). * `ALLOWED_PATH=` violates CSP (#183). * Monitor page: Automatic log refresh reset log page to first one (#190). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Re: [Intelmq-users] IntelMQ on CentOS7 - yum update failling with
by Marco Paulo Machado Teixeira 18 Jun '20

18 Jun '20

Ok Sebastian, I'm holding off until 2.2.0 rpms are released. Thank you once more. Cumprimentos, Marco -----Mensagem original----- De: Sebastian Wagner <wagner(a)cert.at> Enviada: 18 de junho de 2020 13:10 Para: Marco Paulo Machado Teixeira <marco(a)scom.uminho.pt>; intelmq-users(a)lists.cert.at Assunto: Re: [Intelmq-users] IntelMQ on CentOS7 - yum update failling with conflicting files On 6/18/20 1:27 PM, Marco Paulo Machado Teixeira wrote: > Should have mentioned that already tried that, but dependencies ask to remove IntelMQ... Ah, I understand. Will fix that together with the 2.2.0 release. > Would you say it's safe to force removal of the package, ignoring dependencies, and just install the EPEL packet afterwards? In principle yes, or just wait a bit until I fixed the dependency of the python3-intelmq package. Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ on CentOS7 - yum update failling with conflicting files
by Marco Paulo Machado Teixeira 18 Jun '20

18 Jun '20

Good day to all, Trying to update my intelmq install, on CentOS 7, with RPM packages (as per the install instructions). I'm getting a conflict with overlapping RPM packages from EPEL repo and HOME:SEBIX:INTELMQ repo, with the following error (first line only, several more lines below I didn't copy). Already searched the web without luck. Also tried using REPO priorities but did not solve it. Anyone has some advice on the best way to fix this without ruining the RPM database consistency? Or maybe some pointers where to look? === Transaction check error: file /usr/lib/python3.6/site-packages/dns/__init__.py from install of python36-dns-1.16.0-1.el7.noarch conflicts with file from package python3-dnspython-1.15.0-3.6.noarch === Regards, Marco

2 3

IntelMQ 2.2.0 release
by Sebastian Wagner 18 Jun '20

18 Jun '20

Dear community, Today I finalized the 2.2.0 Feature release, more than half a year after 2.1.0. Thanks to all the contributors who made this possible! The release includes six new bots and seven heavily revised bots, and of course a lot of small changes to various bots. The full changelog can be found below. We dropped support for Python 3.4, that means Debian 8.0 and similar operating systems are no longer supported. Installation documentation: https://github.com/certtools/intelmq/blob/2.2.0/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.2.0/docs/UPGRADING.md ### Core - `__init__`: Changes to the path-handling, see [User Guide, section _/opt and LSB paths_](docs/User-Guide.md#opt-and-lsb-paths) for more information - The environment variable `INTELMQ_ROOT_DIR` can be used to set custom root directories instead of `/opt/intelmq/` (#805) in case of non LSB-path installations. - The environment variable `ROOT_DIR` can be used to set custom root directories instead of `/` (#805) in case of LSB-path installations. - `intelmq.lib.exceptions`: Added `MissingDependencyError` for show error messages about a missing library and how to install it (#1471). - Added optional parameter `installed` to show the installed version. - Added optional parameter `additional_text` to show arbitrary text. - Adding more type annotations for core libraries. - `intelmq.lib.pipeline.Pythonlist.sleep`: Drop deprecated method. - `intelmq.lib.utils`: `write_configuration`: Append a newline at end of configuration/file to allow proper comparisons & diffs. - `intelmq.lib.test`: `BotTestCase` drops privileges upon initialization (#1489). - `intelmq.lib.bot`: - New class `OutputBot`: - Method `export_event` to format/export events according to the parameters given by the user. - `ParserBot`: New methods `parse_json_stream` and `recover_line_json_stream`. - `ParserBot.recover_line_json`: Fix format by adding a list around the line data. - `Bot.send_message`: In debugging log level, the path to which the message is sent is now logged too. ### Bots - Bots with dependencies: Use of `intelmq.lib.exceptions.MissingDependencyError`. #### Collectors - `intelmq.bots.collectors.misp.collector`: Deprecate parameter `misp_verify` in favor of generic parameter `http_verify_cert`. - `intelmq.bots.collectors.tcp.collector`: Drop compatibility with Python 3.4. - `intelmq.bots.collectors.stomp.collector`: - Check the stomp.py version and show an error message if it does not match. - For stomp.py versions `>= 5.0.0` redirect the `stomp.PrintingListener` output to debug logging. - `intelmq.bots.collectors.microsoft.collector_azure`: Support current Python library `azure-storage-blob>= 12.0.0`, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details. - `intelmq.bots.collectors.amqp.collector_amqp`: Require `pika` minimum version 1.0. - `intelmq.bots.collectors.github_api.collector_github_contents_api`: Added (PR#1481). #### Parsers - `intelmq.bots.parsers.autoshun.parser`: Drop compatibility with Python 3.4. - `intelmq.bots.parsers.html_table.parser`: Drop compatibility with Python 3.4. - `intelmq.bots.parsers.shadowserver.parser`: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544). - `intelmq.bots.parsers.taichung.parser`: - Migrate to `ParserBot`. - Also parse geolocation information if available. - `intelmq.bots.parsers.cymru.parser_full_bogons`: - Migrate to `ParserBot`. - Add last updated information in raw. - `intelmq.bots.parsers.anubisnetworks.parser`: Add new parameter `use_malware_familiy_as_classification_identifier`. - `intelmq.bots.parsers.microsoft.parser_ctip`: Compatibility for new CTIP data format used provided by the Azure interface. - `intelmq.bots.parsers.cymru.parser_cap_program`: Support for `openresolver` type. - `intelmq.bots.parsers.github_feed.parser`: Added (PR#1481). - `intelmq.bots.parsers.urlvir.parser`: Removed, as the feed is discontinued (#1537). #### Experts - `intelmq.bots.experts.csv_converter`: Added as converter to CSV. - `intelmq.bots.experts.misp`: Added (PR#1475). - `intelmq.bots.experts.modify`: New parameter `maximum_matches`. #### Outputs - `intelmq.bots.outputs.amqptopic`: - Use `OutputBot` and `export_event`. - Allow formatting the routing key with event data by the new parameter `format_routing_key` (boolean). - `intelmq.bots.outputs.file`: Use `OutputBot` and `export_event`. - `intelmq.bots.outputs.files`: Use `OutputBot` and `export_event`. - `intelmq.bots.outputs.misp.output_feed`: Added, creates a MISP Feed (PR#1473). - `intelmq.bots.outputs.misp.output_api`: Added, pushes to MISP via the API (PR#1506, PR#1536). - `intelmq.bots.outputs.elasticsearch.output`: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (#1513). ### Documentation - Document usage of the `INTELMQ_ROOT_DIR` environment variable. - Added document on MISP integration possibilities. - Feeds: - Added "Full Bogons IPv6" feed. - Remove discontinued URLVir Feeds (#1537). ### Packaging - `setup.py` do not try to install any data to `/opt/intelmq/` as the behavior is inconsistent on various systems and with `intelmqsetup` we have a tool to create the structure and files anyway. - `debian/rules`: - Provide a blank state file in the package. - Patches: - Updated `fix-intelmq-paths.patch`. ### Tests - Travis: Use `intelmqsetup` here too. - Install required build dependencies for the Debian package build test. - This version is no longer automatically tested on Python `<` 3.5. - Also run the tests on Python 3.8. - Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8. - Added tests for the new bot `intelmq.bots.outputs.misp.output_feed` (#1473). - Added tests for the new bot `intelmq.bots.experts.misp.expert` (#1473). - Added tests for `intelmq.lib.exceptions`. - Added tests for `intelmq.lib.bot.OutputBot` and `intelmq.lib.bot.OutputBot.export_event`. - Added IPv6 tests for `intelmq.bots.parsers.cymru.parser_full_bogons`. - Added tests for `intelmq.lib.bot.ParserBot`'s new methods `parse_json_stream` and `recover_line_json_stream`. - `intelmq.tests.test_conf`: Set encoding to UTF-8 for reading the `feeds.yaml` file. ### Tools - `intelmqctl`: - `upgrade-config`: - Allow setting the state file location with the `--state-file` parameter. - Do not require a second run anymore, if the state file is newly created (#1491). - New parameter `no_backup`/`--no-backup` to skip creation of `.bak` files for state and configuration files. - Only require `psutil` for the `IntelMQProcessManager`, not for process manager independent calls like `upgrade-config` or `check`. - Add new command `debug` to output some information for debugging. Currently implemented: - paths - environment variables - `IntelMQController`: New argument `--no-file-logging` to disable logging to file. - If dropping privileges does not work, `intelmqctl` will now abort (#1489). - `intelmqsetup`: - Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions. - Call `intelmqctl upgrade-config` and add argument for the state file path (#1491). - `intelmq_generate_misp_objects_templates.py`: Tool to create a MISP object template (#1470). - `intelmqdump`: New parameter `-t` or `--truncate` to optionally give the maximum length of `raw` data to show, 0 for no truncating. ### Contrib - Added `development-tools`. - ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (#1513). - Malware Name Mapping Downloader: - New parameter `--mwnmp-ignore-adware`. - The parameter `--add-default` supports an optional parameter to define the default value. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Corrupt dump files when interrupted during writing (#870). best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ vs AbuseIO
by Patrick Forsberg 01 Jun '20

01 Jun '20

Hi, Anyone have a quick comparison between AbuseIO and IntelMQ? I'm currently in the process of getting IntelMQ to work with IntelMQ-Mailgen to be able to send out abuse-emails to our constituency based on feeds like Shadowserver and since it seems like AbuseIO can do just about the same I would like to know the pros and cons of the systems. Regards, Patrick Forsberg SUNET CERT

1 0

IntelMQ 2.1.3 Bugfix release
by Sebastian Wagner 26 May '20

26 May '20

Dear community, We already collected a very long list of bug fixes since the last release, so it was time to mark the next milestone! As usual, you can find the list of changes below. The pre-built deb/rpm packages will hit the repositories very soon. Installation documentation: https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md Full changelog: ### Requirements - The python library `requests` is (again) listed as dependency of the core (#1519). ### Core - `intelmq.lib.upgrades`: - Harmonization upgrade: Also check and update regular expressions. - Add function to migrate the deprecated parameter `attach_unzip` to `extract_files` for the mail attachment collector. - Add function to migrate changed Taichung URL feed. - Check for discontinued Abuse.CH Zeus Tracker feed. - `intelmq.lib.bot`: - `ParserBot.recover_line`: Parameter `line` needs to be optional, fix usage of fallback value `self.current_line`. - `start`: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494). - `start`: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors. - `_dump_message`: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494). - `intelmq.lib.test`: - `BotTestCase.run_bot`: Add parameters `allowed_error_count` and `allowed_warning_count` to allow set the number per run, not per test class. - Set `source_pipeline_broker` and `destination_pipeline_broker` to `pythonlist` instead of the old `broker`, fixes `intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising`. - Fix test for (allowed) errors and warnings. - `intelmq.lib.exceptions`: - `InvalidKey`: Add `KeyError` as parent class. - `DecodingError`: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494). - `intelmq.lib.pipeline`: - Decode messages in `Pipeline.receive` not in the implementation's `_receive` so that the internal counter is correct in case of decoding errors (#1494). - `intelmq.lib.utils`: - `decode`: Raise new `DecodingError` if decoding fails. ### Harmonization - `protocol.transport`: Adapt regular expression to allow the value `nvp-ii` (protocol 11). ### Bots #### Collectors - `intelmq.bots.collectors.mail.collector_mail_attach`: - Fix handling of deprecated parameter name `attach_unzip`. - Fix handling of attachments without filenames (#1538). - `intelmq.bots.collectors.stomp.collector`: Fix compatibility with stomp.py versions `> 4.1.20` and catch errors on shutdown. - `intelmq.bots.collectors.microsoft`: - Update `REQUIREMENTS.txt` temporarily fixing deprecated Azure library (#1530, PR#1532). - `intelmq.bots.collectors.microsoft.collector_interflow`: Add method for printing the file list. #### Parsers - `intelmq.bots.parsers.cymru.parser_cap_program`: Support for protocol 11 (`nvp-ii`) and `conficker` type. - `intelmq.bots.parsers.taichung.parser`: Support more types/classifications: - Application Compromise: Apache vulnerability & SQL injections - Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks - C2 Sever: Attack controller - DDoS - DoS: DNS, DoS, Excess connection - IDS Alert / known vulnerability exploitation: backdoor - Malware: Malware Proxy - Warn on new unknown types. - `intelmq.bots.parsers.bitcash.parser`: Removed as feed is discontinued. - `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target`: Removed as feed is discontinued. - `intelmq.bots.parsers.malwaredomains.parser`: Correctly classify `C&C` and `phishing` events. - `intelmq.bots.parsers.shadowserver.parser`: More verbose error message for missing report specification (#1507). - `intelmq.bots.parsers.n6.parser_n6stomp`: Always add n6 field `name` as `malware.name` independent of `category`. - `intelmq.bots.parsers.anubisnetworks`: Update parser with new data format. - `intelmq.bots.parsers.bambenek`: Add new feed URLs with Host `faf.bambenekconsulting.com` (#1525, PR#1526). - `intelmq.bots.parsers.abusech.parser_ransomware`: Removed, as the feed is discontinued (#1537). - `intelmq.bots.parsers.nothink.parser`: Removed, as the feed is discontinued (#1537). - `intelmq.bots.parsers.n6.parser`: Remove not allowed characters in the name field for `malware.name` and write original value to `event_description.text` instead. #### Experts - `intelmq.bots.experts.cymru_whois.lib`: Fix parsing of AS names with Unicode characters. #### Outputs - `intelmq.bots.outputs.mongodb`: - Set default port 27017. - Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439). ### Documentation - Feeds: - Remove unavailable feed Abuse.CH Zeus Tracker. - Remove the field `status`, offline feeds should be removed. - Add a new field `public` to differentiate between private and public feeds. - Adding documentation URLs to nearly all feeds. - Remove unavailable Bitcash.cz feed. - Remove unavailable Fraunhofer DDos Attack feeds. - Remove unavailable feed Abuse.CH Ransomware Tracker (#1537). - Update information on Bambenek Feeds, many require a license now (#1525). - Remove discontinued Nothink Honeypot Feeds (#1537). - Developers Guide: Fix the instructions for `/opt/intelmq` file permissions. ### Packaging - Patches: `fix-logrotate-path.patch`: also include path to rotated file in patch. - Fix paths from `/opt` to LSB for `setup.py` and `contrib/logrotate/intelmq` in build process (#1500). - Add runtime dependency `debianutils` for the program `which`, which is required for `intelmqctl`. ### Tests - Dropping Travis tests for 3.4 as required libraries dropped 3.4 support. - `intelmq.tests.bots.experts.cymru_whois`: - Drop missing ASN test, does not work anymore. - IPv6 to IPv4 test: Test for two possible results. - `intelmq.lib.test`: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (#1342). - `intelmq.bots.collectors.tcp.test_collector`: Removing custom mocking and bot starting, not necessary anymore. - Added tests for `intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline`. - Fix and split `tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json`. - Added tests for invalid encodings in input messages in `intelmq.tests.lib.test_bot` and `intelmq.tests.lib.test_pipeline` (#1494). - Travis: Explicitly enable RabbitMQ management plugin. - `intelmq.tests.lib.test_message`: Fix usage of the parameter `blacklist` for Message hash tests (#1539). ### Tools - `intelmqsetup`: Copy missing BOTS file to IntelMQ's root directory (#1498). - `intelmq_gen_docs`: Feed documentation generation: Handle missing/empty parameters. - `intelmqctl`: - `IntelMQProcessManager`: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (#1492). - `IntelMQController`: Fix exit codes of `check` command for JSON output (now 0 on success and 1 on error, was swapped, #1520). - `intelmqdump`: - Handle base64-type messages for show, editor and recovery actions. ### Contrib - `intelmq/bots/experts/asn_lookup/update-asn-data`: Use `pyasn_util_download.py` to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, https://github.com/hadiasghari/pyasn/issues/62). ### Known issues - HTTP stream collector: retry on regular connection problems? (#1435). - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Reverse DNS: Only first record is used (#877). - Corrupt dump files when interrupted during writing (#870). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Request for feedback - IntelMQ 3.0 architecture
by L. Aaron Kaplan 08 May '20

08 May '20

Dear IntelMQ community, I hope everyone is fine and most of us are slowly waking up after the Corona lock-down phase ( at least I am, and it's a really good feeling to be able to work concentrated on things again. Let me tell you, home-office with small children is not fun). In any case, I already had some telcos with some of you regarding our wish list for IntelMQ-3.0. There are some fantastic ideas floating around and since IntelMQ is such a versatile and adaptable tool, it is quite a challenge to capture all the great changes and adaptations people made to it. The next version of IntelMQ (3.0) will try to capture some of the most needed changes (and some wishes) for you. I tried to summarise some of these changes in the following document: https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectu… May I ask you to review it and send me feedback (kaplan(a)cert.at)? Also, I'd be very happy to have a telco with you in case that's easier or more natural for you for giving feedback. Since we would like to start coding soon, it is important to receive *your* input on the next version of IntelMQ. Thanks for your time! All the best, Aaron. -- // L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users