- IntelMQ-users

IntelMQ Manager Release 2.1.0
by Sebastian Wagner 15 Oct '19

15 Oct '19

Dear community, The new 2.1.0 version of the intelmq manager improves the error reporting in the frontend and allows to connect collectors to experts and experts to parsers. Thanks to Edvard for his work on this tool! Installation instructions: https://github.com/certtools/intelmq-manager/blob/2.0.0/docs/INSTALL.md Full changelog: ### Backend - Fix mispelling of the environmental variable `INTELMQ_MANGER_CONTROLER_CMD` to `INTELMQ_MANGER_CONTROLLER_CMD` (you might be required to add the double 'l' to your nginx/apache server configuration) - When displaying a command to be replicated by debugging user, the string "sudo -u {webserver user}" string is prepended so that linux user do not have to bother with sudoing himself on the commonly used user "www-data" – which often can't be sudoed to (no bash provided due to good security measures). (Used in monitor and error reporting.) ### Frontend - Error reporting - Click will enlarge the dialog that contains much more useful info, notably the very command that failed so that it can be easily reproduced. - Error messages are shuffled only when minimized, not when maximized. That would disturb the user trying to read the details. - Invalid Syntax Error message removed. Till now, all error messages generated the string that JSON received is invalid – that wasn't needed, we knew it's invalid because it contained string message. - Double click does not close log window anymore since it would interfere with the user trying to select whole text by mouse. - Escape minimizes the reporting. - For common seen errors, a tip is displayed (preferable with a link to the Github manual). ### Pages #### Configuration - Node group Collector now may connect to Expert and Expert can connect to Parser, however you receive a warning that it is not very common. -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 2.0.2 Bugfix release
by Sebastian Wagner 14 Oct '19

14 Oct '19

Dear community, Today I released version 2.0.2 of intelmq with a bunch of bugfixes, see the changelog below. A feature release is coming soon too. Install documentation: https://github.com/certtools/intelmq/blob/2.0.2/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.0.2/docs/UPGRADING.md As always: read the NEWS file, upgrade according to the documentation and have fun! If you get any errors, please report them here or in the bug tracker. Sebastian Full changelog: ### Core - `intelmq.lib.bot.CollectorBot`: Support the deprecated parameter `feed` until version 2.2 as the documentation was not properly updated (#1445). - `intelmq.lib.bot.Bot`: - `_dump_message`: Wait for up to 60 seconds instead of 50 if the dump file is locked (the log message was said 60, but the code was for only 50). - `intelmq.lib.upgrades.v202_fixes` - Migration of deprecated parameter `feed` for Collectors. - Ripe expert parameter `query_ripe_stat_ip` was not correctly configured in `v110_deprecations`, now use `query_ripe_stat_asn` as default if it does not exist. - `intelmq.lib.upgrades.v110_deprecations`: Fix upgrade of ripe expert configuration. - `intelmq.lib.bot_debugger`: - Fix handling of empty messages generated by parser when user wanted to show the result by "--show-sent" flag. - Fix handling of sent messages for bots using the `path_permissive` paramter (#1453). - `intelmq.lib.pipeline.Amqp`: - use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used. - Reconnect once on sending messages if disconnect detected. ### Bots #### Collectors - `intelmq.bots.collectors.api.collector_api`: - Handle non-existing IO loop in shutdown. - Close socket on shutdown, fixes reloading. - Marked as non-threadable. - `intelmq.bots.collectors.rt.collector_rt`: Check for matching URLs if no `attachment_regex` is given. - `intelmq.bots.collectors.stomp.collector_stomp`: Handle disconnects by actively reconnecting. #### Parsers - `intelmq.bots.cymru.parser_cap_program`: Fix parsing of the new `$certname_$date.txt` report format (#1443): - Support protocol ICMP. - Fix error message for unsupported protocols. - Support fields `destination_port_numbers`, `port`. - Support for all proxy types without ports. - Use Country Code of AS as `source.geolocation.cc`. - Support for 'scanner' and 'spam' categories. - Handle bogus lines with missing separator. - Fix bug preventing use of old format after using the new format. - Handle postfix ` (total_count:..)` for destination port numbers. #### Experts - `intelmq.bots.experts.cymru_whois.expert`: Add optional parameter `overwrite`, current behavior was `True`, default if not given is `True` now, will change to `False` in 3.0.0 (#1452, #1455). - `intelmq.bots.experts.modify.expert`: Add optional parameter `overwrite`, current behavior was `True`, default if not given is `True` now, will change to `False` in 3.0.0 (#1452, #1455). - `intelmq.bots.experts.reverse_dns.expert`: Add optional parameter `overwrite`, current behavior was `True`, default if not given is `True` now, will change to `False` in 3.0.0 (#1452, #1455). #### Outputs - `intelmq.bots.outputs.amqptopic.output`: use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used. ### Packaging - Rules: - Exclude intelmqsetup tool in packages - Include update-rfiprisk-data in packages ### Tests - Tests for `intelmq.lib.upgrades.v202_fixes`. - Tests for `intelmq.lib.upgrades.v110_deprecations`. - Extended tests for `intelmq.bots.parser.cymru.parser_cap_program`. ### Tools - intelmqctl: - More and more precise logging messages for botnet starting and restarting, enable and disable. - No error message for disabled bots on botnet reload. - Fix `upgrade-conf` is state file is empty or not existing. - Use arpgarse's `store_true` action for flags instead of `store_const`. - If the loading of the defaults configuration failed, a variable definition was missing and causing an exception (#1456). ### Contrib - Check MK Statistics Cronjob: - Use `statistics_*` parameters. - Make file executable - Handle None values in `*.temporary.*` keys and treat them as 0. - systemd: - Add `PIDFile` parameter to service file. ### Known issues - MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439) - ctl: shell colorizations are logged (#1436) - http stream collector: retry on regular connection problems? (#1435) - tests: capture logging with context manager (#1342) - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952) - n6 parser: mapping is modified within each run (#905) - reverse DNS: Only first record is used (#877) - Corrupt dump files when interrupted during writing (#870) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 50564167201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Permissions error
by Vaclav Bruzek 28 Aug '19

28 Aug '19

Hi mailing list, I've recently upgraded intelmq from version 1.1.2 to 2.0.1. My intelmq runs in Docker. I have a slight issue with the botnet startup on the new version (the previous version works ok). Intelmq fails to start because of the exception present at the bottom of the mail. Upon further investigation, when the folder was manually created, other issues started to pop up. All of them were due to a permission error on either the files in /opt/intelmq/etc/ or some else that intelmq tried to access during startup. Did something change in the setup requirements that I've overlooked? 28/08/2019 09:48:272019-08-28 07:48:27,796 DEBG 'intelmq' stdout output: 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 657, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 334, in log 28/08/2019 09:48:27 handler = FileHandler("%s/%s.log" % (log_path, name)) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1032, in __init__ 28/08/2019 09:48:27 StreamHandler.__init__(self, self._open()) 28/08/2019 09:48:27 File "/usr/lib/python3.6/logging/__init__.py", line 1061, in _open 28/08/2019 09:48:27 return open(self.baseFilename, self.mode, encoding=self.encoding) 28/08/2019 09:48:27FileNotFoundError: [Errno 2] No such file or directory: '/opt/intelmq/var/log/intelmqctl.log' 28/08/2019 09:48:27 28/08/2019 09:48:27During handling of the above exception, another exception occurred: 28/08/2019 09:48:27 28/08/2019 09:48:27Traceback (most recent call last): 28/08/2019 09:48:27 File "/usr/local/bin/intelmqctl", line 11, in <module> 28/08/2019 09:48:27 load_entry_point('intelmq==2.0.1', 'console_scripts', 'intelmqctl')() 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 1710, in main 28/08/2019 09:48:27 x = IntelMQController(interactive=True) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/bin/intelmqctl.py", line 661, in __init__ 28/08/2019 09:48:27 logging_level_stream=logging_level_stream) 28/08/2019 09:48:27 File "/usr/local/lib/python3.6/dist-packages/intelmq-2.0.1-py3.6.egg/intelmq/lib/utils.py", line 345, in log 28/08/2019 09:48:27 raise ValueError("Invalid configuration, neither log_path is given nor syslog is used.") 28/08/2019 09:48:27ValueError: Invalid configuration, neither log_path is given nor syslog is used. Sincerely, Václav Brůžek

2 4

IntelMQ 2.0.1 Bugfix release
by Sebastian Wagner 23 Aug '19

23 Aug '19

Dear community, I just released version 2.0.1 of IntelMQ, the first bugfix release for the 2.x series, published in May. This is also the first 2.0 release which will hit the deb/rpm stable repositories. If you are using them, you'll receive the update in the next few hours. The version further introduces a new intelmqctl command: intelmqctl upgrade-conf. This command takes care of changed configurations, or newly added parameters. Call it after every upgrade, as documented in the Upgrade documentation linked below. The tools updates the configuration if necessary and creates backup files on it's own. If it can't fix the issues automatically, it will show you an error message. A side effect is, that intelmq now saves a "state" of itself in /var/lib/intelmq/state.json or /opt/intelmq/var/lib/state.json depending on your setup. Install documentation: https://github.com/certtools/intelmq/blob/2.0.1/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.0.1/docs/UPGRADING.md As always: read the NEWS file, upgrade according to the documentation and have fun! If you get any errors, please report them here or in the bug tracker. The full changelog: ### Core - `intelmq.lib.harmonization`: - `IPAddress`: Remove Scope/Zone IDs for IPv6 addresses in sanitation. - All types: Handle `None` for validation and sanitation gracefully. - `intelmq.lib.bot`: - fix parameters of ParserBot and CollectorBot constructors, allowing `intelmqctl run` with these bots again (#1414). - Also run `rate_limit` after retry counter reset (#1431). - `__version_info__`: - is now available in the top level module. - uses integer values now instead of strings for numerical version parts - Also provide (empty) `ROOT_DIR` for non-pip installations. - `intelmq.lib.upgrades`: New library file `upgrades` with upgrade functions. - `intelmq.lib.utils`: - New function `setup_list_logging` for intelmqctl check an possibly others. - Fix return values (#1423). - New function `version_smaller` for version comparisons. - New function `lazy_int` for version conversions. - `parse_logline`: Handle thread IDs. - `log` takes a new argument `logging_level_stream` for the logging level of the console handler. - New constant `LOG_FORMAT_SIMPLE`, used by intelmqctl. - New function `write_configuration` to write dicts to files in the correct json formatting. - New function `create_request_session_from_bot`. - `intelmq.lib.pipeline`: - AMQP: - Actually use `source/destination_pipeline_amqp_virtual_host` parameter. - Support for SSL with `source/destination_pipeline_ssl` parameter. - pipeline base class: add missing dummy methods. - Add missing return types. - Redis: Evaluate return parameter of queue/key deletion. - Variable `STATE_FILE_PATH` added. ### Development - `intelmq.bin.intelmq_gen_docs`: For yaml use `safe_load` instead of unsafe `load`. ### Harmonization - IPAddress type: Remove Scope/Zone IDs for IPv6 addresses in sanitation. - TLP: Sanitation handles now more cases: case-insensitive prefixes and arbitrary whitespace between the prefix and the value (#1420). ### Bots #### Collectors - `intelmq.bots.collectors.http.collector_http`: Use `utils.create_request_session_from_bot`. - `intelmq.bots.collectors.http.collector_http_stream`: Use `utils.create_request_session_from_bot` and thus fix some retries on connection timeouts. - `intelmq.bots.collectors.mail.collector_mail_url`: Use `utils.create_request_session_from_bot`. - `intelmq.bots.collectors.microsoft.collector_interflow`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. - `intelmq.bots.collectors.rt.collector_rt`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. - `intelmq.bots.collectors.twitter.collector_twitter`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts for non-twitter connections. #### Parsers - `intelmq.bots.parsers.n6.parser_n6stomp`: use `malware-generic` instead of `generic-n6-drone` for unknown infected system events. - `intelmq.bots.parsers.abusech.parser_ip`: Support LastOnline column in feodo feed (#1400) and use it for `time.source` if available. - Use lower case malware names as default, should not make a difference in practice. - Fix handling of CSV header for feodotracker (#1417, #1418). - `intelmq.bots.parsers.netlab_360.parser`: Detect feeds with `https://` too. #### Experts - `intelmq.bots.experts.generic_db_lookup`: Recommend psycopg2-binary package. - `intelmq.bots.experts.modify.expert`: - Compile regular expressions (all string rules) at initialization, improves the speed. - Warn about old configuration style deprecation. - `intelmq.bots.experts.do_portal.expert`: - Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts (#1432). - Treat "502 Bad Gateway" as timeout which can be retried. - `intelmq.bots.experts.ripe.expert`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. - `intelmq.bots.experts.url2fqdn.expert`: Support for IP addresses in hostnames (#1416). - `intelmq.bots.experts.national_cert_contact_certat.expert`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. #### Outputs - `intelmq.bots.outputs.postgresql`: Recommend psycopg2-binary package. - `intelmq.bots.outputs.amqptopic`: - Shutdown: Close connection only if connection exists. - Add support for pika > 1. Pika changed the way it indicates (Non-)Acknowledgments of sent messages. - Gracefully handle unroutable messages and give advice. - Support for connections without authentication. - Replace deprecated parameter `type` with `exchange_type` for `exchange_declare`, supporting pika >= 0.11 (#1425). - New parameters `message_hierarchical_output`, `message_with_type`, `message_jsondict_as_string`. - New parameter `use_ssl` for SSL connections. - New parameter `single_key` for sending single fields instead of the full event. - `intelmq.bots.outputs.mongodb.output`: Support for pymongo >= 3.0.0 (#1063, PR#1421). - `intelmq.bots.outputs.file`: `time.*` field serialization: support for microseconds. - `intelmq.bots.outputs.mongodb.output`: Support for authentication in pymongo >= 3.5 (#1062). - `intelmq.bots.outputs.restapi.output`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. ### Documentation - Add certbund-contact to the ecosystem document. - Rename the IDEA expert to "IDEA Converter". - Add the new configuration upgrade function to the docs. - User Guide: - Clarify on Uninstallation ### Packaging - Do not execute the tcp collector tests during debian and ubuntu builds as they fail there. ### Tests - `intelmq.lib.test`: Disable statistics for test runs of bots. - `contrib.malware_name_mapping`: Added tests. - Travis: Also run tests of contrib. ### Tools - `intelmqsetup`: Only change directory ownerships if necessary. - `intelmqctl`: - Provide new command `upgrade-conf` to uprade configuration to a newer version. - Makes backups of configurations files on its own. - Also checks for previously skipped or new functions of older versions and catches up. - Provides logging level on class layer. - Fix `-q` flag for `intelmqctl list queues` by renaming its alternative name to `--non-zero` to avoid a name collision with the global `--quiet` parameter. - For console output the string `intelmqctl: ` at the beginning of each line is no longer present. - `check`: Support for the state file added. Checks if it exists and all upgrade functions have been executed successfully. - Waits for up to 2 seconds when stopping a bot (#1434). - Exits early on restart when stopping a bot did not work (#1434). - `intelmqctl run process -m` debugging: Mock acknowledge method if incoming message is mocked too, otherwise a different message is acknowledged. - Queue listing for AMQP: Support non-default monitoring URLs, see User-Guide. ### Contrib * logcheck rules: Adapt ignore rule to cover the instance IDs of bot names. * malware name mapping: - Ignore lines in mapping starting with '#'. - Optionally include malpedia data. - Fix command line parsing for not arguments (#1427). - bash-completion: Support for `intelmqctl upgrade-config` added. ### Known issues - http stream collector: retry on regular connection problems? (#1435) - tests: capture logging with context manager (#1342) - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952) - n6 parser: mapping is modified within each run (#905) - reverse DNS: Only first record is used (#877) - Corrupt dump files when interrupted during writing (#870) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Max retries issue
by Vaclav Bruzek 01 Aug '19

01 Aug '19

Hi mailinglist, I've went over intelmq logs today and that some bots keep retrying indefinitely when they encounter a failure. However each of those bots have *error_max_retries* set to 1 (and *error_retry_delay *to 60). Which in my mind should limit the bot to just try one more time and than give up and sleep (if sleep is set up) however this is not happening. Am I doing something wrong or does this functionality work differently? My version of intelmq is 1.1.1 in Docker on Ubuntu 18.04 as base. Following is an example from logs os collector collecting from phishtank: 2019-03-01 20:07:51,747 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:07:52,139 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:07:52,140 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:07:52,140 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:07:52,298 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:07:52,299 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:07:52,299 - phishtank-collector - INFO - Bot will continue in 60 seconds. 2019-03-01 20:08:52,359 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:08:52,654 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:08:52,656 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:08:52,656 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:08:52,831 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:08:52,832 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:08:52,832 - phishtank-collector - INFO - Bot will continue in 60 seconds. 2019-03-01 20:09:52,891 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:09:53,138 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:09:53,139 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:09:53,139 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:09:53,315 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:09:53,317 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:09:53,318 - phishtank-collector - INFO - Bot will continue in 60 seconds. Sincerely, Václav Brůžek

2 3

IntelMQ & Manager 2.0.0 Release
by Sebastian Wagner 28 May '19

28 May '19

Dear community, I am pleased to announce the second major versions of IntelMQ and IntelMQ Manager. *A big thanks goes to all the contributors engaging and contributing to this project!* Before migrating, have a look at the NEWS file: https://github.com/certtools/intelmq/blob/maintenance/NEWS.md Installation instructions: https://github.com/certtools/intelmq/blob/2.0.0/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/2.0.0/docs/UPGRADING.md The release is available also in the unstable deb/rpm repositories: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… It is *not* yet available in the stable repositories, I will do that in the next weeks or with the next bugfix release, depending on whether there are some issues popping up. Some numbers on the bots compared to the first major release: 8 new collectors since 1.0, now 21 in total 16 new parsers since 1.0, now 59 in total 37 new documented feeds since 1.0, now 138 in total 2 new experts since 1.0, now 24 in total 2 new outputs since 1.0, now 15 in total On 4th April I wrote a summary about new features to the developer's list: https://lists.cert.at/pipermail/intelmq-dev/2019-April/000293.html Here is a short summary for the core's most important new features including the links to documentation: * AMQP is available as alternative broker additionally to redis. I do *not* recommend yet to migrate if you are not familiar with RabbitMQ (an AQMP implementation). I'd still like to get more feedback on this feature. https://github.com/certtools/intelmq/blob/|maintenance|/docs/User-Guide.md#… * Supervisor can be used as replacement for intelmqctl's own pid-based process management https://github.com/certtools/intelmq/blob/|maintenance|/docs/User-Guide.md#… Note that there is also a script to generate systemd service files for intelmq bots: https://github.com/certtools/intelmq/tree/maintenance/contrib/systemd * Multithreading: By proving the parameter |instances_threads you can set the number of threads for a bot, works for most bots. https://github.com/certtools/intelmq/blob/maintenance/docs/User-Guide.md#mu… * Statistics: Mostly relevant for power users and developers, the bots save load statistics, not stable yet. Discussion is here: https://github.com/certtools/intelmq/issues/1274 * You can switch between /opt and LSB-paths with an environment variable: https://github.com/certtools/intelmq/blob/maintenance/docs/User-Guide.md#op… * "Named queue" or "paths" have already been introduced in 1.1, allowing multiple *different* destination pipelines. The filter expert supports named queues, potentially reducing the number of needed filter bots: https://github.com/certtools/intelmq/blob/maintenance/docs/Bots.md#possible… Using the sieve expert, you can set the paths for the messages too. * The separated malware name mapping since 1.1: https://github.com/certtools/intelmq/tree/maintenance/contrib/malware_name_… * The transparent usage of `extra.*` fields since 1.1. * And *a lot* of changes in the bots, it is impossible to list them here For the IntelMQ Manager: * The configuration tab shows the number of queued messages live, name of queues ("paths"), management and monitoring features. Physics can be permanently turned off (the state is saved). With the new shortcuts, you can do some actions with keyboard only. * List and array parameters are shown correctly as JSON/JS. * The management allows to manage bot groups (intelmqctl's --group feature). * The monitor page has an advanced inspection tool and further allows management of the bot. * Error handling and communication to the user: Instead of alerts a message box shows a summary and counter of errors messages. * Many more and lots of improvements under the hood I am sure that I forgot something here ;) so for the full changelogs please refer to the repository: https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md https://github.com/certtools/intelmq-manager/blob/develop/CHANGELOG.md Co-financed by the Connecting Europe Facility of the European Union -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Pipeline erros
by Vaclav Bruzek 20 May '19

20 May '19

Hi mailing list, I've recently began experiencing issues relating Redis, usually with timeouts. In case this issue happens some bots usually shut down. Most common exception is lib.exceptions.PipelineError (sometimes redis.exceptions.TimeoutError). However, bots shut down after different number of exceptions occur for each of them, without any clear pattern. The behaviour is even stranger since some bots seem to recover from these exceptions and then suspend anyway (attached log). Why is this happening? Shouldn't all bots suspend after the same number of exceptions? i' running intelmq and Redis in Docker, with intelmq version 1.1.2 and with all bots initiated with 'error_procedure': 'pass'. Sincerely, Václav Brůžek

1 0

Logging summary
by Vaclav Bruzek 30 Apr '19

30 Apr '19

Hi mailing list, I've recently encountered and issue with some of my custom parser bots. The default parsers log a line like this: Sent 7176 events and found 0 error(s) This is quite fine piece of information but I am not getting that for my custom parsers. The only info i have is the number of processed messages. Is there a need to do something special apart from 'acknowledge_message()' and 'send_message()'? Sincerely, Václav Brůžek

2 3

IntelMQ 2.0.0 Beta 1 release
by Sebastian Wagner 10 Apr '19

10 Apr '19

Dear community, The long awaited next major release is coming - in May. For now, there's a beta release as it still has a few bugs we should fix. The release will hit the unstable deb/rpm repositories shortly: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… Thanks to all contributors who made IntelMQ what it is today! Installation instructions: https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/UPGRADING.md The full changelog: There are some features considered as beta and marked as such in the documentation, do not use them in production yet. - upgraded all files to python3-only syntax, e.g. use `super()` instead of `super(..., ...)` in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer. ### Removals of deprecated code: - Removed compatibility shim `intelmq.bots.collectors.n6.collector_stomp`, use `intelmq.bots.collectors.stomp.collector` instead (see #1124). - Removed compatibility shim `intelmq.bots.parsers.cymru_full_bogons.parser`, use `intelmq.bots.parsers.cymru.parser_full_bogons` instead. - Removed compatibility shim handing deprecated parameter `feed` for collectors. Use `name` instead. - Removed deprecated and unused method `intelmq.lib.pipeline.Pipeline.sleep`. - Removed support for deprecated parameter `query_ripe_stat` in `intelmq.bots.experts.ripe.expert`, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (#1291). - Removed deprecated and unused function `intelmq.lib.utils.extract_tar`. ### Core - `lib/pipeline`: - Allow setting the broker of source and destination independently. - Support for a new AMQP broker. See User Guide for configuration. (#1179) - `lib/bot`: - Dump messages locks the dump file using unix file locks (#574). - Print idle/rate limit time also in human readable format (#1332). - `set_request_parameters`: Use `{}` as default proxy value instead of `None`. Allows updating of existing proxy dictionaries. - Bots drop privileges if they run as root. - Save statistics on successfully and failed processed messages in the redis database 3. - `lib/utils` - Function `unzip` to extract files from gzipped and/or tar-archives. - New class `ListHandler`: new handler for logging purpose which saves the messages in a list. - Add function `seconds_to_human`. - Add function `drop_privileges`. - `parse_relative`: Strip string before parsing. - `parse_logline`: Do not convert the timestamps to UTC, leave them as is. - `lib/cache`: - Allow ttl to be None explicitly. - Overwrite existing cache keys in the database instead of discarding the new data. - `lib/bot`: - Basic, but easy-to-configure multi-threading using python's `threading` library. See the User-Guide for more information (#111, #186). - `bin/intelmqctl`: - Support for Supervisor as process manager (#693, #1360). ### Harmonization ### Bots #### Collectors - added `intelmq.bots.parsers.opendxl.collector` (#1265). - added `intelmq.bots.collectors.api`: collecting data using an HTTP API (#123, #1187). - added `intelmq.bots.collectors.rsync` (#1286). - `intelmq.bots.collectors.http.collector_http`: - Add support for uncompressing of gzipped-files (#1270). - Add time-delta support for time formatted URLs (#1366). - `intelmq.collectors.blueliv.collector_crimeserver`: Allow setting the API URL by parameter (#1336). - `intelmq.collectors.mail`: - Use internal lib for functionality. - Add `intelmq.bots.collectors.mail.collector_mail_body`. - Support for `ssl_ca_certificate` parameter (#1362). #### Parsers - added `intelmq.bots.parsers.mcafee.parser_atd` (#1265). - `intelmq.bots.parsers.generic.parser_csv`: - New parameter `columns_required` to optionally ignore parse errors for columns. - added `intelmq.bots.parsers.cert_eu.parser_csv` (#1287). - Do not overwrite the local `time.observation` with the data from the feed. The feed's field 'observation time' is now saved in the field `extra.cert_eu_time_observation`. - Fix parsing of `asn` (renamed to `source asn`, `source.asn` internally) and handle existing `feed.accuracy` for parsing `confidence`. - Update columns and mapping to current (2019-04-02) data. - added `intelmq.bots.parsers.surbl.surbl` - added `intelmq.bots.parsers.html_table` (#1381). - `intelmq.bot.parsers.netlab_360.parser`: Handle empty lines containing blank characters (#1393). - `intelmq.bots.parsers.n6.parser_n6stomp`: Handle events without IP addresses. - `intelmq.bots.parsers.cymru.parser_cap_program`: Handle new feed format. - `intelmq.bots.parsers.shadowserver`: - Add support for the `Accessible-FTP` feed (#1391). - `intelmq.bots.parsers.dataplane.parser`: - Fix parse errors and log more context (#1396). - added `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py` (#1373). #### Experts - added `intelmq.bots.experts.recordedfuture_iprisk` (#1267). - added `intelmq.bots.experts.mcafee.expert_mar` (1265). - renamed `intelmq.bots.experts.ripencc_abuse_contact.expert` to `intelmq.bots.experts.ripe.expert`, compatibility shim will be removed in version 3.0. - Added support for geolocation information in ripe expert with a new parameter `query_ripe_stat_geolocation` (#1317). - Restructurize the expert and de-duplicataion (#1384). - Handle '?' in geolocation country data (#1384). - `intelmq.bots.experts.ripe.expert`: - Use a requests session (#1363). - Set the requests parameters once per session. - `intelmq.bots.experts.maxmind_geoip.expert`: New parameter `use_registered` to use the registered country (#1344). - `intelmq.bots.experts.filter.expert`: Support for paths (#1208). #### Outputs - added `intelmq.bots.experts.mcafee.output_esm` (1265). - added `intelmq.bots.outputs.blackhole` (#1279). - `intelmq.bots.outputs.restapi.expert`: - Set the requests parameters once per session. - `intelmq.bots.outputs.redis`: - New parameter `hierarchichal_output` (#1388). - New parameter `with_type`. - `intelmq.bots.outputs.amqptopic.output`: Compatibility with pika 1.0.0 (#1084, #1394). ### Documentation - added documentation for feeds - CyberCrime Tracker - Feodo Tracker Latest - Feeds: Document abuse.ch URLhaus feed (#1379). - Install and Upgrading: Use `intelmqsetup` tool. ### Packaging ### Tests - Add tests of AMQP broker. - Travis: Change the ownership of `/opt/intelmq` to the current user. ### Tools - `intelmqctl check`: Now uses the new `ListHandler` from utils to handle the logging in JSON output mode. - `intelmqctl run`: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behaviour. - `intelmqdump`: Inspecting dumps locks the dump file using unix file locks (#574). - `intelmqctl`: - After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown. - `intelmqsetup`: New tool for initialize an IntelMQ environment. ### Contrib - `malware_name_mapping`: - Added the script `apply_mapping_eventdb.py` to apply the mapping to an eventdb. - Possibility to add local rules using the download tool. - `check_mk`: - Added scripts for monitoring queues and statistics. ### Known issues - Multi-threaded bots require multiple SIGTERMs (#1403) - Stats can't be saved with AMQP if redis is password-protected (#1402) - Update taxonomies to current RSIT and vice-versa (#1380) - stomp collector bot constantly uses 100% of CPU (#1364) - tests: capture logging with context manager (#1342) - Consistent message counter log messages for all kind of bots (#1278) - pymongo 3.0 deprecates used insert method (#1063) - pymongo >= 3.5: authentication changes (#1062) - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952) - n6 parser: mapping is modified within each run (#905) - reverse DNS: Only first record is used (#877) - Corrupt dump files when interrupted during writing (#870) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Re: [Intelmq-users] Uncaught bounce notification
by Sebastian Wagner 05 Apr '19

05 Apr '19

Hi Majid You sent your mail to intelmq-dev-*bounces*(a)lists.cert.at instead of intelmq-users(a)lists.cert.at or intelmq-dev-bounces(a)lists.cert.at The bounces address has a special meaning, it is not for sending mails to the list. Yes, you need one parser per feed and provide the feed's name as parameter to the parser. Sebastian On 05/04/2019 13.35, "Salehi Ghamsari, Majid" <majid.salehi.ghamsari(a)fokus.fraunhofer.de> wrote: > Hi there, > > to parse multiple resports from shadowserver like Open-Redis and Open-SNMP, do i need multiple shadowserver parser or > there is a simple way to parse multiple reposts. > > Thank you for your help > Majid Salehi -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users