- IntelMQ-users

IntelMQ 2.0.1 Bugfix release
by Sebastian Wagner 23 Aug '19

23 Aug '19

Dear community, I just released version 2.0.1 of IntelMQ, the first bugfix release for the 2.x series, published in May. This is also the first 2.0 release which will hit the deb/rpm stable repositories. If you are using them, you'll receive the update in the next few hours. The version further introduces a new intelmqctl command: intelmqctl upgrade-conf. This command takes care of changed configurations, or newly added parameters. Call it after every upgrade, as documented in the Upgrade documentation linked below. The tools updates the configuration if necessary and creates backup files on it's own. If it can't fix the issues automatically, it will show you an error message. A side effect is, that intelmq now saves a "state" of itself in /var/lib/intelmq/state.json or /opt/intelmq/var/lib/state.json depending on your setup. Install documentation: https://github.com/certtools/intelmq/blob/2.0.1/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.0.1/docs/UPGRADING.md As always: read the NEWS file, upgrade according to the documentation and have fun! If you get any errors, please report them here or in the bug tracker. The full changelog: ### Core - `intelmq.lib.harmonization`: - `IPAddress`: Remove Scope/Zone IDs for IPv6 addresses in sanitation. - All types: Handle `None` for validation and sanitation gracefully. - `intelmq.lib.bot`: - fix parameters of ParserBot and CollectorBot constructors, allowing `intelmqctl run` with these bots again (#1414). - Also run `rate_limit` after retry counter reset (#1431). - `__version_info__`: - is now available in the top level module. - uses integer values now instead of strings for numerical version parts - Also provide (empty) `ROOT_DIR` for non-pip installations. - `intelmq.lib.upgrades`: New library file `upgrades` with upgrade functions. - `intelmq.lib.utils`: - New function `setup_list_logging` for intelmqctl check an possibly others. - Fix return values (#1423). - New function `version_smaller` for version comparisons. - New function `lazy_int` for version conversions. - `parse_logline`: Handle thread IDs. - `log` takes a new argument `logging_level_stream` for the logging level of the console handler. - New constant `LOG_FORMAT_SIMPLE`, used by intelmqctl. - New function `write_configuration` to write dicts to files in the correct json formatting. - New function `create_request_session_from_bot`. - `intelmq.lib.pipeline`: - AMQP: - Actually use `source/destination_pipeline_amqp_virtual_host` parameter. - Support for SSL with `source/destination_pipeline_ssl` parameter. - pipeline base class: add missing dummy methods. - Add missing return types. - Redis: Evaluate return parameter of queue/key deletion. - Variable `STATE_FILE_PATH` added. ### Development - `intelmq.bin.intelmq_gen_docs`: For yaml use `safe_load` instead of unsafe `load`. ### Harmonization - IPAddress type: Remove Scope/Zone IDs for IPv6 addresses in sanitation. - TLP: Sanitation handles now more cases: case-insensitive prefixes and arbitrary whitespace between the prefix and the value (#1420). ### Bots #### Collectors - `intelmq.bots.collectors.http.collector_http`: Use `utils.create_request_session_from_bot`. - `intelmq.bots.collectors.http.collector_http_stream`: Use `utils.create_request_session_from_bot` and thus fix some retries on connection timeouts. - `intelmq.bots.collectors.mail.collector_mail_url`: Use `utils.create_request_session_from_bot`. - `intelmq.bots.collectors.microsoft.collector_interflow`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. - `intelmq.bots.collectors.rt.collector_rt`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. - `intelmq.bots.collectors.twitter.collector_twitter`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts for non-twitter connections. #### Parsers - `intelmq.bots.parsers.n6.parser_n6stomp`: use `malware-generic` instead of `generic-n6-drone` for unknown infected system events. - `intelmq.bots.parsers.abusech.parser_ip`: Support LastOnline column in feodo feed (#1400) and use it for `time.source` if available. - Use lower case malware names as default, should not make a difference in practice. - Fix handling of CSV header for feodotracker (#1417, #1418). - `intelmq.bots.parsers.netlab_360.parser`: Detect feeds with `https://` too. #### Experts - `intelmq.bots.experts.generic_db_lookup`: Recommend psycopg2-binary package. - `intelmq.bots.experts.modify.expert`: - Compile regular expressions (all string rules) at initialization, improves the speed. - Warn about old configuration style deprecation. - `intelmq.bots.experts.do_portal.expert`: - Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts (#1432). - Treat "502 Bad Gateway" as timeout which can be retried. - `intelmq.bots.experts.ripe.expert`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. - `intelmq.bots.experts.url2fqdn.expert`: Support for IP addresses in hostnames (#1416). - `intelmq.bots.experts.national_cert_contact_certat.expert`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. #### Outputs - `intelmq.bots.outputs.postgresql`: Recommend psycopg2-binary package. - `intelmq.bots.outputs.amqptopic`: - Shutdown: Close connection only if connection exists. - Add support for pika > 1. Pika changed the way it indicates (Non-)Acknowledgments of sent messages. - Gracefully handle unroutable messages and give advice. - Support for connections without authentication. - Replace deprecated parameter `type` with `exchange_type` for `exchange_declare`, supporting pika >= 0.11 (#1425). - New parameters `message_hierarchical_output`, `message_with_type`, `message_jsondict_as_string`. - New parameter `use_ssl` for SSL connections. - New parameter `single_key` for sending single fields instead of the full event. - `intelmq.bots.outputs.mongodb.output`: Support for pymongo >= 3.0.0 (#1063, PR#1421). - `intelmq.bots.outputs.file`: `time.*` field serialization: support for microseconds. - `intelmq.bots.outputs.mongodb.output`: Support for authentication in pymongo >= 3.5 (#1062). - `intelmq.bots.outputs.restapi.output`: Use `utils.create_request_session_from_bot` and thus fix retries on connection timeouts. ### Documentation - Add certbund-contact to the ecosystem document. - Rename the IDEA expert to "IDEA Converter". - Add the new configuration upgrade function to the docs. - User Guide: - Clarify on Uninstallation ### Packaging - Do not execute the tcp collector tests during debian and ubuntu builds as they fail there. ### Tests - `intelmq.lib.test`: Disable statistics for test runs of bots. - `contrib.malware_name_mapping`: Added tests. - Travis: Also run tests of contrib. ### Tools - `intelmqsetup`: Only change directory ownerships if necessary. - `intelmqctl`: - Provide new command `upgrade-conf` to uprade configuration to a newer version. - Makes backups of configurations files on its own. - Also checks for previously skipped or new functions of older versions and catches up. - Provides logging level on class layer. - Fix `-q` flag for `intelmqctl list queues` by renaming its alternative name to `--non-zero` to avoid a name collision with the global `--quiet` parameter. - For console output the string `intelmqctl: ` at the beginning of each line is no longer present. - `check`: Support for the state file added. Checks if it exists and all upgrade functions have been executed successfully. - Waits for up to 2 seconds when stopping a bot (#1434). - Exits early on restart when stopping a bot did not work (#1434). - `intelmqctl run process -m` debugging: Mock acknowledge method if incoming message is mocked too, otherwise a different message is acknowledged. - Queue listing for AMQP: Support non-default monitoring URLs, see User-Guide. ### Contrib * logcheck rules: Adapt ignore rule to cover the instance IDs of bot names. * malware name mapping: - Ignore lines in mapping starting with '#'. - Optionally include malpedia data. - Fix command line parsing for not arguments (#1427). - bash-completion: Support for `intelmqctl upgrade-config` added. ### Known issues - http stream collector: retry on regular connection problems? (#1435) - tests: capture logging with context manager (#1342) - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952) - n6 parser: mapping is modified within each run (#905) - reverse DNS: Only first record is used (#877) - Corrupt dump files when interrupted during writing (#870) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Max retries issue
by Vaclav Bruzek 01 Aug '19

01 Aug '19

Hi mailinglist, I've went over intelmq logs today and that some bots keep retrying indefinitely when they encounter a failure. However each of those bots have *error_max_retries* set to 1 (and *error_retry_delay *to 60). Which in my mind should limit the bot to just try one more time and than give up and sleep (if sleep is set up) however this is not happening. Am I doing something wrong or does this functionality work differently? My version of intelmq is 1.1.1 in Docker on Ubuntu 18.04 as base. Following is an example from logs os collector collecting from phishtank: 2019-03-01 20:07:51,747 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:07:52,139 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:07:52,140 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:07:52,140 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:07:52,298 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:07:52,299 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:07:52,299 - phishtank-collector - INFO - Bot will continue in 60 seconds. 2019-03-01 20:08:52,359 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:08:52,654 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:08:52,656 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:08:52,656 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:08:52,831 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:08:52,832 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:08:52,832 - phishtank-collector - INFO - Bot will continue in 60 seconds. 2019-03-01 20:09:52,891 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:09:53,138 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:09:53,139 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:09:53,139 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:09:53,315 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:09:53,317 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:09:53,318 - phishtank-collector - INFO - Bot will continue in 60 seconds. Sincerely, Václav Brůžek

2 3

IntelMQ & Manager 2.0.0 Release
by Sebastian Wagner 28 May '19

28 May '19

Dear community, I am pleased to announce the second major versions of IntelMQ and IntelMQ Manager. *A big thanks goes to all the contributors engaging and contributing to this project!* Before migrating, have a look at the NEWS file: https://github.com/certtools/intelmq/blob/maintenance/NEWS.md Installation instructions: https://github.com/certtools/intelmq/blob/2.0.0/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/2.0.0/docs/UPGRADING.md The release is available also in the unstable deb/rpm repositories: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… It is *not* yet available in the stable repositories, I will do that in the next weeks or with the next bugfix release, depending on whether there are some issues popping up. Some numbers on the bots compared to the first major release: 8 new collectors since 1.0, now 21 in total 16 new parsers since 1.0, now 59 in total 37 new documented feeds since 1.0, now 138 in total 2 new experts since 1.0, now 24 in total 2 new outputs since 1.0, now 15 in total On 4th April I wrote a summary about new features to the developer's list: https://lists.cert.at/pipermail/intelmq-dev/2019-April/000293.html Here is a short summary for the core's most important new features including the links to documentation: * AMQP is available as alternative broker additionally to redis. I do *not* recommend yet to migrate if you are not familiar with RabbitMQ (an AQMP implementation). I'd still like to get more feedback on this feature. https://github.com/certtools/intelmq/blob/|maintenance|/docs/User-Guide.md#… * Supervisor can be used as replacement for intelmqctl's own pid-based process management https://github.com/certtools/intelmq/blob/|maintenance|/docs/User-Guide.md#… Note that there is also a script to generate systemd service files for intelmq bots: https://github.com/certtools/intelmq/tree/maintenance/contrib/systemd * Multithreading: By proving the parameter |instances_threads you can set the number of threads for a bot, works for most bots. https://github.com/certtools/intelmq/blob/maintenance/docs/User-Guide.md#mu… * Statistics: Mostly relevant for power users and developers, the bots save load statistics, not stable yet. Discussion is here: https://github.com/certtools/intelmq/issues/1274 * You can switch between /opt and LSB-paths with an environment variable: https://github.com/certtools/intelmq/blob/maintenance/docs/User-Guide.md#op… * "Named queue" or "paths" have already been introduced in 1.1, allowing multiple *different* destination pipelines. The filter expert supports named queues, potentially reducing the number of needed filter bots: https://github.com/certtools/intelmq/blob/maintenance/docs/Bots.md#possible… Using the sieve expert, you can set the paths for the messages too. * The separated malware name mapping since 1.1: https://github.com/certtools/intelmq/tree/maintenance/contrib/malware_name_… * The transparent usage of `extra.*` fields since 1.1. * And *a lot* of changes in the bots, it is impossible to list them here For the IntelMQ Manager: * The configuration tab shows the number of queued messages live, name of queues ("paths"), management and monitoring features. Physics can be permanently turned off (the state is saved). With the new shortcuts, you can do some actions with keyboard only. * List and array parameters are shown correctly as JSON/JS. * The management allows to manage bot groups (intelmqctl's --group feature). * The monitor page has an advanced inspection tool and further allows management of the bot. * Error handling and communication to the user: Instead of alerts a message box shows a summary and counter of errors messages. * Many more and lots of improvements under the hood I am sure that I forgot something here ;) so for the full changelogs please refer to the repository: https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md https://github.com/certtools/intelmq-manager/blob/develop/CHANGELOG.md Co-financed by the Connecting Europe Facility of the European Union -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Pipeline erros
by Vaclav Bruzek 20 May '19

20 May '19

Hi mailing list, I've recently began experiencing issues relating Redis, usually with timeouts. In case this issue happens some bots usually shut down. Most common exception is lib.exceptions.PipelineError (sometimes redis.exceptions.TimeoutError). However, bots shut down after different number of exceptions occur for each of them, without any clear pattern. The behaviour is even stranger since some bots seem to recover from these exceptions and then suspend anyway (attached log). Why is this happening? Shouldn't all bots suspend after the same number of exceptions? i' running intelmq and Redis in Docker, with intelmq version 1.1.2 and with all bots initiated with 'error_procedure': 'pass'. Sincerely, Václav Brůžek

1 0

Logging summary
by Vaclav Bruzek 30 Apr '19

30 Apr '19

Hi mailing list, I've recently encountered and issue with some of my custom parser bots. The default parsers log a line like this: Sent 7176 events and found 0 error(s) This is quite fine piece of information but I am not getting that for my custom parsers. The only info i have is the number of processed messages. Is there a need to do something special apart from 'acknowledge_message()' and 'send_message()'? Sincerely, Václav Brůžek

2 3

IntelMQ 2.0.0 Beta 1 release
by Sebastian Wagner 10 Apr '19

10 Apr '19

Dear community, The long awaited next major release is coming - in May. For now, there's a beta release as it still has a few bugs we should fix. The release will hit the unstable deb/rpm repositories shortly: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… Thanks to all contributors who made IntelMQ what it is today! Installation instructions: https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/UPGRADING.md The full changelog: There are some features considered as beta and marked as such in the documentation, do not use them in production yet. - upgraded all files to python3-only syntax, e.g. use `super()` instead of `super(..., ...)` in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer. ### Removals of deprecated code: - Removed compatibility shim `intelmq.bots.collectors.n6.collector_stomp`, use `intelmq.bots.collectors.stomp.collector` instead (see #1124). - Removed compatibility shim `intelmq.bots.parsers.cymru_full_bogons.parser`, use `intelmq.bots.parsers.cymru.parser_full_bogons` instead. - Removed compatibility shim handing deprecated parameter `feed` for collectors. Use `name` instead. - Removed deprecated and unused method `intelmq.lib.pipeline.Pipeline.sleep`. - Removed support for deprecated parameter `query_ripe_stat` in `intelmq.bots.experts.ripe.expert`, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (#1291). - Removed deprecated and unused function `intelmq.lib.utils.extract_tar`. ### Core - `lib/pipeline`: - Allow setting the broker of source and destination independently. - Support for a new AMQP broker. See User Guide for configuration. (#1179) - `lib/bot`: - Dump messages locks the dump file using unix file locks (#574). - Print idle/rate limit time also in human readable format (#1332). - `set_request_parameters`: Use `{}` as default proxy value instead of `None`. Allows updating of existing proxy dictionaries. - Bots drop privileges if they run as root. - Save statistics on successfully and failed processed messages in the redis database 3. - `lib/utils` - Function `unzip` to extract files from gzipped and/or tar-archives. - New class `ListHandler`: new handler for logging purpose which saves the messages in a list. - Add function `seconds_to_human`. - Add function `drop_privileges`. - `parse_relative`: Strip string before parsing. - `parse_logline`: Do not convert the timestamps to UTC, leave them as is. - `lib/cache`: - Allow ttl to be None explicitly. - Overwrite existing cache keys in the database instead of discarding the new data. - `lib/bot`: - Basic, but easy-to-configure multi-threading using python's `threading` library. See the User-Guide for more information (#111, #186). - `bin/intelmqctl`: - Support for Supervisor as process manager (#693, #1360). ### Harmonization ### Bots #### Collectors - added `intelmq.bots.parsers.opendxl.collector` (#1265). - added `intelmq.bots.collectors.api`: collecting data using an HTTP API (#123, #1187). - added `intelmq.bots.collectors.rsync` (#1286). - `intelmq.bots.collectors.http.collector_http`: - Add support for uncompressing of gzipped-files (#1270). - Add time-delta support for time formatted URLs (#1366). - `intelmq.collectors.blueliv.collector_crimeserver`: Allow setting the API URL by parameter (#1336). - `intelmq.collectors.mail`: - Use internal lib for functionality. - Add `intelmq.bots.collectors.mail.collector_mail_body`. - Support for `ssl_ca_certificate` parameter (#1362). #### Parsers - added `intelmq.bots.parsers.mcafee.parser_atd` (#1265). - `intelmq.bots.parsers.generic.parser_csv`: - New parameter `columns_required` to optionally ignore parse errors for columns. - added `intelmq.bots.parsers.cert_eu.parser_csv` (#1287). - Do not overwrite the local `time.observation` with the data from the feed. The feed's field 'observation time' is now saved in the field `extra.cert_eu_time_observation`. - Fix parsing of `asn` (renamed to `source asn`, `source.asn` internally) and handle existing `feed.accuracy` for parsing `confidence`. - Update columns and mapping to current (2019-04-02) data. - added `intelmq.bots.parsers.surbl.surbl` - added `intelmq.bots.parsers.html_table` (#1381). - `intelmq.bot.parsers.netlab_360.parser`: Handle empty lines containing blank characters (#1393). - `intelmq.bots.parsers.n6.parser_n6stomp`: Handle events without IP addresses. - `intelmq.bots.parsers.cymru.parser_cap_program`: Handle new feed format. - `intelmq.bots.parsers.shadowserver`: - Add support for the `Accessible-FTP` feed (#1391). - `intelmq.bots.parsers.dataplane.parser`: - Fix parse errors and log more context (#1396). - added `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py` (#1373). #### Experts - added `intelmq.bots.experts.recordedfuture_iprisk` (#1267). - added `intelmq.bots.experts.mcafee.expert_mar` (1265). - renamed `intelmq.bots.experts.ripencc_abuse_contact.expert` to `intelmq.bots.experts.ripe.expert`, compatibility shim will be removed in version 3.0. - Added support for geolocation information in ripe expert with a new parameter `query_ripe_stat_geolocation` (#1317). - Restructurize the expert and de-duplicataion (#1384). - Handle '?' in geolocation country data (#1384). - `intelmq.bots.experts.ripe.expert`: - Use a requests session (#1363). - Set the requests parameters once per session. - `intelmq.bots.experts.maxmind_geoip.expert`: New parameter `use_registered` to use the registered country (#1344). - `intelmq.bots.experts.filter.expert`: Support for paths (#1208). #### Outputs - added `intelmq.bots.experts.mcafee.output_esm` (1265). - added `intelmq.bots.outputs.blackhole` (#1279). - `intelmq.bots.outputs.restapi.expert`: - Set the requests parameters once per session. - `intelmq.bots.outputs.redis`: - New parameter `hierarchichal_output` (#1388). - New parameter `with_type`. - `intelmq.bots.outputs.amqptopic.output`: Compatibility with pika 1.0.0 (#1084, #1394). ### Documentation - added documentation for feeds - CyberCrime Tracker - Feodo Tracker Latest - Feeds: Document abuse.ch URLhaus feed (#1379). - Install and Upgrading: Use `intelmqsetup` tool. ### Packaging ### Tests - Add tests of AMQP broker. - Travis: Change the ownership of `/opt/intelmq` to the current user. ### Tools - `intelmqctl check`: Now uses the new `ListHandler` from utils to handle the logging in JSON output mode. - `intelmqctl run`: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behaviour. - `intelmqdump`: Inspecting dumps locks the dump file using unix file locks (#574). - `intelmqctl`: - After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown. - `intelmqsetup`: New tool for initialize an IntelMQ environment. ### Contrib - `malware_name_mapping`: - Added the script `apply_mapping_eventdb.py` to apply the mapping to an eventdb. - Possibility to add local rules using the download tool. - `check_mk`: - Added scripts for monitoring queues and statistics. ### Known issues - Multi-threaded bots require multiple SIGTERMs (#1403) - Stats can't be saved with AMQP if redis is password-protected (#1402) - Update taxonomies to current RSIT and vice-versa (#1380) - stomp collector bot constantly uses 100% of CPU (#1364) - tests: capture logging with context manager (#1342) - Consistent message counter log messages for all kind of bots (#1278) - pymongo 3.0 deprecates used insert method (#1063) - pymongo >= 3.5: authentication changes (#1062) - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952) - n6 parser: mapping is modified within each run (#905) - reverse DNS: Only first record is used (#877) - Corrupt dump files when interrupted during writing (#870) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Re: [Intelmq-users] Uncaught bounce notification
by Sebastian Wagner 05 Apr '19

05 Apr '19

Hi Majid You sent your mail to intelmq-dev-*bounces*(a)lists.cert.at instead of intelmq-users(a)lists.cert.at or intelmq-dev-bounces(a)lists.cert.at The bounces address has a special meaning, it is not for sending mails to the list. Yes, you need one parser per feed and provide the feed's name as parameter to the parser. Sebastian On 05/04/2019 13.35, "Salehi Ghamsari, Majid" <majid.salehi.ghamsari(a)fokus.fraunhofer.de> wrote: > Hi there, > > to parse multiple resports from shadowserver like Open-Redis and Open-SNMP, do i need multiple shadowserver parser or > there is a simple way to parse multiple reposts. > > Thank you for your help > Majid Salehi -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 1.1.2 release
by Sebastian Wagner 25 Mar '19

25 Mar '19

Dear community, This is presumably the last bugfix release before the next feature release. That is expected to be the version 2.0, not 1.2 in the next month. The release will hit the deb/rpm repositories shortly. Thanks to all contributors! Installation instructions: https://github.com/certtools/intelmq/blob/1.1.2/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.2/docs/UPGRADING.md ### Configuration #### Feodotracker * The URL of the "Feodo Tracker IPs" feed has changed. The new one is `https://feodotracker.abuse.ch/downloads/ipblocklist.csv`. If you are using this feed, adapt your configuration accordingly. The parser has been updated to support the new format. * The feed "Feodo Tracker Domains" has been discontinued. The full changelog: ### Core - `intelmq.lib.bot`: - `Bot.__handle_sighup`: Handle exceptions in `shutdown` method of bots. ### Harmonization - FQDN: Disallow `:` in FQDN values to prevent values like '10.0.0.1:8080' (#1235). ### Bots #### Collectors - `intelmq.bots.collectors.stomp.collector` - Fix name of shutdown method, was ineffective in the past. - Ignore `NotConnectedException` errors on disconnect during shutdown. - `intelmq.bots.collectors.mail.collector_mail_url`: Decode body if it is bytes (#1367). - `intelmq.bots.collectors.tcp.collector`: Timeout added. More stable version. #### Parsers - `intelmq.bots.parsers.shadowserver`: - Add support for the `Amplification-DDoS-Victim`, `HTTP-Scanners`, `ICS-Scanners` and `Accessible-Ubiquiti-Discovery-Service` feeds (#1368, #1383) - `intelmq.bots.parsers.microsoft.parser_ctip`: - Workaround for mis-formatted data in `networkdestinationipv4` field (since 2019-03-14). - Ignore "hostname" ("destination.fqdn") if it contains invalid data. - `intelmq.bots.parsers.shodan.parser`: - In `minimal_mode`: - Fix the parsing, previously only `source.geolocation.cc` and `extra.shodan` was correctly filled with information. - Add a `classification.type` = 'other' to all events. - Added tests for this mode. - Normal mode: - Fix the parsing of `timestamp` to `time.source in the normal mode, previously no timezone information has been added and thus every event raised an exception. - ISAKMP: Ignore `isakmp.aggressive`, as the content is same as `isakmp` or less. - `intelmq.bots.parsers.abusech.parser_ip`: Re-structure the bot and support new format of the changed "Feodo Tracker Domains" feed. - `intelmq.bots.parsers.n6.parser`: - Add parsing for fields "confidence", "expires" and "source". - Add support for type "bl-other" (category "other"). #### Experts - `intelmq.bots.experts.sieve.expert`: Fix key definition to allow field names with numbers (`malware.hash.md5`/`sha1`, #1371). #### Outputs - `intelmq.bots.outputs.tcp.output`: Timeout added. When no separator used, awaits that every message is acknowledged by a simple "Ok" string to ensure more stability. ### Documentation - Install: Update operating system versions - Sieve Expert: Fix `elsif` -> `elif`. - Rephrase the description of `time.*` fields. - Feeds: New URL and format of the "Feodo Tracker IPs" feed. "Feodo Tracker Domains" has been discontinued. ### Packaging ### Tests - Add missing `__init__.py` files in 4 bot's test directories. Previously these tests have never been executed. - `intelmq.lib.test`: Allow bot test class names with an arbitrary postfix separated by an underscore. E.g. `TestShodanParserBot_minimal`. ### Tools - intelmqctl: - status: Show commandline differences if a program with the expected PID could be found, but they do not match (previous output was `None`). - Use logging level from defauls configuration if possible, otherwise intelmq's internal default. Previously, DEBUG was used unconditionally. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). - stomp collector bot constantly uses 100% of CPU (#1364). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 1.1.1 released
by Sebastian Wagner 15 Jan '19

15 Jan '19

Dear community, Today I released the first maintenance version of the 1.1.x series. Thanks for all the contributions! Installation instructions: https://github.com/certtools/intelmq/blob/1.1.1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.1/docs/UPGRADING.md The full change log: ### Core - `lib/harmonization.py`: Change `parse_utc_isoformat` of `DateTime` class from private to public (related to #1322). - `lib/utils.py`: Add new function `object_pair_hook_bots`. - `lib.bot.py`: - `ParserBot`'s method `recover_line_csv` now also handles given `tempdata`. - `Bot.acknowledge_message()` deletes `__current_message` to free the memory, saves memory in idling parsers with big reports. - `start()`: Warn once per run if `error_dump_message` is set to false. - `Bot.start()`, `ParserBot.process()`: If errors happen on bots without destination pipeline, the `on_error` path has been queried and lead to an exception being raised. - `start()`: If `error_procedure` is pass and on pipeline errors, the bot retries forever (#1333). - `lib/message.py`: - Fix add('extra', ..., overwrite=True): old extra fields have not been deleted previously (#1335). - Do not ignore empty or ignored (as defined in `_IGNORED_VALUES`) values of `extra.*` fields for backwards compatibility (#1335). - `lib/pipeline.py` (`Redis.receive`): Wait in 1s steps if redis is busy loading its snapshot from disk (#1334). ### Default configuration - Set `error_dump_message` to true by default in `defaults.conf`. - Fixed typo in `defaults.conf`: `proccess_manager` -> `process_manager` ### Development - `bin/rewrite_config_files.py`: Fix ordering of BOTS file (#1327). ### Harmonization Update to 2018-09-26 version. New values are per taxonomy: - Taxonomy 'intrusions': - "application-compromise" - "burglary" - "privileged-account-compromise" - "unprivileged-account-compromise" - Taxonomy 'fraud': - "copyright" - "masquerade" - "unauthorized-use-of-resources" - Taxonomy 'information content security': - "data-loss" - Taxonomy 'vulnerable': - "ddos-amplifier" - "information-disclosure" - "potentially-unwanted-accessible" - "vulnerable-system" - "weak-crypto" - Taxonomy 'availability': - "dos" - "outage" - "sabotage" - Taxonomy 'abusive-content': - "harmful-speech" - "violence" - Taxonomy 'malicious code': - "malware-distribution" - Taxonomy 'information-gathering': - "social-engineering" - "sniffing" - Taxonomy 'information content security': - "Unauthorised-information-access" - "Unauthorised-information-modification" ### Bots #### Collectors - `intelmq.bots.collectors.http.collector_http`: - Fix parameter name `extract_files` in BOTS (#1331). - Fix handling of `extract_files` parameter if the value is an empty string. - Handle not installed dependency library `requests` gracefully. - Explain `extract_files` parameter in docs and use a sane default in BOTS file. - `intelmq.bots.collectors.mail.collector_mail_url`: - Handle HTTP status codes != 2xx the same as HTTP timeouts: No exception, but graceful handling. - Handle HTTP errors (bad status code and timeouts) with `error_procedure` == 'pass' but marking the mail as read and logging the error. - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.collectors.http.collector_http_stream`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.collectors.microsoft.collector_interflow`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.collectors.rt.collector_rt`: - Handle not installed dependency library `requests` gracefully. - added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (#1096). - Correctly check the version of the shodan library, it resulted in wrong comparisons with two digit numbers. - `intelmq.bots.collectors.microsoft.collector_interflow`: - Add check if Cache's TTL is big enough compared to `not_older_than` and throw an error otherwise. #### Parsers - `intelmq.bots.parsers.misp`: Fix Object attribute (#1318). - `intelmq.bots.parsers.cymru.parser_cap_program`: - Add support for new format (extra data about botnet of 'bots'). - Handle AS number 0. - `intelmq.bots.parsers.shadowserver`: - Spam URL reports: remove `src_naics`, `src_sic` columns. - fix parsing of 'spam' events in ShadowServer's 'Botnet Drone Hadoop' Report (#1271). - Add support in parser to ignore some columns in config file by using `False` as intelmq key. - Add support for the `Outdated-DNSSEC-Key` and `Outdated-DNSSEC-Key-IPv6` feeds. - Add support for the `Accessible-Rsync` feed. - Document support for the `Open-LDAP-TCP` feed. - Add support for `Accessible-HTTP` and `Open-DB2-Discovery-Service` (#1349). - Add support for `Accessible-AFP` (#1351). - Add support for `Darknet` (#1353). - `intelmq.bots.parsers.generic.parser_csv`: If the `skip_header` parameter was set to `True`, the header was not part of the `raw` field as returned by the `recover_line` method. The header is now saved and handled correctly by the fixed recovery method. - `intelmq.bots.parsers.cleanmx.parser`: Use field `first` instead of `firsttime` for `time.source` (#1329, #1348). - `intelmq.bots.parsers.twitter.parser`: Support for `url-normalize` >= 1.4.1 and recommend it. Added new optional parameter `default_scheme`, passed to `url-normalize` (#1356). #### Experts - `intelmq.bots.experts.national_cert_contact_certat.expert`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.experts.ripencc_abuse_contact.expert`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.experts.sieve.expert`: - check method: Add missing of the harmonization for the check, caused an error for every check. - Add text and more context to error messages. - README: Fix 'modify' to 'update' (#1340). - Handle empty rules file (#1343). - `intelmq.bots.experts.idea.expert`: Add mappings for new harmonization `classification.type` values, see above. #### Outputs - `intelmq.bots.outputs.redis`: - Fix sending password to redis server. - Fix for redis-py >= 3.0.0: Convert Event to string explicitly (#1354). - Use `Redis` class instead of deprecated `StrictRedis` for redis-py >= 3.0.0 (#1355). - `intelmq.bots.outputs.mongodb`: - New parameter `replacement_char` (default: `'_'`) for non-hierarchical output as dots in key names are not allowed (#1324, #1322). - Save value of fields `time.observation` and `time.source` as native datetime object, not as string (#1322). - `intelmq.bots.outputs.restapi.output`: - Handle not installed dependency library `requests` gracefully. ### Documentation - FAQ - Explanation and solution on orphaned queues. - Section on how and why to remove `raw` data. - Add or fix the tables of contents for all documentation files. - Feeds: - Fix Autoshun Feed URL (#1325). - Add parameters `name` and `provider` to `intelmq/etc/feeds.yaml`, `docs/Feeds.md` and `intelmq/bots/BOTS` (#1321). - Add SECURITY.md file. ### Packaging - Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320). ### Tests - `intelmq.tests.lib.test_bot`: Skip `test_logging_level_other` on python 3.7 because of unclear behavior related to copies of loggers (#1269). - `intelmq.tests.bots.collectors.rt.test_collector`: Remove test because the REST interface of the instance has been closed (see also https://github.com/CZ-NIC/python-rt/issues/28). ### Tools - `intelmqctl check`: Shows more detailed information on orphaned queues. - `intelmqctl`: - Correctly determine the status of bots started with `intelmqctl run`. - Fix output of errors during bot status determination, making it compatible to IntelMQ Manager. - `check` subcommand: Show bot ID for messages also in JSON output. - `run [bot-id] process -m [message]` works also with bots without a configured source pipeline (#1307). ### Contrib - elasticsearch/elasticmapper: Add tlp field (#1308). - `feeds-config-generator/intelmq_gen_feeds_conf`: - Add parameters to write resulting configuration directly to files (#1321). - Handle collector's `feed.name` and `feed.provider` (#1314). ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). - Tests: capture logging with context manager (#1342). - stomp collector bot constantly uses 100% of CPU (#1364). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq-manager release 1.1.0
by Sebastian Wagner 12 Nov '18

12 Nov '18

Dear community, The 1.1.0 contains lots of new features and enhancements. It has better error handling, display of queue sizes and bot status, support for named queues, management of groups, debugging functionalities and much more. It requires intelmq >= 1.1.0. I'll update the deb/rpm packages tomorrow. Installation instructions: https://github.com/certtools/intelmq-manager/blob/1.1.0/docs/INSTALL.md Full changelog: - Add generic MIT license text to make `reuse` license checker happy (#140). ### Backend * Using LESS for writing CSS (so CSS is now readonly) * Fix on several places: intervals won't flood server if there is a lag, they'll wait till previous request has finished. Auto-delay/boost depending on server lag. ### Frontend * Error reporting in a less-disturbing text box rather than an tab-wide javascript alert. Click to enlarge, double click to hide. * Much more detailed Error reporting. What was earlier just mere 'SyntaxError: JSON.parse ...' has now included 'Failed to execute intelmqctl:~/.local/bin/intelmqctl --type json list queues' - see? That's a full command that'll help the developer so much in debugging. (Long error reports got automatically truncated to 200 characters.) * If an error appears again and again (redis down?) it blinks instead of spamming you, updates its time and show number of times it has been thrown. * Current tab marked in menu ### Pages #### Configuration * New menu item linking selected bot with the monitor (just an icon to preserve space on precious menu bar). Supports ctrl+click to new tab on the whole button area. * New menu item to duplicate current bot, its queues included (and focus the new one); the new bot is selected and focused so that the user finds it on a big plan. It is inserted instead of the add button when a bot is selected. * You may now see live data, each bot has its queue length in the label * Button labels (Add bot, Add queue instead of Add node, Add edge) now makes more sense and won't shock a newcomer * Clearly visible bot statuses; if bot is not running, its colour is dimmed. - Manager now supports named queue paths. They are displayed along the edges and editable. - Double click edits the bot or the queue - Code cleanup: New modal window for performing multiple dialog actions. Intercepting visjs internals without using external Hammer library. - Enter key submits the edit bot dialog. - Clicking on the side-menu bot name cycles amongst bot instances in the canvas. - Got rid of the dumb "#load" and "#new" parameter → "#new" converted into a standard URL parameter "configuration=new". - Button to turn on/off physics - Detection for bots not configured in runtime but in pipeline configuration (#166). - Runtime parameters with the names 'group', 'name' or 'module' were not editable (they are read-only for the 'generic' section) (#172). - Message counter for removed queues is correctly reset (#158). #### Management * Control buttons (start/stop...) now are not only part of the management tab but are common even for configuration tab and inspection panel in monitor tab. They need less configuration (thanks to $(document).on global configuration feature) and works better with less code used (single method instead of four of them). There is also a new red state 'error' for non-standard problems or invalid output from intelmqctl. * Clicking bot name support ctrl+click to new tab * Table doesn't redraw all the time (so that a highlighting of a browser search is preserved while clicking on a button) * It is now possible to start/stop a group of bots easily. Additional panels are linked, so that if you eg. start all the experts, but let other bots stopped, Experts panel shows that all bots are running and Botnet panel shows that only some bots are running. #### Monitor * Fixed bot crawling in Monitor section (F5 will take you on the same page, not on the main monitoring page, permanent links, history browsing) * Prefer updating the destination queues overview table over redrawing it all the time (so that you can select some text) * Reloading a page sometimes caused a disturbing 'Error loading' alert in Firefox if an AJAX request had not yet been completed * Fixed bug: when clicking on bot destination queue in bot detail, we won't end in an undefined bot * Inspect panel allows you to communicate via debug functions of `intelmqctl` (see and inject messages, process single message and see output). Ctrl+Enter hits the Process button (hint included in textarea placeholder). Checkbox 'Inject message from above' marked on first text ever written to message textarea. * The new parameters panel that displays the bot's parameters on the bot's monitor page. For parameters which are filenames, the content of these files will be shown as text if they reside in the allowed paths (set in config.php). * When needed, Path column appears, resizing the Queues panel accordingly. * Connected with the configuration panel → a link appears next to the bot title that takes you directly to the bot ### Known issues * Missing CSRF protection (#111). * Graph jumps around on "Add edge" bug component (#148). * wrong error message for new bots with existing ID (#152). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users