- IntelMQ-users

IntelMQ 1.1.2 release
by Sebastian Wagner 25 Mar '19

25 Mar '19

Dear community, This is presumably the last bugfix release before the next feature release. That is expected to be the version 2.0, not 1.2 in the next month. The release will hit the deb/rpm repositories shortly. Thanks to all contributors! Installation instructions: https://github.com/certtools/intelmq/blob/1.1.2/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.2/docs/UPGRADING.md ### Configuration #### Feodotracker * The URL of the "Feodo Tracker IPs" feed has changed. The new one is `https://feodotracker.abuse.ch/downloads/ipblocklist.csv`. If you are using this feed, adapt your configuration accordingly. The parser has been updated to support the new format. * The feed "Feodo Tracker Domains" has been discontinued. The full changelog: ### Core - `intelmq.lib.bot`: - `Bot.__handle_sighup`: Handle exceptions in `shutdown` method of bots. ### Harmonization - FQDN: Disallow `:` in FQDN values to prevent values like '10.0.0.1:8080' (#1235). ### Bots #### Collectors - `intelmq.bots.collectors.stomp.collector` - Fix name of shutdown method, was ineffective in the past. - Ignore `NotConnectedException` errors on disconnect during shutdown. - `intelmq.bots.collectors.mail.collector_mail_url`: Decode body if it is bytes (#1367). - `intelmq.bots.collectors.tcp.collector`: Timeout added. More stable version. #### Parsers - `intelmq.bots.parsers.shadowserver`: - Add support for the `Amplification-DDoS-Victim`, `HTTP-Scanners`, `ICS-Scanners` and `Accessible-Ubiquiti-Discovery-Service` feeds (#1368, #1383) - `intelmq.bots.parsers.microsoft.parser_ctip`: - Workaround for mis-formatted data in `networkdestinationipv4` field (since 2019-03-14). - Ignore "hostname" ("destination.fqdn") if it contains invalid data. - `intelmq.bots.parsers.shodan.parser`: - In `minimal_mode`: - Fix the parsing, previously only `source.geolocation.cc` and `extra.shodan` was correctly filled with information. - Add a `classification.type` = 'other' to all events. - Added tests for this mode. - Normal mode: - Fix the parsing of `timestamp` to `time.source in the normal mode, previously no timezone information has been added and thus every event raised an exception. - ISAKMP: Ignore `isakmp.aggressive`, as the content is same as `isakmp` or less. - `intelmq.bots.parsers.abusech.parser_ip`: Re-structure the bot and support new format of the changed "Feodo Tracker Domains" feed. - `intelmq.bots.parsers.n6.parser`: - Add parsing for fields "confidence", "expires" and "source". - Add support for type "bl-other" (category "other"). #### Experts - `intelmq.bots.experts.sieve.expert`: Fix key definition to allow field names with numbers (`malware.hash.md5`/`sha1`, #1371). #### Outputs - `intelmq.bots.outputs.tcp.output`: Timeout added. When no separator used, awaits that every message is acknowledged by a simple "Ok" string to ensure more stability. ### Documentation - Install: Update operating system versions - Sieve Expert: Fix `elsif` -> `elif`. - Rephrase the description of `time.*` fields. - Feeds: New URL and format of the "Feodo Tracker IPs" feed. "Feodo Tracker Domains" has been discontinued. ### Packaging ### Tests - Add missing `__init__.py` files in 4 bot's test directories. Previously these tests have never been executed. - `intelmq.lib.test`: Allow bot test class names with an arbitrary postfix separated by an underscore. E.g. `TestShodanParserBot_minimal`. ### Tools - intelmqctl: - status: Show commandline differences if a program with the expected PID could be found, but they do not match (previous output was `None`). - Use logging level from defauls configuration if possible, otherwise intelmq's internal default. Previously, DEBUG was used unconditionally. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). - stomp collector bot constantly uses 100% of CPU (#1364). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 1.1.1 released
by Sebastian Wagner 15 Jan '19

15 Jan '19

Dear community, Today I released the first maintenance version of the 1.1.x series. Thanks for all the contributions! Installation instructions: https://github.com/certtools/intelmq/blob/1.1.1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.1/docs/UPGRADING.md The full change log: ### Core - `lib/harmonization.py`: Change `parse_utc_isoformat` of `DateTime` class from private to public (related to #1322). - `lib/utils.py`: Add new function `object_pair_hook_bots`. - `lib.bot.py`: - `ParserBot`'s method `recover_line_csv` now also handles given `tempdata`. - `Bot.acknowledge_message()` deletes `__current_message` to free the memory, saves memory in idling parsers with big reports. - `start()`: Warn once per run if `error_dump_message` is set to false. - `Bot.start()`, `ParserBot.process()`: If errors happen on bots without destination pipeline, the `on_error` path has been queried and lead to an exception being raised. - `start()`: If `error_procedure` is pass and on pipeline errors, the bot retries forever (#1333). - `lib/message.py`: - Fix add('extra', ..., overwrite=True): old extra fields have not been deleted previously (#1335). - Do not ignore empty or ignored (as defined in `_IGNORED_VALUES`) values of `extra.*` fields for backwards compatibility (#1335). - `lib/pipeline.py` (`Redis.receive`): Wait in 1s steps if redis is busy loading its snapshot from disk (#1334). ### Default configuration - Set `error_dump_message` to true by default in `defaults.conf`. - Fixed typo in `defaults.conf`: `proccess_manager` -> `process_manager` ### Development - `bin/rewrite_config_files.py`: Fix ordering of BOTS file (#1327). ### Harmonization Update to 2018-09-26 version. New values are per taxonomy: - Taxonomy 'intrusions': - "application-compromise" - "burglary" - "privileged-account-compromise" - "unprivileged-account-compromise" - Taxonomy 'fraud': - "copyright" - "masquerade" - "unauthorized-use-of-resources" - Taxonomy 'information content security': - "data-loss" - Taxonomy 'vulnerable': - "ddos-amplifier" - "information-disclosure" - "potentially-unwanted-accessible" - "vulnerable-system" - "weak-crypto" - Taxonomy 'availability': - "dos" - "outage" - "sabotage" - Taxonomy 'abusive-content': - "harmful-speech" - "violence" - Taxonomy 'malicious code': - "malware-distribution" - Taxonomy 'information-gathering': - "social-engineering" - "sniffing" - Taxonomy 'information content security': - "Unauthorised-information-access" - "Unauthorised-information-modification" ### Bots #### Collectors - `intelmq.bots.collectors.http.collector_http`: - Fix parameter name `extract_files` in BOTS (#1331). - Fix handling of `extract_files` parameter if the value is an empty string. - Handle not installed dependency library `requests` gracefully. - Explain `extract_files` parameter in docs and use a sane default in BOTS file. - `intelmq.bots.collectors.mail.collector_mail_url`: - Handle HTTP status codes != 2xx the same as HTTP timeouts: No exception, but graceful handling. - Handle HTTP errors (bad status code and timeouts) with `error_procedure` == 'pass' but marking the mail as read and logging the error. - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.collectors.http.collector_http_stream`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.collectors.microsoft.collector_interflow`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.collectors.rt.collector_rt`: - Handle not installed dependency library `requests` gracefully. - added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (#1096). - Correctly check the version of the shodan library, it resulted in wrong comparisons with two digit numbers. - `intelmq.bots.collectors.microsoft.collector_interflow`: - Add check if Cache's TTL is big enough compared to `not_older_than` and throw an error otherwise. #### Parsers - `intelmq.bots.parsers.misp`: Fix Object attribute (#1318). - `intelmq.bots.parsers.cymru.parser_cap_program`: - Add support for new format (extra data about botnet of 'bots'). - Handle AS number 0. - `intelmq.bots.parsers.shadowserver`: - Spam URL reports: remove `src_naics`, `src_sic` columns. - fix parsing of 'spam' events in ShadowServer's 'Botnet Drone Hadoop' Report (#1271). - Add support in parser to ignore some columns in config file by using `False` as intelmq key. - Add support for the `Outdated-DNSSEC-Key` and `Outdated-DNSSEC-Key-IPv6` feeds. - Add support for the `Accessible-Rsync` feed. - Document support for the `Open-LDAP-TCP` feed. - Add support for `Accessible-HTTP` and `Open-DB2-Discovery-Service` (#1349). - Add support for `Accessible-AFP` (#1351). - Add support for `Darknet` (#1353). - `intelmq.bots.parsers.generic.parser_csv`: If the `skip_header` parameter was set to `True`, the header was not part of the `raw` field as returned by the `recover_line` method. The header is now saved and handled correctly by the fixed recovery method. - `intelmq.bots.parsers.cleanmx.parser`: Use field `first` instead of `firsttime` for `time.source` (#1329, #1348). - `intelmq.bots.parsers.twitter.parser`: Support for `url-normalize` >= 1.4.1 and recommend it. Added new optional parameter `default_scheme`, passed to `url-normalize` (#1356). #### Experts - `intelmq.bots.experts.national_cert_contact_certat.expert`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.experts.ripencc_abuse_contact.expert`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.experts.sieve.expert`: - check method: Add missing of the harmonization for the check, caused an error for every check. - Add text and more context to error messages. - README: Fix 'modify' to 'update' (#1340). - Handle empty rules file (#1343). - `intelmq.bots.experts.idea.expert`: Add mappings for new harmonization `classification.type` values, see above. #### Outputs - `intelmq.bots.outputs.redis`: - Fix sending password to redis server. - Fix for redis-py >= 3.0.0: Convert Event to string explicitly (#1354). - Use `Redis` class instead of deprecated `StrictRedis` for redis-py >= 3.0.0 (#1355). - `intelmq.bots.outputs.mongodb`: - New parameter `replacement_char` (default: `'_'`) for non-hierarchical output as dots in key names are not allowed (#1324, #1322). - Save value of fields `time.observation` and `time.source` as native datetime object, not as string (#1322). - `intelmq.bots.outputs.restapi.output`: - Handle not installed dependency library `requests` gracefully. ### Documentation - FAQ - Explanation and solution on orphaned queues. - Section on how and why to remove `raw` data. - Add or fix the tables of contents for all documentation files. - Feeds: - Fix Autoshun Feed URL (#1325). - Add parameters `name` and `provider` to `intelmq/etc/feeds.yaml`, `docs/Feeds.md` and `intelmq/bots/BOTS` (#1321). - Add SECURITY.md file. ### Packaging - Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320). ### Tests - `intelmq.tests.lib.test_bot`: Skip `test_logging_level_other` on python 3.7 because of unclear behavior related to copies of loggers (#1269). - `intelmq.tests.bots.collectors.rt.test_collector`: Remove test because the REST interface of the instance has been closed (see also https://github.com/CZ-NIC/python-rt/issues/28). ### Tools - `intelmqctl check`: Shows more detailed information on orphaned queues. - `intelmqctl`: - Correctly determine the status of bots started with `intelmqctl run`. - Fix output of errors during bot status determination, making it compatible to IntelMQ Manager. - `check` subcommand: Show bot ID for messages also in JSON output. - `run [bot-id] process -m [message]` works also with bots without a configured source pipeline (#1307). ### Contrib - elasticsearch/elasticmapper: Add tlp field (#1308). - `feeds-config-generator/intelmq_gen_feeds_conf`: - Add parameters to write resulting configuration directly to files (#1321). - Handle collector's `feed.name` and `feed.provider` (#1314). ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). - Tests: capture logging with context manager (#1342). - stomp collector bot constantly uses 100% of CPU (#1364). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq-manager release 1.1.0
by Sebastian Wagner 12 Nov '18

12 Nov '18

Dear community, The 1.1.0 contains lots of new features and enhancements. It has better error handling, display of queue sizes and bot status, support for named queues, management of groups, debugging functionalities and much more. It requires intelmq >= 1.1.0. I'll update the deb/rpm packages tomorrow. Installation instructions: https://github.com/certtools/intelmq-manager/blob/1.1.0/docs/INSTALL.md Full changelog: - Add generic MIT license text to make `reuse` license checker happy (#140). ### Backend * Using LESS for writing CSS (so CSS is now readonly) * Fix on several places: intervals won't flood server if there is a lag, they'll wait till previous request has finished. Auto-delay/boost depending on server lag. ### Frontend * Error reporting in a less-disturbing text box rather than an tab-wide javascript alert. Click to enlarge, double click to hide. * Much more detailed Error reporting. What was earlier just mere 'SyntaxError: JSON.parse ...' has now included 'Failed to execute intelmqctl:~/.local/bin/intelmqctl --type json list queues' - see? That's a full command that'll help the developer so much in debugging. (Long error reports got automatically truncated to 200 characters.) * If an error appears again and again (redis down?) it blinks instead of spamming you, updates its time and show number of times it has been thrown. * Current tab marked in menu ### Pages #### Configuration * New menu item linking selected bot with the monitor (just an icon to preserve space on precious menu bar). Supports ctrl+click to new tab on the whole button area. * New menu item to duplicate current bot, its queues included (and focus the new one); the new bot is selected and focused so that the user finds it on a big plan. It is inserted instead of the add button when a bot is selected. * You may now see live data, each bot has its queue length in the label * Button labels (Add bot, Add queue instead of Add node, Add edge) now makes more sense and won't shock a newcomer * Clearly visible bot statuses; if bot is not running, its colour is dimmed. - Manager now supports named queue paths. They are displayed along the edges and editable. - Double click edits the bot or the queue - Code cleanup: New modal window for performing multiple dialog actions. Intercepting visjs internals without using external Hammer library. - Enter key submits the edit bot dialog. - Clicking on the side-menu bot name cycles amongst bot instances in the canvas. - Got rid of the dumb "#load" and "#new" parameter → "#new" converted into a standard URL parameter "configuration=new". - Button to turn on/off physics - Detection for bots not configured in runtime but in pipeline configuration (#166). - Runtime parameters with the names 'group', 'name' or 'module' were not editable (they are read-only for the 'generic' section) (#172). - Message counter for removed queues is correctly reset (#158). #### Management * Control buttons (start/stop...) now are not only part of the management tab but are common even for configuration tab and inspection panel in monitor tab. They need less configuration (thanks to $(document).on global configuration feature) and works better with less code used (single method instead of four of them). There is also a new red state 'error' for non-standard problems or invalid output from intelmqctl. * Clicking bot name support ctrl+click to new tab * Table doesn't redraw all the time (so that a highlighting of a browser search is preserved while clicking on a button) * It is now possible to start/stop a group of bots easily. Additional panels are linked, so that if you eg. start all the experts, but let other bots stopped, Experts panel shows that all bots are running and Botnet panel shows that only some bots are running. #### Monitor * Fixed bot crawling in Monitor section (F5 will take you on the same page, not on the main monitoring page, permanent links, history browsing) * Prefer updating the destination queues overview table over redrawing it all the time (so that you can select some text) * Reloading a page sometimes caused a disturbing 'Error loading' alert in Firefox if an AJAX request had not yet been completed * Fixed bug: when clicking on bot destination queue in bot detail, we won't end in an undefined bot * Inspect panel allows you to communicate via debug functions of `intelmqctl` (see and inject messages, process single message and see output). Ctrl+Enter hits the Process button (hint included in textarea placeholder). Checkbox 'Inject message from above' marked on first text ever written to message textarea. * The new parameters panel that displays the bot's parameters on the bot's monitor page. For parameters which are filenames, the content of these files will be shown as text if they reside in the allowed paths (set in config.php). * When needed, Path column appears, resizing the Queues panel accordingly. * Connected with the configuration panel → a link appears next to the bot title that takes you directly to the bot ### Known issues * Missing CSRF protection (#111). * Graph jumps around on "Add edge" bug component (#148). * wrong error message for new bots with existing ID (#152). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Bots stopping
by Vaclav Bruzek 23 Oct '18

23 Oct '18

Hi, since upgrading to version 1.1.0 it became quite a big problem the stability of bots. Often it happens that bot encounters an exception and logs that the bot is stopped or encounters an exception and logs nothing but status check reports that the bot is not running. I'm using the 'error_procedure' parameter set to 'pass' (with error_max_retries and error_retry_delay set to 0) and I've always thought that this is a sort of 'run forever' parameter that even when exception occurs the bot will keep on doing its job. I'm using intelmq in Docker environment with ubuntu 18.04 as base. Sincerely, Václav Brůžek

3 10

Custom bots from external code base
by Antoine Neuenschwander 16 Oct '18

16 Oct '18

Hello list, We are currently developing custom bots that are very specific to one of our clients' environment. For the sake of clean code management, we intend to keep the custom bots in a separate repository and generate separate pip/rpm packages. So far I think the only obstacle in doing so is that the bots are not registered in the BOTS file, therefore they will not be recognized by intelmqctl. We could append the bots' description in the BOTS file manually, but we'd rather have an automatic way of doing this, when installing the custom bot rpm package. Has anyone already dealt with the same use-case? Any tips? Thanks and best regards, Antoine -- SWITCH -------------------------- Antoine Neuenschwander, SWITCH-CERT Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland phone +41 44 268 15 15, direct +41 44 268 15 29 antoine.neuenschwander(a)switch.ch, http://www.switch.ch Working for a better digital world

3 3

intelmq-manager release 1.0.2
by Sebastian Wagner 25 Sep '18

25 Sep '18

Dear community, Just released the second and most probably last bugfix release for the intelmq-manager 1.0 series. Installation instructions: https://github.com/certtools/intelmq-manager/blob/1.0.2/docs/INSTALL.md The built deb/rpm packages are already available. The full changelog: # Backend - Emit a more specific error message for cases like #160. - The timeout for calls to `intelmqctl` has been raised to 20s (#164). # Pages ## Configuration - Underscore is now allowed for new parameter names (#153). ## Monitor * Fix link to monitor page (#157). # Documentation - Add a FAQ and add a section about the docker issue (#160). - Add instructions for Debian 9, Ubuntu 18.04, and openSUSE Leap 42.3, 15.0 (#168). # Packaging - Include a positions file matching the default configuration of intelmq (#171). # Known issues * Missing CSRF protection (#111). * Missing copyright notices (#140). * Graph jumps around on "Add edge" bug component (#148). * new runtime parameters with _ not possible (#153). * wrong error message for new bots with existing ID (#152). * Queue size for deleted queue remains displayed (#158) best regards, Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Updating datastores
by joanna＠scate.tech 18 Sep '18

18 Sep '18

Hi, Is there a way of updating outputs such as databases when IOCs expire? Don't want to spend time re-inventing the wheel. Thanks.

4 5

IntelMQ 1.1.0 released
by Sebastian Wagner 05 Sep '18

05 Sep '18

Dear community, I just released the feature release 1.1.0 for the 1.x series the first stable release after the initial 1.0.0. This version is already in use in several setups without known issues and introduces many new features and bots. Installation instructions: https://github.com/certtools/intelmq/blob/1.1.0/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.0/docs/UPGRADING.md The deb/rpm packages will be available in some hours in the stable repository (see linked instructions). The full changelog is very long: - Support for Python 3.3 has been dropped in IntelMQ and some dependencies of it. Python 3.3 reached its end of life and Python 3.4 or newer is a hard requirement now. - The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml A tool to convert from yaml to md has been added. ### Tools - `intelmq_gen_feeds_docs` addded to bin directory, allows generating the Feeds.md documentation file from feeds.yaml - `intelmq_gen_docs` merges both `intelmq_gen_feeds_docs` and `intelmq_gen_harm_docs` in one file and automatically updates the documentation files. #### intelmqctl - `intelmqctl start` prints the bot's last error messages if the bot failed to start (#1021). - `intelmqctl start` message "is running" is printed every time. (Until now, it wasn't said when a bot was just starting.) - `intelmqctl start/stop/restart/reload/status` now has a "--group" flag which allows you to specify the group of the bots that should be influenced by the command. - `intelmqctl check` checks for defaults.conf completeness if the shipped file from the package can be found. - `intelmqctl check` shows errors for non-importable bots. - `intelmqctl list bots -q` only prints the IDs of enabled bots. - `intelmqctl list queues-and-status` prints both queues and bots statuses (so that it can be used in eg. intelmq-manager). - `intelmqctl run` parameter for showing a sent message. - `intelmqctl run` if message is sent to a non-default path, it is printed out. - `intelmqctl restart` bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226). - `intelmqctl check`: New parameter `--no-connections` to prevent the command from making connections e.g. to the redis pipeline.s - `intelmqctl list queues`: don't display named paths amongst standard queues. - The process status test failed if the PATH did not include the bot executables and the `which` command failed. Then the proccess's command line could not be compared correctly. The fix warns of this and adds a new status 'unknown' (#1297). ### Contrib - tool `feeds-config-generator` to automatically generate the collector and parser runtime and pipeline configurations. - `malware_name_mapping`: Download and convert tool for malware family name mapping has been added. - Added a systemd script which creates systemd units for bots (#953). - `contrib/cron-jobs/update-asn-data`, `contrib/cron-jobs/update-geoip-data`, `contrib/cron-jobs/update-tor-nodes`: Errors produce proper output. ### Core - lib/bot - use SIGTERM instead of SIGINT to stop bots (#981). - Bots can specify a static method `check(parameters)` which can perform individual checks specific to the bot. These functions will be called by `intelmqctl check` if the bot is configured with the given parameters - top level bot parameters (description, group, module, name) are exposed as members of the class. - The parameter `feed` for collectors is deprecated for 2.0 and has been replaced by the more consistent `name` (#1144). - bug: allow path parameter for CollectorBot class. - Handle errors better when the logger could not be initialized. - `ParserBot`: - For the csv parsing methods, `ParserBot.csv_params` is now used for all these methods. - `ParserBot.parse_csv_dict` now saves the field names in `ParserBot.csv_fieldnames`. - `ParserBot.parse_csv_dict` now saves the raw current line in `ParserBot.current_line`. - `ParserBot.recover_line_csv_dict` now uses the raw current line. - lib/message: - Subitems in fields of type `JSONDict` (see below) can be accessed directly. E.g. you can do: event['extra.foo'] = 'bar' event['extra.foo'] # gives 'bar' It is still possible to set and get the field as whole, however this may be removed or changed in the future: event['extra'] = '{"foo": "bar"}' event['extra'] # gives '{"foo": "bar"}' "Old" bots and configurations compatible with 1.0.x do still work. Also, the extra field is now properly exploded when exporting events, analogous to all other fields. The `in` operator works now for both - the old and the new - behavior. - `Message.add`: The parameter `overwrite` accepts now three different values: `True`, `False` and `None` (new). True: An existing value will be overwritten False: An existing value will not be overwritten (previously an exception has been raised when the value was given). None (default): If the value exists an `KeyExists` exception is thrown (previously the same as False). This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function. - The message class has now the possibility to return a default value for non-exisiting fields, see `Message.set_default_value`. - Message.get behaves the same like `Message.__getitem__` (#1305). - Add `RewindableFileHandle` to utils making handling of CSV files more easy (optionally) - lib/pipeline: * you may now define more than one destination queues path the bot should pass the message to, see [Pipelines](https://github.com/certtools/intelmq/blob/develop/docs/User-Guid… (#1088, #1190). * the special path `"_on_error"` can be used to pass messages to different queues in case of processing errors (#1133). - `lib/harmonization`: Accept `AS` prefix for ASN values (automatically stripped). ### Bots - Removed print statements from various bots. - Replaced various occurences of `self.logger.error()` + `self.stop()` with `raise ValueError`. #### Collectors - `bots.collectors.mail`: - New parameters; `sent_from`: filter messages by sender, `sent_to`: filter messages by recipient - More debug logs - `bots.collectors.n6.collector_stomp`: renamed to `bots.collectors.stomp.collector` (#716) - bots.collectors.rt: - New parameter `search_requestor` to search for field Requestor. - Empty strings and `null` as value for search parameters are ignored. - Empty parameters `attachment_regex` and `url_regex` handled. - `bots.collectors.http.collector_http`: Ability to optionally use the current time in parameter `http_url`, added parameter `http_url_formatting`. - `bots.collectors.stomp.collector`: Heartbeat timeout is now logged with log level info instead of warning. - added `intelmq.bots.collectors.twitter.collector_twitter` - added `intelmq.bots.collectors.tcp.collector` that can be bound to another IntelMQ instance by a TCP output - `bots.collectors.microsoft.collector_interflow`: added for MS interflow API - Automatic ungzipping for .gz files. - added `intelmq.bots.collectors.calidog.collector_certstream` for collecting certstream data (#1120). - added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (#1096). - Add proxy support. - Fix handling of parameter `countries`. #### Parsers - `bots.parsers.shadowserver`: - changed feednames . Please refer to it's README for the exact changes. - If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration. Previously errors like these were only logged and ignored otherwise. - add support for the feeds - `Accessible-Hadoop` (#1231) - `Accessible ADB` (#1285) - Remove deprecated parameter `override`, use `overwrite` instead (#1071). - The `raw` values now are exactly the input with quotes unchanged, the ParserBot methods are now used directly (#1011). - The Generic CSV Parser `bots.parsers.generic.parser_csv`: - It is possible to filter the data before processing them using the new parameters `filter_type` and `filter_text`. - It is possible to specify multiple columns using `|` character in parameter `columns`. - The parameter `time_format` now supports `'epoch_millis'` for seconds since the Epoch, milliseconds are supported but not used. - renamed `bots.parsers.cymru_full_bogons.parser` to `bots.parsers.cymru.parser_full_bogons`, compatibility shim will be removed in version 2.0 - added `bots.parsers.cymru.parser_cap_program` - added `intelmq.bots.parsers.zoneh.parser` for ZoneH feeds - added `intelmq.bots.parsers.sucuri.parser` - added `intelmq.bots.parsers.malwareurl.parser` - added `intelmq.bots.parsers.threatminer.parser` - added `intelmq.bots.parsers.webinspektor.parser` - added `intelmq.bots.parsers.twitter.parser` - added `intelmq.bots.parsers.microsoft.parser_ctip` * ignore the invalid IP '0.0.0.0' for the destination * fix the raw/dumped messages, did not contain the paling list previously. * use the new harmonization field `tlp` instead of `extra.tlp`. - `bots.parsers.alienvault.parser_otx`: Save TLP data in the new harmonization field `tlp`. - added `intelmq.bots.parsers.openphish.parser_commercial` - added `intelmq.bots.parsers.microsoft.parser_bingmurls` - added `intelmq.bots.parsers.calidog.parser_certstream` for parsing certstream data (#1120). - added `intelmq.bots.parsers.shodan.parser` for parsing shodan data (#1096). - change the classification type from 'botnet drone' to infected system' in various parses. - `intelmq.bots.parsers.spamhaus.parser_cert`: Added support for all known bot types. #### Experts - Added sieve expert for filtering and modifying events (#1083) * capable of distributing the event to appropriate named queues - `bots.experts.modify` * default rulesets: all malware name mappings have been migrated to the [Malware Name Mapping repository](https://github.com/certtools/malware_name_mapping) ruleset. See the new added contrib tool for download and conversion. * new parameter `case_sensitive` (default: True) - Added wait expert for sleeping - Added domain suffix expert to extract the TLD/Suffix from a domain name. - `bots.experts.maxmind_geoip`: New (optional) parameter `overwrite`, by default false. The current default was to overwrite! - `intelmq.bots.experts.ripencc_abuse_contact`: * Extend deprecated parameter compatibility `query_ripe_stat` until 2.0 because of a logic bug in the compatibility code, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (#1071, #1291). * Handle HTTP status code 404 for DB AS queries. * Add caching capability. - `intelmq/bots/experts/asn_lookup/update-asn-data`: Errors produce proper output on stdout/stderr. - `intelmq/bots/experts/maxmind_geoip/update-geoip-data`: Errors produce proper output on stdout/stderr. - `intelmq/bots/experts/tor_nodes/update-tor-nodes`: Errors produce proper output on stdout/stderr. #### Outputs - `bots.outputs.file`: - String formatting can be used for file names with new parameter `format_filename`. - New parameter `single_key` to only save one field. - New parameter `encoding_errors_mode` with default value `'strict'` to handle encoding errors for the files written. ### Harmonization - Renamed `JSON` to `JSONDict` and added a new type `JSON`. `JSONDict` saves data internally as JSON, but acts like a dictionary. `JSON` accepts any valid JSON. - fixed regex for `protocol.transport` it previously allowed more values than it should have. - New ASN type. Like integer but checks the range. - added `destination.urlpath` and `source.urlpath` to harmonization. - New field `tlp` for tlp level specification. - New TLP type. Allows all four tlp levels, removes 'TLP:' prefix and converts to upper case. - Added new `classification.type` 'vulnerable client' - Added `(destination|source).domain_suffix` to hold the TLD/domain suffix. - New allowed value for `classification.type`: `infected system` for taxonomy `malicious code` (#1197). ### Requirements - Requests is no longer listed as dependency of the core. For depending bots the requirement is noted in their REQUIREMENTS.txt file. ### Documentation - Use Markdown for README again, as pypi now supports it. - Developers Guide: Add instructions for pre-release testing. ### Packaging - Add logcheck configuration to the packages. - Fix packaging of bash completion script. ### Tests - Travis now correctly stops if a requirement could not be installed (#1257). - New tests for validating `etc/feeds.yaml` and `bots/BOTS` using cerberus and schemes are added (#1166). - New test for checking if `docs/Feeds.md` is up to date with `etc/feeds.yaml`. ### Known bugs - contrib: feeds-config-generator does not add feed name as parameter (#1314). - bot debugger requires configured source pipeline (#1307). - shadowserver parser: drone feed has spam events (#1271). - debug log level on python 3.7 not applied (#1269). - `bots.experts.sieve` does not support textX (#1246). - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 1.0.6 released
by Sebastian Wagner 05 Sep '18

05 Sep '18

Dear community, at the end of last weak I finalized the last planned maintenance release of the 1.0.x series, version 1.0.6. I do not intend to do any further maintenance releases for the 1.0 version, however I am open for discussion here. The branch maintenance-1.0 has all the commits of the 1.0.x series. Installation instructions: https://github.com/certtools/intelmq/blob/1.0.6/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.6/docs/UPGRADING.md The changelog: ### Bots #### Collectors - `bots.collectors.rt.collector_rt`: Log ticket id for downloaded reports. #### Parsers - `bots.parsers.shadowserver`: - if required fields do not exist in data, an exception is raised, so the line will be dumped and not further processed. - fix a bug in the parsing of column `cipher_suite` in ssl poodle reports (#1288). #### Experts - Reverse DNS Expert: ignore all invalid results and use first valid one (#1264). - `intelmq/bots/experts/tor_nodes/update-tor-nodes`: Use check.torproject.org as source as internet2.us is down (#1289). #### Outputs - `bots.output.amqptopic`: - The default exchange must not be declared (#1295). - Unencodable characters are prepended by backslashes by default. Otherwise Unicode characters can't be encoded and sent (#1296). - Gracefully close AMQP connection on shutdown of bot. ### Documentation - Bots: document redis cache parameters. - Installation documentation: Ubuntu needs universe repositories. ### Packaging - Dropped support for Ubuntu 17.10, it reached its End of Life as of 2018-07-19. ### Tests - Drop tests for Python 3.3 for the mode with all requirements, as some optional dependencies do not support Python 3.3 anymore. - `lib.test`: Add parameter `compare_raw` (default: `True`) to `assertMessageEqual`, to optionally skip the comparison of the raw field. - Add tests for RT collector. - Add tests for Shadowserver Parser: - SSL Poodle Reports. - Helper functions. ### Tools - `intelmqctl list` now sorts the output of bots and queues (#1262). - `intelmqctl`: Correctly handle the corner cases with collectors and outputs for getting/sending messages in the bot debugger (#1263). - `intelmqdump`: fix ordering of dumps in a file in runtime. All operations are applied to a sorted list (#1280). ### Contrib - `cron-jobs/update-tor-nodes`: Use check.torproject.org as source as internet2.us is down (#1289). Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

1.1.0 release candidate
by Sebastian Wagner 07 Aug '18

07 Aug '18

Hi, I just released a release candidate version for 1.1.0. If there are no big issues coming up in the next days, there won't be another release candidate and a final release in one or two weeks, depending on the feedback. Everybody is encouraged to give feedback and test this pre release. 1.0.x installations are compatible with the 1.1.0 code. Some things have been deprecated, have a look at warnings and `intelmqctl check`. The deb/rpm packages in the unstable repository will be available tomorrow or next week. This is the full current changelog: Installation instructions: https://github.com/certtools/intelmq/blob/1.1.0rc1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.0rc1/docs/UPGRADING.md - Support for Python 3.3 has been dropped, it reached its end of life. - The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml A tool to convert from yaml to md has been added. # Tools - `intelmq_gen_feeds_docs` add to bin directory, allows generating the Feeds.md documentation file from feeds.yaml - `intelmq_gen_docs` merges both `intelmq_gen_feeds_docs` and `intelmq_gen_harm_docs` in one file and automatically updates the documentation files. ## intelmqctl - `intelmqctl start` prints the bot's last error messages if the bot failed to start (#1021). - `intelmqctl start` message "is running" is printed every time. (Until now, it wasn't said when a bot was just starting.) - `intelmqctl start/stop/restart/reload/status` now have a "--group" flag which allows you to specify the group of the bots that should be influenced by the command. - `intelmqctl check` checks for defaults.conf completeness if the shipped file from the package can be found. - `intelmqctl check` shows errors for non-importable bots. - `intelmqctl list bots -q` only prints the IDs of enabled bots. - `intelmqctl list queues-and-status` prints both queues and bots statuses (so that it can be used in eg. intelmq-manager). - `intelmqctl run` parameter for showing a sent message. - `intelmqctl run` if message is sent to a non-default path, it is printed out. - `intelmqctl restart` bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226). - `intelmqctl check`: New parameter `--no-connections` to prevent the command from making connections e.g. to the redis pipeline.s # Contrib - tool `feeds-config-generator` to automatically generate the collector and parser runtime and pipeline configurations. - `malware_name_mapping`: Download and convert tool for malware family name mapping has been added. - Added a systemd script which creates systemd units for bots (#953). # Core - lib/bot - use SIGTERM instead of SIGINT to stop bots (#981). - Bots can specify a static method `check(parameters)` which can perform individual checks specific to the bot. These functions will be called by `intelmqctl check` if the bot is configured with the given parameters - top level bot parameters (description, group, module, name) are exposed as members of the class. - The parameter `feed` for collectors is deprecated for 2.0 and has been replaced by the more consistent `name` (#1144). - lib/message: - Subitems in fields of type `JSONDict` (see below) can be accessed directly. E.g. you can do: event['extra.foo'] = 'bar' event['extra.foo'] # gives 'bar' It is still possible to set and get the field as whole, however this may be removed or changed in the future: event['extra'] = '{"foo": "bar"}' event['extra'] # gives '{"foo": "bar"}' "Old" bots and configurations compatible with 1.0.x do still work. Also, the extra field is now properly exploded when exporting events, analogous to all other fields. - `Message.add`: The parameter `overwrite` accepts now three different values: `True`, `False` and `None` (new). True: An existing value will be overwritten False: An existing value will not be overwritten (previously an exception has been raised when the value was given). None (default): If the value exists an `KeyExists` exception is thrown (previously the same as False). This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function. - The message class has now the possibility to return a default value for non-exisiting fields, see `Message.set_default_value`. - Add `RewindableFileHandle` to utils making handling of CSV files more easy (optionally) - lib/pipeline: * you may now define more than one destination queues path the bot should pass the message to, see [Pipelines](https://github.com/certtools/intelmq/blob/develop/docs/User-Guid… (#1088, #1190). * the special path `"_on_error"` can be used to pass messages to different queues in case of processing errors (#1133). - `lib/harmonization`: Accept `AS` prefix for ASN values (automatically stripped). # Bots ## Collectors - `bots.collectors.mail`: - New parameters; `sent_from`: filter messages by sender, `sent_to`: filter messages by recipient - More debug logs - `bots.collectors.n6.collector_stomp`: renamed to `bots.collectors.stomp.collector` (#716) - bots.collectors.rt: - New parameter `search_requestor` to search for field Requestor. - Empty strings and `null` as value for search parameters are ignored. - Empty parameters `attachment_regex` and `url_regex` handled. - `bots.collectors.http.collector_http`: Ability to optionally use the current time in parameter `http_url`, added parameter `http_url_formatting`. - `bots.collectors.stomp.collector`: Heartbeat timeout is now logged with log level info instead of warning. - added `intelmq.bots.collectors.twitter.collector_twitter` - added `intelmq.bots.collectors.tcp.collector` that can be bound to another IntelMQ instance by a TCP output - `bots.collectors.microsoft.collector_interflow`: added for MS interflow API - Automatic ungzipping for .gz files. - added `intelmq.bots.collectors.calidog.collector_certstream` for collecting certstream data (#1120). - added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (#1096). ## Parsers - `bots.parsers.shadowserver`: - changed feednames . Please refer to it's README for the exact changes. - If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration. Previously errors like these were only logged and ignored otherwise. - add support for the feed `Accessible-Hadoop` - The Generic CSV Parser `bots.parsers.generic.parser_csv`: - It is possible to filter the data before processing them using the new parameters `filter_type` and `filter_text`. - It is possible to specify multiple columns using `|` character in parameter `columns`. - The parameter `time_format` now supports `'epoch_millis'` for seconds since the Epoch, milliseconds are supported but not used. - renamed `bots.parsers.cymru_full_bogons.parser` to `bots.parsers.cymru.parser_full_bogons`, compatibility shim will be removed in version 2.0 - added `bots.parsers.cymru.parser_cap_program` - added `intelmq.bots.parsers.zoneh.parser` for ZoneH feeds - added `intelmq.bots.parsers.sucuri.parser` - added `intelmq.bots.parsers.malwareurl.parser` - added `intelmq.bots.parsers.threatminer.parser` - added `intelmq.bots.parsers.webinspektor.parser` - added `intelmq.bots.parsers.twitter.parser` - added `intelmq.bots.parsers.microsoft.parser_ctip` * ignore the invalid IP '0.0.0.0' for the destination * fix the raw/dumped messages, did not contain the paling list previously. * use the new harmonization field `tlp` instead of `extra.tlp`. - `bots.parsers.alienvault.parser_otx`: Save TLP data in the new harmonization field `tlp`. - added `intelmq.bots.parsers.openphish.parser_commercial` - added `intelmq.bots.parsers.microsoft.parser_bingmurls` - added `intelmq.bots.parsers.calidog.parser_certstream` for parsing certstream data (#1120). - added `intelmq.bots.parsers.shodan.parser` for parsing shodan data (#1096). ## Experts - Added sieve expert for filtering and modifying events (#1083) * capable of distributing the event to appropriate named queues - `bots.experts.modify` * default rulesets: all malware name mappings have been migrated to the [Malware Name Mapping repository](https://github.com/certtools/malware_name_mapping) ruleset. See the new added contrib tool for download and conversion. * new parameter `case_sensitive` (default: True) - Added wait expert for sleeping - Added domain suffix expert to extract the TLD/Suffix from a domain name. - `bots.experts.maxmind_geoip`: New (optional) parameter `overwrite`, by default false. The current default was to overwrite! ## Outputs - `bots.outputs.file`: - String formatting can be used for file names. - New parameter `single_key` to only save one field. # Harmonization - Renamed `JSON` to `JSONDict` and added a new type `JSON`. `JSONDict` saves data internally as JSON, but acts like a dictionary. `JSON` accepts any valid JSON. - fixed regex for `protocol.transport` it previously allowed more values than it should have. - New ASN type. Like integer but checks the range. - added `destination.urlpath` and `source.urlpath` to harmonization. - New field `tlp` for tlp level specification. - New TLP type. Allows all four tlp levels, removes 'TLP:' prefix and converts to upper case. - Added new `classification.type` 'vulnerable client' - Added `(destination|source).domain_suffix` to hold the TLD/domain suffix. - New allowed value for `classification.type`: `infected system` for taxonomy `malicious code` (#1197). # Requirements - Requests is no longer listed as dependency of the core. For depending bots the requirement is noted in their REQUIREMENTS.txt file. # Documentation - Use Markdown for README again, as pypi now supports it. # Tests - Travis now correctly stops if a requirement could not be installed (#1257). - New tests for validating `etc/feeds.yaml` and `bots/BOTS` using cerberus and schemes are added (#1166). - New test for checking if `docs/Feeds.md` is up to date with `etc/feeds.yaml`. # Known bugs - `bots.experts.sieve` does not support textX (#1246). - performance degradation for extra fields (#1117). - Postgres output: support condensed JSONDicts (#1107). - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users