IntelMQ-users April 2019

intelmq-users@lists.cert.at

2 participants
4 discussions

Max retries issue
by Vaclav Bruzek 01 Aug '19

01 Aug '19

Hi mailinglist, I've went over intelmq logs today and that some bots keep retrying indefinitely when they encounter a failure. However each of those bots have *error_max_retries* set to 1 (and *error_retry_delay *to 60). Which in my mind should limit the bot to just try one more time and than give up and sleep (if sleep is set up) however this is not happening. Am I doing something wrong or does this functionality work differently? My version of intelmq is 1.1.1 in Docker on Ubuntu 18.04 as base. Following is an example from logs os collector collecting from phishtank: 2019-03-01 20:07:51,747 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:07:52,139 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:07:52,140 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:07:52,140 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:07:52,298 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:07:52,299 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:07:52,299 - phishtank-collector - INFO - Bot will continue in 60 seconds. 2019-03-01 20:08:52,359 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:08:52,654 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:08:52,656 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:08:52,656 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:08:52,831 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:08:52,832 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:08:52,832 - phishtank-collector - INFO - Bot will continue in 60 seconds. 2019-03-01 20:09:52,891 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:09:53,138 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:09:53,139 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:09:53,139 - phishtank-collector - INFO - Downloading report from 'https://data.phishtank.com/data/online-valid.csv'. 2019-03-01 20:09:53,315 - phishtank-collector - ERROR - Bot has found a problem. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/lib/bot.py", line 167, in start self.process() File "/usr/local/lib/python3.6/dist-packages/intelmq-1.1.1-py3.6.egg/intelmq/bots/collectors/http/collector_http.py", line 76, in process raise ValueError('HTTP response status code was %i.' % resp.status_code) ValueError: HTTP response status code was 509. 2019-03-01 20:09:53,317 - phishtank-collector - INFO - Current Message(event): None. 2019-03-01 20:09:53,318 - phishtank-collector - INFO - Bot will continue in 60 seconds. Sincerely, Václav Brůžek

2 3

Logging summary
by Vaclav Bruzek 30 Apr '19

30 Apr '19

Hi mailing list, I've recently encountered and issue with some of my custom parser bots. The default parsers log a line like this: Sent 7176 events and found 0 error(s) This is quite fine piece of information but I am not getting that for my custom parsers. The only info i have is the number of processed messages. Is there a need to do something special apart from 'acknowledge_message()' and 'send_message()'? Sincerely, Václav Brůžek

2 3

IntelMQ 2.0.0 Beta 1 release
by Sebastian Wagner 10 Apr '19

10 Apr '19

Dear community, The long awaited next major release is coming - in May. For now, there's a beta release as it still has a few bugs we should fix. The release will hit the unstable deb/rpm repositories shortly: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… Thanks to all contributors who made IntelMQ what it is today! Installation instructions: https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/UPGRADING.md The full changelog: There are some features considered as beta and marked as such in the documentation, do not use them in production yet. - upgraded all files to python3-only syntax, e.g. use `super()` instead of `super(..., ...)` in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer. ### Removals of deprecated code: - Removed compatibility shim `intelmq.bots.collectors.n6.collector_stomp`, use `intelmq.bots.collectors.stomp.collector` instead (see #1124). - Removed compatibility shim `intelmq.bots.parsers.cymru_full_bogons.parser`, use `intelmq.bots.parsers.cymru.parser_full_bogons` instead. - Removed compatibility shim handing deprecated parameter `feed` for collectors. Use `name` instead. - Removed deprecated and unused method `intelmq.lib.pipeline.Pipeline.sleep`. - Removed support for deprecated parameter `query_ripe_stat` in `intelmq.bots.experts.ripe.expert`, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (#1291). - Removed deprecated and unused function `intelmq.lib.utils.extract_tar`. ### Core - `lib/pipeline`: - Allow setting the broker of source and destination independently. - Support for a new AMQP broker. See User Guide for configuration. (#1179) - `lib/bot`: - Dump messages locks the dump file using unix file locks (#574). - Print idle/rate limit time also in human readable format (#1332). - `set_request_parameters`: Use `{}` as default proxy value instead of `None`. Allows updating of existing proxy dictionaries. - Bots drop privileges if they run as root. - Save statistics on successfully and failed processed messages in the redis database 3. - `lib/utils` - Function `unzip` to extract files from gzipped and/or tar-archives. - New class `ListHandler`: new handler for logging purpose which saves the messages in a list. - Add function `seconds_to_human`. - Add function `drop_privileges`. - `parse_relative`: Strip string before parsing. - `parse_logline`: Do not convert the timestamps to UTC, leave them as is. - `lib/cache`: - Allow ttl to be None explicitly. - Overwrite existing cache keys in the database instead of discarding the new data. - `lib/bot`: - Basic, but easy-to-configure multi-threading using python's `threading` library. See the User-Guide for more information (#111, #186). - `bin/intelmqctl`: - Support for Supervisor as process manager (#693, #1360). ### Harmonization ### Bots #### Collectors - added `intelmq.bots.parsers.opendxl.collector` (#1265). - added `intelmq.bots.collectors.api`: collecting data using an HTTP API (#123, #1187). - added `intelmq.bots.collectors.rsync` (#1286). - `intelmq.bots.collectors.http.collector_http`: - Add support for uncompressing of gzipped-files (#1270). - Add time-delta support for time formatted URLs (#1366). - `intelmq.collectors.blueliv.collector_crimeserver`: Allow setting the API URL by parameter (#1336). - `intelmq.collectors.mail`: - Use internal lib for functionality. - Add `intelmq.bots.collectors.mail.collector_mail_body`. - Support for `ssl_ca_certificate` parameter (#1362). #### Parsers - added `intelmq.bots.parsers.mcafee.parser_atd` (#1265). - `intelmq.bots.parsers.generic.parser_csv`: - New parameter `columns_required` to optionally ignore parse errors for columns. - added `intelmq.bots.parsers.cert_eu.parser_csv` (#1287). - Do not overwrite the local `time.observation` with the data from the feed. The feed's field 'observation time' is now saved in the field `extra.cert_eu_time_observation`. - Fix parsing of `asn` (renamed to `source asn`, `source.asn` internally) and handle existing `feed.accuracy` for parsing `confidence`. - Update columns and mapping to current (2019-04-02) data. - added `intelmq.bots.parsers.surbl.surbl` - added `intelmq.bots.parsers.html_table` (#1381). - `intelmq.bot.parsers.netlab_360.parser`: Handle empty lines containing blank characters (#1393). - `intelmq.bots.parsers.n6.parser_n6stomp`: Handle events without IP addresses. - `intelmq.bots.parsers.cymru.parser_cap_program`: Handle new feed format. - `intelmq.bots.parsers.shadowserver`: - Add support for the `Accessible-FTP` feed (#1391). - `intelmq.bots.parsers.dataplane.parser`: - Fix parse errors and log more context (#1396). - added `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py` (#1373). #### Experts - added `intelmq.bots.experts.recordedfuture_iprisk` (#1267). - added `intelmq.bots.experts.mcafee.expert_mar` (1265). - renamed `intelmq.bots.experts.ripencc_abuse_contact.expert` to `intelmq.bots.experts.ripe.expert`, compatibility shim will be removed in version 3.0. - Added support for geolocation information in ripe expert with a new parameter `query_ripe_stat_geolocation` (#1317). - Restructurize the expert and de-duplicataion (#1384). - Handle '?' in geolocation country data (#1384). - `intelmq.bots.experts.ripe.expert`: - Use a requests session (#1363). - Set the requests parameters once per session. - `intelmq.bots.experts.maxmind_geoip.expert`: New parameter `use_registered` to use the registered country (#1344). - `intelmq.bots.experts.filter.expert`: Support for paths (#1208). #### Outputs - added `intelmq.bots.experts.mcafee.output_esm` (1265). - added `intelmq.bots.outputs.blackhole` (#1279). - `intelmq.bots.outputs.restapi.expert`: - Set the requests parameters once per session. - `intelmq.bots.outputs.redis`: - New parameter `hierarchichal_output` (#1388). - New parameter `with_type`. - `intelmq.bots.outputs.amqptopic.output`: Compatibility with pika 1.0.0 (#1084, #1394). ### Documentation - added documentation for feeds - CyberCrime Tracker - Feodo Tracker Latest - Feeds: Document abuse.ch URLhaus feed (#1379). - Install and Upgrading: Use `intelmqsetup` tool. ### Packaging ### Tests - Add tests of AMQP broker. - Travis: Change the ownership of `/opt/intelmq` to the current user. ### Tools - `intelmqctl check`: Now uses the new `ListHandler` from utils to handle the logging in JSON output mode. - `intelmqctl run`: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behaviour. - `intelmqdump`: Inspecting dumps locks the dump file using unix file locks (#574). - `intelmqctl`: - After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown. - `intelmqsetup`: New tool for initialize an IntelMQ environment. ### Contrib - `malware_name_mapping`: - Added the script `apply_mapping_eventdb.py` to apply the mapping to an eventdb. - Possibility to add local rules using the download tool. - `check_mk`: - Added scripts for monitoring queues and statistics. ### Known issues - Multi-threaded bots require multiple SIGTERMs (#1403) - Stats can't be saved with AMQP if redis is password-protected (#1402) - Update taxonomies to current RSIT and vice-versa (#1380) - stomp collector bot constantly uses 100% of CPU (#1364) - tests: capture logging with context manager (#1342) - Consistent message counter log messages for all kind of bots (#1278) - pymongo 3.0 deprecates used insert method (#1063) - pymongo >= 3.5: authentication changes (#1062) - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952) - n6 parser: mapping is modified within each run (#905) - reverse DNS: Only first record is used (#877) - Corrupt dump files when interrupted during writing (#870) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Re: [Intelmq-users] Uncaught bounce notification
by Sebastian Wagner 05 Apr '19

05 Apr '19

Hi Majid You sent your mail to intelmq-dev-*bounces*(a)lists.cert.at instead of intelmq-users(a)lists.cert.at or intelmq-dev-bounces(a)lists.cert.at The bounces address has a special meaning, it is not for sending mails to the list. Yes, you need one parser per feed and provide the feed's name as parameter to the parser. Sebastian On 05/04/2019 13.35, "Salehi Ghamsari, Majid" <majid.salehi.ghamsari(a)fokus.fraunhofer.de> wrote: > Hi there, > > to parse multiple resports from shadowserver like Open-Redis and Open-SNMP, do i need multiple shadowserver parser or > there is a simple way to parse multiple reposts. > > Thank you for your help > Majid Salehi -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users April 2019