[Intelmq-dev] IntelMQ 1.1.1 released

Sebastian Wagner wagner at cert.at
Tue Jan 15 17:23:40 CET 2019


Dear community,

Today I released the first maintenance version of the 1.1.x series.
Thanks for all the contributions!

Installation instructions:
https://github.com/certtools/intelmq/blob/1.1.1/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.1.1/docs/UPGRADING.md

The full change log:

### Core
- `lib/harmonization.py`: Change `parse_utc_isoformat` of `DateTime`
class from private to public (related to #1322).
- `lib/utils.py`: Add new function `object_pair_hook_bots`.
- `lib.bot.py`:
  - `ParserBot`'s method `recover_line_csv` now also handles given
`tempdata`.
  - `Bot.acknowledge_message()` deletes `__current_message` to free the
memory, saves memory in idling parsers with big reports.
  - `start()`: Warn once per run if `error_dump_message` is set to false.
  - `Bot.start()`, `ParserBot.process()`: If errors happen on bots
without destination pipeline, the `on_error` path has been queried and
lead to an exception being raised.
  - `start()`: If `error_procedure` is pass and on pipeline errors, the
bot retries forever (#1333).
- `lib/message.py`:
  - Fix add('extra', ..., overwrite=True): old extra fields have not
been deleted previously (#1335).
  - Do not ignore empty or ignored (as defined in `_IGNORED_VALUES`)
values of `extra.*` fields for backwards compatibility (#1335).
- `lib/pipeline.py` (`Redis.receive`): Wait in 1s steps if redis is busy
loading its snapshot from disk (#1334).

### Default configuration
- Set `error_dump_message` to true by default in `defaults.conf`.
- Fixed typo in `defaults.conf`: `proccess_manager` -> `process_manager`

### Development
- `bin/rewrite_config_files.py`: Fix ordering of BOTS file (#1327).

### Harmonization
Update to 2018-09-26 version. New values are per taxonomy:
- Taxonomy 'intrusions':
  - "application-compromise"
  - "burglary"
  - "privileged-account-compromise"
  - "unprivileged-account-compromise"
- Taxonomy 'fraud':
  - "copyright"
  - "masquerade"
  - "unauthorized-use-of-resources"
- Taxonomy 'information content security':
  - "data-loss"
- Taxonomy 'vulnerable':
  - "ddos-amplifier"
  - "information-disclosure"
  - "potentially-unwanted-accessible"
  - "vulnerable-system"
  - "weak-crypto"
- Taxonomy 'availability':
  - "dos"
  - "outage"
  - "sabotage"
- Taxonomy 'abusive-content':
  - "harmful-speech"
  - "violence"
- Taxonomy 'malicious code':
  - "malware-distribution"
- Taxonomy 'information-gathering':
  - "social-engineering"
  - "sniffing"
- Taxonomy 'information content security':
  - "Unauthorised-information-access"
  - "Unauthorised-information-modification"

### Bots
#### Collectors
- `intelmq.bots.collectors.http.collector_http`:
  - Fix parameter name `extract_files` in BOTS (#1331).
  - Fix handling of `extract_files` parameter if the value is an empty
string.
  - Handle not installed dependency library `requests` gracefully.
  - Explain `extract_files` parameter in docs and use a sane default in
BOTS file.
- `intelmq.bots.collectors.mail.collector_mail_url`:
  - Handle HTTP status codes != 2xx the same as HTTP timeouts: No
exception, but graceful handling.
  - Handle HTTP errors (bad status code and timeouts) with
`error_procedure` == 'pass' but marking the mail as read and logging the
error.
  - Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.collectors.http.collector_http_stream`:
  - Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.collectors.microsoft.collector_interflow`:
  - Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.collectors.rt.collector_rt`:
  - Handle not installed dependency library `requests` gracefully.
- added `intelmq.bots.collectors.shodan.collector_stream` for collecting
shodan stream data (#1096).
  - Correctly check the version of the shodan library, it resulted in
wrong comparisons with two digit numbers.
- `intelmq.bots.collectors.microsoft.collector_interflow`:
  - Add check if Cache's TTL is big enough compared to `not_older_than`
and throw an error otherwise.

#### Parsers
- `intelmq.bots.parsers.misp`: Fix Object attribute (#1318).
- `intelmq.bots.parsers.cymru.parser_cap_program`:
  - Add support for new format (extra data about botnet of 'bots').
  - Handle AS number 0.
- `intelmq.bots.parsers.shadowserver`:
  - Spam URL reports: remove `src_naics`, `src_sic` columns.
  - fix parsing of 'spam' events in ShadowServer's 'Botnet Drone Hadoop'
Report (#1271).
  - Add support in parser to ignore some columns in config file by using
`False` as intelmq key.
  - Add support for the `Outdated-DNSSEC-Key` and
`Outdated-DNSSEC-Key-IPv6` feeds.
  - Add support for the `Accessible-Rsync` feed.
  - Document support for the `Open-LDAP-TCP` feed.
  - Add support for `Accessible-HTTP` and `Open-DB2-Discovery-Service`
(#1349).
  - Add support for `Accessible-AFP` (#1351).
  - Add support for `Darknet` (#1353).
- `intelmq.bots.parsers.generic.parser_csv`: If the `skip_header`
parameter was set to `True`, the header was not part of the `raw` field
as returned by the `recover_line` method. The header is now saved and
handled correctly by the fixed recovery method.
- `intelmq.bots.parsers.cleanmx.parser`: Use field `first` instead of
`firsttime` for `time.source` (#1329, #1348).
- `intelmq.bots.parsers.twitter.parser`: Support for `url-normalize` >=
1.4.1 and recommend it. Added new optional parameter `default_scheme`,
passed to `url-normalize` (#1356).

#### Experts
- `intelmq.bots.experts.national_cert_contact_certat.expert`:
  - Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.experts.ripencc_abuse_contact.expert`:
  - Handle not installed dependency library `requests` gracefully.
- `intelmq.bots.experts.sieve.expert`:
  - check method: Add missing of the harmonization for the check, caused
an error for every check.
  - Add text and more context to error messages.
  - README: Fix 'modify' to 'update' (#1340).
  - Handle empty rules file (#1343).
- `intelmq.bots.experts.idea.expert`: Add mappings for new harmonization
`classification.type` values, see above.

#### Outputs
- `intelmq.bots.outputs.redis`:
  - Fix sending password to redis server.
  - Fix for redis-py >= 3.0.0: Convert Event to string explicitly (#1354).
  - Use `Redis` class instead of deprecated `StrictRedis` for redis-py
>= 3.0.0 (#1355).
- `intelmq.bots.outputs.mongodb`:
  - New parameter `replacement_char` (default: `'_'`) for
non-hierarchical output as dots in key names are not allowed (#1324, #1322).
  - Save value of fields `time.observation` and `time.source` as native
datetime object, not as string (#1322).
- `intelmq.bots.outputs.restapi.output`:
  - Handle not installed dependency library `requests` gracefully.

### Documentation
- FAQ
  - Explanation and solution on orphaned queues.
  - Section on how and why to remove `raw` data.
- Add or fix the tables of contents for all documentation files.
- Feeds:
  - Fix Autoshun Feed URL (#1325).
  - Add parameters `name` and `provider` to `intelmq/etc/feeds.yaml`,
`docs/Feeds.md` and `intelmq/bots/BOTS` (#1321).
- Add SECURITY.md file.

### Packaging
- Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320).

### Tests
- `intelmq.tests.lib.test_bot`: Skip `test_logging_level_other` on
python 3.7 because of unclear behavior related to copies of loggers (#1269).
- `intelmq.tests.bots.collectors.rt.test_collector`: Remove test because
the REST interface of the instance has been closed (see also
https://github.com/CZ-NIC/python-rt/issues/28).

### Tools
- `intelmqctl check`: Shows more detailed information on orphaned queues.
- `intelmqctl`:
  - Correctly determine the status of bots started with `intelmqctl run`.
  - Fix output of errors during bot status determination, making it
compatible to IntelMQ Manager.
  - `check` subcommand: Show bot ID for messages also in JSON output.
  - `run [bot-id] process -m [message]` works also with bots without a
configured source pipeline (#1307).

### Contrib
- elasticsearch/elasticmapper: Add tlp field (#1308).
- `feeds-config-generator/intelmq_gen_feeds_conf`:
  - Add parameters to write resulting configuration directly to files
(#1321).
  - Handle collector's `feed.name` and `feed.provider` (#1314).

### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is restarted
(#952).
- Tests: capture logging with context manager (#1342).
- stomp collector bot constantly uses 100% of CPU (#1364).

-- 
// Sebastian Wagner <wagner at cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20190115/96ea63f7/attachment.sig>


More information about the Intelmq-dev mailing list