Dear IntelMQ community,
I hope everyone is fine and most of us are slowly waking up after the Corona lock-down phase ( at least I am, and it's a really good feeling to be able to work concentrated on things again. Let me tell you, home-office with small children is not fun).
In any case, I already had some telcos with some of you regarding our wish list for IntelMQ-3.0. There are some fantastic ideas floating around and since IntelMQ is such a versatile and adaptable tool, it is quite a challenge to capture all the great changes and adaptations people made to it.
The next version of IntelMQ (3.0) will try to capture some of the most needed changes (and some wishes) for you.
I tried to summarise some of these changes in the following document:
https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectu…
May I ask you to review it and send me feedback (kaplan(a)cert.at)? Also, I'd be very happy to have a telco with you in case that's easier or more natural for you for giving feedback.
Since we would like to start coding soon, it is important to receive *your* input on the next version of IntelMQ.
Thanks for your time!
All the best,
Aaron.
--
// L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
The newest IntelMQ Manager release 2.1.1 fixes a critical security bug.
Please never run the IntelMQ Manager without proper authentication in place!
Installation instructions:
https://github.com/certtools/intelmq-manager/blob/2.1.1/docs/INSTALL.md
Bernhard Herzog (Intevation) discovered that the backend incorrectly
handled messages given by user-input in the "send" functionality of the
Inspect-tool of the Monitor component. An attacker with access to the
IntelMQ Manager could possibly use this issue to execute arbitrary code
with the privileges of the webserver.
Updated deb/rpm-packages are already available in the repositories.
Other changes:
### Backend
- Fix misspelling of the environmental variable
`INTELMQ_MANGER_CONTROLLER_CMD` to `INTELMQ_MANAGER_CONTROLLER_CMD` (an
'a' was missing).
- Fix handling of POST variable `msg` of the message-sending
functionality available in the Inspect-tool.
### Pages
#### Monitor
- Fix running commands with the "inspect" widget by fixing the
definition of the `CONTROLLER_CMD` in the template (PR #194).
### Documentation
- Update supported operating systems in Installation documentation (i.a.
PR #191).
### Known issues
* Missing CSRF protection (#111).
* Graph jumps around on "Add edge" (#148).
* wrong error message for new bots with existing ID (#152).
* `ALLOWED_PATH=` violates CSP (#183).
* Monitor page: Automatic log refresh reset log page to first one (#190).
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
The ElasticSearch bots, tests and tools in IntelMQ need some maintenance
which I am unable to provide. As ES is a very common tool I am sure that
there is know-how available in the community and we are able to continue
the support for it.
The oldest know issue is a broken unittest:
https://github.com/certtools/intelmq/issues/1480
But there are also incompatibilties with current ElasticSearch version,
e.g. I had problems with the elasticmapper tool using ES 7.6.1 (maybe
easy to fix).
Using 7.5.0 failed on the indices tests
https://github.com/certtools/intelmq/issues/1479
Further, the only supported elasticsearch python library version is
currently 'elasticsearch>=5.0.0,<6.0.0' while the latest release is 7.6.0.
Please consider contributing
best regards
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi,
(involving intelmq-dev now so that we can move that discussion to the
developers' list)
Thanks for the hint. That could be a possible replacement. When
analyzing the stat.ripe.net webinterface I also found this endpoint
giving the same result:
https://stat.ripe.net/data/rir-geo/data.json?resource=131.130.254.77
Has anyone a clue why RIPE provides so many different endpoints for the
same data? (With different status which is not properly propagated to
the status code...)
best wishes,
Sebastian
On 3/17/20 10:55 PM, Chris Horsley wrote:
>
> Is this alternative RIPE API endpoint a feasible alternative?
>
> https://stat.ripe.net/data/geoloc/data.json?resource=131.130.254.77/24
>
> Cheers,
>
> Chris
>
> On 18/03/2020 3:14 am, Sebastian Wagner wrote:
>>
>> Hi,
>>
>> I just noticed, that RIPE currently does not provide geolocation
>> information anymore as a result of the MaxMind data license change.
>> That data can/could be queried with the IntelMQ RIPE expert. In case
>> you are still relying on this information, please use another source
>> for geolocation data, like the maxmind geolocation expert and local
>> data. Unfortunately, the returned status code of the API call is 200
>> and the error is only detectable by another field. I am working on
>> changes in the RPIE expert to detect this and raise a warning for it.
>>
>> best regards,
>> Sebastian
>>
>> For example
>> https://stat.ripe.net/data/maxmind-geo-lite/data.json?resource=131.130.254.…
>> says:
>>
>> messages
>> 0
>> 0 "info"
>> 1 "This data is currently unavailable due to maintenance. Please
>> check official announcements for when it will be available again!
>> https://stat.ripe.net/feedback"
>> data_call_status "maintenance - this data call is in maintenance mode"
>>
>> --
>> // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
>> // CERT Austria - https://www.cert.at/
>> // Eine Initiative der nic.at GmbH - https://www.nic.at/
>> // Firmenbuchnummer 172568b, LG Salzburg
>>
>
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear friends of IntelMQ,
just wondering: are their tutorial out there to get a beginner
to have IntelMQ running and doing something useful?
Possible steps to cover:
* Get a first feed in, using a public feed
* Setup a simble "botnet",
e.g. one that filters for my country, ASN or network range
* Do a simple output task, e.g. like creating a DNS RPZ file (once)
Not covering installation, but first setup
Maybe setup with IntelMQ Manager or without.
Can be textual or otherwise.
Background of the question: For new users or development setups, it is needed
to get an up-to-date, working IntelMQ setup. Doing a few searches on the
internet I did not see a tutorial for this and the current documentation is
geared towards being a comprensive reference.
Saw
https://github.com/certtools/intelmq/issues/256 Request for a Video Tutorial
Just to saw: For me videos do not work best and they probably are a lot of
work compared to a classic text and screenshot based tutorial.
So is anything already out there? :)
Regards,
Bernhard
--
www.intevation.de/~bernhard +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
Dear community,
Version 2.1.2, a bugfix release, is out which contains various bugfixes.
Install documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/UPGRADING.md
The full changelog:
### Core
- `__init__`: Resolve absolute path for `STATE_FILE_PATH` variable
(resolves `..`).
- `intelmq.lib.utils`:
- log: Do not raise an exception if logging to neither file nor syslog
is requested.
- logging StreamHandler: Colorize all warning and error messages red.
- logging FileHandler: Strip all shell colorizations from the messages
(#1436).
- `intelmq.lib.message`:
- `Message.to_json`: Set `sort_keys=True` to get reproducible results.
- `drop_privileges`: Handle situations where the user or group
`intelmq` does not exist.
- `intelmq.lib.pipeline`:
- `Amqp._send` and `Amqp._acknowledge`: Log traceback in debug mode in
case of errors and necessary re-connections.
- `Amqp._acknowledge`: Reset delivery tag if acknowledge was successful.
### Bots
#### Collectors
- `intelmq.bots.collectors.misp.collector`:
- Add compatibility with current pymisp versions and versions released
after January 2020 (PR #1468).
#### Parsers
- `intelmq.bots.parsers.shadowserver.config`: Add some missing fields
for the feed `accessible-rdp` (#1463).
- `intelmq.bots.parsers.shadowserver.parser`:
- Feed-detection based on file names: The prefixed date is optional now.
- Feed-detection based on file names: Re-detect feed for every report
received (#1493).
#### Experts
- `intelmq.bots.experts.national_cert_contact_certat`: Handle empty
responses by server (#1467).
- `intelmq.bots.experts.maxmind_geoip`: The script `update-geoip-data`
now requires a license key as second parameter because of upstream
changes (#1484)).
#### Outputs
- `intelmq.bots.outputs.restapi.output`: Fix logging of response body if
response status code was not ok.
### Documentation
- Remove some hardcoded `/opt/intelmq/` paths from code comments and
program outputs.
### Packaging
- debian/rules: Only replace `/opt/intelmq/` with LSB-paths in some
certain files, not the whole tree, avoiding wrong replacements.
- debian/rules and debian/intelmq.install: Do install the examples
configuration directly instead of working around the abandoned examples
directory.
### Tests
- `lib/test_utils`: Skip some tests on Python 3.4 because
`contextlib.redirect_stdout` and `contextlib.redirect_sterr` are not
supported on this version.
- Travis: Stop running tests with all optional dependencies on Python
3.4, as more and more libraries are dropping support for it. Tests on
the core and code without non-optional requirements are not affected.
- `tests.bots.parsers.html_table`: Make tests independent of current year.
### Tools
- `intelmqctl upgrade-config`: Fix missing substitution in error message
"State file %r is not writable.".
### Known issues
- bots trapped in endless loop if decoding of raw message fails (#1494)
- intelmqctl status of processes: need to check bot id too (#1492)
- MongoDB authentication: compatibility on different MongoDB and pymongo
versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear IntelMQ users,
Sebix mentioned something that might be relevant for everyone using IntelMQ (or actually maxmind for the matter).
(quote) "MaxMind has always been committed to an individual’s right to privacy on the internet. We welcome the burgeoning privacy regulations, such as GDPR and CCPA, for the benefit they can provide to internet citizens. However, these new legislative measures place restrictions that impact our ability to continue distributing our GeoLite2 databases on a public page under the Creative Commons Attribution-ShareAlike 4.0 International License."
Maxmind has decided to change the download mechanism of the maxmind GeoLite 2 database. You now need to be registered and need a license key.
Since I know that many IntelMQ users rely on maxmind for geolocation, you might want to register there and get a new license key and make sure it can be used for the most recent database version.
More info at https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-us…
All the best and a happy 2020!
Aaron.
--
// L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
we again collected a bunch of bugfixes in the last weeks, coming almost
one month after 2.1.0.
Install documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/UPGRADING.md
The full changelog:
### Configuration
- Default configuration:
- Remove discontinued feed "Feodo Tracker Domains" from default
configuration.
- Add "Feodo Tracker Browse" feed to default configuration.
### Core
- `intelmq.lib.pipeline`: AMQP: using port 15672 as default (like
RabbitMQ's defaults) for the monitoring interface for getting
statistical data (`intelmqctl_rabbitmq_monitoring_url`).
- `intelmq.lib.upgrades`: Added a generic upgrade function for
harmonization, checking of all message types, it's fields and their types.
- `intelmq.lib.utils`:
- `TimeoutHTTPAdapter`: A subclass of `requests.adapters.HTTPAdapter`
with the possibility to set the timeout per adapter.
- `create_request_session_from_bot`: Use the `TimeoutHTTPAdapter` with
the user-defined timeout. Previously the timeout was not functional.
### Bots
#### Parsers
- `intelmq.bots.parsers.shadowserver.parser`: Fix logging message if the
parameter `feedname` is not present.
- `intelmq.bots.parsers.shodan.parser`: Also add field
`classification.identifier` (`'network-scan'`) in minimal mode.
- `intelmq.bots.parsers.spamhaus.parser_cert`: Add support for category
`'misc'`.
- `intelmq.bots.parsers.cymru.parser_cap_program`:
- Add support for phishing events without URL.
- Add support for protocols >= 143 (unassigned, experiments, testing,
reserved), saving the number to extra, as the data would be bogus.
- `intelmq.bots.parsers.microsoft.parser_bingmurls`:
- Save the `Tags` data as `source.geolocation.cc`.
#### Experts
- `intelmq.bots.experts.modify.expert`: Fix bug with setting non-string
values (#1460).
#### Outputs
- `intelmq.bots.outputs.smtp`:
- Allow non-existent field in text formatting by using a default value
`None` instead of throwing errors.
- Fix Authentication (#1464).
- Fix sending to multiple recipients (#1464).
### Documentation
- Feeds:
- Fix configuration of `Feodo Tracker Browse` feed.
- Bots:
- Sieve expert: Document behavior of `!=` with lists.
### Tests
- Adaption and extension of the test cases to the changes.
### Tools
- `intelmq.bin.intelmqctl`:
- check: Check if running the upgrade function for harmonization is
necessary.
- upgrade-config: Run the upgrade function for harmonization.
- `intelmqctl restart` did throw an error as the message for
restarting was not defined (#1465).
### Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo
versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
After updating to version 2.1.0 the file collector raise exceptions.InvalidKey(key).
How can I find out what the problem is, this bot has worked very well under 1.1.1?
Please help
Regards Majid
Bot has found a problem.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start
self.process()
File "/usr/local/lib/python3.5/dist-packages/intelmq/bots/collectors/file/collector_file.py", line 66, in process
template.add("extra.file_name", f)
File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/message.py", line 234, in add
raise exceptions.InvalidKey(key)
intelmq.lib.exceptions.InvalidKey: invalid key 'extra.file_name'
Majid Salehi Ghamsari
Wiss. Mitarbeiter Digital Public Services
Tel: +49 30 3463-7118
Fax: +49 30 3463-99-7118
E-Mail: majid.salehi.ghamsari(a)fokus.fraunhofer.de
Fraunhofer-Institut
für Offene Kommunikationssysteme FOKUS
Kaiserin-Augusta-Allee 31
10589 Berlin
http://www.fokus.fraunhofer.de