Hi,
We've been running intelmq in a production-like setup for some time now and occasionally we see the Shadowserver Parser bot behave in a way that looks odd. We push reports to it via the mail attachment collector bot. At times, the parser bot ends up treating all reports as being of the type Vulnerable ISAKMP. We don't know what triggers this behaviour but so far we've seen this when the parser parses Sandbox URL and Open TFTP reports, both filling the log (debug level) with notifications of "missing keys" or "optional keys not found" because the parser assumes the feed to be Vulnerable ISAKMP. And before this occurs, the parser has been running for a good while.
We've picked up the the problematic Sandbox URL and Open TFTP reports above and made unit test cases of these - all tests are correctly parsed, the feed name is deduced correctly and the tests are successful, so the parser handles these correctly if these are the first reports parsed. Therefore, our assumption is that this problem occurs only when the parser has been running for a longer time and some state information about the feed type does not get cleared between parsing incoming reports.
Anyone else experiencing similar problems? This is a tricky one to debug so I decided to ask on the list first.
Br, Mika
P.S: Our setup is intelmq 3.0.2 on an Ubuntu 20.04 LTS
Can you please show the configuration (parameters) of your shadowserver parser?
On 9/5/22 9:42 AM, Mika Silander wrote:
Hi,
We've been running intelmq in a production-like setup for some time now and occasionally we see the Shadowserver Parser bot behave in a way that looks odd. We push reports to it via the mail attachment collector bot. At times, the parser bot ends up treating all reports as being of the type Vulnerable ISAKMP. We don't know what triggers this behaviour but so far we've seen this when the parser parses Sandbox URL and Open TFTP reports, both filling the log (debug level) with notifications of "missing keys" or "optional keys not found" because the parser assumes the feed to be Vulnerable ISAKMP. And before this occurs, the parser has been running for a good while.
We've picked up the the problematic Sandbox URL and Open TFTP reports above and made unit test cases of these - all tests are correctly parsed, the feed name is deduced correctly and the tests are successful, so the parser handles these correctly if these are the first reports parsed. Therefore, our assumption is that this problem occurs only when the parser has been running for a longer time and some state information about the feed type does not get cleared between parsing incoming reports.
Anyone else experiencing similar problems? This is a tricky one to debug so I decided to ask on the list first.
Br, Mika
P.S: Our setup is intelmq 3.0.2 on an Ubuntu 20.04 LTS _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Hi,
Currently the parameters section for the parser bot in runtime.yaml is just:
parameters: destination_queues: _default: [fc-set-event-constant-expert-queue] overwrite: true run_mode: continuous
Br, Mika
----- Original Message ----- From: "Sebix" sebix@sebix.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 5 September, 2022 10:43:56 Subject: Re: [IntelMQ-dev] Feed handling bug in shadowserver parser bot?
Can you please show the configuration (parameters) of your shadowserver parser?
On 9/5/22 9:42 AM, Mika Silander wrote:
Hi,
We've been running intelmq in a production-like setup for some time now and occasionally we see the Shadowserver Parser bot behave in a way that looks odd. We push reports to it via the mail attachment collector bot. At times, the parser bot ends up treating all reports as being of the type Vulnerable ISAKMP. We don't know what triggers this behaviour but so far we've seen this when the parser parses Sandbox URL and Open TFTP reports, both filling the log (debug level) with notifications of "missing keys" or "optional keys not found" because the parser assumes the feed to be Vulnerable ISAKMP. And before this occurs, the parser has been running for a good while.
We've picked up the the problematic Sandbox URL and Open TFTP reports above and made unit test cases of these - all tests are correctly parsed, the feed name is deduced correctly and the tests are successful, so the parser handles these correctly if these are the first reports parsed. Therefore, our assumption is that this problem occurs only when the parser has been running for a longer time and some state information about the feed type does not get cleared between parsing incoming reports.
Anyone else experiencing similar problems? This is a tricky one to debug so I decided to ask on the list first.
Br, Mika
P.S: Our setup is intelmq 3.0.2 on an Ubuntu 20.04 LTS _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Hi,
There *was* a bug similar to this, but I fixed that one two years ago in 2.1.2. Just for reference: https://github.com/certtools/intelmq/issues/1493 https://github.com/certtools/intelmq/commit/ac9e442f522e9bc5fcfe1cb5591d7b63...
The shadowserver parsers detects the feed based on the reported file name, as passed on by the collector. I'd clarify if - the field `extra.file_name` of the mail collector's outgoing report is correct - the shadowserver parser correctly determines the feed from it. The shadowserver parser logs (in debug level) "Detected report's file name ..." for every incoming report. That's the first thing I'd check.
Sebastian
On 9/5/22 9:48 AM, Mika Silander wrote:
Hi,
Currently the parameters section for the parser bot in runtime.yaml is just:
parameters: destination_queues: _default: [fc-set-event-constant-expert-queue] overwrite: true run_mode: continuous
Br, Mika
----- Original Message ----- From: "Sebix" sebix@sebix.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 5 September, 2022 10:43:56 Subject: Re: [IntelMQ-dev] Feed handling bug in shadowserver parser bot?
Can you please show the configuration (parameters) of your shadowserver parser?
On 9/5/22 9:42 AM, Mika Silander wrote:
Hi,
We've been running intelmq in a production-like setup for some time now and occasionally we see the Shadowserver Parser bot behave in a way that looks odd. We push reports to it via the mail attachment collector bot. At times, the parser bot ends up treating all reports as being of the type Vulnerable ISAKMP. We don't know what triggers this behaviour but so far we've seen this when the parser parses Sandbox URL and Open TFTP reports, both filling the log (debug level) with notifications of "missing keys" or "optional keys not found" because the parser assumes the feed to be Vulnerable ISAKMP. And before this occurs, the parser has been running for a good while.
We've picked up the the problematic Sandbox URL and Open TFTP reports above and made unit test cases of these - all tests are correctly parsed, the feed name is deduced correctly and the tests are successful, so the parser handles these correctly if these are the first reports parsed. Therefore, our assumption is that this problem occurs only when the parser has been running for a longer time and some state information about the feed type does not get cleared between parsing incoming reports.
Anyone else experiencing similar problems? This is a tricky one to debug so I decided to ask on the list first.
Br, Mika
P.S: Our setup is intelmq 3.0.2 on an Ubuntu 20.04 LTS _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Hi Sebastian,
I tried to find info about this problem from the intelmq repo in github but was unlucky with the search criteria. It could be a similar problem, but if it was fixed in 2.1.2, it is as if a variant of it has sneaked in into 3.0.2. I'll try to debug.
Br, Mika
----- Original Message ----- From: "Sebix" sebix@sebix.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 5 September, 2022 10:58:08 Subject: Re: [IntelMQ-dev] Feed handling bug in shadowserver parser bot?
Hi,
There *was* a bug similar to this, but I fixed that one two years ago in 2.1.2. Just for reference: https://github.com/certtools/intelmq/issues/1493 https://github.com/certtools/intelmq/commit/ac9e442f522e9bc5fcfe1cb5591d7b63...
The shadowserver parsers detects the feed based on the reported file name, as passed on by the collector. I'd clarify if - the field `extra.file_name` of the mail collector's outgoing report is correct - the shadowserver parser correctly determines the feed from it. The shadowserver parser logs (in debug level) "Detected report's file name ..." for every incoming report. That's the first thing I'd check.
Sebastian
On 9/5/22 9:48 AM, Mika Silander wrote:
Hi,
Currently the parameters section for the parser bot in runtime.yaml is just:
parameters: destination_queues: _default: [fc-set-event-constant-expert-queue] overwrite: true run_mode: continuous
Br, Mika
----- Original Message ----- From: "Sebix" sebix@sebix.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 5 September, 2022 10:43:56 Subject: Re: [IntelMQ-dev] Feed handling bug in shadowserver parser bot?
Can you please show the configuration (parameters) of your shadowserver parser?
On 9/5/22 9:42 AM, Mika Silander wrote:
Hi,
We've been running intelmq in a production-like setup for some time now and occasionally we see the Shadowserver Parser bot behave in a way that looks odd. We push reports to it via the mail attachment collector bot. At times, the parser bot ends up treating all reports as being of the type Vulnerable ISAKMP. We don't know what triggers this behaviour but so far we've seen this when the parser parses Sandbox URL and Open TFTP reports, both filling the log (debug level) with notifications of "missing keys" or "optional keys not found" because the parser assumes the feed to be Vulnerable ISAKMP. And before this occurs, the parser has been running for a good while.
We've picked up the the problematic Sandbox URL and Open TFTP reports above and made unit test cases of these - all tests are correctly parsed, the feed name is deduced correctly and the tests are successful, so the parser handles these correctly if these are the first reports parsed. Therefore, our assumption is that this problem occurs only when the parser has been running for a longer time and some state information about the feed type does not get cleared between parsing incoming reports.
Anyone else experiencing similar problems? This is a tricky one to debug so I decided to ask on the list first.
Br, Mika
P.S: Our setup is intelmq 3.0.2 on an Ubuntu 20.04 LTS _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Mika,
short, maybe stupid question: are you using the new shadowserver parser in 3.x?
Best, Aaron.
On 05.09.2022, at 11:16, Mika Silander mika.silander@csc.fi wrote:
Hi Sebastian,
I tried to find info about this problem from the intelmq repo in github but was unlucky with the search criteria. It could be a similar problem, but if it was fixed in 2.1.2, it is as if a variant of it has sneaked in into 3.0.2. I'll try to debug.
Br, Mika
----- Original Message ----- From: "Sebix" sebix@sebix.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 5 September, 2022 10:58:08 Subject: Re: [IntelMQ-dev] Feed handling bug in shadowserver parser bot?
Hi,
There *was* a bug similar to this, but I fixed that one two years ago in 2.1.2. Just for reference: https://github.com/certtools/intelmq/issues/1493 https://github.com/certtools/intelmq/commit/ac9e442f522e9bc5fcfe1cb5591d7b63...
The shadowserver parsers detects the feed based on the reported file name, as passed on by the collector. I'd clarify if
- the field `extra.file_name` of the mail collector's outgoing report is
correct
- the shadowserver parser correctly determines the feed from it. The
shadowserver parser logs (in debug level) "Detected report's file name ..." for every incoming report. That's the first thing I'd check.
Sebastian
On 9/5/22 9:48 AM, Mika Silander wrote:
Hi,
Currently the parameters section for the parser bot in runtime.yaml is just:
parameters: destination_queues: _default: [fc-set-event-constant-expert-queue] overwrite: true run_mode: continuous
Br, Mika
----- Original Message ----- From: "Sebix" sebix@sebix.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 5 September, 2022 10:43:56 Subject: Re: [IntelMQ-dev] Feed handling bug in shadowserver parser bot?
Can you please show the configuration (parameters) of your shadowserver parser?
On 9/5/22 9:42 AM, Mika Silander wrote:
Hi,
We've been running intelmq in a production-like setup for some time now and occasionally we see the Shadowserver Parser bot behave in a way that looks odd. We push reports to it via the mail attachment collector bot. At times, the parser bot ends up treating all reports as being of the type Vulnerable ISAKMP. We don't know what triggers this behaviour but so far we've seen this when the parser parses Sandbox URL and Open TFTP reports, both filling the log (debug level) with notifications of "missing keys" or "optional keys not found" because the parser assumes the feed to be Vulnerable ISAKMP. And before this occurs, the parser has been running for a good while.
We've picked up the the problematic Sandbox URL and Open TFTP reports above and made unit test cases of these - all tests are correctly parsed, the feed name is deduced correctly and the tests are successful, so the parser handles these correctly if these are the first reports parsed. Therefore, our assumption is that this problem occurs only when the parser has been running for a longer time and some state information about the feed type does not get cleared between parsing incoming reports.
Anyone else experiencing similar problems? This is a tricky one to debug so I decided to ask on the list first.
Br, Mika
P.S: Our setup is intelmq 3.0.2 on an Ubuntu 20.04 LTS _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
-- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://sebix.at/ ZVR 1510673578 _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
Hi Aaron,
We are basically using intelmq 3.0.2, i.e. the sw tagged "3.0.2" in the intelmq repo's maintenance branch + a bunch of bots of our own, so the answer is most likely "no". The idea was to use an as stable version as possible for production and then experiment with the development versions in a test environment. This approach is not optimal because the maintenance branch does not get backports nor fixes (correct?), so we end up trailing far behind, including with the Shadowserver Parser bot. Then again, using the newest versions from development branches for production probably leads to more troubleshooting and debugging work due to less tested features and components. A tricky decision.
What is your take on the choice of a basis for a production installation? A newer intelmq maintenance release is behind the corner as well I've understood ...
Br, Mika
----- Original Message ----- From: "L. Aaron Kaplan" aaron@lo-res.org To: "Mika Silander" mika.silander@csc.fi Cc: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 5 September, 2022 12:26:47 Subject: Re: [IntelMQ-dev] Feed handling bug in shadowserver parser bot?
Mika,
short, maybe stupid question: are you using the new shadowserver parser in 3.x?
Best, Aaron.
On 05.09.2022, at 11:16, Mika Silander mika.silander@csc.fi wrote:
Hi Sebastian,
I tried to find info about this problem from the intelmq repo in github but was unlucky with the search criteria. It could be a similar problem, but if it was fixed in 2.1.2, it is as if a variant of it has sneaked in into 3.0.2. I'll try to debug.
Br, Mika
----- Original Message ----- From: "Sebix" sebix@sebix.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 5 September, 2022 10:58:08 Subject: Re: [IntelMQ-dev] Feed handling bug in shadowserver parser bot?
Hi,
There *was* a bug similar to this, but I fixed that one two years ago in 2.1.2. Just for reference: https://github.com/certtools/intelmq/issues/1493 https://github.com/certtools/intelmq/commit/ac9e442f522e9bc5fcfe1cb5591d7b63...
The shadowserver parsers detects the feed based on the reported file name, as passed on by the collector. I'd clarify if
- the field `extra.file_name` of the mail collector's outgoing report is
correct
- the shadowserver parser correctly determines the feed from it. The
shadowserver parser logs (in debug level) "Detected report's file name ..." for every incoming report. That's the first thing I'd check.
Sebastian
On 9/5/22 9:48 AM, Mika Silander wrote:
Hi,
Currently the parameters section for the parser bot in runtime.yaml is just:
parameters: destination_queues: _default: [fc-set-event-constant-expert-queue] overwrite: true run_mode: continuous
Br, Mika
----- Original Message ----- From: "Sebix" sebix@sebix.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 5 September, 2022 10:43:56 Subject: Re: [IntelMQ-dev] Feed handling bug in shadowserver parser bot?
Can you please show the configuration (parameters) of your shadowserver parser?
On 9/5/22 9:42 AM, Mika Silander wrote:
Hi,
We've been running intelmq in a production-like setup for some time now and occasionally we see the Shadowserver Parser bot behave in a way that looks odd. We push reports to it via the mail attachment collector bot. At times, the parser bot ends up treating all reports as being of the type Vulnerable ISAKMP. We don't know what triggers this behaviour but so far we've seen this when the parser parses Sandbox URL and Open TFTP reports, both filling the log (debug level) with notifications of "missing keys" or "optional keys not found" because the parser assumes the feed to be Vulnerable ISAKMP. And before this occurs, the parser has been running for a good while.
We've picked up the the problematic Sandbox URL and Open TFTP reports above and made unit test cases of these - all tests are correctly parsed, the feed name is deduced correctly and the tests are successful, so the parser handles these correctly if these are the first reports parsed. Therefore, our assumption is that this problem occurs only when the parser has been running for a longer time and some state information about the feed type does not get cleared between parsing incoming reports.
Anyone else experiencing similar problems? This is a tricky one to debug so I decided to ask on the list first.
Br, Mika
P.S: Our setup is intelmq 3.0.2 on an Ubuntu 20.04 LTS _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
-- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://sebix.at/ ZVR 1510673578 _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
On 9/5/22 11:58 AM, Mika Silander wrote:
We are basically using intelmq 3.0.2, i.e. the sw tagged "3.0.2" in the intelmq repo's maintenance branch + a bunch of bots of our own, so the answer is most likely "no".
The diff between 3.0.2 and develop in the shadowserver parser is negligible. The develop branch has newer /mappings/, contributed/maintained by Shadowserver, but these mappings need tests/feedback either, and have nothing to do with the parser itself.
The idea was to use an as stable version as possible for production and then experiment with the development versions in a test environment. This approach is not optimal because the maintenance branch does not get backports nor fixes (correct?),
It did, a while ago, but not anymore. The idea was to keep "maintenance" stable, with backported bugfixes, while "develop" has all the new stuff until it is well tested and we do a new major/minor release. However, that was based on my own motivation and when I worked at CERT.at. However that maintenance work was obviously not necessary, as no one did bother to do this after I left there and no one was motivated to keep it going (or fund it).
so we end up trailing far behind, including with the Shadowserver Parser bot. Then again, using the newest versions from development branches for production probably leads to more troubleshooting and debugging work due to less tested features and components. A tricky decision.
What is your take on the choice of a basis for a production installation? A newer intelmq maintenance release is behind the corner as well I've understood ...
Yeah, but for that I'm in need of 1) feedback on the Release Candidate version itself and 2) pull request reviews to fix issues. So we are stuck, basically.
Sebastian
Br, Mika
----- Original Message ----- From: "L. Aaron Kaplan" aaron@lo-res.org To: "Mika Silander" mika.silander@csc.fi Cc: "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 5 September, 2022 12:26:47 Subject: Re: [IntelMQ-dev] Feed handling bug in shadowserver parser bot?
Mika,
short, maybe stupid question: are you using the new shadowserver parser in 3.x?
Best, Aaron.
On 05.09.2022, at 11:16, Mika Silander mika.silander@csc.fi wrote:
Hi Sebastian,
I tried to find info about this problem from the intelmq repo in github but was unlucky with the search criteria. It could be a similar problem, but if it was fixed in 2.1.2, it is as if a variant of it has sneaked in into 3.0.2. I'll try to debug.
Br, Mika
----- Original Message ----- From: "Sebix" sebix@sebix.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 5 September, 2022 10:58:08 Subject: Re: [IntelMQ-dev] Feed handling bug in shadowserver parser bot?
Hi,
There *was* a bug similar to this, but I fixed that one two years ago in 2.1.2. Just for reference: https://github.com/certtools/intelmq/issues/1493 https://github.com/certtools/intelmq/commit/ac9e442f522e9bc5fcfe1cb5591d7b63...
The shadowserver parsers detects the feed based on the reported file name, as passed on by the collector. I'd clarify if
- the field `extra.file_name` of the mail collector's outgoing report is
correct
- the shadowserver parser correctly determines the feed from it. The
shadowserver parser logs (in debug level) "Detected report's file name ..." for every incoming report. That's the first thing I'd check.
Sebastian
On 9/5/22 9:48 AM, Mika Silander wrote:
Hi,
Currently the parameters section for the parser bot in runtime.yaml is just:
parameters: destination_queues: _default: [fc-set-event-constant-expert-queue] overwrite: true run_mode: continuous
Br, Mika
----- Original Message ----- From: "Sebix" sebix@sebix.at To: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Monday, 5 September, 2022 10:43:56 Subject: Re: [IntelMQ-dev] Feed handling bug in shadowserver parser bot?
Can you please show the configuration (parameters) of your shadowserver parser?
On 9/5/22 9:42 AM, Mika Silander wrote:
Hi,
We've been running intelmq in a production-like setup for some time now and occasionally we see the Shadowserver Parser bot behave in a way that looks odd. We push reports to it via the mail attachment collector bot. At times, the parser bot ends up treating all reports as being of the type Vulnerable ISAKMP. We don't know what triggers this behaviour but so far we've seen this when the parser parses Sandbox URL and Open TFTP reports, both filling the log (debug level) with notifications of "missing keys" or "optional keys not found" because the parser assumes the feed to be Vulnerable ISAKMP. And before this occurs, the parser has been running for a good while.
We've picked up the the problematic Sandbox URL and Open TFTP reports above and made unit test cases of these - all tests are correctly parsed, the feed name is deduced correctly and the tests are successful, so the parser handles these correctly if these are the first reports parsed. Therefore, our assumption is that this problem occurs only when the parser has been running for a longer time and some state information about the feed type does not get cleared between parsing incoming reports.
Anyone else experiencing similar problems? This is a tricky one to debug so I decided to ask on the list first.
Br, Mika
P.S: Our setup is intelmq 3.0.2 on an Ubuntu 20.04 LTS _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
-- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://sebix.at/ ZVR 1510673578 _______________________________________________ IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/
IntelMQ-dev mailing list https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev https://intelmq.readthedocs.io/