Hi,
recently I noticed that reverse DNS expert bot doesn't correctly apply the reverse lookup results for IP. Meaning, the right value (result) is not applied for the right JSON event. It's like it's skipping it and then applies it to other event. There are no errors in log file for the bot.
For the illustration: Let say that hostname for IP 1.1.1.1 is "xx.yyy.zz", but instead the mentioned hostname becomes applied under wrong JSON event for the IP which in real has no PTR record in DNS.
Of course, there are events which have applied right PTR record for the IP, but in rare situations.
This case is not the issue with raw events which already contain hostname in origin feed.
Anyone notice such behavor, or could take a look at already processed data and see if the "source.reverse_dns" has right value applied against IP?
Regards,
Hi Tomislav,
That's very weird. My first question is of course: Which version of IntelMQ are you using? Do the logs of the bot indicate any DNS lookup errors?
Sebastian
On 12/12/2019 10.18, Tomislav Protega wrote:
Hi,
recently I noticed that reverse DNS expert bot doesn't correctly apply the reverse lookup results for IP. Meaning, the right value (result) is not applied for the right JSON event. It's like it's skipping it and then applies it to other event. There are no errors in log file for the bot.
For the illustration: Let say that hostname for IP 1.1.1.1 is "xx.yyy.zz", but instead the mentioned hostname becomes applied under wrong JSON event for the IP which in real has no PTR record in DNS.
Of course, there are events which have applied right PTR record for the IP, but in rare situations.
This case is not the issue with raw events which already contain hostname in origin feed.
Anyone notice such behavor, or could take a look at already processed data and see if the "source.reverse_dns" has right value applied against IP?
Regards,
Hi Sebastian,
we are still on older version of IntelMQ (1.1.0 - dev). As I said, there are no errors in logs during the DNS lookup.
When I noticed that, for example, for Shadowserver reports, I opened its .csv file and start to compare the lines from origin input and those one on the ouput, whether it was to a local file or ELK. If one line in .csv file doesn't have hostname value, the reverse DNS bot takes hostname value from other line within the same processed .csv file and applies to the one which doesn't have.
Regards,
-- Tomislav
On 12. 12. 2019. 15:25, Sebastian Wagner wrote:
Hi Tomislav,
That's very weird. My first question is of course: Which version of IntelMQ are you using? Do the logs of the bot indicate any DNS lookup errors?
Sebastian
On 12/12/2019 10.18, Tomislav Protega wrote:
Hi,
recently I noticed that reverse DNS expert bot doesn't correctly apply the reverse lookup results for IP. Meaning, the right value (result) is not applied for the right JSON event. It's like it's skipping it and then applies it to other event. There are no errors in log file for the bot.
For the illustration: Let say that hostname for IP 1.1.1.1 is "xx.yyy.zz", but instead the mentioned hostname becomes applied under wrong JSON event for the IP which in real has no PTR record in DNS.
Of course, there are events which have applied right PTR record for the IP, but in rare situations.
This case is not the issue with raw events which already contain hostname in origin feed.
Anyone notice such behavor, or could take a look at already processed data and see if the "source.reverse_dns" has right value applied against IP?
Regards,
Hi,
On 12/12/2019 16.18, Tomislav Protega wrote:
we are still on older version of IntelMQ (1.1.0 - dev).
Does the problem also exist in currently supported versions (2.1.x)?
I guess you already tried to restart the process? (also make (manually) sure that there are no other instances running of that bot)
As I said, there are no errors in logs during the DNS lookup.
When I noticed that, for example, for Shadowserver reports, I opened its .csv file and start to compare the lines from origin input and those one on the ouput, whether it was to a local file or ELK. If one line in .csv file doesn't have hostname value, the reverse DNS bot takes hostname value from other line within the same processed .csv file and applies to the one which doesn't have.
I will try to reproduce it, but only with the latest stable version.
Sebastian
Regards,
-- Tomislav
On 12. 12. 2019. 15:25, Sebastian Wagner wrote:
Hi Tomislav,
That's very weird. My first question is of course: Which version of IntelMQ are you using? Do the logs of the bot indicate any DNS lookup errors?
Sebastian
On 12/12/2019 10.18, Tomislav Protega wrote:
Hi,
recently I noticed that reverse DNS expert bot doesn't correctly apply the reverse lookup results for IP. Meaning, the right value (result) is not applied for the right JSON event. It's like it's skipping it and then applies it to other event. There are no errors in log file for the bot.
For the illustration: Let say that hostname for IP 1.1.1.1 is "xx.yyy.zz", but instead the mentioned hostname becomes applied under wrong JSON event for the IP which in real has no PTR record in DNS.
Of course, there are events which have applied right PTR record for the IP, but in rare situations.
This case is not the issue with raw events which already contain hostname in origin feed.
Anyone notice such behavor, or could take a look at already processed data and see if the "source.reverse_dns" has right value applied against IP?
Regards,
Hi,
Does the problem also exist in currently supported versions (2.1.x)?
We are still on old mentioned version. Beside this problem, we don't have any other issues with the old version in usage. The plan is to move on 2.1.x or 2.2.x dev version (due to use of custom bots) after we handle some hardware resource issues.
Just to point out this is not the issue with only Shadowserver reports.
Redis_cache_ttl is set to "0". Only invalid response are on "60".
For the info, here's the output of intelmq check:
$ intelmqctl check intelmqctl: Reading configuration files. intelmqctl: Checking defaults configuration. intelmqctl: Checking runtime configuration. intelmqctl: Checking runtime and pipeline configuration. intelmqctl: Checking harmonization configuration. intelmqctl: Checking for bots. intelmqctl: No issues found.
Regards,
-- Tomislav
On 13. 12. 2019. 15:58, Sebastian Wagner wrote:
Hi,
On 12/12/2019 16.18, Tomislav Protega wrote:
we are still on older version of IntelMQ (1.1.0 - dev).
Does the problem also exist in currently supported versions (2.1.x)?
I guess you already tried to restart the process? (also make (manually) sure that there are no other instances running of that bot)
As I said, there are no errors in logs during the DNS lookup.
When I noticed that, for example, for Shadowserver reports, I opened its .csv file and start to compare the lines from origin input and those one on the ouput, whether it was to a local file or ELK. If one line in .csv file doesn't have hostname value, the reverse DNS bot takes hostname value from other line within the same processed .csv file and applies to the one which doesn't have.
I will try to reproduce it, but only with the latest stable version.
Sebastian
Regards,
-- Tomislav
On 12. 12. 2019. 15:25, Sebastian Wagner wrote:
Hi Tomislav,
That's very weird. My first question is of course: Which version of IntelMQ are you using? Do the logs of the bot indicate any DNS lookup errors?
Sebastian
On 12/12/2019 10.18, Tomislav Protega wrote:
Hi,
recently I noticed that reverse DNS expert bot doesn't correctly apply the reverse lookup results for IP. Meaning, the right value (result) is not applied for the right JSON event. It's like it's skipping it and then applies it to other event. There are no errors in log file for the bot.
For the illustration: Let say that hostname for IP 1.1.1.1 is "xx.yyy.zz", but instead the mentioned hostname becomes applied under wrong JSON event for the IP which in real has no PTR record in DNS.
Of course, there are events which have applied right PTR record for the IP, but in rare situations.
This case is not the issue with raw events which already contain hostname in origin feed.
Anyone notice such behavor, or could take a look at already processed data and see if the "source.reverse_dns" has right value applied against IP?
Regards,
Hi Sebastian,
I will try to reproduce it, but only with the latest stable version.
Have you tried to do this?
Regards,
-- Tomislav
On 13. 12. 2019. 15:58, Sebastian Wagner wrote:
Hi,
On 12/12/2019 16.18, Tomislav Protega wrote:
we are still on older version of IntelMQ (1.1.0 - dev).
Does the problem also exist in currently supported versions (2.1.x)?
I guess you already tried to restart the process? (also make (manually) sure that there are no other instances running of that bot)
As I said, there are no errors in logs during the DNS lookup.
When I noticed that, for example, for Shadowserver reports, I opened its .csv file and start to compare the lines from origin input and those one on the ouput, whether it was to a local file or ELK. If one line in .csv file doesn't have hostname value, the reverse DNS bot takes hostname value from other line within the same processed .csv file and applies to the one which doesn't have.
I will try to reproduce it, but only with the latest stable version.
Sebastian
Regards,
-- Tomislav
On 12. 12. 2019. 15:25, Sebastian Wagner wrote:
Hi Tomislav,
That's very weird. My first question is of course: Which version of IntelMQ are you using? Do the logs of the bot indicate any DNS lookup errors?
Sebastian
On 12/12/2019 10.18, Tomislav Protega wrote:
Hi,
recently I noticed that reverse DNS expert bot doesn't correctly apply the reverse lookup results for IP. Meaning, the right value (result) is not applied for the right JSON event. It's like it's skipping it and then applies it to other event. There are no errors in log file for the bot.
For the illustration: Let say that hostname for IP 1.1.1.1 is "xx.yyy.zz", but instead the mentioned hostname becomes applied under wrong JSON event for the IP which in real has no PTR record in DNS.
Of course, there are events which have applied right PTR record for the IP, but in rare situations.
This case is not the issue with raw events which already contain hostname in origin feed.
Anyone notice such behavor, or could take a look at already processed data and see if the "source.reverse_dns" has right value applied against IP?
Regards,