Hi,
While writing tests to individual bots seems quite straightforward, what would be the recommended way for writing (unit?) tests for a chain of bots? Ideally, I'd like to feed in individual test events to the first bot and only check what is returned by the last bot (event or whatever the expected outcome should be).
I suppose there are many bad and clumsy ways of doing this so that's why I air this question on the dev list.
Thanks for all help in advance, Mika
Hi Mika,
this is an old idea which I discussed with Sebastian in ~ 2014 (?) or 2015 already. Somehow it never got implemented. But I agree, it's really important to test correctness.
I think what you would need to do is the following:
1) define a pipeline which has as collector some file collectors. The input shall come via a file 2) define the run parameters for this 3) define filters (filter out certain fields from the events before storing) 4) store the results of the flow into a known output file (file output bot) 5) compare these output file with known good output files. 6) return error on diff
I would say we could call this a "unit-test-flow"
I think this could be easily implemented as a script / docker image which gets deployed via CI/CD.
Gitlab supports this approach. I guess gitHUB supports it as well.
How about writing a small proposal for this together? Then we can put this proposal in an issue and see that it gets implemented.
Best, a.
On 19.02.2021, at 12:25, Mika Silander mika.silander@csc.fi wrote:
Hi,
While writing tests to individual bots seems quite straightforward, what would be the recommended way for writing (unit?) tests for a chain of bots? Ideally, I'd like to feed in individual test events to the first bot and only check what is returned by the last bot (event or whatever the expected outcome should be).
I suppose there are many bad and clumsy ways of doing this so that's why I air this question on the dev list.
Thanks for all help in advance, Mika _______________________________________________ IntelMQ-dev mailing list IntelMQ-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
Hi Aaron,
Before you start something new, let me add that we already created a bunch of vagrant+ansible scripts to test that deb/rpm packages work, including the API and the Manager. As you propose, it already includes configuring and starting bots and checking if they are running. Initially it was meant as just a test, and it got bigger and more complete over the last months. Now even package upgrades (2.2 -> 2.3) are tested. All of that needs to be done locally as of now, but that is already sufficient for our purposes and probably it can be integrated in GitHub Actions.
The repository containing everything is currently located at https://github.com/schacht-certat/intelmq-vagrant You can find the tests as ansible scripts in ansible/tasks/ As the README states, it is still very young and alpha, but it works well. Any ideas how to proceed with it and any contributions are welcome.
Many thanks to Birger for creating all that! :)
best regards Sebastian
On 2/19/21 12:32 PM, L. Aaron Kaplan wrote:
Hi Mika,
this is an old idea which I discussed with Sebastian in ~ 2014 (?) or 2015 already. Somehow it never got implemented. But I agree, it's really important to test correctness.
I think what you would need to do is the following:
- define a pipeline which has as collector some file collectors.
The input shall come via a file 2) define the run parameters for this 3) define filters (filter out certain fields from the events before storing) 4) store the results of the flow into a known output file (file output bot) 5) compare these output file with known good output files. 6) return error on diff
I would say we could call this a "unit-test-flow"
I think this could be easily implemented as a script / docker image which gets deployed via CI/CD.
Gitlab supports this approach. I guess gitHUB supports it as well.
How about writing a small proposal for this together? Then we can put this proposal in an issue and see that it gets implemented.
Best, a.
On 19.02.2021, at 12:25, Mika Silander mika.silander@csc.fi wrote:
Hi,
While writing tests to individual bots seems quite straightforward, what would be the recommended way for writing (unit?) tests for a chain of bots? Ideally, I'd like to feed in individual test events to the first bot and only check what is returned by the last bot (event or whatever the expected outcome should be).
I suppose there are many bad and clumsy ways of doing this so that's why I air this question on the dev list.
Thanks for all help in advance, Mika _______________________________________________ IntelMQ-dev mailing list IntelMQ-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
IntelMQ-dev mailing list IntelMQ-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
On 19.02.2021, at 12:52, Sebastian Wagner wagner@cert.at wrote:
Hi Aaron,
Before you start something new, let me add that we already created a bunch of vagrant+ansible scripts to test that deb/rpm packages work, including the API and the Manager. As you propose, it already includes configuring and starting bots and checking if they are running. Initially it was meant as just a test, and it got bigger and more complete over the last months. Now even package upgrades (2.2 -> 2.3) are tested. All of that needs to be done locally as of now, but that is already sufficient for our purposes and probably it can be integrated in GitHub Actions.
Excellent! So that old idea is taking hold then?
Nice see.
I think if we combine that with GitHub CI/CD and with tutorials on "flows" then we made some good progress.
The repository containing everything is currently located at https://github.com/schacht-certat/intelmq-vagrant
ah, okay. yes, that's why I can't see it in the main repo :)
You can find the tests as ansible scripts in ansible/tasks/ As the README states, it is still very young and alpha, but it works well. Any ideas how to proceed with it and any contributions are welcome.
Many thanks to Birger for creating all that! :)
best regards Sebastian
On 2/19/21 12:32 PM, L. Aaron Kaplan wrote:
Hi Mika,
this is an old idea which I discussed with Sebastian in ~ 2014 (?) or 2015 already. Somehow it never got implemented. But I agree, it's really important to test correctness.
I think what you would need to do is the following:
- define a pipeline which has as collector some file collectors.
The input shall come via a file 2) define the run parameters for this 3) define filters (filter out certain fields from the events before storing) 4) store the results of the flow into a known output file (file output bot) 5) compare these output file with known good output files. 6) return error on diff
I would say we could call this a "unit-test-flow"
I think this could be easily implemented as a script / docker image which gets deployed via CI/CD.
Gitlab supports this approach. I guess gitHUB supports it as well.
How about writing a small proposal for this together? Then we can put this proposal in an issue and see that it gets implemented.
Best, a.
On 19.02.2021, at 12:25, Mika Silander mika.silander@csc.fi wrote:
Hi,
While writing tests to individual bots seems quite straightforward, what would be the recommended way for writing (unit?) tests for a chain of bots? Ideally, I'd like to feed in individual test events to the first bot and only check what is returned by the last bot (event or whatever the expected outcome should be).
I suppose there are many bad and clumsy ways of doing this so that's why I air this question on the dev list.
Thanks for all help in advance, Mika _______________________________________________ IntelMQ-dev mailing list
IntelMQ-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
IntelMQ-dev mailing list
IntelMQ-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
-- // Sebastian Wagner wagner@cert.at
- T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi.
Thanks for your quick answers. I wouldn't mind contributing to this but for the moment getting our own system up and running takes all the time. It should ease in a month or two. As a first impression, vagrant+ansible feels like a very heavy machinery for our needs, but there are probably good arguments backing that approach. I still have a fair amount of home work to do with familiarizing myself with intelmq's details.
Cheers, Mika
----- Original Message ----- From: "L. Aaron Kaplan" aaron@lo-res.org To: "Sebastian Wagner" wagner@cert.at Cc: "Mika Silander" mika.silander@csc.fi, "intelmq-dev" intelmq-dev@lists.cert.at Sent: Friday, 19 February, 2021 13:55:49 Subject: Re: [IntelMQ-dev] Advice for setting up tests for bot chains
On 19.02.2021, at 12:52, Sebastian Wagner wagner@cert.at wrote:
Hi Aaron,
Before you start something new, let me add that we already created a bunch of vagrant+ansible scripts to test that deb/rpm packages work, including the API and the Manager. As you propose, it already includes configuring and starting bots and checking if they are running. Initially it was meant as just a test, and it got bigger and more complete over the last months. Now even package upgrades (2.2 -> 2.3) are tested. All of that needs to be done locally as of now, but that is already sufficient for our purposes and probably it can be integrated in GitHub Actions.
Excellent! So that old idea is taking hold then?
Nice see.
I think if we combine that with GitHub CI/CD and with tutorials on "flows" then we made some good progress.
The repository containing everything is currently located at https://github.com/schacht-certat/intelmq-vagrant
ah, okay. yes, that's why I can't see it in the main repo :)
You can find the tests as ansible scripts in ansible/tasks/ As the README states, it is still very young and alpha, but it works well. Any ideas how to proceed with it and any contributions are welcome.
Many thanks to Birger for creating all that! :)
best regards Sebastian
On 2/19/21 12:32 PM, L. Aaron Kaplan wrote:
Hi Mika,
this is an old idea which I discussed with Sebastian in ~ 2014 (?) or 2015 already. Somehow it never got implemented. But I agree, it's really important to test correctness.
I think what you would need to do is the following:
- define a pipeline which has as collector some file collectors.
The input shall come via a file 2) define the run parameters for this 3) define filters (filter out certain fields from the events before storing) 4) store the results of the flow into a known output file (file output bot) 5) compare these output file with known good output files. 6) return error on diff
I would say we could call this a "unit-test-flow"
I think this could be easily implemented as a script / docker image which gets deployed via CI/CD.
Gitlab supports this approach. I guess gitHUB supports it as well.
How about writing a small proposal for this together? Then we can put this proposal in an issue and see that it gets implemented.
Best, a.
On 19.02.2021, at 12:25, Mika Silander mika.silander@csc.fi wrote:
Hi,
While writing tests to individual bots seems quite straightforward, what would be the recommended way for writing (unit?) tests for a chain of bots? Ideally, I'd like to feed in individual test events to the first bot and only check what is returned by the last bot (event or whatever the expected outcome should be).
I suppose there are many bad and clumsy ways of doing this so that's why I air this question on the dev list.
Thanks for all help in advance, Mika _______________________________________________ IntelMQ-dev mailing list
IntelMQ-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
IntelMQ-dev mailing list
IntelMQ-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
-- // Sebastian Wagner wagner@cert.at
- T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi,
On 2/19/21 12:32 PM, L. Aaron Kaplan wrote:
Hi Mika,
this is an old idea which I discussed with Sebastian in ~ 2014 (?) or 2015 already. Somehow it never got implemented. But I agree, it's really important to test correctness.
Oh, cool, did you take any notes back then? Maybe we can use this to integrate it into the existing setup.
I think what you would need to do is the following:
- define a pipeline which has as collector some file collectors.
The input shall come via a file 2) define the run parameters for this 3) define filters (filter out certain fields from the events before storing) 4) store the results of the flow into a known output file (file output bot) 5) compare these output file with known good output files. 6) return error on diff
Well, but that would leave out any tests of collectors and outputs that are not file based, wouldn't it? I guess for some collectors it shouldn't be too hard to create mockups (i.e. for web APIs, we can simply spin up a webserver and serve static files) and I think its the same with some of the outputs (write to some SQL database, do an sqldump, compare the result).
I would say we could call this a "unit-test-flow"
I would call them end-to-end tests https://www.tutorialspoint.com/software_testing_dictionary/end_to_end_testin...
cheers, Birger
I think this could be easily implemented as a script / docker image which gets deployed via CI/CD.
Gitlab supports this approach. I guess gitHUB supports it as well.
How about writing a small proposal for this together? Then we can put this proposal in an issue and see that it gets implemented.
Best, a.
On 19.02.2021, at 12:25, Mika Silander mika.silander@csc.fi wrote:
Hi,
While writing tests to individual bots seems quite straightforward, what would be the recommended way for writing (unit?) tests for a chain of bots? Ideally, I'd like to feed in individual test events to the first bot and only check what is returned by the last bot (event or whatever the expected outcome should be).
I suppose there are many bad and clumsy ways of doing this so that's why I air this question on the dev list.
Thanks for all help in advance, Mika _______________________________________________ IntelMQ-dev mailing list IntelMQ-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
IntelMQ-dev mailing list IntelMQ-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
On 19.02.2021, at 15:17, Birger Schacht schacht@cert.at wrote:
Signed PGP part Hi,
On 2/19/21 12:32 PM, L. Aaron Kaplan wrote:
Hi Mika, this is an old idea which I discussed with Sebastian in ~ 2014 (?) or 2015 already. Somehow it never got implemented. But I agree, it's really important to test correctness.
Oh, cool, did you take any notes back then? Maybe we can use this to integrate it into the existing setup.
A lot was discussed directly f2f (2 or three people) back then. So, it was for sure written down somewhere for the first CEF project. But I think it was never really implemented.
The idea back then was : we need end-to-end tests. So, basically we would like to see that the chain/flow of multiple bots also works, not just the individual unit tests for every bot.
I think what you would need to do is the following:
- define a pipeline which has as collector some file collectors.
The input shall come via a file 2) define the run parameters for this 3) define filters (filter out certain fields from the events before storing) 4) store the results of the flow into a known output file (file output bot) 5) compare these output file with known good output files. 6) return error on diff
Well, but that would leave out any tests of collectors and outputs that are not file based, wouldn't it?
That is true. You *could* mock some stuff, but I think that's overkill.
The point is: if we want to do a repeatable "flow"/integration test with a chain of bots, then basically we need to have well known, pre-determined input. And that's basically from a file-input bot. So, yes, with this approach I see no easy way to test other collectors. Do you have a better approach to test collectors except for unit tests?
IMHO the combination unit tests of collectors + a "flow"/integration test (with the collector replaced by the file-input bot) should be good enough.
The goal of the "flow"/integration test is to test if the bots pass on the right data and if the whole thing works together.
I guess for some collectors it shouldn't be too hard to create mockups (i.e. for web APIs, we can simply spin up a webserver and serve static files)
yes. But for others (n6 feed comes to mind) we would have to replicate a lot....basically we need to draw a border somewhere.
and I think its the same with some of the outputs (write to some SQL database, do an sqldump, compare the result).
I would say we could call this a "unit-test-flow"
I would call them end-to-end tests https://www.tutorialspoint.com/software_testing_dictionary/end_to_end_testin...
also ok. "integration test" also would fit :) A end-to-end test is a bit bigger for my feeling , but .. it's just terminology. Whatever we call it (I am not insisting on any name), let's just use the same word :)
Bye & have a nice weekend, a.
cheers, Birger
I think this could be easily implemented as a script / docker image which gets deployed via CI/CD. Gitlab supports this approach. I guess gitHUB supports it as well. How about writing a small proposal for this together? Then we can put this proposal in an issue and see that it gets implemented. Best, a.
On 19.02.2021, at 12:25, Mika Silander mika.silander@csc.fi wrote:
Hi,
While writing tests to individual bots seems quite straightforward, what would be the recommended way for writing (unit?) tests for a chain of bots? Ideally, I'd like to feed in individual test events to the first bot and only check what is returned by the last bot (event or whatever the expected outcome should be).
I suppose there are many bad and clumsy ways of doing this so that's why I air this question on the dev list.
Thanks for all help in advance, Mika _______________________________________________ IntelMQ-dev mailing list IntelMQ-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
IntelMQ-dev mailing list IntelMQ-dev@lists.cert.at https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
-- // Birger Schacht schacht@cert.at // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg <OpenPGP_0x3A3C547D2D48D997.asc>