[IntelMQ-dev] Advice for setting up tests for bot chains

[Birger Schacht]

Hi,

On 2/19/21 12:32 PM, L. Aaron Kaplan wrote:
> Hi Mika,
> 
> this is an old idea which I discussed with Sebastian in ~ 2014 (?) or 2015 already.
> Somehow it never got implemented. But I agree, it's really important to test correctness.

Oh, cool, did you take any notes back then? Maybe we can use this to 
integrate it into the existing setup.

> 
> I think what you would need to do is the following:
> 
> 1) define a pipeline which has as collector some file collectors.
> The input shall come via a file
> 2) define the run parameters for this
> 3) define filters (filter out certain fields from the events before storing)
> 4) store the results of the flow into a known output file (file output bot)
> 5) compare these output file with known good output files.
> 6) return error on diff

Well, but that would leave out any tests of collectors and outputs that 
are not file based, wouldn't it? I guess for some collectors it 
shouldn't be too hard to create mockups (i.e. for web APIs, we can 
simply spin up a webserver and serve static files) and I think its the 
same with some of the outputs (write to some SQL database, do an 
sqldump, compare the result).


> I would say we could call this a "unit-test-flow"

I would call them end-to-end tests
https://www.tutorialspoint.com/software_testing_dictionary/end_to_end_testing.htm


cheers,
Birger

> I think this could be easily implemented as a script / docker image which gets deployed via CI/CD.
> 
> Gitlab supports this approach. I guess gitHUB supports it as well.
> 
> How about writing a small proposal for this together?
> Then we can put this proposal in an issue and see that it gets implemented.
> 
> 
> Best,
> a.
> 
> 
>> On 19.02.2021, at 12:25, Mika Silander <mika.silander at csc.fi> wrote:
>>
>> Hi,
>>
>> While writing tests to individual bots seems quite straightforward, what would be the recommended way for writing (unit?) tests for a chain of bots? Ideally, I'd like to feed in individual test events to the first bot and only check what is returned by the last bot (event or whatever the expected outcome should be).
>>
>> I suppose there are many bad and clumsy ways of doing this so that's why I air this question on the dev list.
>>
>> Thanks for all help in advance, Mika
>> _______________________________________________
>> IntelMQ-dev mailing list
>> IntelMQ-dev at lists.cert.at
>> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
> 
> 
> _______________________________________________
> IntelMQ-dev mailing list
> IntelMQ-dev at lists.cert.at
> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
> 

-- 
// Birger Schacht <schacht at cert.at>
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0x3A3C547D2D48D997.asc
Type: application/pgp-keys
Size: 5392 bytes
Desc: not available
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210219/d15e97c5/attachment.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20210219/d15e97c5/attachment.sig>