Re: [IntelMQ-users] Custom harmonization configuration in tests

Hi, I'll try to prepare something.

Sincerely, Václav Brůžek

On Wed, 28 Feb 2024 at 09:39, Kamil Mankowski mankowski@cert.at wrote:

True, you could also - if you had time - propose a change in the base BotTestCase class to allow overriding the harmonization loader without subclassing. It was my plan, but just don't have time for that at the moment.

Best regards

// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 2/28/24 09:31, Vaclav Bruzek wrote:

...

Hi, thanks for the suggestions. So there are two options

1. overwrite the file in intelmq/etc/.
2. merge the loaded harmonization in the specific test cases with the

custom event fields.

This might be a good addition to the documentation.

Sincerely, Václav Brůžek

On Wed, 28 Feb 2024 at 09:04, Kamil Mankowski <mankowski@cert.at mailto:mankowski@cert.at> wrote:

Hi,

I overcome such a problem by creating a custom test base, look at

those

...

snippets as an example:

import intelmq.lib.test as test

ADDITIONAL_HARMONIZATION =
["contrib/constituency.harmonization.part.json"]

def load_harmonization():
      harmonization = pkg_resources.resource_filename(
          "intelmq_extensions", "etc/harmonization.conf"
      )
      extensions = []
      for file in ADDITIONAL_HARMONIZATION:
          with open(pathlib.Path(__file__).parent.parent.parent /
file) as f:
              extensions.append(json.load(f))
      return merge_harmonization(extensions, harmonization)

class TestCaseMixin:
      harmonization = load_harmonization()

class BotTestCase(TestCaseMixin, test.BotTestCase):
      """Provides test class with additional changes required for
extension bots"""

In my case, I'm merging the default harmonization with additional
fields, but you can just load your own.

Best regards

// Kamil Mańkowski <mankowski@cert.at <mailto:mankowski@cert.at>> -
T: +43 676 898 298 7204
// CERT Austria - https://www.cert.at/ <https://www.cert.at/>
// CERT.at GmbH, FB-Nr. 561772k, HG Wien

On 2/28/24 08:52, Sebix wrote:
 > Dear Václav,
 >
 > Which harmonization file did you change?
 >
 > The test library loads the harmonization config from the package
itself:
 >
 >

https://github.com/certtools/intelmq/blob/ac6aa4e306b1431b6db89158e25ed4c2c5... < https://github.com/certtools/intelmq/blob/ac6aa4e306b1431b6db89158e25ed4c2c5...

...

 >
 >      harmonization =
 >

utils.load_configuration(pkg_resources.resource_filename('intelmq',

...

 > 'etc/harmonization.conf'))
 >
 > best regards
 > Sebastian
 >
 > Institute for Common Good Technology
 > gemeinnütziger Kulturverein - nonprofit cultural society
 > https://commongoodtechnology.org/ <

https://commongoodtechnology.org/%3E

...

 > ZVR 1510673578
 >
 > On 2/28/24 08:39, Vaclav Bruzek wrote:
 >> Dear IntelMQ Developers & Users,
 >> I'm trying to incorporate my custom fields in Events and Reports

as

...

 >> well as test cases for my custom bots in unit tests. What I¨m

still

...

 >> encountering is that the Github actions are failing since my

custom

...

 >> fields in harmonization are not known. I've added step which

should

...

 >> create the harmonization file in /opt/intelmq/etc/ like so which

is

...

 >> the only change to the action:
 >>
 >>    - name: Include custom harmonization names
 >>       run: mkdir -p /opt/intelmq/etc/ && cp
 >> docker/fast/harmonization.conf /opt/intelmq/etc/
 >>
 >> However, intelmq still falls back to the default one. If I try
to run
 >>
manually load_configuration("/opt/intelmq/etc/harmonization.conf")

the

...

 >> correct result is displayed.
 >>
 >> Does anybody know how to supply the custom configuration to unit
tests?
 >>
 >> Sincerely,
 >> Václav Brůžek
 >>
 >