Dear IntelMQ Developers & Users, I'm trying to incorporate my custom fields in Events and Reports as well as test cases for my custom bots in unit tests. What I¨m still encountering is that the Github actions are failing since my custom fields in harmonization are not known. I've added step which should create the harmonization file in /opt/intelmq/etc/ like so which is the only change to the action:
- name: Include custom harmonization names run: mkdir -p /opt/intelmq/etc/ && cp docker/fast/harmonization.conf /opt/intelmq/etc/
However, intelmq still falls back to the default one. If I try to run manually load_configuration("/opt/intelmq/etc/harmonization.conf") the correct result is displayed.
Does anybody know how to supply the custom configuration to unit tests?
Sincerely, Václav Brůžek
Dear Václav,
Which harmonization file did you change?
The test library loads the harmonization config from the package itself:
https://github.com/certtools/intelmq/blob/ac6aa4e306b1431b6db89158e25ed4c2c5...
harmonization = utils.load_configuration(pkg_resources.resource_filename('intelmq', 'etc/harmonization.conf'))
best regards Sebastian
Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578
On 2/28/24 08:39, Vaclav Bruzek wrote:
Dear IntelMQ Developers & Users, I'm trying to incorporate my custom fields in Events and Reports as well as test cases for my custom bots in unit tests. What I¨m still encountering is that the Github actions are failing since my custom fields in harmonization are not known. I've added step which should create the harmonization file in /opt/intelmq/etc/ like so which is the only change to the action:
- name: Include custom harmonization names run: mkdir -p /opt/intelmq/etc/ && cp docker/fast/harmonization.conf /opt/intelmq/etc/
However, intelmq still falls back to the default one. If I try to run manually load_configuration("/opt/intelmq/etc/harmonization.conf") the correct result is displayed.
Does anybody know how to supply the custom configuration to unit tests?
Sincerely, Václav Brůžek
Hi,
I overcome such a problem by creating a custom test base, look at those snippets as an example:
import intelmq.lib.test as test
ADDITIONAL_HARMONIZATION = ["contrib/constituency.harmonization.part.json"]
def load_harmonization(): harmonization = pkg_resources.resource_filename( "intelmq_extensions", "etc/harmonization.conf" ) extensions = [] for file in ADDITIONAL_HARMONIZATION: with open(pathlib.Path(__file__).parent.parent.parent / file) as f: extensions.append(json.load(f)) return merge_harmonization(extensions, harmonization)
class TestCaseMixin: harmonization = load_harmonization()
class BotTestCase(TestCaseMixin, test.BotTestCase): """Provides test class with additional changes required for extension bots"""
In my case, I'm merging the default harmonization with additional fields, but you can just load your own.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/28/24 08:52, Sebix wrote:
Dear Václav,
Which harmonization file did you change?
The test library loads the harmonization config from the package itself:
https://github.com/certtools/intelmq/blob/ac6aa4e306b1431b6db89158e25ed4c2c5...
harmonization = utils.load_configuration(pkg_resources.resource_filename('intelmq', 'etc/harmonization.conf'))
best regards Sebastian
Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578
On 2/28/24 08:39, Vaclav Bruzek wrote:
Dear IntelMQ Developers & Users, I'm trying to incorporate my custom fields in Events and Reports as well as test cases for my custom bots in unit tests. What I¨m still encountering is that the Github actions are failing since my custom fields in harmonization are not known. I've added step which should create the harmonization file in /opt/intelmq/etc/ like so which is the only change to the action:
- name: Include custom harmonization names run: mkdir -p /opt/intelmq/etc/ && cp docker/fast/harmonization.conf /opt/intelmq/etc/
However, intelmq still falls back to the default one. If I try to run manually load_configuration("/opt/intelmq/etc/harmonization.conf") the correct result is displayed.
Does anybody know how to supply the custom configuration to unit tests?
Sincerely, Václav Brůžek
Hi, thanks for the suggestions. So there are two options 1. overwrite the file in intelmq/etc/. 2. merge the loaded harmonization in the specific test cases with the custom event fields.
This might be a good addition to the documentation.
Sincerely, Václav Brůžek
On Wed, 28 Feb 2024 at 09:04, Kamil Mankowski mankowski@cert.at wrote:
Hi,
I overcome such a problem by creating a custom test base, look at those snippets as an example:
import intelmq.lib.test as test
ADDITIONAL_HARMONIZATION = ["contrib/constituency.harmonization.part.json"]
def load_harmonization(): harmonization = pkg_resources.resource_filename( "intelmq_extensions", "etc/harmonization.conf" ) extensions = [] for file in ADDITIONAL_HARMONIZATION: with open(pathlib.Path(__file__).parent.parent.parent / file) as f: extensions.append(json.load(f)) return merge_harmonization(extensions, harmonization)
class TestCaseMixin: harmonization = load_harmonization()
class BotTestCase(TestCaseMixin, test.BotTestCase): """Provides test class with additional changes required for extension bots"""
In my case, I'm merging the default harmonization with additional fields, but you can just load your own.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/28/24 08:52, Sebix wrote:
Dear Václav,
Which harmonization file did you change?
The test library loads the harmonization config from the package itself:
https://github.com/certtools/intelmq/blob/ac6aa4e306b1431b6db89158e25ed4c2c5...
harmonization =utils.load_configuration(pkg_resources.resource_filename('intelmq', 'etc/harmonization.conf'))
best regards Sebastian
Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578
On 2/28/24 08:39, Vaclav Bruzek wrote:
Dear IntelMQ Developers & Users, I'm trying to incorporate my custom fields in Events and Reports as well as test cases for my custom bots in unit tests. What I¨m still encountering is that the Github actions are failing since my custom fields in harmonization are not known. I've added step which should create the harmonization file in /opt/intelmq/etc/ like so which is the only change to the action:
- name: Include custom harmonization names run: mkdir -p /opt/intelmq/etc/ && cp
docker/fast/harmonization.conf /opt/intelmq/etc/
However, intelmq still falls back to the default one. If I try to run manually load_configuration("/opt/intelmq/etc/harmonization.conf") the correct result is displayed.
Does anybody know how to supply the custom configuration to unit tests?
Sincerely, Václav Brůžek
True, you could also - if you had time - propose a change in the base BotTestCase class to allow overriding the harmonization loader without subclassing. It was my plan, but just don't have time for that at the moment.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/28/24 09:31, Vaclav Bruzek wrote:
Hi, thanks for the suggestions. So there are two options
- overwrite the file in intelmq/etc/.
- merge the loaded harmonization in the specific test cases with the
custom event fields.
This might be a good addition to the documentation.
Sincerely, Václav Brůžek
On Wed, 28 Feb 2024 at 09:04, Kamil Mankowski <mankowski@cert.at mailto:mankowski@cert.at> wrote:
Hi, I overcome such a problem by creating a custom test base, look at those snippets as an example: import intelmq.lib.test as test ADDITIONAL_HARMONIZATION = ["contrib/constituency.harmonization.part.json"] def load_harmonization(): harmonization = pkg_resources.resource_filename( "intelmq_extensions", "etc/harmonization.conf" ) extensions = [] for file in ADDITIONAL_HARMONIZATION: with open(pathlib.Path(__file__).parent.parent.parent / file) as f: extensions.append(json.load(f)) return merge_harmonization(extensions, harmonization) class TestCaseMixin: harmonization = load_harmonization() class BotTestCase(TestCaseMixin, test.BotTestCase): """Provides test class with additional changes required for extension bots""" In my case, I'm merging the default harmonization with additional fields, but you can just load your own. Best regards // Kamil Mańkowski <mankowski@cert.at <mailto:mankowski@cert.at>> - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ <https://www.cert.at/> // CERT.at GmbH, FB-Nr. 561772k, HG Wien On 2/28/24 08:52, Sebix wrote: > Dear Václav, > > Which harmonization file did you change? > > The test library loads the harmonization config from the package itself: > > https://github.com/certtools/intelmq/blob/ac6aa4e306b1431b6db89158e25ed4c2c5a356bd/intelmq/lib/test.py#L190-L191 <https://github.com/certtools/intelmq/blob/ac6aa4e306b1431b6db89158e25ed4c2c5a356bd/intelmq/lib/test.py#L190-L191> > > harmonization = > utils.load_configuration(pkg_resources.resource_filename('intelmq', > 'etc/harmonization.conf')) > > best regards > Sebastian > > Institute for Common Good Technology > gemeinnütziger Kulturverein - nonprofit cultural society > https://commongoodtechnology.org/ <https://commongoodtechnology.org/> > ZVR 1510673578 > > On 2/28/24 08:39, Vaclav Bruzek wrote: >> Dear IntelMQ Developers & Users, >> I'm trying to incorporate my custom fields in Events and Reports as >> well as test cases for my custom bots in unit tests. What I¨m still >> encountering is that the Github actions are failing since my custom >> fields in harmonization are not known. I've added step which should >> create the harmonization file in /opt/intelmq/etc/ like so which is >> the only change to the action: >> >> - name: Include custom harmonization names >> run: mkdir -p /opt/intelmq/etc/ && cp >> docker/fast/harmonization.conf /opt/intelmq/etc/ >> >> However, intelmq still falls back to the default one. If I try to run >> manually load_configuration("/opt/intelmq/etc/harmonization.conf") the >> correct result is displayed. >> >> Does anybody know how to supply the custom configuration to unit tests? >> >> Sincerely, >> Václav Brůžek >> >
Hi, I'll try to prepare something.
Sincerely, Václav Brůžek
On Wed, 28 Feb 2024 at 09:39, Kamil Mankowski mankowski@cert.at wrote:
True, you could also - if you had time - propose a change in the base BotTestCase class to allow overriding the harmonization loader without subclassing. It was my plan, but just don't have time for that at the moment.
Best regards
// Kamil Mańkowski mankowski@cert.at - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ // CERT.at GmbH, FB-Nr. 561772k, HG Wien
On 2/28/24 09:31, Vaclav Bruzek wrote:
Hi, thanks for the suggestions. So there are two options
- overwrite the file in intelmq/etc/.
- merge the loaded harmonization in the specific test cases with the
custom event fields.
This might be a good addition to the documentation.
Sincerely, Václav Brůžek
On Wed, 28 Feb 2024 at 09:04, Kamil Mankowski <mankowski@cert.at mailto:mankowski@cert.at> wrote:
Hi, I overcome such a problem by creating a custom test base, look atthose
snippets as an example: import intelmq.lib.test as test ADDITIONAL_HARMONIZATION = ["contrib/constituency.harmonization.part.json"] def load_harmonization(): harmonization = pkg_resources.resource_filename( "intelmq_extensions", "etc/harmonization.conf" ) extensions = [] for file in ADDITIONAL_HARMONIZATION: with open(pathlib.Path(__file__).parent.parent.parent / file) as f: extensions.append(json.load(f)) return merge_harmonization(extensions, harmonization) class TestCaseMixin: harmonization = load_harmonization() class BotTestCase(TestCaseMixin, test.BotTestCase): """Provides test class with additional changes required for extension bots""" In my case, I'm merging the default harmonization with additional fields, but you can just load your own. Best regards // Kamil Mańkowski <mankowski@cert.at <mailto:mankowski@cert.at>> - T: +43 676 898 298 7204 // CERT Austria - https://www.cert.at/ <https://www.cert.at/> // CERT.at GmbH, FB-Nr. 561772k, HG Wien On 2/28/24 08:52, Sebix wrote: > Dear Václav, > > Which harmonization file did you change? > > The test library loads the harmonization config from the package itself: > >https://github.com/certtools/intelmq/blob/ac6aa4e306b1431b6db89158e25ed4c2c5... < https://github.com/certtools/intelmq/blob/ac6aa4e306b1431b6db89158e25ed4c2c5...
> > harmonization = >utils.load_configuration(pkg_resources.resource_filename('intelmq',
> 'etc/harmonization.conf')) > > best regards > Sebastian > > Institute for Common Good Technology > gemeinnütziger Kulturverein - nonprofit cultural society > https://commongoodtechnology.org/ <https://commongoodtechnology.org/%3E
> ZVR 1510673578 > > On 2/28/24 08:39, Vaclav Bruzek wrote: >> Dear IntelMQ Developers & Users, >> I'm trying to incorporate my custom fields in Events and Reportsas
>> well as test cases for my custom bots in unit tests. What I¨mstill
>> encountering is that the Github actions are failing since mycustom
>> fields in harmonization are not known. I've added step whichshould
>> create the harmonization file in /opt/intelmq/etc/ like so whichis
>> the only change to the action: >> >> - name: Include custom harmonization names >> run: mkdir -p /opt/intelmq/etc/ && cp >> docker/fast/harmonization.conf /opt/intelmq/etc/ >> >> However, intelmq still falls back to the default one. If I try to run >> manually load_configuration("/opt/intelmq/etc/harmonization.conf")the
>> correct result is displayed. >> >> Does anybody know how to supply the custom configuration to unit tests? >> >> Sincerely, >> Václav Brůžek >> >