[IntelMQ-dev] RFC on IEP007: Running IntelMQ as Python Library

[Sebix]

Dear community,


tl;dr Your opinion on the programming interface matters! Please have a 
look and share your thoughts by the end of May, preferably before. Links 
below.


Thank you for contributing to the discussion around this proposal so far.


I want to take the liberty of summarising the previous discussion as 
follows:

  * the feature itself is welcome and not objected
  * the benefits are not clear to all
  * the existing programming interfaces (especially for bots) must not
    change

Whereas the focus of the former comments was on the nature of the 
proposal itself, we can dive deeper and discuss technical details, such 
as the programming interfaces, such as:

  * how to instantiate bots in "library mode" and to pass settings
    (parameters)
  * how to pass messages to the bot/source pipeline
  * how to receive resulting messages from the bot/destination pipeline

Below I added a few links with code examples.


We'd welcome your thoughts, especially on these topics.


In today's IntelMQ maintainer meeting, we planned to conclude this IEP, 
including its implementation, End of May with a new feature release of 
IntelMQ. So it would be great if we could collect all feedback before.


Programming examples in the current draft of IEP007 itself:

https://github.com/certtools/ieps/tree/iep-007/007/#user-content-examples

and you can also look at the current PoC/draft implementation:

https://github.com/certtools/intelmq/pull/2358/files

or an example use case:

https://github.com/Intevation/intelmq-webinput-csv/blob/f29c6922f3a41a1399b4dbb5f260d9519232831e/intelmq_webinput_csv/serve.py#L344


Best regards

Sebastian


P.S.: A bugfix release is envisaged for the end of next week

Institute for Common Good Technology
gemeinnütziger Kulturverein - nonprofit cultural society
https://sebix.at/
ZVR 1510673578

On 4/24/23 18:31, Sebix wrote:
> Dear community,
>
> I invite you to discuss a new (IEP - IntelMQ Enhancement Proposal)
>
> IEP007: Running IntelMQ as Python Library
>
> Have you ever wondered if you can write a Python script, call a bot's
> process method, pass it some data and get back the enriched/modified data?
> (pseudo code)
> bot_instance = Bot(parameters)
> bot_instance.process_message(input message) -> output messages
>
> Strictly speaking, it *is* actually possible with the current version,
> but it requires some bizarre hacks like re-defining Bot's methods and
> overwriting internal values.
> Staying on the wishlist for quite a while, we intend to implement the
> feature now. I started the IEP007 draft and need some input from you to
> maximize the benefit for all the IntelMQ community (developers):
> https://github.com/certtools/ieps/pull/7/files
> Or in a readable display:https://github.com/certtools/ieps/tree/iep-007/007
>
> What features and interfacing options would you expect when starting the
> bot as a library?
>
> Do you think the `Bot.process` method should be rewritten entirely now,
> removing the calls receive_message/send_message and converting the
> method into a generator (an API-breaking change)? And if yes, should
> this be done in one step, or separated from this bot-as-library feature,
> reducing the complexity of development steps?
>
> Looking forward for your ideas
> Sebastian
>
>
> _______________________________________________
> IntelMQ-dev mailing list
> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
> https://intelmq.readthedocs.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cert.at/pipermail/intelmq-dev/attachments/20230504/f181c423/attachment.htm>