<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>Hi,<br>
</p>
<div class="moz-cite-prefix">On 8/6/21 9:34 AM, Guillaume GRANJON DE
LEPINEY wrote:<br>
</div>
<blockquote type="cite"
cite="mid:PA4PR10MB45443489DD9747254D3835188CF39@PA4PR10MB4544.EURPRD10.PROD.OUTLOOK.COM">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
mso-fareast-language:EN-US;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1"><span lang="EN-US"><o:p></o:p></span><span
lang="EN-US">Thank you for taking the time to answer all my
questions.
<o:p></o:p></span>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I've already learned a
few things from reading the email that I’m going to apply.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">However, during my tests
I had the impression that the messages were dropping when it
didn't have the key.</span></p>
</div>
</blockquote>
<p>Yeah, it depends on the other fields' values. If they are
identical, the events will get dropped. As the message-algorithm
just ignores non-existing fields.<br>
</p>
<p>Sebastian<br>
</p>
<blockquote type="cite"
cite="mid:PA4PR10MB45443489DD9747254D3835188CF39@PA4PR10MB4544.EURPRD10.PROD.OUTLOOK.COM">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">I'll look into the issue
when I'll have more time in the coming weeks.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I will not hesitate to
contact you again.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><b><span style="mso-fareast-language:FR"
lang="EN-US">Guillaume GRANJON de LÉPINEY</span></b><span
style="mso-fareast-language:FR" lang="EN-US"> |
</span><span style="mso-fareast-language:FR"><a
href="mailto:ggranjon@excellium-services.be"
moz-do-not-send="true"><span style="color:#0563C1"
lang="EN-US">ggranjon@excellium-services.be</span></a></span><span
style="mso-fareast-language:FR" lang="EN-US"> | PGP Key
ID:
</span><span style="mso-fareast-language:FR"><a
href="https://pgp.circl.lu/pks/lookup?search=0xE2FD5ED1&fingerprint=on&op=index"
moz-do-not-send="true"><span style="color:#0563C1"
lang="EN-US">0xE2FD5ED1</span></a></span><span
style="mso-fareast-language:FR" lang="EN-US">
<br>
<b>CERT-XLM Incident Handler</b> @ </span><span
style="mso-fareast-language:FR"><a
href="https://excellium-services.com/"
moz-do-not-send="true"><span style="color:#0563C1"
lang="EN-US">excellium-services.com</span></a></span><span
style="mso-fareast-language:FR" lang="EN-US"><br>
<b>CERT-XLM</b> | </span><span
style="mso-fareast-language:FR"><a
href="mailto:cert@excellium-services.com"
moz-do-not-send="true"><span style="color:#0563C1"
lang="EN-US">cert@excellium-services.com</span></a></span><span
style="mso-fareast-language:FR" lang="EN-US"> | PGP Key
ID:
</span><span style="mso-fareast-language:FR"><a
href="http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x67B311E5D74E5AC0"
moz-do-not-send="true"><span style="color:#0563C1"
lang="EN-US">0xD74E5AC0</span></a></span><span
style="mso-fareast-language:FR" lang="EN-US">
<br>
</span><span style="mso-fareast-language:#2000" lang="EN-US">Excellium
Services </span>
<span style="mso-fareast-language:FR" lang="EN-US">Belgium
N.V.</span><span style="mso-fareast-language:#2000"
lang="EN-US"> |
</span><span style="mso-fareast-language:FR" lang="EN-US">Orion
Bldg, Belgicastraat 13, B-1930 Zaventem, Belgium<br>
Mobile: </span><span style="mso-fareast-language:#2000"
lang="EN-US">+32 4 71 98 57 65</span><span
style="mso-fareast-language:FR" lang="EN-US"><br>
Emergency: +352 262 039 64 708 | </span><span
style="mso-fareast-language:FR"><a
href="mailto:emergency@excellium-services.com"
moz-do-not-send="true"><span style="color:#0563C1"
lang="EN-US">emergency@excellium-services.com</span></a></span><span
style="mso-fareast-language:FR" lang="EN-US"> | PGP Key
ID: </span><span style="mso-fareast-language:FR"><a
href="https://excellium-services.com/assets/EMERGENCY_PKEY.asc"
moz-do-not-send="true"><span style="color:#0563C1"
lang="EN-US">0x42662EFE</span></a></span><span
style="mso-fareast-language:#2000" lang="EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span
style="mso-fareast-language:FR">From:</span></b><span
style="mso-fareast-language:FR"> Sebastian Wagner
<a class="moz-txt-link-rfc2396E" href="mailto:wagner@cert.at"><wagner@cert.at></a>
<br>
<b>Sent:</b> vendredi 30 juillet 2021 09:42<br>
<b>To:</b> Guillaume GRANJON DE LEPINEY
<a class="moz-txt-link-rfc2396E" href="mailto:ggranjon@excellium-services.be"><ggranjon@excellium-services.be></a>;
'<a class="moz-txt-link-abbreviated" href="mailto:intelmq-users@lists.cert.at">intelmq-users@lists.cert.at</a>'
<a class="moz-txt-link-rfc2396E" href="mailto:intelmq-users@lists.cert.at"><intelmq-users@lists.cert.at></a><br>
<b>Subject:</b> Re: [IntelMQ-users] [IntelMQ]
Deduplication on an optional field<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Hi,<o:p></o:p></p>
<div>
<p class="MsoNormal">On 7/26/21 3:04 PM, Guillaume GRANJON DE
LEPINEY wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"
style="mso-margin-top-alt:5.0pt;margin-right:36.0pt;margin-bottom:0cm;margin-left:36.0pt;margin-bottom:.0001pt"><span
style="mso-fareast-language:FR" lang="EN-US">I wonder if
there is a simple way to use a Deduplicator bot on an
optional field. Indeed, I noticed when I apply the
deduplicator on an optional field that the null value must
be entered in the redis because all messages (except the
first one) that do not contain the field are dropped. <o:p>
</o:p></span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:0cm;margin-right:36.0pt;margin-bottom:0cm;margin-left:36.0pt;margin-bottom:.0001pt"><span
lang="EN-US">Is there a workaround please?<o:p></o:p></span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:0cm;margin-right:36.0pt;margin-bottom:0cm;margin-left:36.0pt;margin-bottom:.0001pt"><span
lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:0cm;margin-right:36.0pt;margin-bottom:5.0pt;margin-left:36.0pt"><span
lang="EN-US">I could work around this problem by adding
two Sieve bots at the exit of the precedent bot that would
jump the Deduplicator bot if the message doesn't have the
field, but I don't find that to be optimal. Thus, I am
open to any proposal that could help me.<o:p></o:p></span></p>
</blockquote>
<p><span lang="EN-US">The message-hash method ignores any
non-existing key: </span>
<a
href="https://github.com/certtools/intelmq/blob/8a8107ec6b332e710626d056b2b0446ab976775f/intelmq/lib/message.py#L404-L405"
moz-do-not-send="true"><span lang="EN-US">https://github.com/certtools/intelmq/blob/8a8107ec6b332e710626d056b2b0446ab976775f/intelmq/lib/message.py#L404-L405</span></a><span
lang="EN-US"><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"
style="line-height:14.25pt;background:#1E1E1E"><span
style="font-size:10.5pt;font-family:"Courier
New";color:#C586C0;mso-fareast-language:FR"
lang="EN-US">if</span><span
style="font-size:10.5pt;font-family:"Courier
New";color:#D4D4D4;mso-fareast-language:FR"
lang="EN-US"> filter_type == </span><span
style="font-size:10.5pt;font-family:"Courier
New";color:#CE9178;mso-fareast-language:FR"
lang="EN-US">"whitelist"</span><span
style="font-size:10.5pt;font-family:"Courier
New";color:#D4D4D4;mso-fareast-language:FR"
lang="EN-US">
</span><span
style="font-size:10.5pt;font-family:"Courier
New";color:#569CD6;mso-fareast-language:FR"
lang="EN-US">and</span><span
style="font-size:10.5pt;font-family:"Courier
New";color:#D4D4D4;mso-fareast-language:FR"
lang="EN-US"> key
</span><span
style="font-size:10.5pt;font-family:"Courier
New";color:#569CD6;mso-fareast-language:FR"
lang="EN-US">not</span><span
style="font-size:10.5pt;font-family:"Courier
New";color:#D4D4D4;mso-fareast-language:FR"
lang="EN-US">
</span><span
style="font-size:10.5pt;font-family:"Courier
New";color:#569CD6;mso-fareast-language:FR"
lang="EN-US">in</span><span
style="font-size:10.5pt;font-family:"Courier
New";color:#D4D4D4;mso-fareast-language:FR"
lang="EN-US"> filter_keys:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"
style="line-height:14.25pt;background:#1E1E1E"><span
style="font-size:10.5pt;font-family:"Courier
New";color:#D4D4D4;mso-fareast-language:FR"
lang="EN-US">
</span><span
style="font-size:10.5pt;font-family:"Courier
New";color:#C586C0;mso-fareast-language:FR"
lang="EN-US">continue</span><span
style="font-size:10.5pt;font-family:"Courier
New";color:#D4D4D4;mso-fareast-language:FR"
lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<p><span lang="EN-US">You could either filter these messages out
just before the deduplicator, but I don't see a reason for
<i>two</i> sieve bots, one should be sufficient, plus using
paths (see </span><a
href="https://intelmq.readthedocs.io/en/latest/user/bots.html#sieve"
moz-do-not-send="true"><span lang="EN-US">https://intelmq.readthedocs.io/en/latest/user/bots.html#sieve</span></a><span
lang="EN-US">).<o:p></o:p></span></p>
<p><span lang="EN-US">(btw: If someone tackles </span><a
href="https://github.com/certtools/intelmq/issues/1250"
moz-do-not-send="true"><span lang="EN-US">https://github.com/certtools/intelmq/issues/1250</span></a><span
lang="EN-US">, the simpler filter expert would also work)<o:p></o:p></span></p>
<p><span lang="EN-US">If that's not viable for you, then you'd
need to adapt the deduplicator's code a bit, probably also
introducing additional parameters. Using the
Message.set_default_value is not possible either, as that
would set a constant, leading to the same behavior as you
have now.<o:p></o:p></span></p>
<p><span lang="EN-US">I hope that helps a bit<o:p></o:p></span></p>
<p><span lang="EN-US">Sebastian<o:p></o:p></span></p>
<pre><span lang="EN-US">-- <o:p></o:p></span></pre>
<pre><span lang="EN-US">// Sebastian Wagner </span><a href="mailto:wagner@cert.at" moz-do-not-send="true"><span lang="EN-US"><wagner@cert.at></span></a><span lang="EN-US"> - T: +43 676 898 298 7201<o:p></o:p></span></pre>
<pre>// CERT Austria - <a href="https://www.cert.at/" moz-do-not-send="true">https://www.cert.at/</a><o:p></o:p></pre>
<pre>// Eine Initiative der nic.at GmbH - <a href="https://www.nic.at/" moz-do-not-send="true">https://www.nic.at/</a><o:p></o:p></pre>
<pre>// Firmenbuchnummer 172568b, LG Salzburg<o:p></o:p></pre>
</div>
This email is confidential and may contain legally privileged
information. If you are not the intended recipient, you should not
copy, distribute, disclose or use the information it contains,
please e-mail the sender immediately and delete this message from
your system. Note: e-mails are susceptible to corruption,
interception and unauthorised amendment; we do not accept
liability for any such changes, or for their consequences. You
should be aware that we may monitor your e-mails and their
content. Excellium Services SA.
</blockquote>
<pre class="moz-signature" cols="72">--
// Sebastian Wagner <a class="moz-txt-link-rfc2396E" href="mailto:wagner@cert.at"><wagner@cert.at></a> - T: +43 676 898 298 7201
// CERT Austria - <a class="moz-txt-link-freetext" href="https://www.cert.at/">https://www.cert.at/</a>
// Eine Initiative der nic.at GmbH - <a class="moz-txt-link-freetext" href="https://www.nic.at/">https://www.nic.at/</a>
// Firmenbuchnummer 172568b, LG Salzburg</pre>
</body>
</html>