<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
mso-fareast-language:EN-US;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FR" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thank you for taking the time to answer all my questions.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I've already learned a few things from reading the email that I’m going to apply.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">However, during my tests I had the impression that the messages were dropping when it didn't have the key. I'll look into the issue when I'll have more time in the coming weeks.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I will not hesitate to contact you again.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:FR">Guillaume GRANJON de LÉPINEY</span></b><span lang="EN-US" style="mso-fareast-language:FR"> |
</span><span style="mso-fareast-language:FR"><a href="mailto:ggranjon@excellium-services.be"><span lang="EN-US" style="color:#0563C1">ggranjon@excellium-services.be</span></a></span><span lang="EN-US" style="mso-fareast-language:FR"> | PGP Key ID:
</span><span style="mso-fareast-language:FR"><a href="https://pgp.circl.lu/pks/lookup?search=0xE2FD5ED1&fingerprint=on&op=index"><span lang="EN-US" style="color:#0563C1">0xE2FD5ED1</span></a></span><span lang="EN-US" style="mso-fareast-language:FR">
<br>
<b>CERT-XLM Incident Handler</b> @ </span><span style="mso-fareast-language:FR"><a href="https://excellium-services.com/"><span lang="EN-US" style="color:#0563C1">excellium-services.com</span></a></span><span lang="EN-US" style="mso-fareast-language:FR"><br>
<b>CERT-XLM</b> | </span><span style="mso-fareast-language:FR"><a href="mailto:cert@excellium-services.com"><span lang="EN-US" style="color:#0563C1">cert@excellium-services.com</span></a></span><span lang="EN-US" style="mso-fareast-language:FR"> | PGP Key ID:
</span><span style="mso-fareast-language:FR"><a href="http://pgp.circl.lu/pks/lookup?op=vindex&fingerprint=on&search=0x67B311E5D74E5AC0"><span lang="EN-US" style="color:#0563C1">0xD74E5AC0</span></a></span><span lang="EN-US" style="mso-fareast-language:FR">
<br>
</span><span lang="EN-US" style="mso-fareast-language:#2000">Excellium Services </span>
<span lang="EN-US" style="mso-fareast-language:FR">Belgium N.V.</span><span lang="EN-US" style="mso-fareast-language:#2000"> |
</span><span lang="EN-US" style="mso-fareast-language:FR">Orion Bldg, Belgicastraat 13, B-1930 Zaventem, Belgium<br>
Mobile: </span><span lang="EN-US" style="mso-fareast-language:#2000">+32 4 71 98 57 65</span><span lang="EN-US" style="mso-fareast-language:FR"><br>
Emergency: +352 262 039 64 708 | </span><span style="mso-fareast-language:FR"><a href="mailto:emergency@excellium-services.com"><span lang="EN-US" style="color:#0563C1">emergency@excellium-services.com</span></a></span><span lang="EN-US" style="mso-fareast-language:FR">
| PGP Key ID: </span><span style="mso-fareast-language:FR"><a href="https://excellium-services.com/assets/EMERGENCY_PKEY.asc"><span lang="EN-US" style="color:#0563C1">0x42662EFE</span></a></span><span lang="EN-US" style="mso-fareast-language:#2000"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="mso-fareast-language:FR">From:</span></b><span style="mso-fareast-language:FR"> Sebastian Wagner <wagner@cert.at>
<br>
<b>Sent:</b> vendredi 30 juillet 2021 09:42<br>
<b>To:</b> Guillaume GRANJON DE LEPINEY <ggranjon@excellium-services.be>; 'intelmq-users@lists.cert.at' <intelmq-users@lists.cert.at><br>
<b>Subject:</b> Re: [IntelMQ-users] [IntelMQ] Deduplication on an optional field<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Hi,<o:p></o:p></p>
<div>
<p class="MsoNormal">On 7/26/21 3:04 PM, Guillaume GRANJON DE LEPINEY wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:5.0pt;margin-right:36.0pt;margin-bottom:0cm;margin-left:36.0pt;margin-bottom:.0001pt">
<span lang="EN-US" style="mso-fareast-language:FR">I wonder if there is a simple way to use a Deduplicator bot on an optional field. Indeed, I noticed when I apply the deduplicator on an optional field that the null value must be entered in the redis because
all messages (except the first one) that do not contain the field are dropped. <o:p>
</o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:36.0pt;margin-bottom:0cm;margin-left:36.0pt;margin-bottom:.0001pt">
<span lang="EN-US">Is there a workaround please?<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:36.0pt;margin-bottom:0cm;margin-left:36.0pt;margin-bottom:.0001pt">
<span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:36.0pt;margin-bottom:5.0pt;margin-left:36.0pt">
<span lang="EN-US">I could work around this problem by adding two Sieve bots at the exit of the precedent bot that would jump the Deduplicator bot if the message doesn't have the field, but I don't find that to be optimal. Thus, I am open to any proposal that
could help me.<o:p></o:p></span></p>
</blockquote>
<p><span lang="EN-US">The message-hash method ignores any non-existing key: </span>
<a href="https://github.com/certtools/intelmq/blob/8a8107ec6b332e710626d056b2b0446ab976775f/intelmq/lib/message.py#L404-L405"><span lang="EN-US">https://github.com/certtools/intelmq/blob/8a8107ec6b332e710626d056b2b0446ab976775f/intelmq/lib/message.py#L404-L405</span></a><span lang="EN-US"><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="line-height:14.25pt;background:#1E1E1E"><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#C586C0;mso-fareast-language:FR">if</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#D4D4D4;mso-fareast-language:FR">
filter_type == </span><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#CE9178;mso-fareast-language:FR">"whitelist"</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#D4D4D4;mso-fareast-language:FR">
</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#569CD6;mso-fareast-language:FR">and</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#D4D4D4;mso-fareast-language:FR"> key
</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#569CD6;mso-fareast-language:FR">not</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#D4D4D4;mso-fareast-language:FR">
</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#569CD6;mso-fareast-language:FR">in</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#D4D4D4;mso-fareast-language:FR"> filter_keys:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="line-height:14.25pt;background:#1E1E1E"><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#D4D4D4;mso-fareast-language:FR">
</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#C586C0;mso-fareast-language:FR">continue</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Courier New";color:#D4D4D4;mso-fareast-language:FR"><o:p></o:p></span></p>
</div>
</div>
<p><span lang="EN-US">You could either filter these messages out just before the deduplicator, but I don't see a reason for
<i>two</i> sieve bots, one should be sufficient, plus using paths (see </span><a href="https://intelmq.readthedocs.io/en/latest/user/bots.html#sieve"><span lang="EN-US">https://intelmq.readthedocs.io/en/latest/user/bots.html#sieve</span></a><span lang="EN-US">).<o:p></o:p></span></p>
<p><span lang="EN-US">(btw: If someone tackles </span><a href="https://github.com/certtools/intelmq/issues/1250"><span lang="EN-US">https://github.com/certtools/intelmq/issues/1250</span></a><span lang="EN-US">, the simpler filter expert would also work)<o:p></o:p></span></p>
<p><span lang="EN-US">If that's not viable for you, then you'd need to adapt the deduplicator's code a bit, probably also introducing additional parameters. Using the Message.set_default_value is not possible either, as that would set a constant, leading to
the same behavior as you have now.<o:p></o:p></span></p>
<p><span lang="EN-US">I hope that helps a bit<o:p></o:p></span></p>
<p><span lang="EN-US">Sebastian<o:p></o:p></span></p>
<pre><span lang="EN-US">-- <o:p></o:p></span></pre>
<pre><span lang="EN-US">// Sebastian Wagner </span><a href="mailto:wagner@cert.at"><span lang="EN-US"><wagner@cert.at></span></a><span lang="EN-US"> - T: +43 676 898 298 7201<o:p></o:p></span></pre>
<pre>// CERT Austria - <a href="https://www.cert.at/">https://www.cert.at/</a><o:p></o:p></pre>
<pre>// Eine Initiative der nic.at GmbH - <a href="https://www.nic.at/">https://www.nic.at/</a><o:p></o:p></pre>
<pre>// Firmenbuchnummer 172568b, LG Salzburg<o:p></o:p></pre>
</div>
This email is confidential and may contain legally privileged information. If you are not the intended recipient, you should not copy, distribute, disclose or use the information it contains, please e-mail the sender immediately and delete this message from
your system. Note: e-mails are susceptible to corruption, interception and unauthorised amendment; we do not accept liability for any such changes, or for their consequences. You should be aware that we may monitor your e-mails and their content. Excellium
Services SA.
</body>
</html>