- IntelMQ-users

ShadowServer Parser bot feedname
by Peter Chan 02 Aug '18

02 Aug '18

Good day, I wanted to set up a test in intelmq, with the following pipe: Fetch Mail Attachement -> Parse Shadow Server csv -> output to file. The mail and file output bots are running fine, however I am getting the following error with the shadowserver parser bot: 2018-06-05 10:54:51,880 - ShadowServer-Parser - INFO - ShadowserverParserBot initialized with id ShadowServer-Parser and intelmq 1.0.3 and python 3.5.2 (default, Nov 23 2017, 16:37:01) as process 7223. 2018-06-05 10:54:51,881 - ShadowServer-Parser - INFO - Bot is starting. 2018-06-05 10:54:51,888 - ShadowServer-Parser - ERROR - No feedname provided or feedname not in conf. 2018-06-05 10:54:51,889 - ShadowServer-Parser - INFO - Bot stopped. The feedname parameter in my runtime.conf is definitely populated. I have tried defining different feed names, based on the report name from the shadowserver website, but still getting the same error. Any idea what I am missing? I've attached my runtime.conf and pipeline.conf for reference. Unfortunately I cannot share the shadowserver email as it contains device scans in my ASN. Regards, Peter

3 7

shadowserver parser
by Ole Kristoffer Dybvik Apeland 06 Jul '18

06 Jul '18

Hi We had an issue with shadowserver data today where those feeds that only provide a download link would yield a php script. I have spoken with someone connected to the project and they are aware of the problem. Is it possible to let the shadowserver parser drop the messages in these instances? In it's current form only a warning is given (line 92 in bots/parsers/shadowserver/parser.py) even though required_fields are missing in the data. best regards Ole Kristoffer Nkom EkomCERT

2 1

Shadowserver compromised website parser -ERROR
by Tomislav Protega 02 Jul '18

02 Jul '18

Hi, I ran into this error: Shadowserver-Compromised-Website-Parser - ERROR - Could not convert shadowkey: 'http_host', value: '' via conversion function 'validate_fqdn'. More detailed log is attached. This happens when "http_host" field in the shadowserver origin report contains IP instead of domain which is not something unusual. At the end IntelMQ does produce the output data, but there's no 'source.url' field which should contain merged 'http_host' and 'url' parameters from the origin report. Regards, -- Tomislav

2 2

reverse_dns-expert exception InvalidValue
by Tomislav Protega 02 Jul '18

02 Jul '18

Hi, here's another exception I ran into. If Reverse DNS expert bot runs into IP address which doesn't have DNS PTR record in fqdn form, then it gives exception. In attachment are logs. Example: There are cases when DNS gives two records for PTR. $ nslookup 5.157.80.221 Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: 221.80.157.5.in-addr.arpa name = 5.157.80.221. 221.80.157.5.in-addr.arpa name = aliancys.peopleinc.nl. -------------------------------------------------------- $ nslookup 121.201.38.118 Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: 118.38.201.121.in-addr.arpa name = 121.201.38.118. Regards, -- Tomislav

2 1

Question on Intelmq
by LIMITHA THOMAS 28 Jun '18

28 Jun '18

Hi , Can I integrate Intelmq with Sumologic to feed in data ? If yes , how do I do it? Thanks and regards, Limitha

2 1

intelmq 1.0.5 released
by Sebastian Wagner 22 Jun '18

22 Jun '18

Dear community, I just released the next maintenance release 1.0.5 for the 1.0.x series and it could be the last one. I will also release a RC for 1.1.0 very soon. The pre-build deb and rpm packages will be available soon (in some hours). Installation instructions: https://github.com/certtools/intelmq/blob/1.0.5/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.5/docs/UPGRADING.md This is the changelog: # Core - `lib/message`: `Report()` can now create a Report instance from Event instances (#1225). - `lib/bot`: * The first word in the log line `Processed ... messages since last logging.` is now adaptible and set to `Forwarded` in the existing filtering bots (#1237). * Kills oneself again after proper shutdown if the bot is XMPP collector or output (#970). Previously these two bots needed two stop commands to get actually stopped. - `lib/utils`: log: set the name of the `py.warnings` logger to the bot name (#1184). # Bots ## Collectors - `bots.collectors.mail.collector_mail_url`: handle empty downloaded reports (#988). - `bots.collectos.file.collector_file`: handle empty files (#1244). ## Parsers - Shadowserver parser: * SSL FREAK: Remove optional column `device_serial` and add several new ones. * Fixed HTTP URL parsing for multiple feeds (#1243). - Spamhaus CERT parser: * add support for `smtpauth`, `l_spamlink`, `pop`, `imap`, `rdp`, `smb`, `iotscan`, `proxyget`, `iotmicrosoftds`, `automatedtest`, `ioturl`, `iotmirai`, `iotcmd`, `iotlogin` and `iotuser` (#1254). * fix `extra.destination.local_port` -> `extra.source.local_port`. ## Experts - `bots.experts.filter`: Pre-compile regex at bot initialization. # Tests - Ensure that the bots did process all messages (#291). # Tools - `intelmqctl`: * `intelmqctl run` has a new parameter `-l` `--loglevel` to overwrite the log level for the run (#1075). * `intelmqctl run [bot-id] mesage send` can now send report messages (#1077). - `intelmqdump`: * has now command completion for bot names, actions and queue names in interacive console. * automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (#1225). * is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (#1256). * truncated messages are no longer used/saved to the file after being shown (#1255). * now again denies recovery of dumps if the corresponding bot is running. The check was broken (#1258). * now sorts the dump by the time of the dump. Previously, the list was in random order (#1020). # Known issues no known issues Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 1.0.5 release schedule
by Chris Horsley 14 Jun '18

14 Jun '18

Hi, There's currently a few very nice features and bots in the develop branch we'd like to use, and we're trying to work out our local upgrade schedule. Is there any current ballpark date anticipated for the official 1.0.5 release? Many thanks, Chris ----- Chris Horsley CSIRT Foundry http://csirtfoundry.com chris.horsley(a)csirtfoundry.com +61 402 646 653 Skype: chrishorsley

2 2

New OS versions as build targets for native packages
by Sebastian Wagner 07 May '18

07 May '18

Hi all, Some of the recent changes of the build targets for deb/rpm packages: New targets: * Ubuntu 18.04 * Fedora 27 * openSUSE Leap 15.0 Removed targets: * Ubuntu 17.04/17.10 * Fedora 25 * openSUSE 42.2 So this is the full list of supported OSs for native packages currently: * CentOS 7 * RHEL 7 * Debian 8 and 9 * Fedora 26, 27, 28 and Rawhide * openSUSE Leap 42.3, 15.0 and Tumbleweed * Ubuntu 16.04, 17.10 and 18.04 The unstable repository additionally has: * Debian Testing * Fedora Rawhide Docs are here: https://github.com/certtools/intelmq/blob/maintenance/docs/INSTALL.md#nativ… Please let me know your feedback any time! Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq-manager release 1.0.0 & 1.0.1
by Sebastian Wagner 23 Apr '18

23 Apr '18

Dear community, I just released a new stable release 1.0.0 and maintenance release 1.0.1 of intelmq-manager. Thanks to Edvard Reijthar, the project is very active again and many changes can be expected for the next versions too! Version 1.0.0 fixes some bugs and adds an interface for `intelmqctl check`. 1.0.1 fixes the version number in the code (shown in the about page). Installation instructions: https://github.com/certtools/intelmq-manager/blob/1.0.1/docs/INSTALL.md The version is compatible with intelmq >= 1.0.3 Combined changelog: ### Backend * Set content type correctly for JSON data in configuration loading (#112) * Fix version number. ### Pages * All pages are now deliverd by php, reducing the amount of duplicate code drastically. #### Landing page * Added a new block for the new check page, changed the formatting a bit. #### Configuration * Fixed handling of special parameter `run_mode` (#150) * Intelmqctl controller may be set via an env variable `INTELMQ_MANGER_CONTROLER_CMD` #### Check * Added, showing the output of `intelmqctl check` (#118). ### Documentation * Note on header Content-Security-Policy (#113) * Note on security considerations in Readme to avoid misunderstandings * Remove compatibility warning from README ### Third-party libraries * reverted update jQuery to 3.2.1 * reverted update metisMenu to 2.7.0 ### Licenses * Licenses of used and included software is now inventarized and properly declared in LICENSES/ (#134) ### Packaging * fix packaging of positions.conf file for deb-packages (#133). ### Known issues * Missing CSRF protection (#111). * Missing copyright notices (#140). * Graph jumps around on "Add edge" bug component (#148). * new runtime parameters with _ not possible (#153). * wrong error message for new bots with existing ID (#152). Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq release 1.0.4
by Sebastian Wagner 20 Apr '18

20 Apr '18

Dear community, I just released a new maintenance release 1.0.4 of intelmq. It only fixes bugs in the 1.0.x series and may be the last version of the 1.0.x series. Installation instructions: https://github.com/certtools/intelmq/blob/1.0.4/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.4/docs/UPGRADING.md The changes are: - make code style compatible to pycodestyle 2.4.0 - fixed permissions of some files (they were executable but shouldn't be) ### Core - lib/harmonization: * FQDN validation now handles None correctly (raised an Exception). * Fixed several sanitize() methods, the generic sanitation method were called by is_valid, not the sanitize methods (#1219). ### Harmonization ### Bots * Use the new pypi website at https://pypi.org/ everywhere. #### Parsers - Shadowserver parser: * The fields `url` and `http_url` now handle HTTP URL paths and HTTP requests for all feeds (#1204). * The conversion function `validate_fqdn` now handles empty strings correctly. * Feed 'drone (hadoop)': * Correct validation of field `cc_dns`, will now only be added as `destination.fqdn` if correct FQDN, otherwise ignored. Previously this field could be saved in extra containing an IP address. * Adding more mappings for added columns. * A lot of newly added fields and fixed conversions. * Add newly added columns of `Ssl-Scan` feed to parser - Spamhaus CERT parser: * fix parsing and classification for bot names 'openrelay', 'iotrdp', 'sshauth', 'telnetauth', 'iotcmd', 'iotuser', 'wpscanner', 'w_wplogin', 'iotscan' see the NEWS file - Postgresql section - for all changes. - CleanMX phishing parser: handle FQDNs in IP column (#1162). #### Experts - `bots.experts.ripencc_abuse_contact`: Add existing parameter `mode` to BOTS file. ### Tools - intelmqctl check: Fixed and extended message for 'run_mode' check. - `intelmqctl start` botnet. When using `--type json`, no non-json information about wrong bots are output because that would confuse eg. intelmq-manager ### Tests - lib/bot: No dumps will be written during tests (#934). - lib/test: Expand regular expression on python version to match pre-releases (debian testing). ### Packaging * Static data is now included in source tarballs, development files are excluded Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users