IntelMQ-dev

intelmq-dev@lists.cert.at

6 participants
232 discussions

OSError: [Errno 16] Resource busy: '/opt/intelmq/var/log' by update the intelmq
by Salehi Ghamsari, Majid 01 Aug '18

01 Aug '18

Hi, i have strange problem to compile my intelmq dev enviroment. I have stopped intelmq and try to compile, but i get OSError: [Errno 16] Resource busy: '/opt/intelmq/var/log' Has anyone ever had that? Regards Majid git clone https://github.com/certtools/intelmq.git /opt/dev_intelmq cd opt/dev_intelmq/ /opt/dev_intelmq # pip3 install -e . Obtaining file:///opt/dev_intelmq Requirement already satisfied: dnspython>=1.11.1 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: psutil>=1.2.1 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: python-dateutil>=2.5 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: python-termstyle>=0.1.10 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: pytz>=2012c in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: redis>=2.10 in /usr/lib/python3.5/site-packages (from intelmq==1.2.0a1) Requirement already satisfied: six>=1.5 in /usr/lib/python3.5/site-packages (from python-dateutil>=2.5->intelmq==1.2.0a1) Requirement already satisfied: setuptools in /usr/lib/python3.5/site-packages (from python-termstyle>=0.1.10->intelmq==1.2.0a1) Installing collected packages: intelmq Found existing installation: intelmq 1.0.4 Uninstalling intelmq-1.0.4: Exception: Traceback (most recent call last): File "/usr/lib/python3.5/shutil.py", line 538, in move os.rename(src, real_dst) OSError: [Errno 16] Resource busy: '/opt/intelmq/var/log' -> '/tmp/pip-6vus5dcm-uninstall/opt/intelmq/var/log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/usr/lib/python3.5/site-packages/pip/commands/install.py", line 342, in run prefix=options.prefix_path, File "/usr/lib/python3.5/site-packages/pip/req/req_set.py", line 778, in install requirement.uninstall(auto_confirm=True) File "/usr/lib/python3.5/site-packages/pip/req/req_install.py", line 754, in uninstall paths_to_remove.remove(auto_confirm) File "/usr/lib/python3.5/site-packages/pip/req/req_uninstall.py", line 115, in remove renames(path, new_path) File "/usr/lib/python3.5/site-packages/pip/utils/__init__.py", line 267, in renames shutil.move(old, new) File "/usr/lib/python3.5/shutil.py", line 550, in move rmtree(src) File "/usr/lib/python3.5/shutil.py", line 478, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib/python3.5/shutil.py", line 476, in rmtree os.rmdir(path) OSError: [Errno 16] Resource busy: '/opt/intelmq/var/log' You are using pip version 9.0.1, however version 10.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. /opt/dev_intelmq #

2 2

using the intelmq twitter bot
by Salehi Ghamsari, Majid 17 Jul '18

17 Jul '18

Hi there, I try to get the CVEs from this twitter page https://twitter.com/CVEnew but i get an exception that “that page does not exist” >target_timelines: screen_names of twitter accounts to be followed What should I put for target_timelines parameter? I do not know what I am missing. In the documentation we have to put for each bot parameters some examples input for better understanding the parameters. https://github.com/certtools/intelmq/blob/develop/docs/Bots.md#twitter Thank you for your help Majid Salehi Ghamsari Digital Public Services (DPS) Fraunhofer Institute for Open Communication Systems Kaiserin-Augusta-Allee 31, 10589 Berlin, Germany Email: majid.salehi.ghamsari(a)fokus.fraunhofer.de<mailto:majid.salehi.ghamsari@fokus.fraunhofer.de> Office: +49 (0) 30 3463 7118 Skype: salehimajid WEB: fokus.fraunhofer.de "cve-twitter-collector": { "parameters": { "feed": "Twitter", "target_timelines": "https://twitter.com/CVEnew ", "tweet_count": 10, "timelimit": 60000, "include_rts": false, "exclude_replies": true, "follow_urls": "", "consumer_key": "myconsumer_key ", "consumer_secret": "myconsumer_secret ", "access_token_key": "myaccess_token_key ", "access_token_secret": "myaccess_token_secret " }, "name": "Twitter", "group": "Collector", "module": "intelmq.bots.collectors.twitter.collector_twitter", "description": "Collects tweets from target_timelines. Maximum number of tweet_count from each timeline and up to timelimit back in time.", "enabled": true, "run_mode": "continuous" } Traceback (most recent call last): File "/opt/dev_intelmq/intelmq/lib/bot.py", line 167, in start self.process() File "/opt/dev_intelmq/intelmq/bots/collectors/twitter/collector_twitter.py", line 94, in process exclude_replies=self.exclude_replies) File "/usr/local/lib/python3.6/dist-packages/twitter/api.py", line 810, in GetUserTimeline data = self._ParseAndCheckTwitter(resp.content.decode('utf-8')) File "/usr/local/lib/python3.6/dist-packages/twitter/api.py", line 4890, in _ParseAndCheckTwitter self._CheckForTwitterError(data) File "/usr/local/lib/python3.6/dist-packages/twitter/api.py", line 4910, in _CheckForTwitterError raise TwitterError(data['errors']) twitter.error.TwitterError: [{'code': 34, 'message': 'Sorry, that page does not exist.'}]

2 2

changing bot runtime parameter
by Salehi Ghamsari, Majid 11 Jul '18

11 Jul '18

Hello there, I'm trying to develop a simple pipeline with three bots. test-file-collector---> test-message-parser---> test-tcp-output I would like to put or change the IP runtime parameter of the TCP output bot "test-tcp-output" depends on Message that I get from file-collector in the runtime. What is the best method to do this in intelmq? Thank you for your help. Majid

3 4

testing tcpcollecor bot 1.1.0rc1
by Salehi Ghamsari, Majid 05 Jul '18

05 Jul '18

Hi, i try to test the tcpcollector bot, on sender site I send this message { "feed.accuracy":100.0, "feed.name":"FileCollector", "feed.url":"file://localhost/opt/intelmq/var/lib/bots/file-input/event.json", "time.observation":"2018-07-05T08:25:51+00:00", "extra.info":"TC_IMQ_Event_001", "extra.threat_level_id":"1", "raw":"eyJFdmVudCI6IHsiZGlzdHJpYnV0aW9uIjogIjEiLCAiaW5mbyI6ICJUQ19JTVFfRXZlbnRfMDAxIiwgInNoYXJpbmdfZ3JvdXBfaWQiOiAiMCIsICJ0aHJlYXRfbGV2ZWxfaWQiOiAiMSJ9fQ==" } On the other site the TcpColltector bot failed with this message. Bot has found a problem. Traceback (most recent call last): File "/opt/dev_intelmq/intelmq/lib/bot.py", line 167, in start self.process() File "/opt/dev_intelmq/intelmq/bots/collectors/tcp/collector.py", line 47, in process self.send_message(report) File "/opt/dev_intelmq/intelmq/lib/bot.py", line 816, in send_message super(CollectorBot, self).send_message(*messages, path=path) File "/opt/dev_intelmq/intelmq/lib/bot.py", line 797, in __add_report_fields report.add("feed.name", self.parameters.name) AttributeError: 'Parameters' object has no attribute 'name' What do I missing? Regards Majid

2 3

intelmq 1.0.5 released
by Sebastian Wagner 22 Jun '18

22 Jun '18

Dear community, I just released the next maintenance release 1.0.5 for the 1.0.x series and it could be the last one. I will also release a RC for 1.1.0 very soon. The pre-build deb and rpm packages will be available soon (in some hours). Installation instructions: https://github.com/certtools/intelmq/blob/1.0.5/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.5/docs/UPGRADING.md This is the changelog: # Core - `lib/message`: `Report()` can now create a Report instance from Event instances (#1225). - `lib/bot`: * The first word in the log line `Processed ... messages since last logging.` is now adaptible and set to `Forwarded` in the existing filtering bots (#1237). * Kills oneself again after proper shutdown if the bot is XMPP collector or output (#970). Previously these two bots needed two stop commands to get actually stopped. - `lib/utils`: log: set the name of the `py.warnings` logger to the bot name (#1184). # Bots ## Collectors - `bots.collectors.mail.collector_mail_url`: handle empty downloaded reports (#988). - `bots.collectos.file.collector_file`: handle empty files (#1244). ## Parsers - Shadowserver parser: * SSL FREAK: Remove optional column `device_serial` and add several new ones. * Fixed HTTP URL parsing for multiple feeds (#1243). - Spamhaus CERT parser: * add support for `smtpauth`, `l_spamlink`, `pop`, `imap`, `rdp`, `smb`, `iotscan`, `proxyget`, `iotmicrosoftds`, `automatedtest`, `ioturl`, `iotmirai`, `iotcmd`, `iotlogin` and `iotuser` (#1254). * fix `extra.destination.local_port` -> `extra.source.local_port`. ## Experts - `bots.experts.filter`: Pre-compile regex at bot initialization. # Tests - Ensure that the bots did process all messages (#291). # Tools - `intelmqctl`: * `intelmqctl run` has a new parameter `-l` `--loglevel` to overwrite the log level for the run (#1075). * `intelmqctl run [bot-id] mesage send` can now send report messages (#1077). - `intelmqdump`: * has now command completion for bot names, actions and queue names in interacive console. * automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (#1225). * is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (#1256). * truncated messages are no longer used/saved to the file after being shown (#1255). * now again denies recovery of dumps if the corresponding bot is running. The check was broken (#1258). * now sorts the dump by the time of the dump. Previously, the list was in random order (#1020). # Known issues no known issues Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

New OS versions as build targets for native packages
by Sebastian Wagner 07 May '18

07 May '18

Hi all, Some of the recent changes of the build targets for deb/rpm packages: New targets: * Ubuntu 18.04 * Fedora 27 * openSUSE Leap 15.0 Removed targets: * Ubuntu 17.04/17.10 * Fedora 25 * openSUSE 42.2 So this is the full list of supported OSs for native packages currently: * CentOS 7 * RHEL 7 * Debian 8 and 9 * Fedora 26, 27, 28 and Rawhide * openSUSE Leap 42.3, 15.0 and Tumbleweed * Ubuntu 16.04, 17.10 and 18.04 The unstable repository additionally has: * Debian Testing * Fedora Rawhide Docs are here: https://github.com/certtools/intelmq/blob/maintenance/docs/INSTALL.md#nativ… Please let me know your feedback any time! Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq-manager release 1.0.0 & 1.0.1
by Sebastian Wagner 23 Apr '18

23 Apr '18

Dear community, I just released a new stable release 1.0.0 and maintenance release 1.0.1 of intelmq-manager. Thanks to Edvard Reijthar, the project is very active again and many changes can be expected for the next versions too! Version 1.0.0 fixes some bugs and adds an interface for `intelmqctl check`. 1.0.1 fixes the version number in the code (shown in the about page). Installation instructions: https://github.com/certtools/intelmq-manager/blob/1.0.1/docs/INSTALL.md The version is compatible with intelmq >= 1.0.3 Combined changelog: ### Backend * Set content type correctly for JSON data in configuration loading (#112) * Fix version number. ### Pages * All pages are now deliverd by php, reducing the amount of duplicate code drastically. #### Landing page * Added a new block for the new check page, changed the formatting a bit. #### Configuration * Fixed handling of special parameter `run_mode` (#150) * Intelmqctl controller may be set via an env variable `INTELMQ_MANGER_CONTROLER_CMD` #### Check * Added, showing the output of `intelmqctl check` (#118). ### Documentation * Note on header Content-Security-Policy (#113) * Note on security considerations in Readme to avoid misunderstandings * Remove compatibility warning from README ### Third-party libraries * reverted update jQuery to 3.2.1 * reverted update metisMenu to 2.7.0 ### Licenses * Licenses of used and included software is now inventarized and properly declared in LICENSES/ (#134) ### Packaging * fix packaging of positions.conf file for deb-packages (#133). ### Known issues * Missing CSRF protection (#111). * Missing copyright notices (#140). * Graph jumps around on "Add edge" bug component (#148). * new runtime parameters with _ not possible (#153). * wrong error message for new bots with existing ID (#152). Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq release 1.0.4
by Sebastian Wagner 20 Apr '18

20 Apr '18

Dear community, I just released a new maintenance release 1.0.4 of intelmq. It only fixes bugs in the 1.0.x series and may be the last version of the 1.0.x series. Installation instructions: https://github.com/certtools/intelmq/blob/1.0.4/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.4/docs/UPGRADING.md The changes are: - make code style compatible to pycodestyle 2.4.0 - fixed permissions of some files (they were executable but shouldn't be) ### Core - lib/harmonization: * FQDN validation now handles None correctly (raised an Exception). * Fixed several sanitize() methods, the generic sanitation method were called by is_valid, not the sanitize methods (#1219). ### Harmonization ### Bots * Use the new pypi website at https://pypi.org/ everywhere. #### Parsers - Shadowserver parser: * The fields `url` and `http_url` now handle HTTP URL paths and HTTP requests for all feeds (#1204). * The conversion function `validate_fqdn` now handles empty strings correctly. * Feed 'drone (hadoop)': * Correct validation of field `cc_dns`, will now only be added as `destination.fqdn` if correct FQDN, otherwise ignored. Previously this field could be saved in extra containing an IP address. * Adding more mappings for added columns. * A lot of newly added fields and fixed conversions. * Add newly added columns of `Ssl-Scan` feed to parser - Spamhaus CERT parser: * fix parsing and classification for bot names 'openrelay', 'iotrdp', 'sshauth', 'telnetauth', 'iotcmd', 'iotuser', 'wpscanner', 'w_wplogin', 'iotscan' see the NEWS file - Postgresql section - for all changes. - CleanMX phishing parser: handle FQDNs in IP column (#1162). #### Experts - `bots.experts.ripencc_abuse_contact`: Add existing parameter `mode` to BOTS file. ### Tools - intelmqctl check: Fixed and extended message for 'run_mode' check. - `intelmqctl start` botnet. When using `--type json`, no non-json information about wrong bots are output because that would confuse eg. intelmq-manager ### Tests - lib/bot: No dumps will be written during tests (#934). - lib/test: Expand regular expression on python version to match pre-releases (debian testing). ### Packaging * Static data is now included in source tarballs, development files are excluded Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Classification of malware events
by Thomas Hungenberg 15 Mar '18

15 Mar '18

The current classification scheme for malware events in shadowserver/parser/config.py is: 'constant_fields': { 'classification.taxonomy': 'malicious code', 'classification.type': 'botnet drone', 'classification.identifier': 'botnet', }, The modify expert (if used) overwrites the classification.identifier with a malware name (either a "harmonized" name or the value of malware.name as default). Last year, we discussed dropping the term "botnet (drone)" and replace it by "infected system" (as not all malware infected systems are necessarily part of a botnet). The config.py in branch develop currently looks like: 'classification.taxonomy': 'malicious code', 'classification.type': 'botnet drone', 'classification.identifier': 'infected system', However, my intention was to set the *type* to 'infected system' and not the *identifier* (which will be overwritten by the modify expert). So I'd like to propose to change the classification scheme as follows: 'classification.taxonomy': 'malicious code', 'classification.type': 'infected system', 'classification.identifier': 'malware', # default name, will be overwritten by modify expert So the final classification of an event will look like: 'classification.taxonomy': 'malicious code', 'classification.type': 'infected system', 'classification.identifier': 'ramnit', Thoughts? Objections? - Thomas CERT-Bund Incident Response & Malware Analysis Team

3 9

IntelMQ 1.0.3 released
by Sebastian Wagner 05 Feb '18

05 Feb '18

Dear community I have just release a new bugfix release of IntelMQ. Installation instructions: https://github.com/certtools/intelmq/blob/1.0.3/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.0.3/docs/UPGRADING.md The released is published on PyPI, Github and the OpenBuildService (for rpm/deb packages). If you installed intelmq with a package manager, the new released will be installed automatically. Full changelog: ### Contrib * logrotate: use sudo for postrotate script * cron-jobs: use the scripts in the bots' directories and link them (#1056, #1142) ### Core - `lib.harmonization`: Handle idna encoding error in FQDN sanitation (#1175, #1176). - `lib.bot`: - Bots stop when redis gives the error "OOM command not allowed when used memory > 'maxmemory'." (#1138). - warnings of bots are catched by the logger (#1074, #1113). - Fixed exitcodes 0 for graceful shutdowns . - better handling of problems with pipeline and especially it's initialization (#1178). - All parsers using `ParserBot`'s methods now log the sum of successfully parsed and failed lines at the end of each run (#1161). ### Harmonization - Rule for harmonization keys is enforced (#1104, #1141). - New allowed values for `classification.type`: `tor` & `leak` (see n6 parser below ). ### Bots #### Collectors - `bots.collectors.mail.collector_mail_attach`: Support attachment file parsing for imbox versions newer than 0.9.5 (#1134). - `bots.outputs.smtp.output`: Fix STARTTLS, threw an exception (#1152, #1153). #### Parsers - All CSV parsers ignore NULL-bytes now, because the csv-library cannot handle it (#967, #1114). - `bots.experts.modify` default ruleset: changed conficker rule to catch more spellings. - `bots.parsers.shadowserver.parser`: Add Accessible Cisco Smart Install (#1122). - `bots.parsers.cleanmx.parser`: Handle new columns `first` and `last`, rewritten for XML feed. See NEWS.md for upgrade instructions (#1131, #1136, #1163). - `bots.parsers.n6.parser`: Fix classification mappings. See NEWS file for changes values (#738, #1127). ### Documentation - `Release.md` add release procedure documentation - `Bots.md`: fix example configuration for modify expert ### Tools - intelmqctl now exits with exit codes > 0 when errors happened or the operation was not successful. Also, the status operation exits with 1, if bots are stopped, but enabled. (#977, #1143) - `intelmctl check` checks for valid `run_mode` in runtime configuration (#1140). ### Tests - `tests.lib.test_pipeline`: Redis tests clear all queues before and after tests (#1086). - Repaired debian package build on travis (#1169). - Warnings are not allowed by default, an allowed count can be specified (#1129). - `tests.bots.experts.cymru_whois/abusix`: Skipped on travis because of ongoing problems. ### Packaging * cron jobs: fix paths of executables ### Known issues - `bots.collectors/outputs.xmpp` must be killed two times (#970). - When running bots with `intelmqctl run [bot-id]` the log level is always INFO (#1075). - `intelmqctl run [bot-id] message send [msg]` does only support Events, not Reports (#1077). - `python3 setup.py sdist` does not include static files in the resulting tarballs (#1146). - `bots.parsers.cleanmx.parser`: The cleanMX feed may have FQDNs as IPs in rare cases, such lines are dumped (#1162). Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

← Newer
1
...
12
13
14
15
16
17
18
...
24
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev