././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.2052033 masakari_monitors-19.0.0/0000775000175000017500000000000000000000000015346 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/.coveragerc0000664000175000017500000000011500000000000017464 0ustar00zuulzuul00000000000000[run] branch = True source = masakarimonitors [report] ignore_errors = True ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/.mailmap0000664000175000017500000000013100000000000016762 0ustar00zuulzuul00000000000000# Format is: # # ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/.stestr.conf0000664000175000017500000000007600000000000017622 0ustar00zuulzuul00000000000000[DEFAULT] test_path=./masakarimonitors/tests/unit top_dir=./ ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/.yamllint0000664000175000017500000000010700000000000017176 0ustar00zuulzuul00000000000000--- extends: default ignore: | .tox/ rules: line-length: disable ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/.zuul.yaml0000664000175000017500000000107200000000000017307 0ustar00zuulzuul00000000000000--- - project: queue: masakari templates: - check-requirements - openstack-cover-jobs - openstack-python3-jobs - publish-openstack-docs-pti - release-notes-jobs-python3 - periodic-stable-jobs - periodic-jobs-with-oslo-master check: jobs: - openstack-tox-linters - masakari-functional-devstack-multinode - masakari-functional-devstack-ipv6 gate: jobs: - openstack-tox-linters - masakari-functional-devstack-multinode - masakari-functional-devstack-ipv6 ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591461.0 masakari_monitors-19.0.0/AUTHORS0000664000175000017500000000432600000000000016423 0ustar00zuulzuul00000000000000Andreas Jaeger Chuck Short Corey Bryant Dinesh Bhor Dmitriy Rabotyagov Doug Hellmann Ghanshyam Mann Hamid Lotfi Hervé Beraud Keiji Niwa Kengo Takahara Kengo Takahara Liam Young Louie KWAN Maksim Malchuk Mark Goddard Nguyen Hai Nguyen Hung Phuong OpenStack Release Bot Pavlo Shchelokovskyy Radosław Piliszek Sampath Priyankara Sean McGinnis Shilpa Devharakar Takahiro Izumi Takashi Kajinami Takashi NATSUME Takashi Natsume Thomas Bechtold Thomas Goirand Tushar Patil Vasyl Saienko XinxinShen YeHaiyang <6161910042@vip.jiangnan.edu.cn> avnish dengzhaosen dineshbhor ericxiett gugug huang.zhiping jacky06 jayashri bidwe liyingjun nitesh.vanarase openstack poojajadhav shilpa.devharakar songwenping sue sunjia suzhengwei suzhengwei tpatil wangqiangbj wu.shiming yangkun.lc zhaoleilc <15247232416@163.com> zhaoyixin ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/CONTRIBUTING.rst0000664000175000017500000000120300000000000020003 0ustar00zuulzuul00000000000000The source repository for this project can be found at: https://opendev.org/openstack/masakari-monitors Pull requests submitted through GitHub are not monitored. To start contributing to OpenStack, follow the steps in the contribution guide to set up and use Gerrit: https://docs.openstack.org/contributors/code-and-documentation/quick-start.html Bugs should be filed on Launchpad: https://bugs.launchpad.net/masakari-monitors For more specific information about contributing to this repository, see the masakari-monitors contributor guide: https://docs.openstack.org/masakari-monitors/latest/contributor/contributing.html ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591461.0 masakari_monitors-19.0.0/ChangeLog0000664000175000017500000001760600000000000017132 0ustar00zuulzuul00000000000000CHANGES ======= 19.0.0 ------ * If any of the consul agents such as manage, tenant, or warehouse are not set * Fix docs build * Update master for stable/2024.2 18.0.0 ------ * Update master for stable/2024.1 * reno: Update master for unmaintained/xena * reno: Update master for unmaintained/wallaby * reno: Update master for unmaintained/victoria 17.0.1 ------ * Honor libvirt.connection\_uri in introspectivemonitor * [introspectivemonitor] Fix syntax for python3 * Update python classifier in setup.cfg * Fix pep8 for new hacking * Honor test-requirements.txt for pep8 checks * Update master for stable/2023.2 16.0.0 ------ * Update master for stable/2023.1 15.0.0 ------ * not retry to send notification for specific http exception * Fix tox4 issues * [CI] Move queue setting to project level * Switch to 2023.1 Python3 unit tests and generic template name * Update master for stable/zed 14.0.0 ------ * Use daemon property instead of setDaemon method * Update python testing as per zed cycle testing runtime * Libvirt auth support * CI: Fix issue with absent stestr * Fix typo * Add Python3 zed unit tests * Update master for stable/yoga 13.0.0 ------ * Use LOG.warning instead of deprecated LOG.warn * connection too much when large scale failure * Updating python testing classifier as per Yoga testing runtime * host monitor by consul * Updating python testing classifier as per Yoga testing runtime * Fix a typo * Fix some typos * Fix home\_page in setup.cfg * Add note for package maintainers in requirements.txt * Add Python3 yoga unit tests * Update master for stable/xena 12.0.0 ------ * Fix hostmonitor to respect quorum * [CI] Gate on the same jobs as Masakari * [CI] Run cover jobs * Deprecate masakari-processmonitor * Use pre-provided libvirt-python * Use some better linting * Generate PDF documentation * [doc] Add config reference guide * move DriverBase to the base dir * [doc] Add masakari monitors usage * Remove conditionals for an ancient openstacksdk * Fix hostmonitor hanging forever after certain exceptions * Fix typos * Fix several code comment errors in process monitor * Replace "split(' ')" with "split()" in masakari-monitors * [community goal] Update contributor documentation * setup.cfg: Replace dashes with underscores * Fix one reno * Add Python3 xena unit tests * Update master for stable/wallaby 11.0.0 ------ * Repeated check to determine host status * [hostmonitor] Add pacemaker\_node\_type option * Drop CAP\_NET\_ADMIN * Drop unused samples * Revert "remove py37" * remove unused configration * remove unicode from code * remove unused script * remove py37 * Drop lower-constraints * Update TOX\_CONSTRAINTS\_FILE * [CI] Add periodic jobs * Add py38 package metadata * Add Python3 wallaby unit tests * Update master for stable/victoria 10.0.0 ------ * Fix tox for py38 and lower-contraints * [goal] Migrate testing to ubuntu focal * Add .stestr/ to .gitignore * Add missing keystoneauth1 to requirements.txt * Fix constraints URL enforcement for lower-constraints * Drop Babel from reqs * [goal] Migrate testing to ubuntu focal * Use keystoneauth1 config option loading for masakari client * Replace assertRaisesRegexp with assertRaisesRegex * Remove elementtree deprecated methods * Remove unnecessary continue statement * drop mock from lower-constraints * repeated parsing * Switch to newer openstackdocstheme and reno versions * Remove translation sections from setup.cfg * Remove six * Fix hacking min version to 3.0.1 * Add Python3 victoria unit tests * Update master for stable/ussuri * reset nova-compute process name 9.0.0 ----- * Use unittest.mock instead of third party mock * Cleanup py27 support * Update and replace http with https for doc links * Check config file for hostname * Update hacking for Python3 * Use hostname to avoid clash with section * [ussuri][goal] Drop python 2.7 support and testing * Update constraints path to preferred static location * Update master for stable/train * Use crm\_mon for pacemaker-remote deployments 8.0.0 ----- * Add Python 3 Train unit tests * add libvirt-python for libvirt package * Remove deprecated shell scripts * Switch to using stestr * Use template for lower-constraints * OpenDev Migration Patch * Dropping the py35 testing * Switch from oslosphinx to openstackdocstheme * Update master for stable/stein 7.0.0 ----- * Run all jobs by default using python3 * Add line for PyCharm IDE in gitignore file * Removed unnecessary parantheses in yield statements 7.0.0.0b1 --------- * Increment versioning with pbr instruction * import zuul job settings from project-config * fix tox python3 overrides * Pass region\_name and interface parameters during connection initialization * Update reno for stable/rocky 6.0.0 ----- * Remove python-openstackclient from global-requirements 6.0.0.0b3 --------- * add lower-constraints job * Fix stable branch releasenotes * Introspective Instance Monitoring through QEMU Guest Agent 6.0.0.0b2 --------- * Remove dependency on python-masakariclient * Update for new openstacksdk changes to masakari-monitors 6.0.0.0b1 --------- * Updated from global requirements * Updated from global requirements * Update for upcoming openstacksdk changes to masakari-monitors * Replaces yaml.load() with yaml.safe\_load() 5.0.0 ----- * Use status\_code instead of http\_status * fix misspelling of 'configuration' * Support specify connection uri for libvirt 4.0.0 ----- * Updated from global requirements * Remove an extra word in process\_list.yaml.sample * Use os-testr and add PY35 support * Add testcases of hostmonitor * Remove log translations * Pass correct parameters to '\_get\_connection' method * Fix typo of hostmonitor * Change the required olso.privsep version * Fix typo of processmonitor * Add testcases of ha and processmonitor * Add testcases of instancemonitor * Change the condition of success or failure of notifying * Fix syntax errors of README.rst * Remove data\_files definition from setup.cfg 3.0.1 ----- * Add README.rst * Add warning messages about deprecation to process/host monitor * Skip notification retry processing when HTTP status is 409 * Fix global name '\_' is not defined * Return user-friendly error message when monkey\_patch enabled * Add hacking check to ensure \_ is imported * Add implement of calling pre and post script * Add implement of sending notification * Add implement of preventing split-brain * Add implement of sending a notification * Add implement of comparing host status * Add implement of monitoring host * Add implement of restarting processes * Add implement of monitoring processes * Add unit test codes of instancemonitor * Add implement of loading hostmonitor driver * Add python hostmonitor only main process * Add common notification sending functions * Add initial start of processes * Loading the process list written in YAML * Add python processmonitor only main process * Add missing packages in requirements.txt * Using ServiceLauncher instead of ProcessLauncher * Change the section name of settings that related keystone * Update to match latest global-requirements * Add stop method to terminate child process * Fix missing translations for log messages * Remove conversion of 'retry\_interval' parameter * Remove unnecessary return and pass statements * Fix unexpected bash error raised by hostmonitor * Add the start scripts of processmonitor and hostmonitor * Refactor: Move domainEventRegisterAny method in a loop * Fix the regular expression for the check state of RA * Change service type and service name * Allow masakari-instancemonitor command anywhere * To allow processmonitor recognize proc.list which exists anywhere * Fix processmonitor's invalid messages * masakari-instancemonitor: fix incorrect module specification 2.0.0 ----- * Remove unnecessary methods * Add hostmonitor * Add processmonitor * Add implementation of notifying * Add implementation of handling events * Add instancemonitor only main process * Add a mechanism to use the oslo libraries * Initial Cookiecutter Commit * Added .gitreview ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/HACKING.rst0000664000175000017500000000111700000000000017144 0ustar00zuulzuul00000000000000masakari-monitors Style Commandments =============================================== Read the OpenStack Style Commandments http://docs.openstack.org/developer/hacking/ masakari-monitors Specific Commandments --------------------------------------- - [M301] Ensure that the _() function is explicitly imported to ensure proper translations. - [M302] Validate that log messages are not translated. - [M303] Yield must always be followed by a space when yielding a value. - [M304] Check for usage of deprecated assertRaisesRegexp - [M305] LOG.warn is deprecated. Enforce use of LOG.warning. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/LICENSE0000664000175000017500000002363700000000000016366 0ustar00zuulzuul00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/MANIFEST.in0000664000175000017500000000013600000000000017104 0ustar00zuulzuul00000000000000include AUTHORS include ChangeLog exclude .gitignore exclude .gitreview global-exclude *.pyc ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.2052033 masakari_monitors-19.0.0/PKG-INFO0000644000175000017500000000635200000000000016447 0ustar00zuulzuul00000000000000Metadata-Version: 2.1 Name: masakari-monitors Version: 19.0.0 Summary: Monitors for Masakari Home-page: https://docs.openstack.org/masakari-monitors/latest/ Author: OpenStack Author-email: openstack-dev@lists.openstack.org Classifier: Environment :: OpenStack Classifier: Intended Audience :: Information Technology Classifier: Intended Audience :: System Administrators Classifier: License :: OSI Approved :: Apache Software License Classifier: Operating System :: POSIX :: Linux Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: 3 :: Only Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.8 Classifier: Programming Language :: Python :: 3.9 Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Requires-Python: >=3.8 License-File: LICENSE Requires-Dist: automaton>=1.9.0 Requires-Dist: keystoneauth1>=3.4.0 Requires-Dist: openstacksdk>=0.13.0 Requires-Dist: oslo.concurrency>=3.26.0 Requires-Dist: oslo.config>=5.2.0 Requires-Dist: lxml>=4.5.0 Requires-Dist: oslo.cache>=1.26.0 Requires-Dist: oslo.i18n>=3.15.3 Requires-Dist: oslo.log>=3.36.0 Requires-Dist: oslo.middleware>=3.31.0 Requires-Dist: oslo.privsep>=1.23.0 Requires-Dist: oslo.service!=1.28.1,>=1.24.0 Requires-Dist: oslo.utils>=3.33.0 Requires-Dist: pbr!=2.1.0,>=2.0.0 Requires-Dist: python-consul>=1.1.0 =============================== masakari-monitors =============================== Monitors for Masakari ===================== Monitors for Masakari provides Virtual Machine High Availability (VMHA) service for OpenStack clouds by automatically detecting the failure events such as VM process down, provisioning process down, and nova-compute host failure. If it detect the events, it sends notifications to the masakari-api. Original version of Masakari: https://github.com/ntt-sic/masakari Tokyo Summit Session: https://www.youtube.com/watch?v=BmjNKceW_9A Monitors for Masakari is distributed under the terms of the Apache License, Version 2.0. The full terms and conditions of this license are detailed in the LICENSE file. * Free software: Apache license * Documentation: https://docs.openstack.org/masakari-monitors * Source: https://git.openstack.org/cgit/openstack/masakari-monitors * Bugs: https://bugs.launchpad.net/masakari-monitors Configure masakari-monitors --------------------------- #. Clone masakari using:: $ git clone https://github.com/openstack/masakari-monitors.git #. Create masakarimonitors directory in /etc/. #. Run setup.py from masakari-monitors:: $ sudo python setup.py install #. Copy masakarimonitors.conf and process_list.yaml files from masakari-monitors/etc/ to /etc/masakarimonitors folder and make necessary changes to the masakarimonitors.conf and process_list.yaml files. To generate the sample masakarimonitors.conf file, run the following command from the top level of the masakari-monitors directory:: $ tox -egenconfig #. To run masakari-processmonitor, masakari-hostmonitor and masakari-instancemonitor simply use following binary:: $ masakari-processmonitor $ masakari-hostmonitor $ masakari-instancemonitor Features -------- * TODO ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/README.rst0000664000175000017500000000344500000000000017043 0ustar00zuulzuul00000000000000=============================== masakari-monitors =============================== Monitors for Masakari ===================== Monitors for Masakari provides Virtual Machine High Availability (VMHA) service for OpenStack clouds by automatically detecting the failure events such as VM process down, provisioning process down, and nova-compute host failure. If it detect the events, it sends notifications to the masakari-api. Original version of Masakari: https://github.com/ntt-sic/masakari Tokyo Summit Session: https://www.youtube.com/watch?v=BmjNKceW_9A Monitors for Masakari is distributed under the terms of the Apache License, Version 2.0. The full terms and conditions of this license are detailed in the LICENSE file. * Free software: Apache license * Documentation: https://docs.openstack.org/masakari-monitors * Source: https://git.openstack.org/cgit/openstack/masakari-monitors * Bugs: https://bugs.launchpad.net/masakari-monitors Configure masakari-monitors --------------------------- #. Clone masakari using:: $ git clone https://github.com/openstack/masakari-monitors.git #. Create masakarimonitors directory in /etc/. #. Run setup.py from masakari-monitors:: $ sudo python setup.py install #. Copy masakarimonitors.conf and process_list.yaml files from masakari-monitors/etc/ to /etc/masakarimonitors folder and make necessary changes to the masakarimonitors.conf and process_list.yaml files. To generate the sample masakarimonitors.conf file, run the following command from the top level of the masakari-monitors directory:: $ tox -egenconfig #. To run masakari-processmonitor, masakari-hostmonitor and masakari-instancemonitor simply use following binary:: $ masakari-processmonitor $ masakari-hostmonitor $ masakari-instancemonitor Features -------- * TODO ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/bindep.txt0000664000175000017500000000171700000000000017356 0ustar00zuulzuul00000000000000# This is a cross-platform list tracking distribution packages needed for install and tests; # see https://docs.openstack.org/infra/bindep/ for additional information. # Due to the nature of libvirt-python package, in DevStack we use the one # provided in the distro alongside libvirtd - to ensure the two are compatible, # and also to avoid the pip error when it tries to uninstall the distro version # (installed in such a way for Nova in DevStack). # Do note libvirt-python is used only for instance-oriented monitors, so, e.g., # it is not used by any host monitor. # TODO(yoctozepto): Refactor code to not require libvirt-python for unit tests, # basically following how it is handled in nova-compute and ceilometer-compute. # libvirt-dev and pkg-config are required to compile libvirt-python package. libvirt-dev [platform:dpkg test] pkg-config [platform:dpkg test] libxml2-dev [platform:dpkg test] libxslt-devel [platform:rpm test] libxslt1-dev [platform:dpkg test] ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1772006 masakari_monitors-19.0.0/doc/0000775000175000017500000000000000000000000016113 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/requirements.txt0000664000175000017500000000047600000000000021406 0ustar00zuulzuul00000000000000# The order of packages is significant, because pip processes them in the order # of appearance. Changing the order has an impact on the overall integration # process, which may cause wedges in the gate later. sphinx>=2.0.0,!=2.1.0 # BSD openstackdocstheme>=2.2.1 # Apache-2.0 # releasenotes reno>=3.1.0 # Apache-2.0 ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1812007 masakari_monitors-19.0.0/doc/source/0000775000175000017500000000000000000000000017413 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1812007 masakari_monitors-19.0.0/doc/source/_static/0000775000175000017500000000000000000000000021041 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/_static/.gitkeep0000664000175000017500000000000000000000000022460 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/conf.py0000775000175000017500000000563300000000000020724 0ustar00zuulzuul00000000000000# -*- coding: utf-8 -*- # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os import sys sys.path.insert(0, os.path.abspath('../..')) # -- General configuration ---------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = [ 'sphinx.ext.autodoc', 'openstackdocstheme', 'oslo_config.sphinxconfiggen', 'oslo_config.sphinxext', ] # sphinxcontrib.apidoc options config_generator_config_file = ( '../../etc/masakarimonitors/masakarimonitors-config-generator.conf') sample_config_basename = '_static/masakarimonitors' # autodoc generation is a bit aggressive and a nuisance when doing heavy # text edit cycles. # execute "export SPHINX_DEBUG=1" in your terminal to disable # The suffix of source filenames. source_suffix = '.rst' # The master toctree document. master_doc = 'index' # General information about the project. project = 'masakari-monitors' copyright = '2016, OpenStack Foundation' # If true, '()' will be appended to :func: etc. cross-reference text. add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). add_module_names = True # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'native' # -- Options for HTML output -------------------------------------------------- # The theme to use for HTML and HTML Help pages. Major themes that come with # Sphinx are currently 'default' and 'sphinxdoc'. # html_theme_path = ["."] # html_theme = '_theme' # html_static_path = ['static'] html_theme = 'openstackdocs' openstackdocs_repo_name ='openstack/masakari-monitors' openstackdocs_bug_project = 'masakari-monitors' # Output file base name for HTML help builder. htmlhelp_basename = '%sdoc' % project # -- Options for LaTeX output ------------------------------------------------- # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass # [howto/manual]). latex_documents = [ ('index', 'doc-%s.tex' % project, '%s Documentation' % project, 'OpenStack Foundation', 'manual'), ] # Disable usage of xindy https://bugzilla.redhat.com/show_bug.cgi?id=1643664 latex_use_xindy = False # Disable smartquotes, they don't work in latex smartquotes_excludes = {'builders': ['latex']} ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/consul-usage.rst0000664000175000017500000000771300000000000022562 0ustar00zuulzuul00000000000000============= Consul Usage ============= Consul overview ================ Consul is a service mesh solution providing a full featured control plane with service discovery, configuration, and segmentation functionality. Each of these features can be used individually as needed, or they can be used together to build a full service mesh. The Consul agent is the core process of Consul. The Consul agent maintains membership information, registers services, runs checks, responds to queries, and more. Consul clients can provide any number of health checks, either associated with a given service or with the local node. This information can be used by an operator to monitor cluster health. Please refer to `Consul Agent Overview `_. Test Environment ================ There are three controller nodes and two compute nodes in the test environment. Every node has three network interfaces. The first interface is used for management, with an ip such as '192.168.101.*'. The second interface is used to connect to storage, with an ip such as '192.168.102.*'. The third interface is used for tenant, with an ip such as '192.168.103.*'. Download Consul ================ Download Consul package for CentOS. Other OS please refer to `Download Consul `_. .. code-block:: console sudo yum install -y yum-utils sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo sudo yum -y install Consul Configure Consul agent ====================== Consul agent must runs on every node. Consul server agent runs on controller nodes, while Consul client agent runs on compute nodes, which makes up one Consul cluster. The following is an example of a config file for Consul server agent which binds to management interface of the host. management.json .. code-block:: ini { "bind_addr": "192.168.101.1", "datacenter": "management", "data_dir": "/tmp/consul_m", "log_level": "INFO", "server": true, "bootstrap_expect": 3, "node_name": "node01", "addresses": { "http": "192.168.101.1" }, "ports": { "http": 8500, "serf_lan": 8501 }, "retry_join": ["192.168.101.1:8501", "192.168.101.2:8501", "192.168.101.3:8501"] } The following is an example of a config file for Consul client agent which binds to management interface of the host. management.json .. code-block:: ini { "bind_addr": "192.168.101.4", "datacenter": "management", "data_dir": "/tmp/consul_m", "log_level": "INFO", "node_name": "node04", "addresses": { "http": "192.168.101.4" }, "ports": { "http": 8500, "serf_lan": 8501 }, "retry_join": ["192.168.101.1:8501", "192.168.101.2:8501", "192.168.101.3:8501"] } Use the tenant or storage interface ip and ports when config agent in tenant or storage datacenter. Please refer to `Consul Agent Configuration `_. Start Consul agent ================== The Consul agent is started by the following command. .. code-block:: console # Consul agent –config-file management.json Test Consul installation ======================== After all Consul agents installed and started, you can see all nodes in the cluster by the following command. .. code-block:: console # Consul members -http-addr=192.168.101.1:8500 Node Address Status Type Build Protocol DC node01 192.168.101.1:8501 alive server 1.10.2 2 management node02 192.168.101.2:8501 alive server 1.10.2 2 management node03 192.168.101.3:8501 alive server 1.10.2 2 management node04 192.168.101.4:8501 alive client 1.10.2 2 management node05 192.168.101.5:8501 alive client 1.10.2 2 management ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1812007 masakari_monitors-19.0.0/doc/source/contributor/0000775000175000017500000000000000000000000021765 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/contributor/contributing.rst0000664000175000017500000000360600000000000025233 0ustar00zuulzuul00000000000000============================ So You Want to Contribute... ============================ For general information on contributing to OpenStack, please check out the `contributor guide `_ to get started. It covers all the basics that are common to all OpenStack projects: the accounts you need, the basics of interacting with our Gerrit review system, how we communicate as a community, etc. Below will cover the more project specific information you need to get started with masakari-monitors. Communication ~~~~~~~~~~~~~ * IRC channel #openstack-masakari at OFTC * Mailing list (prefix subjects with ``[masakari]`` for faster responses) http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss Contacting the Core Team ~~~~~~~~~~~~~~~~~~~~~~~~ Please refer the `masakari-monitors Core Team `_ contacts. New Feature Planning ~~~~~~~~~~~~~~~~~~~~ masakari-monitors features are tracked on `Launchpad `_. Task Tracking ~~~~~~~~~~~~~ We track our tasks in `Launchpad `_. If you're looking for some smaller, easier work item to pick up and get started on, search for the 'low-hanging-fruit' tag. Reporting a Bug ~~~~~~~~~~~~~~~ You found an issue and want to make sure we are aware of it? You can do so on `Launchpad `_. Getting Your Patch Merged ~~~~~~~~~~~~~~~~~~~~~~~~~ All changes proposed to the masakari-monitors project require one or two +2 votes from masakari-monitors core reviewers before one of the core reviewers can approve patch by giving ``Workflow +1`` vote. Project Team Lead Duties ~~~~~~~~~~~~~~~~~~~~~~~~ All common PTL duties are enumerated in the `PTL guide `_. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/hostmonitor.rst0000664000175000017500000001217500000000000022540 0ustar00zuulzuul00000000000000==================== masakari-hostmonitor ==================== Monitor Overview ------------------ The masakari-hostmonitor provides compute node High Availability for OpenStack clouds by automatically detecting compute nodes failure via monitor driver. How does it work based on pacemaker & corosync? ------------------------------------------------ - Pacemaker or pacemaker-remote is required to install into compute nodes to form a pacemaker cluster. - The compute node's status is depending on the heartbeat between the compute node and the cluster. Once the node lost the heartbeat, masakari-hostmonitor in other nodes will detect the failure and send notifications to masakari-api. How does it work based on consul? ---------------------------------- - If the nodes in the cloud have multiple interfaces to connect to management network, tenant network or storage network, monitor driver based on consul is another choice. Consul agents are required to install into all noedes, which make up multiple consul clusters. Here is an example to show how to make up one consul cluster. .. toctree:: :maxdepth: 1 consul-usage - The compute node's status is depending on assembly of multiple interfaces connectivity status, which are retrieved from multiple consul clusters. Then it sends notifition to trigger host failure recovery according to defined HA strategy - host states and the corresponding actions. Related configurations ------------------------ This section in masakarimonitors.conf shows an example of how to configure the hostmonitor if you choice monitor driver based on pacemaker. .. code-block:: ini [host] # Driver that hostmonitor uses for monitoring hosts. monitoring_driver = default # Monitoring interval(in seconds) of node status. monitoring_interval = 60 # Do not check whether the host is completely down. # Possible values: # * True: Do not check whether the host is completely down. # * False: Do check whether the host is completely down. # If ipmi RA is not set in pacemaker, this value should be set True. disable_ipmi_check = False # Timeout value(in seconds) of the ipmitool command. ipmi_timeout = 5 # Number of ipmitool command retries. ipmi_retry_max = 3 # Retry interval(in seconds) of the ipmitool command. ipmi_retry_interval = 10 # Only monitor pacemaker-remotes, ignore the status of full cluster # members. restrict_to_remotes = False # Standby time(in seconds) until activate STONITH. stonith_wait = 30 # Timeout value(in seconds) of the tcpdump command when monitors # the corosync communication. tcpdump_timeout = 5 # The name of interface that corosync is using for mutual communication # between hosts. # If there are multiple interfaces, specify them in comma-separated # like 'enp0s3,enp0s8'. # The number of interfaces you specify must be equal to the number of # corosync_multicast_ports values and must be in correct order with # relevant ports in corosync_multicast_ports. corosync_multicast_interfaces = enp0s3,enp0s8 # The port numbers that corosync is using for mutual communication # between hosts. # If there are multiple port numbers, specify them in comma-separated # like '5405,5406'. # The number of port numbers you specify must be equal to the number of # corosync_multicast_interfaces values and must be in correct order with # relevant interfaces in corosync_multicast_interfaces. corosync_multicast_ports = 5405,5406 If you want to use or test monitor driver based on consul, please modify following configuration. .. code-block:: ini [host] # Driver that hostmonitor uses for monitoring hosts. monitoring_driver = consul [consul] # Addr for local consul agent in management datacenter. # The addr is make up of the agent's bind_addr and http port, # such as '192.168.101.1:8500'. agent_manage = $(CONSUL_MANAGEMENT_ADDR) # Addr for local consul agent in tenant datacenter. agent_tenant = $(CONSUL_TENANT_ADDR) # Addr for local consul agent in storage datacenter. agent_storage = $(CONSUL_STORAGE_ADDR) # Config file for consul health action matrix. matrix_config_file = /etc/masakarimonitors/matrix.yaml The ``matrix_config_file`` shows the HA strategy. Matrix is combined by host health and actions. The 'health: [x, x, x]', repreasents assembly status of SEQUENCE. Action, means which actions it will trigger if host health turns into, while 'recovery' means it will trigger one host failure recovery workflow. User can define the HA strategy according to the physical environment. For example, if there is just 1 cluster to monitor management network connectivity, the user just need to configurate ``$(CONSUL_MANAGEMENT_ADDR)`` in consul section of the hostmontior' configuration file, and change the HA strategy in ``/etc/masakarimonitors/matrix.yaml`` as following: .. code-block:: yaml sequence: ['manage'] matrix: - health: ['up'] action: [] - health: ['down'] action: ['recovery'] Then the hostmonitor by consul works as same as the hostmonitor by pacemaker. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/index.rst0000664000175000017500000000142400000000000021255 0ustar00zuulzuul00000000000000.. masakari-monitors documentation master file, created by sphinx-quickstart on Tue Jul 9 22:26:36 2013. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. Welcome to masakari-monitors's documentation! ======================================================== Contents: .. toctree:: :maxdepth: 1 readme installation usage reference/conf reference/conf-file .. only:: html For Contributors ================ * If you are a new contributor to Masakari Monitors please refer: :doc:`contributor/contributing` .. toctree:: :hidden: contributor/contributing Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/installation.rst0000664000175000017500000000033400000000000022646 0ustar00zuulzuul00000000000000============ Installation ============ At the command line:: $ pip install masakari-monitors Or, if you have virtualenvwrapper installed:: $ mkvirtualenv masakari-monitors $ pip install masakari-monitors ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/instancemonitor.rst0000664000175000017500000000240300000000000023360 0ustar00zuulzuul00000000000000======================== masakari-instancemonitor ======================== Monitor Overview ------------------ The masakari-instancemonitor provides Virtual Machine High Availability for OpenStack clouds by automatically detecting VMs domain events via libvirt. If it detects specific libvirt events, it sends notifications to the masakari-api. How does it work? ---------------------------------------- - It runs libvirt event loop in a background thread. - Invoking libvirt.virEventRegisterDefaultImpl() will register libvirt's default event loop implementation. - Invoking libvirt.virEventRunDefaultImpl() will perform one iteration of the libvirt default event loop. - Invoking conn.domainEventRegisterAny() will register event callbacks against libvirt connection instances. The callbacks registered will be triggered from the execution context of libvirt.virEventRunDefaultImpl(), which will send notifications to the masakari-api. - It will reconnect to libvirt and reprocess if disconnected. Related configurations ------------------------ This section in masakarimonitors.conf shows an example of how to configure the monitor. .. code-block:: ini [libvirt] # Override the default libvirt URI. connection_uri = qemu:///system ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/introspectiveinstancemonitor.rst0000664000175000017500000000347400000000000026210 0ustar00zuulzuul00000000000000===================================== masakari-introspectiveinstancemonitor ===================================== Monitor Overview ----------------- The masakari-introspectiveinstancemonitor provides Virtual Machine HA for OpenStack clouds by automatically detecting the system-level failure events via QEMU Guest Agent. If it detects VM heartbeat failure events, it sends notifications to the masakari-api. How does it work? ---------------------------------------- - libvirt and QEMU Guest Agent are used as the underlying protocol for messaging to and from VM. - The host-side qemu-agent sockets are used to detemine whether VMs are configured with QEMU Guest Agent. - qemu-guest-ping is used as the monitoring heartbeat. - For the future release, we can pass through arbitrary guest agent commands to check the health of the applications inside a VM. Related configurations ------------------------ This section in masakarimonitors.conf shows an example of how to configure the monitor. .. code-block:: ini [libvirt] # Override the default libvirt URI. connection_uri = qemu:///system [introspectiveinstancemonitor] # Guest monitoring interval of VM status (in seconds). # * The value should not be too low as there should not be false negative # * for reporting QEMU_GUEST_AGENT failures # * VM needs time to do powering-off. # * guest_monitoring_interval should be greater than # * the time to SHUTDOWN VM gracefully. guest_monitoring_interval = 10 # Guest monitoring timeout (in seconds). guest_monitoring_timeout = 2 # Failure threshold before sending notification. guest_monitoring_failure_threshold = 3 # The file path of qemu guest agent sock. qemu_guest_agent_sock_path = \ /var/lib/libvirt/qemu/org\.qemu\.guest_agent\..*\.instance-.*\.sock ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/processmonitor.rst0000664000175000017500000000427100000000000023237 0ustar00zuulzuul00000000000000======================= masakari-processmonitor ======================= Monitor Overview ------------------ The masakari-processmonitor, provides key process High Availability for OpenStack clouds by automatically detecting the process failure. If it detects process failure, it sends notifications to masakari-api. If your OpenStack service runs in container(pod), this processmonitor will not work as expected. It is recommended not to deploy processmonitor. How does it work? ------------------- - Processes to be monitored should be pre-configured in process_list.yaml file. Define one process to be monitored as follows: .. code-block:: ini process_name: [Name of the process as it in 'ps -ef'.] start_command: [Start command of the process.] pre_start_command: [Command which is executed before start_command.] post_start_command: [Command which is executed after start_command.] restart_command: [Restart command of the process.] pre_restart_command: [Command which is executed before restart_command.] post_restart_command: [Command which is executed after restart_command.] run_as_root: [Bool value whether to execute commands as root authority.] Sample of definitions is shown as follows: .. code-block:: ini # nova-compute process_name: /usr/local/bin/nova-compute start_command: systemctl start nova-compute pre_start_command: post_start_command: restart_command: systemctl restart nova-compute pre_restart_command: post_restart_command: run_as_root: True - If masakari-processmonitor detects one process failure, it will try to restart it firstly. After several retries failed, it sends notification to masakari-api. Related configurations ------------------------ This section in masakarimonitors.conf shows an example of how to configure the monitor. .. code-block:: ini [process] # Interval in seconds for checking a process. check_interval = 5 # Number of retries when the failure of restarting a process. restart_retries = 3 # Interval in seconds for restarting a process. restart_interval = 5 # The file path of process list. process_list_path = /etc/masakarimonitors/process_list.yaml ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/readme.rst0000664000175000017500000000003600000000000021401 0ustar00zuulzuul00000000000000.. include:: ../../README.rst ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1812007 masakari_monitors-19.0.0/doc/source/reference/0000775000175000017500000000000000000000000021351 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/reference/conf-file.rst0000664000175000017500000000126100000000000023745 0ustar00zuulzuul00000000000000.. _monitors-config-file: ------------------------------------------- Masakari Monitors Sample Configuration File ------------------------------------------- Configure Masakari Monitors by editing /etc/masakarimonitors/masakarimonitors.conf. No config file is provided with the source code, it will be created during the installation. In case where no configuration file was installed, one can be easily created by running:: tox -e genconfig To see configuration options available, please refer to :ref:`monitors-config`. .. only:: html The following is a sample monitors configuration for adaptation and use. .. literalinclude:: ../_static/masakarimonitors.conf.sample ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/reference/conf.rst0000664000175000017500000000061400000000000023031 0ustar00zuulzuul00000000000000.. _monitors-config: --------------------------------------- Masakari Monitors Configuration Options --------------------------------------- The following is an overview of all available configuration options in masakari-monitors. To see sample configuration file, see :ref:`monitors-config-file`. .. show-options:: :config-file: etc/masakarimonitors/masakarimonitors-config-generator.conf ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/doc/source/usage.rst0000664000175000017500000000024300000000000021250 0ustar00zuulzuul00000000000000======== Usage ======== Monitors for Masakari: .. toctree:: :maxdepth: 1 hostmonitor instancemonitor introspectiveinstancemonitor processmonitor ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1691997 masakari_monitors-19.0.0/etc/0000775000175000017500000000000000000000000016121 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1812007 masakari_monitors-19.0.0/etc/masakarimonitors/0000775000175000017500000000000000000000000021504 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/etc/masakarimonitors/README-masakarimonitors.conf.txt0000664000175000017500000000022000000000000027501 0ustar00zuulzuul00000000000000To generate the sample masakarimonitors.conf file, run the following command from the top level of the masakari directory: tox -egenconfig ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/etc/masakarimonitors/masakarimonitors-config-generator.conf0000664000175000017500000000025500000000000031167 0ustar00zuulzuul00000000000000[DEFAULT] output_file = etc/masakarimonitors/masakarimonitors.conf.sample wrap_width = 80 namespace = masakarimonitors.conf namespace = oslo.log namespace = oslo.middleware ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/etc/masakarimonitors/matrix.yaml.sample0000664000175000017500000000100500000000000025150 0ustar00zuulzuul00000000000000--- sequence: ['manage', 'tenant', 'storage'] matrix: - health: ['up', 'up', 'up'] action: [] - health: ['up', 'up', 'down'] action: ['recovery'] - health: ['up', 'down', 'up'] action: [] - health: ['up', 'down', 'down'] action: ['recovery'] - health: ['down', 'up', 'up'] action: [] - health: ['down', 'up', 'down'] action: ['recovery'] - health: ['down', 'down', 'up'] action: [] - health: ['down', 'down', 'down'] action: ['recovery']././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/etc/masakarimonitors/process_list.yaml.sample0000664000175000017500000000421400000000000026362 0ustar00zuulzuul00000000000000# Define the monitoring processes as follows: # process_name: [Name of the process as it in 'ps -ef'.] # start_command: [Start command of the process.] # pre_start_command: [Command which is executed before start_command.] # post_start_command: [Command which is executed after start_command.] # restart_command: [Restart command of the process.] # pre_restart_command: [Command which is executed before restart_command.] # post_restart_command: [Command which is executed after restart_command.] # run_as_root: [Bool value whether to execute commands as root authority.] # # These definitions need to be set according to the environment to use. # Sample of definitions is shown below. - # libvirt-bin process_name: /usr/sbin/libvirtd start_command: systemctl start libvirt-bin pre_start_command: post_start_command: restart_command: systemctl restart libvirt-bin pre_restart_command: post_restart_command: run_as_root: True - # nova-compute process_name: /usr/local/bin/nova-compute start_command: systemctl start nova-compute pre_start_command: post_start_command: restart_command: systemctl restart nova-compute pre_restart_command: post_restart_command: run_as_root: True - # instancemonitor process_name: /usr/bin/python /usr/local/bin/masakari-instancemonitor start_command: systemctl start masakari-instancemonitor pre_start_command: post_start_command: restart_command: systemctl restart masakari-instancemonitor pre_restart_command: post_restart_command: run_as_root: True - # hostmonitor process_name: /usr/bin/python /usr/local/bin/masakari-hostmonitor start_command: systemctl start masakari-hostmonitor pre_start_command: post_start_command: restart_command: systemctl restart masakari-hostmonitor pre_restart_command: post_restart_command: run_as_root: True - # sshd process_name: /usr/sbin/sshd start_command: systemctl start ssh pre_start_command: post_start_command: restart_command: systemctl restart ssh pre_restart_command: post_restart_command: run_as_root: True ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1691997 masakari_monitors-19.0.0/masakari-monitors/0000775000175000017500000000000000000000000021006 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1691997 masakari_monitors-19.0.0/masakari-monitors/doc/0000775000175000017500000000000000000000000021553 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1691997 masakari_monitors-19.0.0/masakari-monitors/doc/source/0000775000175000017500000000000000000000000023053 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1812007 masakari_monitors-19.0.0/masakari-monitors/doc/source/reference/0000775000175000017500000000000000000000000025011 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakari-monitors/doc/source/reference/conf.rst0000664000175000017500000000062700000000000026475 0ustar00zuulzuul00000000000000.. _monitors-config: --------------------------------------- Masakari Monitors Configuration Options --------------------------------------- The following is an overview of all available configuration options in masakari-monitors. To see sample configuration file, see :ref:`monitors-config-file`. .. _show-options:: :config-file: etc/masakarimonitors/masakarimonitors-config-generator.conf././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.2052033 masakari_monitors-19.0.0/masakari_monitors.egg-info/0000775000175000017500000000000000000000000022562 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591461.0 masakari_monitors-19.0.0/masakari_monitors.egg-info/PKG-INFO0000644000175000017500000000635200000000000023663 0ustar00zuulzuul00000000000000Metadata-Version: 2.1 Name: masakari-monitors Version: 19.0.0 Summary: Monitors for Masakari Home-page: https://docs.openstack.org/masakari-monitors/latest/ Author: OpenStack Author-email: openstack-dev@lists.openstack.org Classifier: Environment :: OpenStack Classifier: Intended Audience :: Information Technology Classifier: Intended Audience :: System Administrators Classifier: License :: OSI Approved :: Apache Software License Classifier: Operating System :: POSIX :: Linux Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: 3 :: Only Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.8 Classifier: Programming Language :: Python :: 3.9 Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Requires-Python: >=3.8 License-File: LICENSE Requires-Dist: automaton>=1.9.0 Requires-Dist: keystoneauth1>=3.4.0 Requires-Dist: openstacksdk>=0.13.0 Requires-Dist: oslo.concurrency>=3.26.0 Requires-Dist: oslo.config>=5.2.0 Requires-Dist: lxml>=4.5.0 Requires-Dist: oslo.cache>=1.26.0 Requires-Dist: oslo.i18n>=3.15.3 Requires-Dist: oslo.log>=3.36.0 Requires-Dist: oslo.middleware>=3.31.0 Requires-Dist: oslo.privsep>=1.23.0 Requires-Dist: oslo.service!=1.28.1,>=1.24.0 Requires-Dist: oslo.utils>=3.33.0 Requires-Dist: pbr!=2.1.0,>=2.0.0 Requires-Dist: python-consul>=1.1.0 =============================== masakari-monitors =============================== Monitors for Masakari ===================== Monitors for Masakari provides Virtual Machine High Availability (VMHA) service for OpenStack clouds by automatically detecting the failure events such as VM process down, provisioning process down, and nova-compute host failure. If it detect the events, it sends notifications to the masakari-api. Original version of Masakari: https://github.com/ntt-sic/masakari Tokyo Summit Session: https://www.youtube.com/watch?v=BmjNKceW_9A Monitors for Masakari is distributed under the terms of the Apache License, Version 2.0. The full terms and conditions of this license are detailed in the LICENSE file. * Free software: Apache license * Documentation: https://docs.openstack.org/masakari-monitors * Source: https://git.openstack.org/cgit/openstack/masakari-monitors * Bugs: https://bugs.launchpad.net/masakari-monitors Configure masakari-monitors --------------------------- #. Clone masakari using:: $ git clone https://github.com/openstack/masakari-monitors.git #. Create masakarimonitors directory in /etc/. #. Run setup.py from masakari-monitors:: $ sudo python setup.py install #. Copy masakarimonitors.conf and process_list.yaml files from masakari-monitors/etc/ to /etc/masakarimonitors folder and make necessary changes to the masakarimonitors.conf and process_list.yaml files. To generate the sample masakarimonitors.conf file, run the following command from the top level of the masakari-monitors directory:: $ tox -egenconfig #. To run masakari-processmonitor, masakari-hostmonitor and masakari-instancemonitor simply use following binary:: $ masakari-processmonitor $ masakari-hostmonitor $ masakari-instancemonitor Features -------- * TODO ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591461.0 masakari_monitors-19.0.0/masakari_monitors.egg-info/SOURCES.txt0000664000175000017500000001601500000000000024451 0ustar00zuulzuul00000000000000.coveragerc .mailmap .stestr.conf .yamllint .zuul.yaml AUTHORS CONTRIBUTING.rst ChangeLog HACKING.rst LICENSE MANIFEST.in README.rst bindep.txt requirements.txt setup.cfg setup.py test-requirements.txt tox.ini doc/requirements.txt doc/source/conf.py doc/source/consul-usage.rst doc/source/hostmonitor.rst doc/source/index.rst doc/source/installation.rst doc/source/instancemonitor.rst doc/source/introspectiveinstancemonitor.rst doc/source/processmonitor.rst doc/source/readme.rst doc/source/usage.rst doc/source/_static/.gitkeep doc/source/contributor/contributing.rst doc/source/reference/conf-file.rst doc/source/reference/conf.rst etc/masakarimonitors/README-masakarimonitors.conf.txt etc/masakarimonitors/masakarimonitors-config-generator.conf etc/masakarimonitors/matrix.yaml.sample etc/masakarimonitors/process_list.yaml.sample masakari-monitors/doc/source/reference/conf.rst masakari_monitors.egg-info/PKG-INFO masakari_monitors.egg-info/SOURCES.txt masakari_monitors.egg-info/dependency_links.txt masakari_monitors.egg-info/entry_points.txt masakari_monitors.egg-info/not-zip-safe masakari_monitors.egg-info/pbr.json masakari_monitors.egg-info/requires.txt masakari_monitors.egg-info/top_level.txt masakarimonitors/__init__.py masakarimonitors/config.py masakarimonitors/i18n.py masakarimonitors/manager.py masakarimonitors/privsep.py masakarimonitors/service.py masakarimonitors/utils.py masakarimonitors/version.py masakarimonitors/cmd/__init__.py masakarimonitors/cmd/hostmonitor.py masakarimonitors/cmd/instancemonitor.py masakarimonitors/cmd/introspectiveinstancemonitor.py masakarimonitors/cmd/processmonitor.py masakarimonitors/common/__init__.py masakarimonitors/common/config.py masakarimonitors/conf/__init__.py masakarimonitors/conf/api.py masakarimonitors/conf/base.py masakarimonitors/conf/consul.py masakarimonitors/conf/host.py masakarimonitors/conf/instance.py masakarimonitors/conf/introspectiveinstancemonitor.py masakarimonitors/conf/opts.py masakarimonitors/conf/process.py masakarimonitors/conf/service.py masakarimonitors/ha/__init__.py masakarimonitors/ha/masakari.py masakarimonitors/hacking/__init__.py masakarimonitors/hacking/checks.py masakarimonitors/hostmonitor/__init__.py masakarimonitors/hostmonitor/driver.py masakarimonitors/hostmonitor/host.py masakarimonitors/hostmonitor/consul_check/__init__.py masakarimonitors/hostmonitor/consul_check/consul_helper.py masakarimonitors/hostmonitor/consul_check/manager.py masakarimonitors/hostmonitor/consul_check/matrix_helper.py masakarimonitors/hostmonitor/host_handler/__init__.py masakarimonitors/hostmonitor/host_handler/handle_host.py masakarimonitors/hostmonitor/host_handler/hold_host_status.py masakarimonitors/hostmonitor/host_handler/parse_cib_xml.py masakarimonitors/hostmonitor/host_handler/parse_crmmon_xml.py masakarimonitors/instancemonitor/__init__.py masakarimonitors/instancemonitor/instance.py masakarimonitors/instancemonitor/libvirt_handler/__init__.py masakarimonitors/instancemonitor/libvirt_handler/callback.py masakarimonitors/instancemonitor/libvirt_handler/eventfilter.py masakarimonitors/instancemonitor/libvirt_handler/eventfilter_table.py masakarimonitors/introspectiveinstancemonitor/README.rst masakarimonitors/introspectiveinstancemonitor/__init__.py masakarimonitors/introspectiveinstancemonitor/cache.py masakarimonitors/introspectiveinstancemonitor/instance.py masakarimonitors/introspectiveinstancemonitor/qemu_utils.py masakarimonitors/introspectiveinstancemonitor/scheduler.py masakarimonitors/objects/__init__.py masakarimonitors/objects/event_constants.py masakarimonitors/processmonitor/__init__.py masakarimonitors/processmonitor/process.py masakarimonitors/processmonitor/process_handler/__init__.py masakarimonitors/processmonitor/process_handler/handle_process.py masakarimonitors/tests/__init__.py masakarimonitors/tests/base.py masakarimonitors/tests/test_masakarimonitors.py masakarimonitors/tests/unit/__init__.py masakarimonitors/tests/unit/test_hacking.py masakarimonitors/tests/unit/ha/__init__.py masakarimonitors/tests/unit/ha/test_masakari.py masakarimonitors/tests/unit/hostmonitor/__init__.py masakarimonitors/tests/unit/hostmonitor/test_host.py masakarimonitors/tests/unit/hostmonitor/consul_check/__init__.py masakarimonitors/tests/unit/hostmonitor/consul_check/test_consul_helper.py masakarimonitors/tests/unit/hostmonitor/consul_check/test_manager.py masakarimonitors/tests/unit/hostmonitor/consul_check/test_matrix_helper.py masakarimonitors/tests/unit/hostmonitor/host_handler/__init__.py masakarimonitors/tests/unit/hostmonitor/host_handler/test_handle_host.py masakarimonitors/tests/unit/hostmonitor/host_handler/test_hold_host_status.py masakarimonitors/tests/unit/hostmonitor/host_handler/test_parse_cib_xml.py masakarimonitors/tests/unit/hostmonitor/host_handler/test_parse_crmmon_xml.py masakarimonitors/tests/unit/instancemonitor/__init__.py masakarimonitors/tests/unit/instancemonitor/test_instance.py masakarimonitors/tests/unit/instancemonitor/libvirt_handler/__init__.py masakarimonitors/tests/unit/instancemonitor/libvirt_handler/test_callback.py masakarimonitors/tests/unit/instancemonitor/libvirt_handler/test_eventfilter.py masakarimonitors/tests/unit/introspectiveinstancemonitor/__init__.py masakarimonitors/tests/unit/introspectiveinstancemonitor/test_monitor_manager.py masakarimonitors/tests/unit/introspectiveinstancemonitor/test_qemu_utils.py masakarimonitors/tests/unit/processmonitor/__init__.py masakarimonitors/tests/unit/processmonitor/test_process.py masakarimonitors/tests/unit/processmonitor/process_handler/__init__.py masakarimonitors/tests/unit/processmonitor/process_handler/test_handle_process.py releasenotes/notes/.placeholder releasenotes/notes/bp-retry-check-when-host-failure-78649c512ef79199.yaml releasenotes/notes/bug-1866660-ef8624f5283b2e5e.yaml releasenotes/notes/bug-1878548-5fab31aec6ba5407.yaml releasenotes/notes/bug-1930361-fa8ce8e9781ea967.yaml releasenotes/notes/deprecate-processmonitor-95c8aefbc749c1ed.yaml releasenotes/notes/drop-cap-net-admin-8d7d7cfb274e9547.yaml releasenotes/notes/drop-py-2-7-b28de816eac45468.yaml releasenotes/notes/hostmonitor-driver-based-on-consul-03f7e619d91e7e06.yaml releasenotes/notes/hostmonitor-systemd-89696f96a654a918.yaml releasenotes/notes/introspectiveinstancemonitor-f4bc71f029b61d49.yaml releasenotes/notes/libvirt-sasl-support-edf1388c556a594b.yaml releasenotes/notes/libvirt-still-required-22a8d817ee8d0be8.yaml releasenotes/notes/pythonize-monitors-081e74dfaf78fe99.yaml releasenotes/source/2023.1.rst releasenotes/source/2023.2.rst releasenotes/source/2024.1.rst releasenotes/source/2024.2.rst releasenotes/source/conf.py releasenotes/source/index.rst releasenotes/source/ocata.rst releasenotes/source/pike.rst releasenotes/source/queens.rst releasenotes/source/rocky.rst releasenotes/source/stein.rst releasenotes/source/train.rst releasenotes/source/unreleased.rst releasenotes/source/ussuri.rst releasenotes/source/victoria.rst releasenotes/source/wallaby.rst releasenotes/source/xena.rst releasenotes/source/yoga.rst releasenotes/source/zed.rst releasenotes/source/_static/.placeholder releasenotes/source/_templates/.placeholder././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591461.0 masakari_monitors-19.0.0/masakari_monitors.egg-info/dependency_links.txt0000664000175000017500000000000100000000000026630 0ustar00zuulzuul00000000000000 ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591461.0 masakari_monitors-19.0.0/masakari_monitors.egg-info/entry_points.txt0000664000175000017500000000200600000000000026056 0ustar00zuulzuul00000000000000[console_scripts] masakari-hostmonitor = masakarimonitors.cmd.hostmonitor:main masakari-instancemonitor = masakarimonitors.cmd.instancemonitor:main masakari-introspectiveinstancemonitor = masakarimonitors.cmd.introspectiveinstancemonitor:main masakari-processmonitor = masakarimonitors.cmd.processmonitor:main [hostmonitor.driver] consul = masakarimonitors.hostmonitor.consul_check.manager:ConsulCheck default = masakarimonitors.hostmonitor.host_handler.handle_host:HandleHost simple = masakarimonitors.hostmonitor.host_handler.handle_host:HandleHost [oslo.config.opts] masakarimonitors.conf = masakarimonitors.conf.opts:list_opts [oslo.config.opts.defaults] masakarimonitors.hostmonitor = masakarimonitors.common.config:set_middleware_defaults masakarimonitors.instancemonitor = masakarimonitors.common.config:set_middleware_defaults masakarimonitors.introspectiveinstancemonitor = masakarimonitors.common.config:set_middleware_defaults masakarimonitors.processmonitor = masakarimonitors.common.config:set_middleware_defaults ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591461.0 masakari_monitors-19.0.0/masakari_monitors.egg-info/not-zip-safe0000664000175000017500000000000100000000000025010 0ustar00zuulzuul00000000000000 ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591461.0 masakari_monitors-19.0.0/masakari_monitors.egg-info/pbr.json0000664000175000017500000000005600000000000024241 0ustar00zuulzuul00000000000000{"git_version": "1f62741", "is_release": true}././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591461.0 masakari_monitors-19.0.0/masakari_monitors.egg-info/requires.txt0000664000175000017500000000045700000000000025170 0ustar00zuulzuul00000000000000automaton>=1.9.0 keystoneauth1>=3.4.0 openstacksdk>=0.13.0 oslo.concurrency>=3.26.0 oslo.config>=5.2.0 lxml>=4.5.0 oslo.cache>=1.26.0 oslo.i18n>=3.15.3 oslo.log>=3.36.0 oslo.middleware>=3.31.0 oslo.privsep>=1.23.0 oslo.service!=1.28.1,>=1.24.0 oslo.utils>=3.33.0 pbr!=2.1.0,>=2.0.0 python-consul>=1.1.0 ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591461.0 masakari_monitors-19.0.0/masakari_monitors.egg-info/top_level.txt0000664000175000017500000000002100000000000025305 0ustar00zuulzuul00000000000000masakarimonitors ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1852012 masakari_monitors-19.0.0/masakarimonitors/0000775000175000017500000000000000000000000020731 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/__init__.py0000664000175000017500000000124100000000000023040 0ustar00zuulzuul00000000000000# -*- coding: utf-8 -*- # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import pbr.version __version__ = pbr.version.VersionInfo( 'masakari-monitors').version_string() ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1852012 masakari_monitors-19.0.0/masakarimonitors/cmd/0000775000175000017500000000000000000000000021474 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/cmd/__init__.py0000664000175000017500000000000000000000000023573 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/cmd/hostmonitor.py0000664000175000017500000000213500000000000024434 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Starter script for Masakari Host Monitor.""" import sys from oslo_log import log as logging import masakarimonitors.conf from masakarimonitors import config from masakarimonitors import service from masakarimonitors import utils CONF = masakarimonitors.conf.CONF def main(): config.parse_args(sys.argv) logging.setup(CONF, "masakarimonitors") utils.monkey_patch() server = service.Service.create(binary='masakarimonitors-hostmonitor') service.serve(server) service.wait() ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/cmd/instancemonitor.py0000664000175000017500000000214500000000000025264 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Starter script for Masakari Instance Monitor.""" import sys from oslo_log import log as logging import masakarimonitors.conf from masakarimonitors import config from masakarimonitors import service from masakarimonitors import utils CONF = masakarimonitors.conf.CONF def main(): config.parse_args(sys.argv) logging.setup(CONF, "masakarimonitors") utils.monkey_patch() server = service.Service.create(binary='masakarimonitors-instancemonitor') service.serve(server) service.wait() ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/cmd/introspectiveinstancemonitor.py0000664000175000017500000000216000000000000030100 0ustar00zuulzuul00000000000000# Copyright(c) 2018 WindRiver Systems # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Starter script for Masakari Introspective Instance Monitor.""" import sys from oslo_log import log as logging import masakarimonitors.conf from masakarimonitors import config from masakarimonitors import service from masakarimonitors import utils CONF = masakarimonitors.conf.CONF def main(): config.parse_args(sys.argv) logging.setup(CONF, "masakarimonitors") utils.monkey_patch() server = service.Service.create( binary='masakarimonitors-introspectiveinstancemonitor') service.serve(server) service.wait() ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/cmd/processmonitor.py0000664000175000017500000000214300000000000025134 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Starter script for Masakari Process Monitor.""" import sys from oslo_log import log as logging import masakarimonitors.conf from masakarimonitors import config from masakarimonitors import service from masakarimonitors import utils CONF = masakarimonitors.conf.CONF def main(): config.parse_args(sys.argv) logging.setup(CONF, "masakarimonitors") utils.monkey_patch() server = service.Service.create(binary='masakarimonitors-processmonitor') service.serve(server) service.wait() ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1852012 masakari_monitors-19.0.0/masakarimonitors/common/0000775000175000017500000000000000000000000022221 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/common/__init__.py0000664000175000017500000000000000000000000024320 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/common/config.py0000664000175000017500000000147500000000000024047 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from oslo_middleware import cors def set_middleware_defaults(): """Update default configuration options for oslo.middleware.""" # CORS Defaults cfg.set_defaults(cors.CORS_OPTS) ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1892016 masakari_monitors-19.0.0/masakarimonitors/conf/0000775000175000017500000000000000000000000021656 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/conf/__init__.py0000664000175000017500000000232600000000000023772 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg from masakarimonitors.conf import api from masakarimonitors.conf import base from masakarimonitors.conf import consul from masakarimonitors.conf import host from masakarimonitors.conf import instance from masakarimonitors.conf import introspectiveinstancemonitor from masakarimonitors.conf import process from masakarimonitors.conf import service CONF = cfg.CONF api.register_opts(CONF) base.register_opts(CONF) consul.register_opts(CONF) host.register_opts(CONF) instance.register_opts(CONF) introspectiveinstancemonitor.register_opts(CONF) process.register_opts(CONF) service.register_opts(CONF) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/conf/api.py0000664000175000017500000000314400000000000023003 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from keystoneauth1 import loading as ks_loading from oslo_config import cfg API_GROUP = 'api' api_group = cfg.OptGroup( API_GROUP, title='Api Options', help=""" Configuration options for sending notifications. """) monitor_api_opts = [ cfg.StrOpt('region', default='RegionOne', help='Region name.'), cfg.StrOpt('api_version', default='v1', help='Masakari API Version.'), cfg.StrOpt('api_interface', default='public', help='Interface of endpoint.'), ] def register_opts(conf): conf.register_group(api_group) conf.register_opts(monitor_api_opts, group=api_group) ks_loading.register_session_conf_options(conf, api_group) ks_loading.register_auth_conf_options(conf, api_group) conf.set_default('auth_type', 'password', group=api_group) def list_opts(): return { api_group: ( monitor_api_opts + ks_loading.get_auth_plugin_conf_options('password')) } ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/conf/base.py0000664000175000017500000000266500000000000023153 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg base_options = [ cfg.StrOpt( 'tempdir', help='Explicitly specify the temporary working directory.'), cfg.BoolOpt( 'monkey_patch', default=False, help=""" Determine if monkey patching should be applied. Related options: * ``monkey_patch_modules``: This must have values set for this option to have any effect """), cfg.ListOpt( 'monkey_patch_modules', default=[], help=""" List of modules/decorators to monkey patch. This option allows you to patch a decorator for all functions in specified modules. Related options: * ``monkey_patch``: This must be set to ``True`` for this option to have any effect """), ] def register_opts(conf): conf.register_opts(base_options) def list_opts(): return {'DEFAULT': base_options} ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/conf/consul.py0000664000175000017500000000225400000000000023536 0ustar00zuulzuul00000000000000# Copyright(c) 2019 Inspur # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg consul_opts = [ cfg.StrOpt('agent_manage', help='Addr for local consul agent in management datacenter.'), cfg.StrOpt('agent_tenant', help='Addr for local consul agent in tenant datacenter.'), cfg.StrOpt('agent_storage', help='Addr for local consul agent in storage datacenter.'), cfg.StrOpt('matrix_config_file', help='Config file for consul health action matrix.'), ] def register_opts(conf): conf.register_opts(consul_opts, group='consul') def list_opts(): return { 'consul': consul_opts } ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/conf/host.py0000664000175000017500000001042700000000000023211 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg monitor_host_opts = [ cfg.StrOpt('monitoring_driver', default='default', help='Driver that hostmonitor uses for monitoring hosts.'), cfg.IntOpt('monitoring_interval', default=60, help='Monitoring interval(in seconds) of node status.'), cfg.IntOpt('monitoring_samples', default=1, help=''' Monitoring probes to collect before making the decision to send Masakari notification about the node status. If and only if ``monitoring_samples`` consecutive reports have the same status, will the Masakari notification be sent. '''), cfg.IntOpt('api_retry_max', default=12, help='Number of retries for send a notification in' ' hostmonitor.'), cfg.IntOpt('api_retry_interval', default=10, help='Trial interval of time of the notification processing' ' is error(in seconds).'), cfg.BoolOpt('disable_ipmi_check', default=False, help=''' Do not check whether the host is completely down. Possible values: * True: Do not check whether the host is completely down. * False: Do check whether the host is completely down. If ipmi RA is not set in pacemaker, this value should be set True. '''), cfg.BoolOpt('restrict_to_remotes', default=False, help='Only monitor pacemaker-remotes, ignore the status of' ' full cluster members.'), cfg.IntOpt('ipmi_timeout', default=5, help='Timeout value(in seconds) of the ipmitool command.'), cfg.IntOpt('ipmi_retry_max', default=3, help='Number of ipmitool command retries.'), cfg.IntOpt('ipmi_retry_interval', default=10, help='Retry interval(in seconds) of the ipmitool command.'), cfg.IntOpt('stonith_wait', default=30, help='Standby time(in seconds) until activate STONITH.'), cfg.IntOpt('tcpdump_timeout', default=5, help='Timeout value(in seconds) of the tcpdump command when' ' monitors the corosync communication.'), cfg.StrOpt('corosync_multicast_interfaces', help=''' The name of interface that corosync is using for mutual communication between hosts. If there are multiple interfaces, specify them in comma-separated like 'enp0s3,enp0s8'. The number of interfaces you specify must be equal to the number of corosync_multicast_ports values and must be in correct order with relevant ports in corosync_multicast_ports. '''), cfg.StrOpt('corosync_multicast_ports', help=''' The port numbers that corosync is using for mutual communication between hosts. If there are multiple port numbers, specify them in comma-separated like '5405,5406'. The number of port numbers you specify must be equal to the number of corosync_multicast_interfaces values and must be in correct order with relevant interfaces in corosync_multicast_interfaces. '''), cfg.StrOpt('pacemaker_node_type', default='autodetect', choices=('autodetect', 'cluster', 'remote'), help=''' Using this option, one can avoid systemd checks that would establish whether this hostmonitor is running alongside Corosync and Pacemaker (the ``cluster`` stack) or Pacemaker Remote (the ``remote`` stack). The default (``autodetect``) ensures backward compatibility and means systemd is used to check the stack. '''), ] def register_opts(conf): conf.register_opts(monitor_host_opts, group='host') def list_opts(): return { 'host': monitor_host_opts } ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/conf/instance.py0000664000175000017500000000261600000000000024041 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg monitor_callback_opts = [ cfg.IntOpt('retry_max', default=12, help='Number of retries when the notification processing' ' is error.'), cfg.IntOpt('retry_interval', default=10, help='Trial interval of time of the notification processing' ' is error(in seconds).'), ] libvirt_opts = [ cfg.StrOpt('connection_uri', default='qemu:///system', help='Override the default libvirt URI.') ] def register_opts(conf): conf.register_opts(monitor_callback_opts, group='callback') conf.register_opts(libvirt_opts, group='libvirt') def list_opts(): return { 'callback': monitor_callback_opts, 'libvirt': libvirt_opts } ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/conf/introspectiveinstancemonitor.py0000664000175000017500000000427100000000000030267 0ustar00zuulzuul00000000000000# Copyright(c) 2018 WindRiver Systems # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg # Note: this string is being used for regex parsing later with re module. # # Use Python's raw string notation for regular expressions and # uses the backslash character ('\') to indicate special # forms or to allow special characters to be used without invoking # their special meaning. SOCK = r'/var/lib/libvirt/qemu/org\.qemu\.guest_agent\..*\.instance-.*\.sock' monitor_opts = [ cfg.IntOpt('guest_monitoring_interval', default=10, help=''' Guest monitoring interval of VM status (in seconds). * The value should not be too low as there should not be false negative * for reporting QEMU_GUEST_AGENT failures * VM needs time to do powering-off. * guest_monitoring_interval should be greater than * the time to SHUTDOWN VM gracefully. * e.g. | 565da9ba-3c0c-4087-83ca | iim1 | ACTIVE | powering-off | Running '''), cfg.IntOpt('guest_monitoring_timeout', default=2, help='Guest monitoring timeout (in seconds).'), cfg.IntOpt('guest_monitoring_failure_threshold', default=3, help='Failure threshold before sending notification.'), cfg.StrOpt('qemu_guest_agent_sock_path', default=SOCK, help=r''' * The file path of qemu guest agent sock. * Please use Python raw string notation as regular expressions. e.g. r'/var/lib/libvirt/qemu/org\.qemu\.guest_agent\..*\.instance-.*\.sock' '''), ] def register_opts(conf): conf.register_opts(monitor_opts, group='introspectiveinstancemonitor') def list_opts(): return { 'introspectiveinstancemonitor': monitor_opts } ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/conf/opts.py0000664000175000017500000000570500000000000023224 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ This is the single point of entry to generate the sample configuration file for Masakari Monitors. It collects all the necessary info from the other modules in this package. It is assumed that: * every other module in this package has a 'list_opts' function which return a dict where * the keys are strings which are the group names * the value of each key is a list of config options for that group * the masakari.conf package doesn't have further packages with config options * this module is only used in the context of sample file generation """ import collections import importlib import os import pkgutil LIST_OPTS_FUNC_NAME = "list_opts" def _tupleize(dct): """Take the dict of options and convert to the 2-tuple format.""" return [(key, val) for key, val in dct.items()] def list_opts(): opts = collections.defaultdict(list) module_names = _list_module_names() imported_modules = _import_modules(module_names) _append_config_options(imported_modules, opts) return _tupleize(opts) def _list_module_names(): module_names = [] package_path = os.path.dirname(os.path.abspath(__file__)) for _, modname, ispkg in pkgutil.iter_modules(path=[package_path]): if modname == "opts" or ispkg: continue else: module_names.append(modname) return module_names def _import_modules(module_names): imported_modules = [] for modname in module_names: mod = importlib.import_module("masakarimonitors.conf." + modname) if not hasattr(mod, LIST_OPTS_FUNC_NAME): msg = "The module 'masakarimonitors.conf.%s' should have a" \ " '%s' function which returns the config options." % \ (modname, LIST_OPTS_FUNC_NAME) raise Exception(msg) else: imported_modules.append(mod) return imported_modules def _process_old_opts(configs): """Convert old-style 2-tuple configs to dicts.""" if isinstance(configs, tuple): configs = [configs] return {label: options for label, options in configs} def _append_config_options(imported_modules, config_options): for mod in imported_modules: configs = mod.list_opts() if not isinstance(configs, dict): configs = _process_old_opts(configs) for key, val in configs.items(): config_options[key].extend(val) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/conf/process.py0000664000175000017500000000334700000000000023715 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_config import cfg monitor_process_opts = [ cfg.IntOpt('check_interval', default=5, help='Interval in seconds for checking a process.'), cfg.IntOpt('restart_retries', default=3, help='Number of retries when the failure of restarting a' ' process.'), cfg.IntOpt('restart_interval', default=5, help='Interval in seconds for restarting a process.'), cfg.IntOpt('api_retry_max', default=12, help='Number of retries for send a notification in' ' processmonitor.'), cfg.IntOpt('api_retry_interval', default=10, help='Interval between re-sending a notification in' ' processmonitor(in seconds).'), cfg.StrOpt('process_list_path', default='/etc/masakarimonitors/process_list.yaml', help='The file path of process list.'), ] def register_opts(conf): conf.register_opts(monitor_process_opts, group='process') def list_opts(): return { 'process': monitor_process_opts } ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/conf/service.py0000664000175000017500000000402000000000000023664 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import socket from oslo_config import cfg service_opts = [ cfg.StrOpt('hostname', default=socket.gethostname(), deprecated_name="host", help=''' Hostname, FQDN or IP address of this host. Must be valid within AMQP key. Possible values: * String with hostname, FQDN or IP address. Default is hostname of this host. '''), cfg.StrOpt('instancemonitor_manager', default='masakarimonitors.instancemonitor.instance' '.InstancemonitorManager', help='Full class name for the Manager for instancemonitor.'), cfg.StrOpt('introspectiveinstancemonitor_manager', default='masakarimonitors.introspectiveinstancemonitor.instance' '.IntrospectiveInstanceMonitorManager', help='Full class name for introspectiveinstancemonitor.'), cfg.StrOpt('processmonitor_manager', default='masakarimonitors.processmonitor.process' '.ProcessmonitorManager', help='Full class name for the Manager for processmonitor.'), cfg.StrOpt('hostmonitor_manager', default='masakarimonitors.hostmonitor.host' '.HostmonitorManager', help='Full class name for the Manager for hostmonitor.'), ] def register_opts(conf): conf.register_opts(service_opts) def list_opts(): return {'DEFAULT': service_opts} ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/config.py0000664000175000017500000000221700000000000022552 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_log import log import masakarimonitors.conf from masakarimonitors import version CONF = masakarimonitors.conf.CONF def parse_args(argv, default_config_files=None): log.register_options(CONF) # We use the oslo.log default log levels which includes suds=INFO # and add only the extra levels that Masakari needs log.set_defaults(default_log_levels=log.get_default_log_levels()) CONF(argv[1:], project='masakarimonitors', version=version.version_string(), default_config_files=default_config_files) ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1892016 masakari_monitors-19.0.0/masakarimonitors/ha/0000775000175000017500000000000000000000000021321 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/ha/__init__.py0000664000175000017500000000000000000000000023420 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/ha/masakari.py0000664000175000017500000000626200000000000023471 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import eventlet from keystoneauth1 import loading as ks_loading from openstack import connection from openstack import exceptions from oslo_log import log as oslo_logging import masakarimonitors.conf LOG = oslo_logging.getLogger(__name__) CONF = masakarimonitors.conf.CONF class SendNotification(object): def __init__(self): self._masakari_client = None @property def masakari_client(self): if not self._masakari_client: self._masakari_client = self._make_client() return self._masakari_client def _make_client(self): auth = ks_loading.load_auth_from_conf_options(CONF, 'api') session = ks_loading.load_session_from_conf_options(CONF, 'api', auth=auth) conn = connection.Connection(session=session, interface=CONF.api.api_interface, region_name=CONF.api.region) return conn.instance_ha def send_notification(self, api_retry_max, api_retry_interval, event): """Send a notification. This method sends a notification to the masakari-api. :param api_retry_max: Number of retries when the notification processing is error. :param api_retry_interval: Trial interval of time of the notification processing is error. :param event: dictionary of event that included in notification. """ LOG.info("Send a notification. %s", event) # Send a notification. retry_count = 0 while True: try: response = self.masakari_client.create_notification( type=event['notification']['type'], hostname=event['notification']['hostname'], generated_time=event['notification']['generated_time'], payload=event['notification']['payload']) LOG.info("Response: %s", response) break except Exception as e: if isinstance(e, exceptions.HttpException): if e.status_code in [400, 409]: LOG.info(e) break if retry_count < api_retry_max: LOG.warning("Retry sending a notification. (%s)", e) retry_count = retry_count + 1 eventlet.greenthread.sleep(api_retry_interval) else: LOG.exception("Exception caught: %s", e) self._masakari_client = None break ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1892016 masakari_monitors-19.0.0/masakarimonitors/hacking/0000775000175000017500000000000000000000000022335 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hacking/__init__.py0000664000175000017500000000000000000000000024434 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hacking/checks.py0000664000175000017500000001035400000000000024152 0ustar00zuulzuul00000000000000# Copyright (c) 2017, NTT Data # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import re from hacking import core """ Guidelines for writing new hacking checks - Use only for masakarimonitors specific tests. OpenStack general tests should be submitted to the common 'hacking' module. - Pick numbers in the range M3xx. Find the current test with the highest allocated number and then pick the next value. - Keep the test method code in the source file ordered based on the M3xx value. - List the new rule in the top level HACKING.rst file - Add test cases for each new rule to masakarimonitors/tests/unit/ test_hacking.py """ UNDERSCORE_IMPORT_FILES = [] translated_log = re.compile( r"(.)*LOG\.(audit|error|info|critical|exception)" r"\(\s*_\(\s*('|\")") string_translation = re.compile(r"[^_]*_\(\s*('|\")") underscore_import_check = re.compile(r"(.)*import _(.)*") underscore_import_check_multi = re.compile(r"(.)*i18n\s+import(.)* _, (.)*") # We need this for cases where they have created their own _ function. custom_underscore_check = re.compile(r"(.)*_\s*=\s*(.)*") log_translation = re.compile( r"(.)*LOG\." r"(audit|debug|error|info|warn|warning|critical|exception)" r"\(" r"(_|_LC|_LE|_LI|_LW)" r"\(") yield_not_followed_by_space = re.compile(r"^\s*yield(?:\(|{|\[|\"|').*$") assert_raises_regexp = re.compile(r"assertRaisesRegexp\(") @core.flake8ext def check_explicit_underscore_import(logical_line, filename): """Check for explicit import of the _ function We need to ensure that any files that are using the _() function to translate logs are explicitly importing the _ function. We can't trust unit test to catch whether the import has been added so we need to check for it here. """ # Build a list of the files that have _ imported. No further # checking needed once it is found. for file in UNDERSCORE_IMPORT_FILES: if file in filename: return if (underscore_import_check.match(logical_line) or underscore_import_check_multi.match(logical_line) or custom_underscore_check.match(logical_line)): UNDERSCORE_IMPORT_FILES.append(filename) elif (translated_log.match(logical_line) or string_translation.match(logical_line)): yield (0, "M301: Found use of _() without explicit import of _ !") @core.flake8ext def no_translate_logs(logical_line): """Check that logging doesn't translate messages M302 """ if log_translation.match(logical_line): yield (0, "M302 Don't translate log messages!") @core.flake8ext def yield_followed_by_space(logical_line): """Yield should be followed by a space. Yield should be followed by a space to clarify that yield is not a function. Adding a space may force the developer to rethink if there are unnecessary parentheses in the written code. Not correct: yield(x), yield(a, b) Correct: yield x, yield (a, b), yield a, b M303 """ if yield_not_followed_by_space.match(logical_line): yield (0, "M303: Yield keyword should be followed by a space.") @core.flake8ext def assert_raisesRegexp(logical_line): """Check that assertRaisesRegexp is not used. M304 """ res = assert_raises_regexp.search(logical_line) if res: yield (0, "M304: assertRaisesRegex must be used instead " "of assertRaisesRegexp") @core.flake8ext def no_log_warn(logical_line): """Disallow 'LOG.warn(' LOG.warn() is deprecated and LOG.warning should be used instead. https://docs.python.org/3/library/logging.html#logging.warning N352 """ msg = ("M305: LOG.warn is deprecated, please use LOG.warning!") if "LOG.warn(" in logical_line: yield (0, msg) ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1892016 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/0000775000175000017500000000000000000000000023316 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/__init__.py0000664000175000017500000000000000000000000025415 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1892016 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/consul_check/0000775000175000017500000000000000000000000025756 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/consul_check/__init__.py0000664000175000017500000000000000000000000030055 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/consul_check/consul_helper.py0000664000175000017500000000747100000000000031203 0ustar00zuulzuul00000000000000# Copyright(c) 2021 Inspur # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Main abstraction layer for retrieving node status from consul """ import consul from masakarimonitors.i18n import _ class ConsulException(Exception): """Base Consul Exception""" msg_fmt = _("An unknown exception occurred.") def __init__(self, message=None, **kwargs): if not message: message = self.msg_fmt % kwargs super(ConsulException, self).__init__(message) class ConsulAgentNotExist(ConsulException): msg_fmt = _("Consul agent of %(cluster)s not exist.") class ConsulGetMembersException(ConsulException): msg_fmt = _("Failed to get members of %(cluster)s: %(err)s.") class ConsulManager(object): """Consul manager class This class helps to pull health data from all consul clusters, and return health data in sequence. """ def __init__(self, CONF): self.agents = {} self.init_agents(CONF) def init_agents(self, CONF): if CONF.consul.agent_manage: addr, port = CONF.consul.agent_manage.split(':') self.agents['manage'] = ConsulAgent('manage', addr, port) if CONF.consul.agent_tenant: addr, port = CONF.consul.agent_tenant.split(':') self.agents['tenant'] = ConsulAgent('tenant', addr, port) if CONF.consul.agent_storage: addr, port = CONF.consul.agent_storage.split(':') self.agents['storage'] = ConsulAgent('storage', addr, port) def valid_agents(self, sequence): for name in sequence: if self.agents.get(name) is None: raise ConsulAgentNotExist(cluster=name) def get_health(self, sequence): hosts_health = {} all_agents = [] for name in sequence: if name in self.agents: consul_agent = self.agents.get(name) agent_health = consul_agent.get_health() hosts_health[name] = agent_health if not all_agents: all_agents = agent_health.keys() sequence_hosts_health = {} for host in all_agents: sequence_hosts_health[host] = [] for name in sequence: if name in self.agents: state = hosts_health[name].get(host) if state: sequence_hosts_health[host].append(state) else: continue return sequence_hosts_health class ConsulAgent(object): """Agent to consul cluster""" def __init__(self, name, addr=None, port=None): self.name = name self.addr = addr self.port = port # connection to consul cluster self.cluster = consul.Consul(host=addr, port=self.port) def get_agents(self): try: members = self.cluster.agent.members() except Exception as e: raise ConsulGetMembersException(cluster=self.name, err=str(e)) return members def get_health(self): agents_health = {} agents = self.get_agents() for agent in agents: host = agent.get('Name') status = agent.get('Status') if status == 1: agents_health[host] = 'up' else: agents_health[host] = 'down' return agents_health ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/consul_check/manager.py0000664000175000017500000001565400000000000027755 0ustar00zuulzuul00000000000000# Copyright(c) 2019 Inspur # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import eventlet import socket from collections import deque from oslo_log import log from oslo_utils import timeutils from masakarimonitors import conf from masakarimonitors.ha import masakari from masakarimonitors.hostmonitor.consul_check import consul_helper from masakarimonitors.hostmonitor.consul_check import matrix_helper from masakarimonitors.hostmonitor import driver from masakarimonitors.objects import event_constants as ec LOG = log.getLogger(__name__) CONF = conf.CONF class ConsulCheck(driver.DriverBase): """Check host status by consul""" def __init__(self): super(ConsulCheck, self).__init__() self.hostname = socket.gethostname() self.monitoring_interval = CONF.host.monitoring_interval self.monitoring_samples = CONF.host.monitoring_samples self.matrix_manager = matrix_helper.MatrixManager(CONF) self.consul_manager = consul_helper.ConsulManager(CONF) self.notifier = masakari.SendNotification() self._matrix = None self._sequence = None self.monitoring_data = {} self.last_host_health = {} @property def matrix(self): if not self._matrix: self._matrix = self.matrix_manager.get_matrix() return self._matrix @property def sequence(self): if not self._sequence: self._sequence = self.matrix_manager.get_sequence() return self._sequence def _formate_health(self, host_health): formate_health = {} for i in range(len(host_health)): layer = "%s-interface" % self.sequence[i] state = host_health[i] formate_health[layer] = state return formate_health def _event(self, host, host_health): host_status = ec.EventConstants.HOST_STATUS_NORMAL if 'down' not in host_health: event_type = ec.EventConstants.EVENT_STARTED cluster_status = ec.EventConstants.CLUSTER_STATUS_ONLINE else: actions = self.get_action_from_matrix(host_health) if 'recovery' in actions: LOG.info("Host %s needs recovery, health status: %s." % (host, str(self._formate_health(host_health)))) event_type = ec.EventConstants.EVENT_STOPPED cluster_status = ec.EventConstants.CLUSTER_STATUS_OFFLINE else: return None # TODO(suzhengwei): Add host status detail in the payload to show # host status in all Consul clusters. event = { 'notification': { 'type': ec.EventConstants.TYPE_COMPUTE_HOST, 'hostname': host, 'generated_time': timeutils.utcnow(), 'payload': { 'event': event_type, 'cluster_status': cluster_status, 'host_status': host_status } } } return event def update_monitoring_data(self): '''update monitoring data from consul clusters''' LOG.debug("update monitoring data from consul.") # Get current host health in sequence [x, y, z]. The example of # the return value is {'node01':[x, y, z], 'node02':[x, y, z]...} cluster_health = self.consul_manager.get_health(self.sequence) LOG.debug("Current cluster state: %s.", cluster_health) # Reassemble host health history with the latest host health. # Example of 'host_health_history' is [[x, y, x], [x, y, z]...] for host, health in cluster_health.items(): host_health_history = self.monitoring_data.get( host, deque([], maxlen=self.monitoring_samples)) host_health_history.append(health) self.monitoring_data[host] = host_health_history def get_host_health(self, host): health_history = self.monitoring_data.get(host, []) if len(health_history) < self.monitoring_samples: LOG.debug("Not enough monitoring data for host %s", host) return None # Caculate host health from host health history. # Only continous 'down' represents the interface 'down', # while continous 'up' represents the interface 'up'. host_sequence_health = [] host_health_history = list(zip(*health_history)) for i in range(0, len(host_health_history)): if ('up' in host_health_history[i] and 'down' in host_health_history[i]): host_sequence_health.append(None) else: host_sequence_health.append(host_health_history[i][0]) return host_sequence_health def _host_health_changed(self, host, health): last_health = self.last_host_health.get(host) if last_health is None: self.last_host_health[host] = health return False if health != last_health: self.last_host_health[host] = health return True else: return False def get_action_from_matrix(self, host_health): for health_action in self.matrix: matrix_health = health_action["health"] matrix_action = health_action["action"] if host_health == matrix_health: return matrix_action return [] def poll_hosts(self): '''poll and check hosts health''' for host in self.monitoring_data.keys(): if host == self.hostname: continue host_health = self.get_host_health(host) if host_health is None: continue if not self._host_health_changed(host, host_health): continue # it will send notifition to trigger host failure recovery # according to defined HA strategy event = self._event(host, host_health) if event: self.notifier.send_notification( CONF.host.api_retry_max, CONF.host.api_retry_interval, event) def stop(self): self.running = False def monitor_hosts(self): self.running = True while self.running: try: self.update_monitoring_data() self.poll_hosts() except Exception as e: LOG.exception("Exception when host-monitor by consul: %s", e) eventlet.greenthread.sleep(self.monitoring_interval) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/consul_check/matrix_helper.py0000664000175000017500000000445100000000000031177 0ustar00zuulzuul00000000000000# Copyright(c) 2021 Inspur # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import os import yaml DEFAULT_SEQUENCE = ['manage', 'tenant', 'storage'] # matrix is combined by health and actions. # health: [x, x, x], repreasents status of DEFAULT_SEQUENCE. # action, means which actions it will trigge if host health turns into. # action choice: 'recovery'. # 'recovery' means it will trigge one host recovery event. DEFAULT_MATRIX = [ {"health": ["up", "up", "up"], "action": []}, {"health": ["up", "up", "down"], "action": ["recovery"]}, {"health": ["up", "down", "up"], "action": []}, {"health": ["up", "down", "down"], "action": ["recovery"]}, {"health": ["down", "up", "up"], "action": []}, {"health": ["down", "up", "down"], "action": ["recovery"]}, {"health": ["down", "down", "up"], "action": []}, {"health": ["down", "down", "down"], "action": ["recovery"]}, ] class MatrixManager(object): """Matrix Manager""" def __init__(self, CONF): cfg_file = CONF.consul.matrix_config_file matrix_conf = self.load_config(cfg_file) if not matrix_conf: self.sequence = DEFAULT_SEQUENCE self.matrix = DEFAULT_MATRIX else: self.sequence = matrix_conf.get("sequence") self.matrix = matrix_conf.get("matrix") self.valid_matrix(self.matrix, self.sequence) def load_config(self, cfg_file): if not cfg_file or not os.path.exists(cfg_file): return None with open(cfg_file) as f: data = f.read() matrix_conf = yaml.safe_load(data) return matrix_conf def get_sequence(self): return self.sequence def get_matrix(self): return self.matrix def valid_matrix(self, matrix, sequence): pass ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/driver.py0000664000175000017500000000161200000000000025163 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import abc class DriverBase(object, metaclass=abc.ABCMeta): """Driver Base class. This class is base of monitoring hosts. """ def __init__(self): pass @abc.abstractmethod def monitor_hosts(self): """Must override monitor_hosts method.""" pass ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/host.py0000664000175000017500000000375000000000000024652 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import os from stevedore import driver from oslo_log import log as oslo_logging import masakarimonitors.conf from masakarimonitors import manager LOG = oslo_logging.getLogger(__name__) CONF = masakarimonitors.conf.CONF class HostmonitorManager(manager.Manager): """Manages the masakari-hostmonitor.""" def __init__(self, *args, **kwargs): super(HostmonitorManager, self).__init__( service_name="hostmonitor", *args, **kwargs) self.driver = None def init_host(self): """Initialization for hostmonitor.""" try: # Determine dynamic load driver from configuration. driver_name = CONF.host.monitoring_driver # Load the driver to global. self.driver = driver.DriverManager( namespace='hostmonitor.driver', name=driver_name, invoke_on_load=True, invoke_args=(), ) except Exception as e: LOG.exception( "Exception caught during initializing hostmonitor: %s", e) os._exit(1) def stop(self): self.driver.driver.stop() def main(self): """Main method.""" try: # Call the host monitoring driver. self.driver.driver.monitor_hosts() except Exception as e: LOG.exception("Exception caught: %s", e) return ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1892016 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/host_handler/0000775000175000017500000000000000000000000025770 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/host_handler/__init__.py0000664000175000017500000000000000000000000030067 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/host_handler/handle_host.py0000664000175000017500000004322200000000000030635 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import socket from collections import deque import eventlet from oslo_log import log as oslo_logging from oslo_utils import timeutils import masakarimonitors.conf from masakarimonitors.ha import masakari import masakarimonitors.hostmonitor.driver as driver from masakarimonitors.hostmonitor.host_handler import hold_host_status from masakarimonitors.hostmonitor.host_handler import parse_cib_xml from masakarimonitors.hostmonitor.host_handler import parse_crmmon_xml from masakarimonitors.objects import event_constants as ec from masakarimonitors import utils LOG = oslo_logging.getLogger(__name__) CONF = masakarimonitors.conf.CONF class CibSchemaCompliantTag(dict): """Create a dict which has the same attributes as a cib node tag. Given a crm node tag convert it to a dict with corresponding cib tag attributes. """ def __init__(self, crmon_entry): self['uname'] = crmon_entry.get('name') online = crmon_entry.get('online') self['crmd'] = 'online' if online == 'true' else 'offline' class HandleHost(driver.DriverBase): """Handle hosts. This class handles the host status. """ def __init__(self): super(HandleHost, self).__init__() self.my_hostname = socket.gethostname() self.xml_parser = parse_cib_xml.ParseCibXml() self.crmmon_xml_parser = parse_crmmon_xml.ParseCrmMonXml() self.status_holder = hold_host_status.HostHoldStatus() self.notifier = masakari.SendNotification() self.monitoring_data = {} def _update_monitoring_data(self, hostname, status): health_history = self.monitoring_data.setdefault( hostname, deque([], maxlen=CONF.host.monitoring_samples)) health_history.append(status) def get_stabilised_host_status(self, hostname): health_history = self.monitoring_data.get(hostname) if len(health_history) < CONF.host.monitoring_samples: LOG.debug("Not enough monitoring data for host %s.", hostname) return '_being_collected' stabilised_status = health_history[0] # If and only if the sequence of host status is consistently the same, # will it return that status. if len(health_history) == health_history.count(stabilised_status): return stabilised_status else: return '_uncertain' def _check_pacemaker_services(self, target_service): try: cmd_str = 'systemctl status ' + target_service command = cmd_str.split() # Execute command. out, err = utils.execute(*command, run_as_root=True) if err: raise Exception return True except Exception: return False def _check_hb_line(self): """Check whether the corosync communication is normal. :returns: 0 if normal, 1 if abnormal, 2 if configuration file is wrong or neither pacemaker nor pacemaker-remote is running. """ if CONF.host.pacemaker_node_type == 'autodetect': # Check whether the Corosync/Pacemaker services are running. corosync_status = self._check_pacemaker_services('corosync') pacemaker_status = self._check_pacemaker_services('pacemaker') pacemaker_remote_status = self._check_pacemaker_services( 'pacemaker_remote') else: corosync_status = ( CONF.host.pacemaker_node_type == 'cluster') pacemaker_status = ( CONF.host.pacemaker_node_type == 'cluster') pacemaker_remote_status = ( CONF.host.pacemaker_node_type == 'remote') if corosync_status is False or pacemaker_status is False: if pacemaker_remote_status is False: LOG.error( "Neither pacemaker nor pacemaker-remote is running.") return 2 else: LOG.info("Works on pacemaker-remote.") return 0 # Check whether the necessary parameters are set. if CONF.host.corosync_multicast_interfaces is None or \ CONF.host.corosync_multicast_ports is None: msg = ("corosync_multicast_interfaces or " "corosync_multicast_ports is not set.") LOG.error("%s", msg) return 2 # Check whether the corosync communication is normal. corosync_multicast_interfaces = \ CONF.host.corosync_multicast_interfaces.split(',') corosync_multicast_ports = \ CONF.host.corosync_multicast_ports.split(',') if len(corosync_multicast_interfaces) != len(corosync_multicast_ports): msg = ("Incorrect parameters corosync_multicast_interfaces or " "corosync_multicast_ports.") LOG.error("%s", msg) return 2 is_nic_normal = False for num in range(0, len(corosync_multicast_interfaces)): cmd_str = ("timeout %s tcpdump -n -c 1 -p -i %s port %s") \ % (CONF.host.tcpdump_timeout, corosync_multicast_interfaces[num], corosync_multicast_ports[num]) command = cmd_str.split() try: # Execute tcpdump command. out, err = utils.execute(*command, run_as_root=True) # If command doesn't raise exception, nic is normal. msg = ("Corosync communication using '%s' is normal.") \ % corosync_multicast_interfaces[num] LOG.info("%s", msg) is_nic_normal = True break except Exception: msg = ("Corosync communication using '%s' is failed.") \ % corosync_multicast_interfaces[num] LOG.warning("%s", msg) if is_nic_normal is False: LOG.error("Corosync communication is failed.") return 1 return 0 def _check_host_status_by_crmadmin(self): try: # Execute crmadmin command. out, err = utils.execute('crmadmin', '-S', self.my_hostname, run_as_root=True) if err: msg = ("crmadmin command output stderr: %s") % err raise Exception(msg) # If own host is stable status, crmadmin outputs # 'S_IDLE' or 'S_NOT_DC' if 'S_IDLE' in out or 'S_NOT_DC' in out: return 0 else: raise Exception( "crmadmin command output unexpected host status.") except Exception as e: LOG.warning("Exception caught: %s", e) LOG.warning("'%s' is unstable state on cluster.", self.my_hostname) return 1 def _get_cib_xml(self): try: # Execute cibadmin command. out, err = utils.execute('cibadmin', '--query', run_as_root=True) if err: msg = ("cibadmin command output stderr: %s") % err raise Exception(msg) except Exception as e: LOG.warning("Exception caught: %s", e) return return out def _get_crmmon_xml(self): """Get summary of cluster's current state in XML format.""" try: # Execute crm_mon command. out, err = utils.execute('crm_mon', '-X', run_as_root=True) if err: msg = ("crmmon command output stderr: %s") % err raise Exception(msg) except Exception as e: LOG.warning("Exception caught: %s", e) return return out def _is_poweroff(self, hostname): ipmi_values = self.xml_parser.get_stonith_ipmi_params(hostname) if ipmi_values is None: LOG.error("Failed to get params of ipmi RA.") return False cmd_str = ("timeout %s ipmitool -U %s -P %s -I %s -H %s " "power status") \ % (str(CONF.host.ipmi_timeout), ipmi_values['userid'], ipmi_values['passwd'], ipmi_values['interface'], ipmi_values['ipaddr']) command = cmd_str.split() retry_count = 0 while True: try: # Execute ipmitool command. out, err = utils.execute(*command, run_as_root=False) if err: msg = ("ipmitool command output stderr: %s") % err raise Exception(msg) msg = ("ipmitool command output stdout: %s") % out if 'Power is off' in out: LOG.info("%s", msg) return True else: raise Exception(msg) except Exception as e: if retry_count < CONF.host.ipmi_retry_max: LOG.warning("Retry executing ipmitool command. (%s)", e) retry_count = retry_count + 1 eventlet.greenthread.sleep(CONF.host.ipmi_retry_interval) else: LOG.error("Exception caught: %s", e) return False def _make_event(self, hostname, current_status): if current_status == 'online': # Set values that host has started. event_type = ec.EventConstants.EVENT_STARTED cluster_status = current_status.upper() host_status = ec.EventConstants.HOST_STATUS_NORMAL else: # Set values that host has stopped. event_type = ec.EventConstants.EVENT_STOPPED cluster_status = current_status.upper() if not CONF.host.disable_ipmi_check: if self._is_poweroff(hostname): # Set value that host status is normal. host_status = ec.EventConstants.HOST_STATUS_NORMAL else: # Set value that host status is unknown. host_status = ec.EventConstants.HOST_STATUS_UNKNOWN else: # Set value that host status is normal. host_status = ec.EventConstants.HOST_STATUS_NORMAL current_time = timeutils.utcnow() event = { 'notification': { 'type': ec.EventConstants.TYPE_COMPUTE_HOST, 'hostname': hostname, 'generated_time': current_time, 'payload': { 'event': event_type, 'cluster_status': cluster_status, 'host_status': host_status } } } return event def _check_if_status_changed(self, node_state_tag_list): # Check if host status changed. for node_state_tag in node_state_tag_list: hostname = node_state_tag.get('uname') # hostmonitor doesn't monitor itself. if hostname == self.my_hostname: continue current_status = node_state_tag.get('crmd') self._update_monitoring_data(hostname, current_status) old_status = self.status_holder.get_host_status(hostname) # If old_status is None, This is first get of host status. if old_status is None: msg = ("Recognized '%s' as a new member of cluster." " Host status is '%s'.") \ % (hostname, current_status) LOG.info("%s", msg) self.status_holder.set_host_status(node_state_tag) continue stabilised_status = self.get_stabilised_host_status(hostname) # Output host status. msg = ("'%s' is '%s' (current: '%s').") % (hostname, stabilised_status, current_status) LOG.info("%s", msg) if stabilised_status == '_being_collected': continue # If host stabilised status changed, send a notification. if stabilised_status != old_status: if stabilised_status not in ['online', 'offline']: # If stabilised_status is not 'online' or 'offline', # hostmonitor doesn't send a notification. msg = ("Since host status is '%s'," " hostmonitor doesn't send a notification.") \ % stabilised_status LOG.info("%s", msg) else: event = self._make_event(hostname, stabilised_status) # Send a notification. self.notifier.send_notification( CONF.host.api_retry_max, CONF.host.api_retry_interval, event) if stabilised_status != '_uncertain': # Update host status. self.status_holder.set_host_status(node_state_tag) def _check_host_status_by_crm_mon(self): crmmon_xml = self._get_crmmon_xml() if crmmon_xml is None: # crm_mon command failure. return 1 # Set to the ParseCrmMonXml object. self.crmmon_xml_parser.set_crmmon_xml(crmmon_xml) # Check if the cluster has quorum. if not self.crmmon_xml_parser.has_quorum(): msg = "Pacemaker cluster doesn't have quorum." LOG.warning("%s", msg) return 2 # Get node_state tag list. node_state_tag_list = self.crmmon_xml_parser.get_node_state_tag_list() if len(node_state_tag_list) == 0: # If crmmon xml doesn't have node_state tag, # it is an unexpected result. raise Exception( "Failed to get nodes tag from crm_mon xml.") node_state_tag_list = [CibSchemaCompliantTag(n) for n in node_state_tag_list if n.get('type') == 'remote'] # Check if status changed. self._check_if_status_changed(node_state_tag_list) return 0 def _check_host_status_by_cibadmin(self): # Get xml of cib info. cib_xml = self._get_cib_xml() if cib_xml is None: # cibadmin command failure. return 1 # Set to the ParseCibXml object. self.xml_parser.set_cib_xml(cib_xml) # Check if pacemaker cluster have quorum. if self.xml_parser.have_quorum() == 0: msg = "Pacemaker cluster doesn't have quorum." LOG.warning("%s", msg) return 2 # Get node_state tag list. node_state_tag_list = self.xml_parser.get_node_state_tag_list() if len(node_state_tag_list) == 0: # If cib xml doesn't have node_state tag, # it is an unexpected result. raise Exception( "Failed to get node_state tag from cib xml.") # Check if status changed. self._check_if_status_changed(node_state_tag_list) return 0 def stop(self): self.running = False def monitor_hosts(self): """Host monitoring main method. This method monitors hosts. """ self.running = True while self.running: try: # Check whether corosync communication between hosts # is normal. ret = self._check_hb_line() if ret == 1: # Because my host may be fenced by stonith due to split # brain condition, sleep for a certain time. eventlet.greenthread.sleep(CONF.host.stonith_wait) elif ret == 2: LOG.warning("hostmonitor skips monitoring hosts.") eventlet.greenthread.sleep(CONF.host.monitoring_interval) continue if CONF.host.pacemaker_node_type == 'autodetect': pacemaker_remote_status = self._check_pacemaker_services( 'pacemaker_remote') else: pacemaker_remote_status = ( CONF.host.pacemaker_node_type == 'remote') # Check the host status is stable or unstable by crmadmin. # It only checks when this process runs on the full cluster # stack of corosync. if pacemaker_remote_status is False: if self._check_host_status_by_crmadmin() != 0: LOG.warning("hostmonitor skips monitoring hosts.") eventlet.greenthread.sleep( CONF.host.monitoring_interval) continue # Check the host status is online or offline. if CONF.host.restrict_to_remotes: status_func = self._check_host_status_by_crm_mon else: status_func = self._check_host_status_by_cibadmin if status_func() != 0: LOG.warning("hostmonitor skips monitoring hosts.") except Exception as e: LOG.exception("Exception caught: %s", e) eventlet.greenthread.sleep(CONF.host.monitoring_interval) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/host_handler/hold_host_status.py0000664000175000017500000000265600000000000031741 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. class HostHoldStatus(object): """Hold host status. This class holds the host status. """ def __init__(self): # host_status is dictionary like {'hostname': 'online'}. self.host_status = {} def set_host_status(self, node_state_tag): """Setter method. This method set host status by node_state tag of cib xml. :params node_state_tag: node_state tag of cib xml. """ self.host_status[node_state_tag.get('uname')] = \ node_state_tag.get('crmd') def get_host_status(self, hostname): """Getter method. This method returns the requested host status. host status is 'online' or 'offline'. :params hostname: Hostname to get status. :returns: Host status. """ return self.host_status.get(hostname) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/host_handler/parse_cib_xml.py0000664000175000017500000001424300000000000031155 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from xml.etree import ElementTree from oslo_log import log as oslo_logging LOG = oslo_logging.getLogger(__name__) class ParseCibXml(object): """ParseCibXml class This class parses the cib xml. """ def __init__(self): self.cib_tag = None def set_cib_xml(self, cib_xml): """Set xml.etree.ElementTree.Element object. This method recieves string of cib xml, and convert it to xml.etree.ElementTree.Element object. :params cib_xml: String of cib xml """ # Convert xml.etree.ElementTree.Element object. self.cib_tag = ElementTree.fromstring(cib_xml) def have_quorum(self): """Returns if cluster has quorum or not. :returns: 0 on no-quorum, 1 if cluster has quorum. """ return int(self.cib_tag.get('have-quorum')) def _get_status_tag(self): # status tag exists in the cib tag. child_list = list(self.cib_tag) for child in child_list: if child.tag == 'status': return child return None def _get_node_states(self, status_tag): node_state_tag_list = [] # node_state tag exists in the status tag. child_list = list(status_tag) for child in child_list: if child.tag == 'node_state': node_state_tag_list.append(child) return node_state_tag_list def get_node_state_tag_list(self): """Get node_state tag list. This method gets node_state tag list from cib xml. :returns: node_state tag list """ # Get status tag. status_tag = self._get_status_tag() if status_tag is None: LOG.error("Cib xml doesn't have status tag.") return [] # Get node_state tag list. node_state_tag_list = self._get_node_states(status_tag) if len(node_state_tag_list) == 0: LOG.error("Cib xml doesn't have node_state tag.") return node_state_tag_list def _parse_instance_attributes_tag(self, instance_attributes_tag, hostname): # Parse nvpair tag under the instance_attributes tag. is_target_ipmi = False ipmi_values = {} nvpair_tag_list = list(instance_attributes_tag) for nvpair_tag in nvpair_tag_list: if nvpair_tag.get('name') == 'hostname' and \ nvpair_tag.get('value') == hostname: is_target_ipmi = True elif nvpair_tag.get('name') == 'ipaddr': ipmi_values['ipaddr'] = nvpair_tag.get('value') elif nvpair_tag.get('name') == 'userid': ipmi_values['userid'] = nvpair_tag.get('value') elif nvpair_tag.get('name') == 'passwd': ipmi_values['passwd'] = nvpair_tag.get('value') elif nvpair_tag.get('name') == 'interface': ipmi_values['interface'] = nvpair_tag.get('value') if is_target_ipmi is True: return ipmi_values else: return None def _parse_primitive_tag(self, primitive_tag, hostname): if primitive_tag.get('type') != 'external/ipmi': return None # Parse instance_attributes tag under the primitive tag. child_list = list(primitive_tag) for child in child_list: if child.tag == 'instance_attributes': ipmi_values = self._parse_instance_attributes_tag( child, hostname) if ipmi_values is not None: return ipmi_values return None def _parse_group_tag(self, group_tag, hostname): # Parse primitive tag under the group tag. child_list = list(group_tag) for child in child_list: if child.tag == 'primitive': ipmi_values = self._parse_primitive_tag(child, hostname) if ipmi_values is not None: return ipmi_values return None def get_stonith_ipmi_params(self, hostname): """Get stonith ipmi params from cib xml. This method gets params of ipmi resource agent(RA) which is set on resources tag. The resources tag exists under the configuration tag. And it is assumed that ipmi RA belongs to some resource group. :params hostname: hostname :returns: Dictionary of ipmi RA's params. They are ipaddr, userid, passwd and interface. """ # Get configuration tag from cib tag. configuration_tag = None child_list = list(self.cib_tag) for child in child_list: if child.tag == 'configuration': configuration_tag = child break if configuration_tag is None: LOG.error("Cib xml doesn't have configuration tag.") return None # Get resources tag from configuration tag. resources_tag = None child_list = list(configuration_tag) for child in child_list: if child.tag == 'resources': resources_tag = child break if resources_tag is None: LOG.error("Cib xml doesn't have resources tag.") return None # They are set at nvpair tag which exists under the # instance_attributes of primitive of group tag. ipmi_values = None child_list = list(resources_tag) for child in child_list: if child.tag == 'group': ipmi_values = self._parse_group_tag(child, hostname) if ipmi_values is not None: break return ipmi_values ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/hostmonitor/host_handler/parse_crmmon_xml.py0000664000175000017500000000606100000000000031712 0ustar00zuulzuul00000000000000# Copyright(c) 2019 Canonical Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from xml.etree import ElementTree from oslo_log import log as oslo_logging LOG = oslo_logging.getLogger(__name__) class ParseCrmMonXml(object): """ParseCrmMonXml class This class parses the crmmon xml. """ def __init__(self): self.crmmon_tag = None def set_crmmon_xml(self, crmmon_xml): """Set xml.etree.ElementTree.Element object. This method receives string of crmmon xml, and convert it to xml.etree.ElementTree.Element object. :params crmmon_xml: String of crmmon xml """ # Convert xml.etree.ElementTree.Element object. self.crmmon_tag = ElementTree.fromstring(crmmon_xml) def has_quorum(self): """Answers if cluster has quorum or not. :returns: True if cluster has quorum, False otherwise. """ current_dc = self._get_current_dc() return current_dc.get('with_quorum') == 'true' def _get_summary(self): child_list = list(self.crmmon_tag) for child in child_list: if child.tag == 'summary': return child return None def _get_current_dc(self): child_list = list(self._get_summary()) for child in child_list: if child.tag == 'current_dc': return child return None def _get_nodes(self): # status tag exists in the crmmon tag. if self.crmmon_tag is None: return None child_list = list(self.crmmon_tag) for child in child_list: if child.tag == 'nodes': return child return None def _get_node_states(self, nodes_tag): node_state_tag_list = [] # node_state tag exists in the status tag. child_list = list(nodes_tag) for child in child_list: if child.tag == 'node': node_state_tag_list.append(child) return node_state_tag_list def get_node_state_tag_list(self): """Get node_state tag list. This method gets node_state tag list from crmmon xml. :returns: node_state tag list """ # Get status tag. nodes_tag = self._get_nodes() if nodes_tag is None: LOG.error("crm_mon xml doesn't have nodes tag.") return [] # Get node_state tag list. node_state_tag_list = self._get_node_states(nodes_tag) if len(node_state_tag_list) == 0: LOG.error("crm_mon xml doesn't have online tag.") return node_state_tag_list ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/i18n.py0000664000175000017500000000165700000000000022073 0ustar00zuulzuul00000000000000# Copyright 2016 NTT DATA # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import oslo_i18n DOMAIN = 'masakarimonitors' _translators = oslo_i18n.TranslatorFactory(domain=DOMAIN) # The primary translation function using the well-known name "_" _ = _translators.primary def translate(value, user_locale): return oslo_i18n.translate(value, user_locale) def get_available_languages(): return oslo_i18n.get_available_languages(DOMAIN) ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1892016 masakari_monitors-19.0.0/masakarimonitors/instancemonitor/0000775000175000017500000000000000000000000024145 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/instancemonitor/__init__.py0000664000175000017500000000000000000000000026244 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/instancemonitor/instance.py0000664000175000017500000001655700000000000026341 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import threading import time import eventlet import libvirt from oslo_config import cfg from oslo_log import log as oslo_logging from masakarimonitors.instancemonitor.libvirt_handler import eventfilter from masakarimonitors import manager LOG = oslo_logging.getLogger(__name__) CONF = cfg.CONF class InstancemonitorManager(manager.Manager): """Manages the masakari-instancemonitor.""" def __init__(self, *args, **kwargs): super(InstancemonitorManager, self).__init__( service_name="instancemonitor", *args, **kwargs) self.evf = eventfilter.EventFilter() # This keeps track of what thread is running the event loop, # (if it is run in a background thread) self.event_loop_thread = None def _vir_event_loop_native_run(self): # Directly run the event loop in the current thread while True: libvirt.virEventRunDefaultImpl() def _vir_event_loop_native_start(self): libvirt.virEventRegisterDefaultImpl() self.event_loop_thread = threading.Thread( target=self._vir_event_loop_native_run, name="lib_virt_eventLoop") self.event_loop_thread.daemon = True self.event_loop_thread.start() def _my_domain_event_callback(self, conn, dom, event, detail, opaque): self.evf.vir_event_filter(libvirt.VIR_DOMAIN_EVENT_ID_LIFECYCLE, event, detail, dom.UUIDString()) def _my_domain_event_reboot_callback(self, conn, dom, opaque): self.evf.vir_event_filter(libvirt.VIR_DOMAIN_EVENT_ID_REBOOT, -1, -1, dom.UUIDString()) def _my_domain_event_rtc_change_callback(self, conn, dom, utcoffset, opaque): self.evf.vir_event_filter(libvirt.VIR_DOMAIN_EVENT_ID_RTC_CHANGE, -1, -1, dom.UUIDString()) def _my_domain_event_watchdog_callback(self, conn, dom, action, opaque): self.evf.vir_event_filter(libvirt.VIR_DOMAIN_EVENT_ID_WATCHDOG, action, -1, dom.UUIDString()) def _my_domain_event_io_error_callback(self, conn, dom, srcpath, devalias, action, opaque): self.evf.vir_event_filter(libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR, action, -1, dom.UUIDString()) def _my_domain_event_graphics_callback(self, conn, dom, phase, localAddr, remoteAddr, authScheme, subject, opaque): self.evf.vir_event_filter(libvirt.VIR_DOMAIN_EVENT_ID_GRAPHICS, -1, phase, dom.UUIDString()) def _my_domain_event_disk_change_callback(self, conn, dom, oldSrcPath, newSrcPath, devAlias, reason, opaque): self.evf.vir_event_filter(libvirt.VIR_DOMAIN_EVENT_ID_DISK_CHANGE, -1, -1, dom.UUIDString()) def _my_domain_event_io_error_reason_callback(self, conn, dom, srcPath, devAlias, action, reason, opaque): self.evf.vir_event_filter(libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR_REASON, -1, -1, dom.UUIDString()) def _my_domain_event_generic_callback(self, conn, dom, opaque): self.evf.vir_event_filter(libvirt.VIR_DOMAIN_EVENT_ID_CONTROL_ERROR, -1, -1, dom.UUIDString()) def _err_handler(self, ctxt, err): LOG.warning("Error from libvirt : %s", err[2]) @staticmethod def _connect_auth_cb(creds, user_data): if len(creds) == 0: return 0 raise Exception("Can not handle authentication request for %d " "credentials" % len(creds)) def _virt_event(self, uri): auth = [[libvirt.VIR_CRED_AUTHNAME, libvirt.VIR_CRED_ECHOPROMPT, libvirt.VIR_CRED_REALM, libvirt.VIR_CRED_PASSPHRASE, libvirt.VIR_CRED_NOECHOPROMPT, libvirt.VIR_CRED_EXTERNAL], InstancemonitorManager._connect_auth_cb, None] flags = libvirt.VIR_CONNECT_RO # Run a background thread with the event loop self._vir_event_loop_native_start() event_callback_handlers = { libvirt.VIR_DOMAIN_EVENT_ID_LIFECYCLE: self._my_domain_event_callback, libvirt.VIR_DOMAIN_EVENT_ID_REBOOT: self._my_domain_event_reboot_callback, libvirt.VIR_DOMAIN_EVENT_ID_RTC_CHANGE: self._my_domain_event_rtc_change_callback, libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR: self._my_domain_event_io_error_callback, libvirt.VIR_DOMAIN_EVENT_ID_WATCHDOG: self._my_domain_event_watchdog_callback, libvirt.VIR_DOMAIN_EVENT_ID_GRAPHICS: self._my_domain_event_graphics_callback, libvirt.VIR_DOMAIN_EVENT_ID_DISK_CHANGE: self._my_domain_event_disk_change_callback, libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR_REASON: self._my_domain_event_io_error_reason_callback, libvirt.VIR_DOMAIN_EVENT_ID_CONTROL_ERROR: self._my_domain_event_generic_callback } # Connect to libvirt - If be disconnected, reprocess. self.running = True while self.running: vc = libvirt.openAuth(uri, auth, flags) # Event callback settings callback_ids = [] for event, callback in event_callback_handlers.items(): cid = vc.domainEventRegisterAny(None, event, callback, None) callback_ids.append(cid) # Connection monitoring. vc.setKeepAlive(5, 3) while vc.isAlive() == 1 and self.running: eventlet.greenthread.sleep(1) # If connection between libvirtd was lost, # clear callback connection. LOG.warning("Libvirt Connection Closed Unexpectedly.") for cid in callback_ids: try: vc.domainEventDeregisterAny(cid) except Exception: pass vc.close() del vc time.sleep(3) def stop(self): self.running = False def main(self): """Main method. Set the URI, error handler, and executes event loop processing. """ uri = CONF.libvirt.connection_uri LOG.debug("Using uri:" + uri) # set error handler & do event loop libvirt.registerErrorHandler(self._err_handler, '_virt_event') self._virt_event(uri) ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1892016 masakari_monitors-19.0.0/masakarimonitors/instancemonitor/libvirt_handler/0000775000175000017500000000000000000000000027315 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/instancemonitor/libvirt_handler/__init__.py0000664000175000017500000000000000000000000031414 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/instancemonitor/libvirt_handler/callback.py0000664000175000017500000000560500000000000031431 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_log import log as oslo_logging import masakarimonitors.conf from masakarimonitors.ha import masakari LOG = oslo_logging.getLogger(__name__) CONF = masakarimonitors.conf.CONF class Callback(object): """Class of callback processing.""" def __init__(self): self.notifier = masakari.SendNotification() def libvirt_event_callback(self, event_id, details, domain_uuid, notice_type, hostname, current_time): """Callback method. Callback processing be executed as result of the libvirt event filter. :param event_id: Event ID notify to the callback function :param details: Event code notify to the callback function :param domain_uuid: Uuid notify to the callback function :param notice_type: Notice type notify to the callback function :param hostname: Server host name of the source an event occur notify to the callback function :param current_time: Event occurred time notify to the callback function """ # Output to the syslog. LOG.info("Libvirt Event: type=%(notice_type)s," " hostname=%(hostname)s," " uuid=%(uuid)s, time=%(current_time)s," " event_id=%(event_id)s," " detail=%(detail)s)" % {'notice_type': notice_type, 'hostname': hostname, 'uuid': domain_uuid, 'current_time': current_time, 'event_id': event_id, 'detail': details}) # Set the event to the dictionary. event = { 'notification': { 'type': notice_type, 'hostname': hostname, 'generated_time': current_time, 'payload': { 'event': event_id, 'instance_uuid': domain_uuid, 'vir_domain_event': details } } } self.notifier.send_notification(CONF.callback.retry_max, CONF.callback.retry_interval, event) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/instancemonitor/libvirt_handler/eventfilter.py0000664000175000017500000000567300000000000032231 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import socket import sys import threading from oslo_log import log as oslo_logging from oslo_utils import excutils from oslo_utils import timeutils import masakarimonitors.conf from masakarimonitors.instancemonitor.libvirt_handler import callback from masakarimonitors.instancemonitor.libvirt_handler \ import eventfilter_table as evft from masakarimonitors.objects import event_constants as ec LOG = oslo_logging.getLogger(__name__) CONF = masakarimonitors.conf.CONF class EventFilter(object): """Class of filtering events.""" def __init__(self): self.callback = callback.Callback() def vir_event_filter(self, eventID, eventType, detail, uuID): """Filter events from libvirt. :param eventID: EventID :param eventType: Event type :param detail: Event name :pram uuID: UUID """ noticeType = ec.EventConstants.TYPE_VM hostname = CONF.hostname or socket.gethostname() currentTime = timeutils.utcnow() # All Event Output if debug mode is on. msg = "libvirt Event Received.type = %s \ hostname = %s uuid = %s time = %s eventID = %d eventType = %d \ detail = %d" % ( noticeType, hostname, uuID, currentTime, eventID, eventType, detail) LOG.debug(msg) try: if detail in evft.event_filter_dic[eventID][eventType]: LOG.debug("Event Filter Matched.") eventID_val = evft.eventID_dic[eventID] detail_val = evft.detail_dic[eventID][eventType][detail] # callback Thread Start thread = threading.Thread( target=self.callback.libvirt_event_callback, args=(eventID_val, detail_val, uuID, noticeType, hostname, currentTime) ) thread.start() else: LOG.debug("Event Filter Unmatched.") except KeyError: LOG.debug("virEventFilter KeyError") except IndexError: LOG.debug("virEventFilter IndexError") except TypeError: LOG.debug("virEventFilter TypeError") except Exception: with excutils.save_and_reraise_exception(): LOG.debug("Unexpected error: %s" % sys.exc_info()[0]) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/instancemonitor/libvirt_handler/eventfilter_table.py0000664000175000017500000001105700000000000033371 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import libvirt # If is not defined internal , -1 is stored. DUMMY = -1 # Enumerate all event that can get. # Comment out events that is not targeted in the callback. event_filter_dic = { libvirt.VIR_DOMAIN_EVENT_ID_LIFECYCLE: { libvirt.VIR_DOMAIN_EVENT_SUSPENDED: ( libvirt.VIR_DOMAIN_EVENT_SUSPENDED_IOERROR, libvirt.VIR_DOMAIN_EVENT_SUSPENDED_WATCHDOG, libvirt.VIR_DOMAIN_EVENT_SUSPENDED_API_ERROR ), libvirt.VIR_DOMAIN_EVENT_STOPPED: ( libvirt.VIR_DOMAIN_EVENT_STOPPED_SHUTDOWN, libvirt.VIR_DOMAIN_EVENT_STOPPED_DESTROYED, libvirt.VIR_DOMAIN_EVENT_STOPPED_FAILED, ), libvirt.VIR_DOMAIN_EVENT_SHUTDOWN: ( libvirt.VIR_DOMAIN_EVENT_SHUTDOWN_FINISHED, ) }, libvirt.VIR_DOMAIN_EVENT_ID_REBOOT: {DUMMY: (DUMMY,)}, libvirt.VIR_DOMAIN_EVENT_ID_WATCHDOG: { libvirt.VIR_DOMAIN_EVENT_WATCHDOG_NONE: (DUMMY,), libvirt.VIR_DOMAIN_EVENT_WATCHDOG_PAUSE: (DUMMY,), libvirt.VIR_DOMAIN_EVENT_WATCHDOG_RESET: (DUMMY,), libvirt.VIR_DOMAIN_EVENT_WATCHDOG_POWEROFF: (DUMMY,), libvirt.VIR_DOMAIN_EVENT_WATCHDOG_SHUTDOWN: (DUMMY,), libvirt.VIR_DOMAIN_EVENT_WATCHDOG_DEBUG: (DUMMY,) }, libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR: { libvirt.VIR_DOMAIN_EVENT_IO_ERROR_NONE: (DUMMY,), libvirt.VIR_DOMAIN_EVENT_IO_ERROR_PAUSE: (DUMMY,), libvirt.VIR_DOMAIN_EVENT_IO_ERROR_REPORT: (DUMMY,) }, libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR_REASON: {DUMMY: (DUMMY,)}, libvirt.VIR_DOMAIN_EVENT_ID_CONTROL_ERROR: {DUMMY: (DUMMY,)} } eventID_dic = { libvirt.VIR_DOMAIN_EVENT_ID_LIFECYCLE: 'LIFECYCLE', libvirt.VIR_DOMAIN_EVENT_ID_REBOOT: 'REBOOT', libvirt.VIR_DOMAIN_EVENT_ID_WATCHDOG: 'WATCHDOG', libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR: 'IO_ERROR', libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR_REASON: 'IO_ERROR_REASON', libvirt.VIR_DOMAIN_EVENT_ID_CONTROL_ERROR: 'CONTROL_ERROR'} detail_dic = { libvirt.VIR_DOMAIN_EVENT_ID_LIFECYCLE: { libvirt.VIR_DOMAIN_EVENT_SUSPENDED: { libvirt.VIR_DOMAIN_EVENT_SUSPENDED_IOERROR: 'SUSPENDED_IOERROR', libvirt.VIR_DOMAIN_EVENT_SUSPENDED_WATCHDOG: 'SUSPENDED_WATCHDOG', libvirt.VIR_DOMAIN_EVENT_SUSPENDED_API_ERROR: 'SUSPENDED_API_ERROR'}, libvirt.VIR_DOMAIN_EVENT_STOPPED: { libvirt.VIR_DOMAIN_EVENT_STOPPED_SHUTDOWN: 'STOPPED_SHUTDOWN', libvirt.VIR_DOMAIN_EVENT_STOPPED_DESTROYED: 'STOPPED_DESTROYED', libvirt.VIR_DOMAIN_EVENT_STOPPED_FAILED: 'STOPPED_FAILED'}, libvirt.VIR_DOMAIN_EVENT_SHUTDOWN: { libvirt.VIR_DOMAIN_EVENT_SHUTDOWN_FINISHED: 'SHUTDOWN_FINISHED'} }, libvirt.VIR_DOMAIN_EVENT_ID_REBOOT: { DUMMY: { DUMMY: 'UNKNOWN'}}, libvirt.VIR_DOMAIN_EVENT_ID_WATCHDOG: { libvirt.VIR_DOMAIN_EVENT_WATCHDOG_NONE: { DUMMY: 'WATCHDOG_NONE'}, libvirt.VIR_DOMAIN_EVENT_WATCHDOG_PAUSE: { DUMMY: 'WATCHDOG_PAUSE'}, libvirt.VIR_DOMAIN_EVENT_WATCHDOG_RESET: { DUMMY: 'WATCHDOG_RESET'}, libvirt.VIR_DOMAIN_EVENT_WATCHDOG_POWEROFF: { DUMMY: 'WATCHDOG_POWEROFF'}, libvirt.VIR_DOMAIN_EVENT_WATCHDOG_SHUTDOWN: { DUMMY: 'WATCHDOG_SHUTDOWN'}, libvirt.VIR_DOMAIN_EVENT_WATCHDOG_DEBUG: { DUMMY: 'WATCHDOG_DEBUG'}}, libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR: { libvirt.VIR_DOMAIN_EVENT_IO_ERROR_NONE: { DUMMY: 'IO_ERROR_NONE'}, libvirt.VIR_DOMAIN_EVENT_IO_ERROR_PAUSE: { DUMMY: 'IO_ERROR_PAUSE'}, libvirt.VIR_DOMAIN_EVENT_IO_ERROR_REPORT: { DUMMY: 'IO_ERROR_REPORT'}}, libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR_REASON: { DUMMY: { DUMMY: 'UNKNOWN'}}, libvirt.VIR_DOMAIN_EVENT_ID_CONTROL_ERROR: { DUMMY: { DUMMY: 'UNKNOWN'}} } ././@PaxHeader0000000000000000000000000000003300000000000011451 xustar000000000000000027 mtime=1743591461.193202 masakari_monitors-19.0.0/masakarimonitors/introspectiveinstancemonitor/0000775000175000017500000000000000000000000026764 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/introspectiveinstancemonitor/README.rst0000664000175000017500000000522100000000000030453 0ustar00zuulzuul00000000000000========================================= masakarimonitors-introspectiveinstancemonitor ========================================= Introspective instance monitor for Masakari ---------------------------------------- - masakarimonitors-introspectiveinstancemonitor, provides Virtual Machine High Availability (VMHA) service for OpenStack clouds by automatically detecting the system-level failure events via QEMU Guest Agent. If it detects VM heartbeat failure events, it sends notifications to the masakari-api. - Based on the QEMU Guest Agent, masakarimonitors-introspectiveinstancemonitor aims to provide access to a system-level agent via standard qemu-ga protocol How does it work? ---------------------------------------- - libvirt and QEMU Guest Agent are used as the underlying protocol for messaging to and from VM. - The host-side qemu-agent sockets are used to determine whether VMs are configured with QEMU Guest Agent. - qemu-guest-ping is used as the monitoring heartbeat. - For the future release, we can pass through arbitrary guest agent commands to check the health of the applications inside a VM. QEMU Guest Agent Installation notes ---------------------------------------- - Set image property: hw_qemu_guest_agent=yes. - This tells NOVA to setup the virtual serial interface thru QEMU to VM - e.g. $ openstack image create --public --disk-format qcow2 --container-format bare --file ~ubuntu/xenial-server-cloudimg-amd64-disk1.img --public --property hw_qemu_guest_agent=yes xenial-server-cloudimg * Inside VM:: $ sudo apt-get install qemu-guest-agent $ sudo systemctl start qemu-guest-agent $ ubuntu@test:~$ ps -ef | fgrep qemu $ ... /usr/sbin/qemu-ga --daemonize -m virtio-serial -p /dev/virtio-ports/org.qemu.guest_agent.0 $ ubuntu@test:~$ ls /dev/virtio-ports/ $ org.qemu.guest_agent.0 Configure masakarimonitors-introspectiveinstancemonitor ---------------------------------------------- #. Clone masakari-monitors using:: $ git clone https://github.com/openstack/masakari-monitors.git #. Create masakarimonitors directory in /etc/. #. Run setup.py from masakari-monitors:: $ sudo python setup.py install #. Copy masakarimonitors.conf and process_list.yaml files from masakari-monitors/etc/ to /etc/masakarimonitors folder and make necessary changes to the masakarimonitors.conf and process_list.yaml files. To generate the sample masakarimonitors.conf file, run the following command from the top level of the masakari-monitors directory:: $ tox -egenconfig #. To run masakari-introspectiveinstancemonitor simply use following binary:: $ masakari-introspectiveinstancemonitor ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/introspectiveinstancemonitor/__init__.py0000664000175000017500000000000000000000000031063 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/introspectiveinstancemonitor/cache.py0000664000175000017500000001176400000000000030412 0ustar00zuulzuul00000000000000# Copyright (c) 2018 WindRiver Systems # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. """ The code related to integration between oslo.cache module and masakarimonitors. Use 'oslo_cache.dict' i.e. dogpile.cache backend that uses dictionary for storage Dogpile consists of two subsystems, one building on top of the other. dogpile provides the concept of a dogpile lock, a control structure which allows a single thread of execution to be selected as the creator of some resource, while allowing other threads of execution to refer to the previous version of this resource as the creation proceeds; if there is no previous version, then those threads block until the object is available. """ from oslo_cache import core from oslo_config import cfg from masakarimonitors.i18n import _ WEEK = 604800 def register_cache_configurations(conf): """Register all configurations required for oslo.cache. The procedure registers all configurations required for oslo.cache. It should be called before configuring of cache region :param conf: instance of configuration :returns: updated configuration """ # register global configurations for caching core.configure(conf) # register specific configurations constraint_cache_group = cfg.OptGroup('constraint_validation_cache') constraint_cache_opts = [ cfg.IntOpt('expiration_time', default=WEEK, help=_( 'TTL, in seconds, for any cached item in the ' 'dogpile.cache region used for caching of validation ' 'constraints.')), cfg.BoolOpt("caching", default=True, help=_( 'Toggle to enable/disable caching when Orchestration ' 'Engine validates property constraints of stack.' 'During property validation with constraints ' 'Orchestration Engine caches requests to other ' 'OpenStack services. Please note that the global ' 'toggle for oslo.cache(enabled=True in [cache] group) ' 'must be enabled to use this feature.')) ] conf.register_group(constraint_cache_group) conf.register_opts(constraint_cache_opts, group=constraint_cache_group) extension_cache_group = cfg.OptGroup('service_extension_cache') extension_cache_opts = [ cfg.IntOpt('expiration_time', default=WEEK, help=_( 'TTL, in seconds, for any cached item in the ' 'dogpile.cache region used for caching of service ' 'extensions.')), cfg.BoolOpt('caching', default=True, help=_( 'Toggle to enable/disable caching when Orchestration ' 'Engine retrieves extensions from other OpenStack ' 'services. Please note that the global toggle for ' 'oslo.cache(enabled=True in [cache] group) must be ' 'enabled to use this feature.')) ] conf.register_group(extension_cache_group) conf.register_opts(extension_cache_opts, group=extension_cache_group) find_cache_group = cfg.OptGroup('resource_finder_cache') find_cache_opts = [ cfg.IntOpt('expiration_time', default=WEEK, help=_( 'TTL, in seconds, for any cached item in the ' 'dogpile.cache region used for caching of OpenStack ' 'service finder functions.')), cfg.BoolOpt('caching', default=True, help=_( 'Toggle to enable/disable caching when Orchestration ' 'Engine looks for other OpenStack service resources ' 'using name or id. Please note that the global ' 'toggle for oslo.cache(enabled=True in [cache] group) ' 'must be enabled to use this feature.')) ] conf.register_group(find_cache_group) conf.register_opts(find_cache_opts, group=find_cache_group) return conf # variable that stores an initialized cache region _REGION = None def get_cache_region(): global _REGION if not _REGION: _REGION = core.create_region() _REGION.configure('oslo_cache.dict', arguments={'expiration_time': WEEK}) core.configure_cache_region( conf=register_cache_configurations(cfg.CONF), region=_REGION) return _REGION ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/introspectiveinstancemonitor/instance.py0000664000175000017500000001404000000000000031141 0ustar00zuulzuul00000000000000# Copyright (c) 2018 WindRiver Systems # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import eventlet import libvirt import socket import sys import threading import time from oslo_config import cfg from oslo_log import log as oslo_logging from oslo_utils import excutils from oslo_utils import timeutils from masakarimonitors.introspectiveinstancemonitor import qemu_utils from masakarimonitors.introspectiveinstancemonitor import scheduler from masakarimonitors import manager from masakarimonitors.objects import event_constants as ec LOG = oslo_logging.getLogger(__name__) CONF = cfg.CONF class IntrospectiveInstanceMonitorManager(manager.Manager): def __init__(self, *args, **kwargs): self.init_tgm() super(IntrospectiveInstanceMonitorManager, self).__init__( service_name="introspectiveinstancemonitor", *args, **kwargs) # This keeps track of what thread is running the event loop, # (if it is run in a background thread) self.event_loop_thread = None def _reset_journal(self, eventID, eventType, detail, uuID): """To reset the monitoring to discovery stage :param eventID: Event ID :param eventType: Event type :param detail: Event code :param uuID: Domain uuid """ noticeType = ec.EventConstants.TYPE_VM hostname = socket.gethostname() currentTime = timeutils.utcnow() def _reset_handler(event_id, event_type, detail, domain_uuid, msg): """Reset monitoring To reset monitoring to discovery stage for the following event: libvirt.VIR_DOMAIN_EVENT_STARTED """ if (event_id == libvirt.VIR_DOMAIN_EVENT_ID_LIFECYCLE) and \ (event_type == libvirt.VIR_DOMAIN_EVENT_STARTED): LOG.debug(msg) qemu_utils.resetJournal(domain_uuid) # All Event Output if debug mode is on. msg = "libvirt Event Received.type = %s \ hostname = %s uuid = %s time = %s eventID = %d eventType = %d \ detail = %d" % ( noticeType, hostname, uuID, currentTime, eventID, eventType, detail) LOG.debug("%s", msg) try: thread = threading.Thread( _reset_handler(eventID, eventType, detail, uuID, msg) ) thread.start() except KeyError as e: LOG.error("virEventFilter KeyError: {0}".format(e)) except IndexError as e: LOG.error("virEventFilter IndexError: {0}".format(e)) except TypeError as e: LOG.error("virEventFilter TypeError: {0}".format(e)) except Exception: with excutils.save_and_reraise_exception(): LOG.error("Unexpected error: %s" % sys.exc_info()[0]) def init_tgm(self): """Manages the masakari-introspectiveinstancemonitor.""" self.TG = scheduler.ThreadGroupManager() def _vir_event_loop_native_run(self): # Directly run the event loop in the current thread while True: libvirt.virEventRunDefaultImpl() def _vir_event_loop_native_start(self): libvirt.virEventRegisterDefaultImpl() self.event_loop_thread = threading.Thread( target=self._vir_event_loop_native_run, name="lib_virt_eventLoop") self.event_loop_thread.daemon = True self.event_loop_thread.start() def _my_domain_event_callback(self, conn, dom, event, detail, opaque): self._reset_journal(libvirt.VIR_DOMAIN_EVENT_ID_LIFECYCLE, event, detail, dom.UUIDString()) def _my_domain_event_reboot_callback(self, conn, dom, opaque): self._reset_journal(libvirt.VIR_DOMAIN_EVENT_ID_REBOOT, -1, -1, dom.UUIDString()) def _err_handler(self, ctxt, err): LOG.warning("Error from libvirt : %s", err[2]) def _virt_event(self, uri): # Run a background thread with the event loop self._vir_event_loop_native_start() event_callback_handlers = { libvirt.VIR_DOMAIN_EVENT_ID_LIFECYCLE: self._my_domain_event_callback, libvirt.VIR_DOMAIN_EVENT_ID_REBOOT: self._my_domain_event_reboot_callback } # Connect to libvirt - If be disconnected, reprocess. self.running = True while self.running: vc = libvirt.openReadOnly(uri) # Event callback settings callback_ids = [] for event, callback in event_callback_handlers.items(): cid = vc.domainEventRegisterAny(None, event, callback, None) callback_ids.append(cid) # Connection monitoring. vc.setKeepAlive(5, 3) while vc.isAlive() == 1 and self.running: eventlet.greenthread.sleep(1) # If connection between libvirtd was lost, # clear callback connection. LOG.warning("Libvirt Connection Closed Unexpectedly.") for cid in callback_ids: try: vc.domainEventDeregisterAny(cid) except Exception: pass vc.close() del vc time.sleep(3) def stop(self): self.running = False def main(self): """Main method. Set the URI, error handler, and executes event loop processing. """ uri = CONF.libvirt.connection_uri LOG.debug("Using uri:" + uri) # set error handler & do event loop libvirt.registerErrorHandler(self._err_handler, '_virt_event') self._virt_event(uri) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/introspectiveinstancemonitor/qemu_utils.py0000775000175000017500000003276200000000000031542 0ustar00zuulzuul00000000000000# Copyright (c) 2018 WindRiver Systems # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # Introspective instance monitoring depends on the qemu guest agent to # monitoring what inside of a VM. # # A few items to get in the way of the design: # # - After VM is active, it needs time to start qemu-guest-agent. # Before error/failure is reported, we need a discovery phase to wait # until VM is guest_pingable. # # - Debouncing is needed to enforce that masakari_notifier function # not calling twice or more for the same failure. # # After reported a masakari notification, sent_notification flag will # need to be reset when there is a QEMU LIFECYCLE event like: # STARTED_BOOTED # SUSPENDED_PAUSED import eventlet import libvirt import libvirtmod_qemu import logging import re import socket import time import traceback # Machine module contains the state machine logic, state and transition events from automaton import machines from dogpile.cache.api import NoValue from lxml import etree from oslo_config import cfg from oslo_utils import timeutils from masakarimonitors.ha import masakari from masakarimonitors.introspectiveinstancemonitor import cache from masakarimonitors.objects import event_constants as ec from masakarimonitors import utils CONF = cfg.CONF ICONF = cfg.CONF.introspectiveinstancemonitor # The VM QEMU Quest Agent states # # discovery = initial state of VM, # remains in this state until it is determined that # the VM has a qemu-agent interface enabling intrusive-instance-monitoring # # healthy = no failure event is detected # # error = An error is recorded on every audit cycle where # the VM is in the error state # # reported = a transient state # to keep track of reporting of notification # # Transitions # # Representation of a transition managed by a ``Machine`` instance. # source (str): Source state of the transition. # dest (str): Destination state of the transition. # trigger (str): The type of triggering event # that advances to the next state in the sequence. def action_on_enter(new_state, triggered_event): pass def action_on_exit(old_state, triggered_event): pass STATE_SPACE = [ { 'name': 'discovery', 'next_states': { 'guest_pingable': 'healthy', 'guest_not_pingable': 'discovery', }, 'on_enter': action_on_enter, 'on_exit': action_on_exit, }, { 'name': 'healthy', 'next_states': { 'guest_pingable': 'healthy', 'guest_not_pingable': 'error', }, 'on_enter': action_on_enter, 'on_exit': action_on_exit, }, { 'name': 'error', 'next_states': { 'report': 'reported', 'guest_pingable': 'healthy', 'guest_not_pingable': 'error', }, 'on_enter': action_on_enter, 'on_exit': action_on_exit, }, { 'name': 'reported', 'next_states': { 'guest_pingable': 'discovery', 'guest_not_pingable': 'reported', }, 'on_enter': action_on_enter, 'on_exit': action_on_exit, }, ] # Journal representation of a managed ``Machine`` instance. class Journal(machines.FiniteMachine): # Factory pattern to create a Journal object @classmethod def factory(cls, domain_uuid): jo = cls.build(STATE_SPACE) jo.default_start_state = 'discovery' jo.initialize() jo.failedCount = 0 # Conditions to reset sent_notification jo.sent_notification = False jo.domain_uuid = domain_uuid jo.lastUsed = time.time() LOG.debug(str(domain_uuid) + ':Journal:__init__:') return jo def resetState(self): self.default_start_state = 'discovery' self.initialize() self.failedCount = 0 self.sent_notification = False LOG.debug(str(self.domain_uuid) + '__resetState__:') def processEvent(self, event): self.process_event(event) self.lastUsed = time.time() def getFailedCount(self): return self.failedCount def incrementFailedCount(self): if (self.current_state == 'error'): self.failedCount += 1 def resetFailedCount(self): self.failedCount = 0 def setSentNotification(self, boolean): self.sent_notification = boolean def getSentNotification(self): return self.sent_notification # libvirt state verbose dictionary STATES = { libvirt.VIR_DOMAIN_NOSTATE: 'no state', libvirt.VIR_DOMAIN_RUNNING: 'running', libvirt.VIR_DOMAIN_BLOCKED: 'blocked on resource', libvirt.VIR_DOMAIN_PAUSED: 'paused by user', libvirt.VIR_DOMAIN_SHUTDOWN: 'being shut down', libvirt.VIR_DOMAIN_SHUTOFF: 'shut off', libvirt.VIR_DOMAIN_CRASHED: 'crashed', } LOG = logging.getLogger(__name__) def get_function_name(): return traceback.extract_stack(None, 2)[0][2] # To reset journal object by domain uuid def resetJournal(domain_uuid): # To get the VM Journal object from the dictionary # :param domain: QEMU domain UUID dict = cache.get_cache_region() jo = None if type(dict.get(domain_uuid)) is NoValue: jo = Journal.factory(domain_uuid) dict.set(domain_uuid, jo) else: jo = dict.get(domain_uuid) jo.resetState() # Qemu guest agent is used to check VM status # The checking pre-conditions are as follows: # - VM is running # - VM has Qemu guest agent installed # # then status is determined by # - VM is guest-agent-pingable or not # # Note: checkGuests function is called by the scheduler class QemuGuestAgent(object): def __init__(self): super(QemuGuestAgent, self).__init__() self.notifier = masakari.SendNotification() # _thresholdsCrossing # # We only issue a notification # to masakari-engine if the VM is 'already' in the error state # when there are consecutive guest_ping failures. # Suggested value for guest_monitoring_failure_threshold >= 3 # # Note: When operators are trying to gracefully shutdown VM, # QEMU may take time to powering-off. # E.g. When you do or # you may see that QEMU is active but monitoring may fail # due to VM is still "powering-off" # Status | Task State | Power State # ACTIVE | powering-off | Running def _thresholdsCrossing(self, domain): if (((self.getVmFsm(domain.UUIDString()).current_state) == 'error') and (self._getJournalObject(domain.UUIDString()).getFailedCount() > ICONF.guest_monitoring_failure_threshold)): LOG.debug('_thresholdsCrossing:' + domain.UUIDString()) LOG.debug(self._getJournalObject( domain.UUIDString()).getFailedCount()) return True else: return False def _masakari_notifier(self, domain_uuid): if self._getJournalObject(domain_uuid).getSentNotification(): LOG.debug('notifier.send_notification Skipped:' + domain_uuid) else: hostname = socket.gethostname() noticeType = ec.EventConstants.TYPE_VM current_time = timeutils.utcnow() event = { 'notification': { 'type': noticeType, 'hostname': hostname, 'generated_time': current_time, 'payload': { 'event': 'QEMU_GUEST_AGENT_ERROR', 'instance_uuid': domain_uuid, 'vir_domain_event': 'STOPPED_FAILED' } } } try: self.notifier.send_notification(CONF.callback.retry_max, CONF.callback.retry_interval, event) self._getJournalObject(domain_uuid).processEvent('report') self._getJournalObject(domain_uuid).setSentNotification(True) except Exception: LOG.warning('Exception :' + domain_uuid + ' @ ' + get_function_name()) pass def _qemuAgentGuestPing(self, domain, timeout, flags=0): def _no_heartbeat(domain_uuid): # Also advance the FSM self.getVmFsm(domain_uuid).processEvent('guest_not_pingable') self._getJournalObject(domain_uuid).incrementFailedCount() def _with_heartbeat(domain_uuid): #The order matters as we want to decrease the counter first self._getJournalObject(domain_uuid).resetFailedCount() self.getVmFsm(domain_uuid).processEvent('guest_pingable') def _record(result): if result is None: LOG.debug(domain.UUIDString() + '\tqemu-ga_guest-ping is not responding.') if self._thresholdsCrossing(domain): self._masakari_notifier(domain.UUIDString()) _no_heartbeat(domain.UUIDString()) else: _with_heartbeat(domain.UUIDString()) """Send a Guest Agent ping to domain """ # must pass domain._o to the c library as virDomainPtr ret = libvirtmod_qemu.virDomainQemuAgentCommand(domain._o, '{"execute": "guest-ping"}', timeout, flags) _record(ret) def _getJournalObject(self, domain_uuid): """Function: To get the dictionary :param domain: QEMU domain :return: the journal object referred by domain_uuid """ dict = cache.get_cache_region() if type(dict.get(domain_uuid)) is NoValue: jo = Journal.factory(domain_uuid) dict.set(domain_uuid, jo) return jo else: return dict.get(domain_uuid) def getVmFsm(self, domain_uuid): """Function: To get the VM Finite State Machine from the dictionary :param domain: QEMU domain :return: FSM object """ dict = cache.get_cache_region() if type(dict.get(domain_uuid)) is NoValue: jo = Journal.factory(domain_uuid) dict.set(domain_uuid, jo) return jo else: return dict.get(domain_uuid) def _hasQemuGuestAgent(self, domain): """Function: To check whether the VM has an QEMU Guest Agent by examining the qemu.guest_agent sock First check if libvirt is running or not, then sock :param domain: QEMU domain :return: true or false """ def qemuGuestAgentPathMatch(path): SOCK = ICONF.qemu_guest_agent_sock_path return re.match('%s' % SOCK, path) state, reason = domain.state() # First check if libvirt is running or not if state != libvirt.VIR_DOMAIN_RUNNING: return False xmlDesc = domain.XMLDesc() tree = etree.fromstring(xmlDesc) ''' Example
''' try: source = tree.find("devices/channel/source") if (source is not None): mode = source.get('mode') path = source.get('path') # There should be a bind for a sock file for qemu guest_agent if (qemuGuestAgentPathMatch(path) and mode == 'bind'): return True except Exception: pass return False def checkGuests(self): """Function: Check QEMU Guests Condition: VM under intrusive monitoring must have QEMU agent client configured, installed and qemu "guest-agent-pingable". """ try: conn = libvirt.open(CONF.libvirt.connection_uri) ids = conn.listDomainsID() running = map(conn.lookupByID, ids) for domain in running: try: if self._hasQemuGuestAgent(domain): @utils.synchronized(domain.UUIDString()) def do_qemuAgentGuestPing(domain, timeout): self._qemuAgentGuestPing(domain, timeout) do_qemuAgentGuestPing(domain, ICONF.guest_monitoring_timeout) except libvirt.libvirtError as le: LOG.warning(le) continue except Exception as e: LOG.warning(e) pass def reschedule(action, sleep_time=1): """Eventlet Sleep for the specified number of seconds. :param sleep_time: seconds to sleep; if None, no sleep; """ LOG.debug('At reschedule') if sleep_time is not None: LOG.debug('Action %s sleep for %s seconds' % ( action.id, sleep_time)) eventlet.sleep(sleep_time) def sleep(sleep_time): """Interface for sleeping.""" eventlet.sleep(sleep_time) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/introspectiveinstancemonitor/scheduler.py0000664000175000017500000000616300000000000031322 0ustar00zuulzuul00000000000000# Copyright (c) 2018 WindRiver Systems # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import eventlet import logging from oslo_config import cfg from oslo_service import threadgroup from masakarimonitors.introspectiveinstancemonitor import qemu_utils LOG = logging.getLogger(__name__) CONF = cfg.CONF class ThreadGroupManager(object): """Thread group manager.""" def init_qemu_ga(self): self.qemuGA = qemu_utils.QemuGuestAgent() LOG.debug('Started QemuGuestAgent') def __init__(self): self.init_qemu_ga() super(ThreadGroupManager, self).__init__() self.threads = {} self.group = threadgroup.ThreadGroup() # Create dummy service task, because when there is nothing queued # on self.tg the process exits self.add_timer( CONF.introspectiveinstancemonitor.guest_monitoring_interval, self._service_task) def _service_task(self): # This is used to trigger periodic monitoring tasks self.qemuGA.checkGuests() def start(self, func, *args, **kwargs): """Run the given method in a sub-thread.""" LOG.debug('add_thread') return self.group.add_thread(func, *args, **kwargs) def add_timer(self, interval, func, *args, **kwargs): """Define a periodic task to be run in the thread group. The task will be executed in a separate green thread. Interval is from cfg.CONF.periodic_interval """ LOG.debug('group.add_timer') self.group.add_timer(interval, func, *args, **kwargs) def stop_timers(self): self.group.stop_timers() def stop(self, graceful=False): """Stop any active threads belong to this threadgroup.""" # Try to stop all threads gracefully self.group.stop(graceful) self.group.wait() # Wait for link()ed functions (i.e. lock release) threads = self.group.threads[:] links_done = dict((th, False) for th in threads) def mark_done(gt, th): links_done[th] = True for th in threads: th.link(mark_done, th) while not all(links_done.values()): eventlet.sleep() def reschedule(action, sleep_time=1): """Eventlet Sleep for the specified number of seconds. :param sleep_time: seconds to sleep; if None, no sleep; """ LOG.debug('At reschedule') if sleep_time is not None: LOG.debug('Action %s sleep for %s seconds' % ( action.id, sleep_time)) eventlet.sleep(sleep_time) def sleep(sleep_time): """Interface for sleeping.""" LOG.debug('sleep') eventlet.sleep(sleep_time) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/manager.py0000664000175000017500000000674400000000000022730 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Base Manager class. Managers are responsible for a certain aspect of the system. It is a logical grouping of code relating to a portion of the system. In general other components should be using the manager to make changes to the components that it is responsible for. We have adopted a basic strategy of Smart managers and dumb data, which means rather than attaching methods to data objects, components should call manager methods that act on the data. Methods on managers that can be executed locally should be called directly. If a particular method must execute on a remote host, this should be done via rpc to the service that wraps the manager. Managers should be responsible for most of the db access, and non-implementation specific data. Anything implementation specific that can't be generalized should be done by the Driver. Managers will often provide methods for initial setup of a host or periodic tasks to a wrapping service. This module provides Manager, a base class for managers. """ from oslo_service import periodic_task import masakarimonitors.conf CONF = masakarimonitors.conf.CONF class PeriodicTasks(periodic_task.PeriodicTasks): def __init__(self): super(PeriodicTasks, self).__init__(CONF) class Manager(PeriodicTasks): def __init__(self, host=None, service_name='undefined'): if not host: host = CONF.hostname self.host = host self.service_name = service_name super(Manager, self).__init__() def periodic_tasks(self, context, raise_on_error=False): """Tasks to be run at a periodic interval.""" return self.run_periodic_tasks(context, raise_on_error=raise_on_error) def init_host(self): """Hook to do additional manager initialization when one requests the service be started. This is called before any service record is created. Child classes should override this method. """ pass def cleanup_host(self): """Hook to do cleanup work when the service shuts down. Child classes should override this method. """ pass def pre_start_hook(self): """Hook to provide the manager the ability to do additional start-up work before any RPC queues/consumers are created. This is called after other initialization has succeeded and a service record is created. Child classes should override this method. """ pass def post_start_hook(self): """Hook to provide the manager the ability to do additional start-up work immediately after a service creates RPC consumers and starts 'running'. Child classes should override this method. """ pass def reset(self): """Hook called on SIGHUP to signal the manager to re-read any dynamic configuration or do any reconfiguration tasks. """ pass ././@PaxHeader0000000000000000000000000000003300000000000011451 xustar000000000000000027 mtime=1743591461.193202 masakari_monitors-19.0.0/masakarimonitors/objects/0000775000175000017500000000000000000000000022362 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/objects/__init__.py0000664000175000017500000000000000000000000024461 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/objects/event_constants.py0000664000175000017500000000202600000000000026151 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. class EventConstants(object): # Define event types. TYPE_PROCESS = "PROCESS" TYPE_COMPUTE_HOST = "COMPUTE_HOST" TYPE_VM = "VM" # Define event details. EVENT_STARTED = "STARTED" EVENT_STOPPED = "STOPPED" # Define host status. HOST_STATUS_NORMAL = "NORMAL" HOST_STATUS_UNKNOWN = "UNKNOWN" # Define cluster status. CLUSTER_STATUS_ONLINE = "ONLINE" CLUSTER_STATUS_OFFLINE = "OFFLINE" ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/privsep.py0000664000175000017500000000147200000000000022777 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from oslo_privsep import priv_context monitors_priv = priv_context.PrivContext( "masakarimonitors", cfg_section="masakarimonitors_privileged", pypath=__name__ + ".monitors_priv", capabilities=[], ) ././@PaxHeader0000000000000000000000000000003300000000000011451 xustar000000000000000027 mtime=1743591461.193202 masakari_monitors-19.0.0/masakarimonitors/processmonitor/0000775000175000017500000000000000000000000024017 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/processmonitor/__init__.py0000664000175000017500000000000000000000000026116 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/processmonitor/process.py0000664000175000017500000000623700000000000026057 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import yaml import eventlet from oslo_log import log as oslo_logging import masakarimonitors.conf from masakarimonitors import manager from masakarimonitors.processmonitor.process_handler import handle_process LOG = oslo_logging.getLogger(__name__) CONF = masakarimonitors.conf.CONF class ProcessmonitorManager(manager.Manager): """Manages the masakari-processmonitor.""" def __init__(self, *args, **kwargs): super(ProcessmonitorManager, self).__init__( service_name="processmonitor", *args, **kwargs) self.process_handler = handle_process.HandleProcess() def _load_process_list(self): try: process_list = yaml.safe_load(open(CONF.process.process_list_path)) LOG.debug("Loaded process list. %s" % process_list) return process_list except yaml.YAMLError as e: LOG.exception("YAMLError caught: %s", e) return except Exception as e: LOG.exception("Exception caught: %s", e) return def stop(self): self.running = False def main(self): """Main method.""" try: # Load process list. process_list = self._load_process_list() if process_list is None: LOG.error("Failed to load process list file.") return # Set process_list object to the process handler. self.process_handler.set_process_list(process_list) # Initial start of processes. self.process_handler.start_processes() self.running = True while self.running: # Monitor processes. down_process_list = self.process_handler.monitor_processes() if len(down_process_list) != 0: # Restart down processes. self.process_handler.restart_processes(down_process_list) else: # Since no down process, clear the restart_failure_list self.process_handler.restart_failure_list[:] = [] # Reload process list and set to the process handler. process_list = self._load_process_list() if process_list is None: LOG.error("Failed to reload process list file.") break self.process_handler.set_process_list(process_list) eventlet.greenthread.sleep(CONF.process.check_interval) except Exception as e: LOG.exception("Exception caught: %s", e) return return ././@PaxHeader0000000000000000000000000000003300000000000011451 xustar000000000000000027 mtime=1743591461.193202 masakari_monitors-19.0.0/masakarimonitors/processmonitor/process_handler/0000775000175000017500000000000000000000000027172 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/processmonitor/process_handler/__init__.py0000664000175000017500000000000000000000000031271 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/processmonitor/process_handler/handle_process.py0000664000175000017500000001662500000000000032547 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import socket import eventlet from oslo_log import log as oslo_logging from oslo_utils import timeutils import masakarimonitors.conf from masakarimonitors.ha import masakari from masakarimonitors.objects import event_constants as ec from masakarimonitors import utils LOG = oslo_logging.getLogger(__name__) CONF = masakarimonitors.conf.CONF class HandleProcess(object): """Handle process.""" def __init__(self): self.process_list = None self.restart_failure_list = [] self.notifier = masakari.SendNotification() def set_process_list(self, process_list): """Set process list object. :param process_list: process list object """ self.process_list = process_list def _execute_cmd(self, cmd_str, run_as_root): # Split command string and delete empty elements. command = cmd_str.split() command = filter(lambda x: x != '', command) try: # Execute start command. out, err = utils.execute(*command, run_as_root=run_as_root) if out: msg = ("CMD '%s' output stdout: %s") % (cmd_str, out) LOG.info("%s", msg) if err: msg = ("CMD '%s' output stderr: %s") % (cmd_str, err) LOG.warning("%s", msg) return 1 except Exception as e: msg = ("CMD '%s' raised exception: %s") % (cmd_str, e) LOG.error("%s", e) return 1 return 0 def start_processes(self): """Initial start of processes. This method starts the processes using start command written in the process list. """ for process in self.process_list: cmd_str = process['start_command'] pre_cmd_str = process['pre_start_command'] post_cmd_str = process['post_start_command'] # Execute pre start command. if pre_cmd_str: ret = self._execute_cmd(pre_cmd_str, process['run_as_root']) if ret != 0: continue # Execute start command. LOG.info("Start of process with executing command: %s", cmd_str) self._execute_cmd(cmd_str, process['run_as_root']) # Execute post start command. if post_cmd_str: ret = self._execute_cmd(post_cmd_str, process['run_as_root']) def monitor_processes(self): """Monitor processes. This method monitors the processes using process name written in the process list. :returns: List of down process """ down_process_list = [] for process in self.process_list: process_name = process['process_name'] try: # Execute monitoring command. out, err = utils.execute('ps', '-ef', run_as_root=False) if process_name in out: LOG.debug("Process '%s' is found." % process_name) else: # Append down_process_list. down_process_list.append(process) LOG.warning("Process '%s' is not found.", process_name) except Exception as e: LOG.error("Monitoring command raised exception: %s", e) return down_process_list def _make_event(self, process_name): hostname = socket.gethostname() current_time = timeutils.utcnow() event = { 'notification': { 'type': ec.EventConstants.TYPE_PROCESS, 'hostname': hostname, 'generated_time': current_time, 'payload': { 'event': ec.EventConstants.EVENT_STOPPED, 'process_name': process_name.split('/')[-1] } } } return event def restart_processes(self, down_process_list): """Restart processes. This method restarts the processes using restart command written in the process list. :param down_process_list: down process list object """ tmp_restart_failure_list = [] for down_process in down_process_list: # The process which failed to restart previously doesn't restart. if down_process['process_name'] in self.restart_failure_list: msg = "Process '%s' doesn't be restarted because it failed" \ " to restart previously." % down_process['process_name'] LOG.warning("%s", msg) tmp_restart_failure_list.append(down_process['process_name']) continue cmd_str = down_process['restart_command'] pre_cmd_str = down_process['pre_restart_command'] post_cmd_str = down_process['post_restart_command'] LOG.info("Restart of process with executing command: %s", cmd_str) for retries in range(0, CONF.process.restart_retries + 1): # Execute pre restart command. if pre_cmd_str: ret = self._execute_cmd(pre_cmd_str, down_process['run_as_root']) if ret != 0: # Failed to restart process. eventlet.greenthread.sleep( CONF.process.restart_interval) continue # Execute restart command. ret = self._execute_cmd(cmd_str, down_process['run_as_root']) if ret != 0: # Failed to restart process. eventlet.greenthread.sleep(CONF.process.restart_interval) continue # Execute post restart command. if post_cmd_str: ret = self._execute_cmd(post_cmd_str, down_process['run_as_root']) if ret != 0: # Failed to restart process. eventlet.greenthread.sleep( CONF.process.restart_interval) continue # Succeeded in restarting process. break if retries == CONF.process.restart_retries: # Send a notification. event = self._make_event(down_process['process_name']) self.notifier.send_notification( CONF.process.api_retry_max, CONF.process.api_retry_interval, event) # Add the process name which failed to restart to the # failure list. tmp_restart_failure_list.append(down_process['process_name']) # Replace the old restart_failure_list with new one self.restart_failure_list = tmp_restart_failure_list[:] ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/service.py0000664000175000017500000000766100000000000022755 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Generic Node base class for all workers that run on hosts.""" import os import sys from oslo_log import log as logging from oslo_service import service from oslo_utils import importutils import masakarimonitors.conf from masakarimonitors.i18n import _ from masakarimonitors import utils LOG = logging.getLogger(__name__) CONF = masakarimonitors.conf.CONF class Service(service.Service): """Service object for binaries running on hosts. A service takes a manager. """ def __init__(self, host, binary, manager): super(Service, self).__init__() self.host = host self.binary = binary self.manager_class_name = manager manager_class = importutils.import_class(self.manager_class_name) self.manager = manager_class(host=self.host) def __repr__(self): return "<%(cls_name)s: host=%(host)s, binary=%(binary)s, " \ "manager_class_name=%(manager)s>" %\ { 'cls_name': self.__class__.__name__, 'host': self.host, 'binary': self.binary, 'manager': self.manager_class_name } def start(self): LOG.info('Starting %s', self.binary) self.basic_config_check() self.manager.init_host() self.manager.main() def __getattr__(self, key): manager = self.__dict__.get('manager', None) return getattr(manager, key) @classmethod def create(cls, host=None, binary=None, manager=None): """Instantiates class and passes back application object. :param host: defaults to CONF.hostname :param binary: defaults to basename of executable :param manager: defaults to CONF._manager """ if not host: host = CONF.hostname if not binary: binary = os.path.basename(sys.argv[0]) if not manager: manager_cls = ('%s_manager' % binary.rpartition('masakarimonitors-')[2]) manager = CONF.get(manager_cls, None) service_obj = cls(host, binary, manager) return service_obj def kill(self): """Destroy the service object in the datastore. NOTE: Although this method is not used anywhere else than tests, it is convenient to have it here, so the tests might easily and in clean way stop and remove the service_ref. """ self.stop() def stop(self): LOG.info('Stopping %s', self.binary) self.manager.stop() super(Service, self).stop() def basic_config_check(self): """Perform basic config checks before starting processing.""" # Make sure the tempdir exists and is writable try: with utils.tempdir(): pass except Exception as e: LOG.error('Temporary directory is invalid: %s', e) sys.exit(1) def reset(self): self.manager.reset() # NOTE: the global launcher is to maintain the existing # functionality of calling service.serve + # service.wait _launcher = None def serve(server, workers=None): global _launcher if _launcher: raise RuntimeError(_('serve() can only be called once')) _launcher = service.launch(CONF, server, workers=workers) def wait(): _launcher.wait() ././@PaxHeader0000000000000000000000000000003300000000000011451 xustar000000000000000027 mtime=1743591461.193202 masakari_monitors-19.0.0/masakarimonitors/tests/0000775000175000017500000000000000000000000022073 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/__init__.py0000664000175000017500000000000000000000000024172 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/base.py0000664000175000017500000000140100000000000023353 0ustar00zuulzuul00000000000000# Copyright 2010-2011 OpenStack Foundation # Copyright (c) 2013 Hewlett-Packard Development Company, L.P. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from oslotest import base class TestCase(base.BaseTestCase): """Test case base class for all unit tests.""" ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/test_masakarimonitors.py0000664000175000017500000000141300000000000027066 0ustar00zuulzuul00000000000000# Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. """ test_masakarimonitors ---------------------------------- Tests for `masakarimonitors` module. """ from masakarimonitors.tests import base class TestMasakarimonitors(base.TestCase): def test_something(self): pass ././@PaxHeader0000000000000000000000000000003300000000000011451 xustar000000000000000027 mtime=1743591461.193202 masakari_monitors-19.0.0/masakarimonitors/tests/unit/0000775000175000017500000000000000000000000023052 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/__init__.py0000664000175000017500000000000000000000000025151 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000003300000000000011451 xustar000000000000000027 mtime=1743591461.193202 masakari_monitors-19.0.0/masakarimonitors/tests/unit/ha/0000775000175000017500000000000000000000000023442 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/ha/__init__.py0000664000175000017500000000000000000000000025541 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/ha/test_masakari.py0000664000175000017500000001304500000000000026646 0ustar00zuulzuul00000000000000# Copyright(c) 2017 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import testtools from unittest import mock import uuid import eventlet from keystoneauth1 import loading as ks_loading from openstack import connection from openstack import exceptions from oslo_utils import timeutils import masakarimonitors.conf from masakarimonitors.ha import masakari from masakarimonitors.objects import event_constants as ec CONF = masakarimonitors.conf.CONF PROFILE_TYPE = "ha" PROFILE_NAME = "masakari" class FakeResponse(object): def __init__(self, status_code=200, headers=None): self.status_code = status_code self.headers = { 'content-type': 'application/json', 'x-openstack-request-id': uuid.uuid4().hex, } class TestSendNotification(testtools.TestCase): def setUp(self): super(TestSendNotification, self).setUp() self.api_retry_max = 3 self.api_retry_interval = 1 self.event = { 'notification': { 'type': ec.EventConstants.TYPE_COMPUTE_HOST, 'hostname': 'compute-node1', 'generated_time': timeutils.utcnow(), 'payload': { 'event': ec.EventConstants.EVENT_STOPPED, 'cluster_status': 'OFFLINE', 'host_status': ec.EventConstants.HOST_STATUS_NORMAL } } } @mock.patch.object(connection, 'Connection') @mock.patch.object(ks_loading, 'load_session_from_conf_options') @mock.patch.object(ks_loading, 'load_auth_from_conf_options') def test_send_notification( self, mock_auth, mock_session, mock_connection): mock_conn = mock.Mock() mock_conn.instance_ha.return_value = mock.Mock() mock_conn.instance_ha.create_notification.return_value = mock.Mock() mock_connection.return_value = mock_conn notifier = masakari.SendNotification() notifier.send_notification( self.api_retry_max, self.api_retry_interval, self.event) mock_conn.instance_ha.create_notification.assert_called_once_with( type=self.event['notification']['type'], hostname=self.event['notification']['hostname'], generated_time=self.event['notification']['generated_time'], payload=self.event['notification']['payload']) mock_auth.assert_called_once_with(CONF, 'api') mock_session.assert_called_once_with(CONF, 'api', auth=mock_auth.return_value) @mock.patch.object(connection, 'Connection') @mock.patch.object(ks_loading, 'load_session_from_conf_options') @mock.patch.object(ks_loading, 'load_auth_from_conf_options') def test_send_notification_409_error( self, mock_auth, mock_session, mock_connection): mock_conn = mock.Mock() mock_conn.instance_ha.return_value = mock.Mock() mock_conn.instance_ha.create_notification.return_value = mock.Mock() mock_connection.return_value = mock_conn response = FakeResponse(status_code=409) status_ex = exceptions.HttpException(response=response) mock_conn.instance_ha.create_notification.side_effect = status_ex notifier = masakari.SendNotification() notifier.send_notification( self.api_retry_max, self.api_retry_interval, self.event) mock_conn.instance_ha.create_notification.assert_called_once_with( type=self.event['notification']['type'], hostname=self.event['notification']['hostname'], generated_time=self.event['notification']['generated_time'], payload=self.event['notification']['payload']) @mock.patch.object(eventlet.greenthread, 'sleep') @mock.patch.object(connection, 'Connection') @mock.patch.object(ks_loading, 'load_session_from_conf_options') @mock.patch.object(ks_loading, 'load_auth_from_conf_options') def test_send_notification_500_error( self, mock_auth, mock_session, mock_connection, mock_sleep): mock_conn = mock.Mock() mock_conn.instance_ha.return_value = mock.Mock() mock_conn.instance_ha.create_notification.return_value = mock.Mock() mock_connection.return_value = mock_conn response = FakeResponse(status_code=500) status_ex = exceptions.HttpException(response=response) mock_conn.instance_ha.create_notification.side_effect = status_ex mock_sleep.return_value = None notifier = masakari.SendNotification() notifier.send_notification( self.api_retry_max, self.api_retry_interval, self.event) mock_conn.instance_ha.create_notification.assert_called_with( type=self.event['notification']['type'], hostname=self.event['notification']['hostname'], generated_time=self.event['notification']['generated_time'], payload=self.event['notification']['payload']) self.assertEqual(self.api_retry_max + 1, mock_conn.instance_ha.create_notification.call_count) ././@PaxHeader0000000000000000000000000000003300000000000011451 xustar000000000000000027 mtime=1743591461.193202 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/0000775000175000017500000000000000000000000025437 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/__init__.py0000664000175000017500000000000000000000000027536 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000003300000000000011451 xustar000000000000000027 mtime=1743591461.193202 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/consul_check/0000775000175000017500000000000000000000000030077 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/consul_check/__init__.py0000664000175000017500000000000000000000000032176 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/consul_check/test_consul_helper.py0000664000175000017500000001136500000000000034360 0ustar00zuulzuul00000000000000# Copyright(c) 2021 Inspur # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import testtools from unittest import mock from oslo_config import fixture as fixture_config from masakarimonitors.hostmonitor.consul_check import consul_helper class FakerAgentMembers(object): def __init__(self): self.agent_members = [] def create_agent(self, name, status=1): agent = { 'Name': name, 'Status': status, 'Port': 'agent_lan_port', 'Addr': 'agent_ip', 'Tags': { 'dc': 'storage', 'role': 'consul', 'port': 'agent_server_port', 'wan_join_port': 'agent_wan_port', 'expect': '3', 'id': 'agent_id', 'vsn_max': '3', 'vsn_min': '2', 'vsn': '2', 'raft_vsn': '2', }, 'ProtocolMax': 5, 'ProtocolMin': 1, 'ProtocolCur': 2, 'DelegateMax': 5, 'DelegateMin': 2, 'DelegateCur': 4, } self.agent_members.append(agent) def generate_agent_members(self): return self.agent_members class TestConsulManager(testtools.TestCase): def setUp(self): super(TestConsulManager, self).setUp() self.CONF = self.useFixture(fixture_config.Config()).conf self.consul_manager = consul_helper.ConsulManager(self.CONF) self.consul_manager.agents = { 'manage': consul_helper.ConsulAgent('manage'), 'tenant': consul_helper.ConsulAgent('tenant'), 'storage': consul_helper.ConsulAgent('storage'), } def test_get_health(self): fake_manage_agents = FakerAgentMembers() fake_manage_agents.create_agent('node01', status=1) fake_manage_agents.create_agent('node02', status=1) fake_manage_agents.create_agent('node03', status=1) agent_manage_members = fake_manage_agents.generate_agent_members() fake_tenant_agents = FakerAgentMembers() fake_tenant_agents.create_agent('node01', status=1) fake_tenant_agents.create_agent('node02', status=1) fake_tenant_agents.create_agent('node03', status=1) agent_tenant_members = fake_tenant_agents.generate_agent_members() fake_storage_agents = FakerAgentMembers() fake_storage_agents.create_agent('node01', status=1) fake_storage_agents.create_agent('node02', status=1) fake_storage_agents.create_agent('node03', status=3) agent_storage_members = fake_storage_agents.generate_agent_members() with mock.patch.object(self.consul_manager.agents['manage'], 'get_agents', return_value=agent_manage_members): with mock.patch.object(self.consul_manager.agents['tenant'], 'get_agents', return_value=agent_tenant_members): with mock.patch.object(self.consul_manager.agents['storage'], 'get_agents', return_value=agent_storage_members): excepted_health = { "node01": ['up', 'up', 'up'], "node02": ['up', 'up', 'up'], "node03": ['up', 'up', 'down'], } sequence = ['manage', 'tenant', 'storage'] agents_health = self.consul_manager.get_health(sequence) self.assertEqual(excepted_health, agents_health) class TestConsulAgent(testtools.TestCase): def setUp(self): super(TestConsulAgent, self).setUp() self.consul_agent = consul_helper.ConsulAgent('test') def test_get_health(self): fake_agents = FakerAgentMembers() fake_agents.create_agent('node01', status=1) fake_agents.create_agent('node02', status=1) fake_agents.create_agent('node03', status=3) agent_members = fake_agents.generate_agent_members() with mock.patch.object(self.consul_agent, 'get_agents', return_value=agent_members): excepted_health = { "node01": 'up', "node02": 'up', "node03": 'down', } agents_health = self.consul_agent.get_health() self.assertEqual(excepted_health, agents_health) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/consul_check/test_manager.py0000664000175000017500000001432400000000000033126 0ustar00zuulzuul00000000000000# Copyright(c) 2021 Inspur # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import testtools from unittest import mock from collections import deque import eventlet from oslo_config import fixture as fixture_config import masakarimonitors.conf from masakarimonitors.ha import masakari from masakarimonitors.hostmonitor.consul_check import consul_helper from masakarimonitors.hostmonitor.consul_check import manager from masakarimonitors.hostmonitor.consul_check import matrix_helper eventlet.monkey_patch(os=False) CONF = masakarimonitors.conf.CONF class TestConsulCheck(testtools.TestCase): def setUp(self): super(TestConsulCheck, self).setUp() self.CONF = self.useFixture(fixture_config.Config()).conf self.host_monitor = manager.ConsulCheck() self.host_monitor.matrix_manager = \ matrix_helper.MatrixManager(self.CONF) self.host_monitor._matrix = matrix_helper.DEFAULT_MATRIX self.host_monitor.consul_manager = \ consul_helper.ConsulManager(self.CONF) self.host_monitor._sequence = ['manage', 'tenant', 'storage'] self.host_monitor.monitoring_data = { "node01": deque([['up', 'up', 'up'], ['up', 'up', 'up'], ['up', 'up', 'up']], maxlen=3), "node02": deque([['up', 'up', 'up'], ['up', 'up', 'up'], ['up', 'up', 'down']], maxlen=3), "node03": deque([['up', 'up', 'up'], ['down', 'up', 'up'], ['down', 'up', 'up']], maxlen=3), } def test_update_monitoring_data(self): mock_health = { 'node01': ['up', 'up', 'up'], 'node02': ['up', 'up', 'up'], 'node03': ['up', 'up', 'up'] } with mock.patch.object(self.host_monitor.consul_manager, 'get_health', return_value=mock_health): self.host_monitor.update_monitoring_data() excepted_monitoring_data = { "node01": deque([['up', 'up', 'up'], ['up', 'up', 'up'], ['up', 'up', 'up']], maxlen=3), "node02": deque([['up', 'up', 'up'], ['up', 'up', 'down'], ['up', 'up', 'up']], maxlen=3), "node03": deque([['down', 'up', 'up'], ['down', 'up', 'up'], ['up', 'up', 'up']], maxlen=3), } self.assertEqual(excepted_monitoring_data, self.host_monitor.monitoring_data) def test_get_host_statistical_health(self): self.assertEqual(['up', 'up', 'up'], self.host_monitor.get_host_health('node01')) self.assertEqual(['up', 'up', None], self.host_monitor.get_host_health('node02')) self.assertEqual([None, 'up', 'up'], self.host_monitor.get_host_health('node03')) def test_host_statistical_health_changed(self): self.host_monitor.last_host_health = { 'node02': ['up', 'up', None], 'node03': ['up', 'up', 'down'] } self.assertFalse(self.host_monitor._host_health_changed( 'node01', ['up', 'up', 'up'])) self.assertTrue(self.host_monitor._host_health_changed( 'node02', ['up', 'up', 'up'])) self.assertTrue(self.host_monitor._host_health_changed( 'node03', ['up', 'up', 'up'])) last_host_health = { 'node01': ['up', 'up', 'up'], 'node02': ['up', 'up', 'up'], 'node03': ['up', 'up', 'up'] } self.assertEqual(self.host_monitor.last_host_health, last_host_health) def test_get_action_from_matrix_by_host_health(self): self.assertEqual( [], self.host_monitor.get_action_from_matrix(['up', 'up', 'up'])) self.assertEqual( ["recovery"], self.host_monitor.get_action_from_matrix(['up', 'up', 'down'])) self.assertEqual( [], self.host_monitor.get_action_from_matrix(['down', 'up', 'up'])) self.assertEqual( ["recovery"], self.host_monitor.get_action_from_matrix(['down', 'down', 'down'])) @mock.patch.object(masakari.SendNotification, 'send_notification') @mock.patch.object(manager.ConsulCheck, '_event') def test_poll_hosts(self, mock_event, mock_send_notification): self.host_monitor.monitoring_data = { "node01": deque([['up', 'up', 'up'], ['up', 'up', 'up'], ['up', 'up', 'up']], maxlen=3), "node02": deque([['up', 'up', 'down'], ['up', 'up', 'down'], ['up', 'up', 'down']], maxlen=3), "node03": deque([['up', 'up', 'up'], ['up', 'up', 'up'], ['up', 'up', 'up']], maxlen=3), } self.host_monitor.last_host_health = { 'node02': ['up', 'up', None], 'node03': ['up', 'up', 'up'] } test_event = {'notification': 'test'} mock_event.return_value = test_event self.host_monitor.poll_hosts() mock_send_notification.assert_called_once_with( CONF.host.api_retry_max, CONF.host.api_retry_interval, test_event) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/consul_check/test_matrix_helper.py0000664000175000017500000000410500000000000034353 0ustar00zuulzuul00000000000000# Copyright(c) 2021 Inspur # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import tempfile from oslo_config import fixture as fixture_config import testtools import yaml from masakarimonitors.hostmonitor.consul_check import matrix_helper class TestMatrixManager(testtools.TestCase): def setUp(self): super(TestMatrixManager, self).setUp() self.CONF = self.useFixture(fixture_config.Config()).conf def test_get_matrix_and_sequence_from_file(self): matrix_cfg = { 'sequence': ['manage', 'tenant', 'storage'], 'matrix': [{"health": ["up", "up", "up"], "action": ['test']}] } tmp_cfg = tempfile.NamedTemporaryFile(mode='w', delete=False) tmp_cfg.write(yaml.safe_dump(matrix_cfg)) tmp_cfg.close() self.CONF.set_override('matrix_config_file', tmp_cfg.name, group='consul') matrix_manager = matrix_helper.MatrixManager(self.CONF) self.assertEqual(matrix_cfg.get('sequence'), matrix_manager.get_sequence()) self.assertEqual(matrix_cfg.get('matrix'), matrix_manager.get_matrix()) def test_get_default_matrix_and_sequence(self): self.CONF.set_override('matrix_config_file', None, group='consul') matrix_manager = matrix_helper.MatrixManager(self.CONF) self.assertEqual(matrix_helper.DEFAULT_SEQUENCE, matrix_manager.get_sequence()) self.assertEqual(matrix_helper.DEFAULT_MATRIX, matrix_manager.get_matrix()) ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1972024 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/host_handler/0000775000175000017500000000000000000000000030111 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/host_handler/__init__.py0000664000175000017500000000000000000000000032210 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/host_handler/test_handle_host.py0000664000175000017500000011515300000000000034020 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import socket import testtools from unittest import mock from xml.etree import ElementTree from collections import deque import eventlet from oslo_utils import timeutils import masakarimonitors.conf from masakarimonitors.ha import masakari from masakarimonitors.hostmonitor.host_handler import handle_host from masakarimonitors.hostmonitor.host_handler import hold_host_status from masakarimonitors.hostmonitor.host_handler import parse_cib_xml from masakarimonitors.hostmonitor.host_handler import parse_crmmon_xml from masakarimonitors.objects import event_constants as ec from masakarimonitors import utils eventlet.monkey_patch(os=False) CONF = masakarimonitors.conf.CONF STATUS_TAG_XML = ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' CRMMON_NODES_TAG_XML = """ """ class TestCibSchemaCompliantTag(testtools.TestCase): def setUp(self): super(TestCibSchemaCompliantTag, self).setUp() def test_init_offline(self): tag = handle_host.CibSchemaCompliantTag( {'name': 'test1', 'online': 'false'}) self.assertEqual(tag['uname'], 'test1') self.assertEqual(tag['crmd'], 'offline') def test_init_online(self): tag = handle_host.CibSchemaCompliantTag( {'name': 'test1', 'online': 'true'}) self.assertEqual(tag['uname'], 'test1') self.assertEqual(tag['crmd'], 'online') class TestHandleHost(testtools.TestCase): def setUp(self): super(TestHandleHost, self).setUp() @mock.patch.object(utils, 'execute') def test_check_pacemaker_services(self, mock_execute): mock_execute.return_value = ('test_stdout', '') obj = handle_host.HandleHost() ret = obj._check_pacemaker_services('corosync') self.assertTrue(ret) mock_execute.assert_called_once_with( 'systemctl', 'status', 'corosync', run_as_root=True) @mock.patch.object(utils, 'execute') def test_check_pacemaker_services_stderr(self, mock_execute): mock_execute.return_value = ('test_stdout', 'test_stderr') obj = handle_host.HandleHost() ret = obj._check_pacemaker_services('corosync') self.assertFalse(ret) mock_execute.assert_called_once_with( 'systemctl', 'status', 'corosync', run_as_root=True) @mock.patch.object(utils, 'execute') @mock.patch.object(handle_host.HandleHost, '_check_pacemaker_services') def test_check_hb_line( self, mock_check_pacemaker_services, mock_execute): mock_check_pacemaker_services.side_effect = [True, True, False] interfaces = "enp0s8" ports = "5405" CONF.host.corosync_multicast_interfaces = interfaces CONF.host.corosync_multicast_ports = ports mock_execute.return_value = ('', '') obj = handle_host.HandleHost() ret = obj._check_hb_line() self.assertEqual(0, ret) self.assertEqual(3, mock_check_pacemaker_services.call_count) mock_check_pacemaker_services.assert_any_call('corosync') mock_check_pacemaker_services.assert_any_call('pacemaker') mock_check_pacemaker_services.assert_any_call('pacemaker_remote') cmd_str = ("timeout %s tcpdump -n -c 1 -p -i %s port %s") \ % (CONF.host.tcpdump_timeout, interfaces, ports) command = cmd_str.split() mock_execute.assert_called_once_with(*command, run_as_root=True) @mock.patch.object(handle_host.HandleHost, '_check_pacemaker_services') def test_check_hb_line_pacemaker_not_running( self, mock_check_pacemaker_services): mock_check_pacemaker_services.side_effect = [False, False, False] obj = handle_host.HandleHost() ret = obj._check_hb_line() self.assertEqual(2, ret) self.assertEqual(3, mock_check_pacemaker_services.call_count) mock_check_pacemaker_services.assert_any_call('corosync') mock_check_pacemaker_services.assert_any_call('pacemaker') mock_check_pacemaker_services.assert_any_call('pacemaker_remote') @mock.patch.object(handle_host.HandleHost, '_check_pacemaker_services') def test_check_hb_line_work_on_pacemaker_remote( self, mock_check_pacemaker_services): mock_check_pacemaker_services.side_effect = [False, False, True] obj = handle_host.HandleHost() ret = obj._check_hb_line() self.assertEqual(0, ret) self.assertEqual(3, mock_check_pacemaker_services.call_count) mock_check_pacemaker_services.assert_any_call('corosync') mock_check_pacemaker_services.assert_any_call('pacemaker') mock_check_pacemaker_services.assert_any_call('pacemaker_remote') @mock.patch.object(handle_host.HandleHost, '_check_pacemaker_services') def test_check_hb_line_conf_interfaces_is_none( self, mock_check_pacemaker_services): mock_check_pacemaker_services.side_effect = [True, True, False] interfaces = None ports = "5405" CONF.host.corosync_multicast_interfaces = interfaces CONF.host.corosync_multicast_ports = ports obj = handle_host.HandleHost() ret = obj._check_hb_line() self.assertEqual(2, ret) self.assertEqual(3, mock_check_pacemaker_services.call_count) mock_check_pacemaker_services.assert_any_call('corosync') mock_check_pacemaker_services.assert_any_call('pacemaker') mock_check_pacemaker_services.assert_any_call('pacemaker_remote') @mock.patch.object(handle_host.HandleHost, '_check_pacemaker_services') def test_check_hb_line_conf_ports_is_none( self, mock_check_pacemaker_services): mock_check_pacemaker_services.side_effect = [True, True, False] interfaces = "enp0s8" ports = None CONF.host.corosync_multicast_interfaces = interfaces CONF.host.corosync_multicast_ports = ports obj = handle_host.HandleHost() ret = obj._check_hb_line() self.assertEqual(2, ret) self.assertEqual(3, mock_check_pacemaker_services.call_count) mock_check_pacemaker_services.assert_any_call('corosync') mock_check_pacemaker_services.assert_any_call('pacemaker') mock_check_pacemaker_services.assert_any_call('pacemaker_remote') @mock.patch.object(handle_host.HandleHost, '_check_pacemaker_services') def test_check_hb_line_incorrect_interfaces( self, mock_check_pacemaker_services): mock_check_pacemaker_services.side_effect = [True, True, False] interfaces = "enp0s3,enp0s8" ports = "5405" CONF.host.corosync_multicast_interfaces = interfaces CONF.host.corosync_multicast_ports = ports obj = handle_host.HandleHost() ret = obj._check_hb_line() self.assertEqual(2, ret) self.assertEqual(3, mock_check_pacemaker_services.call_count) mock_check_pacemaker_services.assert_any_call('corosync') mock_check_pacemaker_services.assert_any_call('pacemaker') mock_check_pacemaker_services.assert_any_call('pacemaker_remote') @mock.patch.object(handle_host.HandleHost, '_check_pacemaker_services') def test_check_hb_line_incorrect_ports( self, mock_check_pacemaker_services): mock_check_pacemaker_services.side_effect = [True, True, False] interfaces = "enp0s8" ports = "5405,5406" CONF.host.corosync_multicast_interfaces = interfaces CONF.host.corosync_multicast_ports = ports obj = handle_host.HandleHost() ret = obj._check_hb_line() self.assertEqual(2, ret) self.assertEqual(3, mock_check_pacemaker_services.call_count) mock_check_pacemaker_services.assert_any_call('corosync') mock_check_pacemaker_services.assert_any_call('pacemaker') mock_check_pacemaker_services.assert_any_call('pacemaker_remote') @mock.patch.object(utils, 'execute') @mock.patch.object(handle_host.HandleHost, '_check_pacemaker_services') def test_check_hb_line_tcpdump_fail( self, mock_check_pacemaker_services, mock_execute): mock_check_pacemaker_services.side_effect = [True, True, False] interfaces = "enp0s8" ports = "5405" CONF.host.corosync_multicast_interfaces = interfaces CONF.host.corosync_multicast_ports = ports mock_execute.side_effect = Exception("Test exception.") obj = handle_host.HandleHost() ret = obj._check_hb_line() self.assertEqual(1, ret) self.assertEqual(3, mock_check_pacemaker_services.call_count) mock_check_pacemaker_services.assert_any_call('corosync') mock_check_pacemaker_services.assert_any_call('pacemaker') mock_check_pacemaker_services.assert_any_call('pacemaker_remote') cmd_str = ("timeout %s tcpdump -n -c 1 -p -i %s port %s") \ % (CONF.host.tcpdump_timeout, interfaces, ports) command = cmd_str.split() mock_execute.assert_called_once_with(*command, run_as_root=True) @mock.patch.object(utils, 'execute') def test_check_host_status_by_crmadmin_idle(self, mock_execute): my_hostname = socket.gethostname() crmadmin_stdout = "Status of crmd@%s: S_IDLE (ok)" % my_hostname mock_execute.return_value = (crmadmin_stdout, '') obj = handle_host.HandleHost() ret = obj._check_host_status_by_crmadmin() self.assertEqual(0, ret) mock_execute.assert_called_once_with( 'crmadmin', '-S', my_hostname, run_as_root=True) @mock.patch.object(utils, 'execute') def test_check_host_status_by_crmadmin_not_dc(self, mock_execute): my_hostname = socket.gethostname() crmadmin_stdout = "Status of crmd@%s: S_NOT_DC (ok)" % my_hostname mock_execute.return_value = (crmadmin_stdout, '') obj = handle_host.HandleHost() ret = obj._check_host_status_by_crmadmin() self.assertEqual(0, ret) mock_execute.assert_called_once_with( 'crmadmin', '-S', my_hostname, run_as_root=True) @mock.patch.object(utils, 'execute') def test_check_host_status_by_crmadmin_output_stderr(self, mock_execute): my_hostname = socket.gethostname() mock_execute.return_value = ('', 'crmadmin: command not found') obj = handle_host.HandleHost() ret = obj._check_host_status_by_crmadmin() self.assertEqual(1, ret) mock_execute.assert_called_once_with( 'crmadmin', '-S', my_hostname, run_as_root=True) @mock.patch.object(utils, 'execute') def test_check_host_status_by_crmadmin_output_unexpected_message( self, mock_execute): my_hostname = socket.gethostname() mock_execute.return_value = ('Unexpected message.', '') obj = handle_host.HandleHost() ret = obj._check_host_status_by_crmadmin() self.assertEqual(1, ret) mock_execute.assert_called_once_with( 'crmadmin', '-S', my_hostname, run_as_root=True) @mock.patch.object(utils, 'execute') def test_get_cib_xml(self, mock_execute): mock_execute.return_value = ('test_stdout', '') obj = handle_host.HandleHost() ret = obj._get_cib_xml() self.assertEqual('test_stdout', ret) mock_execute.assert_called_once_with( 'cibadmin', '--query', run_as_root=True) @mock.patch.object(utils, 'execute') def test_get_cib_xml_output_stderr(self, mock_execute): mock_execute.return_value = ('test_stdout', 'test_stderr') obj = handle_host.HandleHost() ret = obj._get_cib_xml() self.assertIsNone(ret) mock_execute.assert_called_once_with( 'cibadmin', '--query', run_as_root=True) @mock.patch.object(utils, 'execute') def test_get_crmmon_xml(self, mock_execute): mock_execute.return_value = ('test_stdout', '') obj = handle_host.HandleHost() ret = obj._get_crmmon_xml() self.assertEqual('test_stdout', ret) mock_execute.assert_called_once_with( 'crm_mon', '-X', run_as_root=True) @mock.patch.object(utils, 'execute') def test_get_crmmon_xml_stderr(self, mock_execute): mock_execute.return_value = ('test_stdout', 'test_stderr') obj = handle_host.HandleHost() ret = obj._get_crmmon_xml() self.assertIsNone(ret) mock_execute.assert_called_once_with( 'crm_mon', '-X', run_as_root=True) @mock.patch.object(utils, 'execute') @mock.patch.object(parse_cib_xml.ParseCibXml, 'get_stonith_ipmi_params') def test_is_poweroff(self, mock_get_stonith_ipmi_params, mock_execute): ipmi_values = { 'userid': 'admin', 'passwd': 'password', 'interface': 'lanplus', 'ipaddr': '0.0.0.0' } mock_get_stonith_ipmi_params.return_value = ipmi_values mock_execute.return_value = ('Chassis Power is off', '') obj = handle_host.HandleHost() ret = obj._is_poweroff('test_hostname') self.assertTrue(ret) cmd_str = ("timeout %s ipmitool -U %s -P %s -I %s -H %s " "power status") \ % (str(CONF.host.ipmi_timeout), ipmi_values['userid'], ipmi_values['passwd'], ipmi_values['interface'], ipmi_values['ipaddr']) command = cmd_str.split() mock_execute.assert_called_once_with(*command, run_as_root=False) @mock.patch.object(utils, 'execute') @mock.patch.object(parse_cib_xml.ParseCibXml, 'get_stonith_ipmi_params') def test_is_poweroff_ipmi_values_is_none( self, mock_get_stonith_ipmi_params, mock_execute): mock_get_stonith_ipmi_params.return_value = None mock_execute.return_value = ('Chassis Power is off', '') obj = handle_host.HandleHost() ret = obj._is_poweroff('test_hostname') self.assertFalse(ret) mock_execute.assert_not_called() @mock.patch.object(eventlet.greenthread, 'sleep') @mock.patch.object(utils, 'execute') @mock.patch.object(parse_cib_xml.ParseCibXml, 'get_stonith_ipmi_params') def test_is_poweroff_output_stderr( self, mock_get_stonith_ipmi_params, mock_execute, mock_sleep): ipmi_values = { 'userid': 'admin', 'passwd': 'password', 'interface': 'lanplus', 'ipaddr': '0.0.0.0' } mock_get_stonith_ipmi_params.return_value = ipmi_values mock_execute.return_value = ('Chassis Power is off', 'test_stderr') mock_sleep.return_value = None obj = handle_host.HandleHost() ret = obj._is_poweroff('test_hostname') self.assertFalse(ret) cmd_str = ("timeout %s ipmitool -U %s -P %s -I %s -H %s " "power status") \ % (str(CONF.host.ipmi_timeout), ipmi_values['userid'], ipmi_values['passwd'], ipmi_values['interface'], ipmi_values['ipaddr']) command = cmd_str.split() calls = [mock.call(*command, run_as_root=False), mock.call(*command, run_as_root=False), mock.call(*command, run_as_root=False), mock.call(*command, run_as_root=False)] mock_execute.assert_has_calls(calls) @mock.patch.object(eventlet.greenthread, 'sleep') @mock.patch.object(utils, 'execute') @mock.patch.object(parse_cib_xml.ParseCibXml, 'get_stonith_ipmi_params') def test_is_poweroff_output_unexpected_message( self, mock_get_stonith_ipmi_params, mock_execute, mock_sleep): ipmi_values = { 'userid': 'admin', 'passwd': 'password', 'interface': 'lanplus', 'ipaddr': '0.0.0.0' } mock_get_stonith_ipmi_params.return_value = ipmi_values mock_execute.return_value = ('Unexpected message.', '') mock_sleep.return_value = None obj = handle_host.HandleHost() ret = obj._is_poweroff('test_hostname') self.assertFalse(ret) cmd_str = ("timeout %s ipmitool -U %s -P %s -I %s -H %s " "power status") \ % (str(CONF.host.ipmi_timeout), ipmi_values['userid'], ipmi_values['passwd'], ipmi_values['interface'], ipmi_values['ipaddr']) command = cmd_str.split() calls = [mock.call(*command, run_as_root=False), mock.call(*command, run_as_root=False), mock.call(*command, run_as_root=False), mock.call(*command, run_as_root=False)] mock_execute.assert_has_calls(calls) @mock.patch.object(timeutils, 'utcnow') def test_make_event_online(self, mock_utcnow): current_time = timeutils.utcnow() mock_utcnow.return_value = current_time obj = handle_host.HandleHost() hostname = 'test_hostname' current_status = 'online' ret = obj._make_event(hostname, current_status) event_type = ec.EventConstants.EVENT_STARTED cluster_status = current_status.upper() host_status = ec.EventConstants.HOST_STATUS_NORMAL event = { 'notification': { 'type': ec.EventConstants.TYPE_COMPUTE_HOST, 'hostname': hostname, 'generated_time': current_time, 'payload': { 'event': event_type, 'cluster_status': cluster_status, 'host_status': host_status } } } self.assertEqual(event, ret) @mock.patch.object(timeutils, 'utcnow') @mock.patch.object(handle_host.HandleHost, '_is_poweroff') def test_make_event_offline_disable_ipmi_check_false_poweroff( self, mock_is_poweroff, mock_utcnow): mock_is_poweroff.return_value = True current_time = timeutils.utcnow() mock_utcnow.return_value = current_time CONF.host.disable_ipmi_check = False obj = handle_host.HandleHost() hostname = 'test_hostname' current_status = 'offline' ret = obj._make_event(hostname, current_status) event_type = ec.EventConstants.EVENT_STOPPED cluster_status = current_status.upper() host_status = ec.EventConstants.HOST_STATUS_NORMAL event = { 'notification': { 'type': ec.EventConstants.TYPE_COMPUTE_HOST, 'hostname': hostname, 'generated_time': current_time, 'payload': { 'event': event_type, 'cluster_status': cluster_status, 'host_status': host_status } } } self.assertEqual(event, ret) mock_is_poweroff.assert_called_once_with(hostname) @mock.patch.object(timeutils, 'utcnow') @mock.patch.object(handle_host.HandleHost, '_is_poweroff') def test_make_event_offline_disable_ipmi_check_false_poweron( self, mock_is_poweroff, mock_utcnow): mock_is_poweroff.return_value = False current_time = timeutils.utcnow() mock_utcnow.return_value = current_time CONF.host.disable_ipmi_check = False obj = handle_host.HandleHost() hostname = 'test_hostname' current_status = 'offline' ret = obj._make_event(hostname, current_status) event_type = ec.EventConstants.EVENT_STOPPED cluster_status = current_status.upper() host_status = ec.EventConstants.HOST_STATUS_UNKNOWN event = { 'notification': { 'type': ec.EventConstants.TYPE_COMPUTE_HOST, 'hostname': hostname, 'generated_time': current_time, 'payload': { 'event': event_type, 'cluster_status': cluster_status, 'host_status': host_status } } } self.assertEqual(event, ret) mock_is_poweroff.assert_called_once_with(hostname) @mock.patch.object(timeutils, 'utcnow') def test_make_event_offline_disable_ipmi_check_true(self, mock_utcnow): current_time = timeutils.utcnow() mock_utcnow.return_value = current_time CONF.host.disable_ipmi_check = True obj = handle_host.HandleHost() hostname = 'test_hostname' current_status = 'offline' ret = obj._make_event(hostname, current_status) event_type = ec.EventConstants.EVENT_STOPPED cluster_status = current_status.upper() host_status = ec.EventConstants.HOST_STATUS_NORMAL event = { 'notification': { 'type': ec.EventConstants.TYPE_COMPUTE_HOST, 'hostname': hostname, 'generated_time': current_time, 'payload': { 'event': event_type, 'cluster_status': cluster_status, 'host_status': host_status } } } self.assertEqual(event, ret) @mock.patch.object(masakari.SendNotification, 'send_notification') @mock.patch.object(handle_host.HandleHost, '_make_event') @mock.patch.object(hold_host_status.HostHoldStatus, 'set_host_status') @mock.patch.object(hold_host_status.HostHoldStatus, 'get_host_status') @mock.patch.object(socket, 'gethostname') def test_check_if_status_changed( self, mock_gethostname, mock_get_host_status, mock_set_host_status, mock_make_event, mock_send_notification): mock_gethostname.return_value = 'node1' mock_get_host_status.side_effect = \ [None, 'online', 'online', 'online'] mock_set_host_status.return_value = None test_event = {'notification': 'test'} mock_make_event.return_value = test_event status_tag = ElementTree.fromstring(STATUS_TAG_XML) node_state_tag_list = list(status_tag) obj = handle_host.HandleHost() obj._check_if_status_changed(node_state_tag_list) node_state_node2 = node_state_tag_list[1] node_state_node3 = node_state_tag_list[2] node_state_node4 = node_state_tag_list[3] node_state_node5 = node_state_tag_list[4] node2 = node_state_node2.get('uname') node3 = node_state_node3.get('uname') node4 = node_state_node4.get('uname') node5 = node_state_node5.get('uname') calls_get_host_status = [mock.call(node2), mock.call(node3), mock.call(node4), mock.call(node5)] calls_set_host_status = [mock.call(node_state_node4), mock.call(node_state_node5)] mock_get_host_status.assert_has_calls(calls_get_host_status) mock_set_host_status.assert_has_calls(calls_set_host_status) mock_make_event.assert_called_once_with(node4, 'offline') mock_send_notification.assert_called_once_with( CONF.host.api_retry_max, CONF.host.api_retry_interval, test_event) @mock.patch.object(masakari.SendNotification, 'send_notification') @mock.patch.object(handle_host.HandleHost, '_make_event') @mock.patch.object(hold_host_status.HostHoldStatus, 'set_host_status') @mock.patch.object(hold_host_status.HostHoldStatus, 'get_host_status') @mock.patch.object(socket, 'gethostname') def test_check_if_status_changed_with_3_samples( self, mock_gethostname, mock_get_host_status, mock_set_host_status, mock_make_event, mock_send_notification): mock_gethostname.return_value = 'node1' mock_get_host_status.side_effect = \ [None, 'online', 'online', 'online'] mock_set_host_status.return_value = None test_event = {'notification': 'test'} mock_make_event.return_value = test_event status_tag = ElementTree.fromstring(STATUS_TAG_XML) node_state_tag_list = list(status_tag) CONF.host.monitoring_samples = 3 obj = handle_host.HandleHost() obj.monitoring_data = { "node1": deque(['online', 'online', 'online'], maxlen=3), "node2": deque(['offline', 'online', 'online'], maxlen=3), "node3": deque(['offline'], maxlen=3), "node4": deque(['online', 'offline', 'offline'], maxlen=3), "node5": deque(['online', 'online', 'online'], maxlen=3), } obj._check_if_status_changed(node_state_tag_list) self.assertEqual(deque(['online', 'online', 'online'], maxlen=3), obj.monitoring_data.get('node2')) self.assertEqual('online', obj.get_stabilised_host_status('node2')) self.assertIn(mock.call(node_state_tag_list[1]), mock_set_host_status.mock_calls) self.assertEqual(deque(['offline', 'online'], maxlen=3), obj.monitoring_data.get('node3')) self.assertEqual('_being_collected', obj.get_stabilised_host_status('node3')) self.assertNotIn(mock.call(node_state_tag_list[2]), mock_set_host_status.mock_calls) self.assertEqual(deque(['offline', 'offline', 'offline'], maxlen=3), obj.monitoring_data.get('node4')) self.assertEqual('offline', obj.get_stabilised_host_status('node4')) self.assertIn(mock.call(node_state_tag_list[3]), mock_set_host_status.mock_calls) self.assertEqual(deque(['online', 'online', 'other'], maxlen=3), obj.monitoring_data.get('node5')) self.assertEqual('_uncertain', obj.get_stabilised_host_status('node5')) self.assertNotIn(mock.call(node_state_tag_list[4]), mock_set_host_status.mock_calls) mock_make_event.assert_called_once_with("node4", 'offline') mock_send_notification.assert_called_once_with( CONF.host.api_retry_max, CONF.host.api_retry_interval, test_event) @mock.patch.object(handle_host.HandleHost, '_check_if_status_changed') @mock.patch.object(parse_crmmon_xml.ParseCrmMonXml, 'get_node_state_tag_list') @mock.patch.object(parse_crmmon_xml.ParseCrmMonXml, 'has_quorum') @mock.patch.object(parse_crmmon_xml.ParseCrmMonXml, 'set_crmmon_xml') @mock.patch.object(handle_host.HandleHost, '_get_crmmon_xml') def test_check_host_status_by_crm_mon( self, mock_get_crmmon_xml, mock_set_crmmon_xml, mock_has_quorum, mock_get_node_state_tag_list, mock_check_if_status_changed): mock_get_crmmon_xml.return_value = CRMMON_NODES_TAG_XML mock_set_crmmon_xml.return_value = None mock_has_quorum.return_value = True status_tag = ElementTree.fromstring(CRMMON_NODES_TAG_XML) node_state_tag_list = list(status_tag) mock_get_node_state_tag_list.return_value = node_state_tag_list mock_check_if_status_changed.return_value = None obj = handle_host.HandleHost() ret = obj._check_host_status_by_crm_mon() self.assertEqual(0, ret) mock_set_crmmon_xml.assert_called_once_with(CRMMON_NODES_TAG_XML) mock_get_node_state_tag_list.assert_called_once_with() mock_check_if_status_changed.assert_called_once_with( [ {'uname': 'remote1', 'crmd': 'online'}, {'uname': 'remote2', 'crmd': 'online'}, {'uname': 'remote3', 'crmd': 'online'}]) @mock.patch.object(handle_host.HandleHost, '_check_if_status_changed') @mock.patch.object(parse_crmmon_xml.ParseCrmMonXml, 'has_quorum') @mock.patch.object(parse_crmmon_xml.ParseCrmMonXml, 'set_crmmon_xml') @mock.patch.object(handle_host.HandleHost, '_get_crmmon_xml') def test_check_host_status_by_crm_mon_no_quorum( self, mock_get_crmmon_xml, mock_set_crmmon_xml, mock_has_quorum, mock_check_if_status_changed): mock_get_crmmon_xml.return_value = CRMMON_NODES_TAG_XML mock_set_crmmon_xml.return_value = None mock_has_quorum.return_value = False obj = handle_host.HandleHost() ret = obj._check_host_status_by_crm_mon() self.assertEqual(2, ret) mock_set_crmmon_xml.assert_called_once_with(CRMMON_NODES_TAG_XML) mock_check_if_status_changed.assert_not_called() @mock.patch.object(parse_crmmon_xml.ParseCrmMonXml, 'has_quorum') @mock.patch.object(parse_crmmon_xml.ParseCrmMonXml, 'get_node_state_tag_list') @mock.patch.object(parse_crmmon_xml.ParseCrmMonXml, 'set_crmmon_xml') @mock.patch.object(handle_host.HandleHost, '_get_crmmon_xml') def test_check_host_status_by_crm_mon_not_have_node_state_tag( self, mock_get_crmmon_xml, mock_set_crmmon_xml, mock_get_node_state_tag_list, mock_has_quorum): mock_get_crmmon_xml.return_value = CRMMON_NODES_TAG_XML mock_set_crmmon_xml.return_value = None mock_get_node_state_tag_list.return_value = [] mock_has_quorum.return_value = True obj = handle_host.HandleHost() self.assertRaisesRegex( Exception, "Failed to get nodes tag from crm_mon xml.", obj._check_host_status_by_crm_mon) mock_get_crmmon_xml.assert_called_once_with() mock_set_crmmon_xml.assert_called_once_with(CRMMON_NODES_TAG_XML) mock_get_node_state_tag_list.assert_called_once_with() @mock.patch.object(handle_host.HandleHost, '_get_crmmon_xml') def test_check_host_status_by_crm_mon_xml_is_None( self, mock_get_crmmon_xml): mock_get_crmmon_xml.return_value = None obj = handle_host.HandleHost() ret = obj._check_host_status_by_crm_mon() self.assertEqual(1, ret) mock_get_crmmon_xml.assert_called_once_with() @mock.patch.object(handle_host.HandleHost, '_check_if_status_changed') @mock.patch.object(parse_cib_xml.ParseCibXml, 'get_node_state_tag_list') @mock.patch.object(parse_cib_xml.ParseCibXml, 'have_quorum') @mock.patch.object(parse_cib_xml.ParseCibXml, 'set_cib_xml') @mock.patch.object(handle_host.HandleHost, '_get_cib_xml') def test_check_host_status_by_cibadmin( self, mock_get_cib_xml, mock_set_cib_xml, mock_have_quorum, mock_get_node_state_tag_list, mock_check_if_status_changed): mock_get_cib_xml.return_value = STATUS_TAG_XML mock_set_cib_xml.return_value = None mock_have_quorum.return_value = 1 status_tag = ElementTree.fromstring(STATUS_TAG_XML) node_state_tag_list = list(status_tag) mock_get_node_state_tag_list.return_value = node_state_tag_list mock_check_if_status_changed.return_value = None obj = handle_host.HandleHost() ret = obj._check_host_status_by_cibadmin() self.assertEqual(0, ret) mock_get_cib_xml.assert_called_once_with() mock_set_cib_xml.assert_called_once_with(STATUS_TAG_XML) mock_have_quorum.assert_called_once_with() mock_get_node_state_tag_list.assert_called_once_with() mock_check_if_status_changed.assert_called_once_with( node_state_tag_list) @mock.patch.object(handle_host.HandleHost, '_check_if_status_changed') @mock.patch.object(parse_cib_xml.ParseCibXml, 'have_quorum') @mock.patch.object(parse_cib_xml.ParseCibXml, 'set_cib_xml') @mock.patch.object(handle_host.HandleHost, '_get_cib_xml') def test_check_host_status_by_cibadmin_no_quorum( self, mock_get_cib_xml, mock_set_cib_xml, mock_have_quorum, mock_check_if_status_changed): mock_get_cib_xml.return_value = STATUS_TAG_XML mock_set_cib_xml.return_value = None mock_have_quorum.return_value = 0 obj = handle_host.HandleHost() ret = obj._check_host_status_by_cibadmin() self.assertEqual(2, ret) mock_get_cib_xml.assert_called_once_with() mock_set_cib_xml.assert_called_once_with(STATUS_TAG_XML) mock_have_quorum.assert_called_once_with() mock_check_if_status_changed.assert_not_called() @mock.patch.object(handle_host.HandleHost, '_get_cib_xml') def test_check_host_status_by_cibadmin_cib_xml_is_None( self, mock_get_cib_xml): mock_get_cib_xml.return_value = None obj = handle_host.HandleHost() ret = obj._check_host_status_by_cibadmin() self.assertEqual(1, ret) mock_get_cib_xml.assert_called_once_with() @mock.patch.object(parse_cib_xml.ParseCibXml, 'get_node_state_tag_list') @mock.patch.object(parse_cib_xml.ParseCibXml, 'have_quorum') @mock.patch.object(parse_cib_xml.ParseCibXml, 'set_cib_xml') @mock.patch.object(handle_host.HandleHost, '_get_cib_xml') def test_check_host_status_by_cibadmin_not_have_node_state_tag( self, mock_get_cib_xml, mock_set_cib_xml, mock_have_quorum, mock_get_node_state_tag_list): mock_get_cib_xml.return_value = STATUS_TAG_XML mock_set_cib_xml.return_value = None mock_have_quorum.return_value = 1 mock_get_node_state_tag_list.return_value = [] obj = handle_host.HandleHost() self.assertRaisesRegex( Exception, "Failed to get node_state tag from cib xml.", obj._check_host_status_by_cibadmin) mock_get_cib_xml.assert_called_once_with() mock_set_cib_xml.assert_called_once_with(STATUS_TAG_XML) mock_have_quorum.assert_called_once_with() mock_get_node_state_tag_list.assert_called_once_with() def test_stop(self): obj = handle_host.HandleHost() obj.stop() self.assertFalse(obj.running) @mock.patch.object(eventlet.greenthread, 'sleep') @mock.patch.object(handle_host.HandleHost, '_check_host_status_by_cibadmin') @mock.patch.object(handle_host.HandleHost, '_check_host_status_by_crmadmin') @mock.patch.object(handle_host.HandleHost, '_check_pacemaker_services') @mock.patch.object(handle_host.HandleHost, '_check_hb_line') def test_monitor_hosts(self, mock_check_hb_line, mock_check_pacemaker_services, mock_check_host_status_by_crmadmin, mock_check_host_status_by_cibadmin, mock_sleep): mock_check_hb_line.side_effect = \ [0, 1, 2, 0, Exception("Test exception."), 0, KeyboardInterrupt()] mock_check_pacemaker_services.side_effect = [True, False, False, True] mock_check_host_status_by_crmadmin.side_effect = [0, 1] mock_check_host_status_by_cibadmin.side_effect = [0, 1, 0] mock_sleep.return_value = None obj = handle_host.HandleHost() self.assertRaises(KeyboardInterrupt, obj.monitor_hosts) self.assertEqual(7, mock_check_hb_line.call_count) self.assertEqual(4, mock_check_pacemaker_services.call_count) mock_check_pacemaker_services.assert_called_with('pacemaker_remote') self.assertEqual(2, mock_check_host_status_by_crmadmin.call_count) self.assertEqual(3, mock_check_host_status_by_cibadmin.call_count) @mock.patch.object(eventlet.greenthread, 'sleep') @mock.patch.object(handle_host.HandleHost, '_check_host_status_by_crm_mon') @mock.patch.object(handle_host.HandleHost, '_check_pacemaker_services') @mock.patch.object(handle_host.HandleHost, '_check_hb_line') def test_monitor_hosts_remotes_only(self, mock_check_hb_line, mock_check_pacemaker_services, mock_check_host_status_by_crm_mon, mock_sleep): CONF.host.restrict_to_remotes = True mock_check_hb_line.side_effect = \ [0, Exception("Test exception."), 0, KeyboardInterrupt()] mock_check_pacemaker_services.return_value = True mock_check_host_status_by_crm_mon.return_value = 0 mock_sleep.return_value = None obj = handle_host.HandleHost() self.assertRaises(KeyboardInterrupt, obj.monitor_hosts) self.assertEqual(4, mock_check_hb_line.call_count) self.assertEqual(2, mock_check_pacemaker_services.call_count) mock_check_pacemaker_services.assert_called_with('pacemaker_remote') self.assertEqual(2, mock_check_host_status_by_crm_mon.call_count) ././@PaxHeader0000000000000000000000000000020600000000000011453 xustar0000000000000000112 path=masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/host_handler/test_hold_host_status.py 22 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/host_handler/test_hold_host_status.0000664000175000017500000000274200000000000034544 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import testtools from xml.etree import ElementTree import eventlet from masakarimonitors.hostmonitor.host_handler import hold_host_status eventlet.monkey_patch(os=False) NODE_STATE_XML = '' \ ' ' \ '' NODE_STATE_TAG = ElementTree.fromstring(NODE_STATE_XML) class TestHostHoldStatus(testtools.TestCase): def setUp(self): super(TestHostHoldStatus, self).setUp() def test_set_host_status(self): obj = hold_host_status.HostHoldStatus() obj.set_host_status(NODE_STATE_TAG) self.assertEqual('online', obj.get_host_status('masakari-node')) def test_get_host_status(self): obj = hold_host_status.HostHoldStatus() obj.set_host_status(NODE_STATE_TAG) self.assertEqual('online', obj.get_host_status('masakari-node')) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/host_handler/test_parse_cib_xml.py0000664000175000017500000001351000000000000034331 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import testtools from unittest import mock from xml.etree import ElementTree import eventlet from masakarimonitors.hostmonitor.host_handler import parse_cib_xml eventlet.monkey_patch(os=False) CIB_XML = '' \ ' ' \ ' test' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ '' CIB_TAG = ElementTree.fromstring(CIB_XML) class TestParseCibXml(testtools.TestCase): def setUp(self): super(TestParseCibXml, self).setUp() @mock.patch.object(ElementTree, 'fromstring') def test_set_cib_xml(self, mock_fromstring): obj = parse_cib_xml.ParseCibXml() mock_fromstring.return_value = CIB_TAG obj.set_cib_xml(CIB_XML) def test_have_quorum(self): obj = parse_cib_xml.ParseCibXml() obj.set_cib_xml(CIB_XML) self.assertEqual(1, obj.have_quorum()) def test_get_node_state_tag_list(self): obj = parse_cib_xml.ParseCibXml() obj.set_cib_xml(CIB_XML) node_state_tag_list = obj.get_node_state_tag_list() for node_state_tag in node_state_tag_list: self.assertEqual('online', node_state_tag.get('crmd')) def test_get_stonith_ipmi_params(self): obj = parse_cib_xml.ParseCibXml() obj.set_cib_xml(CIB_XML) ipmi_values = obj.get_stonith_ipmi_params('compute-node') self.assertEqual('192.168.10.21', ipmi_values['ipaddr']) self.assertEqual('admin', ipmi_values['userid']) self.assertEqual('password', ipmi_values['passwd']) self.assertEqual('lanplus', ipmi_values['interface']) ././@PaxHeader0000000000000000000000000000020600000000000011453 xustar0000000000000000112 path=masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/host_handler/test_parse_crmmon_xml.py 22 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/host_handler/test_parse_crmmon_xml.0000664000175000017500000000671300000000000034525 0ustar00zuulzuul00000000000000# Copyright(c) 2019 Canonical Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import testtools from masakarimonitors.hostmonitor.host_handler import parse_crmmon_xml CRMMON_XML = '' \ '' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ ' ' \ '' CRMMON_XML_NO_QUORUM = \ '' \ '' \ ' ' \ ' ' \ ' ' \ ' ' \ '' CRMMON_NONODES_XML = '' \ '' \ ' ' \ ' ' \ '' CRMMON_NONODES_TAG_XML = '' \ '' \ '' class TestParseCrmMonXml(testtools.TestCase): def setUp(self): super(TestParseCrmMonXml, self).setUp() def test_set_crmmon_xml(self): obj = parse_crmmon_xml.ParseCrmMonXml() obj.set_crmmon_xml(CRMMON_XML) def test_has_quorum(self): obj = parse_crmmon_xml.ParseCrmMonXml() obj.set_crmmon_xml(CRMMON_XML) self.assertEqual(True, obj.has_quorum()) def test_has_quorum_no_quorum(self): obj = parse_crmmon_xml.ParseCrmMonXml() obj.set_crmmon_xml(CRMMON_XML_NO_QUORUM) self.assertEqual(False, obj.has_quorum()) def test_get_node_state_tag_list(self): obj = parse_crmmon_xml.ParseCrmMonXml() obj.set_crmmon_xml(CRMMON_XML) node_state_tag_list = obj.get_node_state_tag_list() expected = { 'node-1': 'true', 'node-2': 'false', 'node-3': 'true'} for node_state_tag in node_state_tag_list: self.assertEqual( expected[node_state_tag.get('name')], node_state_tag.get('online')) def test_get_node_state_tag_list_unset(self): obj = parse_crmmon_xml.ParseCrmMonXml() self.assertEqual(obj.get_node_state_tag_list(), []) def test_get_node_state_tag_list_nonodes(self): obj = parse_crmmon_xml.ParseCrmMonXml() obj.set_crmmon_xml(CRMMON_NONODES_XML) self.assertEqual(obj.get_node_state_tag_list(), []) def test_get_node_state_tag_list_nonodes_tag(self): obj = parse_crmmon_xml.ParseCrmMonXml() obj.set_crmmon_xml(CRMMON_NONODES_TAG_XML) self.assertEqual(obj.get_node_state_tag_list(), []) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/hostmonitor/test_host.py0000664000175000017500000000726600000000000030040 0ustar00zuulzuul00000000000000# Copyright(c) 2017 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import os import testtools from unittest import mock from stevedore import driver from masakarimonitors.hostmonitor import host class TestHostmonitorManager(testtools.TestCase): def setUp(self): super(TestHostmonitorManager, self).setUp() @mock.patch.object(driver, 'DriverManager') def test_init_host(self, mock_DriverManager): mock_driver = mock.Mock() mock_DriverManager.return_value = mock_driver host_manager = host.HostmonitorManager() host_manager.init_host() mock_DriverManager.assert_called_once_with( namespace='hostmonitor.driver', name='default', invoke_on_load=True, invoke_args=(), ) @mock.patch.object(os, '_exit') @mock.patch.object(driver, 'DriverManager') def test_init_host_exception(self, mock_DriverManager, mock_exit): mock_DriverManager.side_effect = Exception("Test exception.") mock_exit.return_value = None host_manager = host.HostmonitorManager() host_manager.init_host() mock_DriverManager.assert_called_once_with( namespace='hostmonitor.driver', name='default', invoke_on_load=True, invoke_args=(), ) mock_exit.assert_called_once_with(1) @mock.patch.object(driver, 'DriverManager') def test_stop(self, mock_DriverManager): mock_driver = mock.Mock() mock_DriverManager.return_value = mock_driver host_manager = host.HostmonitorManager() host_manager.init_host() host_manager.stop() mock_DriverManager.assert_called_once_with( namespace='hostmonitor.driver', name='default', invoke_on_load=True, invoke_args=(), ) mock_driver.driver.stop.assert_called_once_with() @mock.patch.object(driver, 'DriverManager') def test_main(self, mock_DriverManager): mock_driver = mock.Mock() mock_DriverManager.return_value = mock_driver host_manager = host.HostmonitorManager() host_manager.init_host() ret = host_manager.main() mock_DriverManager.assert_called_once_with( namespace='hostmonitor.driver', name='default', invoke_on_load=True, invoke_args=(), ) mock_driver.driver.monitor_hosts.assert_called_once_with() self.assertIsNone(ret) @mock.patch.object(driver, 'DriverManager') def test_main_exception(self, mock_DriverManager): mock_driver = mock.Mock() mock_DriverManager.return_value = mock_driver mock_driver.driver.monitor_hosts.side_effect = \ Exception("Test exception.") host_manager = host.HostmonitorManager() host_manager.init_host() ret = host_manager.main() mock_DriverManager.assert_called_once_with( namespace='hostmonitor.driver', name='default', invoke_on_load=True, invoke_args=(), ) mock_driver.driver.monitor_hosts.assert_called_once_with() self.assertIsNone(ret) ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1972024 masakari_monitors-19.0.0/masakarimonitors/tests/unit/instancemonitor/0000775000175000017500000000000000000000000026266 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/instancemonitor/__init__.py0000664000175000017500000000000000000000000030365 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1972024 masakari_monitors-19.0.0/masakarimonitors/tests/unit/instancemonitor/libvirt_handler/0000775000175000017500000000000000000000000031436 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/instancemonitor/libvirt_handler/__init__.py0000664000175000017500000000000000000000000033535 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000020500000000000011452 xustar0000000000000000111 path=masakari_monitors-19.0.0/masakarimonitors/tests/unit/instancemonitor/libvirt_handler/test_callback.py 22 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/instancemonitor/libvirt_handler/test_callback.p0000664000175000017500000000411200000000000034410 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import socket import testtools from unittest import mock import uuid import eventlet from oslo_utils import timeutils from masakarimonitors.ha import masakari from masakarimonitors.instancemonitor.libvirt_handler import callback from masakarimonitors.objects import event_constants as ec eventlet.monkey_patch(os=False) class TestCallback(testtools.TestCase): def setUp(self): super(TestCallback, self).setUp() @mock.patch.object(masakari.SendNotification, 'send_notification') def test_libvirt_event_callback(self, mock_send_notification): mock_send_notification.return_value = None obj = callback.Callback() event_id = 0 details = 5 domain_uuid = uuid.uuid4() notice_type = ec.EventConstants.TYPE_VM hostname = socket.gethostname() current_time = timeutils.utcnow() obj.libvirt_event_callback(event_id, details, domain_uuid, notice_type, hostname, current_time) retry_max = 12 retry_interval = 10 event = { 'notification': { 'type': notice_type, 'hostname': hostname, 'generated_time': current_time, 'payload': { 'event': event_id, 'instance_uuid': domain_uuid, 'vir_domain_event': details } } } mock_send_notification.assert_called_once_with( retry_max, retry_interval, event) ././@PaxHeader0000000000000000000000000000021000000000000011446 xustar0000000000000000114 path=masakari_monitors-19.0.0/masakarimonitors/tests/unit/instancemonitor/libvirt_handler/test_eventfilter.py 22 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/instancemonitor/libvirt_handler/test_eventfilte0000664000175000017500000001332100000000000034565 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import socket import testtools import threading from unittest import mock import uuid import eventlet from oslo_utils import excutils from oslo_utils import timeutils from masakarimonitors.instancemonitor.libvirt_handler import callback from masakarimonitors.instancemonitor.libvirt_handler import eventfilter from masakarimonitors.instancemonitor.libvirt_handler \ import eventfilter_table as evft from masakarimonitors.objects import event_constants as ec eventlet.monkey_patch(os=False) class TestEventFilter(testtools.TestCase): def setUp(self): super(TestEventFilter, self).setUp() @mock.patch.object(excutils, 'save_and_reraise_exception') @mock.patch.object(callback.Callback, 'libvirt_event_callback') @mock.patch.object(timeutils, 'utcnow') def test_vir_event_filter(self, mock_utcnow, mock_libvirt_event_callback, mock_save_and_reraise_exception): current_time = timeutils.utcnow() mock_utcnow.return_value = current_time mock_libvirt_event_callback.return_value = None mock_save_and_reraise_exception.return_value = None obj = eventfilter.EventFilter() eventID = 0 eventType = 5 detail = 5 uuID = uuid.uuid4() obj.vir_event_filter(eventID, eventType, detail, uuID) mock_libvirt_event_callback.assert_called_once_with( evft.eventID_dic[eventID], evft.detail_dic[eventID][eventType][detail], uuID, ec.EventConstants.TYPE_VM, socket.gethostname(), current_time) mock_save_and_reraise_exception.assert_not_called() @mock.patch.object(excutils, 'save_and_reraise_exception') @mock.patch.object(callback.Callback, 'libvirt_event_callback') def test_vir_event_filter_unmatched(self, mock_libvirt_event_callback, mock_save_and_reraise_exception): mock_libvirt_event_callback.return_value = None mock_save_and_reraise_exception.return_value = None obj = eventfilter.EventFilter() eventID = 0 eventType = 5 detail = 2 uuID = uuid.uuid4() obj.vir_event_filter(eventID, eventType, detail, uuID) mock_libvirt_event_callback.assert_not_called() mock_save_and_reraise_exception.assert_not_called() @mock.patch.object(excutils, 'save_and_reraise_exception') @mock.patch.object(callback.Callback, 'libvirt_event_callback') def test_vir_event_filter_key_error(self, mock_libvirt_event_callback, mock_save_and_reraise_exception): mock_libvirt_event_callback.return_value = None mock_save_and_reraise_exception.return_value = None obj = eventfilter.EventFilter() eventID = 0 eventType = 0 detail = 0 uuID = uuid.uuid4() obj.vir_event_filter(eventID, eventType, detail, uuID) mock_libvirt_event_callback.assert_not_called() mock_save_and_reraise_exception.assert_not_called() @mock.patch.object(excutils, 'save_and_reraise_exception') @mock.patch.object(callback.Callback, 'libvirt_event_callback') @mock.patch.object(threading, 'Thread') def test_vir_event_filter_type_error(self, mock_Thread, mock_libvirt_event_callback, mock_save_and_reraise_exception): mock_Thread.side_effect = TypeError("Threading exception.") mock_libvirt_event_callback.return_value = None mock_save_and_reraise_exception.return_value = None obj = eventfilter.EventFilter() eventID = 0 eventType = 5 detail = 5 uuID = uuid.uuid4() obj.vir_event_filter(eventID, eventType, detail, uuID) mock_libvirt_event_callback.assert_not_called() mock_save_and_reraise_exception.assert_not_called() @mock.patch.object(excutils, 'save_and_reraise_exception') @mock.patch.object(callback.Callback, 'libvirt_event_callback') @mock.patch.object(threading, 'Thread') def test_vir_event_filter_index_error(self, mock_Thread, mock_libvirt_event_callback, mock_save_and_reraise_exception): mock_Thread.side_effect = IndexError("Threading exception.") mock_libvirt_event_callback.return_value = None mock_save_and_reraise_exception.return_value = None obj = eventfilter.EventFilter() eventID = 0 eventType = 5 detail = 5 uuID = uuid.uuid4() obj.vir_event_filter(eventID, eventType, detail, uuID) mock_libvirt_event_callback.assert_not_called() mock_save_and_reraise_exception.assert_not_called() @mock.patch.object(callback.Callback, 'libvirt_event_callback') @mock.patch.object(threading, 'Thread') def test_vir_event_filter_other_exception(self, mock_Thread, mock_libvirt_event_callback): mock_Thread.side_effect = NameError("Threading exception.") mock_libvirt_event_callback.return_value = None obj = eventfilter.EventFilter() eventID = 0 eventType = 5 detail = 5 uuID = uuid.uuid4() self.assertRaises(NameError, obj.vir_event_filter, eventID, eventType, detail, uuID) mock_libvirt_event_callback.assert_not_called() ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/instancemonitor/test_instance.py0000664000175000017500000002336600000000000031515 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import libvirt import testtools import threading import time from unittest import mock import uuid import eventlet from masakarimonitors.instancemonitor import instance from masakarimonitors.instancemonitor.libvirt_handler import eventfilter eventlet.monkey_patch(os=False) class TestInstancemonitorManager(testtools.TestCase): def setUp(self): super(TestInstancemonitorManager, self).setUp() def _make_callback_params(self): mock_conn = mock.Mock() mock_dom = mock.Mock() test_uuid = uuid.uuid4() mock_dom.UUIDString.return_value = test_uuid mock_opaque = mock.Mock() return mock_conn, mock_dom, mock_opaque, test_uuid @mock.patch.object(libvirt, 'virEventRunDefaultImpl') def test_vir_event_loop_native_run(self, mock_virEventRunDefaultImpl): mock_virEventRunDefaultImpl.side_effect = Exception("Test exception.") obj = instance.InstancemonitorManager() exception_flag = False try: obj._vir_event_loop_native_run() except Exception: exception_flag = True self.assertTrue(exception_flag) mock_virEventRunDefaultImpl.assert_called_once() @mock.patch.object(eventfilter.EventFilter, 'vir_event_filter') def test_my_domain_event_callback(self, mock_vir_event_filter): mock_vir_event_filter.return_value = None mock_conn, mock_dom, mock_opaque, test_uuid = \ self._make_callback_params() obj = instance.InstancemonitorManager() obj._my_domain_event_callback(mock_conn, mock_dom, 0, 1, mock_opaque) mock_vir_event_filter.assert_called_once_with( libvirt.VIR_DOMAIN_EVENT_ID_LIFECYCLE, 0, 1, test_uuid) @mock.patch.object(eventfilter.EventFilter, 'vir_event_filter') def test_my_domain_event_reboot_callback(self, mock_vir_event_filter): mock_vir_event_filter.return_value = None mock_conn, mock_dom, mock_opaque, test_uuid = \ self._make_callback_params() obj = instance.InstancemonitorManager() obj._my_domain_event_reboot_callback(mock_conn, mock_dom, mock_opaque) mock_vir_event_filter.assert_called_once_with( libvirt.VIR_DOMAIN_EVENT_ID_REBOOT, -1, -1, test_uuid) @mock.patch.object(eventfilter.EventFilter, 'vir_event_filter') def test_my_domain_event_rtc_change_callback(self, mock_vir_event_filter): mock_vir_event_filter.return_value = None mock_conn, mock_dom, mock_opaque, test_uuid = \ self._make_callback_params() utcoffset = "" obj = instance.InstancemonitorManager() obj._my_domain_event_rtc_change_callback( mock_conn, mock_dom, utcoffset, mock_opaque) mock_vir_event_filter.assert_called_once_with( libvirt.VIR_DOMAIN_EVENT_ID_RTC_CHANGE, -1, -1, test_uuid) @mock.patch.object(eventfilter.EventFilter, 'vir_event_filter') def test_my_domain_event_watchdog_callback(self, mock_vir_event_filter): mock_vir_event_filter.return_value = None mock_conn, mock_dom, mock_opaque, test_uuid = \ self._make_callback_params() action = 0 obj = instance.InstancemonitorManager() obj._my_domain_event_watchdog_callback( mock_conn, mock_dom, action, mock_opaque) mock_vir_event_filter.assert_called_once_with( libvirt.VIR_DOMAIN_EVENT_ID_WATCHDOG, action, -1, test_uuid) @mock.patch.object(eventfilter.EventFilter, 'vir_event_filter') def test_my_domain_event_io_error_callback(self, mock_vir_event_filter): mock_vir_event_filter.return_value = None mock_conn, mock_dom, mock_opaque, test_uuid = \ self._make_callback_params() srcpath = "" devalias = "" action = 0 obj = instance.InstancemonitorManager() obj._my_domain_event_io_error_callback( mock_conn, mock_dom, srcpath, devalias, action, mock_opaque) mock_vir_event_filter.assert_called_once_with( libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR, action, -1, test_uuid) @mock.patch.object(eventfilter.EventFilter, 'vir_event_filter') def test_my_domain_event_graphics_callback(self, mock_vir_event_filter): mock_vir_event_filter.return_value = None mock_conn, mock_dom, mock_opaque, test_uuid = \ self._make_callback_params() phase = 0 localAddr = "" remoteAddr = "" authScheme = "" subject = "" obj = instance.InstancemonitorManager() obj._my_domain_event_graphics_callback( mock_conn, mock_dom, phase, localAddr, remoteAddr, authScheme, subject, mock_opaque) mock_vir_event_filter.assert_called_once_with( libvirt.VIR_DOMAIN_EVENT_ID_GRAPHICS, -1, phase, test_uuid) @mock.patch.object(eventfilter.EventFilter, 'vir_event_filter') def test_my_domain_event_disk_change_callback( self, mock_vir_event_filter): mock_vir_event_filter.return_value = None mock_conn, mock_dom, mock_opaque, test_uuid = \ self._make_callback_params() oldSrcPath = "" newSrcPath = "" devAlias = "" reason = "" obj = instance.InstancemonitorManager() obj._my_domain_event_disk_change_callback( mock_conn, mock_dom, oldSrcPath, newSrcPath, devAlias, reason, mock_opaque) mock_vir_event_filter.assert_called_once_with( libvirt.VIR_DOMAIN_EVENT_ID_DISK_CHANGE, -1, -1, test_uuid) @mock.patch.object(eventfilter.EventFilter, 'vir_event_filter') def test_my_domain_event_io_error_reason_callback( self, mock_vir_event_filter): mock_vir_event_filter.return_value = None mock_conn, mock_dom, mock_opaque, test_uuid = \ self._make_callback_params() srcPath = "" devAlias = "" action = "" reason = "" obj = instance.InstancemonitorManager() obj._my_domain_event_io_error_reason_callback( mock_conn, mock_dom, srcPath, devAlias, action, reason, mock_opaque) mock_vir_event_filter.assert_called_once_with( libvirt.VIR_DOMAIN_EVENT_ID_IO_ERROR_REASON, -1, -1, test_uuid) @mock.patch.object(eventfilter.EventFilter, 'vir_event_filter') def test_my_domain_event_generic_callback(self, mock_vir_event_filter): mock_vir_event_filter.return_value = None mock_conn, mock_dom, mock_opaque, test_uuid = \ self._make_callback_params() obj = instance.InstancemonitorManager() obj._my_domain_event_generic_callback( mock_conn, mock_dom, mock_opaque) mock_vir_event_filter.assert_called_once_with( libvirt.VIR_DOMAIN_EVENT_ID_CONTROL_ERROR, -1, -1, test_uuid) def test_err_handler(self): obj = instance.InstancemonitorManager() obj._err_handler("Test context.", ('err0.', 'err1', 'err2')) def test_stop(self): obj = instance.InstancemonitorManager() obj.stop() self.assertFalse(obj.running) @mock.patch.object(time, 'sleep') @mock.patch.object(eventlet.greenthread, 'sleep') @mock.patch.object(libvirt, 'openAuth') @mock.patch.object(threading, 'Thread') @mock.patch.object(libvirt, 'virEventRegisterDefaultImpl') def test_main(self, mock_virEventRegisterDefaultImpl, mock_Thread, mock_openAuth, mock_greenthread_sleep, mock_time_sleep): mock_virEventRegisterDefaultImpl.return_value = None mock_event_loop_thread = mock.Mock(return_value=None) mock_Thread.return_value = mock_event_loop_thread mock_vc = mock.Mock() mock_openAuth.return_value = mock_vc mock_vc.domainEventRegisterAny.side_effect = \ [0, 0, 0, 0, 0, 0, 0, 0, 0] mock_vc.setKeepAlive.return_value = None mock_vc.isAlive.side_effect = [1, 0] mock_vc.domainEventDeregisterAny.side_effect = \ [None, None, None, None, None, None, None, None, Exception("Test exception.")] mock_vc.close.return_value = None mock_greenthread_sleep.return_value = None mock_time_sleep.side_effect = Exception("Test exception.") obj = instance.InstancemonitorManager() exception_flag = False try: obj.main() except Exception: exception_flag = True handlers_count = 9 self.assertTrue(exception_flag) mock_virEventRegisterDefaultImpl.assert_called_once() self.assertEqual(True, mock_event_loop_thread.daemon) mock_event_loop_thread.start.assert_called_once() mock_openAuth.assert_called_once_with( "qemu:///system", [[2, 6, 8, 5, 7, 9], instance.InstancemonitorManager._connect_auth_cb, None], 1) self.assertEqual( handlers_count, mock_vc.domainEventRegisterAny.call_count) mock_vc.setKeepAlive.assert_called_once_with(5, 3) self.assertEqual(2, mock_vc.isAlive.call_count) self.assertEqual( handlers_count, mock_vc.domainEventDeregisterAny.call_count) mock_vc.close.assert_called_once() ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1972024 masakari_monitors-19.0.0/masakarimonitors/tests/unit/introspectiveinstancemonitor/0000775000175000017500000000000000000000000031105 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/introspectiveinstancemonitor/__init__.py0000775000175000017500000000000000000000000033207 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000021100000000000011447 xustar0000000000000000115 path=masakari_monitors-19.0.0/masakarimonitors/tests/unit/introspectiveinstancemonitor/test_monitor_manager.py 22 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/introspectiveinstancemonitor/test_monitor_manag0000664000175000017500000000257000000000000034725 0ustar00zuulzuul00000000000000# Copyright(c) 2018 WindRiver Systems # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import eventlet import libvirt import testtools from unittest import mock from masakarimonitors.introspectiveinstancemonitor import instance eventlet.monkey_patch(os=False) class TestMonitorManager(testtools.TestCase): def setUp(self): super(TestMonitorManager, self).setUp() @mock.patch.object(libvirt, 'virEventRunDefaultImpl') def test_vir_event_loop_native_run(self, mock_virEventRunDefaultImpl): mock_virEventRunDefaultImpl.side_effect = Exception("Test exception.") obj = instance.IntrospectiveInstanceMonitorManager() exception_flag = False try: obj._vir_event_loop_native_run() except Exception: exception_flag = True self.assertTrue(exception_flag) mock_virEventRunDefaultImpl.assert_called_once() ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/introspectiveinstancemonitor/test_qemu_utils.py0000775000175000017500000000767500000000000034727 0ustar00zuulzuul00000000000000# Copyright(c) 2018 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import libvirt import testtools from unittest import mock import uuid from masakarimonitors.introspectiveinstancemonitor import instance as object from masakarimonitors.introspectiveinstancemonitor import qemu_utils class TestQemuUtils(testtools.TestCase): def setup(self): super(TestQemuUtils, self).setUp() @mock.patch.object(qemu_utils.libvirt, 'virDomain') def test_getVmFsm(self, mock_domain): """To test the state machines Initial stage should be dicovery and will advance to a healthy stage after enough pingable events. Also, it should reach an error stage from healthy after a pingable event failure. """ reference = qemu_utils.QemuGuestAgent() mock_domain.UUID.return_value = uuid.uuid4() reference.getVmFsm(mock_domain) self.assertEqual(reference.getVmFsm(mock_domain).current_state, 'discovery') reference.getVmFsm(mock_domain).process_event('guest_not_pingable') self.assertEqual(reference.getVmFsm(mock_domain).current_state, 'discovery') reference.getVmFsm(mock_domain).process_event('guest_pingable') self.assertEqual(reference.getVmFsm(mock_domain).current_state, 'healthy') reference.getVmFsm(mock_domain).process_event('guest_pingable') self.assertEqual(reference.getVmFsm(mock_domain).current_state, 'healthy') reference.getVmFsm(mock_domain).process_event('guest_not_pingable') self.assertEqual(reference.getVmFsm(mock_domain).current_state, 'error') @mock.patch.object(qemu_utils.libvirt, 'virDomain') def test_hasQemuGuestAgent(self, mock_domain): mock_domain.UUID.return_value = 'testuuid' mock_domain.state.return_value = libvirt.VIR_DOMAIN_RUNNING, 'reason' mock_domain.XMLDesc.return_value = """ """ obj = qemu_utils.QemuGuestAgent() self.assertFalse(obj._hasQemuGuestAgent(mock_domain)) mock_domain.XMLDesc.return_value = """
""" obj = qemu_utils.QemuGuestAgent() self.assertTrue(obj._hasQemuGuestAgent(mock_domain)) @mock.patch.object(qemu_utils, 'resetJournal') def test_resetJournal(self, mock_resetJournal): mock_resetJournal.return_value = None obj = object.IntrospectiveInstanceMonitorManager() event_id = 0 domain_uuid = uuid.uuid4() event_type = libvirt.VIR_DOMAIN_EVENT_STARTED detail = libvirt.VIR_DOMAIN_EVENT_STARTED_BOOTED obj._reset_journal(event_id, event_type, detail, domain_uuid) mock_resetJournal.assert_called_once_with(domain_uuid) ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1972024 masakari_monitors-19.0.0/masakarimonitors/tests/unit/processmonitor/0000775000175000017500000000000000000000000026140 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/processmonitor/__init__.py0000664000175000017500000000000000000000000030237 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1972024 masakari_monitors-19.0.0/masakarimonitors/tests/unit/processmonitor/process_handler/0000775000175000017500000000000000000000000031313 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/processmonitor/process_handler/__init__.py0000664000175000017500000000000000000000000033412 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000021200000000000011450 xustar0000000000000000116 path=masakari_monitors-19.0.0/masakarimonitors/tests/unit/processmonitor/process_handler/test_handle_process.py 22 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/processmonitor/process_handler/test_handle_proc0000664000175000017500000003612400000000000034561 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import socket import testtools from unittest import mock import ddt import eventlet from oslo_utils import timeutils import masakarimonitors.conf from masakarimonitors.ha import masakari from masakarimonitors.objects import event_constants as ec from masakarimonitors.processmonitor.process_handler import handle_process from masakarimonitors import utils CONF = masakarimonitors.conf.CONF eventlet.monkey_patch(os=False) NOVA_COMPUTE = 'nova-compute' MOCK_PROCESS_LIST = [ { 'id': 1, 'process_name': '/usr/local/bin/nova-compute', 'start_command': 'mock_start_command', 'pre_start_command': 'mock_pre_start_command', 'post_start_command': 'mock_post_start_command', 'restart_command': 'mock_restart_command', 'pre_restart_command': 'mock_pre_restart_command', 'post_restart_command': 'mock_post_restart_command', 'run_as_root': True }, ] MOCK_DOWN_PROCESS_LIST = [ { 'id': 1, 'process_name': '/usr/local/bin/nova-compute', 'start_command': 'mock_start_command', 'pre_start_command': 'mock_pre_start_command', 'post_start_command': 'mock_post_start_command', 'restart_command': 'mock_restart_command', 'pre_restart_command': 'mock_pre_restart_command', 'post_restart_command': 'mock_post_restart_command', 'run_as_root': True }, ] PS_RESULT = \ "\n" \ "UID PID PPID C STIME TTY TIME CMD\n" \ "root 11187 1 0 18:52 ? 00:00:00 /usr/local/bin/nova-compute\n" @ddt.ddt class TestHandleProcess(testtools.TestCase): def setUp(self): super(TestHandleProcess, self).setUp() def test_set_process_list(self): process_list = MOCK_PROCESS_LIST obj = handle_process.HandleProcess() obj.set_process_list(process_list) self.assertEqual(process_list, obj.process_list) @ddt.data("/usr/local/bin/nova-compute", "nova-compute") def test_make_event(self, process_name): obj = handle_process.HandleProcess() event = obj._make_event(process_name) self.assertEqual( NOVA_COMPUTE, event['notification']['payload']['process_name']) @mock.patch.object(utils, 'execute') def test_start_processes(self, mock_execute): process_list = MOCK_PROCESS_LIST obj = handle_process.HandleProcess() obj.set_process_list(process_list) mock_execute.side_effect = [('test_stdout', ''), ('test_stdout', ''), ('test_stdout', 'test_stderr')] obj.start_processes() mock_execute.assert_any_call( MOCK_PROCESS_LIST[0].get('pre_start_command'), run_as_root=MOCK_PROCESS_LIST[0].get('run_as_root')) mock_execute.assert_any_call( MOCK_PROCESS_LIST[0].get('start_command'), run_as_root=MOCK_PROCESS_LIST[0].get('run_as_root')) mock_execute.assert_any_call( MOCK_PROCESS_LIST[0].get('post_start_command'), run_as_root=MOCK_PROCESS_LIST[0].get('run_as_root')) @mock.patch.object(utils, 'execute') def test_start_processes_pre_cmd_fail(self, mock_execute): process_list = MOCK_PROCESS_LIST obj = handle_process.HandleProcess() obj.set_process_list(process_list) mock_execute.return_value = ('test_stdout', 'test_stderr') obj.start_processes() mock_execute.assert_called_once_with( MOCK_PROCESS_LIST[0].get('pre_start_command'), run_as_root=MOCK_PROCESS_LIST[0].get('run_as_root')) @mock.patch.object(utils, 'execute') def test_monitor_processes(self, mock_execute): process_list = MOCK_PROCESS_LIST obj = handle_process.HandleProcess() obj.set_process_list(process_list) mock_execute.return_value = (PS_RESULT, '') down_process_list = obj.monitor_processes() self.assertEqual([], down_process_list) mock_execute.assert_called_once_with( 'ps', '-ef', run_as_root=False) @mock.patch.object(utils, 'execute') def test_monitor_processes_not_found(self, mock_execute): process_list = MOCK_PROCESS_LIST obj = handle_process.HandleProcess() obj.set_process_list(process_list) mock_execute.return_value = ('', '') down_process_list = obj.monitor_processes() self.assertEqual(MOCK_PROCESS_LIST, down_process_list) mock_execute.assert_called_once_with( 'ps', '-ef', run_as_root=False) @mock.patch.object(utils, 'execute') def test_monitor_processes_exception(self, mock_execute): process_list = MOCK_PROCESS_LIST obj = handle_process.HandleProcess() obj.set_process_list(process_list) mock_execute.side_effect = Exception("Test exception.") down_process_list = obj.monitor_processes() self.assertEqual([], down_process_list) @mock.patch.object(utils, 'execute') def test_restart_processes(self, mock_execute): process_list = MOCK_PROCESS_LIST obj = handle_process.HandleProcess() obj.set_process_list(process_list) down_process_list = MOCK_DOWN_PROCESS_LIST mock_execute.side_effect = [('test_stdout', ''), ('test_stdout', ''), ('test_stdout', '')] obj.restart_processes(down_process_list) mock_execute.assert_any_call( down_process_list[0].get('pre_restart_command'), run_as_root=down_process_list[0].get('run_as_root')) mock_execute.assert_any_call( down_process_list[0].get('restart_command'), run_as_root=down_process_list[0].get('run_as_root')) mock_execute.assert_any_call( down_process_list[0].get('post_restart_command'), run_as_root=down_process_list[0].get('run_as_root')) self.assertEqual([], obj.restart_failure_list) @mock.patch.object(utils, 'execute') def test_restart_processes_failed_to_restart_previously( self, mock_execute): process_list = MOCK_PROCESS_LIST obj = handle_process.HandleProcess() obj.set_process_list(process_list) restart_failure_list = [MOCK_DOWN_PROCESS_LIST[0].get('process_name')] obj.restart_failure_list = restart_failure_list down_process_list = MOCK_DOWN_PROCESS_LIST obj.restart_processes(down_process_list) self.assertEqual(restart_failure_list, obj.restart_failure_list) mock_execute.assert_not_called() @mock.patch.object(masakari.SendNotification, 'send_notification') @mock.patch.object(timeutils, 'utcnow') @mock.patch.object(eventlet.greenthread, 'sleep') @mock.patch.object(utils, 'execute') def test_restart_processes_pre_restart_command_retry_over( self, mock_execute, mock_sleep, mock_utcnow, mock_send_notification): process_list = MOCK_PROCESS_LIST obj = handle_process.HandleProcess() obj.set_process_list(process_list) down_process_list = MOCK_DOWN_PROCESS_LIST mock_execute.side_effect = [('test_stdout', 'test_stderr'), ('test_stdout', 'test_stderr'), ('test_stdout', 'test_stderr'), ('test_stdout', 'test_stderr')] mock_sleep.return_value = None current_time = timeutils.utcnow() mock_utcnow.return_value = current_time mock_send_notification.return_value = None obj.restart_processes(down_process_list) pre_execute_count = CONF.process.restart_retries + 1 self.assertEqual(pre_execute_count, mock_execute.call_count) for var in range(0, mock_execute.call_count): args, kwargs = mock_execute.call_args_list[var] self.assertEqual( (down_process_list[0].get('pre_restart_command'),), args) self.assertEqual({'run_as_root': True}, kwargs) event = { 'notification': { 'type': ec.EventConstants.TYPE_PROCESS, 'hostname': socket.gethostname(), 'generated_time': current_time, 'payload': { 'event': ec.EventConstants.EVENT_STOPPED, 'process_name': NOVA_COMPUTE } } } mock_send_notification.assert_called_once_with( CONF.process.api_retry_max, CONF.process.api_retry_interval, event) self.assertEqual( [down_process_list[0].get('process_name')], obj.restart_failure_list) @mock.patch.object(masakari.SendNotification, 'send_notification') @mock.patch.object(timeutils, 'utcnow') @mock.patch.object(eventlet.greenthread, 'sleep') @mock.patch.object(utils, 'execute') def test_restart_processes_restart_command_retry_over( self, mock_execute, mock_sleep, mock_utcnow, mock_send_notification): process_list = MOCK_PROCESS_LIST obj = handle_process.HandleProcess() obj.set_process_list(process_list) down_process_list = MOCK_DOWN_PROCESS_LIST mock_execute.side_effect = [('test_stdout', ''), ('test_stdout', 'test_stderr'), ('test_stdout', ''), ('test_stdout', 'test_stderr'), ('test_stdout', ''), ('test_stdout', 'test_stderr'), ('test_stdout', ''), ('test_stdout', 'test_stderr')] mock_sleep.return_value = None current_time = timeutils.utcnow() mock_utcnow.return_value = current_time mock_send_notification.return_value = None obj.restart_processes(down_process_list) pre_execute_count = CONF.process.restart_retries + 1 execute_count = CONF.process.restart_retries + 1 total_execute_count = pre_execute_count + execute_count self.assertEqual(total_execute_count, mock_execute.call_count) for var in range(0, mock_execute.call_count): # Execute order of restart_command is the second. execute_order = 2 if (var + 1) % execute_order == 0: args, kwargs = mock_execute.call_args_list[var] self.assertEqual( (down_process_list[0].get('restart_command'),), args) self.assertEqual({'run_as_root': True}, kwargs) event = { 'notification': { 'type': ec.EventConstants.TYPE_PROCESS, 'hostname': socket.gethostname(), 'generated_time': current_time, 'payload': { 'event': ec.EventConstants.EVENT_STOPPED, 'process_name': NOVA_COMPUTE } } } mock_send_notification.assert_called_once_with( CONF.process.api_retry_max, CONF.process.api_retry_interval, event) self.assertEqual( [down_process_list[0].get('process_name')], obj.restart_failure_list) @mock.patch.object(masakari.SendNotification, 'send_notification') @mock.patch.object(timeutils, 'utcnow') @mock.patch.object(eventlet.greenthread, 'sleep') @mock.patch.object(utils, 'execute') def test_restart_processes_post_restart_command_retry_over( self, mock_execute, mock_sleep, mock_utcnow, mock_send_notification): process_list = MOCK_PROCESS_LIST obj = handle_process.HandleProcess() obj.set_process_list(process_list) down_process_list = MOCK_DOWN_PROCESS_LIST mock_execute.side_effect = [('test_stdout', ''), ('test_stdout', ''), ('test_stdout', 'test_stderr'), ('test_stdout', ''), ('test_stdout', ''), ('test_stdout', 'test_stderr'), ('test_stdout', ''), ('test_stdout', ''), ('test_stdout', 'test_stderr'), ('test_stdout', ''), ('test_stdout', ''), ('test_stdout', 'test_stderr')] mock_sleep.return_value = None current_time = timeutils.utcnow() mock_utcnow.return_value = current_time mock_send_notification.return_value = None obj.restart_processes(down_process_list) pre_execute_count = CONF.process.restart_retries + 1 execute_count = CONF.process.restart_retries + 1 post_execute_count = CONF.process.restart_retries + 1 total_execute_count = \ pre_execute_count + execute_count + post_execute_count self.assertEqual(total_execute_count, mock_execute.call_count) for var in range(0, mock_execute.call_count): # Execute order of restart_command is the third. execute_order = 3 if (var + 1) % execute_order == 0: args, kwargs = mock_execute.call_args_list[var] self.assertEqual( (down_process_list[0].get('post_restart_command'),), args) self.assertEqual({'run_as_root': True}, kwargs) event = { 'notification': { 'type': ec.EventConstants.TYPE_PROCESS, 'hostname': socket.gethostname(), 'generated_time': current_time, 'payload': { 'event': ec.EventConstants.EVENT_STOPPED, 'process_name': NOVA_COMPUTE } } } mock_send_notification.assert_called_once_with( CONF.process.api_retry_max, CONF.process.api_retry_interval, event) self.assertEqual( [down_process_list[0].get('process_name')], obj.restart_failure_list) @mock.patch.object(utils, 'execute') def test_execute_cmd_exception(self, mock_execute): mock_execute.side_effect = Exception("Test exception.") obj = handle_process.HandleProcess() ret = obj._execute_cmd(MOCK_PROCESS_LIST[0].get('start_command'), MOCK_PROCESS_LIST[0].get('run_as_root')) self.assertEqual(ret, 1) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/processmonitor/test_process.py0000664000175000017500000001440000000000000031226 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import testtools from unittest import mock import yaml import eventlet from masakarimonitors.processmonitor import process as processmonitor_manager from masakarimonitors.processmonitor.process_handler import handle_process eventlet.monkey_patch(os=False) MOCK_PROCESS_LIST = [ { 'process_name': 'mock_process_name_A', 'start_command': 'mock_start_command', 'pre_start_command': 'mock_pre_start_command', 'post_start_command': 'mock_post_start_command', 'restart_command': 'mock_restart_command', 'pre_restart_command': 'mock_pre_restart_command', 'post_restart_command': 'mock_post_restart_command', 'run_as_root': True }, { 'id': 2, 'process_name': 'mock_process_name_B', 'start_command': 'mock_start_command', 'pre_start_command': 'mock_pre_start_command', 'post_start_command': 'mock_post_start_command', 'restart_command': 'mock_restart_command', 'pre_restart_command': 'mock_pre_restart_command', 'post_restart_command': 'mock_post_restart_command', 'run_as_root': True }, ] MOCK_DOWN_PROCESS_LIST = [ { 'process_name': 'mock_process_name_A', 'start_command': 'mock_start_command', 'pre_start_command': 'mock_pre_start_command', 'post_start_command': 'mock_post_start_command', 'restart_command': 'mock_restart_command', 'pre_restart_command': 'mock_pre_restart_command', 'post_restart_command': 'mock_post_restart_command', 'run_as_root': True }, ] class TestProcessmonitorManager(testtools.TestCase): def setUp(self): super(TestProcessmonitorManager, self).setUp() def _get_mock_process_list(self, call_count): if call_count == 0: return MOCK_PROCESS_LIST else: return @mock.patch.object(eventlet.greenthread, 'sleep') @mock.patch.object(handle_process.HandleProcess, 'restart_processes') @mock.patch.object(handle_process.HandleProcess, 'monitor_processes') @mock.patch.object(handle_process.HandleProcess, 'start_processes') @mock.patch.object(handle_process.HandleProcess, 'set_process_list') @mock.patch.object(yaml, 'load') @mock.patch('builtins.open') def test_main(self, mock_file, mock_load, mock_set_process_list, mock_start_processes, mock_monitor_processes, mock_restart_processes, mock_sleep): mock_load.side_effect = [self._get_mock_process_list(0), self._get_mock_process_list(0), self._get_mock_process_list(1)] mock_set_process_list.return_value = None mock_start_processes.return_value = None mock_monitor_processes.side_effect = [MOCK_DOWN_PROCESS_LIST, []] mock_restart_processes.return_value = None mock_sleep.return_value = None obj = processmonitor_manager.ProcessmonitorManager() obj.main() mock_set_process_list.assert_called_with(MOCK_PROCESS_LIST) mock_start_processes.assert_called_once_with() self.assertEqual(2, mock_monitor_processes.call_count) mock_restart_processes.assert_called_once_with(MOCK_DOWN_PROCESS_LIST) @mock.patch.object(handle_process.HandleProcess, 'restart_processes') @mock.patch.object(handle_process.HandleProcess, 'monitor_processes') @mock.patch.object(handle_process.HandleProcess, 'start_processes') @mock.patch.object(handle_process.HandleProcess, 'set_process_list') @mock.patch.object(yaml, 'load') @mock.patch('builtins.open') def test_main_exception(self, mock_file, mock_load, mock_set_process_list, mock_start_processes, mock_monitor_processes, mock_restart_processes): mock_load.return_value = self._get_mock_process_list(0) mock_set_process_list.return_value = None mock_start_processes.side_effect = Exception("Test exception.") obj = processmonitor_manager.ProcessmonitorManager() obj.main() mock_set_process_list.assert_called_once_with(MOCK_PROCESS_LIST) mock_start_processes.assert_called_once_with() mock_monitor_processes.assert_not_called() mock_restart_processes.assert_not_called() @mock.patch.object(handle_process.HandleProcess, 'set_process_list') @mock.patch.object(yaml, 'load') @mock.patch('builtins.open') def test_load_process_list_yaml_error(self, mock_file, mock_load, mock_set_process_list): mock_load.side_effect = yaml.YAMLError obj = processmonitor_manager.ProcessmonitorManager() obj.main() mock_set_process_list.assert_not_called() @mock.patch.object(handle_process.HandleProcess, 'set_process_list') @mock.patch.object(yaml, 'load') @mock.patch('builtins.open') def test_load_process_list_exception(self, mock_file, mock_load, mock_set_process_list): mock_load.side_effect = Exception("Test exception.") obj = processmonitor_manager.ProcessmonitorManager() obj.main() mock_set_process_list.assert_not_called() def test_stop(self): obj = processmonitor_manager.ProcessmonitorManager() obj.stop() self.assertFalse(obj.running) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/tests/unit/test_hacking.py0000664000175000017500000001525700000000000026101 0ustar00zuulzuul00000000000000# Copyright 2017 NTT Data. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import textwrap from unittest import mock import pycodestyle import testtools from masakarimonitors.hacking import checks class HackingTestCase(testtools.TestCase): """This class tests the hacking checks in masakarimonitors.hacking.checks by passing strings to the check methods like the pycodestyle/flake8 parser would. The parser loops over each line in the file and then passes the parameters to the check method. The parameter names in the check method dictate what type of object is passed to the check method. The parameter types are:: logical_line: A processed line with the following modifications: - Multi-line statements converted to a single line. - Stripped left and right. - Contents of strings replaced with "xxx" of same length. - Comments removed. physical_line: Raw line of text from the input file. lines: a list of the raw lines from the input file tokens: the tokens that contribute to this logical line line_number: line number in the input file total_lines: number of lines in the input file blank_lines: blank lines before this one indent_char: indentation character in this file (" " or "\t") indent_level: indentation (with tabs expanded to multiples of 8) previous_indent_level: indentation on previous line previous_logical: previous logical line filename: Path of the file being run through pycodestyle When running a test on a check method the return will be False/None if there is no violation in the sample input. If there is an error a tuple is returned with a position in the line, and a message. So to check the result just assertTrue if the check is expected to fail and assertFalse if it should pass. """ # We are patching pycodestyle so that only the check under test is actually # installed. @mock.patch('pycodestyle._checks', {'physical_line': {}, 'logical_line': {}, 'tree': {}}) def _run_check(self, code, checker, filename=None): pycodestyle.register_check(checker) lines = textwrap.dedent(code).strip().splitlines(True) checker = pycodestyle.Checker(filename=filename, lines=lines) checker.check_all() checker.report._deferred_print.sort() return checker.report._deferred_print def _assert_has_errors(self, code, checker, expected_errors=None, filename=None): actual_errors = [e[:3] for e in self._run_check(code, checker, filename)] self.assertEqual(expected_errors or [], actual_errors) def _assert_has_no_errors(self, code, checker, filename=None): self._assert_has_errors(code, checker, filename=filename) def test_check_explicit_underscore_import(self): self.assertEqual(len(list(checks.check_explicit_underscore_import( "LOG.info(_('My info message'))", "masakarimonitors/tests/other_files.py"))), 1) self.assertEqual(len(list(checks.check_explicit_underscore_import( "msg = _('My message')", "masakarimonitors/tests/other_files.py"))), 1) self.assertEqual(len(list(checks.check_explicit_underscore_import( "from masakarimonitors.i18n import _", "masakarimonitors/tests/other_files.py"))), 0) self.assertEqual(len(list(checks.check_explicit_underscore_import( "LOG.info(_('My info message'))", "masakarimonitors/tests/other_files.py"))), 0) self.assertEqual(len(list(checks.check_explicit_underscore_import( "msg = _('My message')", "masakarimonitors/tests/other_files.py"))), 0) self.assertEqual(len(list(checks.check_explicit_underscore_import( "from masakarimonitors.i18n import _", "masakarimonitors/tests/other_files2.py"))), 0) self.assertEqual(len(list(checks.check_explicit_underscore_import( "msg = _('My message')", "masakarimonitors/tests/other_files2.py"))), 0) self.assertEqual(len(list(checks.check_explicit_underscore_import( "_ = translations.ugettext", "masakarimonitors/tests/other_files3.py"))), 0) self.assertEqual(len(list(checks.check_explicit_underscore_import( "msg = _('My message')", "masakarimonitors/tests/other_files3.py"))), 0) def test_no_translate_logs(self): self.assertEqual(1, len(list(checks.no_translate_logs( "LOG.error(_LE('foo'))")))) self.assertEqual(0, len(list(checks.no_translate_logs( "LOG.debug('foo')")))) self.assertEqual(1, len(list(checks.no_translate_logs( "LOG.info(_LI('foo'))")))) self.assertEqual(1, len(list(checks.no_translate_logs( "LOG.warning(_LW('foo'))")))) self.assertEqual(1, len(list(checks.no_translate_logs( "LOG.critical(_LC('foo'))")))) def test_yield_followed_by_space(self): code = """ yield(x, y) yield{"type": "test"} yield[a, b, c] yield"test" yield'test' """ errors = [(x + 1, 0, 'M303') for x in range(5)] self._assert_has_errors(code, checks.yield_followed_by_space, expected_errors=errors) code = """ yield x yield (x, y) yield {"type": "test"} yield [a, b, c] yield "test" yield 'test' yieldx_func(a, b) """ self._assert_has_no_errors(code, checks.yield_followed_by_space) def test_no_log_warn(self): code = """ LOG.warn("LOG.warn is deprecated") """ errors = [(1, 0, 'M305')] self._assert_has_errors(code, checks.no_log_warn, expected_errors=errors) code = """ LOG.warning("LOG.warn is deprecated") """ self._assert_has_no_errors(code, checks.no_log_warn) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/utils.py0000664000175000017500000001066600000000000022454 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Utilities and helper functions.""" import contextlib import functools import inspect import pyclbr import shutil import sys import tempfile from oslo_concurrency import lockutils from oslo_concurrency import processutils from oslo_log import log as logging from oslo_utils import importutils import masakarimonitors.conf from masakarimonitors.i18n import _ from masakarimonitors import privsep CONF = masakarimonitors.conf.CONF LOG = logging.getLogger(__name__) def monkey_patch(): """monkey_patch function. If the CONF.monkey_patch set as True, this function patches a decorator for all functions in specified modules. You can set decorators for each modules using CONF.monkey_patch_modules. The format is "Module path:Decorator function". name - name of the function function - object of the function """ # If CONF.monkey_patch is not True, this function do nothing. if not CONF.monkey_patch: return def is_method(obj): return inspect.ismethod(obj) or inspect.isfunction(obj) # Get list of modules and decorators for module_and_decorator in CONF.monkey_patch_modules: md_value = module_and_decorator.split(':') if len(md_value) != 2: msg = _("'monkey_patch_modules' config option is not configured " "correctly") raise Exception(msg) module, decorator_name = md_value # import decorator function decorator = importutils.import_class(decorator_name) __import__(module) # Retrieve module information using pyclbr module_data = pyclbr.readmodule_ex(module) for key, value in module_data.items(): # set the decorator for the class methods if isinstance(value, pyclbr.Class): clz = importutils.import_class("%s.%s" % (module, key)) for method, func in inspect.getmembers(clz, is_method): setattr(clz, method, decorator("%s.%s.%s" % (module, key, method), func)) # set the decorator for the function if isinstance(value, pyclbr.Function): func = importutils.import_class("%s.%s" % (module, key)) setattr(sys.modules[module], key, decorator("%s.%s" % (module, key), func)) @contextlib.contextmanager def tempdir(**kwargs): argdict = kwargs.copy() if 'dir' not in argdict: argdict['dir'] = CONF.tempdir tmpdir = tempfile.mkdtemp(**argdict) try: yield tmpdir finally: try: shutil.rmtree(tmpdir) except OSError as e: LOG.error('Could not remove tmpdir: %s', e) @privsep.monitors_priv.entrypoint def privsep_execute(*cmd, **kwargs): return processutils.execute(*cmd, **kwargs) def execute(*cmd, **kwargs): """Convenience wrapper around oslo's execute() method.""" if 'run_as_root' in kwargs and kwargs.get('run_as_root'): return privsep_execute(*cmd, **kwargs) else: return processutils.execute(*cmd, **kwargs) def synchronized(name, semaphores=None, blocking=False): def wrap(f): @functools.wraps(f) def inner(*args, **kwargs): lock_str = 'masakarimonitors-%s' % name int_lock = lockutils.internal_lock(lock_str, semaphores=semaphores) msg = "Lock blocking: %s on resource %s " % (lock_str, f.__name__) """Acquiring lock: %(lock_str)s on resource """ if not int_lock.acquire(blocking=blocking): raise Exception(msg) try: return f(*args, **kwargs) finally: """Releasing lock: %(lock_str)s on resource """ int_lock.release() return inner return wrap ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/masakarimonitors/version.py0000664000175000017500000000460000000000000022770 0ustar00zuulzuul00000000000000# Copyright(c) 2016 Nippon Telegraph and Telephone Corporation # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from pbr import version as pbr_version MONITORS_VENDOR = "OpenStack Foundation" MONITORS_PRODUCT = "OpenStack Masakari Monitors" MONITORS_PACKAGE = None # OS distro package version suffix loaded = False version_info = pbr_version.VersionInfo('masakari-monitors') version_string = version_info.version_string def _load_config(): # Don't load in global context, since we can't assume # these modules are accessible when distutils uses # this module import configparser from oslo_config import cfg from oslo_log import log as logging global loaded, MONITORS_VENDOR, MONITORS_PRODUCT, MONITORS_PACKAGE if loaded: return loaded = True cfgfile = cfg.CONF.find_file("release") if cfgfile is None: return try: cfg = configparser.RawConfigParser() cfg.read(cfgfile) if cfg.has_option("Masakarimonitors", "vendor"): MONITORS_VENDOR = cfg.get("Masakarimonitors", "vendor") if cfg.has_option("Masakarimonitors", "product"): MONITORS_PRODUCT = cfg.get("Masakarimonitors", "product") if cfg.has_option("Masakarimonitors", "package"): MONITORS_PACKAGE = cfg.get("Masakarimonitors", "package") except Exception as ex: LOG = logging.getLogger(__name__) LOG.error("Failed to load %(cfgfile)s: %(ex)s", {'cfgfile': cfgfile, 'ex': ex}) def vendor_string(): _load_config() return MONITORS_VENDOR def product_string(): _load_config() return MONITORS_PRODUCT def package_string(): _load_config() return MONITORS_PACKAGE def version_string_with_package(): if package_string() is None: return version_info.version_string() else: return "%s-%s" % (version_info.version_string(), package_string()) ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.1772006 masakari_monitors-19.0.0/releasenotes/0000775000175000017500000000000000000000000020037 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.2012029 masakari_monitors-19.0.0/releasenotes/notes/0000775000175000017500000000000000000000000021167 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/.placeholder0000664000175000017500000000000000000000000023440 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/bp-retry-check-when-host-failure-78649c512ef79199.yaml0000664000175000017500000000115200000000000032260 0ustar00zuulzuul00000000000000--- features: - | Support for repeated check of node status in hostmonitor. Repeated check is more reliable than single check to determine host status, especially when there is network instability in play. With this feature, the following config option can be set. .. code-block:: ini [host] monitoring_samples = 3 The above means 3 checks will be done before the node status is decided. The default value is 1 which is backwards compatible. `Blueprint retry-check-when-host-failure `__ ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/bug-1866660-ef8624f5283b2e5e.yaml0000664000175000017500000000065500000000000025723 0ustar00zuulzuul00000000000000--- features: - | A 'hostname' option has been added to the Masakari monitors configuration file which replaces the 'host' option. The option should be used to specify the Hostname, FQDN or IP address of the host. deprecations: - | The 'host' option in the 'DEFAULT' section has been deprecated as it clashes with the name of a section in the same file. It has been replaced by the 'hostname' option. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/bug-1878548-5fab31aec6ba5407.yaml0000664000175000017500000000036100000000000026037 0ustar00zuulzuul00000000000000--- fixes: - | Fixes hostmonitor reporting hosts down because of Pacemaker cluster partitioning. Now hostmonitor properly respects the status of Pacemaker cluster quorum. `LP#1878548 `__ ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/bug-1930361-fa8ce8e9781ea967.yaml0000664000175000017500000000021100000000000025774 0ustar00zuulzuul00000000000000--- fixes: - | Fixes hostmonitor hanging forever after certain exceptions. `LP#1930361 `__ ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/deprecate-processmonitor-95c8aefbc749c1ed.yaml0000664000175000017500000000043600000000000031546 0ustar00zuulzuul00000000000000--- deprecations: - | Masakari Process Monitor (``masakari-processmonitor``) is deprecated and may be removed in any later release. Please use modern tools, such as systemd, Docker or Kubernetes (K8s), to control, monitor and ensure the desired processes are running. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/drop-cap-net-admin-8d7d7cfb274e9547.yaml0000664000175000017500000000016600000000000027676 0ustar00zuulzuul00000000000000--- other: - | Masakari hostmonitor and processmonitor will no longer require ``CAP_NET_ADMIN`` capability. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/drop-py-2-7-b28de816eac45468.yaml0000664000175000017500000000034000000000000026172 0ustar00zuulzuul00000000000000--- upgrade: - | Python 2.7 support has been dropped. Last release of masakari-monitors to support python 2.7 is OpenStack Train. The minimum version of Python now supported by masakari-monitors is Python 3.6. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/hostmonitor-driver-based-on-consul-03f7e619d91e7e06.yaml0000664000175000017500000000035600000000000033073 0ustar00zuulzuul00000000000000--- features: - | Added hostmonitor driver based on consul. It can detects interfaces connectivity status via multiple consul clusters, and sends notifition to trigger host failure recovery according to defined HA strategy. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/hostmonitor-systemd-89696f96a654a918.yaml0000664000175000017500000000030000000000000030152 0ustar00zuulzuul00000000000000--- features: - | Adds ``pacemaker_node_type`` option to ``hostmonitor`` to allow skipping systemd service status checks which are often impossible in containerised deployments. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/introspectiveinstancemonitor-f4bc71f029b61d49.yaml0000664000175000017500000000026100000000000032324 0ustar00zuulzuul00000000000000--- features: - | Added Introspective Instance Monitoring through QEMU Guest Agent, in order to detect, report and optionally recover VMs from internal VM faults. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/libvirt-sasl-support-edf1388c556a594b.yaml0000664000175000017500000000051500000000000030431 0ustar00zuulzuul00000000000000--- features: - | Add support for libvirt auth in instancemonitor. Use the standard methods to provide the actual authentication credentials. The SASL library and pluggable authentication modules should be installed on the instancemonitor host, use the packages provided in the distro alongside libvirt-python. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/libvirt-still-required-22a8d817ee8d0be8.yaml0000664000175000017500000000073500000000000031001 0ustar00zuulzuul00000000000000--- other: - | *Note to packagers*: ``libvirt-python`` package is still required by instance-oriented monitors. It is available in most distros as ``python3-libvirt``. It is no longer listed in ``requirements.txt`` so it will not get installed when installing ``masakari-monitors`` package from PyPI. The change was done to follow how Nova handles this requirement. This is to satisfy both libvirtd compatibility and recent pip sanity checks. ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/notes/pythonize-monitors-081e74dfaf78fe99.yaml0000664000175000017500000000050000000000000030272 0ustar00zuulzuul00000000000000--- features: - Added the masakari-hostmonitor and masakari-processmonitor which were implemented in python. deprecations: - The masakari-hostmonitor and masakari-processmonitor which were implemented in bash shell script are deprecated as of the Ocata release and will be removed in the Queens release. ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.2052033 masakari_monitors-19.0.0/releasenotes/source/0000775000175000017500000000000000000000000021337 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/2023.1.rst0000664000175000017500000000020200000000000022610 0ustar00zuulzuul00000000000000=========================== 2023.1 Series Release Notes =========================== .. release-notes:: :branch: stable/2023.1 ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/2023.2.rst0000664000175000017500000000020200000000000022611 0ustar00zuulzuul00000000000000=========================== 2023.2 Series Release Notes =========================== .. release-notes:: :branch: stable/2023.2 ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/2024.1.rst0000664000175000017500000000020200000000000022611 0ustar00zuulzuul00000000000000=========================== 2024.1 Series Release Notes =========================== .. release-notes:: :branch: stable/2024.1 ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/2024.2.rst0000664000175000017500000000020200000000000022612 0ustar00zuulzuul00000000000000=========================== 2024.2 Series Release Notes =========================== .. release-notes:: :branch: stable/2024.2 ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.2052033 masakari_monitors-19.0.0/releasenotes/source/_static/0000775000175000017500000000000000000000000022765 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/_static/.placeholder0000664000175000017500000000000000000000000025236 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.2052033 masakari_monitors-19.0.0/releasenotes/source/_templates/0000775000175000017500000000000000000000000023474 5ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/_templates/.placeholder0000664000175000017500000000000000000000000025745 0ustar00zuulzuul00000000000000././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/conf.py0000664000175000017500000002215200000000000022640 0ustar00zuulzuul00000000000000# -*- coding: utf-8 -*- # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. # Masakarimonitors Release Notes documentation build configuration file, # created by sphinx-quickstart on Tue Oct 25 17:40:50 2016. # # This file is execfile()d with the current directory set to its # containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # sys.path.insert(0, os.path.abspath('.')) # -- General configuration ------------------------------------------------ # If your documentation needs a minimal Sphinx version, state it here. # needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ 'openstackdocstheme', 'reno.sphinxext', ] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' # The encoding of source files. # source_encoding = 'utf-8-sig' # The master toctree document. master_doc = 'index' # General information about the project. project = 'masakarimonitors Release Notes' copyright = '2016, OpenStack Foundation' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. # The full version, including alpha/beta/rc tags. release = '' # The short X.Y version. version = '' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: # today = '' # Else, today_fmt is used as the format for a strftime call. # today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = [] # The reST default role (used for this markup: `text`) to use for all # documents. # default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. # add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). # add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. # show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'native' # A list of ignored prefixes for module index sorting. # modindex_common_prefix = [] # If true, keep warnings as "system message" paragraphs in the built documents. # keep_warnings = False # -- Options for HTML output ---------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'openstackdocs' openstackdocs_repo_name = 'openstack/masakari-monitors' openstackdocs_bug_project = 'masakari-monitors' openstackdocs_auto_name = False # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. # html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. # html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". # html_title = None # A shorter title for the navigation bar. Default is the same as html_title. # html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. # html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. # html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied # directly to the root of the documentation. # html_extra_path = [] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. # html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. # html_use_smartypants = True # Custom sidebar templates, maps document names to template names. # html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. # html_additional_pages = {} # If false, no module index is generated. # html_domain_indices = True # If false, no index is generated. # html_use_index = True # If true, the index is split into individual pages for each letter. # html_split_index = False # If true, links to the reST sources are added to the pages. # html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. # html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. # html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. # html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). # html_file_suffix = None # Output file base name for HTML help builder. htmlhelp_basename = 'MasakarimonitorsReleaseNotesdoc' # -- Options for LaTeX output --------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). # 'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). # 'pointsize': '10pt', # Additional stuff for the LaTeX preamble. # 'preamble': '', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ ('index', 'MasakarimonitorsReleaseNotes.tex', 'Masakarimonitors Release' 'Notes Documentation', 'Masakarimonitors Developers', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. # latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. # latex_use_parts = False # If true, show page references after internal links. # latex_show_pagerefs = False # If true, show URL addresses after external links. # latex_show_urls = False # Documents to append as an appendix to all manuals. # latex_appendices = [] # If false, no module index is generated. # latex_domain_indices = True # -- Options for manual page output --------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ('index', 'Masakarimonitorsreleasenotes', 'Masakarimonitors Release Notes' 'Documentation', ['Masakarimonitors Developers'], 1) ] # If true, show URL addresses after external links. # man_show_urls = False # -- Options for Texinfo output ------------------------------------------- # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ ('index', 'MasakarimonitorsReleaseNotes', 'Masakarimonitors Release Notes' 'Documentation', 'Masakarimonitors Developers', 'MasakarimonitorsReleaseNotes', 'One line description of project.', 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. # texinfo_appendices = [] # If false, no module index is generated. # texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. # texinfo_show_urls = 'footnote' # If true, do not generate a @detailmenu in the "Top" node's menu. # texinfo_no_detailmenu = False # -- Options for Internationalization output ------------------------------ locale_dirs = ['locale/'] ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/index.rst0000664000175000017500000000051000000000000023174 0ustar00zuulzuul00000000000000Welcome to Masakarimonitor Release Notes documentation! ======================================================== Contents ======== .. toctree:: :maxdepth: 1 unreleased 2024.2 2024.1 2023.2 2023.1 zed yoga xena wallaby victoria ussuri train stein rocky queens pike ocata ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/ocata.rst0000664000175000017500000000021100000000000023152 0ustar00zuulzuul00000000000000=========================== Ocata Series Release Notes =========================== .. release-notes:: :branch: origin/stable/ocata ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/pike.rst0000664000175000017500000000020500000000000023016 0ustar00zuulzuul00000000000000========================== Pike Series Release Notes ========================== .. release-notes:: :branch: origin/stable/pike ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/queens.rst0000664000175000017500000000021500000000000023367 0ustar00zuulzuul00000000000000============================ Queens Series Release Notes ============================ .. release-notes:: :branch: origin/stable/queens ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/rocky.rst0000664000175000017500000000022100000000000023213 0ustar00zuulzuul00000000000000=================================== Rocky Series Release Notes =================================== .. release-notes:: :branch: stable/rocky ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/stein.rst0000664000175000017500000000022100000000000023206 0ustar00zuulzuul00000000000000=================================== Stein Series Release Notes =================================== .. release-notes:: :branch: stable/stein ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/train.rst0000664000175000017500000000017600000000000023212 0ustar00zuulzuul00000000000000========================== Train Series Release Notes ========================== .. release-notes:: :branch: stable/train ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/unreleased.rst0000664000175000017500000000016000000000000024215 0ustar00zuulzuul00000000000000============================== Current Series Release Notes ============================== .. release-notes:: ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/ussuri.rst0000664000175000017500000000020200000000000023415 0ustar00zuulzuul00000000000000=========================== Ussuri Series Release Notes =========================== .. release-notes:: :branch: stable/ussuri ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/victoria.rst0000664000175000017500000000022000000000000023703 0ustar00zuulzuul00000000000000============================= Victoria Series Release Notes ============================= .. release-notes:: :branch: unmaintained/victoria ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/wallaby.rst0000664000175000017500000000021400000000000023521 0ustar00zuulzuul00000000000000============================ Wallaby Series Release Notes ============================ .. release-notes:: :branch: unmaintained/wallaby ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/xena.rst0000664000175000017500000000020000000000000023014 0ustar00zuulzuul00000000000000========================= Xena Series Release Notes ========================= .. release-notes:: :branch: unmaintained/xena ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/yoga.rst0000664000175000017500000000017200000000000023030 0ustar00zuulzuul00000000000000========================= Yoga Series Release Notes ========================= .. release-notes:: :branch: stable/yoga ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/releasenotes/source/zed.rst0000664000175000017500000000016600000000000022656 0ustar00zuulzuul00000000000000======================== Zed Series Release Notes ======================== .. release-notes:: :branch: stable/zed ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/requirements.txt0000664000175000017500000000230100000000000020626 0ustar00zuulzuul00000000000000# The order of packages is significant, because pip processes them in the order # of appearance. Changing the order has an impact on the overall integration # process, which may cause wedges in the gate later. automaton>=1.9.0 # Apache-2.0 keystoneauth1>=3.4.0 # Apache-2.0 openstacksdk>=0.13.0 # Apache-2.0 oslo.concurrency>=3.26.0 # Apache-2.0 oslo.config>=5.2.0 # Apache-2.0 lxml>=4.5.0 # BSD oslo.cache>=1.26.0 # Apache-2.0 oslo.i18n>=3.15.3 # Apache-2.0 oslo.log>=3.36.0 # Apache-2.0 oslo.middleware>=3.31.0 # Apache-2.0 oslo.privsep>=1.23.0 # Apache-2.0 oslo.service!=1.28.1,>=1.24.0 # Apache-2.0 oslo.utils>=3.33.0 # Apache-2.0 pbr!=2.1.0,>=2.0.0 # Apache-2.0 python-consul >=1.1.0 # MIT # Due to the nature of libvirt-python package, in DevStack we use the one # provided in the distro alongside libvirtd - to ensure the two are compatible, # and also to avoid the pip error when it tries to uninstall the distro version # (installed in such a way for Nova in DevStack). # Do note libvirt-python is used only for instance-oriented monitors, so, e.g., # it is not used by any host monitor. # # Note for package maintainers: just use the libvirt-python package version # as seen in the Nova dependencies. ././@PaxHeader0000000000000000000000000000003400000000000011452 xustar000000000000000028 mtime=1743591461.2052033 masakari_monitors-19.0.0/setup.cfg0000664000175000017500000000363000000000000017171 0ustar00zuulzuul00000000000000[metadata] name = masakari-monitors summary = Monitors for Masakari description_file = README.rst author = OpenStack author_email = openstack-dev@lists.openstack.org home_page = https://docs.openstack.org/masakari-monitors/latest/ python_requires = >=3.8 classifier = Environment :: OpenStack Intended Audience :: Information Technology Intended Audience :: System Administrators License :: OSI Approved :: Apache Software License Operating System :: POSIX :: Linux Programming Language :: Python Programming Language :: Python :: Implementation :: CPython Programming Language :: Python :: 3 :: Only Programming Language :: Python :: 3 Programming Language :: Python :: 3.8 Programming Language :: Python :: 3.9 Programming Language :: Python :: 3.10 Programming Language :: Python :: 3.11 [files] packages = masakarimonitors [entry_points] oslo.config.opts = masakarimonitors.conf = masakarimonitors.conf.opts:list_opts oslo.config.opts.defaults = masakarimonitors.introspectiveinstancemonitor = masakarimonitors.common.config:set_middleware_defaults masakarimonitors.instancemonitor = masakarimonitors.common.config:set_middleware_defaults masakarimonitors.processmonitor = masakarimonitors.common.config:set_middleware_defaults masakarimonitors.hostmonitor = masakarimonitors.common.config:set_middleware_defaults console_scripts = masakari-introspectiveinstancemonitor = masakarimonitors.cmd.introspectiveinstancemonitor:main masakari-instancemonitor = masakarimonitors.cmd.instancemonitor:main masakari-processmonitor = masakarimonitors.cmd.processmonitor:main masakari-hostmonitor = masakarimonitors.cmd.hostmonitor:main hostmonitor.driver = simple = masakarimonitors.hostmonitor.host_handler.handle_host:HandleHost default = masakarimonitors.hostmonitor.host_handler.handle_host:HandleHost consul = masakarimonitors.hostmonitor.consul_check.manager:ConsulCheck [egg_info] tag_build = tag_date = 0 ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/setup.py0000664000175000017500000000127100000000000017061 0ustar00zuulzuul00000000000000# Copyright (c) 2013 Hewlett-Packard Development Company, L.P. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import setuptools setuptools.setup( setup_requires=['pbr>=2.0.0'], pbr=True) ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/test-requirements.txt0000664000175000017500000000213500000000000021610 0ustar00zuulzuul00000000000000# The order of packages is significant, because pip processes them in the order # of appearance. Changing the order has an impact on the overall integration # process, which may cause wedges in the gate later. stestr>=1.0.0 # Apache-2.0 hacking>=3.0.1,<3.1.0 # Apache-2.0 coverage!=4.4,>=4.0 # Apache-2.0 python-subunit>=1.0.0 # Apache-2.0/BSD oslotest>=3.2.0 # Apache-2.0 os-testr>=1.0.0 # Apache-2.0 testrepository>=0.0.18 # Apache-2.0/BSD testscenarios>=0.4 # Apache-2.0/BSD testtools>=2.2.0 # MIT ddt>=1.0.1 # MIT # Due to the nature of libvirt-python package, in DevStack we use the one # provided in the distro alongside libvirtd - to ensure the two are compatible, # and also to avoid the pip error when it tries to uninstall the distro version # (installed in such a way for Nova in DevStack). # Do note libvirt-python is used only for instance-oriented monitors, so, e.g., # it is not used by any host monitor. # TODO(yoctozepto): Refactor code to not require libvirt-python for unit tests, # basically following how it is handled in nova-compute and ceilometer-compute. libvirt-python>=6.0.0 # LGPLv2+ ././@PaxHeader0000000000000000000000000000002600000000000011453 xustar000000000000000022 mtime=1743591410.0 masakari_monitors-19.0.0/tox.ini0000664000175000017500000000741200000000000016665 0ustar00zuulzuul00000000000000[tox] minversion = 3.1.1 envlist = pep8,py3 ignore_basepython_conflict = True [testenv] basepython = python3 usedevelop = True setenv = VIRTUAL_ENV={envdir} LANGUAGE=en_US LC_ALL=en_US.utf-8 deps = -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt commands = stestr run {posargs} passenv = HTTP_PROXY, HTTPS_PROXY, NO_PROXY, OS_DEBUG, GENERATE_HASHES [testenv:genconfig] commands = oslo-config-generator --config-file=etc/masakarimonitors/masakarimonitors-config-generator.conf [testenv:linters] skip_install = True deps = -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} {[testenv:pep8]deps} {[testenv:doc8]deps} {[testenv:yamllint]deps} commands = {[testenv:pep8]commands} {[testenv:doc8]commands} {[testenv:yamllint]commands} [testenv:pep8] skip_install = True deps = -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} # NOTE(yoctozepto): this is to pin hacking to the tested version -c{toxinidir}/test-requirements.txt hacking commands = flake8 {posargs} [testenv:doc8] skip_install = True deps = -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} doc8 commands = doc8 README.rst CONTRIBUTING.rst HACKING.rst doc/source doc8 releasenotes/source doc8 -e '.yaml' releasenotes/notes [testenv:yamllint] skip_install = True deps = -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} yamllint commands = yamllint -s . [testenv:venv] commands = {posargs} [testenv:cover] setenv = VIRTUAL_ENV={envdir} PYTHON=coverage run --source masakarimonitors --parallel-mode commands = stestr run {posargs} coverage combine coverage html -d cover coverage xml -o cover/coverage.xml [testenv:docs] deps = -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} -r{toxinidir}/requirements.txt -r{toxinidir}/doc/requirements.txt commands = sphinx-build -W -b html doc/source doc/build/html [testenv:pdf-docs] deps = {[testenv:docs]deps} allowlist_externals = make commands = sphinx-build -W -b latex doc/source doc/build/pdf make -C doc/build/pdf [testenv:releasenotes] skip_install = True deps = -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} -r{toxinidir}/doc/requirements.txt allowlist_externals = rm commands = rm -fr releasenotes/build sphinx-build -a -E -W -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html [testenv:debug] commands = oslo_debug_helper {posargs} [flake8] # E123 - closing bracket does not match indentation of opening bracket's line # E125 - continuation line with same indent as next logical line # E128 - continuation line under-indented for visual indent # E265 - block comment should start with '# ' # H405 - multi line docstring summary not separated with an empty line # W503 - line break before binary operator # W504 - line break after binary operator show-source = True ignore = E123,E125,E128,E265,H405,W503,W504 builtins = _ exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,build [hacking] import_exceptions = masakarimonitors.i18n [flake8:local-plugins] extension = M301 = checks:check_explicit_underscore_import M302 = checks:no_translate_logs M303 = checks:yield_followed_by_space M304 = checks:assert_raisesRegexp M305 = checks:no_log_warn paths = ./masakarimonitors/hacking [testenv:bindep] skip_install = True deps = bindep commands = bindep test [doc8] # NOTE(yoctozepto): this is due to multiple violations - it is better to keep # it limited sanely rather than disable the D001 "Line too long" rule altogether max-line-length = 105