barman-3.13.3/0000755000175100001660000000000015002177454011227 5ustar 00000000000000barman-3.13.3/MANIFEST.in0000644000175100001660000000035615002177425012767 0ustar 00000000000000recursive-include barman *.py recursive-include rpm * recursive-include docs/barman.d * include docs/barman.conf include scripts/barman.bash_completion include AUTHORS RELNOTES.md ChangeLog LICENSE MANIFEST.in setup.py INSTALL README.rst barman-3.13.3/PKG-INFO0000644000175100001660000000306015002177454012323 0ustar 00000000000000Metadata-Version: 2.1 Name: barman Version: 3.13.3 Summary: Backup and Recovery Manager for PostgreSQL Home-page: https://www.pgbarman.org/ Author: EnterpriseDB Author-email: barman@enterprisedb.com License: GPL-3.0 Platform: Linux Platform: Mac OS X Classifier: Environment :: Console Classifier: Development Status :: 5 - Production/Stable Classifier: Topic :: System :: Archiving :: Backup Classifier: Topic :: Database Classifier: Topic :: System :: Recovery Tools Classifier: Intended Audience :: System Administrators Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+) Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3.6 Classifier: Programming Language :: Python :: 3.7 Classifier: Programming Language :: Python :: 3.8 Classifier: Programming Language :: Python :: 3.9 Classifier: Programming Language :: Python :: 3.10 Provides-Extra: argcomplete Provides-Extra: aws-snapshots Provides-Extra: azure Provides-Extra: azure-snapshots Provides-Extra: cloud Provides-Extra: google Provides-Extra: google-snapshots Provides-Extra: snappy Provides-Extra: zstandard Provides-Extra: lz4 License-File: LICENSE License-File: AUTHORS Barman (Backup and Recovery Manager) is an open-source administration tool for disaster recovery of PostgreSQL servers written in Python. It allows your organisation to perform remote backups of multiple servers in business critical environments to reduce risk and help DBAs during the recovery phase. Barman is distributed under GNU GPL 3 and maintained by EnterpriseDB. barman-3.13.3/setup.cfg0000644000175100001660000000043015002177454013045 0ustar 00000000000000[bdist_wheel] universal = 1 [aliases] test = pytest [isort] known_first_party = barman known_third_party = setuptools distutils argcomplete dateutil psycopg2 mock pytest boto3 botocore sphinx sphinx_bootstrap_theme skip = .tox [egg_info] tag_build = tag_date = 0 barman-3.13.3/LICENSE0000644000175100001660000010451515002177425012240 0ustar 00000000000000 GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read . barman-3.13.3/scripts/0000755000175100001660000000000015002177454012716 5ustar 00000000000000barman-3.13.3/scripts/barman.bash_completion0000644000175100001660000000014215002177425017241 0ustar 00000000000000eval "$((register-python-argcomplete3 barman || register-python-argcomplete barman) 2>/dev/null)" barman-3.13.3/barman/0000755000175100001660000000000015002177454012467 5ustar 00000000000000barman-3.13.3/barman/storage/0000755000175100001660000000000015002177454014133 5ustar 00000000000000barman-3.13.3/barman/storage/local_file_manager.py0000644000175100001660000000453615002177425020276 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2013-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import os from .file_manager import FileManager from .file_stats import FileStats class LocalFileManager(FileManager): def file_exist(self, file_path): """ Tests if file exists :param file_path: File path :type file_path: string :return: True if file exists False otherwise :rtype: bool """ return os.path.isfile(file_path) def get_file_stats(self, file_path): """ Tests if file exists :param file_path: File path :type file_path: string :return: :rtype: FileStats """ if not self.file_exist(file_path): raise IOError("Missing file " + file_path) sts = os.stat(file_path) return FileStats(sts.st_size, sts.st_mtime) def get_file_list(self, path): """ List all files within a path, including subdirectories :param path: Path to analyze :type path: string :return: List of file path :rtype: list """ if not os.path.isdir(path): raise NotADirectoryError(path) file_list = [] for root, dirs, files in os.walk(path): file_list.extend( list(map(lambda x, prefix=root: os.path.join(prefix, x), files)) ) return file_list def get_file_content(self, file_path, file_mode="rb"): with open(file_path, file_mode) as reader: content = reader.read() return content def save_content_to_file(self, file_path, content, file_mode="wb"): """ """ with open(file_path, file_mode) as writer: writer.write(content) barman-3.13.3/barman/storage/file_stats.py0000644000175100001660000000321715002177425016643 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2013-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . from datetime import datetime try: from datetime import timezone utc = timezone.utc except ImportError: # python 2.7 compatibility from dateutil import tz utc = tz.tzutc() class FileStats: def __init__(self, size, last_modified): """ Arbitrary timezone set to UTC. There is probably possible improvement here. :param size: file size in bytes :type size: int :param last_modified: Time of last modification in seconds :type last_modified: int """ self.size = size self.last_modified = datetime.fromtimestamp(last_modified, tz=utc) def get_size(self): """ """ return self.size def get_last_modified(self, datetime_format="%Y-%m-%d %H:%M:%S"): """ :param datetime_format: Format to apply on datetime object :type datetime_format: str """ return self.last_modified.strftime(datetime_format) barman-3.13.3/barman/storage/file_manager.py0000644000175100001660000000341015002177425017112 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2013-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . from abc import ABCMeta, abstractmethod from barman.utils import with_metaclass class FileManager(with_metaclass(ABCMeta)): @abstractmethod def file_exist(self, file_path): """ Tests if file exists :param file_path: File path :type file_path: string :return: True if file exists False otherwise :rtype: bool """ @abstractmethod def get_file_stats(self, file_path): """ Tests if file exists :param file_path: File path :type file_path: string :return: :rtype: FileStats """ @abstractmethod def get_file_list(self, path): """ List all files within a path, including subdirectories :param path: Path to analyze :type path: string :return: List of file path :rtype: list """ @abstractmethod def get_file_content(self, file_path, file_mode="rb"): """ """ @abstractmethod def save_content_to_file(self, file_path, content, file_mode="wb"): """ """ barman-3.13.3/barman/storage/__init__.py0000644000175100001660000000132415002177425016242 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2013-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . barman-3.13.3/barman/clients/0000755000175100001660000000000015002177454014130 5ustar 00000000000000barman-3.13.3/barman/clients/cloud_restore.py0000644000175100001660000003762215002177425017363 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import logging import os from abc import ABCMeta, abstractmethod from contextlib import closing from barman.clients.cloud_cli import ( CLIErrorExit, GeneralErrorExit, NetworkErrorExit, OperationErrorExit, create_argument_parser, ) from barman.cloud import CloudBackupCatalog, configure_logging from barman.cloud_providers import ( get_cloud_interface, get_snapshot_interface_from_backup_info, ) from barman.exceptions import ConfigurationException from barman.fs import UnixLocalCommand from barman.recovery_executor import SnapshotRecoveryExecutor from barman.utils import ( check_tli, force_str, get_backup_id_from_target_lsn, get_backup_id_from_target_time, get_backup_id_from_target_tli, get_last_backup_id, parse_target_tli, with_metaclass, ) def _validate_config(config, backup_info): """ Additional validation for config such as mutually inclusive options. Raises a ConfigurationException if any options are missing or incompatible. :param argparse.Namespace config: The backup options provided at the command line. :param BackupInfo backup_info: The backup info for the backup to restore """ if backup_info.snapshots_info: if config.tablespace != []: raise ConfigurationException( "Backup %s is a snapshot backup therefore tablespace relocation rules " "cannot be used." % backup_info.backup_id, ) def main(args=None): """ The main script entry point :param list[str] args: the raw arguments list. When not provided it defaults to sys.args[1:] """ config = parse_arguments(args) configure_logging(config) try: cloud_interface = get_cloud_interface(config) with closing(cloud_interface): if not cloud_interface.test_connectivity(): raise NetworkErrorExit() # If test is requested, just exit after connectivity test elif config.test: raise SystemExit(0) if not cloud_interface.bucket_exists: logging.error("Bucket %s does not exist", cloud_interface.bucket_name) raise OperationErrorExit() catalog = CloudBackupCatalog(cloud_interface, config.server_name) backup_id = None if config.backup_id != "auto": backup_id = catalog.parse_backup_id(config.backup_id) else: target_options = ["target_time", "target_lsn"] target_option = None for option in target_options: target = getattr(config, option, None) if target is not None: target_option = option break # "Parse" the string value to integer for `target_tli` if passed as a # string ("current", "latest") target_tli = parse_target_tli(obj=catalog, target_tli=config.target_tli) available_backups = catalog.get_backup_list().values() if target_option is None: if target_tli is not None: backup_id = get_backup_id_from_target_tli( available_backups, target_tli ) else: backup_id = get_last_backup_id(available_backups) elif target_option == "target_time": backup_id = get_backup_id_from_target_time( available_backups, target, target_tli ) elif target_option == "target_lsn": backup_id = get_backup_id_from_target_lsn( available_backups, target, target_tli ) # If no candidate backup_id is found, error out. if backup_id is None: logging.error("Cannot find any candidate backup for recovery.") raise OperationErrorExit() backup_info = catalog.get_backup_info(backup_id) logging.info("Restoring from backup_id: %s" % backup_id) if not backup_info: logging.error( "Backup %s for server %s does not exists", backup_id, config.server_name, ) raise OperationErrorExit() _validate_config(config, backup_info) if backup_info.snapshots_info: snapshot_interface = get_snapshot_interface_from_backup_info( backup_info, config ) snapshot_interface.validate_restore_config(config) downloader = CloudBackupDownloaderSnapshot( cloud_interface, catalog, snapshot_interface ) downloader.download_backup( backup_info, config.recovery_dir, config.snapshot_recovery_instance, ) else: downloader = CloudBackupDownloaderObjectStore(cloud_interface, catalog) downloader.download_backup( backup_info, config.recovery_dir, tablespace_map(config.tablespace), ) except KeyboardInterrupt as exc: logging.error("Barman cloud restore was interrupted by the user") logging.debug("Exception details:", exc_info=exc) raise OperationErrorExit() except Exception as exc: logging.error("Barman cloud restore exception: %s", force_str(exc)) logging.debug("Exception details:", exc_info=exc) raise GeneralErrorExit() def parse_arguments(args=None): """ Parse command line arguments :return: The options parsed """ parser, s3_arguments, azure_arguments = create_argument_parser( description="This script can be used to download a backup " "previously made with barman-cloud-backup command." "Currently AWS S3, Azure Blob Storage and Google Cloud Storage are supported.", ) parser.add_argument("backup_id", help="the backup ID") parser.add_argument("recovery_dir", help="the path to a directory for recovery.") parser.add_argument( "--tablespace", help="tablespace relocation rule", metavar="NAME:LOCATION", action="append", default=[], ) parser.add_argument( "--snapshot-recovery-instance", help="Instance where the disks recovered from the snapshots are attached", ) parser.add_argument( "--snapshot-recovery-zone", help=( "Zone containing the instance and disks for the snapshot recovery " "(deprecated: replaced by --gcp-zone)" ), dest="gcp_zone", ) s3_arguments.add_argument( "--aws-region", help=( "Name of the AWS region where the instance and disks for snapshot " "recovery are located" ), ) gcs_arguments = parser.add_argument_group( "Extra options for google-cloud-storage cloud provider" ) gcs_arguments.add_argument( "--gcp-zone", help="Zone containing the instance and disks for the snapshot recovery", ) azure_arguments.add_argument( "--azure-resource-group", help="Resource group containing the instance and disks for the snapshot recovery", ) parser.add_argument("--target-tli", help="target timeline", type=check_tli) target_args = parser.add_mutually_exclusive_group() target_args.add_argument("--target-lsn", help="target LSN (Log Sequence Number)") target_args.add_argument( "--target-time", help="target time. You can use any valid unambiguous representation. " 'e.g: "YYYY-MM-DD HH:MM:SS.mmm"', ) return parser.parse_args(args=args) def tablespace_map(rules): """ Return a mapping from tablespace names to locations built from any `--tablespace name:/loc/ation` rules specified. """ tablespaces = {} for rule in rules: try: tablespaces.update([rule.split(":", 1)]) except ValueError: logging.error( "Invalid tablespace relocation rule '%s'\n" "HINT: The valid syntax for a relocation rule is " "NAME:LOCATION", rule, ) raise CLIErrorExit() return tablespaces class CloudBackupDownloader(with_metaclass(ABCMeta)): """ Restore a backup from cloud storage. """ def __init__(self, cloud_interface, catalog): """ Object responsible for handling interactions with cloud storage :param CloudInterface cloud_interface: The interface to use to upload the backup :param str server_name: The name of the server as configured in Barman :param CloudBackupCatalog catalog: The cloud backup catalog """ self.cloud_interface = cloud_interface self.catalog = catalog @abstractmethod def download_backup(self, backup_id, destination_dir): """ Download a backup from cloud storage :param str backup_id: The backup id to restore :param str destination_dir: Path to the destination directory """ class CloudBackupDownloaderObjectStore(CloudBackupDownloader): """ Cloud storage download client for an object store backup """ def download_backup(self, backup_info, destination_dir, tablespaces): """ Download a backup from cloud storage :param BackupInfo backup_info: The backup info for the backup to restore :param str destination_dir: Path to the destination directory """ # Validate the destination directory before starting recovery if os.path.exists(destination_dir) and os.listdir(destination_dir): logging.error( "Destination %s already exists and it is not empty", destination_dir ) raise OperationErrorExit() backup_files = self.catalog.get_backup_files(backup_info) # We must download and restore a bunch of .tar files that contain PGDATA # and each tablespace. First, we determine a target directory to extract # each tar file into and record these in copy_jobs. For each tablespace, # the location may be overridden by `--tablespace name:/new/location` on # the command-line; and we must also add an entry to link_jobs to create # a symlink from $PGDATA/pg_tblspc/oid to the correct location after the # downloads. copy_jobs = [] link_jobs = [] for oid in backup_files: file_info = backup_files[oid] # PGDATA is restored where requested (destination_dir) if oid is None: target_dir = destination_dir else: for tblspc in backup_info.tablespaces: if oid == tblspc.oid: target_dir = tblspc.location if tblspc.name in tablespaces: target_dir = os.path.realpath(tablespaces[tblspc.name]) logging.debug( "Tablespace %s (oid=%s) will be located at %s", tblspc.name, oid, target_dir, ) link_jobs.append( ["%s/pg_tblspc/%s" % (destination_dir, oid), target_dir] ) break else: raise AssertionError( "The backup file oid '%s' must be present " "in backupinfo.tablespaces list" ) # Validate the destination directory before starting recovery if os.path.exists(target_dir) and os.listdir(target_dir): logging.error( "Destination %s already exists and it is not empty", target_dir ) raise OperationErrorExit() copy_jobs.append([file_info, target_dir]) for additional_file in file_info.additional_files: copy_jobs.append([additional_file, target_dir]) # Now it's time to download the files for file_info, target_dir in copy_jobs: # Download the file logging.debug( "Extracting %s to %s (%s)", file_info.path, target_dir, ( "decompressing " + file_info.compression if file_info.compression else "no compression" ), ) self.cloud_interface.extract_tar(file_info.path, target_dir) for link, target in link_jobs: os.symlink(target, link) # If we did not restore the pg_wal directory from one of the uploaded # backup files, we must recreate it here. (If pg_wal was originally a # symlink, it would not have been uploaded.) wal_path = os.path.join(destination_dir, backup_info.wal_directory()) if not os.path.exists(wal_path): os.mkdir(wal_path) class CloudBackupDownloaderSnapshot(CloudBackupDownloader): """A minimal downloader for cloud backups which just retrieves the backup label.""" def __init__(self, cloud_interface, catalog, snapshot_interface): """ Object responsible for handling interactions with cloud storage :param CloudInterface cloud_interface: The interface to use to upload the backup :param str server_name: The name of the server as configured in Barman :param CloudBackupCatalog catalog: The cloud backup catalog :param CloudSnapshotInterface snapshot_interface: Interface for managing snapshots via a cloud provider API. """ super(CloudBackupDownloaderSnapshot, self).__init__(cloud_interface, catalog) self.snapshot_interface = snapshot_interface def download_backup( self, backup_info, destination_dir, recovery_instance, ): """ Download a backup from cloud storage :param BackupInfo backup_info: The backup info for the backup to restore :param str destination_dir: Path to the destination directory :param str recovery_instance: The name of the VM instance to which the disks cloned from the backup snapshots are attached. """ attached_volumes = SnapshotRecoveryExecutor.get_attached_volumes_for_backup( self.snapshot_interface, backup_info, recovery_instance, ) cmd = UnixLocalCommand() SnapshotRecoveryExecutor.check_mount_points(backup_info, attached_volumes, cmd) SnapshotRecoveryExecutor.check_recovery_dir_exists(destination_dir, cmd) # If the target directory does not exist then we will fail here because # it tells us the snapshot has not been restored. return self.cloud_interface.download_file( "/".join((self.catalog.prefix, backup_info.backup_id, "backup_label")), os.path.join(destination_dir, "backup_label"), decompress=None, ) if __name__ == "__main__": main() barman-3.13.3/barman/clients/walarchive.py0000755000175100001660000002764315002177425016644 0ustar 00000000000000# -*- coding: utf-8 -*- # walarchive - Remote Barman WAL archive command for PostgreSQL # # This script remotely sends WAL files to Barman via SSH, on demand. # It is intended to be used as archive_command in PostgreSQL configuration. # # See the help page for usage information. # # © Copyright EnterpriseDB UK Limited 2019-2025 # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . from __future__ import print_function import argparse import copy import hashlib import os import subprocess import sys import tarfile import time from contextlib import closing from io import BytesIO import barman DEFAULT_USER = "barman" BUFSIZE = 16 * 1024 def main(args=None): """ The main script entry point :param list[str] args: the raw arguments list. When not provided it defaults to sys.args[1:] """ config = parse_arguments(args) # Do connectivity test if requested if config.test: connectivity_test(config) return # never reached # Check WAL destination is not a directory if os.path.isdir(config.wal_path): exit_with_error("WAL_PATH cannot be a directory: %s" % config.wal_path) try: # Execute barman put-wal through the ssh connection ssh_process = RemotePutWal(config, config.wal_path) except EnvironmentError as exc: exit_with_error("Error executing ssh: %s" % exc) return # never reached # Wait for termination of every subprocess. If CTRL+C is pressed, # terminate all of them RemotePutWal.wait_for_all() # If the command succeeded exit here if ssh_process.returncode == 0: return # Report the exit code, remapping ssh failure code (255) to 3 if ssh_process.returncode == 255: exit_with_error("Connection problem with ssh", 3) else: exit_with_error( "Remote 'barman put-wal' command has failed!", ssh_process.returncode ) def build_ssh_command(config): """ Prepare an ssh command according to the arguments passed on command line :param argparse.Namespace config: the configuration from command line :return list[str]: the ssh command as list of string """ ssh_command = ["ssh"] if config.port is not None: ssh_command += ["-p", config.port] ssh_command += [ "-q", # quiet mode - suppress warnings "-T", # disable pseudo-terminal allocation "%s@%s" % (config.user, config.barman_host), "barman", ] if config.config: ssh_command.append("--config='%s'" % config.config) ssh_command.extend(["put-wal", config.server_name]) if config.test: ssh_command.append("--test") return ssh_command def exit_with_error(message, status=2): """ Print ``message`` and terminate the script with ``status`` :param str message: message to print :param int status: script exit code """ print("ERROR: %s" % message, file=sys.stderr) sys.exit(status) def connectivity_test(config): """ Invoke remote put-wal --test to test the connection with Barman server :param argparse.Namespace config: the configuration from command line """ ssh_command = build_ssh_command(config) try: pipe = subprocess.Popen( ssh_command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT ) output = pipe.communicate() print(output[0].decode("utf-8")) sys.exit(pipe.returncode) except subprocess.CalledProcessError as e: exit_with_error("Impossible to invoke remote put-wal: %s" % e) def parse_arguments(args=None): """ Parse the command line arguments :param list[str] args: the raw arguments list. When not provided it defaults to sys.args[1:] :rtype: argparse.Namespace """ parser = argparse.ArgumentParser( description="This script will be used as an 'archive_command' " "based on the put-wal feature of Barman. " "A ssh connection will be opened to the Barman host.", ) parser.add_argument( "-V", "--version", action="version", version="%%(prog)s %s" % barman.__version__ ) parser.add_argument( "-U", "--user", default=DEFAULT_USER, help="The user used for the ssh connection to the Barman server. " "Defaults to '%(default)s'.", ) parser.add_argument( "--port", help="The port used for the ssh connection to the Barman server.", ) parser.add_argument( "-c", "--config", metavar="CONFIG", help="configuration file on the Barman server", ) parser.add_argument( "-t", "--test", action="store_true", help="test both the connection and the configuration of the " "requested PostgreSQL server in Barman for WAL retrieval. " "With this option, the 'wal_name' mandatory argument is " "ignored.", ) parser.add_argument( "--md5", action="store_true", help="Use MD5 as the hash algorithm to maintain compatibility between " "mismatched client and server versions.", ) parser.add_argument( "barman_host", metavar="BARMAN_HOST", help="The host of the Barman server.", ) parser.add_argument( "server_name", metavar="SERVER_NAME", help="The server name configured in Barman from which WALs are taken.", ) parser.add_argument( "wal_path", metavar="WAL_PATH", help="The value of the '%%p' keyword (according to 'archive_command').", ) return parser.parse_args(args=args) def hashCopyfileobj(src, dst, length=None, hash_algorithm="sha256"): """ Copy length bytes from fileobj src to fileobj dst. If length is None, copy the entire content. This method is used by the ChecksumTarFile.addfile(). Returns the checksum for the specified hashing algorithm. """ checksum = hashlib.new(hash_algorithm) if length == 0: return checksum.hexdigest() if length is None: while 1: buf = src.read(BUFSIZE) if not buf: break checksum.update(buf) dst.write(buf) return checksum.hexdigest() blocks, remainder = divmod(length, BUFSIZE) for _ in range(blocks): buf = src.read(BUFSIZE) if len(buf) < BUFSIZE: raise IOError("end of file reached") checksum.update(buf) dst.write(buf) if remainder != 0: buf = src.read(remainder) if len(buf) < remainder: raise IOError("end of file reached") checksum.update(buf) dst.write(buf) return checksum.hexdigest() class ChecksumTarInfo(tarfile.TarInfo): """ Special TarInfo that can hold a file checksum """ def __init__(self, *args, **kwargs): super(ChecksumTarInfo, self).__init__(*args, **kwargs) self.data_checksum = None class ChecksumTarFile(tarfile.TarFile): """ Custom TarFile class that automatically calculates hash checksum of each file and appends a file called 'MD5SUMS' or 'SHA256SUMS' to the stream, depending on the hash algorithm specified. """ def __init__(self, *args, **kwargs): super(ChecksumTarFile, self).__init__(*args, **kwargs) self.hash_algorithm = "sha256" self.HASHSUMS_FILE = "SHA256SUMS" tarinfo = ChecksumTarInfo # The default TarInfo class used by TarFile format = tarfile.PAX_FORMAT # Use PAX format to better preserve metadata def addfile(self, tarinfo, fileobj=None): """ Add the provided fileobj to the tar using hashCopyfileobj and saves the file hash in the provided ChecksumTarInfo object. This method completely replaces TarFile.addfile() """ self._check("aw") tarinfo = copy.copy(tarinfo) buf = tarinfo.tobuf(self.format, self.encoding, self.errors) self.fileobj.write(buf) self.offset += len(buf) # If there's data to follow, append it. if fileobj is not None: tarinfo.data_checksum = hashCopyfileobj( fileobj, self.fileobj, tarinfo.size, self.hash_algorithm ) blocks, remainder = divmod(tarinfo.size, tarfile.BLOCKSIZE) if remainder > 0: self.fileobj.write(tarfile.NUL * (tarfile.BLOCKSIZE - remainder)) blocks += 1 self.offset += blocks * tarfile.BLOCKSIZE self.members.append(tarinfo) def close(self): """ Add a :attr:`HASHSUMS_FILE` file to the tar just before closing. This method extends TarFile.close(). """ if self.closed: return if self.mode in "aw": with BytesIO() as hashsums: for tarinfo in self.members: line = "%s *%s\n" % (tarinfo.data_checksum, tarinfo.name) hashsums.write(line.encode()) hashsums.seek(0, os.SEEK_END) size = hashsums.tell() hashsums.seek(0, os.SEEK_SET) tarinfo = self.tarinfo(self.HASHSUMS_FILE) tarinfo.size = size self.addfile(tarinfo, hashsums) super(ChecksumTarFile, self).close() class RemotePutWal(object): """ Spawn a process that sends a WAL to a remote Barman server. :param argparse.Namespace config: the configuration from command line :param wal_path: The name of WAL to upload """ processes = set() """ The list of processes that has been spawned by RemotePutWal """ def __init__(self, config, wal_path): self.config = config self.wal_path = wal_path self.dest_file = None # Spawn a remote put-wal process self.ssh_process = subprocess.Popen( build_ssh_command(config), stdin=subprocess.PIPE ) # Register the spawned processes in the class registry self.processes.add(self.ssh_process) # Check if md5 flag was used. hash_settings = {True: ("md5", "MD5SUMS"), False: ("sha256", "SHA256SUMS")} hash_algorithm, HASHSUMS_FILE = hash_settings[config.md5] # Send the data as a tar file (containing checksums) with self.ssh_process.stdin as dest_file: with closing(ChecksumTarFile.open(mode="w|", fileobj=dest_file)) as tar: tar.hash_algorithm = hash_algorithm tar.HASHSUMS_FILE = HASHSUMS_FILE tar.add(wal_path, os.path.basename(wal_path)) @classmethod def wait_for_all(cls): """ Wait for the termination of all the registered spawned processes. """ try: while cls.processes: time.sleep(0.1) for process in cls.processes.copy(): if process.poll() is not None: cls.processes.remove(process) except KeyboardInterrupt: # If a SIGINT has been received, make sure that every subprocess # terminate for process in cls.processes: process.kill() exit_with_error("SIGINT received! Terminating.") @property def returncode(self): """ Return the exit code of the RemoteGetWal processes. :return: exit code of the RemoteGetWal processes """ if self.ssh_process.returncode != 0: return self.ssh_process.returncode return 0 if __name__ == "__main__": main() barman-3.13.3/barman/clients/cloud_walarchive.py0000755000175100001660000002776315002177425020035 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import logging import os import os.path from contextlib import closing from barman.clients.cloud_cli import ( CLIErrorExit, GeneralErrorExit, NetworkErrorExit, UrlArgumentType, add_tag_argument, create_argument_parser, ) from barman.clients.cloud_compression import compress from barman.cloud import configure_logging from barman.cloud_providers import get_cloud_interface from barman.exceptions import BarmanException from barman.utils import check_positive, check_size, force_str from barman.xlog import hash_dir, is_any_xlog_file, is_history_file def __is_hook_script(): """Check the environment and determine if we are running as a hook script""" if "BARMAN_HOOK" in os.environ and "BARMAN_PHASE" in os.environ: if ( os.getenv("BARMAN_HOOK") in ("archive_script", "archive_retry_script") and os.getenv("BARMAN_PHASE") == "pre" ): return True else: raise BarmanException( "barman-cloud-wal-archive called as unsupported hook script: %s_%s" % (os.getenv("BARMAN_PHASE"), os.getenv("BARMAN_HOOK")) ) else: return False def main(args=None): """ The main script entry point :param list[str] args: the raw arguments list. When not provided it defaults to sys.args[1:] """ config = parse_arguments(args) configure_logging(config) # Read wal_path from environment if we're a hook script if __is_hook_script(): if "BARMAN_FILE" not in os.environ: raise BarmanException("Expected environment variable BARMAN_FILE not set") config.wal_path = os.getenv("BARMAN_FILE") else: if config.wal_path is None: raise BarmanException("the following arguments are required: wal_path") # Validate the WAL file name before uploading it if not is_any_xlog_file(config.wal_path): logging.error("%s is an invalid name for a WAL file" % config.wal_path) raise CLIErrorExit() try: cloud_interface = get_cloud_interface(config) with closing(cloud_interface): uploader = CloudWalUploader( cloud_interface=cloud_interface, server_name=config.server_name, compression=config.compression, ) if not cloud_interface.test_connectivity(): raise NetworkErrorExit() # If test is requested, just exit after connectivity test elif config.test: raise SystemExit(0) # TODO: Should the setup be optional? cloud_interface.setup_bucket() upload_kwargs = {} if is_history_file(config.wal_path): upload_kwargs["override_tags"] = config.history_tags uploader.upload_wal(config.wal_path, **upload_kwargs) except Exception as exc: logging.error("Barman cloud WAL archiver exception: %s", force_str(exc)) logging.debug("Exception details:", exc_info=exc) raise GeneralErrorExit() def parse_arguments(args=None): """ Parse command line arguments :return: The options parsed """ parser, s3_arguments, azure_arguments = create_argument_parser( description="This script can be used in the `archive_command` " "of a PostgreSQL server to ship WAL files to the Cloud. " "Currently AWS S3, Azure Blob Storage and Google Cloud Storage are supported.", source_or_destination=UrlArgumentType.destination, ) parser.add_argument( "wal_path", nargs="?", help="the value of the '%%p' keyword (according to 'archive_command').", default=None, ) compression = parser.add_mutually_exclusive_group() compression.add_argument( "-z", "--gzip", help="gzip-compress the WAL while uploading to the cloud", action="store_const", const="gzip", dest="compression", ) compression.add_argument( "-j", "--bzip2", help="bzip2-compress the WAL while uploading to the cloud", action="store_const", const="bzip2", dest="compression", ) compression.add_argument( "--xz", help="xz-compress the WAL while uploading to the cloud", action="store_const", const="xz", dest="compression", ) compression.add_argument( "--snappy", help="snappy-compress the WAL while uploading to the cloud " "(requires optional python-snappy library)", action="store_const", const="snappy", dest="compression", ) compression.add_argument( "--zstd", help="zstd-compress the WAL while uploading to the cloud " "(requires optional zstandard library)", action="store_const", const="zstd", dest="compression", ) compression.add_argument( "--lz4", help="lz4-compress the WAL while uploading to the cloud " "(requires optional lz4 library)", action="store_const", const="lz4", dest="compression", ) add_tag_argument( parser, name="tags", help="Tags to be added to archived WAL files in cloud storage", ) add_tag_argument( parser, name="history-tags", help="Tags to be added to archived history files in cloud storage", ) gcs_arguments = parser.add_argument_group( "Extra options for google-cloud-storage cloud provider" ) gcs_arguments.add_argument( "--kms-key-name", help="The name of the GCP KMS key which should be used for encrypting the " "uploaded data in GCS.", ) s3_arguments.add_argument( "-e", "--encryption", help="The encryption algorithm used when storing the uploaded data in S3. " "Allowed values: 'AES256'|'aws:kms'.", choices=["AES256", "aws:kms"], metavar="ENCRYPTION", ) s3_arguments.add_argument( "--sse-kms-key-id", help="The AWS KMS key ID that should be used for encrypting the uploaded data " "in S3. Can be specified using the key ID on its own or using the full ARN for " "the key. Only allowed if `-e/--encryption` is set to `aws:kms`.", ) azure_arguments.add_argument( "--encryption-scope", help="The name of an encryption scope defined in the Azure Blob Storage " "service which is to be used to encrypt the data in Azure", ) azure_arguments.add_argument( "--max-block-size", help="The chunk size to be used when uploading an object via the " "concurrent chunk method (default: 4MB).", type=check_size, default="4MB", ) azure_arguments.add_argument( "--max-concurrency", help="The maximum number of chunks to be uploaded concurrently (default: 1).", type=check_positive, default=1, ) azure_arguments.add_argument( "--max-single-put-size", help="Maximum size for which the Azure client will upload an object in a " "single request (default: 64MB). If this is set lower than the PostgreSQL " "WAL segment size after any applied compression then the concurrent chunk " "upload method for WAL archiving will be used.", default="64MB", type=check_size, ) return parser.parse_args(args=args) class CloudWalUploader(object): """ Cloud storage upload client """ def __init__(self, cloud_interface, server_name, compression=None): """ Object responsible for handling interactions with cloud storage :param CloudInterface cloud_interface: The interface to use to upload the backup :param str server_name: The name of the server as configured in Barman :param str compression: Compression algorithm to use """ self.cloud_interface = cloud_interface self.compression = compression self.server_name = server_name def upload_wal(self, wal_path, override_tags=None): """ Upload a WAL file from postgres to cloud storage :param str wal_path: Full path of the WAL file :param List[tuple] override_tags: List of k,v tuples which should override any tags already defined in the cloud interface """ # Extract the WAL file wal_name = self.retrieve_wal_name(wal_path) # Use the correct file object for the upload (simple|gzip|bz2) file_object = self.retrieve_file_obj(wal_path) # Correctly format the destination path destination = os.path.join( self.cloud_interface.path, self.server_name, "wals", hash_dir(wal_path), wal_name, ) # Put the file in the correct bucket. # The put method will handle automatically multipart upload self.cloud_interface.upload_fileobj( fileobj=file_object, key=destination, override_tags=override_tags ) def retrieve_file_obj(self, wal_path): """ Create the correct type of file object necessary for the file transfer. If no compression is required a simple File object is returned. In case of compression, a BytesIO object is returned, containing the result of the compression. NOTE: the Wal files are actually compressed straight into memory, thanks to the usual small dimension of the WAL. This could change in the future because the WAL files dimension could be more than 16MB on some postgres install. TODO: Evaluate using tempfile if the WAL is bigger than 16MB :param str wal_path: :return File: simple or compressed file object """ # Read the wal_file in binary mode wal_file = open(wal_path, "rb") # return the opened file if is uncompressed if not self.compression: return wal_file return compress(wal_file, self.compression) def retrieve_wal_name(self, wal_path): """ Extract the name of the WAL file from the complete path. If no compression is specified, then the simple file name is returned. In case of compression, the correct file extension is applied to the WAL file name. :param str wal_path: the WAL file complete path :return str: WAL file name """ # Extract the WAL name wal_name = os.path.basename(wal_path) # return the plain file name if no compression is specified if not self.compression: return wal_name if self.compression == "gzip": # add gz extension return "%s.gz" % wal_name elif self.compression == "bzip2": # add bz2 extension return "%s.bz2" % wal_name elif self.compression == "xz": # add xz extension return "%s.xz" % wal_name elif self.compression == "snappy": # add snappy extension return "%s.snappy" % wal_name elif self.compression == "zstd": # add zst extension return "%s.zst" % wal_name elif self.compression == "lz4": # add lz4 extension return "%s.lz4" % wal_name else: raise ValueError("Unknown compression type: %s" % self.compression) if __name__ == "__main__": main() barman-3.13.3/barman/clients/cloud_backup_show.py0000644000175100001660000000721015002177425020173 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . from __future__ import print_function import json import logging from contextlib import closing from barman.clients.cloud_cli import ( GeneralErrorExit, NetworkErrorExit, OperationErrorExit, create_argument_parser, ) from barman.cloud import CloudBackupCatalog, configure_logging from barman.cloud_providers import get_cloud_interface from barman.output import ConsoleOutputWriter from barman.utils import force_str def main(args=None): """ The main script entry point :param list[str] args: the raw arguments list. When not provided it defaults to sys.args[1:] """ config = parse_arguments(args) configure_logging(config) try: cloud_interface = get_cloud_interface(config) with closing(cloud_interface): catalog = CloudBackupCatalog( cloud_interface=cloud_interface, server_name=config.server_name ) if not cloud_interface.test_connectivity(): raise NetworkErrorExit() # If test is requested, just exit after connectivity test elif config.test: raise SystemExit(0) if not cloud_interface.bucket_exists: logging.error("Bucket %s does not exist", cloud_interface.bucket_name) raise OperationErrorExit() backup_id = catalog.parse_backup_id(config.backup_id) backup_info = catalog.get_backup_info(backup_id) if not backup_info: logging.error( "Backup %s for server %s does not exist", backup_id, config.server_name, ) raise OperationErrorExit() # Output if config.format == "console": ConsoleOutputWriter.render_show_backup(backup_info.to_dict(), print) else: # Match the `barman show-backup` top level structure json_output = {backup_info.server_name: backup_info.to_json()} print(json.dumps(json_output)) except Exception as exc: logging.error("Barman cloud backup show exception: %s", force_str(exc)) logging.debug("Exception details:", exc_info=exc) raise GeneralErrorExit() def parse_arguments(args=None): """ Parse command line arguments :param list[str] args: The raw arguments list :return: The options parsed """ parser, _, _ = create_argument_parser( description="This script can be used to show metadata for backups " "made with barman-cloud-backup command. " "Currently AWS S3, Azure Blob Storage and Google Cloud Storage are supported.", ) parser.add_argument("backup_id", help="the backup ID") parser.add_argument( "--format", default="console", help="Output format (console or json). Default console.", ) return parser.parse_args(args=args) if __name__ == "__main__": main() barman-3.13.3/barman/clients/cloud_walrestore.py0000644000175100001660000001561415002177425020064 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import logging import os import sys from contextlib import closing from barman.clients.cloud_cli import ( CLIErrorExit, GeneralErrorExit, NetworkErrorExit, OperationErrorExit, create_argument_parser, ) from barman.cloud import ALLOWED_COMPRESSIONS, configure_logging from barman.cloud_providers import get_cloud_interface from barman.exceptions import BarmanException from barman.utils import force_str from barman.xlog import hash_dir, is_any_xlog_file, is_backup_file, is_partial_file def main(args=None): """ The main script entry point :param list[str] args: the raw arguments list. When not provided it defaults to sys.args[1:] """ config = parse_arguments(args) configure_logging(config) # Validate the WAL file name before downloading it if not is_any_xlog_file(config.wal_name): logging.error("%s is an invalid name for a WAL file" % config.wal_name) raise CLIErrorExit() try: cloud_interface = get_cloud_interface(config) with closing(cloud_interface): downloader = CloudWalDownloader( cloud_interface=cloud_interface, server_name=config.server_name ) if not cloud_interface.test_connectivity(): raise NetworkErrorExit() # If test is requested, just exit after connectivity test elif config.test: raise SystemExit(0) if not cloud_interface.bucket_exists: logging.error("Bucket %s does not exist", cloud_interface.bucket_name) raise OperationErrorExit() downloader.download_wal(config.wal_name, config.wal_dest, config.no_partial) except Exception as exc: logging.error("Barman cloud WAL restore exception: %s", force_str(exc)) logging.debug("Exception details:", exc_info=exc) raise GeneralErrorExit() def parse_arguments(args=None): """ Parse command line arguments :return: The options parsed """ parser, _, _ = create_argument_parser( description="This script can be used as a `restore_command` " "to download WAL files previously archived with " "barman-cloud-wal-archive command. " "Currently AWS S3, Azure Blob Storage and Google Cloud Storage are supported.", ) parser.add_argument( "--no-partial", help="Do not download partial WAL files", action="store_true", default=False, ) parser.add_argument( "wal_name", help="The value of the '%%f' keyword (according to 'restore_command').", ) parser.add_argument( "wal_dest", help="The value of the '%%p' keyword (according to 'restore_command').", ) return parser.parse_args(args=args) class CloudWalDownloader(object): """ Cloud storage download client """ def __init__(self, cloud_interface, server_name): """ Object responsible for handling interactions with cloud storage :param CloudInterface cloud_interface: The interface to use to upload the backup :param str server_name: The name of the server as configured in Barman """ self.cloud_interface = cloud_interface self.server_name = server_name def download_wal(self, wal_name, wal_dest, no_partial): """ Download a WAL file from cloud storage :param str wal_name: Name of the WAL file :param str wal_dest: Full path of the destination WAL file :param bool no_partial: Do not download partial WAL files """ # Correctly format the source path on s3 source_dir = os.path.join( self.cloud_interface.path, self.server_name, "wals", hash_dir(wal_name) ) # Add a path separator if needed if not source_dir.endswith(os.path.sep): source_dir += os.path.sep wal_path = os.path.join(source_dir, wal_name) remote_name = None # Automatically detect compression based on the file extension compression = None for item in self.cloud_interface.list_bucket(wal_path): # perfect match (uncompressed file) if item == wal_path: remote_name = item continue # look for compressed files or .partial files # Detect compression basename = item for e, c in ALLOWED_COMPRESSIONS.items(): if item[-len(e) :] == e: # Strip extension basename = basename[: -len(e)] compression = c break # Check basename is a known xlog file (.partial?) if not is_any_xlog_file(basename): logging.warning("Unknown WAL file: %s", item) continue # Exclude backup informative files (not needed in recovery) elif is_backup_file(basename): logging.info("Skipping backup file: %s", item) continue # Exclude partial files if required elif no_partial and is_partial_file(basename): logging.info("Skipping partial file: %s", item) continue # Found candidate remote_name = item logging.info( "Found WAL %s for server %s as %s", wal_name, self.server_name, remote_name, ) break if not remote_name: logging.info( "WAL file %s for server %s does not exists", wal_name, self.server_name ) raise OperationErrorExit() if compression and sys.version_info < (3, 0, 0): raise BarmanException( "Compressed WALs cannot be restored with Python 2.x - " "please upgrade to a supported version of Python 3" ) # Download the file logging.debug( "Downloading %s to %s (%s)", remote_name, wal_dest, "decompressing " + compression if compression else "no compression", ) self.cloud_interface.download_file(remote_name, wal_dest, compression) if __name__ == "__main__": main() barman-3.13.3/barman/clients/cloud_cli.py0000644000175100001660000001460115002177425016437 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import argparse import csv import logging import barman from barman.utils import force_str class OperationErrorExit(SystemExit): """ Dedicated exit code for errors where connectivity to the cloud provider was ok but the operation still failed. """ def __init__(self): super(OperationErrorExit, self).__init__(1) class NetworkErrorExit(SystemExit): """Dedicated exit code for network related errors.""" def __init__(self): super(NetworkErrorExit, self).__init__(2) class CLIErrorExit(SystemExit): """Dedicated exit code for CLI level errors.""" def __init__(self): super(CLIErrorExit, self).__init__(3) class GeneralErrorExit(SystemExit): """Dedicated exit code for general barman cloud errors.""" def __init__(self): super(GeneralErrorExit, self).__init__(4) class UrlArgumentType(object): source = "source" destination = "destination" def get_missing_attrs(config, attrs): """ Returns list of each attr not found in config. :param argparse.Namespace config: The backup options provided at the command line. :param list[str] attrs: List of attribute names to be searched for in the config. :rtype: list[str] :return: List of all items in attrs which were not found as attributes of config. """ missing_options = [] for attr in attrs: if not getattr(config, attr): missing_options.append(attr) return missing_options def __parse_tag(tag): """Parse key,value tag with csv reader""" try: rows = list(csv.reader([tag], delimiter=",")) except csv.Error as exc: logging.error( "Error parsing tag %s: %s", tag, force_str(exc), ) raise CLIErrorExit() if len(rows) != 1 or len(rows[0]) != 2: logging.error( "Invalid tag format: %s", tag, ) raise CLIErrorExit() return tuple(rows[0]) def add_tag_argument(parser, name, help): parser.add_argument( "--%s" % name, type=__parse_tag, nargs="*", help=help, ) class CloudArgumentParser(argparse.ArgumentParser): """ArgumentParser which exits with CLIErrorExit on errors.""" def error(self, message): try: super(CloudArgumentParser, self).error(message) except SystemExit: raise CLIErrorExit() def create_argument_parser(description, source_or_destination=UrlArgumentType.source): """ Create a barman-cloud argument parser with the given description. Returns an `argparse.ArgumentParser` object which parses the core arguments and options for barman-cloud commands. """ parser = CloudArgumentParser( description=description, add_help=False, ) parser.add_argument( "%s_url" % source_or_destination, help=( "URL of the cloud %s, such as a bucket in AWS S3." " For example: `s3://bucket/path/to/folder`." ) % source_or_destination, ) parser.add_argument( "server_name", help="the name of the server as configured in Barman." ) parser.add_argument( "-V", "--version", action="version", version="%%(prog)s %s" % barman.__version__ ) parser.add_argument("--help", action="help", help="show this help message and exit") verbosity = parser.add_mutually_exclusive_group() verbosity.add_argument( "-v", "--verbose", action="count", default=0, help="increase output verbosity (e.g., -vv is more than -v)", ) verbosity.add_argument( "-q", "--quiet", action="count", default=0, help="decrease output verbosity (e.g., -qq is less than -q)", ) parser.add_argument( "-t", "--test", help="Test cloud connectivity and exit", action="store_true", default=False, ) parser.add_argument( "--cloud-provider", help="The cloud provider to use as a storage backend", choices=["aws-s3", "azure-blob-storage", "google-cloud-storage"], default="aws-s3", ) s3_arguments = parser.add_argument_group( "Extra options for the aws-s3 cloud provider" ) s3_arguments.add_argument( "--endpoint-url", help="Override default S3 endpoint URL with the given one", ) s3_arguments.add_argument( "-P", "--aws-profile", help="profile name (e.g. INI section in AWS credentials file)", ) s3_arguments.add_argument( "--profile", help="profile name (deprecated: replaced by --aws-profile)", dest="aws_profile", ) s3_arguments.add_argument( "--read-timeout", type=int, help="the time in seconds until a timeout is raised when waiting to " "read from a connection (defaults to 60 seconds)", ) azure_arguments = parser.add_argument_group( "Extra options for the azure-blob-storage cloud provider" ) azure_arguments.add_argument( "--azure-credential", "--credential", "--default", choices=["azure-cli", "managed-identity", "default"], help="Optionally specify the type of credential to use when authenticating " "with Azure. If omitted then Azure Blob Storage credentials will be obtained " "from the environment and the default Azure authentication flow will be used " "for authenticating with all other Azure services. If no credentials can be " "found in the environment then the default Azure authentication flow will " "also be used for Azure Blob Storage.", dest="azure_credential", ) return parser, s3_arguments, azure_arguments barman-3.13.3/barman/clients/cloud_backup_list.py0000644000175100001660000001052615002177425020172 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import json import logging from contextlib import closing from barman.clients.cloud_cli import ( GeneralErrorExit, NetworkErrorExit, OperationErrorExit, create_argument_parser, ) from barman.cloud import CloudBackupCatalog, configure_logging from barman.cloud_providers import get_cloud_interface from barman.infofile import BackupInfo from barman.utils import force_str def main(args=None): """ The main script entry point :param list[str] args: the raw arguments list. When not provided it defaults to sys.args[1:] """ config = parse_arguments(args) configure_logging(config) try: cloud_interface = get_cloud_interface(config) with closing(cloud_interface): catalog = CloudBackupCatalog( cloud_interface=cloud_interface, server_name=config.server_name ) if not cloud_interface.test_connectivity(): raise NetworkErrorExit() # If test is requested, just exit after connectivity test elif config.test: raise SystemExit(0) if not cloud_interface.bucket_exists: logging.error("Bucket %s does not exist", cloud_interface.bucket_name) raise OperationErrorExit() backup_list = catalog.get_backup_list() # Output if config.format == "console": COLUMNS = "{:<20}{:<25}{:<30}{:<17}{:<20}" print( COLUMNS.format( "Backup ID", "End Time", "Begin Wal", "Archival Status", "Name", ) ) for backup_id in sorted(backup_list): item = backup_list[backup_id] if item and item.status == BackupInfo.DONE: keep_target = catalog.get_keep_target(item.backup_id) keep_status = ( keep_target and "KEEP:%s" % keep_target.upper() or "" ) print( COLUMNS.format( item.backup_id, item.end_time.strftime("%Y-%m-%d %H:%M:%S"), item.begin_wal, keep_status, item.backup_name or "", ) ) else: print( json.dumps( { "backups_list": [ backup_list[backup_id].to_json() for backup_id in sorted(backup_list) ] } ) ) except Exception as exc: logging.error("Barman cloud backup list exception: %s", force_str(exc)) logging.debug("Exception details:", exc_info=exc) raise GeneralErrorExit() def parse_arguments(args=None): """ Parse command line arguments :return: The options parsed """ parser, _, _ = create_argument_parser( description="This script can be used to list backups " "made with barman-cloud-backup command. " "Currently AWS S3, Azure Blob Storage and Google Cloud Storage are supported.", ) parser.add_argument( "--format", default="console", help="Output format (console or json). Default console.", ) return parser.parse_args(args=args) if __name__ == "__main__": main() barman-3.13.3/barman/clients/cloud_compression.py0000644000175100001660000001615115002177425020233 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import bz2 import gzip import lzma import shutil from abc import ABCMeta, abstractmethod from io import BytesIO from barman.compression import _try_import_lz4, _try_import_zstd from barman.utils import with_metaclass def _try_import_snappy(): try: import snappy except ImportError: raise SystemExit("Missing required python module: python-snappy") return snappy class ChunkedCompressor(with_metaclass(ABCMeta, object)): """ Base class for all ChunkedCompressors """ @abstractmethod def add_chunk(self, data): """ Compresses the supplied data and returns all the compressed bytes. :param bytes data: The chunk of data to be compressed :return: The compressed data :rtype: bytes """ @abstractmethod def decompress(self, data): """ Decompresses the supplied chunk of data and returns at least part of the uncompressed data. :param bytes data: The chunk of data to be decompressed :return: The decompressed data :rtype: bytes """ class SnappyCompressor(ChunkedCompressor): """ A ChunkedCompressor implementation based on python-snappy """ def __init__(self): snappy = _try_import_snappy() self.compressor = snappy.StreamCompressor() self.decompressor = snappy.StreamDecompressor() def add_chunk(self, data): """ Compresses the supplied data and returns all the compressed bytes. :param bytes data: The chunk of data to be compressed :return: The compressed data :rtype: bytes """ return self.compressor.add_chunk(data) def decompress(self, data): """ Decompresses the supplied chunk of data and returns at least part of the uncompressed data. :param bytes data: The chunk of data to be decompressed :return: The decompressed data :rtype: bytes """ return self.decompressor.decompress(data) def get_compressor(compression): """ Helper function which returns a ChunkedCompressor for the specified compression algorithm. Currently only snappy is supported. The other compression algorithms supported by barman cloud use the decompression built into TarFile. :param str compression: The compression algorithm to use. Can be set to snappy or any compression supported by the TarFile mode string. :return: A ChunkedCompressor capable of compressing and decompressing using the specified compression. :rtype: ChunkedCompressor """ if compression == "snappy": return SnappyCompressor() return None def compress(wal_file, compression): """ Compresses the supplied wal_file and returns a file-like object containing the compressed data. :param IOBase wal_file: A file-like object containing the WAL file data. :param str compression: The compression algorithm to apply. Can be one of: bzip2, gzip, snappy, zstd, lz4, xz. :return: The compressed data :rtype: BytesIO """ if compression == "snappy": in_mem_snappy = BytesIO() snappy = _try_import_snappy() snappy.stream_compress(wal_file, in_mem_snappy) in_mem_snappy.seek(0) return in_mem_snappy elif compression == "zstd": in_mem_zstd = BytesIO() zstd = _try_import_zstd() zstd.ZstdCompressor().copy_stream(wal_file, in_mem_zstd) in_mem_zstd.seek(0) return in_mem_zstd elif compression == "lz4": lz4 = _try_import_lz4() in_mem_lz4 = BytesIO(lz4.frame.compress(wal_file.read())) in_mem_lz4.seek(0) return in_mem_lz4 elif compression == "gzip": # Create a BytesIO for in memory compression in_mem_gzip = BytesIO() with gzip.GzipFile(fileobj=in_mem_gzip, mode="wb") as gz: # copy the gzipped data in memory shutil.copyfileobj(wal_file, gz) in_mem_gzip.seek(0) return in_mem_gzip elif compression == "bzip2": # Create a BytesIO for in memory compression in_mem_bz2 = BytesIO(bz2.compress(wal_file.read())) in_mem_bz2.seek(0) return in_mem_bz2 elif compression == "xz": in_mem_xz = BytesIO(lzma.compress(wal_file.read())) in_mem_xz.seek(0) return in_mem_xz else: raise ValueError("Unknown compression type: %s" % compression) def get_streaming_tar_mode(mode, compression): """ Helper function used in streaming uploads and downloads which appends the supplied compression to the raw filemode (either r or w) and returns the result. Any compression algorithms supported by barman-cloud but not Python TarFile are ignored so that barman-cloud can apply them itself. :param str mode: The file mode to use, either r or w. :param str compression: The compression algorithm to use. Can be set to snappy or any compression supported by the TarFile mode string. :return: The full filemode for a streaming tar file :rtype: str """ if compression == "snappy" or compression is None: return "%s|" % mode else: return "%s|%s" % (mode, compression) def decompress_to_file(blob, dest_file, compression): """ Decompresses the supplied blob of data into the dest_file file-like object using the specified compression. :param IOBase blob: A file-like object containing the compressed data. :param IOBase dest_file: A file-like object into which the uncompressed data should be written. :param str compression: The compression algorithm to apply. Can be one of: bzip2, gzip, snappy, zstd, lz4, xz. :rtype: None """ if compression == "snappy": snappy = _try_import_snappy() snappy.stream_decompress(blob, dest_file) return elif compression == "zstd": zstd = _try_import_zstd() source_file = zstd.ZstdDecompressor().stream_reader(blob) elif compression == "lz4": lz4 = _try_import_lz4() source_file = lz4.frame.open(blob, mode="rb") elif compression == "gzip": source_file = gzip.GzipFile(fileobj=blob, mode="rb") elif compression == "bzip2": source_file = bz2.BZ2File(blob, "rb") elif compression == "xz": source_file = lzma.open(blob, "rb") else: raise ValueError("Unknown compression type: %s" % compression) with source_file: shutil.copyfileobj(source_file, dest_file) barman-3.13.3/barman/clients/cloud_check_wal_archive.py0000644000175100001660000000611315002177425021310 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import logging from barman.clients.cloud_cli import ( GeneralErrorExit, NetworkErrorExit, OperationErrorExit, UrlArgumentType, create_argument_parser, ) from barman.cloud import CloudBackupCatalog, configure_logging from barman.cloud_providers import get_cloud_interface from barman.exceptions import WalArchiveContentError from barman.utils import check_positive, force_str from barman.xlog import check_archive_usable def main(args=None): """ The main script entry point :param list[str] args: the raw arguments list. When not provided it defaults to sys.args[1:] """ config = parse_arguments(args) configure_logging(config) try: cloud_interface = get_cloud_interface(config) if not cloud_interface.test_connectivity(): # Deliberately raise an error if we cannot connect raise NetworkErrorExit() # If test is requested, just exit after connectivity test elif config.test: raise SystemExit(0) if not cloud_interface.bucket_exists: # If the bucket does not exist then the check should pass return catalog = CloudBackupCatalog(cloud_interface, config.server_name) wals = list(catalog.get_wal_paths().keys()) check_archive_usable( wals, timeline=config.timeline, ) except WalArchiveContentError as err: logging.error( "WAL archive check failed for server %s: %s", config.server_name, force_str(err), ) raise OperationErrorExit() except Exception as exc: logging.error("Barman cloud WAL archive check exception: %s", force_str(exc)) logging.debug("Exception details:", exc_info=exc) raise GeneralErrorExit() def parse_arguments(args=None): """ Parse command line arguments :return: The options parsed """ parser, _, _ = create_argument_parser( description="Checks that the WAL archive on the specified cloud storage " "can be safely used for a new PostgreSQL server.", source_or_destination=UrlArgumentType.destination, ) parser.add_argument( "--timeline", help="The earliest timeline whose WALs should cause the check to fail", type=check_positive, ) return parser.parse_args(args=args) if __name__ == "__main__": main() barman-3.13.3/barman/clients/cloud_backup_keep.py0000644000175100001660000001037015002177425020140 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import logging from contextlib import closing from barman.annotations import KeepManager from barman.clients.cloud_cli import ( GeneralErrorExit, NetworkErrorExit, OperationErrorExit, create_argument_parser, ) from barman.cloud import CloudBackupCatalog, configure_logging from barman.cloud_providers import get_cloud_interface from barman.infofile import BackupInfo from barman.utils import force_str def main(args=None): """ The main script entry point :param list[str] args: the raw arguments list. When not provided it defaults to sys.args[1:] """ config = parse_arguments(args) configure_logging(config) try: cloud_interface = get_cloud_interface(config) with closing(cloud_interface): if not cloud_interface.test_connectivity(): raise NetworkErrorExit() # If test is requested, just exit after connectivity test elif config.test: raise SystemExit(0) if not cloud_interface.bucket_exists: logging.error("Bucket %s does not exist", cloud_interface.bucket_name) raise OperationErrorExit() catalog = CloudBackupCatalog(cloud_interface, config.server_name) backup_id = catalog.parse_backup_id(config.backup_id) if config.release: catalog.release_keep(backup_id) elif config.status: target = catalog.get_keep_target(backup_id) if target: print("Keep: %s" % target) else: print("Keep: nokeep") else: backup_info = catalog.get_backup_info(backup_id) if backup_info.status == BackupInfo.DONE: catalog.keep_backup(backup_id, config.target) else: logging.error( "Cannot add keep to backup %s because it has status %s. " "Only backups with status DONE can be kept.", backup_id, backup_info.status, ) raise OperationErrorExit() except Exception as exc: logging.error("Barman cloud keep exception: %s", force_str(exc)) logging.debug("Exception details:", exc_info=exc) raise GeneralErrorExit() def parse_arguments(args=None): """ Parse command line arguments :return: The options parsed """ parser, _, _ = create_argument_parser( description="This script can be used to tag backups in cloud storage as " "archival backups such that they will not be deleted. " "Currently AWS S3, Azure Blob Storage and Google Cloud Storage are supported.", ) parser.add_argument( "backup_id", help="the backup ID of the backup to be kept", ) keep_options = parser.add_mutually_exclusive_group(required=True) keep_options.add_argument( "-r", "--release", help="If specified, the command will remove the keep annotation and the " "backup will be eligible for deletion", action="store_true", ) keep_options.add_argument( "-s", "--status", help="Print the keep status of the backup", action="store_true", ) keep_options.add_argument( "--target", help="Specify the recovery target for this backup", choices=[KeepManager.TARGET_FULL, KeepManager.TARGET_STANDALONE], ) return parser.parse_args(args=args) if __name__ == "__main__": main() barman-3.13.3/barman/clients/walrestore.py0000755000175100001660000004410115002177425016672 0ustar 00000000000000# -*- coding: utf-8 -*- # walrestore - Remote Barman WAL restore command for PostgreSQL # # This script remotely fetches WAL files from Barman via SSH, on demand. # It is intended to be used in restore_command in recovery configuration files # of PostgreSQL standby servers. Supports parallel fetching and # protects against SSH failures. # # See the help page for usage information. # # © Copyright EnterpriseDB UK Limited 2016-2025 # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . from __future__ import print_function import argparse import os import shutil import subprocess import sys import time from tempfile import NamedTemporaryFile from types import SimpleNamespace import barman from barman.compression import CompressionManager, InternalCompressor from barman.utils import force_str DEFAULT_USER = "barman" DEFAULT_SPOOL_DIR = "/var/tmp/walrestore" # The string_types list is used to identify strings # in a consistent way between python 2 and 3 if sys.version_info[0] == 3: string_types = (str,) else: string_types = (basestring,) # noqa def main(args=None): """ The main script entry point """ config = parse_arguments(args) # Do connectivity test if requested if config.test: connectivity_test(config) return # never reached if config.compression is not None: print( "WARNING: `%s` option is deprecated and will be removed in future versions. " "For WAL compression, please make sure to enable it directly on the Barman " "server via the `compression` configuration option" % config.compression ) # Check WAL destination is not a directory if os.path.isdir(config.wal_dest): exit_with_error( "WAL_DEST cannot be a directory: %s" % config.wal_dest, status=3 ) # Open the destination file try: dest_file = open(config.wal_dest, "wb") except EnvironmentError as e: exit_with_error( "Cannot open '%s' (WAL_DEST) for writing: %s" % (config.wal_dest, e), status=3, ) return # never reached # If the file is present in SPOOL_DIR use it and terminate try_deliver_from_spool(config, dest_file.name) # If required load the list of files to download in parallel additional_files = peek_additional_files(config) try: # Execute barman get-wal through the ssh connection ssh_process = RemoteGetWal(config, config.wal_name, dest_file) except EnvironmentError as e: exit_with_error('Error executing "ssh": %s' % e, sleep=config.sleep) return # never reached # Spawn a process for every additional file parallel_ssh_processes = spawn_additional_process(config, additional_files) # Wait for termination of every subprocess. If CTRL+C is pressed, # terminate all of them try: RemoteGetWal.wait_for_all() finally: # Cleanup failed spool files in case of errors for process in parallel_ssh_processes: if process.returncode != 0: os.unlink(process.dest_file) # If the command succeeded exit here if ssh_process.returncode == 0: sys.exit(0) # Report the exit code, remapping ssh failure code (255) to 2 if ssh_process.returncode == 255: exit_with_error("Connection problem with ssh", 2, sleep=config.sleep) else: exit_with_error( "Remote 'barman get-wal' command has failed!", ssh_process.returncode, sleep=config.sleep, ) def spawn_additional_process(config, additional_files): """ Execute additional barman get-wal processes :param argparse.Namespace config: the configuration from command line :param additional_files: A list of WAL file to be downloaded in parallel :return list[subprocess.Popen]: list of created processes """ processes = [] for wal_name in additional_files: spool_file_name = os.path.join(config.spool_dir, wal_name) try: # Spawn a process and write the output in the spool dir process = RemoteGetWal(config, wal_name, spool_file_name) processes.append(process) except EnvironmentError: # If execution has failed make sure the spool file is unlinked try: os.unlink(spool_file_name) except EnvironmentError: # Suppress unlink errors pass return processes def peek_additional_files(config): """ Invoke remote get-wal --peek to receive a list of wal files to copy :param argparse.Namespace config: the configuration from command line :returns set: a set of WAL file names from the peek command """ # If parallel downloading is not required return an empty array if not config.parallel: return [] # Make sure the SPOOL_DIR exists try: if not os.path.exists(config.spool_dir): os.mkdir(config.spool_dir) except EnvironmentError as e: exit_with_error("Cannot create '%s' directory: %s" % (config.spool_dir, e)) # Retrieve the list of files from remote additional_files = execute_peek(config) # Sanity check if len(additional_files) == 0 or additional_files[0] != config.wal_name: exit_with_error("The required file is not available: %s" % config.wal_name) # Remove the first element, as now we know is identical to config.wal_name del additional_files[0] return additional_files def build_ssh_command(config, wal_name, peek=0): """ Prepare an ssh command according to the arguments passed on command line :param argparse.Namespace config: the configuration from command line :param str wal_name: the wal_name get-wal parameter :param int peek: in :return list[str]: the ssh command as list of string """ ssh_command = ["ssh"] if config.port is not None: ssh_command += ["-p", config.port] ssh_command += [ "-q", # quiet mode - suppress warnings "-T", # disable pseudo-terminal allocation "%s@%s" % (config.user, config.barman_host), "barman", ] if config.config: ssh_command.append("--config %s" % config.config) options = [] if config.test: options.append("--test") if peek: options.append("--peek '%s'" % peek) if config.compression: options.append("--%s" % config.compression) if config.keep_compression: options.append("--keep-compression") if config.partial: options.append("--partial") if options: get_wal_command = "get-wal %s '%s' '%s'" % ( " ".join(options), config.server_name, wal_name, ) else: get_wal_command = "get-wal '%s' '%s'" % (config.server_name, wal_name) ssh_command.append(get_wal_command) return ssh_command def execute_peek(config): """ Invoke remote get-wal --peek to receive a list of wal file to copy :param argparse.Namespace config: the configuration from command line :returns set: a set of WAL file names from the peek command """ # Build the peek command ssh_command = build_ssh_command(config, config.wal_name, config.parallel) # Issue the command try: output = subprocess.Popen(ssh_command, stdout=subprocess.PIPE).communicate() return list(output[0].decode().splitlines()) except subprocess.CalledProcessError as e: exit_with_error("Impossible to invoke remote get-wal --peek: %s" % e) def try_deliver_from_spool(config, dest_file): """ Search for the requested file in the spool directory. If is already present, then copy it locally and exit, return otherwise. :param argparse.Namespace config: the configuration from command line :param dest_file: The path to the destination file """ spool_file = str(os.path.join(config.spool_dir, config.wal_name)) # id the file is not present, give up if not os.path.exists(spool_file): return try: shutil.move(spool_file, dest_file) sys.exit(0) except IOError as e: exit_with_error("Failure moving %s to %s: %s" % (spool_file, dest_file, e)) def exit_with_error(message, status=2, sleep=0): """ Print ``message`` and terminate the script with ``status`` :param str message: message to print :param int status: script exit code :param int sleep: second to sleep before exiting """ print("ERROR: %s" % message, file=sys.stderr) # Sleep for config.sleep seconds if required if sleep: print("Sleeping for %d seconds." % sleep, file=sys.stderr) time.sleep(sleep) sys.exit(status) def connectivity_test(config): """ Invoke remote get-wal --test to test the connection with Barman server :param argparse.Namespace config: the configuration from command line """ # Build the peek command ssh_command = build_ssh_command(config, "dummy_wal_name") # Issue the command try: pipe = subprocess.Popen( ssh_command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT ) output = pipe.communicate() print(force_str(output[0])) sys.exit(pipe.returncode) except subprocess.CalledProcessError as e: exit_with_error("Impossible to invoke remote get-wal: %s" % e) def parse_arguments(args=None): """ Parse the command line arguments :param list[str] args: the raw arguments list. When not provided it defaults to sys.args[1:] :rtype: argparse.Namespace """ parser = argparse.ArgumentParser( description="This script will be used as a 'restore_command' " "based on the get-wal feature of Barman. " "A ssh connection will be opened to the Barman host.", ) parser.add_argument( "-V", "--version", action="version", version="%%(prog)s %s" % barman.__version__ ) parser.add_argument( "-U", "--user", default=DEFAULT_USER, help="The user used for the ssh connection to the Barman server. " "Defaults to '%(default)s'.", ) parser.add_argument( "--port", help="The port used for the ssh connection to the Barman server.", ) parser.add_argument( "-s", "--sleep", default=0, type=int, metavar="SECONDS", help="Sleep for SECONDS after a failure of get-wal request. " "Defaults to 0 (nowait).", ) parser.add_argument( "-p", "--parallel", default=0, type=int, metavar="JOBS", help="Specifies the number of files to peek and transfer " "in parallel. " "Defaults to 0 (disabled).", ) parser.add_argument( "--spool-dir", default=DEFAULT_SPOOL_DIR, metavar="SPOOL_DIR", help="Specifies spool directory for WAL files. Defaults to " "'{0}'.".format(DEFAULT_SPOOL_DIR), ) parser.add_argument( "-P", "--partial", help="retrieve also partial WAL files (.partial)", action="store_true", dest="partial", default=False, ) compression_parser = parser.add_mutually_exclusive_group() compression_parser.add_argument( "-z", "--gzip", help="Transfer the WAL files compressed with gzip", action="store_const", const="gzip", dest="compression", ) compression_parser.add_argument( "-j", "--bzip2", help="Transfer the WAL files compressed with bzip2", action="store_const", const="bzip2", dest="compression", ) compression_parser.add_argument( "--keep-compression", help="Preserve compression during transfer, decompress once received", action="store_true", dest="keep_compression", ) parser.add_argument( "-c", "--config", metavar="CONFIG", help="configuration file on the Barman server", ) parser.add_argument( "-t", "--test", action="store_true", help="test both the connection and the configuration of the " "requested PostgreSQL server in Barman to make sure it is " "ready to receive WAL files. With this option, " "the 'wal_name' and 'wal_dest' mandatory arguments are ignored.", ) parser.add_argument( "barman_host", metavar="BARMAN_HOST", help="The host of the Barman server.", ) parser.add_argument( "server_name", metavar="SERVER_NAME", help="The server name configured in Barman from which WALs are taken.", ) parser.add_argument( "wal_name", metavar="WAL_NAME", help="The value of the '%%f' keyword (according to 'restore_command').", ) parser.add_argument( "wal_dest", metavar="WAL_DEST", help="The value of the '%%p' keyword (according to 'restore_command').", ) return parser.parse_args(args=args) class RemoteGetWal(object): processes = set() """ The list of processes that has been spawned by RemoteGetWal """ def __init__(self, config, wal_name, dest_file): """ Spawn a process that download a WAL from remote. If needed decompress the remote stream on the fly. :param argparse.Namespace config: the configuration from command line :param wal_name: The name of WAL to download :param dest_file: The destination file name or a writable file object """ self.config = config self.wal_name = wal_name self.source_file = None self.dest_file = None self.decompressor_process = None # If a string has been passed, it's the name of the destination file # We convert it in a writable binary file object if isinstance(dest_file, string_types): self.dest_file = dest_file dest_file = open(dest_file, "wb") # Spawn a remote get-wal process self.ssh_process = subprocess.Popen( build_ssh_command(config, wal_name), stdout=subprocess.PIPE ) # Create a temporary file with the WAL content received self.source_file = NamedTemporaryFile( mode="r+b", prefix=".%s." % os.path.basename(wal_name) ) shutil.copyfileobj(self.ssh_process.stdout, self.source_file) self.source_file.seek(0) # Close the pipe descriptor, letting the ssh process receive the SIGPIPE self.ssh_process.stdout.close() # Identify the WAL compression, if any server_config = self._get_server_config_minimal(config) compression_manager = CompressionManager(server_config, None) compression = compression_manager.identify_compression(self.source_file.name) # If there's no compression then just copy the content to the destination file if compression is None: shutil.copyfileobj(self.source_file, dest_file) else: # If compression is present we proceed differently depending on the compressor compressor = compression_manager.get_compressor(compression) # If InternalCompressor, we can decompress directly to the destination file if isinstance(compressor, InternalCompressor): compressor.decompress(self.source_file.name, dest_file.name) else: # Otherwise it's a CommandCompressor so we spawn the local decompressor self.decompressor_process = subprocess.Popen( [config.compression, "-d"], stdin=self.source_file, stdout=dest_file, ) # close the opened file dest_file.close() # Register the spawned processes in the class registry self.processes.add(self.ssh_process) if self.decompressor_process: self.processes.add(self.decompressor_process) def _get_server_config_minimal(self, config): """ Returns a placeholder for a server config object with all compression parameters relevant to ``CompressionManager`` filled. :param argparse.Namespace config: the configuration from command line """ return SimpleNamespace( compression=config.compression, custom_compression_magic=None, custom_compression_filter=None, custom_decompression_filter=None, ) @classmethod def wait_for_all(cls): """ Wait for the termination of all the registered spawned processes. """ try: while len(cls.processes): time.sleep(0.1) for process in cls.processes.copy(): if process.poll() is not None: cls.processes.remove(process) except KeyboardInterrupt: # If a SIGINT has been received, make sure that every subprocess # terminate for process in cls.processes: process.kill() exit_with_error("SIGINT received! Terminating.") @property def returncode(self): """ Return the exit code of the RemoteGetWal processes. A remote get-wal process return code is 0 only if both the remote get-wal process and the eventual decompressor return 0 :return: exit code of the RemoteGetWal processes """ if self.ssh_process.returncode != 0: return self.ssh_process.returncode if self.decompressor_process: if self.decompressor_process.returncode != 0: return self.decompressor_process.returncode return 0 if __name__ == "__main__": main() barman-3.13.3/barman/clients/cloud_backup_delete.py0000644000175100001660000004510115002177425020456 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import logging import os from contextlib import closing from operator import attrgetter from barman import xlog from barman.backup import BackupManager from barman.clients.cloud_cli import ( CLIErrorExit, GeneralErrorExit, NetworkErrorExit, OperationErrorExit, create_argument_parser, ) from barman.cloud import CloudBackupCatalog, configure_logging from barman.cloud_providers import ( get_cloud_interface, get_snapshot_interface_from_backup_info, ) from barman.exceptions import BadXlogPrefix, InvalidRetentionPolicy from barman.retention_policies import RetentionPolicyFactory from barman.utils import check_non_negative, force_str def _get_files_for_backup(catalog, backup_info): backup_files = [] # Sort the files by OID so that we always get a stable order. The PGDATA dir # has no OID so we use a -1 for sorting purposes, such that it always sorts # ahead of the tablespaces. for oid, backup_file in sorted( catalog.get_backup_files(backup_info, allow_missing=True).items(), key=lambda x: x[0] if x[0] else -1, ): key = oid or "PGDATA" for file_info in [backup_file] + sorted( backup_file.additional_files, key=attrgetter("path") ): # Silently skip files which could not be found - if they don't exist # then not being able to delete them is not an error condition here if file_info.path is not None: logging.debug( "Will delete archive for %s at %s" % (key, file_info.path) ) backup_files.append(file_info.path) return backup_files def _remove_wals_for_backup( cloud_interface, catalog, deleted_backup, dry_run, skip_wal_cleanup_if_standalone=True, ): # An implementation of BackupManager.remove_wal_before_backup which does not # use xlogdb, since xlogdb is not available to barman-cloud should_remove_wals, wal_ranges_to_protect = BackupManager.should_remove_wals( deleted_backup, catalog.get_backup_list(), keep_manager=catalog, skip_wal_cleanup_if_standalone=skip_wal_cleanup_if_standalone, ) next_backup = BackupManager.find_next_backup_in( catalog.get_backup_list(), deleted_backup.backup_id ) wals_to_delete = {} if should_remove_wals: # There is no previous backup or all previous backups are archival # standalone backups, so we can remove unused WALs (those WALs not # required by standalone archival backups). # If there is a next backup then all unused WALs up to the begin_wal # of the next backup can be removed. # If there is no next backup then there are no remaining backups, # because we must assume non-exclusive backups are taken, we can only # safely delete unused WALs up to begin_wal of the deleted backup. # See comments in barman.backup.BackupManager.delete_backup. if next_backup: remove_until = next_backup else: remove_until = deleted_backup # A WAL is only a candidate for deletion if it is on the same timeline so we # use BackupManager to get a set of all other timelines with backups so that # we can preserve all WALs on other timelines. timelines_to_protect = BackupManager.get_timelines_to_protect( remove_until=remove_until, deleted_backup=deleted_backup, available_backups=catalog.get_backup_list(), ) # Identify any prefixes under which all WALs are no longer needed. # This is a shortcut which allows us to delete all WALs under a prefix without # checking each individual WAL. try: wal_prefixes = catalog.get_wal_prefixes() except NotImplementedError: # If fetching WAL prefixes isn't supported by the cloud provider then # the old method of checking each WAL must be used for all WALs. wal_prefixes = [] deletable_prefixes = [] for wal_prefix in wal_prefixes: try: tli_and_log = wal_prefix.split("/")[-2] tli, log = xlog.decode_hash_dir(tli_and_log) except (BadXlogPrefix, IndexError): # If the prefix does not appear to be a tli and log we output a warning # and move on to the next prefix rather than error out. logging.warning( "Ignoring malformed WAL object prefix: {}".format(wal_prefix) ) continue # If this prefix contains a timeline which should be protected then we # cannot delete the WALS under it so advance to the next prefix. if tli in timelines_to_protect: continue # If the tli and log fall are inclusively between the tli and log for the # begin and end WAL of any protected WAL range then this prefix cannot be # deleted outright. for begin_wal, end_wal in wal_ranges_to_protect: begin_tli, begin_log, _ = xlog.decode_segment_name(begin_wal) end_tli, end_log, _ = xlog.decode_segment_name(end_wal) if ( tli >= begin_tli and log >= begin_log and tli <= end_tli and log <= end_log ): break else: # The prefix tli and log do not match any protected timelines or # protected WAL ranges so all WALs are eligible for deletion if the tli # is the same timeline and the log is below the begin_wal log of the # backup being deleted. until_begin_tli, until_begin_log, _ = xlog.decode_segment_name( remove_until.begin_wal ) if tli == until_begin_tli and log < until_begin_log: # All WALs under this prefix pre-date the backup being deleted so they # can be deleted in one request. deletable_prefixes.append(wal_prefix) for wal_prefix in deletable_prefixes: if not dry_run: cloud_interface.delete_under_prefix(wal_prefix) else: print( "Skipping deletion of all objects under prefix %s " "due to --dry-run option" % wal_prefix ) try: wal_paths = catalog.get_wal_paths() except Exception as exc: logging.error( "Cannot clean up WALs for backup %s because an error occurred listing WALs: %s", deleted_backup.backup_id, force_str(exc), ) return for wal_name, wal in wal_paths.items(): # If the wal starts with a prefix we deleted then ignore it so that the # dry-run output is accurate if any(wal.startswith(prefix) for prefix in deletable_prefixes): continue if xlog.is_history_file(wal_name): continue if timelines_to_protect: tli, _, _ = xlog.decode_segment_name(wal_name) if tli in timelines_to_protect: continue # Check if the WAL is in a protected range, required by an archival # standalone backup - so do not delete it if xlog.is_backup_file(wal_name): # If we have a backup file, truncate the name for the range check range_check_wal_name = wal_name[:24] else: range_check_wal_name = wal_name if any( range_check_wal_name >= begin_wal and range_check_wal_name <= end_wal for begin_wal, end_wal in wal_ranges_to_protect ): continue if wal_name < remove_until.begin_wal: wals_to_delete[wal_name] = wal # Explicitly sort because dicts are not ordered in python < 3.6 wal_paths_to_delete = sorted(wals_to_delete.values()) if len(wal_paths_to_delete) > 0: if not dry_run: try: cloud_interface.delete_objects(wal_paths_to_delete) except Exception as exc: logging.error( "Could not delete the following WALs for backup %s: %s, Reason: %s", deleted_backup.backup_id, wal_paths_to_delete, force_str(exc), ) # Return early so that we leave the WALs in the local cache so they # can be cleaned up should there be a subsequent backup deletion. return else: print( "Skipping deletion of objects %s due to --dry-run option" % wal_paths_to_delete ) for wal_name in wals_to_delete.keys(): catalog.remove_wal_from_cache(wal_name) def _delete_backup( cloud_interface, catalog, backup_id, config, skip_wal_cleanup_if_standalone=True, ): backup_info = catalog.get_backup_info(backup_id) if not backup_info: logging.warning("Backup %s does not exist", backup_id) return if backup_info.snapshots_info: logging.debug( "Will delete the following snapshots: %s", ", ".join( snapshot.identifier for snapshot in backup_info.snapshots_info.snapshots ), ) if not config.dry_run: snapshot_interface = get_snapshot_interface_from_backup_info( backup_info, config ) snapshot_interface.delete_snapshot_backup(backup_info) else: print("Skipping deletion of snapshots due to --dry-run option") # Delete the backup_label for snapshots backups as this is not stored in the # same format used by the non-snapshot backups. backup_label_path = os.path.join( catalog.prefix, backup_info.backup_id, "backup_label" ) if not config.dry_run: cloud_interface.delete_objects([backup_label_path]) else: print("Skipping deletion of %s due to --dry-run option" % backup_label_path) objects_to_delete = _get_files_for_backup(catalog, backup_info) backup_info_path = os.path.join( catalog.prefix, backup_info.backup_id, "backup.info" ) logging.debug("Will delete backup.info file at %s" % backup_info_path) if not config.dry_run: try: cloud_interface.delete_objects(objects_to_delete) # Do not try to delete backup.info until we have successfully deleted # everything else so that it is possible to retry the operation should # we fail to delete any backup file cloud_interface.delete_objects([backup_info_path]) except Exception as exc: logging.error("Could not delete backup %s: %s", backup_id, force_str(exc)) raise OperationErrorExit() else: print( "Skipping deletion of objects %s due to --dry-run option" % (objects_to_delete + [backup_info_path]) ) _remove_wals_for_backup( cloud_interface, catalog, backup_info, config.dry_run, skip_wal_cleanup_if_standalone, ) # It is important that the backup is removed from the catalog after cleaning # up the WALs because the code in _remove_wals_for_backup depends on the # deleted backup existing in the backup catalog catalog.remove_backup_from_cache(backup_id) def main(args=None): """ The main script entry point :param list[str] args: the raw arguments list. When not provided it defaults to sys.args[1:] """ config = parse_arguments(args) configure_logging(config) try: cloud_interface = get_cloud_interface(config) with closing(cloud_interface): if not cloud_interface.test_connectivity(): raise NetworkErrorExit() # If test is requested, just exit after connectivity test elif config.test: raise SystemExit(0) if not cloud_interface.bucket_exists: logging.error("Bucket %s does not exist", cloud_interface.bucket_name) raise OperationErrorExit() catalog = CloudBackupCatalog( cloud_interface=cloud_interface, server_name=config.server_name ) # Call catalog.get_backup_list now so we know we can read the whole catalog # (the results are cached so this does not result in extra calls to cloud # storage) catalog.get_backup_list() if len(catalog.unreadable_backups) > 0: logging.error( "Cannot read the following backups: %s\n" "Unsafe to proceed with deletion due to failure reading backup catalog" % catalog.unreadable_backups ) raise OperationErrorExit() if config.backup_id: backup_id = catalog.parse_backup_id(config.backup_id) # Because we only care about one backup, skip the annotation cache # because it is only helpful when dealing with multiple backups if catalog.should_keep_backup(backup_id, use_cache=False): logging.error( "Skipping delete of backup %s for server %s " "as it has a current keep request. If you really " "want to delete this backup please remove the keep " "and try again.", backup_id, config.server_name, ) raise OperationErrorExit() if config.minimum_redundancy > 0: if config.minimum_redundancy >= len(catalog.get_backup_list()): logging.error( "Skipping delete of backup %s for server %s " "due to minimum redundancy requirements " "(minimum redundancy = %s, " "current redundancy = %s)", backup_id, config.server_name, config.minimum_redundancy, len(catalog.get_backup_list()), ) raise OperationErrorExit() _delete_backup(cloud_interface, catalog, backup_id, config) elif config.retention_policy: try: retention_policy = RetentionPolicyFactory.create( "retention_policy", config.retention_policy, server_name=config.server_name, catalog=catalog, minimum_redundancy=config.minimum_redundancy, ) except InvalidRetentionPolicy as exc: logging.error( "Could not create retention policy %s: %s", config.retention_policy, force_str(exc), ) raise CLIErrorExit() # Sort to ensure that we delete the backups in ascending order, that is # from oldest to newest. This ensures that the relevant WALs will be cleaned # up after each backup is deleted. backups_to_delete = sorted( [ backup_id for backup_id, status in retention_policy.report().items() if status == "OBSOLETE" ] ) for backup_id in backups_to_delete: _delete_backup( cloud_interface, catalog, backup_id, config, skip_wal_cleanup_if_standalone=False, ) except Exception as exc: logging.error("Barman cloud backup delete exception: %s", force_str(exc)) logging.debug("Exception details:", exc_info=exc) raise GeneralErrorExit() def parse_arguments(args=None): """ Parse command line arguments :return: The options parsed """ parser, _, _ = create_argument_parser( description="This script can be used to delete backups " "made with barman-cloud-backup command. " "Currently AWS S3, Azure Blob Storage and Google Cloud Storage are supported.", ) delete_arguments = parser.add_mutually_exclusive_group(required=True) delete_arguments.add_argument( "-b", "--backup-id", help="Backup ID of the backup to be deleted", ) parser.add_argument( "-m", "--minimum-redundancy", type=check_non_negative, help="The minimum number of backups that should always be available.", default=0, ) delete_arguments.add_argument( "-r", "--retention-policy", help="If specified, delete all backups eligible for deletion according to the " "supplied retention policy. Syntax: REDUNDANCY value | RECOVERY WINDOW OF " "value {DAYS | WEEKS | MONTHS}", ) parser.add_argument( "--dry-run", action="store_true", help="Find the objects which need to be deleted but do not delete them", ) parser.add_argument( "--batch-size", dest="delete_batch_size", type=int, help="The maximum number of objects to be deleted in a single request to the " "cloud provider. If unset then the maximum allowed batch size for the " "specified cloud provider will be used (1000 for aws-s3, 256 for " "azure-blob-storage and 100 for google-cloud-storage).", ) return parser.parse_args(args=args) if __name__ == "__main__": main() barman-3.13.3/barman/clients/__init__.py0000644000175100001660000000132415002177425016237 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2019-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . barman-3.13.3/barman/clients/cloud_backup.py0000755000175100001660000004231615002177425017144 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import logging import os import re import tempfile from contextlib import closing from shutil import rmtree from barman.clients.cloud_cli import ( GeneralErrorExit, NetworkErrorExit, OperationErrorExit, UrlArgumentType, add_tag_argument, create_argument_parser, ) from barman.cloud import ( CloudBackupSnapshot, CloudBackupUploader, CloudBackupUploaderBarman, configure_logging, ) from barman.cloud_providers import get_cloud_interface, get_snapshot_interface from barman.exceptions import ( BarmanException, ConfigurationException, PostgresConnectionError, UnrecoverableHookScriptError, ) from barman.postgres import PostgreSQLConnection from barman.utils import ( check_aws_expiration_date_format, check_aws_snapshot_lock_cool_off_period_range, check_aws_snapshot_lock_duration_range, check_backup_name, check_positive, check_size, force_str, ) _find_space = re.compile(r"[\s]").search def __is_hook_script(): """Check the environment and determine if we are running as a hook script""" if "BARMAN_HOOK" in os.environ and "BARMAN_PHASE" in os.environ: if ( os.getenv("BARMAN_HOOK") in ("backup_script", "backup_retry_script") and os.getenv("BARMAN_PHASE") == "post" ): return True else: raise BarmanException( "barman-cloud-backup called as unsupported hook script: %s_%s" % (os.getenv("BARMAN_PHASE"), os.getenv("BARMAN_HOOK")) ) else: return False def quote_conninfo(value): """ Quote a connection info parameter :param str value: :rtype: str """ if not value: return "''" if not _find_space(value): return value return "'%s'" % value.replace("\\", "\\\\").replace("'", "\\'") def build_conninfo(config): """ Build a DSN to connect to postgres using command-line arguments """ conn_parts = [] # If -d specified a conninfo string, just return it if config.dbname is not None: if config.dbname == "" or "=" in config.dbname: return config.dbname if config.host: conn_parts.append("host=%s" % quote_conninfo(config.host)) if config.port: conn_parts.append("port=%s" % quote_conninfo(config.port)) if config.user: conn_parts.append("user=%s" % quote_conninfo(config.user)) if config.dbname: conn_parts.append("dbname=%s" % quote_conninfo(config.dbname)) return " ".join(conn_parts) def _validate_config(config): """ Additional validation for config such as mutually inclusive options. Raises a ConfigurationException if any options are missing or incompatible. :param argparse.Namespace config: The backup options provided at the command line. """ required_snapshot_variables = ( "snapshot_disks", "snapshot_instance", ) is_snapshot_backup = any( [getattr(config, var) for var in required_snapshot_variables] ) if is_snapshot_backup: if getattr(config, "compression"): raise ConfigurationException( "Compression options cannot be used with snapshot backups" ) if getattr(config, "aws_snapshot_lock_mode", None) == "governance" and getattr( config, "aws_snapshot_lock_cool_off_period", None ): raise ConfigurationException( "'aws_snapshot_lock_mode' = 'governance' cannot be used with " "'aws_snapshot_lock_cool_off_period'" ) def main(args=None): """ The main script entry point :param list[str] args: the raw arguments list. When not provided it defaults to sys.args[1:] """ config = parse_arguments(args) configure_logging(config) tempdir = tempfile.mkdtemp(prefix="barman-cloud-backup-") try: _validate_config(config) # Create any temporary file in the `tempdir` subdirectory tempfile.tempdir = tempdir cloud_interface = get_cloud_interface(config) if not cloud_interface.test_connectivity(): raise NetworkErrorExit() # If test is requested, just exit after connectivity test elif config.test: raise SystemExit(0) with closing(cloud_interface): # TODO: Should the setup be optional? cloud_interface.setup_bucket() # Perform the backup uploader_kwargs = { "server_name": config.server_name, "compression": config.compression, "max_archive_size": config.max_archive_size, "min_chunk_size": config.min_chunk_size, "max_bandwidth": config.max_bandwidth, "cloud_interface": cloud_interface, } if __is_hook_script(): if config.backup_name: raise BarmanException( "Cannot set backup name when running as a hook script" ) if "BARMAN_BACKUP_DIR" not in os.environ: raise BarmanException( "BARMAN_BACKUP_DIR environment variable not set" ) if "BARMAN_BACKUP_ID" not in os.environ: raise BarmanException( "BARMAN_BACKUP_ID environment variable not set" ) if os.getenv("BARMAN_STATUS") != "DONE": raise UnrecoverableHookScriptError( "backup in '%s' has status '%s' (status should be: DONE)" % (os.getenv("BARMAN_BACKUP_DIR"), os.getenv("BARMAN_STATUS")) ) uploader = CloudBackupUploaderBarman( backup_dir=os.getenv("BARMAN_BACKUP_DIR"), backup_id=os.getenv("BARMAN_BACKUP_ID"), backup_info_path=os.getenv("BARMAN_BACKUP_INFO_PATH"), **uploader_kwargs, ) uploader.backup() else: conninfo = build_conninfo(config) postgres = PostgreSQLConnection( conninfo, config.immediate_checkpoint, application_name="barman_cloud_backup", ) try: postgres.connect() except PostgresConnectionError as exc: logging.error("Cannot connect to postgres: %s", force_str(exc)) logging.debug("Exception details:", exc_info=exc) raise OperationErrorExit() with closing(postgres): # Take snapshot backups if snapshot backups were specified if config.snapshot_disks or config.snapshot_instance: snapshot_interface = get_snapshot_interface(config) snapshot_interface.validate_backup_config(config) snapshot_backup = CloudBackupSnapshot( config.server_name, cloud_interface, snapshot_interface, postgres, config.snapshot_instance, config.snapshot_disks, config.backup_name, ) snapshot_backup.backup() # Otherwise upload everything to the object store else: uploader = CloudBackupUploader( postgres=postgres, backup_name=config.backup_name, **uploader_kwargs, ) uploader.backup() except KeyboardInterrupt as exc: logging.error("Barman cloud backup was interrupted by the user") logging.debug("Exception details:", exc_info=exc) raise OperationErrorExit() except UnrecoverableHookScriptError as exc: logging.error("Barman cloud backup exception: %s", force_str(exc)) logging.debug("Exception details:", exc_info=exc) raise SystemExit(63) except Exception as exc: logging.error("Barman cloud backup exception: %s", force_str(exc)) logging.debug("Exception details:", exc_info=exc) raise GeneralErrorExit() finally: # Remove the temporary directory and all the contained files rmtree(tempdir, ignore_errors=True) def parse_arguments(args=None): """ Parse command line arguments :return: The options parsed """ parser, s3_arguments, azure_arguments = create_argument_parser( description="This script can be used to perform a backup " "of a local PostgreSQL instance and ship " "the resulting tarball(s) to the Cloud. " "Currently AWS S3, Azure Blob Storage and Google Cloud Storage are supported.", source_or_destination=UrlArgumentType.destination, ) compression = parser.add_mutually_exclusive_group() compression.add_argument( "-z", "--gzip", help="gzip-compress the backup while uploading to the cloud", action="store_const", const="gz", dest="compression", ) compression.add_argument( "-j", "--bzip2", help="bzip2-compress the backup while uploading to the cloud", action="store_const", const="bz2", dest="compression", ) compression.add_argument( "--snappy", help="snappy-compress the backup while uploading to the cloud ", action="store_const", const="snappy", dest="compression", ) parser.add_argument( "-h", "--host", help="host or Unix socket for PostgreSQL connection " "(default: libpq settings)", ) parser.add_argument( "-p", "--port", help="port for PostgreSQL connection (default: libpq settings)", ) parser.add_argument( "-U", "--user", help="user name for PostgreSQL connection (default: libpq settings)", ) parser.add_argument( "--immediate-checkpoint", help="forces the initial checkpoint to be done as quickly as possible", action="store_true", ) parser.add_argument( "-J", "--jobs", type=check_positive, help="number of subprocesses to upload data to cloud storage (default: 2)", default=2, ) parser.add_argument( "-S", "--max-archive-size", type=check_size, help="maximum size of an archive when uploading to cloud storage " "(default: 100GB)", default="100GB", ) parser.add_argument( "--min-chunk-size", type=check_size, help="minimum size of an individual chunk when uploading to cloud storage " "(default: 5MB for aws-s3, 64KB for azure-blob-storage, not applicable for " "google-cloud-storage)", default=None, # Defer to the cloud interface if nothing is specified ) parser.add_argument( "--max-bandwidth", type=check_size, help="the maximum amount of data to be uploaded per second when backing up to " "either AWS S3 or Azure Blob Storage (default: no limit)", default=None, ) parser.add_argument( "-d", "--dbname", help="Database name or conninfo string for Postgres connection (default: postgres)", default="postgres", ) parser.add_argument( "-n", "--name", help="a name which can be used to reference this backup in commands " "such as barman-cloud-restore and barman-cloud-backup-delete", default=None, type=check_backup_name, dest="backup_name", ) parser.add_argument( "--snapshot-instance", help="Instance where the disks to be backed up as snapshots are attached", ) parser.add_argument( "--snapshot-disk", help="Name of a disk from which snapshots should be taken", metavar="NAME", action="append", default=[], dest="snapshot_disks", ) parser.add_argument( "--snapshot-zone", help=( "Zone of the disks from which snapshots should be taken (deprecated: " "replaced by --gcp-zone)" ), dest="gcp_zone", ) gcs_arguments = parser.add_argument_group( "Extra options for google-cloud-storage cloud provider" ) gcs_arguments.add_argument( "--snapshot-gcp-project", help=( "GCP project under which disk snapshots should be stored (deprecated: " "replaced by --gcp-project)" ), dest="gcp_project", ) gcs_arguments.add_argument( "--gcp-project", help="GCP project under which disk snapshots should be stored", ) gcs_arguments.add_argument( "--kms-key-name", help="The name of the GCP KMS key which should be used for encrypting the " "uploaded data in GCS.", ) gcs_arguments.add_argument( "--gcp-zone", help="Zone of the disks from which snapshots should be taken", ) add_tag_argument( parser, name="tags", help="Tags to be added to all uploaded files in cloud storage", ) s3_arguments.add_argument( "-e", "--encryption", help="The encryption algorithm used when storing the uploaded data in S3. " "Allowed values: 'AES256'|'aws:kms'.", choices=["AES256", "aws:kms"], ) s3_arguments.add_argument( "--sse-kms-key-id", help="The AWS KMS key ID that should be used for encrypting the uploaded data " "in S3. Can be specified using the key ID on its own or using the full ARN for " "the key. Only allowed if `-e/--encryption` is set to `aws:kms`.", ) s3_arguments.add_argument( "--aws-region", help="The name of the AWS region containing the EC2 VM and storage volumes " "defined by the --snapshot-instance and --snapshot-disk arguments.", ) s3_arguments.add_argument( "--aws-await-snapshots-timeout", default=3600, help="The length of time in seconds to wait for snapshots to be created in AWS before " "timing out (default: 3600 seconds)", type=check_positive, ) s3_arguments.add_argument( "--aws-snapshot-lock-mode", help="The lock mode to apply to the snapshot. Allowed values: " "'governance'|'compliance'.", choices=["governance", "compliance"], ) s3_arguments.add_argument( "--aws-snapshot-lock-cool-off-period", help="Specifies the cool-off period (in hours) for a snapshot locked in " "'compliance' mode, allowing you to unlock or modify lock settings after it is " "locked. Range must be from 1 to 72. To lock the snapshot immediately without " "a cool-off period, leave this option unset.", type=check_aws_snapshot_lock_cool_off_period_range, ) s3_lock_target_group = s3_arguments.add_mutually_exclusive_group() s3_lock_target_group.add_argument( "--aws-snapshot-lock-expiration-date", help="The expiration date for a locked snapshot in the format " "YYYY-MM-DDThh:mm:ss.sssZ. To lock a snapshot, you must specify either this " "argument or --aws-snapshot-lock-duration, but not both.", type=check_aws_expiration_date_format, ) s3_lock_target_group.add_argument( "--aws-snapshot-lock-duration", help="The duration (in days) for which the snapshot should be locked. Range " "must be from 1 to 36500. To lock a snapshopt, you must specify either this " "argument or --aws-snapshot-lock-expiration-date, but not both.", type=check_aws_snapshot_lock_duration_range, ) azure_arguments.add_argument( "--encryption-scope", help="The name of an encryption scope defined in the Azure Blob Storage " "service which is to be used to encrypt the data in Azure", ) azure_arguments.add_argument( "--azure-subscription-id", help="The ID of the Azure subscription which owns the instance and storage " "volumes defined by the --snapshot-instance and --snapshot-disk arguments.", ) azure_arguments.add_argument( "--azure-resource-group", help="The name of the Azure resource group to which the compute instance and " "disks defined by the --snapshot-instance and --snapshot-disk arguments belong.", ) parsed_args = parser.parse_args(args=args) return parsed_args if __name__ == "__main__": main() barman-3.13.3/barman/cli.py0000644000175100001660000024465715002177425013630 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module implements the interface with the command line and the logger. """ import argparse import json import logging import os import sys from argparse import SUPPRESS, ArgumentParser, ArgumentTypeError, HelpFormatter from collections import OrderedDict from contextlib import closing import barman.config import barman.diagnose import barman.utils from barman import output from barman.annotations import KeepManager from barman.backup_manifest import BackupManifest from barman.config import ConfigChangesProcessor, RecoveryOptions, parse_staging_path from barman.exceptions import ( BadXlogSegmentName, LockFileBusy, RecoveryException, SyncError, WalArchiveContentError, ) from barman.infofile import BackupInfo, WalFileInfo from barman.lockfile import ConfigUpdateLock from barman.server import Server from barman.storage.local_file_manager import LocalFileManager from barman.utils import ( RESERVED_BACKUP_IDS, SHA256, BarmanEncoder, check_backup_name, check_non_negative, check_positive, check_tli, configure_logging, drop_privileges, force_str, get_backup_id_using_shortcut, get_log_levels, parse_log_level, parse_target_tli, ) from barman.xlog import check_archive_usable if sys.version_info.major < 3: from argparse import Action, _ActionsContainer, _SubParsersAction try: import argcomplete except ImportError: argcomplete = None _logger = logging.getLogger(__name__) # Support aliases for argparse in python2. # Derived from https://gist.github.com/sampsyo/471779 and based on the # initial patchset for CPython for supporting aliases in argparse. # Licensed under CC0 1.0 if sys.version_info.major < 3: class AliasedSubParsersAction(_SubParsersAction): old_init = staticmethod(_ActionsContainer.__init__) @staticmethod def _containerInit( self, description, prefix_chars, argument_default, conflict_handler ): AliasedSubParsersAction.old_init( self, description, prefix_chars, argument_default, conflict_handler ) self.register("action", "parsers", AliasedSubParsersAction) class _AliasedPseudoAction(Action): def __init__(self, name, aliases, help): dest = name if aliases: dest += " (%s)" % ",".join(aliases) sup = super(AliasedSubParsersAction._AliasedPseudoAction, self) sup.__init__(option_strings=[], dest=dest, help=help) def add_parser(self, name, **kwargs): aliases = kwargs.pop("aliases", []) parser = super(AliasedSubParsersAction, self).add_parser(name, **kwargs) # Make the aliases work. for alias in aliases: self._name_parser_map[alias] = parser # Make the help text reflect them, first removing old help entry. if "help" in kwargs: help_text = kwargs.pop("help") self._choices_actions.pop() pseudo_action = self._AliasedPseudoAction(name, aliases, help_text) self._choices_actions.append(pseudo_action) return parser # override argparse to register new subparser action by default _ActionsContainer.__init__ = AliasedSubParsersAction._containerInit class OrderedHelpFormatter(HelpFormatter): def _format_usage(self, usage, actions, groups, prefix): for action in actions: if not action.option_strings: action.choices = OrderedDict(sorted(action.choices.items())) return super(OrderedHelpFormatter, self)._format_usage( usage, actions, groups, prefix ) p = ArgumentParser( epilog="Barman by EnterpriseDB (www.enterprisedb.com)", formatter_class=OrderedHelpFormatter, ) p.add_argument( "-v", "--version", action="version", version="%s\n\nBarman by EnterpriseDB (www.enterprisedb.com)" % barman.__version__, ) p.add_argument( "-c", "--config", help="uses a configuration file " "(defaults: %s)" % ", ".join(barman.config.Config.CONFIG_FILES), default=SUPPRESS, ) p.add_argument( "--color", "--colour", help="Whether to use colors in the output", choices=["never", "always", "auto"], default="auto", ) p.add_argument( "--log-level", help="Override the default log level", choices=list(get_log_levels()), default=SUPPRESS, ) p.add_argument("-q", "--quiet", help="be quiet", action="store_true") p.add_argument("-d", "--debug", help="debug output", action="store_true") p.add_argument( "-f", "--format", help="output format", choices=output.AVAILABLE_WRITERS.keys(), default=output.DEFAULT_WRITER, ) subparsers = p.add_subparsers(dest="command") def argument(*name_or_flags, **kwargs): """Convenience function to properly format arguments to pass to the command decorator. """ # Remove the completer keyword argument from the dictionary completer = kwargs.pop("completer", None) return (list(name_or_flags), completer, kwargs) def command(args=None, parent=subparsers, cmd_aliases=None): """Decorator to define a new subcommand in a sanity-preserving way. The function will be stored in the ``func`` variable when the parser parses arguments so that it can be called directly like so:: args = cli.parse_args() args.func(args) Usage example:: @command([argument("-d", help="Enable debug mode", action="store_true")]) def command(args): print(args) Then on the command line:: $ python cli.py command -d """ if args is None: args = [] if cmd_aliases is None: cmd_aliases = [] def decorator(func): parser = parent.add_parser( func.__name__.replace("_", "-"), description=func.__doc__, help=func.__doc__, aliases=cmd_aliases, ) parent._choices_actions = sorted(parent._choices_actions, key=lambda x: x.dest) for arg in args: if arg[1]: parser.add_argument(*arg[0], **arg[2]).completer = arg[1] else: parser.add_argument(*arg[0], **arg[2]) parser.set_defaults(func=func) return func return decorator @command() def help(args=None): """ show this help message and exit """ p.print_help() def check_target_action(value): """ Check the target action option :param value: str containing the value to check """ if value is None: return None if value in ("pause", "shutdown", "promote"): return value raise ArgumentTypeError("'%s' is not a valid recovery target action" % value) @command( [argument("--minimal", help="machine readable output", action="store_true")], cmd_aliases=["list-server"], ) def list_servers(args): """ List available servers, with useful information """ # Get every server, both inactive and temporarily disabled servers = get_server_list() for name in sorted(servers): server = servers[name] # Exception: manage_server_command is not invoked here # Normally you would call manage_server_command to check if the # server is None and to report inactive and disabled servers, but here # we want all servers and the server cannot be None output.init("list_server", name, minimal=args.minimal) description = server.config.description or "" # If the server has been manually disabled if not server.config.active: description += " (inactive)" # If server has configuration errors elif server.config.disabled: description += " (WARNING: disabled)" # If server is a passive node if server.passive_node: description += " (Passive)" output.result("list_server", name, description) output.close_and_exit() @command( [ argument( "--keep-descriptors", help="Keep the stdout and the stderr streams attached to Barman subprocesses", action="store_true", ) ] ) def cron(args): """ Run maintenance tasks (global command) """ # Before doing anything, check if the configuration file has been updated try: with ConfigUpdateLock(barman.__config__.barman_lock_directory): procesor = ConfigChangesProcessor(barman.__config__) procesor.process_conf_changes_queue() except LockFileBusy: output.warning("another process is updating barman configuration files") # Skip inactive and temporarily disabled servers servers = get_server_list( skip_inactive=True, skip_disabled=True, wal_streaming=True ) for name in sorted(servers): server = servers[name] # Exception: manage_server_command is not invoked here # Normally you would call manage_server_command to check if the # server is None and to report inactive and disabled servers, # but here we have only active and well configured servers. try: server.cron(keep_descriptors=args.keep_descriptors) except Exception: # A cron should never raise an exception, so this code # should never be executed. However, it is here to protect # unrelated servers in case of unexpected failures. output.exception( "Unable to run cron on server '%s', " "please look in the barman log file for more details.", name, ) # Lockfile directory cleanup barman.utils.lock_files_cleanup( barman.__config__.barman_lock_directory, barman.__config__.lock_directory_cleanup, ) output.close_and_exit() @command(cmd_aliases=["lock-directory-cleanup"]) def lock_directory_cleanup(args=None): """ Cleanup command for the lock directory, takes care of leftover lock files. """ barman.utils.lock_files_cleanup(barman.__config__.barman_lock_directory, True) output.close_and_exit() # noinspection PyUnusedLocal def server_completer(prefix, parsed_args, **kwargs): global_config(parsed_args) for conf in barman.__config__.servers(): if conf.name.startswith(prefix): yield conf.name # noinspection PyUnusedLocal def server_completer_all(prefix, parsed_args, **kwargs): global_config(parsed_args) current_list = getattr(parsed_args, "server_name", None) or () for conf in barman.__config__.servers(): if conf.name.startswith(prefix) and conf.name not in current_list: yield conf.name if len(current_list) == 0 and "all".startswith(prefix): yield "all" # noinspection PyUnusedLocal def backup_completer(prefix, parsed_args, **kwargs): global_config(parsed_args) server = get_server(parsed_args) backups = server.get_available_backups() for backup_id in sorted(backups, reverse=True): if backup_id.startswith(prefix): yield backup_id for special_id in RESERVED_BACKUP_IDS: if len(backups) > 0 and special_id.startswith(prefix): yield special_id @command( [ argument( "server_name", completer=server_completer_all, nargs="+", help="specifies the server names for the backup command " "('all' will show all available servers)", ), argument( "--immediate-checkpoint", help="forces the initial checkpoint to be done as quickly as possible", dest="immediate_checkpoint", action="store_true", default=SUPPRESS, ), argument( "--no-immediate-checkpoint", help="forces the initial checkpoint to be spread", dest="immediate_checkpoint", action="store_false", default=SUPPRESS, ), argument( "--incremental", completer=backup_completer, dest="backup_id", help="performs an incremental backup. An ID of a previous backup must " "be provided ('latest' and 'latest-full' are also available options)", ), argument( "--reuse-backup", nargs="?", choices=barman.config.REUSE_BACKUP_VALUES, default=None, const="link", help="use the previous backup to improve transfer-rate. " 'If no argument is given "link" is assumed', ), argument( "--retry-times", help="Number of retries after an error if base backup copy fails.", type=check_non_negative, ), argument( "--retry-sleep", help="Wait time after a failed base backup copy, before retrying.", type=check_non_negative, ), argument( "--no-retry", help="Disable base backup copy retry logic.", dest="retry_times", action="store_const", const=0, ), argument( "--jobs", "-j", help="Run the copy in parallel using NJOBS processes.", type=check_positive, metavar="NJOBS", ), argument( "--jobs-start-batch-period", help="The time period in seconds over which a single batch of jobs will " "be started.", type=check_positive, ), argument( "--jobs-start-batch-size", help="The maximum number of parallel Rsync jobs to start in a single " "batch.", type=check_positive, ), argument( "--bwlimit", help="maximum transfer rate in kilobytes per second. " "A value of 0 means no limit. Overrides 'bandwidth_limit' " "configuration option.", metavar="KBPS", type=check_non_negative, default=SUPPRESS, ), argument( "--wait", "-w", help="wait for all the required WAL files to be archived", dest="wait", action="store_true", default=False, ), argument( "--wait-timeout", help="the time, in seconds, spent waiting for the required " "WAL files to be archived before timing out", dest="wait_timeout", metavar="TIMEOUT", default=None, type=check_non_negative, ), argument( "--keepalive-interval", help="An interval, in seconds, at which a heartbeat query will be sent " "to the server to keep the libpq connection alive during an Rsync backup.", dest="keepalive_interval", type=check_non_negative, ), argument( "--name", help="a name which can be used to reference this backup in barman " "commands such as restore and delete", dest="backup_name", default=None, type=check_backup_name, ), argument( "--manifest", help="forces the creation of the backup manifest file for the " "rsync backup method", dest="automatic_manifest", action="store_true", default=SUPPRESS, ), argument( "--no-manifest", help="disables the creation of the backup manifest file for the " "rsync backup method", dest="automatic_manifest", action="store_false", default=SUPPRESS, ), ] ) def backup(args): """ Perform a full backup for the given server (supports 'all') """ servers = get_server_list(args, skip_inactive=True, skip_passive=True) for name in sorted(servers): server = servers[name] # Skip the server (apply general rule) if not manage_server_command(server, name): continue incremental_kwargs = {} if args.backup_id is not None: parent_backup_info = parse_backup_id(server, args) if parent_backup_info: incremental_kwargs["parent_backup_id"] = parent_backup_info.backup_id if args.reuse_backup is not None: server.config.reuse_backup = args.reuse_backup if args.retry_sleep is not None: server.config.basebackup_retry_sleep = args.retry_sleep if args.retry_times is not None: server.config.basebackup_retry_times = args.retry_times if args.keepalive_interval is not None: server.config.keepalive_interval = args.keepalive_interval if hasattr(args, "immediate_checkpoint"): # As well as overriding the immediate_checkpoint value in the config # we must also update the immediate_checkpoint attribute on the # postgres connection because it has already been set from the config server.config.immediate_checkpoint = args.immediate_checkpoint server.postgres.immediate_checkpoint = args.immediate_checkpoint if hasattr(args, "automatic_manifest"): # Override the set value for the autogenerate_manifest config option. # The backup executor class will automatically ignore --manifest requests # for backup methods different from rsync. server.config.autogenerate_manifest = args.automatic_manifest if args.jobs is not None: server.config.parallel_jobs = args.jobs if args.jobs_start_batch_size is not None: server.config.parallel_jobs_start_batch_size = args.jobs_start_batch_size if args.jobs_start_batch_period is not None: server.config.parallel_jobs_start_batch_period = ( args.jobs_start_batch_period ) if hasattr(args, "bwlimit"): server.config.bandwidth_limit = args.bwlimit with closing(server): server.backup( wait=args.wait, wait_timeout=args.wait_timeout, backup_name=args.backup_name, **incremental_kwargs, ) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer_all, nargs="+", help="specifies the server name for the command " "('all' will show all available servers)", ), argument("--minimal", help="machine readable output", action="store_true"), ], cmd_aliases=["list-backup"], ) def list_backups(args): """ List available backups for the given server (supports 'all') """ servers = get_server_list(args, skip_inactive=True) for name in sorted(servers): server = servers[name] # Skip the server (apply general rule) if not manage_server_command(server, name): continue output.init("list_backup", name, minimal=args.minimal) with closing(server): server.list_backups() output.close_and_exit() @command( [ argument( "server_name", completer=server_completer_all, nargs="+", help="specifies the server name for the command", ) ] ) def status(args): """ Shows live information and status of the PostgreSQL server """ servers = get_server_list(args, skip_inactive=True) for name in sorted(servers): server = servers[name] # Skip the server (apply general rule) if not manage_server_command(server, name): continue output.init("status", name) with closing(server): server.status() output.close_and_exit() @command( [ argument( "server_name", completer=server_completer_all, nargs="+", help="specifies the server name for the command " "('all' will show all available servers)", ), argument("--minimal", help="machine readable output", action="store_true"), argument( "--target", choices=("all", "hot-standby", "wal-streamer"), default="all", help=""" Possible values are: 'hot-standby' (only hot standby servers), 'wal-streamer' (only WAL streaming clients, such as pg_receivewal), 'all' (any of them). Defaults to %(default)s""", ), argument( "--source", choices=("backup-host", "wal-host"), default="backup-host", help=""" Possible values are: 'backup-host' (list clients using the backup conninfo for a server) or `wal-host` (list clients using the WAL streaming conninfo for a server). Defaults to %(default)s""", ), ] ) def replication_status(args): """ Shows live information and status of any streaming client """ wal_streaming = args.source == "wal-host" servers = get_server_list( args, skip_inactive=True, skip_passive=True, wal_streaming=wal_streaming ) for name in sorted(servers): server = servers[name] # Skip the server (apply general rule) if not manage_server_command(server, name): continue with closing(server): output.init("replication_status", name, minimal=args.minimal) server.replication_status(args.target) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer_all, nargs="+", help="specifies the server name for the command ", ) ] ) def rebuild_xlogdb(args): """ Rebuild the WAL file database guessing it from the disk content. """ servers = get_server_list(args, skip_inactive=True) for name in sorted(servers): server = servers[name] # Skip the server (apply general rule) if not manage_server_command(server, name): continue with closing(server): server.rebuild_xlogdb() output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command ", ), argument("--target-tli", help="target timeline", type=check_tli), argument( "--target-time", help="target time. You can use any valid unambiguous representation. " 'e.g: "YYYY-MM-DD HH:MM:SS.mmm"', ), argument("--target-xid", help="target transaction ID"), argument("--target-lsn", help="target LSN (Log Sequence Number)"), argument( "--target-name", help="target name created previously with " "pg_create_restore_point() function call", ), argument( "--target-immediate", help="end recovery as soon as a consistent state is reached", action="store_true", default=False, ), argument( "--exclusive", help="set target to be non inclusive", action="store_true" ), argument( "--tablespace", help="tablespace relocation rule", metavar="NAME:LOCATION", action="append", ), argument( "--remote-ssh-command", metavar="SSH_COMMAND", help="This options activates remote recovery, by specifying the secure " "shell command to be launched on a remote host. It is " 'the equivalent of the "ssh_command" server option in ' "the configuration file for remote recovery. " 'Example: "ssh postgres@db2"', ), argument( "backup_id", completer=backup_completer, help="specifies the backup ID to restore", ), argument( "destination_directory", help="the directory where the new server is created", ), argument( "--staging-wal-directory", help="a staging directory in the target host for WAL files when performing " "PITR. If unspecified, it uses a `barman_wal` directory inside the " "destination directory.", ), argument( "--bwlimit", help="maximum transfer rate in kilobytes per second. " "A value of 0 means no limit. Overrides 'bandwidth_limit' " "configuration option.", metavar="KBPS", type=check_non_negative, default=SUPPRESS, ), argument( "--retry-times", help="Number of retries after an error if base backup copy fails.", type=check_non_negative, ), argument( "--retry-sleep", help="Wait time after a failed base backup copy, before retrying.", type=check_non_negative, ), argument( "--no-retry", help="Disable base backup copy retry logic.", dest="retry_times", action="store_const", const=0, ), argument( "--jobs", "-j", help="Run the copy in parallel using NJOBS processes.", type=check_positive, metavar="NJOBS", ), argument( "--jobs-start-batch-period", help="The time period in seconds over which a single batch of jobs will " "be started.", type=check_positive, ), argument( "--jobs-start-batch-size", help="The maximum number of Rsync jobs to start in a single batch.", type=check_positive, ), argument( "--get-wal", help="Enable the get-wal option during the recovery.", dest="get_wal", action="store_true", default=SUPPRESS, ), argument( "--no-get-wal", help="Disable the get-wal option during recovery.", dest="get_wal", action="store_false", default=SUPPRESS, ), argument( "--network-compression", help="Enable network compression during remote recovery.", dest="network_compression", action="store_true", default=SUPPRESS, ), argument( "--no-network-compression", help="Disable network compression during remote recovery.", dest="network_compression", action="store_false", default=SUPPRESS, ), argument( "--target-action", help="Specifies what action the server should take once the " "recovery target is reached. This option is not allowed for " "PostgreSQL < 9.1. If PostgreSQL is between 9.1 and 9.4 included " 'the only allowed value is "pause". If PostgreSQL is 9.5 or newer ' 'the possible values are "shutdown", "pause", "promote".', dest="target_action", type=check_target_action, default=SUPPRESS, ), argument( "--standby-mode", dest="standby_mode", action="store_true", default=SUPPRESS, help="Enable standby mode when starting the restored PostgreSQL instance", ), argument( "--recovery-staging-path", dest="recovery_staging_path", help=( "A path to a location on the recovery host where compressed backup " "files will be staged during the recovery. This location must have " "enough available space to temporarily hold the full compressed " "backup. This option is *required* when restoring from a compressed " "backup." ), ), argument( "--local-staging-path", help=( "A path to a location on the local host where incremental backups " "will be combined during the recovery. This location must have " "enough available space to temporarily hold the new synthetic " "backup. This option is *required* when restoring from an " "incremental backup." ), ), argument( "--recovery-conf-filename", dest="recovery_conf_filename", help=( "Name of the file to which recovery configuration options will be " "added for PostgreSQL 12 and later (default: postgresql.auto.conf)." ), ), argument( "--snapshot-recovery-instance", help="Instance where the disks recovered from the snapshots are attached", ), argument( "--snapshot-recovery-zone", help=( "Zone containing the instance and disks for the snapshot recovery " "(deprecated: replaced by --gcp-zone)" ), ), argument( "--gcp-zone", help="Zone containing the instance and disks for the snapshot recovery", ), argument( "--azure-resource-group", help="Azure resource group containing the instance and disks for recovery " "of a snapshot backup", ), argument( "--aws-region", help="The name of the AWS region containing the EC2 VM and storage " "volumes for recovery of a snapshot backup", ), ], cmd_aliases=["recover"], ) def restore(args): """ Restore a server at a given time, name, LSN or xid """ server = get_server(args) # PostgreSQL supports multiple parameters to specify when the recovery # process will end, and in that case the last entry in recovery # configuration files will be used. See [1] # # Since the meaning of the target options is not dependent on the order # of parameters, we decided to make the target options mutually exclusive. # # [1]: https://www.postgresql.org/docs/current/static/ # recovery-target-settings.html target_options = [ "target_time", "target_xid", "target_lsn", "target_name", "target_immediate", ] specified_target_options = [ option for option in target_options if getattr(args, option, None) ] if len(specified_target_options) > 1: output.error("You cannot specify multiple targets for the recovery operation") output.close_and_exit() target_option = ( specified_target_options[0] if len(specified_target_options) == 1 else None ) target_tli = None backup_info = None if args.backup_id != "auto": backup_info = parse_backup_id(server, args) else: target = getattr(args, target_option) if target_option else None # "Parse" the string value to integer for target_tli if passed as a string # ("current", "latest") target_tli = parse_target_tli( obj=server.backup_manager, target_tli=args.target_tli ) # Error out on recovery targets that are not allowed. if target_option in {"target_immediate", "target_xid", "target_name"}: output.error( "For PITR without a backup_id, the only possible recovery targets " "are target_time and target_lsn. '%s' recovery target is not " "allowed without a backup_id." % target_option ) output.close_and_exit() # Search for a candidate backup based on recovery targets if "backup_id" is None elif target_option is None: if target_tli is not None: backup_id = server.get_last_backup_id_from_target_tli(target_tli) else: backup_id = server.get_last_backup_id() elif target_option == "target_time": backup_id = server.get_closest_backup_id_from_target_time( target, target_tli ) elif target_option == "target_lsn": backup_id = server.get_closest_backup_id_from_target_lsn(target, target_tli) # If no candidate backup_id is found, error out. if backup_id is None: output.error("Cannot find any candidate backup for recovery.") output.close_and_exit() backup_info = server.get_backup(backup_id) if backup_info.status not in BackupInfo.STATUS_COPY_DONE: output.error( "Cannot restore from backup '%s' of server '%s': " "backup status is not DONE", args.backup_id, server.config.name, ) output.close_and_exit() # If the backup to be recovered is compressed then there are additional # checks to be carried out if backup_info.compression is not None: # Set the recovery staging path from the cli if it is set if args.recovery_staging_path is not None: try: recovery_staging_path = parse_staging_path(args.recovery_staging_path) except ValueError as exc: output.error("Cannot parse recovery staging path: %s", str(exc)) output.close_and_exit() server.config.recovery_staging_path = recovery_staging_path # If the backup is compressed but there is no recovery_staging_path # then this is an error - the user *must* tell barman where recovery # data can be staged. if server.config.recovery_staging_path is None: output.error( "Cannot restore from backup '%s' of server '%s': " "backup is compressed with %s compression but no recovery " "staging path is provided. Either set recovery_staging_path " "in the Barman config or use the --recovery-staging-path " "argument.", args.backup_id, server.config.name, backup_info.compression, ) output.close_and_exit() # If the backup to be recovered is incremental then there are additional # checks to be carried out if backup_info.is_incremental: # Set the local staging path from the cli if it is set if args.local_staging_path is not None: try: local_staging_path = parse_staging_path(args.local_staging_path) except ValueError as exc: output.error("Cannot parse local staging path: %s", str(exc)) output.close_and_exit() server.config.local_staging_path = local_staging_path # If the backup is incremental but there is no local_staging_path # then this is an error - the user *must* tell barman where recovery # data can be staged. if server.config.local_staging_path is None: output.error( "Cannot restore from backup '%s' of server '%s': " "backup will be combined with pg_combinebackup in the " "barman host but no local staging path is provided. " "Either set local_staging_path in the Barman config " "or use the --local-staging-path argument.", args.backup_id, server.config.name, ) output.close_and_exit() # decode the tablespace relocation rules tablespaces = {} if args.tablespace: for rule in args.tablespace: try: tablespaces.update([rule.split(":", 1)]) except ValueError: output.error( "Invalid tablespace relocation rule '%s'\n" "HINT: The valid syntax for a relocation rule is " "NAME:LOCATION", rule, ) output.close_and_exit() # validate the rules against the tablespace list valid_tablespaces = [] if backup_info.tablespaces: valid_tablespaces = [ tablespace_data.name for tablespace_data in backup_info.tablespaces ] for item in tablespaces: if item not in valid_tablespaces: output.error( "Invalid tablespace name '%s'\n" "HINT: Please use any of the following " "tablespaces: %s", item, ", ".join(valid_tablespaces), ) output.close_and_exit() # explicitly disallow the rsync remote syntax (common mistake) if ":" in args.destination_directory: output.error( "The destination directory parameter " "cannot contain the ':' character\n" "HINT: If you want to do a remote recovery you have to use " "the --remote-ssh-command option" ) output.close_and_exit() if args.retry_sleep is not None: server.config.basebackup_retry_sleep = args.retry_sleep if args.retry_times is not None: server.config.basebackup_retry_times = args.retry_times if hasattr(args, "get_wal"): if args.get_wal: server.config.recovery_options.add(RecoveryOptions.GET_WAL) elif RecoveryOptions.GET_WAL in server.config.recovery_options: server.config.recovery_options.remove(RecoveryOptions.GET_WAL) if args.jobs is not None: server.config.parallel_jobs = args.jobs if args.jobs_start_batch_size is not None: server.config.parallel_jobs_start_batch_size = args.jobs_start_batch_size if args.jobs_start_batch_period is not None: server.config.parallel_jobs_start_batch_period = args.jobs_start_batch_period if hasattr(args, "bwlimit"): server.config.bandwidth_limit = args.bwlimit if hasattr(args, "network_compression"): if args.network_compression and args.remote_ssh_command is None: output.error( "Network compression can only be used with " "remote recovery.\n" "HINT: If you want to do a remote recovery " "you have to use the --remote-ssh-command option" ) output.close_and_exit() server.config.network_compression = args.network_compression if backup_info.snapshots_info is not None: missing_args = [] if not args.snapshot_recovery_instance: missing_args.append("--snapshot-recovery-instance") if len(missing_args) > 0: output.error( "Backup %s is a snapshot backup and the following required arguments " "have not been provided: %s", backup_info.backup_id, ", ".join(missing_args), ) output.close_and_exit() if tablespaces != {}: output.error( "Backup %s is a snapshot backup therefore tablespace relocation rules " "cannot be used.", backup_info.backup_id, ) output.close_and_exit() # Set the snapshot keyword arguments to be passed to the recovery executor snapshot_kwargs = { "recovery_instance": args.snapshot_recovery_instance, } # Special handling for deprecated snapshot_recovery_zone arg if args.gcp_zone is None and args.snapshot_recovery_zone is not None: args.gcp_zone = args.snapshot_recovery_zone # Override provider-specific options in the config for arg in ( "aws_region", "azure_resource_group", "gcp_zone", ): value = getattr(args, arg) if value is not None: setattr(server.config, arg, value) else: unexpected_args = [] if args.snapshot_recovery_instance: unexpected_args.append("--snapshot-recovery-instance") if len(unexpected_args) > 0: output.error( "Backup %s is not a snapshot backup but the following snapshot " "arguments have been used: %s", backup_info.backup_id, ", ".join(unexpected_args), ) output.close_and_exit() # An empty dict is used so that snapshot-specific arguments are not passed to # non-snapshot recovery executors snapshot_kwargs = {} with closing(server): try: server.recover( backup_info, args.destination_directory, wal_dest=args.staging_wal_directory, tablespaces=tablespaces, target_tli=args.target_tli, target_time=args.target_time, target_xid=args.target_xid, target_lsn=args.target_lsn, target_name=args.target_name, target_immediate=args.target_immediate, exclusive=args.exclusive, remote_command=args.remote_ssh_command, target_action=getattr(args, "target_action", None), standby_mode=getattr(args, "standby_mode", None), recovery_conf_filename=args.recovery_conf_filename, **snapshot_kwargs, ) except RecoveryException as exc: output.error(force_str(exc)) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer_all, nargs="+", help="specifies the server names to show " "('all' will show all available servers)", ) ], cmd_aliases=["show-server"], ) def show_servers(args): """ Show all configuration parameters for the specified servers """ servers = get_server_list(args) for name in sorted(servers): server = servers[name] # Skip the server (apply general rule) if not manage_server_command( server, name, skip_inactive=False, skip_disabled=False, disabled_is_error=False, ): continue # If the server has been manually disabled if not server.config.active: description = "(inactive)" # If server has configuration errors elif server.config.disabled: description = "(WARNING: disabled)" else: description = None output.init("show_server", name, description=description) with closing(server): server.show() output.close_and_exit() @command( [ argument( "server_name", completer=server_completer_all, nargs="+", help="specifies the server name target of the switch-wal command", ), argument( "--force", help="forces the switch of a WAL by executing a checkpoint before", dest="force", action="store_true", default=False, ), argument( "--archive", help="wait for one WAL file to be archived", dest="archive", action="store_true", default=False, ), argument( "--archive-timeout", help="the time, in seconds, the archiver will wait for a new WAL file " "to be archived before timing out", metavar="TIMEOUT", default="30", type=check_non_negative, ), ], cmd_aliases=["switch-xlog"], ) def switch_wal(args): """ Execute the switch-wal command on the target server """ servers = get_server_list(args, skip_inactive=True) for name in sorted(servers): server = servers[name] # Skip the server (apply general rule) if not manage_server_command(server, name): continue with closing(server): server.switch_wal(args.force, args.archive, args.archive_timeout) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer_all, nargs="+", help="specifies the server names to check " "('all' will check all available servers)", ), argument( "--nagios", help="Nagios plugin compatible output", action="store_true" ), ] ) def check(args): """ Check if the server configuration is working. This command returns success if every checks pass, or failure if any of these fails """ if args.nagios: output.set_output_writer(output.NagiosOutputWriter()) servers = get_server_list(args) for name in sorted(servers): server = servers[name] # Validate the returned server if not manage_server_command( server, name, skip_inactive=False, skip_disabled=False, disabled_is_error=False, ): continue output.init("check", name, server.config.active, server.config.disabled) with closing(server): server.check() output.close_and_exit() @command( [ argument( "--show-config-source", help="Include the source file which provides the effective value " "for each configuration option", action="store_true", ) ], ) def diagnose(args=None): """ Diagnostic command (for support and problems detection purpose) """ # Get every server (both inactive and temporarily disabled) servers = get_server_list(on_error_stop=False, suppress_error=True) models = get_models_list() # errors list with duplicate paths between servers errors_list = barman.__config__.servers_msg_list barman.diagnose.exec_diagnose(servers, models, errors_list, args.show_config_source) output.close_and_exit() @command( [ argument( "--primary", help="execute the sync-info on the primary node (if set)", action="store_true", default=SUPPRESS, ), argument( "server_name", completer=server_completer, help="specifies the server name for the command", ), argument( "last_wal", help="specifies the name of the latest WAL read", nargs="?" ), argument( "last_position", nargs="?", type=check_positive, help="the last position read from xlog database (in bytes)", ), ] ) def sync_info(args): """ Output the internal synchronisation status. Used to sync_backup with a passive node """ server = get_server(args) try: # if called with --primary option if getattr(args, "primary", False): primary_info = server.primary_node_info(args.last_wal, args.last_position) output.info( json.dumps(primary_info, cls=BarmanEncoder, indent=4), log=False ) else: server.sync_status(args.last_wal, args.last_position) except SyncError as e: # Catch SyncError exceptions and output only the error message, # preventing from logging the stack trace output.error(e) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command", ), argument( "backup_id", help="specifies the backup ID to be copied on the passive node" ), ] ) def sync_backup(args): """ Command that synchronises a backup from a master to a passive node """ server = get_server(args) try: server.sync_backup(args.backup_id) except SyncError as e: # Catch SyncError exceptions and output only the error message, # preventing from logging the stack trace output.error(e) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command", ) ] ) def sync_wals(args): """ Command that synchronises WAL files from a master to a passive node """ server = get_server(args) try: server.sync_wals() except SyncError as e: # Catch SyncError exceptions and output only the error message, # preventing from logging the stack trace output.error(e) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command", ), argument( "backup_id", completer=backup_completer, help="specifies the backup ID" ), ], cmd_aliases=["show-backups"], ) def show_backup(args): """ This method shows a single backup information """ server = get_server(args) # Retrieves the backup backup_info = parse_backup_id(server, args) with closing(server): server.show_backup(backup_info) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command", ), argument( "backup_id", completer=backup_completer, help="specifies the backup ID" ), argument( "--target", choices=("standalone", "data", "wal", "full"), default="standalone", help=""" Possible values are: data (just the data files), standalone (base backup files, including required WAL files), wal (just WAL files between the beginning of base backup and the following one (if any) or the end of the log) and full (same as data + wal). Defaults to %(default)s""", ), ] ) def list_files(args): """ List all the files for a single backup """ server = get_server(args) # Retrieves the backup backup_info = parse_backup_id(server, args) try: for line in backup_info.get_list_of_files(args.target): output.info(line, log=False) except BadXlogSegmentName as e: output.error( "invalid xlog segment name %r\n" 'HINT: Please run "barman rebuild-xlogdb %s" ' "to solve this issue", force_str(e), server.config.name, ) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command", ), argument( "backup_id", completer=backup_completer, help="specifies the backup ID" ), ] ) def delete(args): """ Delete a backup """ server = get_server(args) # Retrieves the backup backup_id = parse_backup_id(server, args) with closing(server): if not server.delete_backup(backup_id): output.error( "Cannot delete backup (%s %s)" % (server.config.name, backup_id) ) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command", ), argument("wal_name", help="the WAL file to get"), argument( "--output-directory", "-o", help="put the retrieved WAL file in this directory with the original name", default=SUPPRESS, ), argument( "--partial", "-P", help="retrieve also partial WAL files (.partial)", action="store_true", dest="partial", default=False, ), argument( "--gzip", "-z", "-x", help="compress the output with gzip", action="store_const", const="gzip", dest="compression", default=SUPPRESS, ), argument( "--bzip2", "-j", help="compress the output with bzip2", action="store_const", const="bzip2", dest="compression", default=SUPPRESS, ), argument( "--keep-compression", help="do not decompress the output if compressed", action="store_true", dest="keep_compression", ), argument( "--peek", "-p", help="peek from the WAL archive up to 'SIZE' WAL files, starting " "from the requested one. 'SIZE' must be an integer >= 1. " "When invoked with this option, get-wal returns a list of " "zero to 'SIZE' WAL segment names, one per row.", metavar="SIZE", type=check_positive, default=SUPPRESS, ), argument( "--test", "-t", help="test both the connection and the configuration of the requested " "PostgreSQL server in Barman for WAL retrieval. With this option, " "the 'wal_name' mandatory argument is ignored.", action="store_true", default=SUPPRESS, ), ] ) def get_wal(args): """ Retrieve WAL_NAME file from SERVER_NAME archive. The content will be streamed on standard output unless the --output-directory option is specified. """ server = get_server(args, inactive_is_error=True) if getattr(args, "test", None): output.info( "Ready to retrieve WAL files from the server %s", server.config.name ) return # Retrieve optional arguments. If an argument is not specified, # the namespace doesn't contain it due to SUPPRESS default. # In that case we pick 'None' using getattr third argument. compression = getattr(args, "compression", None) keep_compression = getattr(args, "keep_compression", False) output_directory = getattr(args, "output_directory", None) peek = getattr(args, "peek", None) if compression and keep_compression: output.error( "argument `%s` not allowed with argument `keep-compression`" % compression ) output.close_and_exit() with closing(server): server.get_wal( args.wal_name, compression=compression, keep_compression=keep_compression, output_directory=output_directory, peek=peek, partial=args.partial, ) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command", ), argument( "--test", "-t", help="test both the connection and the configuration of the requested " "PostgreSQL server in Barman to make sure it is ready to receive " "WAL files.", action="store_true", default=SUPPRESS, ), ] ) def put_wal(args): """ Receive a WAL file from SERVER_NAME and securely store it in the incoming directory. The file will be read from standard input in tar format. """ server = get_server(args, inactive_is_error=True) if getattr(args, "test", None): output.info("Ready to accept WAL files for the server %s", server.config.name) return try: # Python 3.x stream = sys.stdin.buffer except AttributeError: # Python 2.x stream = sys.stdin with closing(server): server.put_wal(stream) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command", ) ] ) def archive_wal(args): """ Execute maintenance operations on WAL files for a given server. This command processes any incoming WAL files for the server and archives them along the catalogue. """ server = get_server(args) with closing(server): server.archive_wal() output.close_and_exit() @command( [ argument( "--stop", help="stop the receive-wal subprocess for the server", action="store_true", ), argument( "--reset", help="reset the status of receive-wal removing any status files", action="store_true", ), argument( "--create-slot", help="create the replication slot, if it does not exist", action="store_true", ), argument( "--drop-slot", help="drop the replication slot, if it exists", action="store_true", ), argument( "server_name", completer=server_completer, help="specifies the server name for the command", ), ] ) def receive_wal(args): """ Start a receive-wal process. The process uses the streaming protocol to receive WAL files from the PostgreSQL server. """ should_skip_inactive = not ( args.create_slot or args.drop_slot or args.stop or args.reset ) server = get_server(args, skip_inactive=should_skip_inactive, wal_streaming=True) if args.stop and args.reset: output.error("--stop and --reset options are not compatible") # If the caller requested to shutdown the receive-wal process deliver the # termination signal, otherwise attempt to start it elif args.stop: server.kill("receive-wal") elif args.create_slot: with closing(server): server.create_physical_repslot() elif args.drop_slot: with closing(server): server.drop_repslot() else: with closing(server): server.receive_wal(reset=args.reset) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command", ), argument( "backup_id", completer=backup_completer, help="specifies the backup ID" ), ] ) def check_backup(args): """ Make sure that all the required WAL files to check the consistency of a physical backup (that is, from the beginning to the end of the full backup) are correctly archived. This command is automatically invoked by the cron command and at the end of every backup operation. """ server = get_server(args) # Retrieves the backup backup_info = parse_backup_id(server, args) with closing(server): server.check_backup(backup_info) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command ", ), argument( "backup_id", completer=backup_completer, help="specifies the backup ID" ), ], cmd_aliases=["verify"], ) def verify_backup(args): """ verify a backup for the given server and backup id """ # get barman.server.Server server = get_server(args) # Raises an error if wrong backup backup_info = parse_backup_id(server, args) # get backup path output.info( "Verifying backup '%s' on server %s" % (args.backup_id, args.server_name) ) server.backup_manager.verify_backup(backup_info) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command ", ), argument( "backup_id", completer=backup_completer, help="specifies the backup ID" ), ], ) def generate_manifest(args): """ Generate a manifest-backup for the given server and backup id """ server = get_server(args) # Raises an error if wrong backup backup_info = parse_backup_id(server, args) # know context (remote backup? local?) local_file_manager = LocalFileManager() backup_manifest = BackupManifest( backup_info.get_data_directory(), local_file_manager, SHA256() ) backup_manifest.create_backup_manifest() output.info( "Backup manifest for backup '%s' successfully generated for server %s" % (args.backup_id, args.server_name) ) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command", ), argument( "backup_id", completer=backup_completer, help="specifies the backup ID" ), argument( "-r", "--release", help="remove the keep annotation", action="store_true" ), argument( "-s", "--status", help="return the keep status of the backup", action="store_true", ), argument( "--target", help="keep this backup with the specified recovery target", choices=[KeepManager.TARGET_FULL, KeepManager.TARGET_STANDALONE], ), ] ) def keep(args): """ Tag the specified backup so that it will never be deleted """ if not any((args.release, args.status, args.target)): output.error( "one of the arguments -r/--release -s/--status --target is required" ) output.close_and_exit() server = get_server(args) backup_info = parse_backup_id(server, args) backup_manager = server.backup_manager if args.status: output.init("status", server.config.name) target = backup_manager.get_keep_target(backup_info.backup_id) if target: output.result("status", server.config.name, "keep_status", "Keep", target) else: output.result("status", server.config.name, "keep_status", "Keep", "nokeep") elif args.release: backup_manager.release_keep(backup_info.backup_id) else: if backup_info.status != BackupInfo.DONE: msg = ( "Cannot add keep to backup %s because it has status %s. " "Only backups with status DONE can be kept." ) % (backup_info.backup_id, backup_info.status) output.error(msg) output.close_and_exit() if backup_info.is_incremental: msg = ( "Unable to execute the keep command on backup %s: is an incremental backup.\n" "Only full backups are eligible for the use of the keep command." ) % (backup_info.backup_id) output.error(msg) output.close_and_exit() backup_manager.keep_backup(backup_info.backup_id, args.target) @command( [ argument( "server_name", completer=server_completer, help="specifies the server name for the command", ), argument( "--timeline", help="the earliest timeline whose WALs should cause the check to fail", type=check_positive, ), ] ) def check_wal_archive(args): """ Check the WAL archive can be safely used for a new server. This will fail if there are any existing WALs in the archive. If the --timeline option is used then any WALs on earlier timelines than that specified will not cause the check to fail. """ server = get_server(args) output.init("check_wal_archive", server.config.name) with server.xlogdb() as fxlogdb: wals = [WalFileInfo.from_xlogdb_line(w).name for w in fxlogdb] try: check_archive_usable( wals, timeline=args.timeline, ) output.result("check_wal_archive", server.config.name) except WalArchiveContentError as err: msg = "WAL archive check failed for server %s: %s" % ( server.config.name, force_str(err), ) logging.error(msg) output.error(msg) output.close_and_exit() @command( [ argument( "server_name", completer=server_completer, help="specifies the name of the server which configuration should " "be overriden by the model", ), argument( "model_name", help="specifies the name of the model which configuration should " "override the server configuration. This is an optional argument " "and will not be used when called with the '--reset' flag.", nargs="?", ), argument( "--reset", help="indicates that we should unapply the currently active model " "for the server", action="store_true", ), ] ) def config_switch(args): """ Change the active configuration for a server by applying a named model on top of it, or by resetting the active model. """ if args.model_name is None and not args.reset: output.error("Either a model name or '--reset' flag need to be given") return server = get_server(args, skip_inactive=False) if server is not None: if args.reset: server.config.reset_model() else: model = get_model(args) if model is not None: server.config.apply_model(model, True) server.restart_processes() @command( [ argument( "json_changes", help="specifies the configuration changes to apply, in json format ", ), ] ) def config_update(args): """ Receives a set of configuration changes in json format and applies them. """ json_changes = json.loads(args.json_changes) # this prevents multiple concurrent executions of the config-update command with ConfigUpdateLock(barman.__config__.barman_lock_directory): processor = ConfigChangesProcessor(barman.__config__) processor.receive_config_changes(json_changes) processor.process_conf_changes_queue() for change in processor.applied_changes: server = get_server( argparse.Namespace(server_name=change.section), # skip_disabled=True, inactive_is_error=False, disabled_is_error=False, on_error_stop=False, suppress_error=True, ) if server: server.restart_processes() def pretty_args(args): """ Prettify the given argparse namespace to be human readable :type args: argparse.Namespace :return: the human readable content of the namespace """ values = dict(vars(args)) # Retrieve the command name with recent argh versions if "_functions_stack" in values: values["command"] = values["_functions_stack"][0].__name__ del values["_functions_stack"] # Older argh versions only have the matching function in the namespace elif "function" in values: values["command"] = values["function"].__name__ del values["function"] return "%r" % values def global_config(args): """ Set the configuration file """ if hasattr(args, "config"): filename = args.config else: try: filename = os.environ["BARMAN_CONFIG_FILE"] except KeyError: filename = None config = barman.config.Config(filename) barman.__config__ = config # change user if needed try: drop_privileges(config.user) except OSError: msg = "ERROR: please run barman as %r user" % config.user raise SystemExit(msg) except KeyError: msg = "ERROR: the configured user %r does not exists" % config.user raise SystemExit(msg) # configure logging if hasattr(args, "log_level"): config.log_level = args.log_level log_level = parse_log_level(config.log_level) configure_logging( config.log_file, log_level or barman.config.DEFAULT_LOG_LEVEL, config.log_format ) if log_level is None: _logger.warning("unknown log_level in config file: %s", config.log_level) # Configure output if args.format != output.DEFAULT_WRITER or args.quiet or args.debug: output.set_output_writer(args.format, quiet=args.quiet, debug=args.debug) # Configure color output if args.color == "auto": # Enable colored output if both stdout and stderr are TTYs output.ansi_colors_enabled = sys.stdout.isatty() and sys.stderr.isatty() else: output.ansi_colors_enabled = args.color == "always" # Load additional configuration files config.load_configuration_files_directory() # Handle the autoconf file, load it only if exists autoconf_path = "%s/.barman.auto.conf" % config.get("barman", "barman_home") if os.path.exists(autoconf_path): config.load_config_file(autoconf_path) # We must validate the configuration here in order to have # both output and logging configured config.validate_global_config() _logger.debug( "Initialised Barman version %s (config: %s, args: %s)", barman.__version__, config.config_file, pretty_args(args), ) def get_server( args, skip_inactive=True, skip_disabled=False, skip_passive=False, inactive_is_error=False, disabled_is_error=True, on_error_stop=True, suppress_error=False, wal_streaming=False, ): """ Get a single server retrieving its configuration (wraps get_server_list()) Returns a Server object or None if the required server is unknown and on_error_stop is False. WARNING: this function modifies the 'args' parameter :param args: an argparse namespace containing a single server_name parameter WARNING: the function modifies the content of this parameter :param bool skip_inactive: do nothing if the server is inactive :param bool skip_disabled: do nothing if the server is disabled :param bool skip_passive: do nothing if the server is passive :param bool inactive_is_error: treat inactive server as error :param bool on_error_stop: stop if an error is found :param bool suppress_error: suppress display of errors (e.g. diagnose) :param bool wal_streaming: create the :class:`barman.server.Server` using WAL streaming conninfo (if available in the configuration) :rtype: Server|None """ # This function must to be called with in a single-server context name = args.server_name assert isinstance(name, str) # The 'all' special name is forbidden in this context if name == "all": output.error("You cannot use 'all' in a single server context") output.close_and_exit() # The following return statement will never be reached # but it is here for clarity return None # Builds a list from a single given name args.server_name = [name] # Skip_inactive is reset if inactive_is_error is set, because # it needs to retrieve the inactive server to emit the error. skip_inactive &= not inactive_is_error # Retrieve the requested server servers = get_server_list( args, skip_inactive, skip_disabled, skip_passive, on_error_stop, suppress_error, wal_streaming, ) # The requested server has been excluded from get_server_list result if len(servers) == 0: output.close_and_exit() # The following return statement will never be reached # but it is here for clarity return None # retrieve the server object server = servers[name] # Apply standard validation control and skips # the server if inactive or disabled, displaying standard # error messages. If on_error_stop (default) exits x = not manage_server_command( server, name, inactive_is_error, disabled_is_error, skip_inactive, skip_disabled, suppress_error, ) if x and on_error_stop: output.close_and_exit() # The following return statement will never be reached # but it is here for clarity return None # Returns the filtered server return server def get_server_list( args=None, skip_inactive=False, skip_disabled=False, skip_passive=False, on_error_stop=True, suppress_error=False, wal_streaming=False, ): """ Get the server list from the configuration If args the parameter is None or arg.server_name is ['all'] returns all defined servers :param args: an argparse namespace containing a list server_name parameter :param bool skip_inactive: skip inactive servers when 'all' is required :param bool skip_disabled: skip disabled servers when 'all' is required :param bool skip_passive: skip passive servers when 'all' is required :param bool on_error_stop: stop if an error is found :param bool suppress_error: suppress display of errors (e.g. diagnose) :param bool wal_streaming: create :class:`barman.server.Server` objects using WAL streaming conninfo (if available in the configuration) :rtype: dict[str,Server] """ server_dict = {} # This function must to be called with in a multiple-server context assert not args or isinstance(args.server_name, list) # Generate the list of servers (required for global errors) available_servers = barman.__config__.server_names() # Get a list of configuration errors from all the servers global_error_list = barman.__config__.servers_msg_list # Global errors have higher priority if global_error_list: # Output the list of global errors if not suppress_error: for error in global_error_list: output.error(error) # If requested, exit on first error if on_error_stop: output.close_and_exit() # The following return statement will never be reached # but it is here for clarity return {} # Handle special 'all' server cases # - args is None # - 'all' special name if not args or "all" in args.server_name: # When 'all' is used, it must be the only specified argument if args and len(args.server_name) != 1: output.error("You cannot use 'all' with other server names") server_names = available_servers else: # Put servers in a set, so multiple occurrences are counted only once server_names = set(args.server_name) # Loop through all the requested servers for server_name in server_names: conf = barman.__config__.get_server(server_name) if conf is None: # Unknown server server_dict[server_name] = None else: if wal_streaming: conf.streaming_conninfo, conf.conninfo = conf.get_wal_conninfo() server_object = Server(conf) # Skip inactive servers, if requested if skip_inactive and not server_object.config.active: output.info("Skipping inactive server '%s'" % conf.name) continue # Skip disabled servers, if requested if skip_disabled and server_object.config.disabled: output.info("Skipping temporarily disabled server '%s'" % conf.name) continue # Skip passive nodes, if requested if skip_passive and server_object.passive_node: output.info("Skipping passive server '%s'", conf.name) continue server_dict[server_name] = server_object return server_dict def manage_server_command( server, name=None, inactive_is_error=False, disabled_is_error=True, skip_inactive=True, skip_disabled=True, suppress_error=False, ): """ Standard and consistent method for managing server errors within a server command execution. By default, suggests to skip any inactive and disabled server; it also emits errors for disabled servers by default. Returns True if the command has to be executed for this server. :param barman.server.Server server: server to be checked for errors :param str name: name of the server, in a multi-server command :param bool inactive_is_error: treat inactive server as error :param bool disabled_is_error: treat disabled server as error :param bool skip_inactive: skip if inactive :param bool skip_disabled: skip if disabled :return: True if the command has to be executed on this server :rtype: boolean """ # Unknown server (skip it) if not server: if not suppress_error: output.error("Unknown server '%s'" % name) return False if not server.config.active: # Report inactive server as error if inactive_is_error: output.error("Inactive server: %s" % server.config.name) return False if skip_inactive: return False # Report disabled server as error if server.config.disabled: # Output all the messages as errors, and exit terminating the run. if disabled_is_error: for message in server.config.msg_list: output.error(message) return False if skip_disabled: return False # All ok, execute the command return True def get_models_list(args=None): """Get the model list from the configuration. If the *args* parameter is ``None`` returns all defined servers. :param args: an :class:`argparse.Namespace` containing a list ``model_name`` parameter. :return: a :class:`dict` -- each key is a model name, and its value the corresponding :class:`ModelConfig` instance. """ model_dict = {} # This function must to be called with in a multiple-model context assert not args or isinstance(args.model_name, list) # Generate the list of models (required for global errors) available_models = barman.__config__.model_names() # Handle special *args* is ``None`` case if not args: model_names = available_models else: # Put models in a set, so multiple occurrences are counted only once model_names = set(args.model_name) # Loop through all the requested models for model_name in model_names: model = barman.__config__.get_model(model_name) if model is None: # Unknown model model_dict[model_name] = None else: model_dict[model_name] = model return model_dict def manage_model_command(model, name=None): """ Standard and consistent method for managing model errors within a model command execution. :param model: :class:`ModelConfig` to be checked for errors. :param name: name of the model. :return: ``True`` if the command has to be executed with this model. """ # Unknown model (skip it) if not model: output.error("Unknown model '%s'" % name) return False # All ok, execute the command return True def get_model(args, on_error_stop=True): """ Get a single model retrieving its configuration (wraps :func:`get_models_list`). .. warning:: This function modifies the *args* parameter. :param args: an :class:`argparse.Namespace` containing a single ``model_name`` parameter. :param on_error_stop: stop if an error is found. :return: a :class:`ModelConfig` or ``None`` if the required model is unknown and *on_error_stop* is ``False``. """ # This function must to be called with in a single-model context name = args.model_name assert isinstance(name, str) # Builds a list from a single given name args.model_name = [name] # Retrieve the requested model models = get_models_list(args) # The requested model has been excluded from :func:`get_models_list`` result if len(models) == 0: output.close_and_exit() # The following return statement will never be reached # but it is here for clarity return None # retrieve the model object model = models[name] # Apply standard validation control and skips # the model if invalid, displaying standard # error messages. If on_error_stop (default) exits if not manage_model_command(model, name) and on_error_stop: output.close_and_exit() # The following return statement will never be reached # but it is here for clarity return None # Returns the filtered model return model def parse_backup_id(server, args): """ Parses backup IDs including special words such as latest, oldest, etc. Exit with error if the backup id doesn't exist. :param Server server: server object to search for the required backup :param args: command line arguments namespace :rtype: barman.infofile.LocalBackupInfo """ backup_id = get_backup_id_using_shortcut(server, args.backup_id, BackupInfo) if backup_id is None: try: backup_id = server.get_backup_id_from_name(args.backup_id) except ValueError as exc: output.error(str(exc)) output.close_and_exit() backup_info = server.get_backup(backup_id) if backup_info is None: output.error( "Unknown backup '%s' for server '%s'", args.backup_id, server.config.name ) output.close_and_exit() return backup_info def main(): """ The main method of Barman """ # noinspection PyBroadException try: if argcomplete: argcomplete.autocomplete(p) args = p.parse_args() global_config(args) if args.command is None: p.print_help() else: args.func(args) except KeyboardInterrupt: msg = "Process interrupted by user (KeyboardInterrupt)" output.error(msg) except Exception as e: msg = "%s\nSee log file for more details." % e output.exception(msg) # cleanup output API and exit honoring output.error_occurred and # output.error_exit_code output.close_and_exit() if __name__ == "__main__": # This code requires the mock module and allow us to test # bash completion inside the IDE debugger try: # noinspection PyUnresolvedReferences import mock sys.stdout = mock.Mock(wraps=sys.stdout) sys.stdout.isatty.return_value = True os.dup2(2, 8) except ImportError: pass main() barman-3.13.3/barman/infofile.py0000644000175100001660000012443615002177425014644 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2013-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import ast import collections import inspect import logging import os import re import dateutil.parser import dateutil.tz from barman import xlog from barman.cloud_providers import snapshots_info_from_dict from barman.exceptions import BackupInfoBadInitialisation from barman.utils import fsync_dir # Named tuple representing a Tablespace with 'name' 'oid' and 'location' # as property. Tablespace = collections.namedtuple("Tablespace", "name oid location") # Named tuple representing a file 'path' with an associated 'file_type' TypedFile = collections.namedtuple("ConfFile", "file_type path") def output_snapshots_info(snapshots_info): return null_repr(snapshots_info.to_dict()) def load_snapshots_info(string): obj = ast.literal_eval(string) return snapshots_info_from_dict(obj) _logger = logging.getLogger(__name__) def output_tablespace_list(tablespaces): """ Return the literal representation of tablespaces as a Python string :param tablespaces tablespaces: list of Tablespaces objects :return str: Literal representation of tablespaces """ if tablespaces: return repr([tuple(item) for item in tablespaces]) else: return None def load_tablespace_list(string): """ Load the tablespaces as a Python list of namedtuple Uses ast to evaluate information about tablespaces. The returned list is used to create a list of namedtuple :param str string: :return list: list of namedtuple representing all the tablespaces """ obj = ast.literal_eval(string) if obj: return [Tablespace._make(item) for item in obj] else: return None def null_repr(obj): """ Return the literal representation of an object :param object obj: object to represent :return str|None: Literal representation of an object or None """ return repr(obj) if obj else None def load_datetime_tz(time_str): """ Load datetime and ensure the result is timezone-aware. If the parsed timestamp is naive, transform it into a timezone-aware one using the local timezone. :param str time_str: string representing a timestamp :return datetime: the parsed timezone-aware datetime """ # dateutil parser returns naive or tz-aware string depending on the format # of the input string timestamp = dateutil.parser.parse(time_str) # if the parsed timestamp is naive, forces it to local timezone if timestamp.tzinfo is None: timestamp = timestamp.replace(tzinfo=dateutil.tz.tzlocal()) return timestamp def dump_backup_ids(ids): """ Dump a list of backup IDs to disk as a string. :param list[str]|None ids: list of backup IDs, if any :return str|None: the dumped string. """ if ids: return ",".join(ids) else: return None def load_backup_ids(string): """ Load a list of backup IDs from disk as a :class:`list`. :param string: the string to be loaded as a list. :return list[str]|None: the list of backup IDs, if any. """ if string: return string.split(",") else: return None class Field(object): def __init__(self, name, dump=None, load=None, default=None, doc=None): """ Field descriptor to be used with a FieldListFile subclass. The resulting field is like a normal attribute with two optional associated function: to_str and from_str The Field descriptor can also be used as a decorator class C(FieldListFile): x = Field('x') @x.dump def x(val): return '0x%x' % val @x.load def x(val): return int(val, 16) :param str name: the name of this attribute :param callable dump: function used to dump the content to a disk :param callable load: function used to reload the content from disk :param default: default value for the field :param str doc: docstring of the filed """ self.name = name self.to_str = dump self.from_str = load self.default = default self.__doc__ = doc # noinspection PyUnusedLocal def __get__(self, obj, objtype=None): if obj is None: return self if not hasattr(obj, "_fields"): obj._fields = {} return obj._fields.setdefault(self.name, self.default) def __set__(self, obj, value): if not hasattr(obj, "_fields"): obj._fields = {} obj._fields[self.name] = value def __delete__(self, obj): raise AttributeError("can't delete attribute") def dump(self, to_str): return type(self)(self.name, to_str, self.from_str, self.__doc__) def load(self, from_str): return type(self)(self.name, self.to_str, from_str, self.__doc__) class FieldListFile(object): __slots__ = ("_fields", "filename") # A list of fields which should be hidden if they are not set. # Such fields will not be written to backup.info files or included in the # backup.info items unles they are set to a non-None value. # Any fields listed here should be removed from the list at the next major # version increase. _hide_if_null = () def __init__(self, **kwargs): """ Represent a predefined set of keys with the associated value. The constructor build the object assigning every keyword argument to the corresponding attribute. If a provided keyword argument doesn't has a corresponding attribute an AttributeError exception is raised. The values provided to the constructor must be of the appropriate type for the corresponding attribute. The constructor will not attempt any validation or conversion on them. This class is meant to be an abstract base class. :raises: AttributeError """ self._fields = {} self.filename = None for name in kwargs: field = getattr(type(self), name, None) if isinstance(field, Field): setattr(self, name, kwargs[name]) else: raise AttributeError("unknown attribute %s" % name) @classmethod def from_meta_file(cls, filename): """ Factory method that read the specified file and build an object with its content. :param str filename: the file to read """ o = cls() o.load(filename) return o def save(self, filename=None, file_object=None): """ Serialize the object to the specified file or file object If a file_object is specified it will be used. If the filename is not specified it uses the one memorized in the filename attribute. If neither the filename attribute and parameter are set a ValueError exception is raised. :param str filename: path of the file to write :param file file_object: a file like object to write in :param str filename: the file to write :raises: ValueError """ if file_object: info = file_object else: filename = filename or self.filename if filename: info = open(filename + ".tmp", "wb") else: info = None if not info: raise ValueError( "either a valid filename or a file_object must be specified" ) try: for name, field in sorted(inspect.getmembers(type(self))): value = getattr(self, name, None) if value is None and name in self._hide_if_null: continue if isinstance(field, Field): if callable(field.to_str): value = field.to_str(value) info.write(("%s=%s\n" % (name, value)).encode("UTF-8")) finally: if not file_object: info.close() if not file_object: os.rename(filename + ".tmp", filename) fsync_dir(os.path.normpath(os.path.dirname(filename))) def load(self, filename=None, file_object=None): """ Replaces the current object content with the one deserialized from the provided file. This method set the filename attribute. A ValueError exception is raised if the provided file contains any invalid line. :param str filename: path of the file to read :param file file_object: a file like object to read from :param str filename: the file to read :raises: ValueError """ if file_object: info = file_object elif filename: info = open(filename, "rb") else: raise ValueError("either filename or file_object must be specified") # detect the filename if a file_object is passed if not filename and file_object: if hasattr(file_object, "name"): filename = file_object.name # canonicalize filename if filename: self.filename = os.path.abspath(filename) else: self.filename = None filename = "" # This is only for error reporting with info: for line in info: line = line.decode("UTF-8") # skip spaces and comments if line.isspace() or line.rstrip().startswith("#"): continue # parse the line of form "key = value" try: name, value = [x.strip() for x in line.split("=", 1)] except ValueError: raise ValueError( "invalid line %s in file %s" % (line.strip(), filename) ) # use the from_str function to parse the value field = getattr(type(self), name, None) if value == "None": value = None elif isinstance(field, Field) and callable(field.from_str): value = field.from_str(value) setattr(self, name, value) def items(self): """ Return a generator returning a list of (key, value) pairs. If a filed has a dump function defined, it will be used. """ for name, field in sorted(inspect.getmembers(type(self))): value = getattr(self, name, None) if value is None and name in self._hide_if_null: continue if isinstance(field, Field): if callable(field.to_str): value = field.to_str(value) yield (name, value) def __repr__(self): return "%s(%s)" % ( self.__class__.__name__, ", ".join(["%s=%r" % x for x in self.items()]), ) class WalFileInfo(FieldListFile): """ Metadata of a WAL file. """ __slots__ = ("orig_filename",) name = Field("name", doc="base name of WAL file") size = Field("size", load=int, doc="WAL file size after compression") time = Field( "time", load=float, doc="WAL file modification time (seconds since epoch)" ) compression = Field("compression", doc="compression type") @classmethod def from_file( cls, filename, compression_manager=None, unidentified_compression=None, **kwargs ): """ Factory method to generate a WalFileInfo from a WAL file. Every keyword argument will override any attribute from the provided file. If a keyword argument doesn't has a corresponding attribute an AttributeError exception is raised. :param str filename: the file to inspect :param Compressionmanager compression_manager: a compression manager which will be used to identify the compression :param str unidentified_compression: the compression to set if the current schema is not identifiable """ stat = os.stat(filename) kwargs.setdefault("name", os.path.basename(filename)) kwargs.setdefault("size", stat.st_size) kwargs.setdefault("time", stat.st_mtime) if "compression" not in kwargs: kwargs["compression"] = ( compression_manager.identify_compression(filename) or unidentified_compression ) obj = cls(**kwargs) obj.filename = "%s.meta" % filename obj.orig_filename = filename return obj def to_xlogdb_line(self): """ Format the content of this object as a xlogdb line. """ return "%s\t%s\t%s\t%s\n" % (self.name, self.size, self.time, self.compression) @classmethod def from_xlogdb_line(cls, line): """ Parse a line from xlog catalogue :param str line: a line in the wal database to parse :rtype: WalFileInfo """ try: name, size, time, compression = line.split() except ValueError: # Old format compatibility (no compression) compression = None try: name, size, time = line.split() except ValueError: raise ValueError("cannot parse line: %r" % (line,)) # The to_xlogdb_line method writes None values as literal 'None' if compression == "None": compression = None size = int(size) time = float(time) return cls(name=name, size=size, time=time, compression=compression) def to_json(self): """ Return an equivalent dictionary that can be encoded in json """ return dict(self.items()) def relpath(self): """ Returns the WAL file path relative to the server's wals_directory """ return os.path.join(xlog.hash_dir(self.name), self.name) def fullpath(self, server): """ Returns the WAL file full path :param barman.server.Server server: the server that owns the wal file """ return os.path.join(server.config.wals_directory, self.relpath()) class BackupInfo(FieldListFile): #: Conversion to string EMPTY = "EMPTY" STARTED = "STARTED" FAILED = "FAILED" WAITING_FOR_WALS = "WAITING_FOR_WALS" DONE = "DONE" SYNCING = "SYNCING" STATUS_COPY_DONE = (WAITING_FOR_WALS, DONE) STATUS_ALL = (EMPTY, STARTED, WAITING_FOR_WALS, DONE, SYNCING, FAILED) STATUS_NOT_EMPTY = (STARTED, WAITING_FOR_WALS, DONE, SYNCING, FAILED) STATUS_ARCHIVING = (STARTED, WAITING_FOR_WALS, DONE, SYNCING) #: Status according to retention policies OBSOLETE = "OBSOLETE" VALID = "VALID" POTENTIALLY_OBSOLETE = "OBSOLETE*" NONE = "-" KEEP_FULL = "KEEP:FULL" KEEP_STANDALONE = "KEEP:STANDALONE" RETENTION_STATUS = ( OBSOLETE, VALID, POTENTIALLY_OBSOLETE, KEEP_FULL, KEEP_STANDALONE, NONE, ) version = Field("version", load=int) pgdata = Field("pgdata") # Parse the tablespaces as a literal Python list of namedtuple # Output the tablespaces as a literal Python list of tuple tablespaces = Field( "tablespaces", load=load_tablespace_list, dump=output_tablespace_list ) # Timeline is an integer timeline = Field("timeline", load=int) begin_time = Field("begin_time", load=load_datetime_tz) begin_xlog = Field("begin_xlog") begin_wal = Field("begin_wal") begin_offset = Field("begin_offset", load=int) size = Field("size", load=int) deduplicated_size = Field("deduplicated_size", load=int) end_time = Field("end_time", load=load_datetime_tz) end_xlog = Field("end_xlog") end_wal = Field("end_wal") end_offset = Field("end_offset", load=int) status = Field("status", default=EMPTY) server_name = Field("server_name") error = Field("error") mode = Field("mode") config_file = Field("config_file") hba_file = Field("hba_file") ident_file = Field("ident_file") included_files = Field("included_files", load=ast.literal_eval, dump=null_repr) backup_label = Field("backup_label", load=ast.literal_eval, dump=null_repr) copy_stats = Field("copy_stats", load=ast.literal_eval, dump=null_repr) xlog_segment_size = Field( "xlog_segment_size", load=int, default=xlog.DEFAULT_XLOG_SEG_SIZE ) systemid = Field("systemid") compression = Field("compression") backup_name = Field("backup_name") snapshots_info = Field( "snapshots_info", load=load_snapshots_info, dump=output_snapshots_info ) data_checksums = Field("data_checksums") summarize_wal = Field("summarize_wal") parent_backup_id = Field("parent_backup_id") children_backup_ids = Field( "children_backup_ids", dump=dump_backup_ids, load=load_backup_ids, ) cluster_size = Field("cluster_size", load=int) __slots__ = "backup_id", "backup_version" _hide_if_null = ("backup_name", "snapshots_info") def __init__(self, backup_id, **kwargs): """ Stores meta information about a single backup :param str,None backup_id: """ self.backup_version = 2 self.backup_id = backup_id super(BackupInfo, self).__init__(**kwargs) def get_required_wal_segments(self): """ Get the list of required WAL segments for the current backup """ return xlog.generate_segment_names( self.begin_wal, self.end_wal, self.version, self.xlog_segment_size ) def get_external_config_files(self): """ Identify all the configuration files that reside outside the PGDATA. Returns a list of TypedFile objects. :rtype: list[TypedFile] """ config_files = [] for file_type in ("config_file", "hba_file", "ident_file"): config_file = getattr(self, file_type, None) if config_file: # Consider only those that reside outside of the original # PGDATA directory if config_file.startswith(self.pgdata): _logger.debug( "Config file '%s' already in PGDATA", config_file[len(self.pgdata) + 1 :], ) continue config_files.append(TypedFile(file_type, config_file)) # Check for any include directives in PostgreSQL configuration # Currently, include directives are not supported for files that # reside outside PGDATA. These files must be manually backed up. # Barman will emit a warning and list those files if self.included_files: for included_file in self.included_files: if not included_file.startswith(self.pgdata): config_files.append(TypedFile("include", included_file)) return config_files def set_attribute(self, key, value): """ Set a value for a given key """ setattr(self, key, value) @property def is_incremental(self): """ Only checks if the backup_info is an incremental backup .. note:: This property only makes sense in the context of local backups stored in the Barman server. However, this property is used for retention policies processing, code which is shared among local and cloud backups. As this property always returns ``False`` for cloud backups, it can safely be reused in their code paths as well, though. :return bool: ``True`` if this backup has a parent, ``False`` otherwise. """ return self.parent_backup_id is not None @property def has_children(self): """ Only checks if the backup_info has children .. note:: This property only makes sense in the context of local backups stored in the Barman server. However, this property is used for retention policies processing, code which is shared among local and cloud backups. As this property always returns ``False`` for cloud backups, it can safely be reused in their code paths as well, though. :return bool: ``True`` if this backup has at least one child, ``False`` otherwise. """ return self.children_backup_ids is not None @property def backup_type(self): """ Returns a string with the backup type label. .. note:: Even though this property is available in this base class, it is not expected to be used in the context of cloud backups. The backup type can be one of the following: - ``snapshot``: If the backup mode starts with ``snapshot``. - ``rsync``: If the backup mode starts with ``rsync``. - ``incremental``: If the mode is ``postgres`` and the backup is incremental. - ``full``: If the mode is ``postgres`` and the backup is not incremental. :return str: The backup type label. """ if self.mode.startswith("snapshot"): return "snapshot" elif self.mode.startswith("rsync"): return "rsync" return "incremental" if self.is_incremental else "full" @property def deduplication_ratio(self): """ Returns a value between and including ``0`` and ``1`` related to the estimate deduplication ratio of the backup. .. note:: For ``rsync`` backups, the :attr:`size` of the backup, which is the sum of all file sizes in basebackup directory, is used to calculate the ratio. For ``postgres`` backups, the :attr:`cluster_size` is used, which contains the estimated size of the Postgres cluster at backup time. We perform this calculation to make an estimation of how much network and disk I/O has been saved when taking an incremental backup through ``rsync`` or through ``pg_basebackup``. We abuse of the term "deduplication" here. It makes more sense to ``rsync`` than to ``postgres`` method. However, the idea is the same in both cases: get an estimation of resources saving. .. note:: Even though this property is available in this base class, it is not expected to be used in the context of cloud backups. :return float: The backup deduplication ratio. """ size = self.cluster_size if self.backup_type == "rsync": size = self.size if size and self.deduplicated_size: return 1 - (self.deduplicated_size / size) return 0 def to_dict(self): """ Return the backup_info content as a simple dictionary :return dict: """ result = dict(self.items()) top_level_fields = ( "backup_id", "server_name", "mode", "tablespaces", "included_files", "copy_stats", "snapshots_info", ) for field_name in top_level_fields: field_value = getattr(self, field_name) if field_value is not None or field_name not in self._hide_if_null: result.update({field_name: field_value}) if self.snapshots_info is not None: result.update({"snapshots_info": self.snapshots_info.to_dict()}) return result def to_json(self): """ Return an equivalent dictionary that uses only json-supported types """ data = self.to_dict() # Convert fields which need special types not supported by json if data.get("tablespaces") is not None: data["tablespaces"] = [list(item) for item in data["tablespaces"]] # Note on the `begin_time_iso`` and `end_time_iso`` fields: # as ctime is not timezone-aware and mostly for human-readable output, # we want to migrate to isoformat for datetime objects representation. # To retain, for now, compatibility with the previous version of the output # we add two new _iso fields to the json document. if data.get("begin_time") is not None: begin_time = data["begin_time"] data["begin_time"] = begin_time.ctime() data["begin_time_iso"] = begin_time.isoformat() if data.get("end_time") is not None: end_time = data["end_time"] data["end_time"] = end_time.ctime() data["end_time_iso"] = end_time.isoformat() return data @classmethod def from_json(cls, server, json_backup_info): """ Factory method that builds a BackupInfo object from a json dictionary :param barman.Server server: the server related to the Backup :param dict json_backup_info: the data set containing values from json """ data = dict(json_backup_info) # Convert fields which need special types not supported by json if data.get("tablespaces") is not None: data["tablespaces"] = [ Tablespace._make(item) for item in data["tablespaces"] ] if data.get("begin_time_iso") is not None: data["begin_time"] = load_datetime_tz(data["begin_time_iso"]) del data["begin_time_iso"] elif data.get("begin_time") is not None: data["begin_time"] = load_datetime_tz(data["begin_time"]) if data.get("end_time_iso") is not None: data["end_time"] = load_datetime_tz(data["end_time_iso"]) del data["end_time_iso"] elif data.get("end_time") is not None: data["end_time"] = load_datetime_tz(data["end_time"]) # Instantiate a BackupInfo object using the converted fields return cls(server, **data) def pg_major_version(self): """ Returns the major version of the PostgreSQL instance from which the backup was made taking into account the change in versioning scheme between PostgreSQL < 10.0 and PostgreSQL >= 10.0. """ major = int(self.version / 10000) if major < 10: minor = int(self.version / 100 % 100) return "%d.%d" % (major, minor) else: return str(major) def wal_directory(self): """ Returns "pg_wal" (v10 and above) or "pg_xlog" (v9.6 and below) based on the Postgres version represented by this backup """ return "pg_wal" if self.version >= 100000 else "pg_xlog" class LocalBackupInfo(BackupInfo): __slots__ = "server", "config", "backup_manager" def __init__(self, server, info_file=None, backup_id=None, **kwargs): """ Stores meta information about a single backup :param Server server: :param file,str,None info_file: :param str,None backup_id: :raise BackupInfoBadInitialisation: if the info_file content is invalid or neither backup_info or """ # Initialises the attributes for the object # based on the predefined keys super(LocalBackupInfo, self).__init__(backup_id=backup_id, **kwargs) self.server = server self.config = server.config self.backup_manager = self.server.backup_manager self.server_name = self.config.name self.mode = self.backup_manager.mode if backup_id: # Cannot pass both info_file and backup_id if info_file: raise BackupInfoBadInitialisation( "both info_file and backup_id parameters are set" ) self.backup_id = backup_id self.filename = self.get_filename() # Check if a backup info file for a given server and a given ID # already exists. If so load the values from the file. if os.path.exists(self.filename): self.load(filename=self.filename) elif info_file: if hasattr(info_file, "read"): # We have been given a file-like object self.load(file_object=info_file) else: # Just a file name self.load(filename=info_file) self.backup_id = self.detect_backup_id() elif not info_file: raise BackupInfoBadInitialisation( "backup_id and info_file parameters are both unset" ) # Manage backup version for new backup structure try: # the presence of pgdata directory is the marker of version 1 if self.backup_id is not None and os.path.exists( os.path.join(self.get_basebackup_directory(), "pgdata") ): self.backup_version = 1 except Exception as e: _logger.warning( "Error detecting backup_version, use default: 2. Failure reason: %s", e, ) def get_list_of_files(self, target): """ Get the list of files for the current backup """ # Walk down the base backup directory if target in ("data", "standalone", "full"): for root, _, files in os.walk(self.get_basebackup_directory()): files.sort() for f in files: yield os.path.join(root, f) if target in "standalone": # List all the WAL files for this backup for x in self.get_required_wal_segments(): yield self.server.get_wal_full_path(x) if target in ("wal", "full"): for wal_info in self.server.get_wal_until_next_backup( self, include_history=True ): yield wal_info.fullpath(self.server) def detect_backup_id(self): """ Detect the backup ID from the file name or parent directory name. .. note:: The ``backup.info`` file was relocated and renamed in version 3.13.2 to also contain the backup id as a prefix. In previous versions, the parent directory of the file was named with the backup id. This method handles both cases. """ if self.filename: match = re.match(r"^(.+?)-backup\.info$", os.path.basename(self.filename)) if match: return match.group(1) return os.path.basename(os.path.dirname(self.filename)) def get_basebackup_directory(self): """ Get the default filename for the backup.info file based on backup ID and server directory for base backups """ return os.path.join(self.config.basebackups_directory, self.backup_id) def get_data_directory(self, tablespace_oid=None): """ Get path to the backup data dir according with the backup version If tablespace_oid is passed, build the path to the tablespace base directory, according with the backup version :param int tablespace_oid: the oid of a valid tablespace """ # Check if a tablespace oid is passed and if is a valid oid if tablespace_oid is not None: if self.tablespaces is None: raise ValueError("Invalid tablespace OID %s" % tablespace_oid) invalid_oid = all( str(tablespace_oid) != str(tablespace.oid) for tablespace in self.tablespaces ) if invalid_oid: raise ValueError("Invalid tablespace OID %s" % tablespace_oid) # Build the requested path according to backup_version value path = [self.get_basebackup_directory()] # Check the version of the backup if self.backup_version == 2: # If an oid has been provided, we are looking for a tablespace if tablespace_oid is not None: # Append the oid to the basedir of the backup path.append(str(tablespace_oid)) else: # Looking for the data dir path.append("data") else: # Backup v1, use pgdata as base path.append("pgdata") # If a oid has been provided, we are looking for a tablespace. if tablespace_oid is not None: # Append the path to pg_tblspc/oid folder inside pgdata path.extend(("pg_tblspc", str(tablespace_oid))) # Return the built path return os.path.join(*path) def get_filename(self, write=False): """ Get the default file path for the backup.info file. :param bool write: if the file is to be written or not. .. note:: The ``backup.info`` file was relocated and renamed in version 3.13.2 to also contain the backup id as a prefix. In previous versions, it lived inside the backup directory alongside the data directory. This method handles both paths. When writing, always write to the new path. When reading, it could be that a user just migrated to the new version, so we also handle reading from the old path. """ path = os.path.join(self.server.meta_directory, f"{self.backup_id}-backup.info") old_path = os.path.join(self.get_basebackup_directory(), "backup.info") if write or os.path.exists(path) or not os.path.exists(old_path): return path return old_path def save(self, filename=None, file_object=None): # Update the filename before saving to ensure we're using the correct value # It could be that a user just migrated to version 3.13.2, which relocated the # backup.info path. In this case, the file was read from the old path but has # to be written to the new path if not filename: self.filename = self.get_filename(write=True) if not file_object: # Make sure the containing directory exists filename = filename or self.filename dir_name = os.path.dirname(filename) if not os.path.exists(dir_name): os.makedirs(dir_name) super(LocalBackupInfo, self).save(filename=filename, file_object=file_object) def get_backup_manifest_path(self): """ Get the full path to the backup manifest file :return str: the full path to the backup manifest file. """ return os.path.join(self.get_data_directory(), "backup_manifest") def get_parent_backup_info(self): """ If the backup is incremental, build the :class:`LocalBackupInfo` object for the parent backup and return it. If the backup is not incremental OR the status of the parent backup is ``EMPTY``, return ``None``. :return LocalBackupInfo|None: the parent backup info object, or None if it does not exist or is empty. """ if self.is_incremental: backup_info = LocalBackupInfo( self.server, backup_id=self.parent_backup_id, ) if backup_info.status != BackupInfo.EMPTY: return backup_info return None def get_child_backup_info(self, child_backup_id): """ Allow to retrieve a specific child of the current incremental backup, if there are any. If the child backup exists, the LocalBackupInfo object for it is returned. If does not exist or its status is `EMPTY`, return None :param str child_backup_id: the ID of the child backup to retrieve :return LocalBackupInfo|None: the child backup info object, or None if it does not exist or is empty. """ if self.children_backup_ids: if child_backup_id in self.children_backup_ids: backup_info = LocalBackupInfo( self.server, backup_id=child_backup_id, ) if backup_info.status != BackupInfo.EMPTY: return backup_info return None def walk_backups_tree(self, return_self=True): """ Walk through all the children backups of the current backup. .. note:: The objects are returned with a bottom-up approach, including all children backups plus the caller backup. :param bool return_self: Whether to return the current backup. Default to ``True``. :yields: a generator of :class:`LocalBackupInfo` objects for each backup, walking from the leaves to self. """ if self.children_backup_ids: for child_backup_id in self.children_backup_ids: backup_info = LocalBackupInfo( self.server, backup_id=child_backup_id, ) yield from backup_info.walk_backups_tree() if not return_self: return yield self def walk_to_root(self, return_self=True): """ Walk through all the parent backups of the current backup. .. note:: The objects are returned with a bottom-up approach, including all parents backups plus the caller backup if *return_self* is ``True``. :param bool return_self: Whether to return the current backup. Default to ``True``. :yield: a generator of :class:`LocalBackupInfo` objects for each parent backup. """ if return_self: yield self backup_info = self.get_parent_backup_info() while backup_info: yield backup_info backup_info = backup_info.get_parent_backup_info() def is_checksum_consistent(self): """ Check if all backups in the chain are consistent with their checksums configurations. The backup chain is considered inconsistent if the current backup was taken with ``data_checksums`` enabled and any of its ascendants were taken with it disabled. It is considered consistent otherwise. .. note:: While this method was created to check inconsistencies in chains of one or more (Postgres 17+ core) incremental backups, it can be safely used with any Postgres version and with any Barman backup method. That is true because it always returns ``True`` when called for a Postgres full backup or for a rsync backup. :return bool: ``True`` if it is consistent, ``False`` otherwise. """ if self.data_checksums != "on" or not self.is_incremental: return True for backup in self.walk_to_root(return_self=False): if backup.data_checksums == "off": return False return True @property def is_full(self): """ Check if this is a full backup. :return bool: ``True`` if it's a full backup or ``False`` if not. """ return self.backup_type not in ("snapshot", "incremental") @property def is_orphan(self): """ Determine if the backup is an orphan. An orphan backup is defined as a backup directory that contains only a non-empty backup.info file. This may indicate an incomplete delete operation. :return bool: ``True`` if the backup is an orphan, ``False`` otherwise. .. note:: The ``backup.info`` file was relocated and renamed in version 3.13.2 to also contain the backup id as a prefix. In previous versions, the parent directory of the file was named with the backup id. This method handles both cases. """ if self.status == BackupInfo.EMPTY: return False backup_dir = self.get_basebackup_directory() # In the >= 3.13.2 structure, it is considered orphan if the backup.info exists # in the server meta directory while no directory related to the backup exists backup_info_path = self.get_filename() if os.path.exists(backup_info_path) and not os.path.exists(backup_dir): return True # In the < 3.13.2 structure, the backup.info lived inside the backup directory. # In this case, it is considered orphan if the backup.info exists alone in there old_backup_info_path = os.path.join(backup_dir, "backup.info") if os.path.exists(backup_dir) and os.path.exists(old_backup_info_path): if len(os.listdir(backup_dir)) == 1: return True return False class SyntheticBackupInfo(LocalBackupInfo): def __init__( self, server, base_directory, backup_id=None, info_file=None, **kwargs ): """ Stores meta information about a single synthetic backup. .. note:: A synthetic backup is a base backup which was artificially created through ``pg_combinebackup``. A synthetic backup is not part of the Barman backup catalog, and only exists so we are able to recover a backup created by ``pg_combinebackup`` utility, as almost all functions and methods require a backup info object. The only difference from this class to its parent :class:`LocalBackupInfo` is that it accepts a custom base directory for the backup as synthetic backups are expected to live on directories other than the default ``/base`` path. :param barman.server.Server server: the server that owns the synthetic backup :param str base_directory: the root directory where this synthetic backup resides, essentially an override to the ``server.config.basebackups_directory`` configuration. :param str|None backup_id: the backup id of this backup :param None|str|TextIO info_file: path or file descriptor of an existing synthetic ``backup.info`` file """ self.base_directory = base_directory super(SyntheticBackupInfo, self).__init__( server, info_file, backup_id, **kwargs ) def get_basebackup_directory(self): """Get the backup directory based on its base directory""" return os.path.join(self.base_directory, self.backup_id) barman-3.13.3/barman/cloud_providers/0000755000175100001660000000000015002177454015672 5ustar 00000000000000barman-3.13.3/barman/cloud_providers/azure_blob_storage.py0000644000175100001660000011523015002177425022114 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see import logging import os import re from io import SEEK_END, BytesIO, RawIOBase import requests from barman.clients.cloud_compression import decompress_to_file from barman.cloud import ( DEFAULT_DELIMITER, CloudInterface, CloudProviderError, CloudSnapshotInterface, DecompressingStreamingIO, SnapshotMetadata, SnapshotsInfo, VolumeMetadata, ) from barman.exceptions import CommandException, SnapshotBackupException try: # Python 3.x from urllib.parse import urlparse except ImportError: # Python 2.x from urlparse import urlparse try: from azure.core.exceptions import ( HttpResponseError, ResourceNotFoundError, ServiceRequestError, ) from azure.storage.blob import ContainerClient, PartialBatchErrorException except ImportError: raise SystemExit("Missing required python module: azure-storage-blob") # Domain for azure blob URIs # See https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata#resource-uri-syntax AZURE_BLOB_STORAGE_DOMAIN = "blob.core.windows.net" class StreamingBlobIO(RawIOBase): """ Wrap an azure-storage-blob StorageStreamDownloader in the IOBase API. Inherits the IOBase defaults of seekable() -> False and writable() -> False. """ def __init__(self, blob): self._chunks = blob.chunks() self._current_chunk = BytesIO() def readable(self): return True def read(self, n=1): """ Read at most n bytes from the stream. Fetches new chunks from the StorageStreamDownloader until the requested number of bytes have been read. :param int n: Number of bytes to read from the stream :return: Up to n bytes from the stream :rtype: bytes """ n = None if n < 0 else n blob_bytes = self._current_chunk.read(n) bytes_count = len(blob_bytes) try: while bytes_count < n: self._current_chunk = BytesIO(self._chunks.next()) new_blob_bytes = self._current_chunk.read(n - bytes_count) bytes_count += len(new_blob_bytes) blob_bytes += new_blob_bytes except StopIteration: pass return blob_bytes class AzureCloudInterface(CloudInterface): # Azure block blob limitations # https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs MAX_CHUNKS_PER_FILE = 50000 # Minimum block size allowed in Azure Blob Storage is 64KB MIN_CHUNK_SIZE = 64 << 10 # Azure Blob Storage permit a maximum of 4.75TB per file # This is a hard limit, while our upload procedure can go over the specified # MAX_ARCHIVE_SIZE - so we set a maximum of 1TB per file MAX_ARCHIVE_SIZE = 1 << 40 MAX_DELETE_BATCH_SIZE = 256 # The size of each chunk in a single object upload when the size of the # object exceeds max_single_put_size. We default to 2MB in order to # allow the default max_concurrency of 8 to be achieved when uploading # uncompressed WAL segments of the default 16MB size. DEFAULT_MAX_BLOCK_SIZE = 2 << 20 # The maximum amount of concurrent chunks allowed in a single object upload # where the size exceeds max_single_put_size. We default to 8 based on # experiments with in-region and inter-region transfers within Azure. DEFAULT_MAX_CONCURRENCY = 8 # The largest file size which will be uploaded in a single PUT request. This # should be lower than the size of the compressed WAL segment in order to # force the Azure client to use concurrent chunk upload for archiving WAL files. DEFAULT_MAX_SINGLE_PUT_SIZE = 4 << 20 # The maximum size of the requests connection pool used by the Azure client # to upload objects. REQUESTS_POOL_MAXSIZE = 32 def __init__( self, url, jobs=2, encryption_scope=None, credential=None, tags=None, delete_batch_size=None, max_block_size=DEFAULT_MAX_BLOCK_SIZE, max_concurrency=DEFAULT_MAX_CONCURRENCY, max_single_put_size=DEFAULT_MAX_SINGLE_PUT_SIZE, ): """ Create a new Azure Blob Storage interface given the supplied account url :param str url: Full URL of the cloud destination/source :param int jobs: How many sub-processes to use for asynchronous uploading, defaults to 2. :param int|None delete_batch_size: the maximum number of objects to be deleted in a single request """ super(AzureCloudInterface, self).__init__( url=url, jobs=jobs, tags=tags, delete_batch_size=delete_batch_size, ) self.encryption_scope = encryption_scope self.credential = credential self.max_block_size = max_block_size self.max_concurrency = max_concurrency self.max_single_put_size = max_single_put_size parsed_url = urlparse(url) if parsed_url.netloc.endswith(AZURE_BLOB_STORAGE_DOMAIN): # We have an Azure Storage URI so we use the following form: # ://..core.windows.net/ # where is /. # Note that although Azure supports an implicit root container, we require # that the container is always included. self.account_url = parsed_url.netloc try: self.bucket_name = parsed_url.path.split("/")[1] except IndexError: raise ValueError("azure blob storage URL %s is malformed" % url) path = parsed_url.path.split("/")[2:] else: # We are dealing with emulated storage so we use the following form: # http://:// logging.info("Using emulated storage URL: %s " % url) if "AZURE_STORAGE_CONNECTION_STRING" not in os.environ: raise ValueError( "A connection string must be provided when using emulated storage" ) try: self.bucket_name = parsed_url.path.split("/")[2] except IndexError: raise ValueError("emulated storage URL %s is malformed" % url) path = parsed_url.path.split("/")[3:] self.path = "/".join(path) self.bucket_exists = None self._reinit_session() def _reinit_session(self): """ Create a new session """ if self.credential: # Any supplied credential takes precedence over the environment credential = self.credential elif "AZURE_STORAGE_CONNECTION_STRING" in os.environ: logging.info("Authenticating to Azure with connection string") self.container_client = ContainerClient.from_connection_string( conn_str=os.getenv("AZURE_STORAGE_CONNECTION_STRING"), container_name=self.bucket_name, ) return else: if "AZURE_STORAGE_SAS_TOKEN" in os.environ: logging.info("Authenticating to Azure with SAS token") credential = os.getenv("AZURE_STORAGE_SAS_TOKEN") elif "AZURE_STORAGE_KEY" in os.environ: logging.info("Authenticating to Azure with shared key") credential = os.getenv("AZURE_STORAGE_KEY") else: logging.info("Authenticating to Azure with default credentials") # azure-identity is not part of azure-storage-blob so only import # it if needed try: from azure.identity import DefaultAzureCredential except ImportError: raise SystemExit("Missing required python module: azure-identity") credential = DefaultAzureCredential() session = requests.Session() adapter = requests.adapters.HTTPAdapter(pool_maxsize=self.REQUESTS_POOL_MAXSIZE) session.mount("https://", adapter) self.container_client = ContainerClient( account_url=self.account_url, container_name=self.bucket_name, credential=credential, max_single_put_size=self.max_single_put_size, max_block_size=self.max_block_size, session=session, ) @property def _extra_upload_args(self): optional_args = {} if self.encryption_scope: optional_args["encryption_scope"] = self.encryption_scope return optional_args def test_connectivity(self): """ Test Azure connectivity by trying to access a container """ try: # We are not even interested in the existence of the bucket, # we just want to see if Azure blob service is reachable. self.bucket_exists = self._check_bucket_existence() return True except (HttpResponseError, ServiceRequestError) as exc: logging.error("Can't connect to cloud provider: %s", exc) return False def _check_bucket_existence(self): """ Chck Azure Blob Storage for the target container Although there is an `exists` function it cannot be called by container-level shared access tokens. We therefore check for existence by calling list_blobs on the container. :return: True if the container exists, False otherwise :rtype: bool """ try: self.container_client.list_blobs().next() except ResourceNotFoundError: return False except StopIteration: # The bucket is empty but it does exist pass return True def _create_bucket(self): """ Create the container in cloud storage """ # By default public access is disabled for newly created containers. # Unlike S3 there is no concept of regions for containers (this is at # the storage account level in Azure) self.container_client.create_container() def list_bucket(self, prefix="", delimiter=DEFAULT_DELIMITER): """ List bucket content in a directory manner :param str prefix: :param str delimiter: :return: List of objects and dirs right under the prefix :rtype: List[str] """ res = self.container_client.walk_blobs( name_starts_with=prefix, delimiter=delimiter ) for item in res: yield item.name def download_file(self, key, dest_path, decompress=None): """ Download a file from Azure Blob Storage :param str key: The key to download :param str dest_path: Where to put the destination file :param str|None decompress: Compression scheme to use for decompression """ obj = self.container_client.download_blob(key) with open(dest_path, "wb") as dest_file: if decompress is None: obj.download_to_stream(dest_file) return blob = StreamingBlobIO(obj) decompress_to_file(blob, dest_file, decompress) def remote_open(self, key, decompressor=None): """ Open a remote Azure Blob Storage object and return a readable stream :param str key: The key identifying the object to open :param barman.clients.cloud_compression.ChunkedCompressor decompressor: A ChunkedCompressor object which will be used to decompress chunks of bytes as they are read from the stream :return: A file-like object from which the stream can be read or None if the key does not exist """ try: obj = self.container_client.download_blob(key) resp = StreamingBlobIO(obj) if decompressor: return DecompressingStreamingIO(resp, decompressor) else: return resp except ResourceNotFoundError: return None def upload_fileobj( self, fileobj, key, override_tags=None, ): """ Synchronously upload the content of a file-like object to a cloud key :param fileobj IOBase: File-like object to upload :param str key: The key to identify the uploaded object :param List[tuple] override_tags: List of tags as k,v tuples to be added to the uploaded object """ # Find length of the file so we can pass it to the Azure client fileobj.seek(0, SEEK_END) length = fileobj.tell() fileobj.seek(0) extra_args = self._extra_upload_args.copy() tags = override_tags or self.tags if tags is not None: extra_args["tags"] = dict(tags) self.container_client.upload_blob( name=key, data=fileobj, overwrite=True, length=length, max_concurrency=self.max_concurrency, **extra_args ) def create_multipart_upload(self, key): """No-op method because Azure has no concept of multipart uploads Instead of multipart upload, blob blocks are staged and then committed. However this does not require anything to be created up front. This method therefore does nothing. """ pass def _upload_part(self, upload_metadata, key, body, part_number): """ Upload a single block of this block blob. Uses the supplied part number to generate the block ID and returns it as the "PartNumber" in the part metadata. :param dict upload_metadata: Provider-specific metadata about the upload (not used in Azure) :param str key: The key to use in the cloud service :param object body: A stream-like object to upload :param int part_number: Part number, starting from 1 :return: The part metadata :rtype: dict[str, None|str] """ # Block IDs must be the same length for all bocks in the blob # and no greater than 64 characters. Given there is a limit of # 50000 blocks per blob we zero-pad the part_number to five # places. block_id = str(part_number).zfill(5) blob_client = self.container_client.get_blob_client(key) blob_client.stage_block(block_id, body, **self._extra_upload_args) return {"PartNumber": block_id} def _complete_multipart_upload(self, upload_metadata, key, parts): """ Finish a "multipart upload" by committing all blocks in the blob. :param dict upload_metadata: Provider-specific metadata about the upload (not used in Azure) :param str key: The key to use in the cloud service :param parts: The list of block IDs for the blocks which compose this blob """ blob_client = self.container_client.get_blob_client(key) block_list = [part["PartNumber"] for part in parts] extra_args = self._extra_upload_args.copy() if self.tags is not None: extra_args["tags"] = dict(self.tags) blob_client.commit_block_list(block_list, **extra_args) def _abort_multipart_upload(self, upload_metadata, key): """ Abort the upload of a block blob The objective of this method is to clean up any dangling resources - in this case those resources are uncommitted blocks. :param dict upload_metadata: Provider-specific metadata about the upload (not used in Azure) :param str key: The key to use in the cloud service """ # Ideally we would clean up uncommitted blocks at this point # however there is no way of doing that. # Uncommitted blocks will be discarded after 7 days or when # the blob is committed (if they're not included in the commit). # We therefore create an empty blob (thereby discarding all uploaded # blocks for that blob) and then delete it. blob_client = self.container_client.get_blob_client(key) blob_client.commit_block_list([], **self._extra_upload_args) blob_client.delete_blob() def _delete_objects_batch(self, paths): """ Delete the objects at the specified paths :param List[str] paths: """ super(AzureCloudInterface, self)._delete_objects_batch(paths) try: # If paths is empty because the files have already been deleted then # delete_blobs will return successfully so we just call it with whatever # we were given responses = self.container_client.delete_blobs(*paths) except PartialBatchErrorException as exc: # Although the docs imply any errors will be returned in the response # object, in practice a PartialBatchErrorException is raised which contains # the response objects in its `parts` attribute. # We therefore set responses to reference the response in the exception and # treat it the same way we would a regular response. logging.warning( "PartialBatchErrorException received from Azure: %s" % exc.message ) responses = exc.parts # resp is an iterator of HttpResponse objects so we check the status codes # which should all be 202 if successful errors = False for resp in responses: if resp.status_code == 404: logging.warning( "Deletion of object %s failed because it could not be found" % resp.request.url ) elif resp.status_code != 202: errors = True logging.error( 'Deletion of object %s failed with error code: "%s"' % (resp.request.url, resp.status_code) ) if errors: raise CloudProviderError() def get_prefixes(self, prefix): """ Return only the common prefixes under the supplied prefix. :param str prefix: The object key prefix under which the common prefixes will be found. :rtype: Iterator[str] :return: A list of unique prefixes immediately under the supplied prefix. """ raise NotImplementedError() def delete_under_prefix(self, prefix): """ Delete all objects under the specified prefix. :param str prefix: The object key prefix under which all objects should be deleted. """ raise NotImplementedError() def import_azure_mgmt_compute(): """ Import and return the azure.mgmt.compute module. This particular import happens in a function so that it can be deferred until needed while still allowing tests to easily mock the library. """ try: import azure.mgmt.compute as compute except ImportError: raise SystemExit("Missing required python module: azure-mgmt-compute") return compute def import_azure_identity(): """ Import and return the azure.identity module. This particular import happens in a function so that it can be deferred until needed while still allowing tests to easily mock the library. """ try: import azure.identity as identity except ImportError: raise SystemExit("Missing required python module: azure-identity") return identity class AzureCloudSnapshotInterface(CloudSnapshotInterface): """ Implementation of CloudSnapshotInterface for managed disk snapshots in Azure, as described at: https://learn.microsoft.com/en-us/azure/virtual-machines/snapshot-copy-managed-disk """ _required_config_for_backup = CloudSnapshotInterface._required_config_for_backup + ( "azure_resource_group", ) _required_config_for_restore = ( CloudSnapshotInterface._required_config_for_restore + ("azure_resource_group",) ) def __init__(self, subscription_id, resource_group=None, credential=None): """ Imports the azure-mgmt-compute library and creates the clients necessary for creating and managing snapshots. :param str subscription_id: A Microsoft Azure subscription ID to which all resources accessed through this interface belong. :param str resource_group|None: The resource_group to which the resources accessed through this interface belong. :param azure.identity.AzureCliCredential|azure.identity.ManagedIdentityCredential| azure.identity.DefaultAzureCredential The Azure credential to be used when authenticating against the Azure API. If omitted then a DefaultAzureCredential will be created and used. """ if subscription_id is None: raise TypeError("subscription_id cannot be None") self.subscription_id = subscription_id self.resource_group = resource_group if credential is None: identity = import_azure_identity() credential = identity.DefaultAzureCredential self.credential = credential() # Import of azure-mgmt-compute is deferred until this point so that it does not # become a hard dependency of this module. compute = import_azure_mgmt_compute() self.client = compute.ComputeManagementClient( self.credential, self.subscription_id ) def _get_instance_metadata(self, instance_name): """ Retrieve the metadata for the named instance. :rtype: azure.mgmt.compute.v2022_11_01.models.VirtualMachine :return: An object representing the named compute instance. """ try: return self.client.virtual_machines.get(self.resource_group, instance_name) except ResourceNotFoundError: raise SnapshotBackupException( "Cannot find instance with name %s in resource group %s " "in subscription %s" % (instance_name, self.resource_group, self.subscription_id) ) def _get_disk_metadata(self, disk_name): """ Retrieve the metadata for the named disk in the specified zone. :rtype: azure.mgmt.compute.v2022_11_01.models.Disk :return: An object representing the disk. """ try: return self.client.disks.get(self.resource_group, disk_name) except ResourceNotFoundError: raise SnapshotBackupException( "Cannot find disk with name %s in resource group %s " "in subscription %s" % (disk_name, self.resource_group, self.subscription_id) ) def _take_snapshot(self, backup_info, resource_group, location, disk_name, disk_id): """ Take a snapshot of a managed disk in Azure. :param barman.infofile.LocalBackupInfo backup_info: Backup information. :param str resource_group: The resource_group to which the snapshot disks and instance belong. :param str location: The location of the source disk for the snapshot. :param str disk_name: The name of the source disk for the snapshot. :param str disk_id: The Azure identifier for the source disk. :rtype: str :return: The name used to reference the snapshot with Azure. """ snapshot_name = "%s-%s" % (disk_name, backup_info.backup_id.lower()) logging.info("Taking snapshot '%s' of disk '%s'", snapshot_name, disk_name) resp = self.client.snapshots.begin_create_or_update( resource_group, snapshot_name, { "location": location, "incremental": True, "creation_data": {"create_option": "Copy", "source_uri": disk_id}, }, ) logging.info("Waiting for snapshot '%s' completion", snapshot_name) resp.wait() if ( resp.status().lower() != "succeeded" or resp.result().provisioning_state.lower() != "succeeded" ): raise CloudProviderError( "Snapshot '%s' failed with error code %s: %s" % (snapshot_name, resp.status(), resp.result()) ) logging.info("Snapshot '%s' completed", snapshot_name) return snapshot_name def take_snapshot_backup(self, backup_info, instance_name, volumes): """ Take a snapshot backup for the named instance. Creates a snapshot for each named disk and saves the required metadata to backup_info.snapshots_info as an AzureSnapshotsInfo object. :param barman.infofile.LocalBackupInfo backup_info: Backup information. :param str instance_name: The name of the VM instance to which the disks to be backed up are attached. :param dict[str,barman.cloud.VolumeMetadata] volumes: Metadata describing the volumes to be backed up. """ instance_metadata = self._get_instance_metadata(instance_name) snapshots = [] for disk_name, volume_metadata in volumes.items(): attached_disks = [ d for d in instance_metadata.storage_profile.data_disks if d.name == disk_name ] if len(attached_disks) == 0: raise SnapshotBackupException( "Disk %s not attached to instance %s" % (disk_name, instance_name) ) # We should always have exactly one attached disk matching the name assert len(attached_disks) == 1 snapshot_name = self._take_snapshot( backup_info, self.resource_group, volume_metadata.location, disk_name, attached_disks[0].managed_disk.id, ) snapshots.append( AzureSnapshotMetadata( lun=attached_disks[0].lun, snapshot_name=snapshot_name, location=volume_metadata.location, mount_point=volume_metadata.mount_point, mount_options=volume_metadata.mount_options, ) ) backup_info.snapshots_info = AzureSnapshotsInfo( snapshots=snapshots, subscription_id=self.subscription_id, resource_group=self.resource_group, ) def _delete_snapshot(self, snapshot_name, resource_group): """ Delete the specified snapshot. :param str snapshot_name: The short name used to reference the snapshot within Azure. :param str resource_group: The resource_group to which the snapshot belongs. """ # The call to begin_delete will raise a ResourceNotFoundError if the resource # group cannot be found. This is deliberately not caught here because it is # an error condition which we cannot do anything about. # If the snapshot itself cannot be found then the response status will be # `succeeded`, exactly as if it did exist and was successfully deleted. resp = self.client.snapshots.begin_delete( resource_group, snapshot_name, ) resp.wait() if resp.status().lower() != "succeeded": raise CloudProviderError( "Deletion of snapshot %s failed with error code %s: %s" % (snapshot_name, resp.status(), resp.result()) ) logging.info("Snapshot %s deleted", snapshot_name) def delete_snapshot_backup(self, backup_info): """ Delete all snapshots for the supplied backup. :param barman.infofile.LocalBackupInfo backup_info: Backup information. """ for snapshot in backup_info.snapshots_info.snapshots: logging.info( "Deleting snapshot '%s' for backup %s", snapshot.identifier, backup_info.backup_id, ) self._delete_snapshot( snapshot.identifier, backup_info.snapshots_info.resource_group ) def get_attached_volumes(self, instance_name, disks=None, fail_on_missing=True): """ Returns metadata for the volumes attached to this instance. Queries Azure for metadata relating to the volumes attached to the named instance and returns a dict of `VolumeMetadata` objects, keyed by disk name. If the optional disks parameter is supplied then this method returns metadata for the disks in the supplied list only. If fail_on_missing is set to True then a SnapshotBackupException is raised if any of the supplied disks are not found to be attached to the instance. If the disks parameter is not supplied then this method returns a VolumeMetadata object for every disk attached to this instance. :param str instance_name: The name of the VM instance to which the disks are attached. :param list[str]|None disks: A list containing the names of disks to be backed up. :param bool fail_on_missing: Fail with a SnapshotBackupException if any specified disks are not attached to the instance. :rtype: dict[str, VolumeMetadata] :return: A dict of VolumeMetadata objects representing each volume attached to the instance, keyed by volume identifier. """ instance_metadata = self._get_instance_metadata(instance_name) attached_volumes = {} for attachment_metadata in instance_metadata.storage_profile.data_disks: disk_name = attachment_metadata.name if disks and disk_name not in disks: continue assert disk_name not in attached_volumes disk_metadata = self._get_disk_metadata(disk_name) attached_volumes[disk_name] = AzureVolumeMetadata( attachment_metadata, disk_metadata ) # Check all requested disks were found and complain if necessary if disks is not None and fail_on_missing: unattached_disks = [] for disk_name in disks: if disk_name not in attached_volumes: # Verify the disk definitely exists by fetching the metadata self._get_disk_metadata(disk_name) # Append to list of unattached disks unattached_disks.append(disk_name) if len(unattached_disks) > 0: raise SnapshotBackupException( "Disks not attached to instance %s: %s" % (instance_name, ", ".join(unattached_disks)) ) return attached_volumes def instance_exists(self, instance_name): """ Determine whether the named instance exists. :param str instance_name: The name of the VM instance to which the disks to be backed up are attached. :rtype: bool :return: True if the named instance exists, False otherwise. """ try: self.client.virtual_machines.get(self.resource_group, instance_name) except ResourceNotFoundError: return False return True class AzureVolumeMetadata(VolumeMetadata): """ Specialization of VolumeMetadata for Azure managed disks. This class uses the LUN obtained from the Azure API in order to resolve the mount point and options via using a documented symlink. """ def __init__(self, attachment_metadata=None, disk_metadata=None): """ Creates an AzureVolumeMetadata instance using metadata obtained from the Azure API. Uses attachment_metadata to obtain the LUN of the attached volume and disk_metadata to obtain the location of the disk. :param azure.mgmt.compute.v2022_11_01.models.DataDisk|None attachment_metadata: Metadata for the attached volume. :param azure.mgmt.compute.v2022_11_01.models.Disk|None disk_metadata: Metadata for the managed disk. """ super(AzureVolumeMetadata, self).__init__() self.location = None self._lun = None self._snapshot_name = None if attachment_metadata is not None: self._lun = attachment_metadata.lun if disk_metadata is not None: # Record the location because this is needed when creating snapshots # (even though snapshots can only be created in the same location as the # source disk, Azure requires us to specify the location anyway). self.location = disk_metadata.location # Figure out whether this disk was cloned from a snapshot. if ( disk_metadata.creation_data.create_option == "Copy" and "providers/Microsoft.Compute/snapshots" in disk_metadata.creation_data.source_resource_id ): # Extract the snapshot name from the source_resource_id in the disk # metadata. We do not care about the source subscription or resource # group - these may vary depending on whether the user has copied the # snapshot between resource groups or subscriptions. We only care about # the name because this is the part of the resource ID which Barman # associates with backups. resource_regex = ( r"/subscriptions/(?!/).*/resourceGroups/(?!/).*" "/providers/Microsoft.Compute" r"/snapshots/(?P.*)" ) match = re.search( resource_regex, disk_metadata.creation_data.source_resource_id ) if match is None or match.group("snapshot_name") == "": raise SnapshotBackupException( "Could not determine source snapshot for disk %s with source resource ID %s" % ( disk_metadata.name, disk_metadata.creation_data.source_resource_id, ) ) self._snapshot_name = match.group("snapshot_name") def resolve_mounted_volume(self, cmd): """ Resolve the mount point and mount options using shell commands. Uses findmnt to retrieve the mount point and mount options for the device path at which this volume is mounted. :param UnixLocalCommand cmd: An object which can be used to run shell commands on a local (or remote, via the UnixRemoteCommand subclass) instance. """ if self._lun is None: raise SnapshotBackupException("Cannot resolve mounted volume: LUN unknown") try: # This symlink path is created by the Azure linux agent on boot. It is a # direct symlink to the actual device path of the attached volume. This # symlink will be consistent across reboots of the VM but the device path # will not. We therefore call findmnt directly on this symlink. # See the following documentation for more context: # - https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/troubleshoot-device-names-problems#identify-disk-luns lun_symlink = "/dev/disk/azure/scsi1/lun{}".format(self._lun) mount_point, mount_options = cmd.findmnt(lun_symlink) except CommandException as e: raise SnapshotBackupException( "Error finding mount point for volume with lun %s: %s" % (self._lun, e) ) if mount_point is None: raise SnapshotBackupException( "Could not find volume with lun %s at any mount point" % self._lun ) self._mount_point = mount_point self._mount_options = mount_options @property def source_snapshot(self): """ An identifier which can reference the snapshot via the cloud provider. :rtype: str :return: The snapshot short name. """ return self._snapshot_name class AzureSnapshotMetadata(SnapshotMetadata): """ Specialization of SnapshotMetadata for Azure managed disk snapshots. Stores the location, lun and snapshot_name in the provider-specific field. """ _provider_fields = ("location", "lun", "snapshot_name") def __init__( self, mount_options=None, mount_point=None, lun=None, snapshot_name=None, location=None, ): """ Constructor saves additional metadata for Azure snapshots. :param str mount_options: The mount options used for the source disk at the time of the backup. :param str mount_point: The mount point of the source disk at the time of the backup. :param int lun: The lun identifying the disk from which the snapshot was taken on the instance it was attached to at the time of the backup. :param str snapshot_name: The snapshot name used in the Azure API. :param str location: The location of the disk from which the snapshot was taken at the time of the backup. """ super(AzureSnapshotMetadata, self).__init__(mount_options, mount_point) self.lun = lun self.snapshot_name = snapshot_name self.location = location @property def identifier(self): """ An identifier which can reference the snapshot via the cloud provider. :rtype: str :return: The snapshot short name. """ return self.snapshot_name class AzureSnapshotsInfo(SnapshotsInfo): """ Represents the snapshots_info field for Azure managed disk snapshots. """ _provider_fields = ("subscription_id", "resource_group") _snapshot_metadata_cls = AzureSnapshotMetadata def __init__(self, snapshots=None, subscription_id=None, resource_group=None): """ Constructor saves the list of snapshots if it is provided. :param list[SnapshotMetadata] snapshots: A list of metadata objects for each snapshot. """ super(AzureSnapshotsInfo, self).__init__(snapshots) self.provider = "azure" self.subscription_id = subscription_id self.resource_group = resource_group barman-3.13.3/barman/cloud_providers/aws_s3.py0000644000175100001660000013226015002177425017445 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see import json import logging import math import shutil from datetime import datetime from io import RawIOBase from barman.clients.cloud_compression import decompress_to_file from barman.cloud import ( DEFAULT_DELIMITER, CloudInterface, CloudProviderError, CloudSnapshotInterface, DecompressingStreamingIO, SnapshotMetadata, SnapshotsInfo, VolumeMetadata, ) from barman.exceptions import ( CommandException, SnapshotBackupException, SnapshotInstanceNotFoundException, ) try: # Python 3.x from urllib.parse import urlencode, urlparse except ImportError: # Python 2.x from urllib import urlencode from urlparse import urlparse try: import boto3 from boto3.s3.transfer import TransferConfig from botocore.config import Config from botocore.exceptions import ClientError, EndpointConnectionError except ImportError: raise SystemExit("Missing required python module: boto3") class StreamingBodyIO(RawIOBase): """ Wrap a boto StreamingBody in the IOBase API. """ def __init__(self, body): self.body = body def readable(self): return True def read(self, n=-1): n = None if n < 0 else n return self.body.read(n) class S3CloudInterface(CloudInterface): # S3 multipart upload limitations # http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html MAX_CHUNKS_PER_FILE = 10000 MIN_CHUNK_SIZE = 5 << 20 # S3 permit a maximum of 5TB per file # https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html # This is a hard limit, while our upload procedure can go over the specified # MAX_ARCHIVE_SIZE - so we set a maximum of 1TB per file MAX_ARCHIVE_SIZE = 1 << 40 MAX_DELETE_BATCH_SIZE = 1000 # The minimum size for a file to be uploaded using multipart upload in upload_fileobj # 100MB is the AWS recommendation for when to start considering using multipart upload # https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html MULTIPART_THRESHOLD = 104857600 def __getstate__(self): state = self.__dict__.copy() # Remove boto3 client reference from the state as it cannot be pickled # in Python >= 3.8 and multiprocessing will pickle the object when the # worker processes are created. # The worker processes create their own boto3 sessions so do not need # the boto3 session from the parent process. del state["s3"] return state def __setstate__(self, state): self.__dict__.update(state) def __init__( self, url, encryption=None, jobs=2, profile_name=None, endpoint_url=None, tags=None, delete_batch_size=None, read_timeout=None, sse_kms_key_id=None, ): """ Create a new S3 interface given the S3 destination url and the profile name :param str url: Full URL of the cloud destination/source :param str|None encryption: Encryption type string :param int jobs: How many sub-processes to use for asynchronous uploading, defaults to 2. :param str profile_name: Amazon auth profile identifier :param str endpoint_url: override default endpoint detection strategy with this one :param int|None delete_batch_size: the maximum number of objects to be deleted in a single request :param int|None read_timeout: the time in seconds until a timeout is raised when waiting to read from a connection :param str|None sse_kms_key_id: the AWS KMS key ID that should be used for encrypting uploaded data in S3 """ super(S3CloudInterface, self).__init__( url=url, jobs=jobs, tags=tags, delete_batch_size=delete_batch_size, ) self.profile_name = profile_name self.encryption = encryption self.endpoint_url = endpoint_url self.read_timeout = read_timeout self.sse_kms_key_id = sse_kms_key_id # Extract information from the destination URL parsed_url = urlparse(url) # If netloc is not present, the s3 url is badly formatted. if parsed_url.netloc == "" or parsed_url.scheme != "s3": raise ValueError("Invalid s3 URL address: %s" % url) self.bucket_name = parsed_url.netloc self.bucket_exists = None self.path = parsed_url.path.lstrip("/") # initialize the config object to be used in uploads self.config = TransferConfig(multipart_threshold=self.MULTIPART_THRESHOLD) # Build a session, so we can extract the correct resource self._reinit_session() def _reinit_session(self): """ Create a new session """ config_kwargs = {} if self.read_timeout is not None: config_kwargs["read_timeout"] = self.read_timeout config = Config(**config_kwargs) session = boto3.Session(profile_name=self.profile_name) self.s3 = session.resource("s3", endpoint_url=self.endpoint_url, config=config) @property def _extra_upload_args(self): """ Return a dict containing ExtraArgs to be passed to certain boto3 calls Because some boto3 calls accept `ExtraArgs: {}` and others do not, we return a nested dict which can be expanded with `**` in the boto3 call. """ additional_args = {} if self.encryption: additional_args["ServerSideEncryption"] = self.encryption if self.sse_kms_key_id: additional_args["SSEKMSKeyId"] = self.sse_kms_key_id return additional_args def test_connectivity(self): """ Test AWS connectivity by trying to access a bucket """ try: # We are not even interested in the existence of the bucket, # we just want to try if aws is reachable self.bucket_exists = self._check_bucket_existence() return True except EndpointConnectionError as exc: logging.error("Can't connect to cloud provider: %s", exc) return False def _check_bucket_existence(self): """ Check cloud storage for the target bucket :return: True if the bucket exists, False otherwise :rtype: bool """ try: # Search the bucket on s3 self.s3.meta.client.head_bucket(Bucket=self.bucket_name) return True except ClientError as exc: # If a client error is thrown, then check the error code. # If code was 404, then the bucket does not exist error_code = exc.response["Error"]["Code"] if error_code == "404": return False # Otherwise there is nothing else to do than re-raise the original # exception raise def _create_bucket(self): """ Create the bucket in cloud storage """ # Get the current region from client. # Do not use session.region_name here because it may be None region = self.s3.meta.client.meta.region_name logging.info( "Bucket '%s' does not exist, creating it on region '%s'", self.bucket_name, region, ) create_bucket_config = { "ACL": "private", } # The location constraint is required during bucket creation # for all regions outside of us-east-1. This constraint cannot # be specified in us-east-1; specifying it in this region # results in a failure, so we will only # add it if we are deploying outside of us-east-1. # See https://github.com/boto/boto3/issues/125 if region != "us-east-1": create_bucket_config["CreateBucketConfiguration"] = { "LocationConstraint": region, } self.s3.Bucket(self.bucket_name).create(**create_bucket_config) def list_bucket(self, prefix="", delimiter=DEFAULT_DELIMITER): """ List bucket content in a directory manner :param str prefix: :param str delimiter: :return: List of objects and dirs right under the prefix :rtype: List[str] """ if prefix.startswith(delimiter): prefix = prefix.lstrip(delimiter) paginator = self.s3.meta.client.get_paginator("list_objects_v2") pages = paginator.paginate( Bucket=self.bucket_name, Prefix=prefix, Delimiter=delimiter ) for page in pages: # List "folders" keys = page.get("CommonPrefixes") if keys is not None: for k in keys: yield k.get("Prefix") # List "files" objects = page.get("Contents") if objects is not None: for o in objects: yield o.get("Key") def download_file(self, key, dest_path, decompress): """ Download a file from S3 :param str key: The S3 key to download :param str dest_path: Where to put the destination file :param str|None decompress: Compression scheme to use for decompression """ # Open the remote file obj = self.s3.Object(self.bucket_name, key) remote_file = obj.get()["Body"] # Write the dest file in binary mode with open(dest_path, "wb") as dest_file: # If the file is not compressed, just copy its content if decompress is None: shutil.copyfileobj(remote_file, dest_file) return decompress_to_file(remote_file, dest_file, decompress) def remote_open(self, key, decompressor=None): """ Open a remote S3 object and returns a readable stream :param str key: The key identifying the object to open :param barman.clients.cloud_compression.ChunkedCompressor decompressor: A ChunkedCompressor object which will be used to decompress chunks of bytes as they are read from the stream :return: A file-like object from which the stream can be read or None if the key does not exist """ try: obj = self.s3.Object(self.bucket_name, key) resp = StreamingBodyIO(obj.get()["Body"]) if decompressor: return DecompressingStreamingIO(resp, decompressor) else: return resp except ClientError as exc: error_code = exc.response["Error"]["Code"] if error_code == "NoSuchKey": return None else: raise def upload_fileobj(self, fileobj, key, override_tags=None): """ Synchronously upload the content of a file-like object to a cloud key :param fileobj IOBase: File-like object to upload :param str key: The key to identify the uploaded object :param List[tuple] override_tags: List of k,v tuples which should override any tags already defined in the cloud interface """ extra_args = self._extra_upload_args.copy() tags = override_tags or self.tags if tags is not None: extra_args["Tagging"] = urlencode(tags) self.s3.meta.client.upload_fileobj( Fileobj=fileobj, Bucket=self.bucket_name, Key=key, ExtraArgs=extra_args, Config=self.config, ) def create_multipart_upload(self, key): """ Create a new multipart upload :param key: The key to use in the cloud service :return: The multipart upload handle :rtype: dict[str, str] """ extra_args = self._extra_upload_args.copy() if self.tags is not None: extra_args["Tagging"] = urlencode(self.tags) return self.s3.meta.client.create_multipart_upload( Bucket=self.bucket_name, Key=key, **extra_args ) def _upload_part(self, upload_metadata, key, body, part_number): """ Upload a part into this multipart upload :param dict upload_metadata: The multipart upload handle :param str key: The key to use in the cloud service :param object body: A stream-like object to upload :param int part_number: Part number, starting from 1 :return: The part handle :rtype: dict[str, None|str] """ part = self.s3.meta.client.upload_part( Body=body, Bucket=self.bucket_name, Key=key, UploadId=upload_metadata["UploadId"], PartNumber=part_number, ) return { "PartNumber": part_number, "ETag": part["ETag"], } def _complete_multipart_upload(self, upload_metadata, key, parts): """ Finish a certain multipart upload :param dict upload_metadata: The multipart upload handle :param str key: The key to use in the cloud service :param parts: The list of parts composing the multipart upload """ self.s3.meta.client.complete_multipart_upload( Bucket=self.bucket_name, Key=key, UploadId=upload_metadata["UploadId"], MultipartUpload={"Parts": parts}, ) def _abort_multipart_upload(self, upload_metadata, key): """ Abort a certain multipart upload :param dict upload_metadata: The multipart upload handle :param str key: The key to use in the cloud service """ self.s3.meta.client.abort_multipart_upload( Bucket=self.bucket_name, Key=key, UploadId=upload_metadata["UploadId"] ) def _delete_objects_batch(self, paths): """ Delete the objects at the specified paths :param List[str] paths: """ super(S3CloudInterface, self)._delete_objects_batch(paths) resp = self.s3.meta.client.delete_objects( Bucket=self.bucket_name, Delete={ "Objects": [{"Key": path} for path in paths], "Quiet": True, }, ) if "Errors" in resp: for error_dict in resp["Errors"]: logging.error( 'Deletion of object %s failed with error code: "%s", message: "%s"' % (error_dict["Key"], error_dict["Code"], error_dict["Message"]) ) raise CloudProviderError() def get_prefixes(self, prefix): """ Return only the common prefixes under the supplied prefix. :param str prefix: The object key prefix under which the common prefixes will be found. :rtype: Iterator[str] :return: A list of unique prefixes immediately under the supplied prefix. """ for wal_prefix in self.list_bucket(prefix + "/", delimiter="/"): if wal_prefix.endswith("/"): yield wal_prefix def delete_under_prefix(self, prefix): """ Delete all objects under the specified prefix. :param str prefix: The object key prefix under which all objects should be deleted. """ if len(prefix) == 0 or prefix == "/" or not prefix.endswith("/"): raise ValueError( "Deleting all objects under prefix %s is not allowed" % prefix ) bucket = self.s3.Bucket(self.bucket_name) for resp in bucket.objects.filter(Prefix=prefix).delete(): response_metadata = resp["ResponseMetadata"] if response_metadata["HTTPStatusCode"] != 200: logging.error( 'Deletion of objects under %s failed with error code: "%s"' % (prefix, response_metadata["HTTPStatusCode"]) ) raise CloudProviderError() class AwsCloudSnapshotInterface(CloudSnapshotInterface): """ Implementation of CloudSnapshotInterface for EBS snapshots as implemented in AWS as documented at: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html """ def __init__( self, profile_name=None, region=None, await_snapshots_timeout=3600, lock_mode=None, lock_duration=None, lock_cool_off_period=None, lock_expiration_date=None, tags=None, ): """ Creates the client necessary for creating and managing snapshots. :param str profile_name: AWS auth profile identifier. :param str region: The AWS region in which snapshot resources are located. :param int await_snapshots_timeout: The maximum time in seconds to wait for snapshots to complete. :param str lock_mode: The lock mode to apply to the snapshot. :param int lock_duration: The duration (in days) for which the snapshot should be locked. :param int lock_cool_off_period: The cool-off period (in hours) for the snapshot. :param str lock_expiration_date: The expiration date for the snapshot in the format ``YYYY-MM-DDThh:mm:ss.sssZ``. :param List[Tuple[str, str]] tags: Key value pairs for tags to be applied. """ self.session = boto3.Session(profile_name=profile_name) # If a specific region was provided then this overrides any region which may be # defined in the profile self.region = region or self.session.region_name self.ec2_client = self.session.client("ec2", region_name=self.region) self.await_snapshots_timeout = await_snapshots_timeout self.tags = tags self.lock_mode = lock_mode self.lock_duration = lock_duration self.lock_cool_off_period = lock_cool_off_period self.lock_expiration_date = lock_expiration_date def _get_waiter_config(self): delay = 15 # Use ceil so that we always wait for at least the specified timeout max_attempts = math.ceil(self.await_snapshots_timeout / delay) return { "Delay": delay, # Ensure we always try waiting at least once "MaxAttempts": max(max_attempts, 1), } def _get_instance_metadata(self, instance_identifier): """ Retrieve the boto3 describe_instances metadata for the specified instance. The supplied instance_identifier can be either an AWS instance ID or a name. If an instance ID is supplied then this function will look it up directly. If a name is supplied then the `tag:Name` filter will be used to query the AWS API for instances with the matching `Name` tag. :param str instance_identifier: The instance ID or name of the VM instance. :rtype: dict :return: A dict containing the describe_instances metadata for the specified VM instance. """ # Consider all states other than `terminated` as valid instances allowed_states = ["pending", "running", "shutting-down", "stopping", "stopped"] # If the identifier looks like an instance ID then we attempt to look it up resp = None if instance_identifier.startswith("i-"): try: resp = self.ec2_client.describe_instances( InstanceIds=[instance_identifier], Filters=[ {"Name": "instance-state-name", "Values": allowed_states}, ], ) except ClientError as exc: error_code = exc.response["Error"]["Code"] # If we have a malformed instance ID then continue and treat it # like a name, otherwise re-raise the original error if error_code != "InvalidInstanceID.Malformed": raise # If we do not have a response then try looking up by name if resp is None: resp = self.ec2_client.describe_instances( Filters=[ {"Name": "tag:Name", "Values": [instance_identifier]}, {"Name": "instance-state-name", "Values": allowed_states}, ] ) # Check for non-unique reservations and instances before returning the instance # because tag uniqueness is not a thing reservations = resp["Reservations"] if len(reservations) == 1: if len(reservations[0]["Instances"]) == 1: return reservations[0]["Instances"][0] elif len(reservations[0]["Instances"]) > 1: raise CloudProviderError( "Cannot find a unique EC2 instance matching {}".format( instance_identifier ) ) elif len(reservations) > 1: raise CloudProviderError( "Cannot find a unique EC2 reservation containing instance {}".format( instance_identifier ) ) raise SnapshotInstanceNotFoundException( "Cannot find instance {}".format(instance_identifier) ) def _has_tag(self, resource, tag_name, tag_value): """ Determine whether the resource metadata contains a specified tag. :param dict resource: Metadata describing an AWS resource. :parma str tag_name: The name of the tag to be checked. :param str tag_value: The value of the tag to be checked. :rtype: bool :return: True if a tag with the specified name and value was found, False otherwise. """ if "Tags" in resource: for tag in resource["Tags"]: if tag["Key"] == tag_name and tag["Value"] == tag_value: return True return False def _lookup_volume(self, attached_volumes, volume_identifier): """ Searches a supplied list of describe_volumes metadata for the specified volume. :param list[dict] attached_volumes: A list of volumes in the format provided by the boto3 describe_volumes function. :param str volume_identifier: The volume ID or name of the volume to be looked up. :rtype: dict|None :return: describe_volume metadata for the volume matching the supplied identifier. """ # Check whether volume_identifier matches a VolumeId matching_volumes = [ volume for volume in attached_volumes if volume["VolumeId"] == volume_identifier ] # If we do not have a match, try again but search for a matching Name tag if not matching_volumes: matching_volumes = [ volume for volume in attached_volumes if self._has_tag(volume, "Name", volume_identifier) ] # If there is more than one matching volume then it's an error condition if len(matching_volumes) > 1: raise CloudProviderError( "Duplicate volumes found matching {}: {}".format( volume_identifier, ", ".join(v["VolumeId"] for v in matching_volumes), ) ) # If no matching volumes were found then return None - it is up to the calling # code to decide if this is an error elif len(matching_volumes) == 0: return None # Otherwise, we found exactly one matching volume and return its metadata else: return matching_volumes[0] def _get_requested_volumes(self, instance_metadata, disks=None): """ Fetch describe_volumes metadata for disks attached to a specified VM instance. Queries the AWS API for metadata describing the volumes attached to the instance described in instance_metadata. If `disks` is specified then metadata is only returned for the volumes that are included in the list and attached to the instance. Volumes which are requested in the `disks` list but not attached to the instance are not included in the response - it is up to calling code to decide whether this is an error condition. Entries in `disks` can be either volume IDs or names. The value provided for each volume will be included in the response under the key `identifier`. If `disks` is not provided then every non-root volume attached to the instance will be included in the response. :param dict instance_metadata: A dict containing the describe_instances metadata for a VM instance. :param list[str] disks: A list of volume IDs or volume names. If specified then only volumes in this list which are attached to the instance described by instance_metadata will be included in the response. :rtype: list[dict[str,str|dict]] :return: A list of dicts containing identifiers and describe_volumes metadata for the requested volumes. """ # Pre-fetch the describe_volumes output for all volumes attached to the instance attached_volumes = self.ec2_client.describe_volumes( Filters=[ { "Name": "attachment.instance-id", "Values": [instance_metadata["InstanceId"]], }, ] )["Volumes"] # If disks is None then use a list of all Ebs volumes attached to the instance requested_volumes = [] if disks is None: disks = [ device["Ebs"]["VolumeId"] for device in instance_metadata["BlockDeviceMappings"] if "Ebs" in device ] # For each requested volume, look it up in the describe_volumes output using # _lookup_volume which will handle both volume IDs and volume names for volume_identifier in disks: volume = self._lookup_volume(attached_volumes, volume_identifier) if volume is not None: attachment_metadata = None for attachment in volume["Attachments"]: if attachment["InstanceId"] == instance_metadata["InstanceId"]: attachment_metadata = attachment break if attachment_metadata is not None: # Ignore the root volume if ( attachment_metadata["Device"] == instance_metadata["RootDeviceName"] ): continue snapshot_id = None if "SnapshotId" in volume and volume["SnapshotId"] != "": snapshot_id = volume["SnapshotId"] requested_volumes.append( { "identifier": volume_identifier, "attachment_metadata": attachment_metadata, "source_snapshot": snapshot_id, } ) return requested_volumes def _create_snapshot(self, backup_info, volume_name, volume_id): """ Create a snapshot of an EBS volume in AWS. Unlike its counterparts in AzureCloudSnapshotInterface and GcpCloudSnapshotInterface, this function does not wait for the snapshot to enter a successful completed state and instead relies on the calling code to perform any necessary waiting. :param barman.infofile.LocalBackupInfo backup_info: Backup information. :param str volume_name: The user-supplied identifier for the volume. Used when creating the snapshot name. :param str volume_id: The AWS volume ID. Used when calling the AWS API to create the snapshot. :rtype: (str, dict) :return: The snapshot name and the snapshot metadata returned by AWS. """ snapshot_name = "%s-%s" % ( volume_name, backup_info.backup_id.lower(), ) logging.info( "Taking snapshot '%s' of disk '%s' (%s)", snapshot_name, volume_name, volume_id, ) tags = [ {"Key": "Name", "Value": snapshot_name}, ] if self.tags is not None: for key, value in self.tags: tags.append({"Key": key, "Value": value}) resp = self.ec2_client.create_snapshot( TagSpecifications=[ { "ResourceType": "snapshot", "Tags": tags, } ], VolumeId=volume_id, ) if resp["State"] == "error": raise CloudProviderError( "Snapshot '{}' failed: {}".format(snapshot_name, resp) ) return snapshot_name, resp def take_snapshot_backup(self, backup_info, instance_identifier, volumes): """ Take a snapshot backup for the named instance. Creates a snapshot for each named disk and saves the required metadata to backup_info.snapshots_info as an AwsSnapshotsInfo object. :param barman.infofile.LocalBackupInfo backup_info: Backup information. :param str instance_identifier: The instance ID or name of the VM instance to which the disks to be backed up are attached. :param dict[str,barman.cloud_providers.aws_s3.AwsVolumeMetadata] volumes: Metadata describing the volumes to be backed up. """ instance_metadata = self._get_instance_metadata(instance_identifier) attachment_metadata = instance_metadata["BlockDeviceMappings"] snapshots = [] for volume_identifier, volume_metadata in volumes.items(): attached_volumes = [ v for v in attachment_metadata if v["Ebs"]["VolumeId"] == volume_metadata.id ] if len(attached_volumes) == 0: raise SnapshotBackupException( "Disk %s not attached to instance %s" % (volume_identifier, instance_identifier) ) assert len(attached_volumes) == 1 snapshot_name, snapshot_resp = self._create_snapshot( backup_info, volume_identifier, volume_metadata.id ) # Apply lock on snapshot if lock mode is specified if self.lock_mode: self._lock_snapshot( snapshot_resp["SnapshotId"], self.lock_mode, self.lock_duration, self.lock_cool_off_period, self.lock_expiration_date, ) snapshots.append( AwsSnapshotMetadata( snapshot_id=snapshot_resp["SnapshotId"], snapshot_name=snapshot_name, snapshot_lock_mode=self.lock_mode, device_name=attached_volumes[0]["DeviceName"], mount_options=volume_metadata.mount_options, mount_point=volume_metadata.mount_point, ) ) # Await completion of all snapshots using a boto3 waiter. This will call # `describe_snapshots` every 15 seconds until all snapshot IDs are in a # successful state. If the successful state is not reached after the maximum # number of attempts (default: 40) then a WaiterError is raised. snapshot_ids = [snapshot.identifier for snapshot in snapshots] logging.info("Waiting for completion of snapshots: %s", ", ".join(snapshot_ids)) waiter = self.ec2_client.get_waiter("snapshot_completed") waiter.wait( SnapshotIds=snapshot_ids, WaiterConfig=self._get_waiter_config(), ) backup_info.snapshots_info = AwsSnapshotsInfo( snapshots=snapshots, region=self.region, # All snapshots will have the same OwnerId so we get it from the last # snapshot response. account_id=snapshot_resp["OwnerId"], ) def _lock_snapshot( self, snapshot_id, lock_mode, lock_duration, lock_cool_off_period, lock_expiration_date, ): lock_snapshot_default_args = {"LockMode": lock_mode, "SnapshotId": snapshot_id} if lock_duration: lock_snapshot_default_args["LockDuration"] = lock_duration if lock_cool_off_period: lock_snapshot_default_args["CoolOffPeriod"] = lock_cool_off_period if lock_expiration_date: lock_snapshot_default_args["ExpirationDate"] = lock_expiration_date resp = self.ec2_client.lock_snapshot(**lock_snapshot_default_args) _output = {} for key, value in resp.items(): if key != "ResponseMetadata": if isinstance(value, datetime): value = value.isoformat() _output[key] = value logging.info("Snapshot locked: \n%s" % json.dumps(_output, indent=4)) def _delete_snapshot(self, snapshot_id): """ Delete the specified snapshot. :param str snapshot_id: The ID of the snapshot to be deleted. """ try: self.ec2_client.delete_snapshot(SnapshotId=snapshot_id) except ClientError as exc: error_code = exc.response["Error"]["Code"] # If the snapshot could not be found then deletion is considered successful # otherwise we raise a CloudProviderError if error_code == "InvalidSnapshot.NotFound": logging.warning("Snapshot {} could not be found".format(snapshot_id)) elif error_code == "SnapshotLocked": raise SystemExit( "Locked snapshot: %s.\n" "Before deleting a snapshot, please ensure that it is not locked " "or that the lock has expired." % snapshot_id, ) else: raise CloudProviderError( "Deletion of snapshot %s failed with error code %s: %s" % (snapshot_id, error_code, exc.response["Error"]) ) logging.info("Snapshot %s deleted", snapshot_id) def delete_snapshot_backup(self, backup_info): """ Delete all snapshots for the supplied backup. :param barman.infofile.LocalBackupInfo backup_info: Backup information. """ for snapshot in backup_info.snapshots_info.snapshots: logging.info( "Deleting snapshot '%s' for backup %s", snapshot.identifier, backup_info.backup_id, ) self._delete_snapshot(snapshot.identifier) def get_attached_volumes( self, instance_identifier, disks=None, fail_on_missing=True ): """ Returns metadata for the non-root volumes attached to this instance. Queries AWS for metadata relating to the volumes attached to the named instance and returns a dict of `VolumeMetadata` objects, keyed by volume identifier. The volume identifier will be either: - The value supplied in the disks parameter, which can be either the AWS assigned volume ID or a name which corresponds to a unique `Name` tag assigned to a volume. - The AWS assigned volume ID, if the disks parameter is unused. If the optional disks parameter is supplied then this method returns metadata for the disks in the supplied list only. If fail_on_missing is set to True then a SnapshotBackupException is raised if any of the supplied disks are not found to be attached to the instance. If the disks parameter is not supplied then this method returns a VolumeMetadata object for every non-root disk attached to this instance. :param str instance_identifier: Either an instance ID or the name of the VM instance to which the disks are attached. :param list[str]|None disks: A list containing either the volume IDs or names of disks backed up. :param bool fail_on_missing: Fail with a SnapshotBackupException if any specified disks are not attached to the instance. :rtype: dict[str, VolumeMetadata] :return: A dict where the key is the volume identifier and the value is the device path for that disk on the specified instance. """ instance_metadata = self._get_instance_metadata(instance_identifier) requested_volumes = self._get_requested_volumes(instance_metadata, disks) attached_volumes = {} for requested_volume in requested_volumes: attached_volumes[requested_volume["identifier"]] = AwsVolumeMetadata( requested_volume["attachment_metadata"], virtualization_type=instance_metadata["VirtualizationType"], source_snapshot=requested_volume["source_snapshot"], ) if disks is not None and fail_on_missing: unattached_volumes = [] for disk_identifier in disks: if disk_identifier not in attached_volumes: unattached_volumes.append(disk_identifier) if len(unattached_volumes) > 0: raise SnapshotBackupException( "Disks not attached to instance {}: {}".format( instance_identifier, ", ".join(unattached_volumes) ) ) return attached_volumes def instance_exists(self, instance_identifier): """ Determine whether the instance exists. :param str instance_identifier: A string identifying the VM instance to be checked. Can be either an instance ID or a name. If a name is provided it is expected to match the value of a `Name` tag for a single EC2 instance. :rtype: bool :return: True if the named instance exists, False otherwise. """ try: self._get_instance_metadata(instance_identifier) except SnapshotInstanceNotFoundException: return False return True class AwsVolumeMetadata(VolumeMetadata): """ Specialization of VolumeMetadata for AWS EBS volumes. This class uses the device name obtained from the AWS API together with the virtualization type of the VM to which it is attached in order to resolve the mount point and mount options for the volume. """ def __init__( self, attachment_metadata=None, virtualization_type=None, source_snapshot=None ): """ Creates an AwsVolumeMetadata instance using metadata obtained from the AWS API. :param dict attachment_metadata: An `Attachments` entry in the describe_volumes metadata for this volume. :param str virtualization_type: The type of virtualzation used by the VM to which this volume is attached - either "hvm" or "paravirtual". :param str source_snapshot: The snapshot ID of the source snapshot from which volume was created. """ super(AwsVolumeMetadata, self).__init__() # The `id` property is used to store the volume ID so that we always have a # reference to the canonical ID of the volume. This is essential when creating # snapshots via the AWS API. self.id = None self._device_name = None self._virtualization_type = virtualization_type self._source_snapshot = source_snapshot if attachment_metadata: if "Device" in attachment_metadata: self._device_name = attachment_metadata["Device"] if "VolumeId" in attachment_metadata: self.id = attachment_metadata["VolumeId"] def resolve_mounted_volume(self, cmd): """ Resolve the mount point and mount options using shell commands. Uses `findmnt` to find the mount point and options for this volume by building a list of candidate device names and checking each one. Candidate device names are: - The device name reported by the AWS API. - A subsitution of the device name depending on virtualization type, with the same trailing letter. This is based on information provided by AWS about device renaming in EC2: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html :param UnixLocalCommand cmd: An object which can be used to run shell commands on a local (or remote, via the UnixRemoteCommand subclass) instance. """ if self._device_name is None: raise SnapshotBackupException( "Cannot resolve mounted volume: device name unknown" ) # Determine a list of candidate device names device_names = [self._device_name] device_prefix = "/dev/sd" if self._virtualization_type == "hvm": if self._device_name.startswith(device_prefix): device_names.append( self._device_name.replace(device_prefix, "/dev/xvd") ) elif self._virtualization_type == "paravirtual": if self._device_name.startswith(device_prefix): device_names.append(self._device_name.replace(device_prefix, "/dev/hd")) # Try to find the device name reported by the EC2 API for candidate_device in device_names: try: mount_point, mount_options = cmd.findmnt(candidate_device) if mount_point is not None: self._mount_point = mount_point self._mount_options = mount_options return except CommandException as e: raise SnapshotBackupException( "Error finding mount point for device path %s: %s" % (self._device_name, e) ) raise SnapshotBackupException( "Could not find device %s at any mount point" % self._device_name ) @property def source_snapshot(self): """ An identifier which can reference the snapshot via the cloud provider. :rtype: str :return: The snapshot ID """ return self._source_snapshot class AwsSnapshotMetadata(SnapshotMetadata): """ Specialization of SnapshotMetadata for AWS EBS snapshots. Stores the device_name, snapshot_id, snapshot_name and snapshot_lock_mode in the provider-specific field. """ _provider_fields = ( "device_name", "snapshot_id", "snapshot_name", "snapshot_lock_mode", ) def __init__( self, mount_options=None, mount_point=None, device_name=None, snapshot_id=None, snapshot_name=None, snapshot_lock_mode=None, ): """ Constructor saves additional metadata for AWS snapshots. :param str mount_options: The mount options used for the source disk at the time of the backup. :param str mount_point: The mount point of the source disk at the time of the backup. :param str device_name: The device name used in the AWS API. :param str snapshot_id: The snapshot ID used in the AWS API. :param str snapshot_name: The snapshot name stored in the `Name` tag. :param str snapshot_lock_mode: The mode with which the snapshot has been locked (``governance`` or ``compliance``), if set. :param str project: The AWS project name. """ super(AwsSnapshotMetadata, self).__init__(mount_options, mount_point) self.device_name = device_name self.snapshot_id = snapshot_id self.snapshot_name = snapshot_name self.snapshot_lock_mode = snapshot_lock_mode @property def identifier(self): """ An identifier which can reference the snapshot via the cloud provider. :rtype: str :return: The snapshot ID. """ return self.snapshot_id class AwsSnapshotsInfo(SnapshotsInfo): """ Represents the snapshots_info field for AWS EBS snapshots. """ _provider_fields = ( "account_id", "region", ) _snapshot_metadata_cls = AwsSnapshotMetadata def __init__(self, snapshots=None, account_id=None, region=None): """ Constructor saves the list of snapshots if it is provided. :param list[SnapshotMetadata] snapshots: A list of metadata objects for each snapshot. :param str account_id: The AWS account to which the snapshots belong, as reported by the `OwnerId` field in the snapshots metadata returned by AWS at snapshot creation time. :param str region: The AWS region in which snapshot resources are located. """ super(AwsSnapshotsInfo, self).__init__(snapshots) self.provider = "aws" self.account_id = account_id self.region = region barman-3.13.3/barman/cloud_providers/google_cloud_storage.py0000644000175100001660000007550515002177425022444 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import logging import os import posixpath from barman.clients.cloud_compression import decompress_to_file from barman.cloud import ( DEFAULT_DELIMITER, CloudInterface, CloudProviderError, CloudSnapshotInterface, DecompressingStreamingIO, SnapshotMetadata, SnapshotsInfo, VolumeMetadata, ) from barman.exceptions import CommandException, SnapshotBackupException try: # Python 3.x from urllib.parse import urlparse except ImportError: # Python 2.x from urlparse import urlparse try: from google.api_core.exceptions import Conflict, GoogleAPIError, NotFound from google.cloud import storage except ImportError: raise SystemExit("Missing required python module: google-cloud-storage") _logger = logging.getLogger(__name__) BASE_URL = "https://console.cloud.google.com/storage/browser/" class GoogleCloudInterface(CloudInterface): """ This class implements CloudInterface for GCS with the scope of using JSON API storage client documentation: https://googleapis.dev/python/storage/latest/client.html JSON API documentation: https://cloud.google.com/storage/docs/json_api/v1/objects """ # This implementation uses JSON API . does not support real parallel upload. # <> MAX_CHUNKS_PER_FILE = 1 # Since there is only on chunk min size is the same as max archive size MIN_CHUNK_SIZE = 1 << 40 # https://cloud.google.com/storage/docs/json_api/v1/objects/insert # Google json api permit a maximum of 5TB per file # This is a hard limit, while our upload procedure can go over the specified # MAX_ARCHIVE_SIZE - so we set a maximum of 1TB per file MAX_ARCHIVE_SIZE = 1 << 40 MAX_DELETE_BATCH_SIZE = 100 def __init__( self, url, jobs=1, tags=None, delete_batch_size=None, kms_key_name=None ): """ Create a new Google cloud Storage interface given the supplied account url :param str url: Full URL of the cloud destination/source (ex: ) :param int jobs: How many sub-processes to use for asynchronous uploading, defaults to 1. :param List[tuple] tags: List of tags as k,v tuples to be added to all uploaded objects :param int|None delete_batch_size: the maximum number of objects to be deleted in a single request :param str|None kms_key_name: the name of the KMS key which should be used for encrypting the uploaded data in GCS """ self.bucket_name, self.path = self._parse_url(url) super(GoogleCloudInterface, self).__init__( url=url, jobs=jobs, tags=tags, delete_batch_size=delete_batch_size, ) self.kms_key_name = kms_key_name self.bucket_exists = None self._reinit_session() @staticmethod def _parse_url(url): """ Parse url and return bucket name and path. Raise ValueError otherwise. """ if not url.startswith(BASE_URL) and not url.startswith("gs://"): msg = "Google cloud storage URL {} is malformed. Expected format are '{}' or '{}'".format( url, os.path.join(BASE_URL, "bucket-name/some/path"), "gs://bucket-name/some/path", ) raise ValueError(msg) gs_url = url.replace(BASE_URL, "gs://") parsed_url = urlparse(gs_url) if not parsed_url.netloc: raise ValueError( "Google cloud storage URL {} is malformed. Bucket name not found".format( url ) ) return parsed_url.netloc, parsed_url.path.strip("/") def _reinit_session(self): """ Create a new session Creates a client using "GOOGLE_APPLICATION_CREDENTIALS" env. An error will be raised if the variable is missing. """ self.client = storage.Client() self.container_client = self.client.bucket(self.bucket_name) def test_connectivity(self): """ Test gcs connectivity by trying to access a container """ try: # We are not even interested in the existence of the bucket, # we just want to see if google cloud storage is reachable. self.bucket_exists = self._check_bucket_existence() return True except GoogleAPIError as exc: logging.error("Can't connect to cloud provider: %s", exc) return False def _check_bucket_existence(self): """ Check google bucket :return: True if the container exists, False otherwise :rtype: bool """ return self.container_client.exists() def _create_bucket(self): """ Create the bucket in cloud storage It will try to create the bucket according to credential provided with 'GOOGLE_APPLICATION_CREDENTIALS' env. This imply the Bucket creation requires following gcsBucket access: 'storage.buckets.create'. Storage Admin role is suited for that. It is advised to have the bucket already created. Bucket creation can use a lot of parameters (region, project, dataclass, access control ...). Barman cloud does not provide a way to customise this creation and will use only bucket for creation . You can check detailed documentation here to learn more about default values https://googleapis.dev/python/storage/latest/client.html -> create_bucket """ try: self.client.create_bucket(self.container_client) except Conflict as e: logging.warning("It seems there was a Conflict creating bucket.") logging.warning(e.message) logging.warning("The bucket already exist, so we continue.") def list_bucket(self, prefix="", delimiter=DEFAULT_DELIMITER): """ List bucket content in a directory manner :param str prefix: Prefix used to filter blobs :param str delimiter: Delimiter, used with prefix to emulate hierarchy :return: List of objects and dirs right under the prefix :rtype: List[str] """ logging.debug("list_bucket: {}, {}".format(prefix, delimiter)) blobs = self.client.list_blobs( self.container_client, prefix=prefix, delimiter=delimiter ) objects = list(map(lambda blob: blob.name, blobs)) dirs = list(blobs.prefixes) logging.debug("objects {}".format(objects)) logging.debug("dirs {}".format(dirs)) return objects + dirs def download_file(self, key, dest_path, decompress): """ Download a file from cloud storage :param str key: The key identifying the file to download :param str dest_path: Where to put the destination file :param str|None decompress: Compression scheme to use for decompression """ logging.debug("GCS.download_file") blob = storage.Blob(key, self.container_client) with open(dest_path, "wb") as dest_file: if decompress is None: self.client.download_blob_to_file(blob, dest_file) return with blob.open(mode="rb") as blob_reader: decompress_to_file(blob_reader, dest_file, decompress) def remote_open(self, key, decompressor=None): """ Open a remote object in cloud storage and returns a readable stream :param str key: The key identifying the object to open :param barman.clients.cloud_compression.ChunkedCompressor decompressor: A ChunkedCompressor object which will be used to decompress chunks of bytes as they are read from the stream :return: google.cloud.storage.fileio.BlobReader | DecompressingStreamingIO | None A file-like object from which the stream can be read or None if the key does not exist """ logging.debug("GCS.remote_open") blob = storage.Blob(key, self.container_client) if not blob.exists(): logging.debug("Key: {} does not exist".format(key)) return None blob_reader = blob.open("rb") if decompressor: return DecompressingStreamingIO(blob_reader, decompressor) return blob_reader def upload_fileobj(self, fileobj, key, override_tags=None): """ Synchronously upload the content of a file-like object to a cloud key :param fileobj IOBase: File-like object to upload :param str key: The key to identify the uploaded object :param List[tuple] override_tags: List of tags as k,v tuples to be added to the uploaded object """ tags = override_tags or self.tags logging.debug("upload_fileobj to {}".format(key)) extra_args = {} if self.kms_key_name is not None: extra_args["kms_key_name"] = self.kms_key_name blob = self.container_client.blob(key, **extra_args) if tags is not None: blob.metadata = dict(tags) logging.debug("blob initiated") try: blob.upload_from_file(fileobj) except GoogleAPIError as e: logging.error(type(e)) logging.error(e) raise e def create_multipart_upload(self, key): """ JSON API does not allow this kind of multipart. https://cloud.google.com/storage/docs/uploads-downloads#uploads Closest solution is Parallel composite uploads. It is implemented in gsutil. It basically behave as follow: * file to upload is split in chunks * each chunk is sent to a specific path * when all chunks ar uploaded, compose call will assemble them into one file * chunk files can then be deleted For now parallel upload is a simple upload. :param key: The key to use in the cloud service :return: The multipart upload metadata :rtype: dict[str, str]|None """ return [] def _upload_part(self, upload_metadata, key, body, part_number): """ Upload a file The part metadata will included in a list of metadata for all parts of the upload which is passed to the _complete_multipart_upload method. :param dict upload_metadata: Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3 :param str key: The key to use in the cloud service :param object body: A stream-like object to upload :param int part_number: Part number, starting from 1 :return: The part metadata :rtype: dict[str, None|str] """ self.upload_fileobj(body, key) return { "PartNumber": part_number, } def _complete_multipart_upload(self, upload_metadata, key, parts_metadata): """ Finish a certain multipart upload There is nothing to do here as we are not using multipart. :param dict upload_metadata: Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3 :param str key: The key to use in the cloud service :param List[dict] parts_metadata: The list of metadata for the parts composing the multipart upload. Each part is guaranteed to provide a PartNumber and may optionally contain additional metadata returned by the cloud provider such as ETags. """ pass def _abort_multipart_upload(self, upload_metadata, key): """ Abort a certain multipart upload The implementation of this method should clean up any dangling resources left by the incomplete upload. :param dict upload_metadata: Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3 :param str key: The key to use in the cloud service """ # Probably delete things here in case it has already been uploaded ? # Maybe catch some exceptions like file not found (equivalent) try: self.delete_objects(key) except GoogleAPIError as e: logging.error(e) raise e def _delete_objects_batch(self, paths): """ Delete the objects at the specified paths. The maximum possible number of calls in a batch is 100. :param List[str] paths: """ super(GoogleCloudInterface, self)._delete_objects_batch(paths) failures = {} with self.client.batch(): for path in list(set(paths)): try: blob = self.container_client.blob(path) blob.delete() except GoogleAPIError as e: failures[path] = [str(e.__class__), e.__str__()] if failures: logging.error(failures) raise CloudProviderError() def get_prefixes(self, prefix): """ Return only the common prefixes under the supplied prefix. :param str prefix: The object key prefix under which the common prefixes will be found. :rtype: Iterator[str] :return: A list of unique prefixes immediately under the supplied prefix. """ raise NotImplementedError() def delete_under_prefix(self, prefix): """ Delete all objects under the specified prefix. :param str prefix: The object key prefix under which all objects should be deleted. """ raise NotImplementedError() def import_google_cloud_compute(): """ Import and return the google.cloud.compute module. This particular import happens in a function so that it can be deferred until needed while still allowing tests to easily mock the library. """ try: from google.cloud import compute except ImportError: raise SystemExit("Missing required python module: google-cloud-compute") return compute class GcpCloudSnapshotInterface(CloudSnapshotInterface): """ Implementation of ClourSnapshotInterface for persistend disk snapshots as implemented in Google Cloud Platform as documented at: https://cloud.google.com/compute/docs/disks/create-snapshots """ _required_config_for_backup = CloudSnapshotInterface._required_config_for_backup + ( "gcp_zone", ) _required_config_for_restore = ( CloudSnapshotInterface._required_config_for_restore + ("gcp_zone",) ) DEVICE_PREFIX = "/dev/disk/by-id/google-" def __init__(self, project, zone=None): """ Imports the google cloud compute library and creates the clients necessary for creating and managing snapshots. :param str project: The name of the GCP project to which all resources related to the snapshot backups belong. :param str|None zone: The zone in which resources accessed through this snapshot interface reside. """ if project is None: raise TypeError("project cannot be None") self.project = project self.zone = zone # The import of this module is deferred until this constructor so that it # does not become a spurious dependency of the main cloud interface. Doing # so would break backup to GCS for anyone unable to install # google-cloud-compute (which includes anyone using python 2.7). compute = import_google_cloud_compute() self.client = compute.SnapshotsClient() self.disks_client = compute.DisksClient() self.instances_client = compute.InstancesClient() def _get_instance_metadata(self, instance_name): """ Retrieve the metadata for the named instance in the specified zone. :rtype: google.cloud.compute_v1.types.Instance :return: An object representing the compute instance. """ try: return self.instances_client.get( instance=instance_name, zone=self.zone, project=self.project, ) except NotFound: raise SnapshotBackupException( "Cannot find instance with name %s in zone %s for project %s" % (instance_name, self.zone, self.project) ) def _get_disk_metadata(self, disk_name): """ Retrieve the metadata for the named disk in the specified zone. :rtype: google.cloud.compute_v1.types.Disk :return: An object representing the disk. """ try: return self.disks_client.get( disk=disk_name, zone=self.zone, project=self.project ) except NotFound: raise SnapshotBackupException( "Cannot find disk with name %s in zone %s for project %s" % (disk_name, self.zone, self.project) ) def _take_snapshot(self, backup_info, disk_zone, disk_name): """ Take a snapshot of a persistent disk in GCP. :param barman.infofile.LocalBackupInfo backup_info: Backup information. :param str disk_zone: The zone in which the disk resides. :param str disk_name: The name of the source disk for the snapshot. :rtype: str :return: The name used to reference the snapshot with GCP. """ snapshot_name = "%s-%s" % ( disk_name, backup_info.backup_id.lower(), ) _logger.info("Taking snapshot '%s' of disk '%s'", snapshot_name, disk_name) resp = self.client.insert( { "project": self.project, "snapshot_resource": { "name": snapshot_name, "source_disk": "projects/%s/zones/%s/disks/%s" % ( self.project, disk_zone, disk_name, ), }, } ) _logger.info("Waiting for snapshot '%s' completion", snapshot_name) resp.result() if resp.error_code: raise CloudProviderError( "Snapshot '%s' failed with error code %s: %s" % (snapshot_name, resp.error_code, resp.error_message) ) if resp.warnings: prefix = "Warnings encountered during snapshot %s: " % snapshot_name _logger.warning( prefix + ", ".join( "%s:%s" % (warning.code, warning.message) for warning in resp.warnings ) ) _logger.info("Snapshot '%s' completed", snapshot_name) return snapshot_name def take_snapshot_backup(self, backup_info, instance_name, volumes): """ Take a snapshot backup for the named instance. Creates a snapshot for each named disk and saves the required metadata to backup_info.snapshots_info as a GcpSnapshotsInfo object. :param barman.infofile.LocalBackupInfo backup_info: Backup information. :param str instance_name: The name of the VM instance to which the disks to be backed up are attached. :param dict[str,barman.cloud.VolumeMetadata] volumes: Metadata for the volumes to be backed up. """ instance_metadata = self._get_instance_metadata(instance_name) snapshots = [] for disk_name, volume_metadata in volumes.items(): snapshot_name = self._take_snapshot(backup_info, self.zone, disk_name) # Save useful metadata attachment_metadata = [ d for d in instance_metadata.disks if d.source.endswith(disk_name) ][0] snapshots.append( GcpSnapshotMetadata( snapshot_name=snapshot_name, snapshot_project=self.project, device_name=attachment_metadata.device_name, mount_options=volume_metadata.mount_options, mount_point=volume_metadata.mount_point, ) ) # Add snapshot metadata to BackupInfo backup_info.snapshots_info = GcpSnapshotsInfo( project=self.project, snapshots=snapshots ) def _delete_snapshot(self, snapshot_name): """ Delete the specified snapshot. :param str snapshot_name: The short name used to reference the snapshot within GCP. """ try: resp = self.client.delete( { "project": self.project, "snapshot": snapshot_name, } ) except NotFound: # If the snapshot cannot be found then deletion is considered successful return resp.result() if resp.error_code: raise CloudProviderError( "Deletion of snapshot %s failed with error code %s: %s" % (snapshot_name, resp.error_code, resp.error_message) ) if resp.warnings: prefix = "Warnings encountered during deletion of %s: " % snapshot_name _logger.warning( prefix + ", ".join( "%s:%s" % (warning.code, warning.message) for warning in resp.warnings ) ) _logger.info("Snapshot %s deleted", snapshot_name) def delete_snapshot_backup(self, backup_info): """ Delete all snapshots for the supplied backup. :param barman.infofile.LocalBackupInfo backup_info: Backup information. """ for snapshot in backup_info.snapshots_info.snapshots: _logger.info( "Deleting snapshot '%s' for backup %s", snapshot.identifier, backup_info.backup_id, ) self._delete_snapshot(snapshot.identifier) def get_attached_volumes(self, instance_name, disks=None, fail_on_missing=True): """ Returns metadata for the volumes attached to this instance. Queries GCP for metadata relating to the volumes attached to the named instance and returns a dict of `VolumeMetadata` objects, keyed by disk name. If the optional disks parameter is supplied then this method returns metadata for the disks in the supplied list only. If fail_on_missing is set to True then a SnapshotBackupException is raised if any of the supplied disks are not found to be attached to the instance. If the disks parameter is not supplied then this method returns a VolumeMetadata for all disks attached to this instance. :param str instance_name: The name of the VM instance to which the disks to be backed up are attached. :param list[str]|None disks: A list containing the names of disks to be backed up. :param bool fail_on_missing: Fail with a SnapshotBackupException if any specified disks are not attached to the instance. :rtype: dict[str, VolumeMetadata] :return: A dict of VolumeMetadata objects representing each volume attached to the instance, keyed by volume identifier. """ instance_metadata = self._get_instance_metadata(instance_name) attached_volumes = {} for attachment_metadata in instance_metadata.disks: disk_name = posixpath.split(urlparse(attachment_metadata.source).path)[-1] if disks and disk_name not in disks: continue if disk_name == "": raise SnapshotBackupException( "Could not parse disk name for source %s attached to instance %s" % (attachment_metadata.source, instance_name) ) assert disk_name not in attached_volumes disk_metadata = self._get_disk_metadata(disk_name) attached_volumes[disk_name] = GcpVolumeMetadata( attachment_metadata, disk_metadata, ) # Check all requested disks were found and complain if necessary if disks is not None and fail_on_missing: unattached_disks = [] for disk_name in disks: if disk_name not in attached_volumes: # Verify the disk definitely exists by fetching the metadata self._get_disk_metadata(disk_name) # Append to list of unattached disks unattached_disks.append(disk_name) if len(unattached_disks) > 0: raise SnapshotBackupException( "Disks not attached to instance %s: %s" % (instance_name, ", ".join(unattached_disks)) ) return attached_volumes def instance_exists(self, instance_name): """ Determine whether the named instance exists. :param str instance_name: The name of the VM instance to which the disks to be backed up are attached. :rtype: bool :return: True if the named instance exists, False otherwise. """ try: self.instances_client.get( instance=instance_name, zone=self.zone, project=self.project, ) except NotFound: return False return True class GcpVolumeMetadata(VolumeMetadata): """ Specialization of VolumeMetadata for GCP persistent disks. This class uses the device name obtained from the GCP API to determine the full path to the device on the compute instance. This path is then resolved to the mount point using findmnt. """ def __init__(self, attachment_metadata=None, disk_metadata=None): """ Creates a GcpVolumeMetadata instance using metadata obtained from the GCP API. Uses attachment_metadata to obtain the device name and resolves this to the full device path on the instance using a documented prefix. Uses disk_metadata to obtain the source snapshot name, if such a snapshot exists. :param google.cloud.compute_v1.types.AttachedDisk attachment_metadata: An object representing the disk as attached to the instance. :param google.cloud.compute_v1.types.Disk disk_metadata: An object representing the disk. """ super(GcpVolumeMetadata, self).__init__() self._snapshot_name = None self._device_path = None if ( attachment_metadata is not None and attachment_metadata.device_name is not None ): self._device_path = ( GcpCloudSnapshotInterface.DEVICE_PREFIX + attachment_metadata.device_name ) if disk_metadata is not None: if disk_metadata.source_snapshot is not None: attached_snapshot_name = posixpath.split( urlparse(disk_metadata.source_snapshot).path )[-1] else: attached_snapshot_name = "" if attached_snapshot_name != "": self._snapshot_name = attached_snapshot_name def resolve_mounted_volume(self, cmd): """ Resolve the mount point and mount options using shell commands. Uses findmnt to retrieve the mount point and mount options for the device path at which this volume is mounted. """ if self._device_path is None: raise SnapshotBackupException( "Cannot resolve mounted volume: Device path unknown" ) try: mount_point, mount_options = cmd.findmnt(self._device_path) except CommandException as e: raise SnapshotBackupException( "Error finding mount point for device %s: %s" % (self._device_path, e) ) if mount_point is None: raise SnapshotBackupException( "Could not find device %s at any mount point" % self._device_path ) self._mount_point = mount_point self._mount_options = mount_options @property def source_snapshot(self): """ An identifier which can reference the snapshot via the cloud provider. :rtype: str :return: The snapshot short name. """ return self._snapshot_name class GcpSnapshotMetadata(SnapshotMetadata): """ Specialization of SnapshotMetadata for GCP persistent disk snapshots. Stores the device_name, snapshot_name and snapshot_project in the provider-specific field and uses the short snapshot name as the identifier. """ _provider_fields = ("device_name", "snapshot_name", "snapshot_project") def __init__( self, mount_options=None, mount_point=None, device_name=None, snapshot_name=None, snapshot_project=None, ): """ Constructor saves additional metadata for GCP snapshots. :param str mount_options: The mount options used for the source disk at the time of the backup. :param str mount_point: The mount point of the source disk at the time of the backup. :param str device_name: The short device name used in the GCP API. :param str snapshot_name: The short snapshot name used in the GCP API. :param str snapshot_project: The GCP project name. """ super(GcpSnapshotMetadata, self).__init__(mount_options, mount_point) self.device_name = device_name self.snapshot_name = snapshot_name self.snapshot_project = snapshot_project @property def identifier(self): """ An identifier which can reference the snapshot via the cloud provider. :rtype: str :return: The snapshot short name. """ return self.snapshot_name class GcpSnapshotsInfo(SnapshotsInfo): """ Represents the snapshots_info field for GCP persistent disk snapshots. """ _provider_fields = ("project",) _snapshot_metadata_cls = GcpSnapshotMetadata def __init__(self, snapshots=None, project=None): """ Constructor saves the list of snapshots if it is provided. :param list[SnapshotMetadata] snapshots: A list of metadata objects for each snapshot. :param str project: The GCP project name. """ super(GcpSnapshotsInfo, self).__init__(snapshots) self.provider = "gcp" self.project = project barman-3.13.3/barman/cloud_providers/__init__.py0000644000175100001660000003420015002177425020000 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see from barman.exceptions import BarmanException, ConfigurationException class CloudProviderUnsupported(BarmanException): """ Exception raised when an unsupported cloud provider is requested """ class CloudProviderOptionUnsupported(BarmanException): """ Exception raised when a supported cloud provider is given an unsupported option """ def _update_kwargs(kwargs, config, args): """ Helper which adds the attributes of config specified in args to the supplied kwargs dict if they exist. """ for arg in args: if arg in config: kwargs[arg] = getattr(config, arg) def _make_s3_cloud_interface(config, cloud_interface_kwargs): from barman.cloud_providers.aws_s3 import S3CloudInterface cloud_interface_kwargs.update( { "profile_name": config.aws_profile, "endpoint_url": config.endpoint_url, "read_timeout": config.read_timeout, } ) if "encryption" in config: cloud_interface_kwargs["encryption"] = config.encryption if "sse_kms_key_id" in config: if ( config.sse_kms_key_id is not None and "encryption" in config and config.encryption != "aws:kms" ): raise CloudProviderOptionUnsupported( 'Encryption type must be "aws:kms" if SSE KMS Key ID is specified' ) cloud_interface_kwargs["sse_kms_key_id"] = config.sse_kms_key_id return S3CloudInterface(**cloud_interface_kwargs) def _get_azure_credential(credential_type): if credential_type is None: return None try: from azure.identity import ( AzureCliCredential, DefaultAzureCredential, ManagedIdentityCredential, ) except ImportError: raise SystemExit("Missing required python module: azure-identity") supported_credentials = { "azure-cli": AzureCliCredential, "default": DefaultAzureCredential, "managed-identity": ManagedIdentityCredential, } try: return supported_credentials[credential_type] except KeyError: raise CloudProviderOptionUnsupported( "Unsupported credential: %s" % credential_type ) def _make_azure_cloud_interface(config, cloud_interface_kwargs): from barman.cloud_providers.azure_blob_storage import AzureCloudInterface _update_kwargs( cloud_interface_kwargs, config, ( "encryption_scope", "max_block_size", "max_concurrency", "max_single_put_size", ), ) if "azure_credential" in config: credential = _get_azure_credential(config.azure_credential) if credential is not None: cloud_interface_kwargs["credential"] = credential() return AzureCloudInterface(**cloud_interface_kwargs) def _make_google_cloud_interface(config, cloud_interface_kwargs): """ :param config: Not used yet :param cloud_interface_kwargs: common parameters :return: GoogleCloudInterface """ from barman.cloud_providers.google_cloud_storage import GoogleCloudInterface cloud_interface_kwargs["jobs"] = 1 if "kms_key_name" in config: if ( config.kms_key_name is not None and "snapshot_instance" in config and config.snapshot_instance is not None ): raise CloudProviderOptionUnsupported( "KMS key cannot be specified for snapshot backups" ) cloud_interface_kwargs["kms_key_name"] = config.kms_key_name return GoogleCloudInterface(**cloud_interface_kwargs) def get_cloud_interface(config): """ Factory function that creates CloudInterface for the specified cloud_provider :param: argparse.Namespace config :returns: A CloudInterface for the specified cloud_provider :rtype: CloudInterface """ cloud_interface_kwargs = { "url": config.source_url if "source_url" in config else config.destination_url } _update_kwargs( cloud_interface_kwargs, config, ("jobs", "tags", "delete_batch_size") ) if config.cloud_provider == "aws-s3": return _make_s3_cloud_interface(config, cloud_interface_kwargs) elif config.cloud_provider == "azure-blob-storage": return _make_azure_cloud_interface(config, cloud_interface_kwargs) elif config.cloud_provider == "google-cloud-storage": return _make_google_cloud_interface(config, cloud_interface_kwargs) else: raise CloudProviderUnsupported( "Unsupported cloud provider: %s" % config.cloud_provider ) def get_snapshot_interface(config): """ Factory function that creates CloudSnapshotInterface for the cloud provider specified in the supplied config. :param argparse.Namespace config: The backup options provided at the command line. :rtype: CloudSnapshotInterface :returns: A CloudSnapshotInterface for the specified snapshot_provider. """ if config.cloud_provider == "google-cloud-storage": from barman.cloud_providers.google_cloud_storage import ( GcpCloudSnapshotInterface, ) if config.gcp_project is None: raise ConfigurationException( "--gcp-project option must be set for snapshot backups " "when cloud provider is google-cloud-storage" ) return GcpCloudSnapshotInterface(config.gcp_project, config.gcp_zone) elif config.cloud_provider == "azure-blob-storage": from barman.cloud_providers.azure_blob_storage import ( AzureCloudSnapshotInterface, ) if config.azure_subscription_id is None: raise ConfigurationException( "--azure-subscription-id option must be set for snapshot " "backups when cloud provider is azure-blob-storage" ) return AzureCloudSnapshotInterface( config.azure_subscription_id, resource_group=config.azure_resource_group, credential=_get_azure_credential(config.azure_credential), ) elif config.cloud_provider == "aws-s3": from barman.cloud_providers.aws_s3 import AwsCloudSnapshotInterface args = [ config.aws_profile, config.aws_region, config.aws_await_snapshots_timeout, config.aws_snapshot_lock_mode, config.aws_snapshot_lock_duration, config.aws_snapshot_lock_cool_off_period, config.aws_snapshot_lock_expiration_date, config.tags, ] return AwsCloudSnapshotInterface(*args) else: raise CloudProviderUnsupported( "No snapshot provider for cloud provider: %s" % config.cloud_provider ) def get_snapshot_interface_from_server_config(server_config): """ Factory function that creates CloudSnapshotInterface for the snapshot provider specified in the supplied config. :param barman.config.Config server_config: The barman configuration object for a specific server. :rtype: CloudSnapshotInterface :returns: A CloudSnapshotInterface for the specified snapshot_provider. """ if server_config.snapshot_provider == "gcp": from barman.cloud_providers.google_cloud_storage import ( GcpCloudSnapshotInterface, ) gcp_project = server_config.gcp_project or server_config.snapshot_gcp_project if gcp_project is None: raise ConfigurationException( "gcp_project option must be set when snapshot_provider is gcp" ) gcp_zone = server_config.gcp_zone or server_config.snapshot_zone return GcpCloudSnapshotInterface(gcp_project, gcp_zone) elif server_config.snapshot_provider == "azure": from barman.cloud_providers.azure_blob_storage import ( AzureCloudSnapshotInterface, ) if server_config.azure_subscription_id is None: raise ConfigurationException( "azure_subscription_id option must be set when snapshot_provider " "is azure" ) return AzureCloudSnapshotInterface( server_config.azure_subscription_id, resource_group=server_config.azure_resource_group, credential=_get_azure_credential(server_config.azure_credential), ) elif server_config.snapshot_provider == "aws": from barman.cloud_providers.aws_s3 import AwsCloudSnapshotInterface return AwsCloudSnapshotInterface( server_config.aws_profile, server_config.aws_region, server_config.aws_await_snapshots_timeout, server_config.aws_snapshot_lock_mode, server_config.aws_snapshot_lock_duration, server_config.aws_snapshot_lock_cool_off_period, server_config.aws_snapshot_lock_expiration_date, ) else: raise CloudProviderUnsupported( "Unsupported snapshot provider: %s" % server_config.snapshot_provider ) def get_snapshot_interface_from_backup_info(backup_info, config=None): """ Factory function that creates CloudSnapshotInterface for the snapshot provider specified in the supplied backup info. :param barman.infofile.BackupInfo backup_info: The metadata for a specific backup. cloud provider. :param argparse.Namespace|barman.config.Config config: The backup options provided by the command line or the Barman configuration. :rtype: CloudSnapshotInterface :returns: A CloudSnapshotInterface for the specified snapshot provider. """ if backup_info.snapshots_info.provider == "gcp": from barman.cloud_providers.google_cloud_storage import ( GcpCloudSnapshotInterface, ) if backup_info.snapshots_info.project is None: raise BarmanException( "backup_info has snapshot provider 'gcp' but project is not set" ) gcp_zone = config is not None and config.gcp_zone or None return GcpCloudSnapshotInterface( backup_info.snapshots_info.project, gcp_zone, ) elif backup_info.snapshots_info.provider == "azure": from barman.cloud_providers.azure_blob_storage import ( AzureCloudSnapshotInterface, ) # When creating a snapshot interface for dealing with existing backups we use # the subscription ID from that backup and the resource group specified in # provider_args. This means that: # 1. Resources will always belong to the same subscription. # 2. Recovery resources can be in a different resource group to the one used # to create the backup. if backup_info.snapshots_info.subscription_id is None: raise ConfigurationException( "backup_info has snapshot provider 'azure' but " "subscription_id is not set" ) resource_group = None azure_credential = None if config is not None: if hasattr(config, "azure_resource_group"): resource_group = config.azure_resource_group if hasattr(config, "azure_credential"): azure_credential = config.azure_credential return AzureCloudSnapshotInterface( backup_info.snapshots_info.subscription_id, resource_group=resource_group, credential=_get_azure_credential(azure_credential), ) elif backup_info.snapshots_info.provider == "aws": from barman.cloud_providers.aws_s3 import AwsCloudSnapshotInterface # When creating a snapshot interface for existing backups we use the region # from the backup_info, unless a region is set in the config in which case the # config region takes precedence. region = None profile = None if config is not None: if hasattr(config, "aws_region"): region = config.aws_region if hasattr(config, "aws_profile"): profile = config.aws_profile if region is None: region = backup_info.snapshots_info.region return AwsCloudSnapshotInterface(profile, region) else: raise CloudProviderUnsupported( "Unsupported snapshot provider in backup info: %s" % backup_info.snapshots_info.provider ) def snapshots_info_from_dict(snapshots_info): """ Factory function which creates a SnapshotInfo object for the supplied dict of snapshot backup metadata. :param dict snapshots_info: Dictionary of snapshots info from a backup.info :rtype: SnapshotsInfo :return: A SnapshotInfo subclass for the snapshots provider listed in the `provider` field of the snapshots_info. """ if "provider" in snapshots_info and snapshots_info["provider"] == "gcp": from barman.cloud_providers.google_cloud_storage import GcpSnapshotsInfo return GcpSnapshotsInfo.from_dict(snapshots_info) elif "provider" in snapshots_info and snapshots_info["provider"] == "azure": from barman.cloud_providers.azure_blob_storage import AzureSnapshotsInfo return AzureSnapshotsInfo.from_dict(snapshots_info) elif "provider" in snapshots_info and snapshots_info["provider"] == "aws": from barman.cloud_providers.aws_s3 import AwsSnapshotsInfo return AwsSnapshotsInfo.from_dict(snapshots_info) else: raise CloudProviderUnsupported( "Unsupported snapshot provider in backup info: %s" % snapshots_info["provider"] ) barman-3.13.3/barman/version.py0000644000175100001660000000144615002177453014532 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module contains the current Barman version. """ __version__ = '3.13.3' barman-3.13.3/barman/compression.py0000644000175100001660000011604315002177425015405 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module is responsible to manage the compression features of Barman """ import binascii import bz2 import gzip import logging import lzma import shutil from abc import ABCMeta, abstractmethod, abstractproperty from contextlib import closing from distutils.version import LooseVersion as Version import barman.infofile from barman.command_wrappers import Command from barman.exceptions import ( CommandFailedException, CompressionException, CompressionIncompatibility, FileNotFoundException, ) from barman.fs import unix_command_factory from barman.utils import force_str, with_metaclass _logger = logging.getLogger(__name__) class CompressionManager(object): def __init__(self, config, path): """ :param config: barman.config.ServerConfig :param path: str """ self.config = config self.path = path self.unidentified_compression = None if self.config.compression == "custom": # If Barman is set to use the custom compression and no magic is # configured, it assumes that every unidentified file is custom # compressed. if self.config.custom_compression_magic is None: self.unidentified_compression = self.config.compression # If custom_compression_magic is set then we should not assume # unidentified files are custom compressed and should rely on the # magic for identification instead. elif isinstance(config.custom_compression_magic, str): # Since we know the custom compression magic we can now add it # to the class property. compression_registry["custom"].MAGIC = binascii.unhexlify( config.custom_compression_magic[2:] ) # Set the longest string needed to identify a compression schema. # This happens at instantiation time because we need to include the # custom_compression_magic from the config (if set). self.MAGIC_MAX_LENGTH = max( len(x.MAGIC or "") for x in compression_registry.values() ) def check(self, compression=None): """ This method returns True if the compression specified in the configuration file is present in the register, otherwise False """ if not compression: compression = self.config.compression if compression not in compression_registry: return False return True def get_default_compressor(self): """ Returns a new default compressor instance """ return self.get_compressor(self.config.compression) def get_compressor(self, compression): """ Returns a new compressor instance :param str compression: Compression name or none """ # Check if the requested compression mechanism is allowed if compression and self.check(compression): return compression_registry[compression]( config=self.config, compression=compression, path=self.path ) return None def get_wal_file_info(self, filename): """ Populate a WalFileInfo object taking into account the server configuration. Set compression to 'custom' if no compression is identified and Barman is configured to use custom compression. :param str filename: the path of the file to identify :rtype: barman.infofile.WalFileInfo """ return barman.infofile.WalFileInfo.from_file( filename, compression_manager=self, unidentified_compression=self.unidentified_compression, ) def identify_compression(self, filename): """ Try to guess the compression algorithm of a file :param str filename: the path of the file to identify :rtype: str """ # TODO: manage multiple decompression methods for the same # compression algorithm (e.g. what to do when gzip is detected? # should we use gzip or pigz?) with open(filename, "rb") as f: file_start = f.read(self.MAGIC_MAX_LENGTH) for file_type, cls in sorted(compression_registry.items()): if cls.validate(file_start): return file_type return None class Compressor(with_metaclass(ABCMeta, object)): """ Base class for all the compressors """ MAGIC = None def __init__(self, config, compression, path=None): """ :param config: barman.config.ServerConfig :param compression: str compression name :param path: str|None """ self.config = config self.compression = compression self.path = path @classmethod def validate(cls, file_start): """ Guess if the first bytes of a file are compatible with the compression implemented by this class :param file_start: a binary string representing the first few bytes of a file :rtype: bool """ return cls.MAGIC and file_start.startswith(cls.MAGIC) @abstractmethod def compress(self, src, dst): """ Abstract Method for compression method :param str src: source file path :param str dst: destination file path """ @abstractmethod def decompress(self, src, dst): """ Abstract method for decompression method :param str src: source file path :param str dst: destination file path """ class CommandCompressor(Compressor): """ Base class for compressors built on external commands """ def __init__(self, config, compression, path=None): """ :param config: barman.config.ServerConfig :param compression: str compression name :param path: str|None """ super(CommandCompressor, self).__init__(config, compression, path) self._compress = None self._decompress = None def compress(self, src, dst): """ Compress using the specific command defined in the subclass :param src: source file to compress :param dst: destination of the decompression """ return self._compress(src, dst) def decompress(self, src, dst): """ Decompress using the specific command defined in the subclass :param src: source file to decompress :param dst: destination of the decompression """ return self._decompress(src, dst) def _build_command(self, pipe_command): """ Build the command string and create the actual Command object :param pipe_command: the command used to compress/decompress :rtype: Command """ command = "barman_command(){ " command += pipe_command command += ' > "$2" < "$1"' command += ";}; barman_command" return Command(command, shell=True, check=True, path=self.path) class InternalCompressor(Compressor): """ Base class for compressors built on python libraries """ def compress(self, src, dst): """ Compress using the object defined in the subclass :param src: source file to compress :param dst: destination of the decompression """ try: with open(src, "rb") as istream: with closing(self._compressor(dst)) as ostream: shutil.copyfileobj(istream, ostream) except Exception as e: # you won't get more information from the compressors anyway raise CommandFailedException(dict(ret=None, err=force_str(e), out=None)) return 0 def decompress(self, src, dst): """ Decompress using the object defined in the subclass :param src: source file to decompress :param dst: destination of the decompression """ try: with closing(self._decompressor(src)) as istream: with open(dst, "wb") as ostream: shutil.copyfileobj(istream, ostream) except Exception as e: # you won't get more information from the compressors anyway raise CommandFailedException(dict(ret=None, err=force_str(e), out=None)) return 0 @abstractmethod def _decompressor(self, src): """ Abstract decompressor factory method :param src: source file path :return: a file-like readable decompressor object """ @abstractmethod def _compressor(self, dst): """ Abstract compressor factory method :param dst: destination file path :return: a file-like writable compressor object """ class GZipCompressor(CommandCompressor): """ Predefined compressor with GZip """ MAGIC = b"\x1f\x8b\x08" def __init__(self, config, compression, path=None): """ :param config: barman.config.ServerConfig :param compression: str compression name :param path: str|None """ super(GZipCompressor, self).__init__(config, compression, path) self._compress = self._build_command("gzip -c") self._decompress = self._build_command("gzip -c -d") class PyGZipCompressor(InternalCompressor): """ Predefined compressor that uses GZip Python libraries """ MAGIC = b"\x1f\x8b\x08" def __init__(self, config, compression, path=None): """ :param config: barman.config.ServerConfig :param compression: str compression name :param path: str|None """ super(PyGZipCompressor, self).__init__(config, compression, path) # Default compression level used in system gzip utility self._level = -1 # Z_DEFAULT_COMPRESSION constant of zlib def _compressor(self, name): return gzip.GzipFile(name, mode="wb", compresslevel=self._level) def _decompressor(self, name): return gzip.GzipFile(name, mode="rb") class PigzCompressor(CommandCompressor): """ Predefined compressor with Pigz Note that pigz on-disk is the same as gzip, so the MAGIC value of this class is the same """ MAGIC = b"\x1f\x8b\x08" def __init__(self, config, compression, path=None): """ :param config: barman.config.ServerConfig :param compression: str compression name :param path: str|None """ super(PigzCompressor, self).__init__(config, compression, path) self._compress = self._build_command("pigz -c") self._decompress = self._build_command("pigz -c -d") class BZip2Compressor(CommandCompressor): """ Predefined compressor with BZip2 """ MAGIC = b"\x42\x5a\x68" def __init__(self, config, compression, path=None): """ :param config: barman.config.ServerConfig :param compression: str compression name :param path: str|None """ super(BZip2Compressor, self).__init__(config, compression, path) self._compress = self._build_command("bzip2 -c") self._decompress = self._build_command("bzip2 -c -d") class PyBZip2Compressor(InternalCompressor): """ Predefined compressor with BZip2 Python libraries """ MAGIC = b"\x42\x5a\x68" def __init__(self, config, compression, path=None): """ :param config: barman.config.ServerConfig :param compression: str compression name :param path: str|None """ super(PyBZip2Compressor, self).__init__(config, compression, path) # Default compression level used in system gzip utility self._level = 9 def _compressor(self, name): return bz2.BZ2File(name, mode="wb", compresslevel=self._level) def _decompressor(self, name): return bz2.BZ2File(name, mode="rb") class XZCompressor(InternalCompressor): """ Predefined compressor with XZ Python library """ MAGIC = b"\xfd7zXZ\x00" def _compressor(self, dst): return lzma.open(dst, mode="wb") def _decompressor(self, src): return lzma.open(src, mode="rb") def _try_import_zstd(): try: import zstandard except ImportError: raise SystemExit("Missing required python module: zstandard") return zstandard class ZSTDCompressor(InternalCompressor): """ Predefined compressor with zstd """ MAGIC = b"(\xb5/\xfd" def __init__(self, config, compression, path=None): """ Constructor. :param config: barman.config.ServerConfig :param compression: str compression name :param path: str|None """ super(ZSTDCompressor, self).__init__(config, compression, path) self._zstd = None @property def zstd(self): if self._zstd is None: self._zstd = _try_import_zstd() return self._zstd def _compressor(self, dst): return self.zstd.ZstdCompressor().stream_writer(open(dst, mode="wb")) def _decompressor(self, src): return self.zstd.ZstdDecompressor().stream_reader(open(src, mode="rb")) def _try_import_lz4(): try: import lz4.frame except ImportError: raise SystemExit("Missing required python module: lz4") return lz4 class LZ4Compressor(InternalCompressor): """ Predefined compressor with lz4 """ MAGIC = b"\x04\x22\x4d\x18" def __init__(self, config, compression, path=None): """ Constructor. :param config: barman.config.ServerConfig :param compression: str compression name :param path: str|None """ super(LZ4Compressor, self).__init__(config, compression, path) self._lz4 = None @property def lz4(self): if self._lz4 is None: self._lz4 = _try_import_lz4() return self._lz4 def _compressor(self, dst): return self.lz4.frame.open(dst, mode="wb") def _decompressor(self, src): return self.lz4.frame.open(src, mode="rb") class CustomCompressor(CommandCompressor): """ Custom compressor """ def __init__(self, config, compression, path=None): """ :param config: barman.config.ServerConfig :param compression: str compression name :param path: str|None """ if config.custom_compression_filter is None or not isinstance( config.custom_compression_filter, str ): raise CompressionIncompatibility("custom_compression_filter") if config.custom_decompression_filter is None or not isinstance( config.custom_decompression_filter, str ): raise CompressionIncompatibility("custom_decompression_filter") super(CustomCompressor, self).__init__(config, compression, path) self._compress = self._build_command(config.custom_compression_filter) self._decompress = self._build_command(config.custom_decompression_filter) # a dictionary mapping all supported compression schema # to the class implementing it # WARNING: items in this dictionary are extracted using alphabetical order # It's important that gzip and bzip2 are positioned before their variants compression_registry = { "gzip": GZipCompressor, "pigz": PigzCompressor, "bzip2": BZip2Compressor, "pygzip": PyGZipCompressor, "pybzip2": PyBZip2Compressor, "xz": XZCompressor, "zstd": ZSTDCompressor, "lz4": LZ4Compressor, "custom": CustomCompressor, } def get_pg_basebackup_compression(server): """ Factory method which returns an instantiated PgBaseBackupCompression subclass for the backup_compression option in config for the supplied server. :param barman.server.Server server: the server for which the PgBaseBackupCompression should be constructed :return GZipPgBaseBackupCompression """ if server.config.backup_compression is None: return pg_base_backup_cfg = PgBaseBackupCompressionConfig( server.config.backup_compression, server.config.backup_compression_format, server.config.backup_compression_level, server.config.backup_compression_location, server.config.backup_compression_workers, ) base_backup_compression_option = None compression = None if server.config.backup_compression == GZipCompression.name: # Create PgBaseBackupCompressionOption base_backup_compression_option = GZipPgBaseBackupCompressionOption( pg_base_backup_cfg ) compression = GZipCompression(unix_command_factory()) if server.config.backup_compression == LZ4Compression.name: base_backup_compression_option = LZ4PgBaseBackupCompressionOption( pg_base_backup_cfg ) compression = LZ4Compression(unix_command_factory()) if server.config.backup_compression == ZSTDCompression.name: base_backup_compression_option = ZSTDPgBaseBackupCompressionOption( pg_base_backup_cfg ) compression = ZSTDCompression(unix_command_factory()) if server.config.backup_compression == NoneCompression.name: base_backup_compression_option = NonePgBaseBackupCompressionOption( pg_base_backup_cfg ) compression = NoneCompression(unix_command_factory()) if base_backup_compression_option is None or compression is None: # We got to the point where the compression is not handled raise CompressionException( "Barman does not support pg_basebackup compression: %s" % server.config.backup_compression ) return PgBaseBackupCompression( pg_base_backup_cfg, base_backup_compression_option, compression ) class PgBaseBackupCompressionConfig(object): """Should become a dataclass""" def __init__( self, backup_compression, backup_compression_format, backup_compression_level, backup_compression_location, backup_compression_workers, ): self.type = backup_compression self.format = backup_compression_format self.level = backup_compression_level self.location = backup_compression_location self.workers = backup_compression_workers class PgBaseBackupCompressionOption(object): """This class is in charge of validating pg_basebackup compression options""" def __init__(self, pg_base_backup_config): """ :param pg_base_backup_config: PgBaseBackupCompressionConfig """ self.config = pg_base_backup_config def validate(self, pg_server_version, remote_status): """ Validate pg_basebackup compression options. :param pg_server_version int: the server for which the compression options should be validated. :param dict remote_status: the status of the pg_basebackup command :return List: List of Issues (str) or empty list """ issues = [] if self.config.location is not None and self.config.location == "server": # "backup_location = server" requires pg_basebackup >= 15 if remote_status["pg_basebackup_version"] < Version("15"): issues.append( "backup_compression_location = server requires " "pg_basebackup 15 or greater" ) # "backup_location = server" requires PostgreSQL >= 15 if pg_server_version < 150000: issues.append( "backup_compression_location = server requires " "PostgreSQL 15 or greater" ) # plain backup format is only allowed when compression is on the server if self.config.format == "plain" and self.config.location != "server": issues.append( "backup_compression_format plain is not compatible with " "backup_compression_location %s" % self.config.location ) return issues class GZipPgBaseBackupCompressionOption(PgBaseBackupCompressionOption): def validate(self, pg_server_version, remote_status): """ Validate gzip-specific options. :param pg_server_version int: the server for which the compression options should be validated. :param dict remote_status: the status of the pg_basebackup command :return List: List of Issues (str) or empty list """ issues = super(GZipPgBaseBackupCompressionOption, self).validate( pg_server_version, remote_status ) levels = list(range(1, 10)) levels.append(-1) if self.config.level is not None and remote_status[ "pg_basebackup_version" ] < Version("15"): # version prior to 15 allowed gzip compression 0 levels.append(0) if self.config.level not in levels: issues.append( "backup_compression_level %d unsupported by compression algorithm." " %s expects a compression level between -1 and 9 (-1 will use default compression level)." % (self.config.level, self.config.type) ) if ( self.config.level is not None and remote_status["pg_basebackup_version"] >= Version("15") and self.config.level not in levels ): msg = ( "backup_compression_level %d unsupported by compression algorithm." " %s expects a compression level between 1 and 9 (-1 will use default compression level)." % (self.config.level, self.config.type) ) if self.config.level == 0: msg += "\nIf you need to create an archive not compressed, you should set `backup_compression = none`." issues.append(msg) if self.config.workers is not None: issues.append( "backup_compression_workers is not compatible with compression %s" % self.config.type ) return issues class LZ4PgBaseBackupCompressionOption(PgBaseBackupCompressionOption): def validate(self, pg_server_version, remote_status): """ Validate lz4-specific options. :param pg_server_version int: the server for which the compression options should be validated. :param dict remote_status: the status of the pg_basebackup command :return List: List of Issues (str) or empty list """ issues = super(LZ4PgBaseBackupCompressionOption, self).validate( pg_server_version, remote_status ) # "lz4" compression requires pg_basebackup >= 15 if remote_status["pg_basebackup_version"] < Version("15"): issues.append( "backup_compression = %s requires " "pg_basebackup 15 or greater" % self.config.type ) if self.config.level is not None and ( self.config.level < 0 or self.config.level > 12 ): issues.append( "backup_compression_level %d unsupported by compression algorithm." " %s expects a compression level between 1 and 12 (0 will use default compression level)." % (self.config.level, self.config.type) ) if self.config.workers is not None: issues.append( "backup_compression_workers is not compatible with compression %s." % self.config.type ) return issues class ZSTDPgBaseBackupCompressionOption(PgBaseBackupCompressionOption): def validate(self, pg_server_version, remote_status): """ Validate zstd-specific options. :param pg_server_version int: the server for which the compression options should be validated. :param dict remote_status: the status of the pg_basebackup command :return List: List of Issues (str) or empty list """ issues = super(ZSTDPgBaseBackupCompressionOption, self).validate( pg_server_version, remote_status ) # "zstd" compression requires pg_basebackup >= 15 if remote_status["pg_basebackup_version"] < Version("15"): issues.append( "backup_compression = %s requires " "pg_basebackup 15 or greater" % self.config.type ) # Minimal config level comes from zstd library `STD_minCLevel()` and is # commonly set to -131072. if self.config.level is not None and ( self.config.level < -131072 or self.config.level > 22 ): issues.append( "backup_compression_level %d unsupported by compression algorithm." " '%s' expects a compression level between -131072 and 22 (3 will use default compression level)." % (self.config.level, self.config.type) ) if self.config.workers is not None and ( type(self.config.workers) is not int or self.config.workers < 0 ): issues.append( "backup_compression_workers should be a positive integer: '%s' is invalid." % self.config.workers ) return issues class NonePgBaseBackupCompressionOption(PgBaseBackupCompressionOption): def validate(self, pg_server_version, remote_status): """ Validate none compression specific options. :param pg_server_version int: the server for which the compression options should be validated. :param dict remote_status: the status of the pg_basebackup command :return List: List of Issues (str) or empty list """ issues = super(NonePgBaseBackupCompressionOption, self).validate( pg_server_version, remote_status ) if self.config.level is not None and (self.config.level != 0): issues.append( "backup_compression %s only supports backup_compression_level 0." % self.config.type ) if self.config.workers is not None: issues.append( "backup_compression_workers is not compatible with compression '%s'." % self.config.type ) return issues class PgBaseBackupCompression(object): """ Represents the pg_basebackup compression options and provides functionality required by the backup process which depends on those options. This is a facade that interacts with appropriate classes """ def __init__( self, pg_basebackup_compression_cfg, pg_basebackup_compression_option, compression, ): """ Constructor for the PgBaseBackupCompression facade that handles base_backup class related. :param pg_basebackup_compression_cfg PgBaseBackupCompressionConfig: pg_basebackup compression configuration :param pg_basebackup_compression_option PgBaseBackupCompressionOption: :param compression Compression: """ self.config = pg_basebackup_compression_cfg self.options = pg_basebackup_compression_option self.compression = compression def with_suffix(self, basename): """ Append the suffix to the supplied basename. :param str basename: The basename (without compression suffix) of the file to be opened. """ return "%s.%s" % (basename, self.compression.file_extension) def get_file_content(self, filename, archive): """ Returns archive specific file content :param filename: str :param archive: str :return: str """ return self.compression.get_file_content(filename, archive) def validate(self, pg_server_version, remote_status): """ Validate pg_basebackup compression options. :param pg_server_version int: the server for which the compression options should be validated. :param dict remote_status: the status of the pg_basebackup command :return List: List of Issues (str) or empty list """ return self.options.validate(pg_server_version, remote_status) class Compression(with_metaclass(ABCMeta, object)): """ Abstract class meant to represent compression interface """ @abstractproperty def name(self): """ :return: """ @abstractproperty def file_extension(self): """ :return: """ @abstractmethod def uncompress(self, src, dst, exclude=None, include_args=None): """ :param src: source file path without compression extension :param dst: destination path :param exclude: list of filepath in the archive to exclude from the extraction :param include_args: list of filepath in the archive to extract. :return: """ @abstractmethod def get_file_content(self, filename, archive): """ :param filename: str file to search for in the archive (requires its full path within the archive) :param archive: str archive path/name without extension :return: string content """ def validate_src_and_dst(self, src): if src is None or src == "": raise ValueError("Source path should be a string") def validate_dst(self, dst): if dst is None or dst == "": raise ValueError("Destination path should be a string") class GZipCompression(Compression): name = "gzip" file_extension = "tar.gz" def __init__(self, command): """ :param command: barman.fs.UnixLocalCommand """ self.command = command def uncompress(self, src, dst, exclude=None, include_args=None): """ :param src: source file path without compression extension :param dst: destination path :param exclude: list of filepath in the archive to exclude from the extraction :param include_args: list of filepath in the archive to extract. :return: """ self.validate_dst(src) self.validate_dst(dst) exclude = [] if exclude is None else exclude exclude_args = [] for name in exclude: exclude_args.append("--exclude") exclude_args.append(name) include_args = [] if include_args is None else include_args args = ["-xzf", src, "--directory", dst] args.extend(exclude_args) args.extend(include_args) ret = self.command.cmd("tar", args=args) out, err = self.command.get_last_output() if ret != 0: raise CommandFailedException( "Error decompressing %s into %s: %s" % (src, dst, err) ) else: return self.command.get_last_output() def get_file_content(self, filename, archive): """ :param filename: str file to search for in the archive (requires its full path within the archive) :param archive: str archive path/name without extension :return: string content """ full_archive_name = "%s.%s" % (archive, self.file_extension) args = ["-xzf", full_archive_name, "-O", filename, "--occurrence"] ret = self.command.cmd("tar", args=args) out, err = self.command.get_last_output() if ret != 0: if "Not found in archive" in err: raise FileNotFoundException( err + "archive name: %s" % full_archive_name ) else: raise CommandFailedException( "Error reading %s into archive %s: (%s)" % (filename, full_archive_name, err) ) else: return out class LZ4Compression(Compression): name = "lz4" file_extension = "tar.lz4" def __init__(self, command): """ :param command: barman.fs.UnixLocalCommand """ self.command = command def uncompress(self, src, dst, exclude=None, include_args=None): """ :param src: source file path without compression extension :param dst: destination path :param exclude: list of filepath in the archive to exclude from the extraction :param include_args: list of filepath in the archive to extract. :return: """ self.validate_dst(src) self.validate_dst(dst) exclude = [] if exclude is None else exclude exclude_args = [] for name in exclude: exclude_args.append("--exclude") exclude_args.append(name) include_args = [] if include_args is None else include_args args = ["--use-compress-program", "lz4", "-xf", src, "--directory", dst] args.extend(exclude_args) args.extend(include_args) ret = self.command.cmd("tar", args=args) out, err = self.command.get_last_output() if ret != 0: raise CommandFailedException( "Error decompressing %s into %s: %s" % (src, dst, err) ) else: return self.command.get_last_output() def get_file_content(self, filename, archive): """ :param filename: str file to search for in the archive (requires its full path within the archive) :param archive: str archive path/name without extension :return: string content """ full_archive_name = "%s.%s" % (archive, self.file_extension) args = [ "--use-compress-program", "lz4", "-xf", full_archive_name, "-O", filename, "--occurrence", ] ret = self.command.cmd("tar", args=args) out, err = self.command.get_last_output() if ret != 0: if "Not found in archive" in err: raise FileNotFoundException( err + "archive name: %s" % full_archive_name ) else: raise CommandFailedException( "Error reading %s into archive %s: (%s)" % (filename, full_archive_name, err) ) else: return out class ZSTDCompression(Compression): name = "zstd" file_extension = "tar.zst" def __init__(self, command): """ :param command: barman.fs.UnixLocalCommand """ self.command = command def uncompress(self, src, dst, exclude=None, include_args=None): """ :param src: source file path without compression extension :param dst: destination path :param exclude: list of filepath in the archive to exclude from the extraction :param include_args: list of filepath in the archive to extract. :return: """ self.validate_dst(src) self.validate_dst(dst) exclude = [] if exclude is None else exclude exclude_args = [] for name in exclude: exclude_args.append("--exclude") exclude_args.append(name) include_args = [] if include_args is None else include_args args = ["--use-compress-program", "zstd", "-xf", src, "--directory", dst] args.extend(exclude_args) args.extend(include_args) ret = self.command.cmd("tar", args=args) out, err = self.command.get_last_output() if ret != 0: raise CommandFailedException( "Error decompressing %s into %s: %s" % (src, dst, err) ) else: return self.command.get_last_output() def get_file_content(self, filename, archive): """ :param filename: str file to search for in the archive (requires its full path within the archive) :param archive: str archive path/name without extension :return: string content """ full_archive_name = "%s.%s" % (archive, self.file_extension) args = [ "--use-compress-program", "zstd", "-xf", full_archive_name, "-O", filename, "--occurrence", ] ret = self.command.cmd("tar", args=args) out, err = self.command.get_last_output() if ret != 0: if "Not found in archive" in err: raise FileNotFoundException( err + "archive name: %s" % full_archive_name ) else: raise CommandFailedException( "Error reading %s into archive %s: (%s)" % (filename, full_archive_name, err) ) else: return out class NoneCompression(Compression): name = "none" file_extension = "tar" def __init__(self, command): """ :param command: barman.fs.UnixLocalCommand """ self.command = command def uncompress(self, src, dst, exclude=None, include_args=None): """ :param src: source file path without compression extension :param dst: destination path :param exclude: list of filepath in the archive to exclude from the extraction :param include_args: list of filepath in the archive to extract. :return: """ self.validate_dst(src) self.validate_dst(dst) exclude = [] if exclude is None else exclude exclude_args = [] for name in exclude: exclude_args.append("--exclude") exclude_args.append(name) include_args = [] if include_args is None else include_args args = ["-xf", src, "--directory", dst] args.extend(exclude_args) args.extend(include_args) ret = self.command.cmd("tar", args=args) out, err = self.command.get_last_output() if ret != 0: raise CommandFailedException( "Error decompressing %s into %s: %s" % (src, dst, err) ) else: return self.command.get_last_output() def get_file_content(self, filename, archive): """ :param filename: str file to search for in the archive (requires its full path within the archive) :param archive: str archive path/name without extension :return: string content """ full_archive_name = "%s.%s" % (archive, self.file_extension) args = ["-xf", full_archive_name, "-O", filename, "--occurrence"] ret = self.command.cmd("tar", args=args) out, err = self.command.get_last_output() if ret != 0: if "Not found in archive" in err: raise FileNotFoundException( err + "archive name: %s" % full_archive_name ) else: raise CommandFailedException( "Error reading %s into archive %s: (%s)" % (filename, full_archive_name, err) ) else: return out barman-3.13.3/barman/hooks.py0000644000175100001660000002631015002177425014164 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module contains the logic to run hook scripts """ import json import logging import time from barman import version from barman.command_wrappers import Command from barman.exceptions import AbortedRetryHookScript, UnknownBackupIdException from barman.utils import force_str _logger = logging.getLogger(__name__) class HookScriptRunner(object): def __init__( self, backup_manager, name, phase=None, error=None, retry=False, **extra_env ): """ Execute a hook script managing its environment """ self.backup_manager = backup_manager self.name = name self.extra_env = extra_env self.phase = phase self.error = error self.retry = retry self.environment = None self.exit_status = None self.exception = None self.script = None self.reset() def reset(self): """ Reset the status of the class. """ self.environment = dict(self.extra_env) config_file = self.backup_manager.config.config.config_file self.environment.update( { "BARMAN_VERSION": version.__version__, "BARMAN_SERVER": self.backup_manager.config.name, "BARMAN_CONFIGURATION": config_file, "BARMAN_HOOK": self.name, "BARMAN_RETRY": str(1 if self.retry else 0), } ) if self.error: self.environment["BARMAN_ERROR"] = force_str(self.error) if self.phase: self.environment["BARMAN_PHASE"] = self.phase script_config_name = "%s_%s" % (self.phase, self.name) else: script_config_name = self.name self.script = getattr(self.backup_manager.config, script_config_name, None) self.exit_status = None self.exception = None def env_from_backup_info(self, backup_info): """ Prepare the environment for executing a script :param BackupInfo backup_info: the backup metadata """ try: previous_backup = self.backup_manager.get_previous_backup( backup_info.backup_id ) if previous_backup: previous_backup_id = previous_backup.backup_id else: previous_backup_id = "" except UnknownBackupIdException: previous_backup_id = "" try: next_backup = self.backup_manager.get_next_backup(backup_info.backup_id) if next_backup: next_backup_id = next_backup.backup_id else: next_backup_id = "" except UnknownBackupIdException: next_backup_id = "" self.environment.update( { "BARMAN_BACKUP_DIR": backup_info.get_basebackup_directory(), "BARMAN_BACKUP_INFO_PATH": backup_info.get_filename(), "BARMAN_BACKUP_ID": backup_info.backup_id, "BARMAN_PREVIOUS_ID": previous_backup_id, "BARMAN_NEXT_ID": next_backup_id, "BARMAN_STATUS": backup_info.status, "BARMAN_ERROR": backup_info.error or "", } ) def env_from_wal_info(self, wal_info, full_path=None, error=None): """ Prepare the environment for executing a script :param WalFileInfo wal_info: the backup metadata :param str full_path: override wal_info.fullpath() result :param str|Exception error: An error message in case of failure """ self.environment.update( { "BARMAN_SEGMENT": wal_info.name, "BARMAN_FILE": str( full_path if full_path is not None else wal_info.fullpath(self.backup_manager.server) ), "BARMAN_SIZE": str(wal_info.size), "BARMAN_TIMESTAMP": str(wal_info.time), "BARMAN_COMPRESSION": wal_info.compression or "", "BARMAN_ERROR": force_str(error or ""), } ) def env_from_recover( self, backup_info, dest, tablespaces, remote_command, error=None, **kwargs ): """ Prepare the environment for executing a script :param BackupInfo backup_info: the backup metadata :param str dest: the destination directory :param dict[str,str]|None tablespaces: a tablespace name -> location map (for relocation) :param str|None remote_command: default None. The remote command to recover the base backup, in case of remote backup. :param str|Exception error: An error message in case of failure """ self.env_from_backup_info(backup_info) # Prepare a JSON representation of tablespace map tablespaces_map = "" if tablespaces: tablespaces_map = json.dumps(tablespaces, sort_keys=True) # Prepare a JSON representation of additional recovery options # Skip any empty argument kwargs_filtered = dict([(k, v) for k, v in kwargs.items() if v]) recover_options = "" if kwargs_filtered: recover_options = json.dumps(kwargs_filtered, sort_keys=True) self.environment.update( { "BARMAN_DESTINATION_DIRECTORY": str(dest), "BARMAN_TABLESPACES": tablespaces_map, "BARMAN_REMOTE_COMMAND": str(remote_command or ""), "BARMAN_RECOVER_OPTIONS": recover_options, "BARMAN_ERROR": force_str(error or ""), } ) def run(self): """ Run a a hook script if configured. This method must never throw any exception """ # noinspection PyBroadException try: if self.script: _logger.debug("Attempt to run %s: %s", self.name, self.script) cmd = Command( self.script, env_append=self.environment, path=self.backup_manager.server.path, shell=True, check=False, ) self.exit_status = cmd() if self.exit_status != 0: details = "%s returned %d\nOutput details:\n" % ( self.script, self.exit_status, ) details += cmd.out details += cmd.err _logger.warning(details) else: _logger.debug("%s returned %d", self.script, self.exit_status) return self.exit_status except Exception as e: _logger.exception("Exception running %s", self.name) self.exception = e return None class RetryHookScriptRunner(HookScriptRunner): """ A 'retry' hook script is a special kind of hook script that Barman tries to run indefinitely until it either returns a SUCCESS or ABORT exit code. Retry hook scripts are executed immediately before (pre) and after (post) the command execution. Standard hook scripts are executed immediately before (pre) and after (post) the retry hook scripts. """ # Failed attempts before sleeping for NAP_TIME seconds ATTEMPTS_BEFORE_NAP = 5 # Short break after a failure (in seconds) BREAK_TIME = 3 # Long break (nap, in seconds) after ATTEMPTS_BEFORE_NAP failures NAP_TIME = 60 # ABORT (and STOP) exit code EXIT_ABORT_STOP = 63 # ABORT (and CONTINUE) exit code EXIT_ABORT_CONTINUE = 62 # SUCCESS exit code EXIT_SUCCESS = 0 def __init__(self, backup_manager, name, phase=None, error=None, **extra_env): super(RetryHookScriptRunner, self).__init__( backup_manager, name, phase, error, retry=True, **extra_env ) def run(self): """ Run a a 'retry' hook script, if required by configuration. Barman will retry to run the script indefinitely until it returns a EXIT_SUCCESS, or an EXIT_ABORT_CONTINUE, or an EXIT_ABORT_STOP code. There are BREAK_TIME seconds of sleep between every try. Every ATTEMPTS_BEFORE_NAP failures, Barman will sleep for NAP_TIME seconds. """ # If there is no script, exit if self.script is not None: # Keep track of the number of attempts attempts = 1 while True: # Run the script using the standard hook method (inherited) super(RetryHookScriptRunner, self).run() # Run the script until it returns EXIT_ABORT_CONTINUE, # or an EXIT_ABORT_STOP, or EXIT_SUCCESS if self.exit_status in ( self.EXIT_ABORT_CONTINUE, self.EXIT_ABORT_STOP, self.EXIT_SUCCESS, ): break # Check for the number of attempts if attempts <= self.ATTEMPTS_BEFORE_NAP: attempts += 1 # Take a short break _logger.debug("Retry again in %d seconds", self.BREAK_TIME) time.sleep(self.BREAK_TIME) else: # Reset the attempt number and take a longer nap _logger.debug( "Reached %d failures. Take a nap " "then retry again in %d seconds", self.ATTEMPTS_BEFORE_NAP, self.NAP_TIME, ) attempts = 1 time.sleep(self.NAP_TIME) # Outside the loop check for the exit code. if self.exit_status == self.EXIT_ABORT_CONTINUE: # Warn the user if the script exited with EXIT_ABORT_CONTINUE # Notify EXIT_ABORT_CONTINUE exit status because success and # failures are already managed in the superclass run method _logger.warning( "%s was aborted (got exit status %d, Barman resumes)", self.script, self.exit_status, ) elif self.exit_status == self.EXIT_ABORT_STOP: # Log the error and raise AbortedRetryHookScript exception _logger.error( "%s was aborted (got exit status %d, Barman requested to stop)", self.script, self.exit_status, ) raise AbortedRetryHookScript(self) return self.exit_status barman-3.13.3/barman/recovery_executor.py0000644000175100001660000027675015002177425016634 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module contains the methods necessary to perform a recovery """ from __future__ import print_function import collections import datetime import logging import os import re import shutil import socket import tempfile from functools import partial from io import BytesIO import dateutil.parser import dateutil.tz import barman.fs as fs from barman import output, xlog from barman.cloud_providers import get_snapshot_interface_from_backup_info from barman.command_wrappers import PgCombineBackup, RsyncPgData from barman.compression import ( GZipCompression, LZ4Compression, NoneCompression, ZSTDCompression, ) from barman.config import RecoveryOptions from barman.copy_controller import RsyncCopyController from barman.exceptions import ( BadXlogSegmentName, CommandFailedException, DataTransferFailure, FsOperationFailed, RecoveryInvalidTargetException, RecoveryPreconditionException, RecoveryStandbyModeException, RecoveryTargetActionException, SnapshotBackupException, ) from barman.infofile import BackupInfo, LocalBackupInfo, SyntheticBackupInfo from barman.utils import force_str, mkpath, parse_target_tli, total_seconds # generic logger for this module _logger = logging.getLogger(__name__) # regexp matching a single value in Postgres configuration file PG_CONF_SETTING_RE = re.compile(r"^\s*([^\s=]+)\s*=?\s*(.*)$") # create a namedtuple object called Assertion # with 'filename', 'line', 'key' and 'value' as properties Assertion = collections.namedtuple("Assertion", "filename line key value") # noinspection PyMethodMayBeStatic class RecoveryExecutor(object): """ Class responsible of recovery operations """ def __init__(self, backup_manager): """ Constructor :param barman.backup.BackupManager backup_manager: the BackupManager owner of the executor """ self.backup_manager = backup_manager self.server = backup_manager.server self.config = backup_manager.config self.temp_dirs = [] def recover( self, backup_info, dest, wal_dest=None, tablespaces=None, remote_command=None, target_tli=None, target_time=None, target_xid=None, target_lsn=None, target_name=None, target_immediate=False, exclusive=False, target_action=None, standby_mode=None, recovery_conf_filename=None, ): """ Performs a recovery of a backup This method should be called in a closing context :param barman.infofile.BackupInfo backup_info: the backup to recover :param str dest: the destination directory :param str|None wal_dest: the destination directory for WALs when doing PITR. See :meth:`~barman.recovery_executor.RecoveryExecutor._set_pitr_targets` for more details. :param dict[str,str]|None tablespaces: a tablespace name -> location map (for relocation) :param str|None remote_command: The remote command to recover the base backup, in case of remote backup. :param str|None target_tli: the target timeline :param str|None target_time: the target time :param str|None target_xid: the target xid :param str|None target_lsn: the target LSN :param str|None target_name: the target name created previously with pg_create_restore_point() function call :param str|None target_immediate: end recovery as soon as consistency is reached :param bool exclusive: whether the recovery is exclusive or not :param str|None target_action: The recovery target action :param bool|None standby_mode: standby mode :param str|None recovery_conf_filename: filename for storing recovery configurations """ # Run the cron to be sure the wal catalog is up to date # Prepare a map that contains all the objects required for a recovery recovery_info = self._setup( backup_info, remote_command, dest, recovery_conf_filename ) output.info( "Starting %s restore for server %s using backup %s", recovery_info["recovery_dest"], self.server.config.name, backup_info.backup_id, ) output.info("Destination directory: %s", dest) if remote_command: output.info("Remote command: %s", remote_command) # If the backup we are recovering is still not validated and we # haven't requested the get-wal feature, display a warning message if not recovery_info["get_wal"]: if backup_info.status == BackupInfo.WAITING_FOR_WALS: output.warning( "IMPORTANT: You have requested a recovery operation for " "a backup that does not have yet all the WAL files that " "are required for consistency." ) # Set targets for PITR self._set_pitr_targets( recovery_info, backup_info, dest, wal_dest, target_name, target_time, target_tli, target_xid, target_lsn, target_immediate, target_action, ) # Retrieve the safe_horizon for smart copy self._retrieve_safe_horizon(recovery_info, backup_info, dest) # check destination directory. If doesn't exist create it try: recovery_info["cmd"].create_dir_if_not_exists(dest, mode="700") except FsOperationFailed as e: output.error("unable to initialise destination directory '%s': %s", dest, e) output.close_and_exit() # Initialize tablespace directories if backup_info.tablespaces: self._prepare_tablespaces( backup_info, recovery_info["cmd"], dest, tablespaces ) # Copy the base backup self._start_backup_copy_message() try: self._backup_copy( backup_info, dest, tablespaces=tablespaces, remote_command=remote_command, safe_horizon=recovery_info["safe_horizon"], recovery_info=recovery_info, ) except DataTransferFailure as e: self._backup_copy_failure_message(e) output.close_and_exit() # Copy the backup.info file in the destination as # ".barman-recover.info" if remote_command: try: recovery_info["rsync"]( backup_info.filename, ":%s/.barman-recover.info" % dest ) except CommandFailedException as e: output.error("copy of recovery metadata file failed: %s", e) output.close_and_exit() else: backup_info.save(os.path.join(dest, ".barman-recover.info")) # Rename the backup_manifest file by adding a backup ID suffix if recovery_info["cmd"].exists(os.path.join(dest, "backup_manifest")): recovery_info["cmd"].move( os.path.join(dest, "backup_manifest"), os.path.join(dest, "backup_manifest.%s" % backup_info.backup_id), ) # Standby mode is not available for PostgreSQL older than 9.0 if backup_info.version < 90000 and standby_mode: raise RecoveryStandbyModeException( "standby_mode is available only from PostgreSQL 9.0" ) # Restore the WAL segments. If GET_WAL option is set, skip this phase # as they will be retrieved using the wal-get command. if not recovery_info["get_wal"]: # If the backup we restored is still waiting for WALS, read the # backup info again and check whether it has been validated. # Notify the user if it is still not DONE. if backup_info.status == BackupInfo.WAITING_FOR_WALS: data = LocalBackupInfo(self.server, backup_info.filename) if data.status == BackupInfo.WAITING_FOR_WALS: output.warning( "IMPORTANT: The backup we have restored IS NOT " "VALID. Required WAL files for consistency are " "missing. Please verify that WAL archiving is " "working correctly or evaluate using the 'get-wal' " "option for recovery" ) # check WALs destination directory. If doesn't exist create it # we use the value from recovery_info as it contains the final path try: recovery_info["cmd"].create_dir_if_not_exists( recovery_info["wal_dest"], mode="700" ) except FsOperationFailed as e: output.error( "unable to initialise WAL destination directory '%s': %s", wal_dest, e, ) output.close_and_exit() output.info("Copying required WAL segments.") required_xlog_files = () # Makes static analysers happy try: # TODO: Stop early if target-immediate # Retrieve a list of required log files required_xlog_files = tuple( self.server.get_required_xlog_files( backup_info, target_tli, None, None, target_lsn, target_immediate, ) ) # Restore WAL segments into the wal_dest directory self._xlog_copy( required_xlog_files, recovery_info["wal_dest"], remote_command ) except DataTransferFailure as e: output.error("Failure copying WAL files: %s", e) output.close_and_exit() except BadXlogSegmentName as e: output.error( "invalid xlog segment name %r\n" 'HINT: Please run "barman rebuild-xlogdb %s" ' "to solve this issue", force_str(e), self.config.name, ) output.close_and_exit() # If WAL files are put directly in the pg_xlog directory, # avoid shipping of just recovered files # by creating the corresponding archive status file if not recovery_info["is_pitr"]: output.info("Generating archive status files") self._generate_archive_status( recovery_info, remote_command, required_xlog_files ) # Generate recovery.conf file (only if needed by PITR or get_wal) is_pitr = recovery_info["is_pitr"] get_wal = recovery_info["get_wal"] if is_pitr or get_wal or standby_mode: output.info("Generating recovery configuration") self._generate_recovery_conf( recovery_info, backup_info, dest, target_immediate, exclusive, remote_command, target_name, target_time, target_tli, target_xid, target_lsn, standby_mode, ) # Create archive_status directory if necessary archive_status_dir = os.path.join(recovery_info["wal_dest"], "archive_status") try: recovery_info["cmd"].create_dir_if_not_exists(archive_status_dir) except FsOperationFailed as e: output.error( "unable to create the archive_status directory '%s': %s", archive_status_dir, e, ) output.close_and_exit() # As last step, analyse configuration files in order to spot # harmful options. Barman performs automatic conversion of # some options as well as notifying users of their existence. # # This operation is performed in three steps: # 1) mapping # 2) analysis # 3) copy output.info("Identify dangerous settings in destination directory.") self._map_temporary_config_files(recovery_info, backup_info, remote_command) self._analyse_temporary_config_files(recovery_info) self._copy_temporary_config_files(dest, remote_command, recovery_info) return recovery_info def _setup(self, backup_info, remote_command, dest, recovery_conf_filename): """ Prepare the recovery_info dictionary for the recovery, as well as temporary working directory :param barman.infofile.LocalBackupInfo backup_info: representation of a backup :param str remote_command: ssh command for remote connection :param str|None recovery_conf_filename: filename for storing recovery configurations :return dict: recovery_info dictionary, holding the basic values for a recovery """ # Calculate the name of the WAL directory if backup_info.version < 100000: wal_dest = os.path.join(dest, "pg_xlog") else: wal_dest = os.path.join(dest, "pg_wal") tempdir = tempfile.mkdtemp(prefix="barman_recovery-") self.temp_dirs.append(fs.LocalLibPathDeletionCommand(tempdir)) recovery_info = { "cmd": fs.unix_command_factory(remote_command, self.server.path), "recovery_dest": "local", "rsync": None, "configuration_files": [], "destination_path": dest, "temporary_configuration_files": [], "tempdir": tempdir, "is_pitr": False, "wal_dest": wal_dest, "get_wal": RecoveryOptions.GET_WAL in self.config.recovery_options, } # A map that will keep track of the results of the recovery. # Used for output generation results = { "changes": [], "warnings": [], "missing_files": [], "get_wal": False, "recovery_start_time": datetime.datetime.now(dateutil.tz.tzlocal()), } recovery_info["results"] = results # Set up a list of configuration files recovery_info["configuration_files"].append("postgresql.conf") # Always add postgresql.auto.conf to the list of configuration files even if # it is not the specified destination for recovery settings, because there may # be other configuration options which need to be checked by Barman. if backup_info.version >= 90400: recovery_info["configuration_files"].append("postgresql.auto.conf") # Determine the destination file for recovery options. This will normally be # postgresql.auto.conf (or recovery.conf for PostgreSQL versions earlier than # 12) however there are certain scenarios (such as postgresql.auto.conf being # deliberately symlinked to /dev/null) which mean a user might have specified # an alternative destination. If an alternative has been specified, via # recovery_conf_filename, then it should be set as the recovery configuration # file. if recovery_conf_filename: # There is no need to also add the file to recovery_info["configuration_files"] # because that is only required for files which may already exist and # therefore contain options which Barman should check for safety. results["recovery_configuration_file"] = recovery_conf_filename # Otherwise, set the recovery configuration file based on the PostgreSQL # version used to create the backup. else: results["recovery_configuration_file"] = "postgresql.auto.conf" if backup_info.version < 120000: # The recovery.conf file is created for the recovery and therefore # Barman does not need to check the content. The file therefore does # not need to be added to recovery_info["configuration_files"] and # just needs to be set as the recovery configuration file. results["recovery_configuration_file"] = "recovery.conf" # Handle remote recovery options if remote_command: recovery_info["recovery_dest"] = "remote" recovery_info["rsync"] = RsyncPgData( path=self.server.path, ssh=remote_command, bwlimit=self.config.bandwidth_limit, network_compression=self.config.network_compression, ) return recovery_info def _set_pitr_targets( self, recovery_info, backup_info, dest, wal_dest, target_name, target_time, target_tli, target_xid, target_lsn, target_immediate, target_action, ): """ Set PITR targets - as specified by the user :param dict recovery_info: Dictionary containing all the recovery parameters :param barman.infofile.LocalBackupInfo backup_info: representation of a backup :param str dest: destination directory of the recovery :param str|None wal_dest: the destination directory for WALs when doing PITR :param str|None target_name: recovery target name for PITR :param str|None target_time: recovery target time for PITR :param str|None target_tli: recovery target timeline for PITR :param str|None target_xid: recovery target transaction id for PITR :param str|None target_lsn: recovery target LSN for PITR :param bool|None target_immediate: end recovery as soon as consistency is reached :param str|None target_action: recovery target action for PITR """ target_datetime = None # Calculate the integer value of TLI if a keyword is provided calculated_target_tli = parse_target_tli( self.backup_manager, target_tli, backup_info ) d_immediate = backup_info.version >= 90400 and target_immediate d_lsn = backup_info.version >= 100000 and target_lsn # Detect PITR if any([target_time, target_xid, target_tli, target_name, d_immediate, d_lsn]): recovery_info["is_pitr"] = True targets = {} if target_time: try: target_datetime = dateutil.parser.parse(target_time) except ValueError as e: raise RecoveryInvalidTargetException( "Unable to parse the target time parameter %r: %s" % (target_time, e) ) except TypeError: # this should not happen, but there is a known bug in # dateutil.parser.parse() implementation # ref: https://bugs.launchpad.net/dateutil/+bug/1247643 raise RecoveryInvalidTargetException( "Unable to parse the target time parameter %r" % target_time ) # If the parsed timestamp is naive, forces it to local timezone if target_datetime.tzinfo is None: target_datetime = target_datetime.replace( tzinfo=dateutil.tz.tzlocal() ) output.warning( "No time zone has been specified through '--target-time' " "command-line option. Barman assumed the same time zone from " "the Barman host.", ) # Check if the target time is reachable from the # selected backup if backup_info.end_time > target_datetime: raise RecoveryInvalidTargetException( "The requested target time %s " "is before the backup end time %s" % (target_datetime, backup_info.end_time) ) targets["time"] = str(target_datetime) if target_xid: targets["xid"] = str(target_xid) if d_lsn: targets["lsn"] = str(d_lsn) if target_tli: targets["timeline"] = str(calculated_target_tli) if target_name: targets["name"] = str(target_name) if d_immediate: targets["immediate"] = d_immediate # Manage the target_action option if backup_info.version < 90100: if target_action: raise RecoveryTargetActionException( "Illegal target action '%s' " "for this version of PostgreSQL" % target_action ) elif 90100 <= backup_info.version < 90500: if target_action == "pause": recovery_info["pause_at_recovery_target"] = "on" elif target_action: raise RecoveryTargetActionException( "Illegal target action '%s' " "for this version of PostgreSQL" % target_action ) else: if target_action in ("pause", "shutdown", "promote"): recovery_info["recovery_target_action"] = target_action elif target_action: raise RecoveryTargetActionException( "Illegal target action '%s' " "for this version of PostgreSQL" % target_action ) output.info( "Doing PITR. Recovery target %s", (", ".join(["%s: %r" % (k, v) for k, v in targets.items()])), ) # If a custom WALs directory has been given, use it, otherwise defaults to # using a `barman_wal` directory inside the destination directory if wal_dest: recovery_info["wal_dest"] = wal_dest else: recovery_info["wal_dest"] = os.path.join(dest, "barman_wal") else: # Raise an error if target_lsn is used with a pgversion < 10 if backup_info.version < 100000: if target_lsn: raise RecoveryInvalidTargetException( "Illegal use of recovery_target_lsn '%s' " "for this version of PostgreSQL " "(version 10 minimum required)" % target_lsn ) if target_immediate: raise RecoveryInvalidTargetException( "Illegal use of recovery_target_immediate " "for this version of PostgreSQL " "(version 9.4 minimum required)" ) if target_action: raise RecoveryTargetActionException( "Can't enable recovery target action when PITR is not required" ) recovery_info["target_datetime"] = target_datetime def _retrieve_safe_horizon(self, recovery_info, backup_info, dest): """ Retrieve the safe_horizon for smart copy If the target directory contains a previous recovery, it is safe to pick the least of the two backup "begin times" (the one we are recovering now and the one previously recovered in the target directory). Set the value in the given recovery_info dictionary. :param dict recovery_info: Dictionary containing all the recovery parameters :param barman.infofile.LocalBackupInfo backup_info: a backup representation :param str dest: recovery destination directory """ # noinspection PyBroadException try: backup_begin_time = backup_info.begin_time # Retrieve previously recovered backup metadata (if available) dest_info_txt = recovery_info["cmd"].get_file_content( os.path.join(dest, ".barman-recover.info") ) dest_info = LocalBackupInfo( self.server, info_file=BytesIO(dest_info_txt.encode("utf-8")) ) dest_begin_time = dest_info.begin_time # Pick the earlier begin time. Both are tz-aware timestamps because # BackupInfo class ensure it safe_horizon = min(backup_begin_time, dest_begin_time) output.info( "Using safe horizon time for smart rsync copy: %s", safe_horizon ) except FsOperationFailed as e: # Setting safe_horizon to None will effectively disable # the time-based part of smart_copy method. However it is still # faster than running all the transfers with checksum enabled. # # FsOperationFailed means the .barman-recover.info is not available # on destination directory safe_horizon = None _logger.warning( "Unable to retrieve safe horizon time for smart rsync copy: %s", e ) except Exception as e: # Same as above, but something failed decoding .barman-recover.info # or comparing times, so log the full traceback safe_horizon = None _logger.exception( "Error retrieving safe horizon time for smart rsync copy: %s", e ) recovery_info["safe_horizon"] = safe_horizon def _prepare_tablespaces(self, backup_info, cmd, dest, tablespaces): """ Prepare the directory structure for required tablespaces, taking care of tablespaces relocation, if requested. :param barman.infofile.LocalBackupInfo backup_info: backup representation :param barman.fs.UnixLocalCommand cmd: Object for filesystem interaction :param str dest: destination dir for the recovery :param dict tablespaces: dict of all the tablespaces and their location """ tblspc_dir = os.path.join(dest, "pg_tblspc") try: # check for pg_tblspc dir into recovery destination folder. # if it does not exists, create it cmd.create_dir_if_not_exists(tblspc_dir) except FsOperationFailed as e: output.error( "unable to initialise tablespace directory '%s': %s", tblspc_dir, e ) output.close_and_exit() for item in backup_info.tablespaces: # build the filename of the link under pg_tblspc directory pg_tblspc_file = os.path.join(tblspc_dir, str(item.oid)) # by default a tablespace goes in the same location where # it was on the source server when the backup was taken location = item.location # if a relocation has been requested for this tablespace, # use the target directory provided by the user if tablespaces and item.name in tablespaces: location = tablespaces[item.name] try: # remove the current link in pg_tblspc, if it exists cmd.delete_if_exists(pg_tblspc_file) # create tablespace location, if does not exist # (raise an exception if it is not possible) cmd.create_dir_if_not_exists(location) # check for write permissions on destination directory cmd.check_write_permission(location) # create symlink between tablespace and recovery folder cmd.create_symbolic_link(location, pg_tblspc_file) except FsOperationFailed as e: output.error( "unable to prepare '%s' tablespace (destination '%s'): %s", item.name, location, e, ) output.close_and_exit() output.info("\t%s, %s, %s", item.oid, item.name, location) def _start_backup_copy_message(self): """ Write the start backup copy message to the output. """ output.info("Copying the base backup.") def _backup_copy_failure_message(self, e): """ Write the backup failure message to the output. """ output.error("Failure copying base backup: %s", e) def _backup_copy( self, backup_info, dest, tablespaces=None, remote_command=None, safe_horizon=None, recovery_info=None, ): """ Perform the actual copy of the base backup for recovery purposes First, it copies one tablespace at a time, then the PGDATA directory. Bandwidth limitation, according to configuration, is applied in the process. TODO: manage configuration files if outside PGDATA. :param barman.infofile.LocalBackupInfo backup_info: the backup to recover :param str dest: the destination directory :param dict[str,str]|None tablespaces: a tablespace name -> location map (for relocation) :param str|None remote_command: default None. The remote command to recover the base backup, in case of remote backup. :param datetime.datetime|None safe_horizon: anything after this time has to be checked with checksum """ # Set a ':' prefix to remote destinations dest_prefix = "" if remote_command: dest_prefix = ":" # Create the copy controller object, specific for rsync, # which will drive all the copy operations. Items to be # copied are added before executing the copy() method controller = RsyncCopyController( path=self.server.path, ssh_command=remote_command, network_compression=self.config.network_compression, safe_horizon=safe_horizon, retry_times=self.config.basebackup_retry_times, retry_sleep=self.config.basebackup_retry_sleep, workers=self.config.parallel_jobs, workers_start_batch_period=self.config.parallel_jobs_start_batch_period, workers_start_batch_size=self.config.parallel_jobs_start_batch_size, ) # Dictionary for paths to be excluded from rsync exclude_and_protect = [] # Process every tablespace if backup_info.tablespaces: for tablespace in backup_info.tablespaces: # By default a tablespace goes in the same location where # it was on the source server when the backup was taken location = tablespace.location # If a relocation has been requested for this tablespace # use the user provided target directory if tablespaces and tablespace.name in tablespaces: location = tablespaces[tablespace.name] # If the tablespace location is inside the data directory, # exclude and protect it from being deleted during # the data directory copy if location.startswith(dest): exclude_and_protect += [location[len(dest) :]] # Exclude and protect the tablespace from being deleted during # the data directory copy exclude_and_protect.append("/pg_tblspc/%s" % tablespace.oid) # Add the tablespace directory to the list of objects # to be copied by the controller controller.add_directory( label=tablespace.name, src="%s/" % backup_info.get_data_directory(tablespace.oid), dst=dest_prefix + location, bwlimit=self.config.get_bwlimit(tablespace), item_class=controller.TABLESPACE_CLASS, ) # Add the PGDATA directory to the list of objects to be copied # by the controller controller.add_directory( label="pgdata", src="%s/" % backup_info.get_data_directory(), dst=dest_prefix + dest, bwlimit=self.config.get_bwlimit(), exclude=[ "/pg_log/*", "/log/*", "/pg_xlog/*", "/pg_wal/*", "/postmaster.pid", "/recovery.conf", "/tablespace_map", ], exclude_and_protect=exclude_and_protect, item_class=controller.PGDATA_CLASS, ) # TODO: Manage different location for configuration files # TODO: that were not within the data directory # Execute the copy try: controller.copy() # TODO: Improve the exception output except CommandFailedException as e: msg = "data transfer failure" raise DataTransferFailure.from_command_error("rsync", e, msg) def _xlog_copy(self, required_xlog_files, wal_dest, remote_command): """ Restore WAL segments :param required_xlog_files: list of all required WAL files :param wal_dest: the destination directory for xlog recover :param remote_command: default None. The remote command to recover the xlog, in case of remote backup. """ # List of required WAL files partitioned by containing directory xlogs = collections.defaultdict(list) # add '/' suffix to ensure it is a directory wal_dest = "%s/" % wal_dest # Map of every compressor used with any WAL file in the archive, # to be used during this recovery compressors = {} compression_manager = self.backup_manager.compression_manager # Fill xlogs and compressors maps from required_xlog_files for wal_info in required_xlog_files: hashdir = xlog.hash_dir(wal_info.name) xlogs[hashdir].append(wal_info) # If a compressor is required, make sure it exists in the cache if ( wal_info.compression is not None and wal_info.compression not in compressors ): compressors[wal_info.compression] = compression_manager.get_compressor( compression=wal_info.compression ) rsync = RsyncPgData( path=self.server.path, ssh=remote_command, bwlimit=self.config.bandwidth_limit, network_compression=self.config.network_compression, ) # If compression is used and this is a remote recovery, we need a # temporary directory where to spool uncompressed files, # otherwise we either decompress every WAL file in the local # destination, or we ship the uncompressed file remotely if compressors: if remote_command: # Decompress to a temporary spool directory wal_decompression_dest = tempfile.mkdtemp(prefix="barman_wal-") else: # Decompress directly to the destination directory wal_decompression_dest = wal_dest # Make sure wal_decompression_dest exists mkpath(wal_decompression_dest) else: # If no compression wal_decompression_dest = None if remote_command: # If remote recovery tell rsync to copy them remotely # add ':' prefix to mark it as remote wal_dest = ":%s" % wal_dest total_wals = sum(map(len, xlogs.values())) partial_count = 0 for prefix in sorted(xlogs): batch_len = len(xlogs[prefix]) partial_count += batch_len source_dir = os.path.join(self.config.wals_directory, prefix) _logger.info( "Starting copy of %s WAL files %s/%s from %s to %s", batch_len, partial_count, total_wals, xlogs[prefix][0], xlogs[prefix][-1], ) # If at least one compressed file has been found, activate # compression check and decompression for each WAL files if compressors: for segment in xlogs[prefix]: dst_file = os.path.join(wal_decompression_dest, segment.name) if segment.compression is not None: compressors[segment.compression].decompress( os.path.join(source_dir, segment.name), dst_file ) else: shutil.copy2(os.path.join(source_dir, segment.name), dst_file) if remote_command: try: # Transfer the WAL files rsync.from_file_list( list(segment.name for segment in xlogs[prefix]), wal_decompression_dest, wal_dest, ) except CommandFailedException as e: msg = ( "data transfer failure while copying WAL files " "to directory '%s'" ) % (wal_dest[1:],) raise DataTransferFailure.from_command_error("rsync", e, msg) # Cleanup files after the transfer for segment in xlogs[prefix]: file_name = os.path.join(wal_decompression_dest, segment.name) try: os.unlink(file_name) except OSError as e: output.warning( "Error removing temporary file '%s': %s", file_name, e ) else: try: rsync.from_file_list( list(segment.name for segment in xlogs[prefix]), "%s/" % os.path.join(self.config.wals_directory, prefix), wal_dest, ) except CommandFailedException as e: msg = ( "data transfer failure while copying WAL files " "to directory '%s'" % (wal_dest[1:],) ) raise DataTransferFailure.from_command_error("rsync", e, msg) _logger.info("Finished copying %s WAL files.", total_wals) # Remove local decompression target directory if different from the # destination directory (it happens when compression is in use during a # remote recovery if wal_decompression_dest and wal_decompression_dest != wal_dest: shutil.rmtree(wal_decompression_dest) def _generate_archive_status( self, recovery_info, remote_command, required_xlog_files ): """ Populate the archive_status directory :param dict recovery_info: Dictionary containing all the recovery parameters :param str remote_command: ssh command for remote connection :param tuple required_xlog_files: list of required WAL segments """ if remote_command: status_dir = recovery_info["tempdir"] else: status_dir = os.path.join(recovery_info["wal_dest"], "archive_status") mkpath(status_dir) for wal_info in required_xlog_files: with open(os.path.join(status_dir, "%s.done" % wal_info.name), "a") as f: f.write("") if remote_command: try: recovery_info["rsync"]( "%s/" % status_dir, ":%s" % os.path.join(recovery_info["wal_dest"], "archive_status"), ) except CommandFailedException as e: output.error("unable to populate archive_status directory: %s", e) output.close_and_exit() def _generate_recovery_conf( self, recovery_info, backup_info, dest, immediate, exclusive, remote_command, target_name, target_time, target_tli, target_xid, target_lsn, standby_mode, ): """ Generate recovery configuration for PITR :param dict recovery_info: Dictionary containing all the recovery parameters :param barman.infofile.LocalBackupInfo backup_info: representation of a backup :param str dest: destination directory of the recovery :param bool|None immediate: end recovery as soon as consistency is reached :param boolean exclusive: exclusive backup or concurrent :param str remote_command: ssh command for remote connection :param str target_name: recovery target name for PITR :param str target_time: recovery target time for PITR :param str target_tli: recovery target timeline for PITR :param str target_xid: recovery target transaction id for PITR :param str target_lsn: recovery target LSN for PITR :param bool|None standby_mode: standby mode """ wal_dest = recovery_info["wal_dest"] recovery_conf_lines = [] # If GET_WAL has been set, use the get-wal command to retrieve the # required wal files. Otherwise use the unix command "cp" to copy # them from the wal_dest directory if recovery_info["get_wal"]: partial_option = "" if not standby_mode: partial_option = "-P" # We need to create the right restore command. # If we are doing a remote recovery, # the barman-cli package is REQUIRED on the server that is hosting # the PostgreSQL server. # We use the machine FQDN and the barman_user # setting to call the barman-wal-restore correctly. # If local recovery, we use barman directly, assuming # the postgres process will be executed with the barman user. # It MUST to be reviewed by the user in any case. if remote_command: fqdn = socket.getfqdn() recovery_conf_lines.append( "# The 'barman-wal-restore' command " "is provided in the 'barman-cli' package" ) recovery_conf_lines.append( "restore_command = 'barman-wal-restore %s -U %s " "%s %s %%f %%p'" % (partial_option, self.config.config.user, fqdn, self.config.name) ) else: recovery_conf_lines.append( "# The 'barman get-wal' command " "must run as '%s' user" % self.config.config.user ) recovery_conf_lines.append( "restore_command = 'sudo -u %s " "barman get-wal %s %s %%f > %%p'" % (self.config.config.user, partial_option, self.config.name) ) recovery_info["results"]["get_wal"] = True elif not standby_mode: # We copy all the needed WAL files to the wal_dest directory when get-wal # is not requested, except when we are in standby mode. In the case of # standby mode, the server will not exit recovery, so the # recovery_end_command would never be executed. # For this reason, with standby_mode, we need to copy the WAL files # directly in the pg_wal directory. recovery_conf_lines.append(f"restore_command = 'cp {wal_dest}/%f %p'") recovery_conf_lines.append(f"recovery_end_command = 'rm -fr {wal_dest}'") # Writes recovery target if target_time: # 'target_time' is the value as it came from '--target-time' command-line # option, which may be without a time zone. When writing the actual Postgres # configuration we should use a value with an explicit time zone set, so we # avoid hitting pitfalls. We use the 'target_datetime' which was prevously # added to 'recovery_info'. It already handles the cases where the user # specifies no time zone, and uses the Barman host time zone as a fallback. # In short: if 'target_time' is present it means the user asked for a # specific point in time, but we need a sanitized value to use in the # Postgres configuration, so we use 'target_datetime'. # See '_set_pitr_targets'. recovery_conf_lines.append( "recovery_target_time = '%s'" % recovery_info["target_datetime"], ) if target_xid: recovery_conf_lines.append("recovery_target_xid = '%s'" % target_xid) if target_lsn: recovery_conf_lines.append("recovery_target_lsn = '%s'" % target_lsn) if target_name: recovery_conf_lines.append("recovery_target_name = '%s'" % target_name) # TODO: log a warning if PostgreSQL < 9.4 and --immediate if backup_info.version >= 90400 and immediate: recovery_conf_lines.append("recovery_target = 'immediate'") # Manage what happens after recovery target is reached if (target_xid or target_time or target_lsn) and exclusive: recovery_conf_lines.append( "recovery_target_inclusive = '%s'" % (not exclusive) ) if target_tli: recovery_conf_lines.append("recovery_target_timeline = %s" % target_tli) # Write recovery target action if "pause_at_recovery_target" in recovery_info: recovery_conf_lines.append( "pause_at_recovery_target = '%s'" % recovery_info["pause_at_recovery_target"] ) if "recovery_target_action" in recovery_info: recovery_conf_lines.append( "recovery_target_action = '%s'" % recovery_info["recovery_target_action"] ) # Set the standby mode if backup_info.version >= 120000: signal_file = "recovery.signal" if standby_mode: signal_file = "standby.signal" if remote_command: recovery_file = os.path.join(recovery_info["tempdir"], signal_file) else: recovery_file = os.path.join(dest, signal_file) open(recovery_file, "ab").close() recovery_info["auto_conf_append_lines"] = recovery_conf_lines else: if standby_mode: recovery_conf_lines.append("standby_mode = 'on'") if remote_command: recovery_file = os.path.join(recovery_info["tempdir"], "recovery.conf") else: recovery_file = os.path.join(dest, "recovery.conf") with open(recovery_file, "wb") as recovery: recovery.write(("\n".join(recovery_conf_lines) + "\n").encode("utf-8")) if remote_command: plain_rsync = RsyncPgData( path=self.server.path, ssh=remote_command, bwlimit=self.config.bandwidth_limit, network_compression=self.config.network_compression, ) try: plain_rsync.from_file_list( [os.path.basename(recovery_file)], recovery_info["tempdir"], ":%s" % dest, ) except CommandFailedException as e: output.error( "remote copy of %s failed: %s", os.path.basename(recovery_file), e ) output.close_and_exit() def _conf_files_exist(self, conf_files, backup_info, recovery_info): """ Determine whether the conf files in the supplied list exist in the backup represented by backup_info. Returns a map of conf_file:exists. """ exists = {} for conf_file in conf_files: source_path = os.path.join(backup_info.get_data_directory(), conf_file) exists[conf_file] = os.path.exists(source_path) return exists def _copy_conf_files_to_tempdir( self, backup_info, recovery_info, remote_command=None ): """ Copy conf files from the backup location to a temporary directory so that they can be checked and mangled. Returns a list of the paths to the temporary conf files. """ conf_file_paths = [] for conf_file in recovery_info["configuration_files"]: conf_file_path = os.path.join(recovery_info["tempdir"], conf_file) shutil.copy2( os.path.join(backup_info.get_data_directory(), conf_file), conf_file_path, ) conf_file_paths.append(conf_file_path) return conf_file_paths def _map_temporary_config_files(self, recovery_info, backup_info, remote_command): """ Map configuration files, by filling the 'temporary_configuration_files' array, depending on remote or local recovery. This array will be used by the subsequent methods of the class. :param dict recovery_info: Dictionary containing all the recovery parameters :param barman.infofile.LocalBackupInfo backup_info: a backup representation :param str remote_command: ssh command for remote recovery """ # Cycle over postgres configuration files which my be missing. # If a file is missing, we will be unable to restore it and # we will warn the user. # This can happen if we are using pg_basebackup and # a configuration file is located outside the data dir. # This is not an error condition, so we check also for # `pg_ident.conf` which is an optional file. hardcoded_files = ["pg_hba.conf", "pg_ident.conf"] conf_files = recovery_info["configuration_files"] + hardcoded_files conf_files_exist = self._conf_files_exist( conf_files, backup_info, recovery_info ) for conf_file, exists in conf_files_exist.items(): if not exists: recovery_info["results"]["missing_files"].append(conf_file) # Remove the file from the list of configuration files if conf_file in recovery_info["configuration_files"]: recovery_info["configuration_files"].remove(conf_file) conf_file_paths = [] if remote_command: # If the recovery is remote, copy the postgresql.conf # file in a temp dir conf_file_paths = self._copy_conf_files_to_tempdir( backup_info, recovery_info, remote_command ) else: conf_file_paths = [ os.path.join(recovery_info["destination_path"], conf_file) for conf_file in recovery_info["configuration_files"] ] recovery_info["temporary_configuration_files"].extend(conf_file_paths) if backup_info.version >= 120000: # Make sure the recovery configuration file ('postgresql.auto.conf', unless # a custom alternative was specified via recovery_conf_filename) exists in # recovery_info['temporary_configuration_files'] because the recovery # settings will end up there. conf_file = recovery_info["results"]["recovery_configuration_file"] # If the file did not exist it will have been removed from # recovery_info["configuration_files"] earlier in this method. if conf_file not in recovery_info["configuration_files"]: if remote_command: conf_file_path = os.path.join(recovery_info["tempdir"], conf_file) else: conf_file_path = os.path.join( recovery_info["destination_path"], conf_file ) # Touch the file into existence open(conf_file_path, "ab").close() recovery_info["temporary_configuration_files"].append(conf_file_path) def _analyse_temporary_config_files(self, recovery_info): """ Analyse temporary configuration files and identify dangerous options Mark all the dangerous options for the user to review. This procedure also changes harmful options such as 'archive_command'. :param dict recovery_info: dictionary holding all recovery parameters """ results = recovery_info["results"] config_mangeler = ConfigurationFileMangeler() validator = ConfigIssueDetection() # Check for dangerous options inside every config file for conf_file in recovery_info["temporary_configuration_files"]: append_lines = None conf_file_suffix = results["recovery_configuration_file"] if conf_file.endswith(conf_file_suffix): append_lines = recovery_info.get("auto_conf_append_lines") # Identify and comment out dangerous options, replacing them with # the appropriate values results["changes"] += config_mangeler.mangle_options( conf_file, "%s.origin" % conf_file, append_lines ) # Identify dangerous options and warn users about their presence results["warnings"] += validator.detect_issues(conf_file) def _copy_temporary_config_files(self, dest, remote_command, recovery_info): """ Copy modified configuration files using rsync in case of remote recovery :param str dest: destination directory of the recovery :param str remote_command: ssh command for remote connection :param dict recovery_info: Dictionary containing all the recovery parameters """ if remote_command: # If this is a remote recovery, rsync the modified files from the # temporary local directory to the remote destination directory. # The list of files is built from `temporary_configuration_files` instead # of `configuration_files` because `configuration_files` is not guaranteed # to include the recovery configuration file. file_list = [] for conf_path in recovery_info["temporary_configuration_files"]: conf_file = os.path.basename(conf_path) file_list.append("%s" % conf_file) file_list.append("%s.origin" % conf_file) try: recovery_info["rsync"].from_file_list( file_list, recovery_info["tempdir"], ":%s" % dest ) except CommandFailedException as e: output.error("remote copy of configuration files failed: %s", e) output.close_and_exit() def close(self): """ Cleanup operations for a recovery """ # Remove the temporary directories for temp_dir in self.temp_dirs: temp_dir.delete() self.temp_dirs = [] class RemoteConfigRecoveryExecutor(RecoveryExecutor): """ Recovery executor which retrieves config files from the recovery directory instead of the backup directory. Useful when the config files are not available in the backup directory (e.g. compressed backups). """ def _conf_files_exist(self, conf_files, backup_info, recovery_info): """ Determine whether the conf files in the supplied list exist in the backup represented by backup_info. :param list[str] conf_files: List of config files to be checked. :param BackupInfo backup_info: Backup information for the backup being recovered. :param dict recovery_info: Dictionary of recovery information. :rtype: dict[str,bool] :return: A dict representing a map of conf_file:exists. """ exists = {} for conf_file in conf_files: source_path = os.path.join(recovery_info["destination_path"], conf_file) exists[conf_file] = recovery_info["cmd"].exists(source_path) return exists def _copy_conf_files_to_tempdir( self, backup_info, recovery_info, remote_command=None ): """ Copy conf files from the backup location to a temporary directory so that they can be checked and mangled. :param BackupInfo backup_info: Backup information for the backup being recovered. :param dict recovery_info: Dictionary of recovery information. :param str remote_command: The ssh command to be used when copying the files. :rtype: list[str] :return: A list of paths to the destination conf files. """ conf_file_paths = [] rsync = RsyncPgData( path=self.server.path, ssh=remote_command, bwlimit=self.config.bandwidth_limit, network_compression=self.config.network_compression, ) rsync.from_file_list( recovery_info["configuration_files"], ":" + recovery_info["destination_path"], recovery_info["tempdir"], ) conf_file_paths.extend( [ os.path.join(recovery_info["tempdir"], conf_file) for conf_file in recovery_info["configuration_files"] ] ) return conf_file_paths class TarballRecoveryExecutor(RemoteConfigRecoveryExecutor): """ A specialised recovery method for compressed backups. Inheritence is not necessarily the best thing here since the two RecoveryExecutor classes only differ by this one method, and the same will be true for future RecoveryExecutors (i.e. ones which handle encryption). Nevertheless for a wip "make it work" effort this will do. """ BASE_TARBALL_NAME = "base" def __init__(self, backup_manager, compression): """ Constructor :param barman.backup.BackupManager backup_manager: the BackupManager owner of the executor :param compression Compression. """ super(TarballRecoveryExecutor, self).__init__(backup_manager) self.compression = compression def _backup_copy( self, backup_info, dest, tablespaces=None, remote_command=None, safe_horizon=None, recovery_info=None, ): # Set a ':' prefix to remote destinations dest_prefix = "" if remote_command: dest_prefix = ":" # Instead of adding the `data` directory and `tablespaces` to a copy # controller we instead want to copy just the tarballs to a staging # location via the copy controller and then untar into place. # Create the staging area staging_dir = os.path.join( self.config.recovery_staging_path, "barman-staging-{}-{}".format(self.config.name, backup_info.backup_id), ) output.info( "Staging compressed backup files on the recovery host in: %s", staging_dir ) recovery_info["cmd"].create_dir_if_not_exists(staging_dir, mode="700") recovery_info["cmd"].validate_file_mode(staging_dir, mode="700") recovery_info["staging_dir"] = staging_dir self.temp_dirs.append( fs.UnixCommandPathDeletionCommand(staging_dir, recovery_info["cmd"]) ) # Create the copy controller object, specific for rsync. # Network compression is always disabled because we are copying # data which has already been compressed. controller = RsyncCopyController( path=self.server.path, ssh_command=remote_command, network_compression=False, retry_times=self.config.basebackup_retry_times, retry_sleep=self.config.basebackup_retry_sleep, workers=self.config.parallel_jobs, workers_start_batch_period=self.config.parallel_jobs_start_batch_period, workers_start_batch_size=self.config.parallel_jobs_start_batch_size, ) # Add the tarballs to the controller if backup_info.tablespaces: for tablespace in backup_info.tablespaces: tablespace_file = "%s.%s" % ( tablespace.oid, self.compression.file_extension, ) tablespace_path = "%s/%s" % ( backup_info.get_data_directory(), tablespace_file, ) controller.add_file( label=tablespace.name, src=tablespace_path, dst="%s/%s" % (dest_prefix + staging_dir, tablespace_file), item_class=controller.TABLESPACE_CLASS, bwlimit=self.config.get_bwlimit(tablespace), ) base_file = "%s.%s" % (self.BASE_TARBALL_NAME, self.compression.file_extension) base_path = "%s/%s" % ( backup_info.get_data_directory(), base_file, ) controller.add_file( label="pgdata", src=base_path, dst="%s/%s" % (dest_prefix + staging_dir, base_file), item_class=controller.PGDATA_CLASS, bwlimit=self.config.get_bwlimit(), ) controller.add_file( label="pgdata", src=os.path.join(backup_info.get_data_directory(), "backup_manifest"), dst=os.path.join(dest_prefix + dest, "backup_manifest"), item_class=controller.PGDATA_CLASS, bwlimit=self.config.get_bwlimit(), ) # Execute the copy try: controller.copy() except CommandFailedException as e: msg = "data transfer failure" raise DataTransferFailure.from_command_error("rsync", e, msg) # Untar the results files to their intended location if backup_info.tablespaces: for tablespace in backup_info.tablespaces: # By default a tablespace goes in the same location where # it was on the source server when the backup was taken tablespace_dst_path = tablespace.location # If a relocation has been requested for this tablespace # use the user provided target directory if tablespaces and tablespace.name in tablespaces: tablespace_dst_path = tablespaces[tablespace.name] tablespace_file = "%s.%s" % ( tablespace.oid, self.compression.file_extension, ) tablespace_src_path = "%s/%s" % (staging_dir, tablespace_file) _logger.debug( "Uncompressing tablespace %s from %s to %s", tablespace.name, tablespace_src_path, tablespace_dst_path, ) cmd_output = self.compression.uncompress( tablespace_src_path, tablespace_dst_path ) _logger.debug( "Uncompression output for tablespace %s: %s", tablespace.name, cmd_output, ) base_src_path = "%s/%s" % (staging_dir, base_file) _logger.debug("Uncompressing base tarball from %s to %s.", base_src_path, dest) cmd_output = self.compression.uncompress( base_src_path, dest, exclude=["recovery.conf", "tablespace_map"] ) _logger.debug("Uncompression output for base tarball: %s", cmd_output) class SnapshotRecoveryExecutor(RemoteConfigRecoveryExecutor): """ Recovery executor which performs barman recovery tasks for a backup taken with backup_method snapshot. It is responsible for: - Checking that disks cloned from the snapshots in the backup are attached to the recovery instance and that they are mounted at the correct location with the expected options. - Copying the backup_label into place. - Applying the requested recovery options to the PostgreSQL configuration. It does not handle the creation of the recovery instance, the creation of new disks from the snapshots or the attachment of the disks to the recovery instance. These are expected to have been performed before the `barman recover` runs. """ def _prepare_tablespaces(self, backup_info, cmd, dest, tablespaces): """ There is no need to prepare tablespace directories because they will already be present on the recovery instance through the cloning of disks from the backup snapshots. This function is therefore a no-op. """ pass @staticmethod def check_recovery_dir_exists(recovery_dir, cmd): """ Verify that the recovery directory already exists. :param str recovery_dir: Path to the recovery directory on the recovery instance :param UnixLocalCommand cmd: The command wrapper for running commands on the recovery instance. """ if not cmd.check_directory_exists(recovery_dir): message = ( "Recovery directory '{}' does not exist on the recovery instance. " "Check all required disks have been created, attached and mounted." ).format(recovery_dir) raise RecoveryPreconditionException(message) @staticmethod def get_attached_volumes_for_backup(snapshot_interface, backup_info, instance_name): """ Verifies that disks cloned from the snapshots specified in the supplied backup_info are attached to the named instance and returns them as a dict where the keys are snapshot names and the values are the names of the attached devices. If any snapshot associated with this backup is not found as the source for any disk attached to the instance then a RecoveryPreconditionException is raised. :param CloudSnapshotInterface snapshot_interface: Interface for managing snapshots via a cloud provider API. :param BackupInfo backup_info: Backup information for the backup being recovered. :param str instance_name: The name of the VM instance to which the disks to be backed up are attached. :rtype: dict[str,str] :return: A dict where the key is the snapshot name and the value is the device path for the source disk for that snapshot on the specified instance. """ if backup_info.snapshots_info is None: return {} attached_volumes = snapshot_interface.get_attached_volumes(instance_name) attached_volumes_for_backup = {} missing_snapshots = [] for source_snapshot in backup_info.snapshots_info.snapshots: try: disk, attached_volume = [ (k, v) for k, v in attached_volumes.items() if v.source_snapshot == source_snapshot.identifier ][0] attached_volumes_for_backup[disk] = attached_volume except IndexError: missing_snapshots.append(source_snapshot.identifier) if len(missing_snapshots) > 0: raise RecoveryPreconditionException( "The following snapshots are not attached to recovery instance %s: %s" % (instance_name, ", ".join(missing_snapshots)) ) else: return attached_volumes_for_backup @staticmethod def check_mount_points(backup_info, attached_volumes, cmd): """ Check that each disk cloned from a snapshot is mounted at the same mount point as the original disk and with the same mount options. Raises a RecoveryPreconditionException if any of the devices supplied in attached_snapshots are not mounted at the mount point or with the mount options specified in the snapshot metadata. :param BackupInfo backup_info: Backup information for the backup being recovered. :param dict[str,barman.cloud.VolumeMetadata] attached_volumes: Metadata for the volumes attached to the recovery instance. :param UnixLocalCommand cmd: The command wrapper for running commands on the recovery instance. """ mount_point_errors = [] mount_options_errors = [] for disk, volume in sorted(attached_volumes.items()): try: volume.resolve_mounted_volume(cmd) mount_point = volume.mount_point mount_options = volume.mount_options except SnapshotBackupException as e: mount_point_errors.append( "Error finding mount point for disk %s: %s" % (disk, e) ) continue if mount_point is None: mount_point_errors.append( "Could not find disk %s at any mount point" % disk ) continue snapshot_metadata = next( metadata for metadata in backup_info.snapshots_info.snapshots if metadata.identifier == volume.source_snapshot ) expected_mount_point = snapshot_metadata.mount_point expected_mount_options = snapshot_metadata.mount_options if mount_point != expected_mount_point: mount_point_errors.append( "Disk %s cloned from snapshot %s is mounted at %s but %s was " "expected." % (disk, volume.source_snapshot, mount_point, expected_mount_point) ) if mount_options != expected_mount_options: mount_options_errors.append( "Disk %s cloned from snapshot %s is mounted with %s but %s was " "expected." % ( disk, volume.source_snapshot, mount_options, expected_mount_options, ) ) if len(mount_point_errors) > 0: raise RecoveryPreconditionException( "Error checking mount points: %s" % ", ".join(mount_point_errors) ) if len(mount_options_errors) > 0: raise RecoveryPreconditionException( "Error checking mount options: %s" % ", ".join(mount_options_errors) ) def recover( self, backup_info, dest, wal_dest=None, tablespaces=None, remote_command=None, target_tli=None, target_time=None, target_xid=None, target_lsn=None, target_name=None, target_immediate=False, exclusive=False, target_action=None, standby_mode=None, recovery_conf_filename=None, recovery_instance=None, ): """ Performs a recovery of a snapshot backup. This method should be called in a closing context. :param barman.infofile.BackupInfo backup_info: the backup to recover :param str dest: the destination directory :param str|None wal_dest: the destination directory for WALs when doing PITR. See :meth:`~barman.recovery_executor.RecoveryExecutor._set_pitr_targets` for more details. :param dict[str,str]|None tablespaces: a tablespace name -> location map (for relocation) :param str|None remote_command: The remote command to recover the base backup, in case of remote backup. :param str|None target_tli: the target timeline :param str|None target_time: the target time :param str|None target_xid: the target xid :param str|None target_lsn: the target LSN :param str|None target_name: the target name created previously with pg_create_restore_point() function call :param str|None target_immediate: end recovery as soon as consistency is reached :param bool exclusive: whether the recovery is exclusive or not :param str|None target_action: The recovery target action :param bool|None standby_mode: standby mode :param str|None recovery_conf_filename: filename for storing recovery configurations :param str|None recovery_instance: The name of the recovery node as it is known by the cloud provider """ snapshot_interface = get_snapshot_interface_from_backup_info( backup_info, self.server.config ) attached_volumes = self.get_attached_volumes_for_backup( snapshot_interface, backup_info, recovery_instance ) cmd = fs.unix_command_factory(remote_command, self.server.path) SnapshotRecoveryExecutor.check_mount_points(backup_info, attached_volumes, cmd) self.check_recovery_dir_exists(dest, cmd) return super(SnapshotRecoveryExecutor, self).recover( backup_info, dest, wal_dest=wal_dest, tablespaces=None, remote_command=remote_command, target_tli=target_tli, target_time=target_time, target_xid=target_xid, target_lsn=target_lsn, target_name=target_name, target_immediate=target_immediate, exclusive=exclusive, target_action=target_action, standby_mode=standby_mode, recovery_conf_filename=recovery_conf_filename, ) def _start_backup_copy_message(self): """ Write the start backup copy message to the output. """ output.info("Copying the backup label.") def _backup_copy_failure_message(self, e): """ Write the backup failure message to the output. """ output.error("Failure copying the backup label: %s", e) def _backup_copy(self, backup_info, dest, remote_command=None, **kwargs): """ Copy any files from the backup directory which are required by the snapshot recovery (currently only the backup_label). :param barman.infofile.LocalBackupInfo backup_info: the backup to recover :param str dest: the destination directory """ # Set a ':' prefix to remote destinations dest_prefix = "" if remote_command: dest_prefix = ":" # Create the copy controller object, specific for rsync, # which will drive all the copy operations. Items to be # copied are added before executing the copy() method controller = RsyncCopyController( path=self.server.path, ssh_command=remote_command, network_compression=self.config.network_compression, retry_times=self.config.basebackup_retry_times, retry_sleep=self.config.basebackup_retry_sleep, workers=self.config.parallel_jobs, workers_start_batch_period=self.config.parallel_jobs_start_batch_period, workers_start_batch_size=self.config.parallel_jobs_start_batch_size, ) backup_label_file = "%s/%s" % (backup_info.get_data_directory(), "backup_label") controller.add_file( label="pgdata", src=backup_label_file, dst="%s/%s" % (dest_prefix + dest, "backup_label"), item_class=controller.PGDATA_CLASS, bwlimit=self.config.get_bwlimit(), ) # Execute the copy try: controller.copy() except CommandFailedException as e: msg = "data transfer failure" raise DataTransferFailure.from_command_error("rsync", e, msg) class IncrementalRecoveryExecutor(RemoteConfigRecoveryExecutor): """ Recovery executor for recovery of Postgres incremental backups. This class implements the combine backup process as well as the recovery of the newly combined backup by reusing some of the logic from the :class:`RecoveryExecutor` class. """ def __init__(self, backup_manager): """ Constructor :param barman.backup.BackupManager backup_manager: the :class:`BackupManager` owner of the executor """ super(IncrementalRecoveryExecutor, self).__init__(backup_manager) self.combine_start_time = None self.combine_end_time = None def recover(self, backup_info, dest, wal_dest=None, remote_command=None, **kwargs): """ Performs the recovery of an incremental backup. It first combines all backups in the backup chain, full to incremental, then proceeds with the recovery of the generated synthetic backup. This method should be called in a :func:`contextlib.closing` context. :param barman.infofile.BackupInfo backup_info: the incremental backup to recover :param str dest: the destination directory :param str|None wal_dest: the destination directory for WALs when doing PITR. See :meth:`~barman.recovery_executor.RecoveryExecutor._set_pitr_targets` for more details. :param str|None remote_command: The remote command to recover the base backup, in case of remote backup. :return dict: ``recovery_info`` dictionary, holding the values related with the recovery process. """ # First combine the backups, generating a new synthetic backup in the staging area combine_directory = self.config.local_staging_path synthetic_backup_info = self._combine_backups(backup_info, combine_directory) # Add the backup directory created in the staging area to be deleted after recovery synthetic_backup_dir = synthetic_backup_info.get_basebackup_directory() self.temp_dirs.append(fs.LocalLibPathDeletionCommand(synthetic_backup_dir)) # Perform the standard recovery process passing the synthetic backup recovery_info = super(IncrementalRecoveryExecutor, self).recover( synthetic_backup_info, dest, wal_dest, remote_command=remote_command, **kwargs, ) # If the checksum configuration is not consistent among all backups in the chain, we # raise a warning at the end so the user can optionally take action about it if not backup_info.is_checksum_consistent(): output.warning( "You restored from an incremental backup where checksums were enabled on " "that backup, but not all backups in the chain. It is advised to disable, and " "optionally re-enable, checksums on the destination directory to avoid failures." ) return recovery_info def _combine_backups(self, backup_info, dest): """ Combines the backup chain into a single synthetic backup using the ``pg_combinebackup`` utility. :param barman.infofile.LocalBackupInfo backup_info: the incremental backup to be recovered :param str dest: the directory where the synthetic backup is going to be mounted on :return barman.infofile.SyntheticBackupInfo: the backup info file of the combined backup """ self.combine_start_time = datetime.datetime.now() # Build the synthetic backup info from the incremental backup as it has # the most recent data relevant to the recovery. Also, the combine process # should be transparent to the end user so e.g. the .barman-recover.info file # that is created on destination and also the backup_id that is appended to the # manifest file in further steps of the recovery should be the same as the incremental synthetic_backup_info = SyntheticBackupInfo( self.server, base_directory=dest, backup_id=backup_info.backup_id, ) synthetic_backup_info.load(filename=backup_info.filename) dest_dirs = [synthetic_backup_info.get_data_directory()] # Maps the tablespaces from the old backup directory to the new synthetic # backup directory. This mapping is passed to the pg_combinebackup as input tbs_map = {} if backup_info.tablespaces: for tablespace in backup_info.tablespaces: source = backup_info.get_data_directory(tablespace_oid=tablespace.oid) destination = synthetic_backup_info.get_data_directory( tablespace_oid=tablespace.oid ) tbs_map[source] = destination dest_dirs.append(destination) # Prepare the destination directories for pgdata and tablespaces for _dir in dest_dirs: self._prepare_destination(_dir) # Retrieve pg_combinebackup version information remote_status = self._fetch_remote_status() # Get the backup chain data paths to be passed to the pg_combinebackup backups_chain = self._get_backup_chain_paths(backup_info) self._start_message(synthetic_backup_info) pg_combinebackup = PgCombineBackup( destination=synthetic_backup_info.get_data_directory(), command=remote_status["pg_combinebackup_path"], version=remote_status["pg_combinebackup_version"], app_name=None, tbs_mapping=tbs_map, retry_times=self.config.basebackup_retry_times, retry_sleep=self.config.basebackup_retry_sleep, retry_handler=partial(self._retry_handler, dest_dirs), out_handler=PgCombineBackup.make_logging_handler(logging.INFO), args=backups_chain, ) # Do the actual combine try: pg_combinebackup() except CommandFailedException as e: msg = "Combine action failure on directory '%s'" % dest raise DataTransferFailure.from_command_error("pg_combinebackup", e, msg) self._end_message(synthetic_backup_info) self.combine_end_time = datetime.datetime.now() combine_time = total_seconds(self.combine_end_time - self.combine_start_time) synthetic_backup_info.copy_stats = { "combine_time": combine_time, } return synthetic_backup_info def _backup_copy( self, backup_info, dest, tablespaces=None, remote_command=None, **kwargs, ): """ Perform the actual copy/move of the synthetic backup to destination :param barman.infofile.SyntheticBackupInfo backup_info: the synthetic backup info file :param str dest: the destination directory :param dict[str,str]|None tablespaces: a tablespace name -> location map (for relocation) :param str|None remote_command: default ``None``. The remote command to recover the backup, in case of remote backup """ # If it is a remote recovery we just follow the standard rsync copy process if remote_command: super(IncrementalRecoveryExecutor, self)._backup_copy( backup_info, dest, tablespaces, remote_command, **kwargs ) return # If it is a local recovery we move the content from staging to destination # Starts with tablespaces if backup_info.tablespaces: for tablespace in backup_info.tablespaces: # By default a tablespace goes in the same location where # it was on the source server when the backup was taken destination = tablespace.location # If a relocation has been requested for this tablespace # use the user provided target directory if tablespaces and tablespace.name in tablespaces: destination = tablespaces[tablespace.name] # Move the content of the tablespace directory to destination directory self._prepare_destination(destination) tbs_source = backup_info.get_data_directory( tablespace_oid=tablespace.oid ) self._move_to_destination(source=tbs_source, destination=destination) # Then procede to move the content of the data directory # We don't move the pg_tblspc as the _prepare_tablespaces method called earlier # in the process already created this directory and required symlinks in the destination # We also ignore any of the log directories and files not useful for the recovery data_source = backup_info.get_data_directory() self._move_to_destination( source=data_source, destination=dest, exclude_path_names={ "pg_tblspc", "pg_log", "log", "pg_xlog", "pg_wal", "postmaster.pid", "recovery.conf", "tablespace_map", }, ) def _move_to_destination(self, source, destination, exclude_path_names=set()): """ Move all files and directories contained within *source* to *destination*. :param str source: the source directory path from which underlying files and directories will be moved :param str destination: the destination directory path where to move the files and directories contained within *source* :param set[str] exclude_path_names: name of directories or files to be excluded from the moving action. """ for file_or_dir in os.listdir(source): if file_or_dir not in exclude_path_names: file_or_dir_path = os.path.join(source, file_or_dir) try: shutil.move(file_or_dir_path, destination) except shutil.Error: output.error( "Destination directory '%s' must be empty." % destination ) output.close_and_exit() def _get_backup_chain_paths(self, backup_info): """ Get the path of each backup in the chain, from the full backup to the specified incremental backup. :param barman.infofile.LocalBackupInfo backup_info: The incremental backup :return Iterator[barman.infofile.LocalBackupInfo]: iterator of paths of the backups in the chain, going from the full to the incremental backup pointed by *backup_info* """ return reversed( [backup.get_data_directory() for backup in backup_info.walk_to_root()] ) def _prepare_destination(self, dest_dir): """ Prepare the destination directory or file before moving it. This method is responsible for removing a directory if it already exists, then (re)creating it and ensuring the correct permissions on the directory. :param str dest_dir: destination directory """ # Remove a dir if exists. Ignore eventual errors shutil.rmtree(dest_dir, ignore_errors=True) # create the dir mkpath(dest_dir) # Ensure the right permissions to the destination directory # chmod 0700 octal os.chmod(dest_dir, 448) def _retry_handler(self, dest_dirs, attempt): """ Handler invoked during a combine backup in case of retry. The method simply warn the user of the failure and remove the already existing directories of the backup. :param list[str] dest_dirs: destination directories :param int attempt: attempt number (starting from 0) """ output.warning( "Failure combining backups using pg_combinebackup (attempt %s)", attempt ) output.warning( "The files created so far will be removed and " "the combine process will restart in %s seconds", "30", ) # Remove all the destination directories and reinit the backup for _dir in dest_dirs: self._prepare_destination(_dir) def _fetch_remote_status(self): """ Gather info from the remote server. This method does not raise any exception in case of errors, but set the missing values to ``None`` in the resulting dictionary. :return dict[str, str|bool]: the pg_combinebackup client information of the remote server. """ remote_status = dict.fromkeys( ( "pg_combinebackup_installed", "pg_combinebackup_path", "pg_combinebackup_version", ), None, ) # Test pg_combinebackup existence version_info = PgCombineBackup.get_version_info(self.server.path) if version_info["full_path"]: remote_status["pg_combinebackup_installed"] = True remote_status["pg_combinebackup_path"] = version_info["full_path"] remote_status["pg_combinebackup_version"] = version_info["full_version"] else: remote_status["pg_combinebackup_installed"] = False return remote_status def _start_message(self, backup_info): output.info( "Start combining backup via pg_combinebackup for backup %s on %s", backup_info.backup_id, backup_info.base_directory, ) def _end_message(self, backup_info): output.info( "End combining backup via pg_combinebackup for backup %s", backup_info.backup_id, ) def recovery_executor_factory(backup_manager, command, backup_info): """ Method in charge of building adequate RecoveryExecutor depending on the context :param: backup_manager :param: command barman.fs.UnixLocalCommand :return: RecoveryExecutor instance """ if backup_info.is_incremental: return IncrementalRecoveryExecutor(backup_manager) if backup_info.snapshots_info is not None: return SnapshotRecoveryExecutor(backup_manager) compression = backup_info.compression if compression is None: return RecoveryExecutor(backup_manager) if compression == GZipCompression.name: return TarballRecoveryExecutor(backup_manager, GZipCompression(command)) if compression == LZ4Compression.name: return TarballRecoveryExecutor(backup_manager, LZ4Compression(command)) if compression == ZSTDCompression.name: return TarballRecoveryExecutor(backup_manager, ZSTDCompression(command)) if compression == NoneCompression.name: return TarballRecoveryExecutor(backup_manager, NoneCompression(command)) raise AttributeError("Unexpected compression format: %s" % compression) class ConfigurationFileMangeler: # List of options that, if present, need to be forced to a specific value # during recovery, to avoid data losses OPTIONS_TO_MANGLE = { # Dangerous options "archive_command": "false", # Recovery options that may interfere with recovery targets "recovery_target": None, "recovery_target_name": None, "recovery_target_time": None, "recovery_target_xid": None, "recovery_target_lsn": None, "recovery_target_inclusive": None, "recovery_target_timeline": None, "recovery_target_action": None, } def mangle_options(self, filename, backup_filename=None, append_lines=None): """ This method modifies the given PostgreSQL configuration file, commenting out the given settings, and adding the ones generated by Barman. If backup_filename is passed, keep a backup copy. :param filename: the PostgreSQL configuration file :param backup_filename: config file backup copy. Default is None. :param append_lines: Additional lines to add to the config file :return [Assertion] """ # Read the full content of the file in memory with open(filename, "rb") as f: content = f.readlines() # Rename the original file to backup_filename or to a temporary name # if backup_filename is missing. We need to keep it to preserve # permissions. if backup_filename: orig_filename = backup_filename else: orig_filename = "%s.config_mangle.old" % filename shutil.move(filename, orig_filename) # Write the mangled content mangled = [] with open(filename, "wb") as f: last_line = None for l_number, line in enumerate(content): rm = PG_CONF_SETTING_RE.match(line.decode("utf-8")) if rm: key = rm.group(1) if key in self.OPTIONS_TO_MANGLE: value = self.OPTIONS_TO_MANGLE[key] f.write("#BARMAN#".encode("utf-8") + line) # If value is None, simply comment the old line if value is not None: changes = "%s = %s\n" % (key, value) f.write(changes.encode("utf-8")) mangled.append( Assertion._make( [os.path.basename(f.name), l_number, key, value] ) ) continue last_line = line f.write(line) # Append content of append_lines array if append_lines: # Ensure we have end of line character at the end of the file before adding new lines if last_line and last_line[-1] != "\n".encode("utf-8"): f.write("\n".encode("utf-8")) f.write(("\n".join(append_lines) + "\n").encode("utf-8")) # Restore original permissions shutil.copymode(orig_filename, filename) # If a backup copy of the file is not requested, # unlink the orig file if not backup_filename: os.unlink(orig_filename) return mangled class ConfigIssueDetection: # Potentially dangerous options list, which need to be revised by the user # after a recovery DANGEROUS_OPTIONS = [ "data_directory", "config_file", "hba_file", "ident_file", "external_pid_file", "ssl_cert_file", "ssl_key_file", "ssl_ca_file", "ssl_crl_file", "unix_socket_directory", "unix_socket_directories", "include", "include_dir", "include_if_exists", ] def detect_issues(self, filename): """ This method looks for any possible issue with PostgreSQL location options such as data_directory, config_file, etc. It returns a dictionary with the dangerous options that have been found. :param filename str: the Postgres configuration file :return clashes [Assertion] """ clashes = [] with open(filename) as f: content = f.readlines() # Read line by line and identify dangerous options for l_number, line in enumerate(content): rm = PG_CONF_SETTING_RE.match(line) if rm: key = rm.group(1) if key in self.DANGEROUS_OPTIONS: clashes.append( Assertion._make( [os.path.basename(f.name), l_number, key, rm.group(2)] ) ) return clashes barman-3.13.3/barman/config.py0000644000175100001660000022716415002177425014320 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module is responsible for all the things related to Barman configuration, such as parsing configuration file. """ import collections import datetime import inspect import json import logging.handlers import os import re import sys from copy import deepcopy from glob import iglob from typing import List from barman import output, utils try: from ConfigParser import ConfigParser, NoOptionError except ImportError: from configparser import ConfigParser, NoOptionError # create a namedtuple object called PathConflict with 'label' and 'server' PathConflict = collections.namedtuple("PathConflict", "label server") _logger = logging.getLogger(__name__) FORBIDDEN_SERVER_NAMES = ["all"] DEFAULT_USER = "barman" DEFAULT_CLEANUP = "true" DEFAULT_LOG_LEVEL = logging.INFO DEFAULT_LOG_FORMAT = "%(asctime)s [%(process)s] %(name)s %(levelname)s: %(message)s" _TRUE_RE = re.compile(r"""^(true|t|yes|1|on)$""", re.IGNORECASE) _FALSE_RE = re.compile(r"""^(false|f|no|0|off)$""", re.IGNORECASE) _TIME_INTERVAL_RE = re.compile( r""" ^\s* # N (day|month|week|hour) with optional 's' (\d+)\s+(day|month|week|hour)s? \s*$ """, re.IGNORECASE | re.VERBOSE, ) _SLOT_NAME_RE = re.compile("^[0-9a-z_]+$") _SI_SUFFIX_RE = re.compile(r"""(\d+)\s*(k|Ki|M|Mi|G|Gi|T|Ti)?\s*$""") REUSE_BACKUP_VALUES = ("copy", "link", "off") # Possible copy methods for backups (must be all lowercase) BACKUP_METHOD_VALUES = ["rsync", "postgres", "local-rsync", "snapshot"] CREATE_SLOT_VALUES = ["manual", "auto"] # Config values relating to pg_basebackup compression BASEBACKUP_COMPRESSIONS = ["gzip", "lz4", "zstd", "none"] class CsvOption(set): """ Base class for CSV options. Given a comma delimited string, this class is a list containing the submitted options. Internally, it uses a set in order to avoid option replication. Allowed values for the CSV option are contained in the 'value_list' attribute. The 'conflicts' attribute specifies for any value, the list of values that are prohibited (and thus generate a conflict). If a conflict is found, raises a ValueError exception. """ value_list = [] conflicts = {} def __init__(self, value, key, source): # Invoke parent class init and initialize an empty set super(CsvOption, self).__init__() # Parse not None values if value is not None: self.parse(value, key, source) # Validates the object structure before returning the new instance self.validate(key, source) def parse(self, value, key, source): """ Parses a list of values and correctly assign the set of values (removing duplication) and checking for conflicts. """ if not value: return values_list = value.split(",") for val in sorted(values_list): val = val.strip().lower() if val in self.value_list: # check for conflicting values. if a conflict is # found the option is not valid then, raise exception. if val in self.conflicts and self.conflicts[val] in self: raise ValueError( "Invalid configuration value '%s' for " "key %s in %s: cannot contain both " "'%s' and '%s'." "Configuration directive ignored." % (val, key, source, val, self.conflicts[val]) ) else: # otherwise use parsed value self.add(val) else: # not allowed value, reject the configuration raise ValueError( "Invalid configuration value '%s' for " "key %s in %s: Unknown option" % (val, key, source) ) def validate(self, key, source): """ Override this method for special validation needs """ def to_json(self): """ Output representation of the obj for JSON serialization The result is a string which can be parsed by the same class """ return ",".join(self) class BackupOptions(CsvOption): """ Extends CsvOption class providing all the details for the backup_options field """ # constants containing labels for allowed values EXCLUSIVE_BACKUP = "exclusive_backup" CONCURRENT_BACKUP = "concurrent_backup" EXTERNAL_CONFIGURATION = "external_configuration" # list holding all the allowed values for the BackupOption class value_list = [EXCLUSIVE_BACKUP, CONCURRENT_BACKUP, EXTERNAL_CONFIGURATION] # map holding all the possible conflicts between the allowed values conflicts = { EXCLUSIVE_BACKUP: CONCURRENT_BACKUP, CONCURRENT_BACKUP: EXCLUSIVE_BACKUP, } class RecoveryOptions(CsvOption): """ Extends CsvOption class providing all the details for the recovery_options field """ # constants containing labels for allowed values GET_WAL = "get-wal" # list holding all the allowed values for the RecoveryOptions class value_list = [GET_WAL] def parse_boolean(value): """ Parse a string to a boolean value :param str value: string representing a boolean :raises ValueError: if the string is an invalid boolean representation """ if _TRUE_RE.match(value): return True if _FALSE_RE.match(value): return False raise ValueError( "Invalid boolean representation (must be one in: " "true|t|yes|1|on | false|f|no|0|off)" ) def parse_time_interval(value): """ Parse a string, transforming it in a time interval. Accepted format: N (day|month|week)s :param str value: the string to evaluate """ # if empty string or none return none if value is None or value == "": return None result = _TIME_INTERVAL_RE.match(value) # if the string doesn't match, the option is invalid if not result: raise ValueError("Invalid value for a time interval %s" % value) # if the int conversion value = int(result.groups()[0]) unit = result.groups()[1][0].lower() # Calculates the time delta if unit == "d": time_delta = datetime.timedelta(days=value) elif unit == "w": time_delta = datetime.timedelta(weeks=value) elif unit == "m": time_delta = datetime.timedelta(days=(31 * value)) elif unit == "h": time_delta = datetime.timedelta(hours=value) else: # This should never happen raise ValueError("Invalid unit time %s" % unit) return time_delta def parse_si_suffix(value): """ Parse a string, transforming it into integer and multiplying by the SI or IEC suffix eg a suffix of Ki multiplies the integer value by 1024 and returns the new value Accepted format: N (k|Ki|M|Mi|G|Gi|T|Ti) :param str value: the string to evaluate """ # if empty string or none return none if value is None or value == "": return None result = _SI_SUFFIX_RE.match(value) if not result: raise ValueError("Invalid value for a number %s" % value) # if the int conversion value = int(result.groups()[0]) unit = result.groups()[1] # Calculates the value if unit == "k": value *= 1000 elif unit == "Ki": value *= 1024 elif unit == "M": value *= 1000000 elif unit == "Mi": value *= 1048576 elif unit == "G": value *= 1000000000 elif unit == "Gi": value *= 1073741824 elif unit == "T": value *= 1000000000000 elif unit == "Ti": value *= 1099511627776 return value def parse_reuse_backup(value): """ Parse a string to a valid reuse_backup value. Valid values are "copy", "link" and "off" :param str value: reuse_backup value :raises ValueError: if the value is invalid """ if value is None: return None if value.lower() in REUSE_BACKUP_VALUES: return value.lower() raise ValueError( "Invalid value (use '%s' or '%s')" % ("', '".join(REUSE_BACKUP_VALUES[:-1]), REUSE_BACKUP_VALUES[-1]) ) def parse_backup_compression(value): """ Parse a string to a valid backup_compression value. :param str value: backup_compression value :raises ValueError: if the value is invalid """ if value is None: return None if value.lower() in BASEBACKUP_COMPRESSIONS: return value.lower() raise ValueError( "Invalid value '%s'(must be one in: %s)" % (value, BASEBACKUP_COMPRESSIONS) ) def parse_backup_compression_format(value): """ Parse a string to a valid backup_compression format value. Valid values are "plain" and "tar" :param str value: backup_compression_location value :raises ValueError: if the value is invalid """ if value is None: return None if value.lower() in ("plain", "tar"): return value.lower() raise ValueError("Invalid value (must be either `plain` or `tar`)") def parse_backup_compression_location(value): """ Parse a string to a valid backup_compression location value. Valid values are "client" and "server" :param str value: backup_compression_location value :raises ValueError: if the value is invalid """ if value is None: return None if value.lower() in ("client", "server"): return value.lower() raise ValueError("Invalid value (must be either `client` or `server`)") def parse_backup_method(value): """ Parse a string to a valid backup_method value. Valid values are contained in BACKUP_METHOD_VALUES list :param str value: backup_method value :raises ValueError: if the value is invalid """ if value is None: return None if value.lower() in BACKUP_METHOD_VALUES: return value.lower() raise ValueError( "Invalid value (must be one in: '%s')" % ("', '".join(BACKUP_METHOD_VALUES)) ) def parse_staging_path(value): if value is None or os.path.isabs(value): return value raise ValueError("Invalid value : '%s' (must be an absolute path)" % value) def parse_slot_name(value): """ Replication slot names may only contain lower case letters, numbers, and the underscore character. This function parse a replication slot name :param str value: slot_name value :return: """ if value is None: return None value = value.lower() if not _SLOT_NAME_RE.match(value): raise ValueError( "Replication slot names may only contain lower case letters, " "numbers, and the underscore character." ) return value def parse_snapshot_disks(value): """ Parse a comma separated list of names used to reference disks managed by a cloud provider. :param str value: Comma separated list of disk names :return: List of disk names """ disk_names = value.split(",") # Verify each parsed disk is not an empty string for disk_name in disk_names: if disk_name == "": raise ValueError(disk_names) return disk_names def parse_create_slot(value): """ Parse a string to a valid create_slot value. Valid values are "manual" and "auto" :param str value: create_slot value :raises ValueError: if the value is invalid """ if value is None: return None value = value.lower() if value in CREATE_SLOT_VALUES: return value raise ValueError( "Invalid value (use '%s' or '%s')" % ("', '".join(CREATE_SLOT_VALUES[:-1]), CREATE_SLOT_VALUES[-1]) ) class BaseConfig(object): """ Contains basic methods for handling configuration of Servers and Models. You are expected to inherit from this class and define at least the :cvar:`PARSERS` dictionary with a mapping of parsers for each suported configuration option. """ PARSERS = {} def invoke_parser(self, key, source, value, new_value): """ Function used for parsing configuration values. If needed, it uses special parsers from the PARSERS map, and handles parsing exceptions. Uses two values (value and new_value) to manage configuration hierarchy (server config overwrites global config). :param str key: the name of the configuration option :param str source: the section that contains the configuration option :param value: the old value of the option if present. :param str new_value: the new value that needs to be parsed :return: the parsed value of a configuration option """ # If the new value is None, returns the old value if new_value is None: return value # If we have a parser for the current key, use it to obtain the # actual value. If an exception is thrown, print a warning and # ignore the value. # noinspection PyBroadException if key in self.PARSERS: parser = self.PARSERS[key] try: # If the parser is a subclass of the CsvOption class # we need a different invocation, which passes not only # the value to the parser, but also the key name # and the section that contains the configuration if inspect.isclass(parser) and issubclass(parser, CsvOption): value = parser(new_value, key, source) else: value = parser(new_value) except Exception as e: output.warning( "Ignoring invalid configuration value '%s' for key %s in %s: %s", new_value, key, source, e, ) else: value = new_value return value class ServerConfig(BaseConfig): """ This class represents the configuration for a specific Server instance. """ KEYS = [ "active", "archiver", "archiver_batch_size", "autogenerate_manifest", "aws_await_snapshots_timeout", "aws_snapshot_lock_mode", "aws_snapshot_lock_duration", "aws_snapshot_lock_cool_off_period", "aws_snapshot_lock_expiration_date", "aws_profile", "aws_region", "azure_credential", "azure_resource_group", "azure_subscription_id", "backup_compression", "backup_compression_format", "backup_compression_level", "backup_compression_location", "backup_compression_workers", "backup_directory", "backup_method", "backup_options", "bandwidth_limit", "basebackup_retry_sleep", "basebackup_retry_times", "basebackups_directory", "check_timeout", "cluster", "compression", "conninfo", "custom_compression_filter", "custom_decompression_filter", "custom_compression_magic", "description", "disabled", "errors_directory", "forward_config_path", "gcp_project", "gcp_zone", "immediate_checkpoint", "incoming_wals_directory", "keepalive_interval", "last_backup_maximum_age", "last_backup_minimum_size", "last_wal_maximum_age", "local_staging_path", "max_incoming_wals_queue", "minimum_redundancy", "network_compression", "parallel_jobs", "parallel_jobs_start_batch_period", "parallel_jobs_start_batch_size", "path_prefix", "post_archive_retry_script", "post_archive_script", "post_backup_retry_script", "post_backup_script", "post_delete_script", "post_delete_retry_script", "post_recovery_retry_script", "post_recovery_script", "post_wal_delete_script", "post_wal_delete_retry_script", "pre_archive_retry_script", "pre_archive_script", "pre_backup_retry_script", "pre_backup_script", "pre_delete_script", "pre_delete_retry_script", "pre_recovery_retry_script", "pre_recovery_script", "pre_wal_delete_script", "pre_wal_delete_retry_script", "primary_checkpoint_timeout", "primary_conninfo", "primary_ssh_command", "recovery_options", "recovery_staging_path", "create_slot", "retention_policy", "retention_policy_mode", "reuse_backup", "slot_name", "snapshot_disks", "snapshot_gcp_project", # Deprecated, replaced by gcp_project "snapshot_instance", "snapshot_provider", "snapshot_zone", # Deprecated, replaced by gcp_zone "ssh_command", "streaming_archiver", "streaming_archiver_batch_size", "streaming_archiver_name", "streaming_backup_name", "streaming_conninfo", "streaming_wals_directory", "tablespace_bandwidth_limit", "wal_conninfo", "wal_retention_policy", "wal_streaming_conninfo", "wals_directory", "worm_mode", "xlogdb_directory", ] BARMAN_KEYS = [ "archiver", "archiver_batch_size", "autogenerate_manifest", "aws_await_snapshots_timeout", "aws_snapshot_lock_mode", "aws_snapshot_lock_duration", "aws_snapshot_lock_cool_off_period", "aws_snapshot_lock_expiration_date", "aws_profile", "aws_region", "azure_credential", "azure_resource_group", "azure_subscription_id", "backup_compression", "backup_compression_format", "backup_compression_level", "backup_compression_location", "backup_compression_workers", "backup_method", "backup_options", "bandwidth_limit", "basebackup_retry_sleep", "basebackup_retry_times", "check_timeout", "compression", "configuration_files_directory", "create_slot", "custom_compression_filter", "custom_decompression_filter", "custom_compression_magic", "forward_config_path", "gcp_project", "immediate_checkpoint", "keepalive_internval", "last_backup_maximum_age", "last_backup_minimum_size", "last_wal_maximum_age", "local_staging_path", "max_incoming_wals_queue", "minimum_redundancy", "network_compression", "parallel_jobs", "parallel_jobs_start_batch_period", "parallel_jobs_start_batch_size", "path_prefix", "post_archive_retry_script", "post_archive_script", "post_backup_retry_script", "post_backup_script", "post_delete_script", "post_delete_retry_script", "post_recovery_retry_script", "post_recovery_script", "post_wal_delete_script", "post_wal_delete_retry_script", "pre_archive_retry_script", "pre_archive_script", "pre_backup_retry_script", "pre_backup_script", "pre_delete_script", "pre_delete_retry_script", "pre_recovery_retry_script", "pre_recovery_script", "pre_wal_delete_script", "pre_wal_delete_retry_script", "primary_ssh_command", "recovery_options", "recovery_staging_path", "retention_policy", "retention_policy_mode", "reuse_backup", "slot_name", "snapshot_gcp_project", # Deprecated, replaced by gcp_project "snapshot_provider", "streaming_archiver", "streaming_archiver_batch_size", "streaming_archiver_name", "streaming_backup_name", "tablespace_bandwidth_limit", "wal_retention_policy", "worm_mode", "xlogdb_directory", ] DEFAULTS = { "active": "true", "archiver": "off", "archiver_batch_size": "0", "autogenerate_manifest": "false", "aws_await_snapshots_timeout": "3600", "backup_directory": "%(barman_home)s/%(name)s", "backup_method": "rsync", "backup_options": "", "basebackup_retry_sleep": "30", "basebackup_retry_times": "0", "basebackups_directory": "%(backup_directory)s/base", "check_timeout": "30", "cluster": "%(name)s", "disabled": "false", "errors_directory": "%(backup_directory)s/errors", "forward_config_path": "false", "immediate_checkpoint": "false", "incoming_wals_directory": "%(backup_directory)s/incoming", "keepalive_interval": "60", "minimum_redundancy": "0", "network_compression": "false", "parallel_jobs": "1", "parallel_jobs_start_batch_period": "1", "parallel_jobs_start_batch_size": "10", "primary_checkpoint_timeout": "0", "recovery_options": "", "create_slot": "manual", "retention_policy_mode": "auto", "streaming_archiver": "off", "streaming_archiver_batch_size": "0", "streaming_archiver_name": "barman_receive_wal", "streaming_backup_name": "barman_streaming_backup", "streaming_conninfo": "%(conninfo)s", "streaming_wals_directory": "%(backup_directory)s/streaming", "wal_retention_policy": "main", "wals_directory": "%(backup_directory)s/wals", "worm_mode": "off", "xlogdb_directory": "%(wals_directory)s", } FIXED = [ "disabled", ] PARSERS = { "active": parse_boolean, "archiver": parse_boolean, "archiver_batch_size": int, "autogenerate_manifest": parse_boolean, "aws_await_snapshots_timeout": int, "aws_snapshot_lock_duration": int, "aws_snapshot_lock_cool_off_period": int, "backup_compression": parse_backup_compression, "backup_compression_format": parse_backup_compression_format, "backup_compression_level": int, "backup_compression_location": parse_backup_compression_location, "backup_compression_workers": int, "backup_method": parse_backup_method, "backup_options": BackupOptions, "basebackup_retry_sleep": int, "basebackup_retry_times": int, "check_timeout": int, "disabled": parse_boolean, "forward_config_path": parse_boolean, "keepalive_interval": int, "immediate_checkpoint": parse_boolean, "last_backup_maximum_age": parse_time_interval, "last_backup_minimum_size": parse_si_suffix, "last_wal_maximum_age": parse_time_interval, "local_staging_path": parse_staging_path, "max_incoming_wals_queue": int, "network_compression": parse_boolean, "parallel_jobs": int, "parallel_jobs_start_batch_period": int, "parallel_jobs_start_batch_size": int, "primary_checkpoint_timeout": int, "recovery_options": RecoveryOptions, "recovery_staging_path": parse_staging_path, "create_slot": parse_create_slot, "reuse_backup": parse_reuse_backup, "snapshot_disks": parse_snapshot_disks, "streaming_archiver": parse_boolean, "streaming_archiver_batch_size": int, "slot_name": parse_slot_name, "worm_mode": parse_boolean, } def __init__(self, config, name): self.msg_list = [] self.config = config self.name = name self.barman_home = config.barman_home self.barman_lock_directory = config.barman_lock_directory self.lock_directory_cleanup = config.lock_directory_cleanup self.config_changes_queue = config.config_changes_queue config.validate_server_config(self.name) for key in ServerConfig.KEYS: value = None # Skip parameters that cannot be configured by users if key not in ServerConfig.FIXED: # Get the setting from the [name] section of config file # A literal None value is converted to an empty string new_value = config.get(name, key, self.__dict__, none_value="") source = "[%s] section" % name value = self.invoke_parser(key, source, value, new_value) # If the setting isn't present in [name] section of config file # check if it has to be inherited from the [barman] section if value is None and key in ServerConfig.BARMAN_KEYS: new_value = config.get("barman", key, self.__dict__, none_value="") source = "[barman] section" value = self.invoke_parser(key, source, value, new_value) # If the setting isn't present in [name] section of config file # and is not inherited from global section use its default # (if present) if value is None and key in ServerConfig.DEFAULTS: new_value = ServerConfig.DEFAULTS[key] % self.__dict__ source = "DEFAULTS" value = self.invoke_parser(key, source, value, new_value) # An empty string is a None value (bypassing inheritance # from global configuration) if value is not None and value == "" or value == "None": value = None setattr(self, key, value) self._active_model_file = os.path.join( self.backup_directory, ".active-model.auto" ) self.active_model = None def apply_model(self, model, from_cli=False): """Apply config from a model named *name*. :param model: the model to be applied. :param from_cli: ``True`` if this function has been called by the user through a command, e.g. ``barman-config-switch``. ``False`` if it has been called internally by Barman. ``INFO`` messages are written in the first case, ``DEBUG`` messages in the second case. """ writer_func = getattr(output, "info" if from_cli else "debug") if self.cluster != model.cluster: output.error( "Model '%s' has 'cluster=%s', which is not compatible with " "'cluster=%s' from server '%s'" % ( model.name, model.cluster, self.cluster, self.name, ) ) return # No need to apply the same model twice if self.active_model is not None and model.name == self.active_model.name: writer_func( "Model '%s' is already active for server '%s', " "skipping..." % (model.name, self.name) ) return writer_func("Applying model '%s' to server '%s'" % (model.name, self.name)) for option, value in model.get_override_options(): old_value = getattr(self, option) if old_value != value: writer_func( "Changing value of option '%s' for server '%s' " "from '%s' to '%s' through the model '%s'" % (option, self.name, old_value, value, model.name) ) setattr(self, option, value) if from_cli: # If the request came from the CLI, like from 'barman config-switch' # then we need to persist the change to disk. On the other hand, if # Barman is calling this method on its own, that's because it previously # already read the active model from that file, so there is no need # to persist it again to disk with open(self._active_model_file, "w") as f: f.write(model.name) self.active_model = model def reset_model(self): """Reset the active model for this server, if any.""" output.info("Resetting the active model for the server %s" % (self.name)) if os.path.isfile(self._active_model_file): os.remove(self._active_model_file) self.active_model = None def to_json(self, with_source=False): """ Return an equivalent dictionary that can be encoded in json :param with_source: if we should include the source file that provides the effective value for each configuration option. :return: a dictionary. The structure depends on *with_source* argument: * If ``False``: key is the option name, value is its value; * If ``True``: key is the option name, value is a dict with a couple keys: * ``value``: the value of the option; * ``source``: the file which provides the effective value, if the option has been configured by the user, otherwise ``None``. """ json_dict = dict(vars(self)) # remove references that should not go inside the # `servers -> SERVER -> config` key in the barman diagnose output # ideally we should change this later so we only consider configuration # options, as things like `msg_list` are going to the `config` key, # i.e. we might be interested in considering only `ServerConfig.KEYS` # here instead of `vars(self)` for key in ["config", "_active_model_file", "active_model"]: del json_dict[key] # options that are override by the model override_options = set() if self.active_model: override_options = { option for option, _ in self.active_model.get_override_options() } if with_source: for option, value in json_dict.items(): name = self.name if option in override_options: name = self.active_model.name json_dict[option] = { "value": value, "source": self.config.get_config_source(name, option), } return json_dict def get_bwlimit(self, tablespace=None): """ Return the configured bandwidth limit for the provided tablespace If tablespace is None, it returns the global bandwidth limit :param barman.infofile.Tablespace tablespace: the tablespace to copy :rtype: str """ # Default to global bandwidth limit bwlimit = self.bandwidth_limit if tablespace: # A tablespace can be copied using a per-tablespace bwlimit tbl_bw_limit = self.tablespace_bandwidth_limit if tbl_bw_limit and tablespace.name in tbl_bw_limit: bwlimit = tbl_bw_limit[tablespace.name] return bwlimit def update_msg_list_and_disable_server(self, msg_list): """ Will take care of upgrading msg_list :param msg_list: str|list can be either a string or a list of strings """ if not msg_list: return if type(msg_list) is not list: msg_list = [msg_list] self.msg_list.extend(msg_list) self.disabled = True def get_wal_conninfo(self): """ Return WAL-specific conninfo strings for this server. Returns the value of ``wal_streaming_conninfo`` and ``wal_conninfo`` if they are set in the configuration. If ``wal_conninfo`` is unset then it will be given the value of ``wal_streaming_conninfo``. If ``wal_streaming_conninfo`` is unset then fall back to ``streaming_conninfo`` and ``conninfo``. :rtype: (str,str) :return: Tuple consisting of the ``wal_streaming_conninfo`` and ``wal_conninfo``. """ # If `wal_streaming_conninfo` is not set, fall back to `streaming_conninfo` wal_streaming_conninfo = self.wal_streaming_conninfo or self.streaming_conninfo # If `wal_conninfo` is not set, fall back to `wal_streaming_conninfo`. If # `wal_streaming_conninfo` is not set, fall back to `conninfo`. if self.wal_conninfo is not None: wal_conninfo = self.wal_conninfo elif self.wal_streaming_conninfo is not None: wal_conninfo = self.wal_streaming_conninfo else: wal_conninfo = self.conninfo return wal_streaming_conninfo, wal_conninfo class ModelConfig(BaseConfig): """ This class represents the configuration for a specific model of a server. :cvar KEYS: list of configuration options that are allowed in a model. :cvar REQUIRED_KEYS: list of configuration options that must always be set when defining a configuration model. :cvar PARSERS: mapping of parsers for the configuration options, if they need special handling. """ # Keys from ServerConfig which are not allowed in a configuration model. # They are mostly related with paths or hooks, which are not expected to # be changed at all with a model. _KEYS_BLACKLIST = { # Path related options "backup_directory", "basebackups_directory", "errors_directory", "incoming_wals_directory", "streaming_wals_directory", "wals_directory", # Although xlogdb_directory could be set with the same value for two # servers (the xlog.db is now called SERVER-xlog.db, avoiding conflicts) # we exclude it from models to follow the same pattern defined for all # the path settings. "xlogdb_directory", # Hook related options "post_archive_retry_script", "post_archive_script", "post_backup_retry_script", "post_backup_script", "post_delete_script", "post_delete_retry_script", "post_recovery_retry_script", "post_recovery_script", "post_wal_delete_script", "post_wal_delete_retry_script", "pre_archive_retry_script", "pre_archive_script", "pre_backup_retry_script", "pre_backup_script", "pre_delete_script", "pre_delete_retry_script", "pre_recovery_retry_script", "pre_recovery_script", "pre_wal_delete_script", "pre_wal_delete_retry_script", } KEYS = list((set(ServerConfig.KEYS) | {"model"}) - _KEYS_BLACKLIST) REQUIRED_KEYS = [ "cluster", "model", ] PARSERS = deepcopy(ServerConfig.PARSERS) PARSERS.update({"model": parse_boolean}) for key in _KEYS_BLACKLIST: PARSERS.pop(key, None) def __init__(self, config, name): self.config = config self.name = name config.validate_model_config(self.name) for key in ModelConfig.KEYS: value = None # Get the setting from the [name] section of config file # A literal None value is converted to an empty string new_value = config.get(name, key, self.__dict__, none_value="") source = "[%s] section" % name value = self.invoke_parser(key, source, value, new_value) # An empty string is a None value if value is not None and value == "" or value == "None": value = None setattr(self, key, value) def get_override_options(self): """ Get a list of options which values in the server should be override. :yield: tuples os option name and value which should override the value specified in the server with the value specified in the model. """ for option in set(self.KEYS) - set(self.REQUIRED_KEYS): value = getattr(self, option) if value is not None: yield option, value def to_json(self, with_source=False): """ Return an equivalent dictionary that can be encoded in json :param with_source: if we should include the source file that provides the effective value for each configuration option. :return: a dictionary. The structure depends on *with_source* argument: * If ``False``: key is the option name, value is its value; * If ``True``: key is the option name, value is a dict with a couple keys: * ``value``: the value of the option; * ``source``: the file which provides the effective value, if the option has been configured by the user, otherwise ``None``. """ json_dict = {} for option in self.KEYS: value = getattr(self, option) if with_source: value = { "value": value, "source": self.config.get_config_source(self.name, option), } json_dict[option] = value return json_dict class ConfigMapping(ConfigParser): """Wrapper for :class:`ConfigParser`. Extend the facilities provided by a :class:`ConfigParser` object, and additionally keep track of the source file for each configuration option. This is very useful as Barman allows the user to provide configuration options spread over multiple files in the system, so one can know which file provides the value for a configuration option in use. .. note:: When using this class you are expected to use :meth:`read_config` instead of any ``read*`` method exposed by :class:`ConfigParser`. """ def __init__(self, *args, **kwargs): """Create a new instance of :class:`ConfigMapping`. .. note:: We save *args* and *kwargs* so we can instantiate a temporary :class:`ConfigParser` with similar options on :meth:`read_config`. :param args: positional arguments to be passed down to :class:`ConfigParser`. :param kwargs: keyword arguments to be passed down to :class:`ConfigParser`. """ self._args = args self._kwargs = kwargs self._mapping = {} super().__init__(*args, **kwargs) def read_config(self, filename): """ Read and merge configuration options from *filename*. :param filename: path to a configuration file or its file descriptor in reading mode. :return: a list of file names which were able to be parsed, so we are compliant with the return value of :meth:`ConfigParser.read`. In practice the list will always contain at most one item. If *filename* is a descriptor with no ``name`` attribute, the corresponding entry in the list will be ``None``. """ filenames = [] tmp_parser = ConfigParser(*self._args, **self._kwargs) # A file descriptor if hasattr(filename, "read"): try: # Python 3.x tmp_parser.read_file(filename) except AttributeError: # Python 2.x tmp_parser.readfp(filename) if hasattr(filename, "name"): filenames.append(filename.name) else: filenames.append(None) # A file path else: for name in tmp_parser.read(filename): filenames.append(name) # Merge configuration options from the temporary parser into the global # parser, and update the mapping of options for section in tmp_parser.sections(): if not self.has_section(section): self.add_section(section) self._mapping[section] = {} for option, value in tmp_parser[section].items(): self.set(section, option, value) self._mapping[section][option] = filenames[0] return filenames def get_config_source(self, section, option): """Get the source INI file from which a config value comes from. :param section: the section of the configuration option. :param option: the name of the configuraion option. :return: the file that provides the effective value for *section* -> *option*. If no such configuration exists in the mapping, we assume it has a default value and return the ``default`` string. """ source = self._mapping.get(section, {}).get(option, None) # The config was not defined on the server section, but maybe under # `barman` section? if source is None and section != "barman": source = self._mapping.get("barman", {}).get(option, None) return source or "default" class Config(object): """This class represents the barman configuration. Default configuration files are /etc/barman.conf, /etc/barman/barman.conf and ~/.barman.conf for a per-user configuration """ CONFIG_FILES = [ "~/.barman.conf", "/etc/barman.conf", "/etc/barman/barman.conf", ] _QUOTE_RE = re.compile(r"""^(["'])(.*)\1$""") def __init__(self, filename=None): # In Python 3 ConfigParser has changed to be strict by default. # Barman wants to preserve the Python 2 behavior, so we are # explicitly building it passing strict=False. try: # Python 3.x self._config = ConfigMapping(strict=False) except TypeError: # Python 2.x self._config = ConfigMapping() if filename: # If it is a file descriptor if hasattr(filename, "read"): self._config.read_config(filename) # If it is a path else: # check for the existence of the user defined file if not os.path.exists(filename): sys.exit("Configuration file '%s' does not exist" % filename) self._config.read_config(os.path.expanduser(filename)) else: # Check for the presence of configuration files # inside default directories for path in self.CONFIG_FILES: full_path = os.path.expanduser(path) if os.path.exists(full_path) and full_path in self._config.read_config( full_path ): filename = full_path break else: sys.exit( "Could not find any configuration file at " "default locations.\n" "Check Barman's documentation for more help." ) self.config_file = filename self._servers = None self._models = None self.servers_msg_list = [] self._parse_global_config() def get(self, section, option, defaults=None, none_value=None): """Method to get the value from a given section from Barman configuration """ if not self._config.has_section(section): return None try: value = self._config.get(section, option, raw=False, vars=defaults) if value == "None": value = none_value if value is not None: value = self._QUOTE_RE.sub(lambda m: m.group(2), value) return value except NoOptionError: return None def get_config_source(self, section, option): """Get the source INI file from which a config value comes from. .. seealso: See :meth:`ConfigMapping.get_config_source` for details on the interface as this method is just a wrapper for that. """ return self._config.get_config_source(section, option) def _parse_global_config(self): """ This method parses the global [barman] section """ self.barman_home = self.get("barman", "barman_home") self.config_changes_queue = ( self.get("barman", "config_changes_queue") or "%s/cfg_changes.queue" % self.barman_home ) self.barman_lock_directory = ( self.get("barman", "barman_lock_directory") or self.barman_home ) self.lock_directory_cleanup = parse_boolean( self.get("barman", "lock_directory_cleanup") or DEFAULT_CLEANUP ) self.user = self.get("barman", "barman_user") or DEFAULT_USER self.log_file = self.get("barman", "log_file") self.log_format = self.get("barman", "log_format") or DEFAULT_LOG_FORMAT self.log_level = self.get("barman", "log_level") or DEFAULT_LOG_LEVEL # save the raw barman section to be compared later in # _is_global_config_changed() method self._global_config = set(self._config.items("barman")) def global_config_to_json(self, with_source=False): """ Return an equivalent dictionary that can be encoded in json :param with_source: if we should include the source file that provides the effective value for each configuration option. :return: a dictionary. The structure depends on *with_source* argument: * If ``False``: key is the option name, value is its value; * If ``True``: key is the option name, value is a dict with a couple keys: * ``value``: the value of the option; * ``source``: the file which provides the effective value, if the option has been configured by the user, otherwise ``None``. """ json_dict = dict(self._global_config) if with_source: for option, value in json_dict.items(): json_dict[option] = { "value": value, "source": self.get_config_source("barman", option), } return json_dict def _is_global_config_changed(self): """Return true if something has changed in global configuration""" return self._global_config != set(self._config.items("barman")) def load_configuration_files_directory(self): """ Read the "configuration_files_directory" option and load all the configuration files with the .conf suffix that lie in that folder """ config_files_directory = self.get("barman", "configuration_files_directory") if not config_files_directory: return if not os.path.isdir(os.path.expanduser(config_files_directory)): _logger.warn( 'Ignoring the "configuration_files_directory" option as "%s" ' "is not a directory", config_files_directory, ) return for cfile in sorted( iglob(os.path.join(os.path.expanduser(config_files_directory), "*.conf")) ): self.load_config_file(cfile) def load_config_file(self, cfile): filename = os.path.basename(cfile) if os.path.exists(cfile): if os.path.isfile(cfile): # Load a file _logger.debug("Including configuration file: %s", filename) self._config.read_config(cfile) if self._is_global_config_changed(): msg = ( "the configuration file %s contains a not empty [barman] section" % filename ) _logger.fatal(msg) raise SystemExit("FATAL: %s" % msg) else: # Add an warning message that a file has been discarded _logger.warn("Discarding configuration file: %s (not a file)", filename) else: # Add an warning message that a file has been discarded _logger.warn("Discarding configuration file: %s (not found)", filename) def _is_model(self, name): """ Check if section *name* is a model. :param name: name of the config section. :return: ``True`` if section *name* is a model, ``False`` otherwise. :raises: :exc:`ValueError`: re-raised if thrown by :func:`parse_boolean`. """ try: value = self._config.get(name, "model") except NoOptionError: return False try: return parse_boolean(value) except ValueError as exc: raise exc def _populate_servers_and_models(self): """ Populate server list and model list from configuration file Also check for paths errors in configuration. If two or more paths overlap in a single server, that server is disabled. If two or more directory paths overlap between different servers an error is raised. """ # Populate servers if self._servers is not None and self._models is not None: return self._servers = {} self._models = {} # Cycle all the available configurations sections for section in self._config.sections(): if section == "barman": # skip global settings continue # Exit if the section has a reserved name if section in FORBIDDEN_SERVER_NAMES: msg = ( "the reserved word '%s' is not allowed as server name." "Please rename it." % section ) _logger.fatal(msg) raise SystemExit("FATAL: %s" % msg) if self._is_model(section): # Create a ModelConfig object self._models[section] = ModelConfig(self, section) else: # Create a ServerConfig object self._servers[section] = ServerConfig(self, section) # Check for conflicting paths in Barman configuration self._check_conflicting_paths() # Apply models if the hidden files say so self._apply_models() def _check_conflicting_paths(self): """ Look for conflicting paths intra-server and inter-server """ # All paths in configuration servers_paths = {} # Global errors list self.servers_msg_list = [] # Cycle all the available configurations sections for section in sorted(self.server_names()): # Paths map section_conf = self._servers[section] config_paths = { "backup_directory": section_conf.backup_directory, "basebackups_directory": section_conf.basebackups_directory, "errors_directory": section_conf.errors_directory, "incoming_wals_directory": section_conf.incoming_wals_directory, "streaming_wals_directory": section_conf.streaming_wals_directory, "wals_directory": section_conf.wals_directory, } # Check for path errors for label, path in sorted(config_paths.items()): # If the path does not conflict with the others, add it to the # paths map real_path = os.path.realpath(path) if real_path not in servers_paths: servers_paths[real_path] = PathConflict(label, section) else: if section == servers_paths[real_path].server: # Internal path error. # Insert the error message into the server.msg_list if real_path == path: self._servers[section].msg_list.append( "Conflicting path: %s=%s conflicts with " "'%s' for server '%s'" % ( label, path, servers_paths[real_path].label, servers_paths[real_path].server, ) ) else: # Symbolic link self._servers[section].msg_list.append( "Conflicting path: %s=%s (symlink to: %s) " "conflicts with '%s' for server '%s'" % ( label, path, real_path, servers_paths[real_path].label, servers_paths[real_path].server, ) ) # Disable the server self._servers[section].disabled = True else: # Global path error. # Insert the error message into the global msg_list if real_path == path: self.servers_msg_list.append( "Conflicting path: " "%s=%s for server '%s' conflicts with " "'%s' for server '%s'" % ( label, path, section, servers_paths[real_path].label, servers_paths[real_path].server, ) ) else: # Symbolic link self.servers_msg_list.append( "Conflicting path: " "%s=%s (symlink to: %s) for server '%s' " "conflicts with '%s' for server '%s'" % ( label, path, real_path, section, servers_paths[real_path].label, servers_paths[real_path].server, ) ) def _apply_models(self): """ For each Barman server, check for a pre-existing active model. If a hidden file with a pre-existing active model file exists, apply that on top of the server configuration. """ for server in self.servers(): active_model = None try: with open(server._active_model_file, "r") as f: active_model = f.read().strip() except FileNotFoundError: # If a file does not exist, even if the server has models # defined, none of them has ever been applied continue if active_model.strip() == "": # Try to protect itself from a bogus file continue model = self.get_model(active_model) if model is None: # The model used to exist, but it's no longer avaialble for # some reason server.update_msg_list_and_disable_server( [ "Model '%s' is set as the active model for the server " "'%s' but the model does not exist." % (active_model, server.name) ] ) continue server.apply_model(model) def server_names(self): """This method returns a list of server names""" self._populate_servers_and_models() return self._servers.keys() def servers(self): """This method returns a list of server parameters""" self._populate_servers_and_models() return self._servers.values() def get_server(self, name): """ Get the configuration of the specified server :param str name: the server name """ self._populate_servers_and_models() return self._servers.get(name, None) def model_names(self): """Get a list of model names. :return: a :class:`list` of configured model names. """ self._populate_servers_and_models() return self._models.keys() def models(self): """Get a list of models. :return: a :class:`list` of configured :class:`ModelConfig` objects. """ self._populate_servers_and_models() return self._models.values() def get_model(self, name): """Get the configuration of the specified model. :param name: the model name. :return: a :class:`ModelConfig` if the model exists, otherwise ``None``. """ self._populate_servers_and_models() return self._models.get(name, None) def validate_global_config(self): """ Validate global configuration parameters """ # Check for the existence of unexpected parameters in the # global section of the configuration file required_keys = [ "barman_home", ] self._detect_missing_keys(self._global_config, required_keys, "barman") keys = [ "barman_home", "barman_lock_directory", "barman_user", "lock_directory_cleanup", "config_changes_queue", "log_file", "log_level", "configuration_files_directory", ] keys.extend(ServerConfig.KEYS) self._validate_with_keys(self._global_config, keys, "barman") def validate_server_config(self, server): """ Validate configuration parameters for a specified server :param str server: the server name """ # Check for the existence of unexpected parameters in the # server section of the configuration file self._validate_with_keys(self._config.items(server), ServerConfig.KEYS, server) def validate_model_config(self, model): """ Validate configuration parameters for a specified model. :param model: the model name. """ # Check for the existence of unexpected parameters in the # model section of the configuration file self._validate_with_keys(self._config.items(model), ModelConfig.KEYS, model) # Check for keys that are missing, but which are required self._detect_missing_keys( self._config.items(model), ModelConfig.REQUIRED_KEYS, model ) @staticmethod def _detect_missing_keys(config_items, required_keys, section): """ Check config for any missing required keys :param config_items: list of tuples containing provided parameters along with their values :param required_keys: list of required keys :param section: source section (for error reporting) """ missing_key_detected = False config_keys = [item[0] for item in config_items] for req_key in required_keys: # if a required key is not found, then print an error if req_key not in config_keys: output.error( 'Parameter "%s" is required in [%s] section.' % (req_key, section), ) missing_key_detected = True if missing_key_detected: raise SystemExit( "Your configuration is missing required parameters. Exiting." ) @staticmethod def _validate_with_keys(config_items, allowed_keys, section): """ Check every config parameter against a list of allowed keys :param config_items: list of tuples containing provided parameters along with their values :param allowed_keys: list of allowed keys :param section: source section (for error reporting) """ for parameter in config_items: # if the parameter name is not in the list of allowed values, # then output a warning name = parameter[0] if name not in allowed_keys: output.warning( 'Invalid configuration option "%s" in [%s] ' "section.", name, section, ) class BaseChange: """ Base class for change objects. Provides methods for equality comparison, hashing, and conversion to tuple and dictionary. """ _fields = [] def __eq__(self, other): """ Equality support. :param other: other object to compare this one against. """ if isinstance(other, self.__class__): return self.as_tuple() == other.as_tuple() return False def __hash__(self): """ Hash/set support. :return: a hash of the tuple created though :meth:`as_tuple`. """ return hash(self.as_tuple()) def as_tuple(self) -> tuple: """ Convert to a tuple, ordered as :attr:`_fields`. :return: tuple of values for :attr:`_fields`. """ return tuple(vars(self)[k] for k in self._fields) def as_dict(self): """ Convert to a dictionary, using :attr:`_fields` as keys. :return: a dictionary where keys are taken from :attr:`_fields` and values are the corresponding values for those fields. """ return {k: vars(self)[k] for k in self._fields} class ConfigChange(BaseChange): """ Represents a configuration change received. :ivar key str: The key of the configuration change. :ivar value str: The value of the configuration change. :ivar config_file Optional[str]: The configuration file associated with the change, or ``None``. """ _fields = ["key", "value", "config_file"] def __init__(self, key, value, config_file=None): """ Initialize a :class:`ConfigChange` object. :param key str: the configuration setting to be changed. :param value str: the new configuration value. :param config_file Optional[str]: configuration file associated with the change, if any, or ``None``. """ self.key = key self.value = value self.config_file = config_file @classmethod def from_dict(cls, obj): """ Factory method for creating :class:`ConfigChange` objects from a dictionary. :param obj: Dictionary representing the configuration change. :type obj: :class:`dict` :return: Configuration change object. :rtype: :class:`ConfigChange` :raises: :exc:`ValueError`: If the dictionary is malformed. """ if set(obj.keys()) == set(cls._fields): return cls(**obj) raise ValueError("Malformed configuration change serialization: %r" % obj) class ConfigChangeSet(BaseChange): """Represents a set of :class:`ConfigChange` for a given configuration section. :ivar section str: name of the configuration section related with the changes. :ivar changes_set List[:class:`ConfigChange`]: list of configuration changes to be applied to the section. """ _fields = ["section", "changes_set"] def __init__(self, section, changes_set=None): """Initialize a new :class:`ConfigChangeSet` object. :param section str: name of the configuration section related with the changes. :param changes_set List[ConfigChange]: list of configuration changes to be applied to the *section*. """ self.section = section self.changes_set = changes_set if self.changes_set is None: self.changes_set = [] @classmethod def from_dict(cls, obj): """ Factory for configuration change objects. Generates configuration change objects starting from a dictionary with the same fields. .. note:: Handles both :class:`ConfigChange` and :class:`ConfigChangeSet` mapping. :param obj: Dictionary representing the configuration changes set. :type obj: :class:`dict` :return: Configuration set of changes. :rtype: :class:`ConfigChangeSet` :raises: :exc:`ValueError`: If the dictionary is malformed. """ if set(obj.keys()) == set(cls._fields): if len(obj["changes_set"]) > 0 and not isinstance( obj["changes_set"][0], ConfigChange ): obj["changes_set"] = [ ConfigChange.from_dict(c) for c in obj["changes_set"] ] return cls(**obj) if set(obj.keys()) == set(ConfigChange._fields): return ConfigChange(**obj) raise ValueError("Malformed configuration change serialization: %r" % obj) class ConfigChangesQueue: """ Wraps the management of the config changes queue. The :class:`ConfigChangesQueue` class provides methods to read, write, and manipulate a queue of configuration changes. It is designed to be used as a context manager to ensure proper opening and closing of the queue file. Once instantiated the queue can be accessed using the :attr:`queue` property. """ def __init__(self, queue_file): """ Initialize the :class:`ConfigChangesQueue` object. :param queue_file str: file where to persist the queue of changes to be processed. """ self.queue_file = queue_file self._queue = None self.open() @staticmethod def read_file(path) -> List[ConfigChangeSet]: """ Reads a json file containing a list of configuration changes. :return: the list of :class:`ConfigChangeSet` to be applied to Barman configuration sections. """ try: with open(path, "r") as queue_file: # Read the queue if exists return json.load(queue_file, object_hook=ConfigChangeSet.from_dict) except FileNotFoundError: return [] except json.JSONDecodeError: output.warning( "Malformed or empty configuration change queue: %s" % queue_file.name ) return [] def __enter__(self): """ Enter method for context manager. """ return self def __exit__(self, exc_type, exc_val, exc_tb): """ Closes the resource when exiting the context manager. """ self.close() @property def queue(self): """ Returns the queue object. If the queue object is not yet initialized, it will be opened before returning. :return: the queue object. """ if self._queue is None: self.open() return self._queue def open(self): """Open and parse the :attr:`queue_file` into :attr:`_queue`.""" self._queue = self.read_file(self.queue_file) def close(self): """Write the new content and close the :attr:`queue_file`.""" with open(self.queue_file + ".tmp", "w") as queue_file: # Dump the configuration change list into the queue file json.dump(self._queue, queue_file, cls=ConfigChangeSetEncoder, indent=2) # Juggle with the queue files to ensure consistency of # the queue even if Shelver is interrupted abruptly old_file_name = self.queue_file + ".old" try: os.rename(self.queue_file, old_file_name) except FileNotFoundError: old_file_name = None os.rename(self.queue_file + ".tmp", self.queue_file) if old_file_name: os.remove(old_file_name) self._queue = None class ConfigChangesProcessor: """ The class is responsible for processing the config changes to apply to the barman config """ def __init__(self, config): """Initialize a new :class:`ConfigChangesProcessor` object, :param config Config: the Barman configuration. """ self.config = config self.applied_changes = [] def receive_config_changes(self, changes): """ Process all the configuration *changes*. :param changes Dict[str, str]: each key is the name of a section to be updated, and the value is a dictionary of configuration options along with their values that should be updated in such section. """ # Get all the available configuration change files in order changes_list = [] for section in changes: original_section = deepcopy(section) section_name = None scope = section.pop("scope") if scope not in ["server", "model"]: output.warning( "%r has been ignored because 'scope' is " "invalid: '%s'. It should be either 'server' " "or 'model'.", original_section, scope, ) continue elif scope == "server": try: section_name = section.pop("server_name") except KeyError: output.warning( "%r has been ignored because 'server_name' is missing.", original_section, ) continue elif scope == "model": try: section_name = section.pop("model_name") except KeyError: output.warning( "%r has been ignored because 'model_name' is missing.", original_section, ) continue server_obj = self.config.get_server(section_name) model_obj = self.config.get_model(section_name) if scope == "server": # the section already exists as a model if model_obj is not None: output.warning( "%r has been ignored because '%s' is a model, not a server.", original_section, section_name, ) continue elif scope == "model": # the section already exists as a server if server_obj is not None: output.warning( "%r has been ignored because '%s' is a server, not a model.", original_section, section_name, ) continue # If the model does not exist yet in Barman if model_obj is None: # 'model=on' is required for models, so force that if the # user forgot 'model' or set it to something invalid section["model"] = "on" if "cluster" not in section: output.warning( "%r has been ignored because it is a " "new model but 'cluster' is missing.", original_section, ) continue # Instantiate the ConfigChangeSet object chg_set = ConfigChangeSet(section=section_name) for json_cng in section: file_name = self.config._config.get_config_source( section_name, json_cng ) # if the configuration change overrides a default value # then the source file is ".barman.auto.conf" if file_name == "default": file_name = os.path.expanduser( "%s/.barman.auto.conf" % self.config.barman_home ) chg = None # Instantiate the configuration change object chg = ConfigChange( json_cng, section[json_cng], file_name, ) chg_set.changes_set.append(chg) changes_list.append(chg_set) # If there are no configuration change we've nothing to do here if len(changes_list) == 0: _logger.debug("No valid changes submitted") return # Extend the queue with the new changes with ConfigChangesQueue(self.config.config_changes_queue) as changes_queue: changes_queue.queue.extend(changes_list) def process_conf_changes_queue(self): """ Process the configuration changes in the queue. This method iterates over the configuration changes in the queue and applies them one by one. If an error occurs while applying a change, it logs the error and raises an exception. :raises: :exc:`Exception`: If an error occurs while applying a change. """ try: chgs_set = None with ConfigChangesQueue(self.config.config_changes_queue) as changes_queue: # Cycle and apply the configuration changes while len(changes_queue.queue) > 0: chgs_set = changes_queue.queue[0] try: self.apply_change(chgs_set) except Exception as e: # Log that something went horribly wrong and re-raise msg = "Unable to process a set of changes. Exiting." output.error(msg) _logger.debug( "Error while processing %s. \nError: %s" % ( json.dumps( chgs_set, cls=ConfigChangeSetEncoder, indent=2 ), e, ), ) raise e # Remove the configuration change once succeeded changes_queue.queue.pop(0) self.applied_changes.append(chgs_set) except Exception as err: _logger.error("Cannot execute %s: %s", chgs_set, err) def apply_change(self, changes): """ Apply the given changes to the configuration files. :param changes List[ConfigChangeSet]: list of sections and their configuration options to be updated. """ changed_files = dict() for chg in changes.changes_set: changed_files[chg.config_file] = utils.edit_config( chg.config_file, changes.section, chg.key, chg.value, changed_files.get(chg.config_file), ) output.info( "Changing value of option '%s' for section '%s' " "from '%s' to '%s' through config-update." % ( chg.key, changes.section, self.config.get(changes.section, chg.key), chg.value, ) ) for file, lines in changed_files.items(): with open(file, "w") as cfg_file: cfg_file.writelines(lines) class ConfigChangeSetEncoder(json.JSONEncoder): """ JSON encoder for :class:`ConfigChange` and :class:`ConfigChangeSet` objects. """ def default(self, obj): if isinstance(obj, (ConfigChange, ConfigChangeSet)): # Let the base class default method raise the TypeError return dict(obj.as_dict()) return super().default(obj) # easy raw config diagnostic with python -m # noinspection PyProtectedMember def _main(): print("Active configuration settings:") r = Config() r.load_configuration_files_directory() for section in r._config.sections(): print("Section: %s" % section) for option in r._config.options(section): print( "\t%s = %s (from %s)" % (option, r.get(section, option), r.get_config_source(section, option)) ) if __name__ == "__main__": _main() barman-3.13.3/barman/postgres.py0000644000175100001660000021667015002177425014721 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module represents the interface towards a PostgreSQL server. """ import atexit import datetime import logging import os import signal import threading import time from abc import ABCMeta from multiprocessing import Process, Queue import psycopg2 from psycopg2.errorcodes import DUPLICATE_OBJECT, OBJECT_IN_USE, UNDEFINED_OBJECT from psycopg2.extensions import STATUS_IN_TRANSACTION, STATUS_READY from psycopg2.extras import DictCursor, NamedTupleCursor from barman.exceptions import ( BackupFunctionsAccessRequired, ConninfoException, PostgresAppNameError, PostgresCheckpointPrivilegesRequired, PostgresConnectionError, PostgresConnectionLost, PostgresDuplicateReplicationSlot, PostgresException, PostgresInvalidReplicationSlot, PostgresIsInRecovery, PostgresObsoleteFeature, PostgresReplicationSlotInUse, PostgresReplicationSlotsFull, PostgresUnsupportedFeature, ) from barman.infofile import Tablespace from barman.postgres_plumbing import function_name_map from barman.remote_status import RemoteStatusMixin from barman.utils import force_str, simplify_version, with_metaclass try: from queue import Empty except ImportError: from Queue import Empty # This is necessary because the CONFIGURATION_LIMIT_EXCEEDED constant # has been added in psycopg2 2.5, but Barman supports version 2.4.2+ so # in case of import error we declare a constant providing the correct value. try: from psycopg2.errorcodes import CONFIGURATION_LIMIT_EXCEEDED except ImportError: CONFIGURATION_LIMIT_EXCEEDED = "53400" _logger = logging.getLogger(__name__) _live_connections = [] """ List of connections to be closed at the interpreter shutdown """ @atexit.register def _atexit(): """ Ensure that all the connections are correctly closed at interpreter shutdown """ # Take a copy of the list because the conn.close() method modify it for conn in list(_live_connections): _logger.warning( "Forcing %s cleanup during process shut down.", conn.__class__.__name__ ) conn.close() class PostgreSQL(with_metaclass(ABCMeta, RemoteStatusMixin)): """ This abstract class represents a generic interface to a PostgreSQL server. """ CHECK_QUERY = "SELECT 1" MINIMAL_VERSION = 90600 def __init__(self, conninfo): """ Abstract base class constructor for PostgreSQL interface. :param str conninfo: Connection information (aka DSN) """ super(PostgreSQL, self).__init__() self.conninfo = conninfo self._conn = None self.allow_reconnect = True # Build a dictionary with connection info parameters # This is mainly used to speed up search in conninfo try: self.conn_parameters = self.parse_dsn(conninfo) except (ValueError, TypeError) as e: _logger.debug(e) raise ConninfoException( 'Cannot connect to postgres: "%s" ' "is not a valid connection string" % conninfo ) @staticmethod def parse_dsn(dsn): """ Parse connection parameters from 'conninfo' :param str dsn: Connection information (aka DSN) :rtype: dict[str,str] """ # TODO: this might be made more robust in the future return dict(x.split("=", 1) for x in dsn.split()) @staticmethod def encode_dsn(parameters): """ Build a connection string from a dictionary of connection parameters :param dict[str,str] parameters: Connection parameters :rtype: str """ # TODO: this might be made more robust in the future return " ".join(["%s=%s" % (k, v) for k, v in sorted(parameters.items())]) def get_connection_string(self, application_name=None): """ Return the connection string, adding the application_name parameter if requested, unless already defined by user in the connection string :param str application_name: the application_name to add :return str: the connection string """ conn_string = self.conninfo # check if the application name is already defined by user if application_name and "application_name" not in self.conn_parameters: # Then add the it to the connection string conn_string += " application_name=%s" % application_name # adopt a secure schema-usage pattern. See: # https://www.postgresql.org/docs/current/libpq-connect.html if "options" not in self.conn_parameters: conn_string += " options=-csearch_path=" return conn_string def connect(self): """ Generic function for Postgres connection (using psycopg2) """ if not self._check_connection(): try: self._conn = psycopg2.connect(self.conninfo) self._conn.autocommit = True # If psycopg2 fails to connect to the host, # raise the appropriate exception except psycopg2.DatabaseError as e: raise PostgresConnectionError(force_str(e).strip()) # Register the connection to the list of live connections _live_connections.append(self) return self._conn def _check_connection(self): """ Return false if the connection is broken :rtype: bool """ # If the connection is not present return False if not self._conn: return False # Check if the connection works by running 'SELECT 1' cursor = None initial_status = None try: initial_status = self._conn.status cursor = self._conn.cursor() cursor.execute(self.CHECK_QUERY) # Rollback if initial status was IDLE because the CHECK QUERY # has started a new transaction. if initial_status == STATUS_READY: self._conn.rollback() except psycopg2.DatabaseError: # Connection is broken, so we need to reconnect self.close() # Raise an error if reconnect is not allowed if not self.allow_reconnect: raise PostgresConnectionError( "Connection lost, reconnection not allowed" ) return False finally: if cursor: cursor.close() return True def close(self): """ Close the connection to PostgreSQL """ if self._conn: # If the connection is still alive, rollback and close it if not self._conn.closed: if self._conn.status == STATUS_IN_TRANSACTION: self._conn.rollback() self._conn.close() # Remove the connection from the live connections list self._conn = None _live_connections.remove(self) def _cursor(self, *args, **kwargs): """ Return a cursor """ conn = self.connect() return conn.cursor(*args, **kwargs) @property def server_version(self): """ Version of PostgreSQL (returned by psycopg2) """ conn = self.connect() return conn.server_version @property def server_txt_version(self): """ Human readable version of PostgreSQL (calculated from server_version) :rtype: str|None """ try: conn = self.connect() return self.int_version_to_string_version(conn.server_version) except PostgresConnectionError as e: _logger.debug( "Error retrieving PostgreSQL version: %s", force_str(e).strip() ) return None @property def minimal_txt_version(self): """ Human readable version of PostgreSQL (calculated from server_version) :rtype: str|None """ return self.int_version_to_string_version(self.MINIMAL_VERSION) @staticmethod def int_version_to_string_version(int_version): """ takes an int version :param int_version: ex: 10.22 121200 or 130800 :return: str ex 10.22.00 12.12.00 13.8.00 """ major = int(int_version / 10000) minor = int(int_version / 100 % 100) patch = int(int_version % 100) if major < 10: return "%d.%d.%d" % (major, minor, patch) if minor != 0: _logger.warning( "Unexpected non zero minor version %s in %s", minor, int_version, ) return "%d.%d" % (major, patch) @property def server_major_version(self): """ PostgreSQL major version (calculated from server_txt_version) :rtype: str|None """ result = self.server_txt_version if result is not None: return simplify_version(result) return None def is_minimal_postgres_version(self): """Checks if postgres version has at least minimal version""" return self.server_version >= self.MINIMAL_VERSION class StreamingConnection(PostgreSQL): """ This class represents a streaming connection to a PostgreSQL server. """ CHECK_QUERY = "IDENTIFY_SYSTEM" def __init__(self, conninfo): """ Streaming connection constructor :param str conninfo: Connection information (aka DSN) """ super(StreamingConnection, self).__init__(conninfo) # Make sure we connect using the 'replication' option which # triggers streaming replication protocol communication self.conn_parameters["replication"] = "true" # ensure that the datestyle is set to iso, working around an # issue in some psycopg2 versions self.conn_parameters["options"] = "-cdatestyle=iso" # Override 'dbname' parameter. This operation is required to mimic # the behaviour of pg_receivexlog and pg_basebackup self.conn_parameters["dbname"] = "replication" # Rebuild the conninfo string from the modified parameter lists self.conninfo = self.encode_dsn(self.conn_parameters) def connect(self): """ Connect to the PostgreSQL server. It reuses an existing connection. :returns: the connection to the server """ if self._check_connection(): return self._conn # Build a connection self._conn = super(StreamingConnection, self).connect() return self._conn def fetch_remote_status(self): """ Returns the status of the connection to the PostgreSQL server. This method does not raise any exception in case of errors, but set the missing values to None in the resulting dictionary. :rtype: dict[str, None|str] """ result = dict.fromkeys( ( "connection_error", "streaming_supported", "streaming", "streaming_systemid", "timeline", "xlogpos", "version_supported", ), None, ) try: # This needs to be protected by the try/except because # `self.is_minimal_postgres_version` can raise a PostgresConnectionError result["version_supported"] = self.is_minimal_postgres_version() if not self.is_minimal_postgres_version(): return result # streaming is always supported result["streaming_supported"] = True # Execute a IDENTIFY_SYSTEM to check the connection cursor = self._cursor() cursor.execute("IDENTIFY_SYSTEM") row = cursor.fetchone() # If something has been returned, barman is connected # to a replication backend if row: result["streaming"] = True # IDENTIFY_SYSTEM always returns at least two values result["streaming_systemid"] = row[0] result["timeline"] = row[1] # PostgreSQL 9.1+ returns also the current xlog flush location if len(row) > 2: result["xlogpos"] = row[2] except psycopg2.ProgrammingError: # This is not a streaming connection result["streaming"] = False except PostgresConnectionError as e: result["connection_error"] = force_str(e).strip() _logger.warning( "Error retrieving PostgreSQL status: %s", force_str(e).strip() ) return result def create_physical_repslot(self, slot_name): """ Create a physical replication slot using the streaming connection :param str slot_name: Replication slot name """ cursor = self._cursor() try: # In the following query, the slot name is directly passed # to the CREATE_REPLICATION_SLOT command, without any # quoting. This is a characteristic of the streaming # connection, otherwise if will fail with a generic # "syntax error" cursor.execute("CREATE_REPLICATION_SLOT %s PHYSICAL" % slot_name) _logger.info("Replication slot '%s' successfully created", slot_name) except psycopg2.DatabaseError as exc: if exc.pgcode == DUPLICATE_OBJECT: # A replication slot with the same name exists raise PostgresDuplicateReplicationSlot() elif exc.pgcode == CONFIGURATION_LIMIT_EXCEEDED: # Unable to create a new physical replication slot. # All slots are full. raise PostgresReplicationSlotsFull() else: raise PostgresException(force_str(exc).strip()) def drop_repslot(self, slot_name): """ Drop a physical replication slot using the streaming connection :param str slot_name: Replication slot name """ cursor = self._cursor() try: # In the following query, the slot name is directly passed # to the DROP_REPLICATION_SLOT command, without any # quoting. This is a characteristic of the streaming # connection, otherwise if will fail with a generic # "syntax error" cursor.execute("DROP_REPLICATION_SLOT %s" % slot_name) _logger.info("Replication slot '%s' successfully dropped", slot_name) except psycopg2.DatabaseError as exc: if exc.pgcode == UNDEFINED_OBJECT: # A replication slot with the that name does not exist raise PostgresInvalidReplicationSlot() if exc.pgcode == OBJECT_IN_USE: # The replication slot is still in use raise PostgresReplicationSlotInUse() else: raise PostgresException(force_str(exc).strip()) class PostgreSQLConnection(PostgreSQL): """ This class represents a standard client connection to a PostgreSQL server. """ # Streaming replication client types STANDBY = 1 WALSTREAMER = 2 ANY_STREAMING_CLIENT = (STANDBY, WALSTREAMER) HEARTBEAT_QUERY = "SELECT 1" def __init__( self, conninfo, immediate_checkpoint=False, slot_name=None, application_name="barman", ): """ PostgreSQL connection constructor. :param str conninfo: Connection information (aka DSN) :param bool immediate_checkpoint: Whether to do an immediate checkpoint when start a backup :param str|None slot_name: Replication slot name """ super(PostgreSQLConnection, self).__init__(conninfo) self.immediate_checkpoint = immediate_checkpoint self.slot_name = slot_name self.application_name = application_name self.configuration_files = None def connect(self): """ Connect to the PostgreSQL server. It reuses an existing connection. """ if self._check_connection(): return self._conn self._conn = super(PostgreSQLConnection, self).connect() if "application_name" not in self.conn_parameters: try: cur = self._conn.cursor() # Do not use parameter substitution with SET cur.execute("SET application_name TO %s" % self.application_name) cur.close() # If psycopg2 fails to set the application name, # raise the appropriate exception except psycopg2.ProgrammingError as e: raise PostgresAppNameError(force_str(e).strip()) return self._conn @property def has_connection(self): """Checks if the Postgres connection has already been set""" return True if self._conn is not None else False @property def server_txt_version(self): """ Human readable version of PostgreSQL (returned by the server). Note: The return value of this function is used when composing include patterns which are passed to rsync when copying tablespaces. If the value does not exactly match the PostgreSQL version then Barman may fail to copy tablespace files during a backup. """ try: cur = self._cursor() cur.execute("SELECT version()") version_string = cur.fetchone()[0] platform, version = version_string.split()[:2] # EPAS <= 10 will return a version string which starts with # EnterpriseDB followed by the PostgreSQL version with an # additional version field. This additional field must be discarded # so that we return the exact PostgreSQL version. Later versions of # EPAS report the PostgreSQL version directly so do not need # special handling. if platform == "EnterpriseDB": return ".".join(version.split(".")[:-1]) else: return version except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error retrieving PostgreSQL version: %s", force_str(e).strip() ) return None @property def is_in_recovery(self): """ Returns true if PostgreSQL server is in recovery mode (hot standby) """ try: cur = self._cursor() cur.execute("SELECT pg_is_in_recovery()") return cur.fetchone()[0] except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error calling pg_is_in_recovery() function: %s", force_str(e).strip() ) return None @property def is_superuser(self): """ Returns true if current user has superuser privileges """ try: cur = self._cursor() cur.execute("SELECT usesuper FROM pg_user WHERE usename = CURRENT_USER") return cur.fetchone()[0] except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error calling is_superuser() function: %s", force_str(e).strip() ) return None @property def has_backup_privileges(self): """ Returns true if current user is superuser or, for PostgreSQL 10 or above, is a standard user that has grants to read server settings and to execute all the functions needed for exclusive/concurrent backup control and WAL control. """ # pg_monitor / pg_read_all_settings only available from v10 if self.server_version < 100000: return self.is_superuser stop_fun_check = "" if self.server_version < 150000: pg_backup_start_args = "text,bool,bool" pg_backup_stop_args = "bool,bool" stop_fun_check = ( "has_function_privilege(" "CURRENT_USER, '{pg_backup_stop}()', 'EXECUTE') OR " ).format(**self.name_map) else: pg_backup_start_args = "text,bool" pg_backup_stop_args = "bool" start_fun_check = ( "has_function_privilege(" "CURRENT_USER, '{pg_backup_start}({pg_backup_start_args})', 'EXECUTE')" ).format(pg_backup_start_args=pg_backup_start_args, **self.name_map) stop_fun_check += ( "has_function_privilege(CURRENT_USER, " "'{pg_backup_stop}({pg_backup_stop_args})', 'EXECUTE')" ).format(pg_backup_stop_args=pg_backup_stop_args, **self.name_map) backup_check_query = """ SELECT usesuper OR ( ( pg_has_role(CURRENT_USER, 'pg_monitor', 'USAGE') OR ( pg_has_role(CURRENT_USER, 'pg_read_all_settings', 'USAGE') AND pg_has_role(CURRENT_USER, 'pg_read_all_stats', 'USAGE') ) ) AND ( {start_fun_check} ) AND ( {stop_fun_check} ) AND has_function_privilege( CURRENT_USER, 'pg_switch_wal()', 'EXECUTE') AND has_function_privilege( CURRENT_USER, 'pg_create_restore_point(text)', 'EXECUTE') ) FROM pg_user WHERE usename = CURRENT_USER """.format( start_fun_check=start_fun_check, stop_fun_check=stop_fun_check, **self.name_map ) try: cur = self._cursor() cur.execute(backup_check_query) return cur.fetchone()[0] except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error checking privileges for functions needed for backups: %s", force_str(e).strip(), ) return None @property def has_checkpoint_privileges(self): """ Returns true if the current user is a superuser or if, for PostgreSQL 14 and above, the user has the "pg_checkpoint" role. """ if self.server_version < 140000: return self.is_superuser if self.is_superuser: return True else: role_check_query = ( "select pg_has_role(CURRENT_USER ,'pg_checkpoint', 'USAGE');" ) try: cur = self._cursor() cur.execute(role_check_query) return cur.fetchone()[0] except (PostgresConnectionError, psycopg2.Error) as e: _logger.warning( "Error checking privileges for functions needed for creating checkpoints: %s", force_str(e).strip(), ) return None @property def has_monitoring_privileges(self): """ Check whether the current user can access monitoring information. Returns ``True`` if the current user is a superuser or if the user has the necessary privileges to monitor system status. :rtype: bool :return: ``True`` if the current user can access monitoring information. """ if self.is_superuser: return True else: monitoring_check_query = """ SELECT ( pg_has_role(CURRENT_USER, 'pg_monitor', 'USAGE') OR ( pg_has_role(CURRENT_USER, 'pg_read_all_settings', 'USAGE') AND pg_has_role(CURRENT_USER, 'pg_read_all_stats', 'USAGE') ) ) """ try: cur = self._cursor() cur.execute(monitoring_check_query) return cur.fetchone()[0] except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error checking privileges for functions needed for monitoring: %s", force_str(e).strip(), ) return None @property def current_xlog_info(self): """ Get detailed information about the current WAL position in PostgreSQL. This method returns a dictionary containing the following data: * location * file_name * file_offset * timestamp When executed on a standby server file_name and file_offset are always None :rtype: psycopg2.extras.DictRow """ try: cur = self._cursor(cursor_factory=DictCursor) if not self.is_in_recovery: cur.execute( "SELECT location, " "({pg_walfile_name_offset}(location)).*, " "CURRENT_TIMESTAMP AS timestamp " "FROM {pg_current_wal_lsn}() AS location".format(**self.name_map) ) return cur.fetchone() else: cur.execute( "SELECT location, " "NULL AS file_name, " "NULL AS file_offset, " "CURRENT_TIMESTAMP AS timestamp " "FROM {pg_last_wal_replay_lsn}() AS location".format( **self.name_map ) ) return cur.fetchone() except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error retrieving current xlog detailed information: %s", force_str(e).strip(), ) return None @property def current_xlog_file_name(self): """ Get current WAL file from PostgreSQL :return str: current WAL file in PostgreSQL """ current_xlog_info = self.current_xlog_info if current_xlog_info is not None: return current_xlog_info["file_name"] return None @property def xlog_segment_size(self): """ Retrieve the size of one WAL file. In PostgreSQL 11, users will be able to change the WAL size at runtime. Up to PostgreSQL 10, included, the WAL size can be changed at compile time :return: The wal size (In bytes) """ try: cur = self._cursor(cursor_factory=DictCursor) # We can't use the `get_setting` method here, because it # use `SHOW`, returning an human readable value such as "16MB", # while we prefer a raw value such as 16777216. cur.execute("SELECT setting FROM pg_settings WHERE name='wal_segment_size'") result = cur.fetchone() wal_segment_size = int(result[0]) # Prior to PostgreSQL 11, the wal segment size is returned in # blocks if self.server_version < 110000: cur.execute( "SELECT setting FROM pg_settings WHERE name='wal_block_size'" ) result = cur.fetchone() wal_block_size = int(result[0]) wal_segment_size *= wal_block_size return wal_segment_size except ValueError as e: _logger.error( "Error retrieving current xlog segment size: %s", force_str(e).strip(), ) return None @property def current_xlog_location(self): """ Get current WAL location from PostgreSQL :return str: current WAL location in PostgreSQL """ current_xlog_info = self.current_xlog_info if current_xlog_info is not None: return current_xlog_info["location"] return None @property def current_size(self): """ Returns the total size of the PostgreSQL server (requires superuser or pg_read_all_stats) """ if not self.has_backup_privileges: return None try: cur = self._cursor() cur.execute("SELECT sum(pg_tablespace_size(oid)) FROM pg_tablespace") return cur.fetchone()[0] except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error retrieving PostgreSQL total size: %s", force_str(e).strip() ) return None @property def archive_timeout(self): """ Retrieve the archive_timeout setting in PostgreSQL :return: The archive timeout (in seconds) """ try: cur = self._cursor(cursor_factory=DictCursor) # We can't use the `get_setting` method here, because it # uses `SHOW`, returning an human readable value such as "5min", # while we prefer a raw value such as 300. cur.execute("SELECT setting FROM pg_settings WHERE name='archive_timeout'") result = cur.fetchone() archive_timeout = int(result[0]) return archive_timeout except ValueError as e: _logger.error("Error retrieving archive_timeout: %s", force_str(e).strip()) return None @property def checkpoint_timeout(self): """ Retrieve the checkpoint_timeout setting in PostgreSQL :return: The checkpoint timeout (in seconds) """ try: cur = self._cursor(cursor_factory=DictCursor) # We can't use the `get_setting` method here, because it # uses `SHOW`, returning an human readable value such as "5min", # while we prefer a raw value such as 300. cur.execute( "SELECT setting FROM pg_settings WHERE name='checkpoint_timeout'" ) result = cur.fetchone() checkpoint_timeout = int(result[0]) return checkpoint_timeout except ValueError as e: _logger.error( "Error retrieving checkpoint_timeout: %s", force_str(e).strip() ) return None def get_archiver_stats(self): """ This method gathers statistics from pg_stat_archiver. Only for Postgres 9.4+ or greater. If not available, returns None. :return dict|None: a dictionary containing Postgres statistics from pg_stat_archiver or None """ try: cur = self._cursor(cursor_factory=DictCursor) # Select from pg_stat_archiver statistics view, # retrieving statistics about WAL archiver process activity, # also evaluating if the server is archiving without issues # and the archived WALs per second rate. # # We are using current_settings to check for archive_mode=always. # current_setting does normalise its output so we can just # check for 'always' settings using a direct string # comparison cur.execute( "SELECT *, " "current_setting('archive_mode') IN ('on', 'always') " "AND (last_failed_wal IS NULL " "OR last_failed_wal LIKE '%.history' " "AND substring(last_failed_wal from 1 for 8) " "<= substring(last_archived_wal from 1 for 8) " "OR last_failed_time <= last_archived_time) " "AS is_archiving, " "CAST (archived_count AS NUMERIC) " "/ EXTRACT (EPOCH FROM age(now(), stats_reset)) " "AS current_archived_wals_per_second " "FROM pg_stat_archiver" ) return cur.fetchone() except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error retrieving pg_stat_archive data: %s", force_str(e).strip() ) return None def fetch_remote_status(self): """ Get the status of the PostgreSQL server This method does not raise any exception in case of errors, but set the missing values to None in the resulting dictionary. :rtype: dict[str, None|str] """ # PostgreSQL settings to get from the server (requiring superuser) pg_superuser_settings = ["data_directory"] # PostgreSQL settings to get from the server pg_settings = [] pg_query_keys = [ "server_txt_version", "is_superuser", "is_in_recovery", "current_xlog", "replication_slot_support", "replication_slot", "synchronous_standby_names", "postgres_systemid", "version_supported", ] # Initialise the result dictionary setting all the values to None result = dict.fromkeys( pg_superuser_settings + pg_settings + pg_query_keys, None ) try: # Retrieve wal_level, hot_standby and max_wal_senders # only if version is >= 9.0 pg_settings.extend( [ "wal_level", "hot_standby", "max_wal_senders", "data_checksums", "max_replication_slots", "wal_compression", ] ) # Retrieve wal_keep_segments from version 9.0 onwards, until # version 13.0, where it was renamed to wal_keep_size if self.server_version < 130000: pg_settings.append("wal_keep_segments") else: pg_settings.append("wal_keep_size") # retrieves superuser settings if self.has_backup_privileges: for name in pg_superuser_settings: result[name] = self.get_setting(name) # retrieves standard settings for name in pg_settings: result[name] = self.get_setting(name) result["is_superuser"] = self.is_superuser result["has_backup_privileges"] = self.has_backup_privileges result["has_monitoring_privileges"] = self.has_monitoring_privileges result["is_in_recovery"] = self.is_in_recovery result["server_txt_version"] = self.server_txt_version result["version_supported"] = self.is_minimal_postgres_version() current_xlog_info = self.current_xlog_info if current_xlog_info: result["current_lsn"] = current_xlog_info["location"] result["current_xlog"] = current_xlog_info["file_name"] else: result["current_lsn"] = None result["current_xlog"] = None result["current_size"] = self.current_size result["archive_timeout"] = self.archive_timeout result["checkpoint_timeout"] = self.checkpoint_timeout result["xlog_segment_size"] = self.xlog_segment_size result.update(self.get_configuration_files()) # Retrieve the replication_slot status result["replication_slot_support"] = True if self.slot_name is not None: result["replication_slot"] = self.get_replication_slot(self.slot_name) # Retrieve the list of synchronous standby names result["synchronous_standby_names"] = self.get_synchronous_standby_names() result["postgres_systemid"] = self.get_systemid() except (PostgresConnectionError, psycopg2.Error) as e: _logger.warning( "Error retrieving PostgreSQL status: %s", force_str(e).strip() ) return result def get_systemid(self): """ Get a Postgres instance systemid """ try: cur = self._cursor() cur.execute("SELECT system_identifier::text FROM pg_control_system()") return cur.fetchone()[0] except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error retrieving PostgreSQL system Id: %s", force_str(e).strip() ) return None def get_setting(self, name): """ Get a Postgres setting with a given name :param name: a parameter name """ try: cur = self._cursor() cur.execute('SHOW "%s"' % name.replace('"', '""')) return cur.fetchone()[0] except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error retrieving PostgreSQL setting '%s': %s", name.replace('"', '""'), force_str(e).strip(), ) return None def get_tablespaces(self): """ Returns a list of tablespaces or None if not present """ try: cur = self._cursor() cur.execute( "SELECT spcname, oid, " "pg_tablespace_location(oid) AS spclocation " "FROM pg_tablespace " "WHERE pg_tablespace_location(oid) != ''" ) # Generate a list of tablespace objects return [Tablespace._make(item) for item in cur.fetchall()] except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error retrieving PostgreSQL tablespaces: %s", force_str(e).strip() ) return None def get_configuration_files(self): """ Get postgres configuration files or an empty dictionary in case of error :rtype: dict """ if self.configuration_files: return self.configuration_files try: self.configuration_files = {} cur = self._cursor() cur.execute( "SELECT name, setting FROM pg_settings " "WHERE name IN ('config_file', 'hba_file', 'ident_file')" ) for cname, cpath in cur.fetchall(): self.configuration_files[cname] = cpath # Retrieve additional configuration files cur.execute( "SELECT DISTINCT sourcefile AS included_file " "FROM pg_settings " "WHERE sourcefile IS NOT NULL " "AND sourcefile NOT IN " "(SELECT setting FROM pg_settings " "WHERE name = 'config_file') " "ORDER BY 1" ) # Extract the values from the containing single element tuples included_files = [included_file for included_file, in cur.fetchall()] if len(included_files) > 0: self.configuration_files["included_files"] = included_files except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error retrieving PostgreSQL configuration files location: %s", force_str(e).strip(), ) self.configuration_files = {} return self.configuration_files def create_restore_point(self, target_name): """ Create a restore point with the given target name The method executes the pg_create_restore_point() function through a PostgreSQL connection. Only for Postgres versions >= 9.1 when not in replication. If requirements are not met, the operation is skipped. :param str target_name: name of the restore point :returns: the restore point LSN :rtype: str|None """ # Not possible if on a standby # Called inside the pg_connect context to reuse the connection if self.is_in_recovery: return None try: cur = self._cursor() cur.execute("SELECT pg_create_restore_point(%s)", [target_name]) _logger.info("Restore point '%s' successfully created", target_name) return cur.fetchone()[0] except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error issuing pg_create_restore_point() command: %s", force_str(e).strip(), ) return None def start_exclusive_backup(self, label): """ Calls pg_backup_start() on the PostgreSQL server This method returns a dictionary containing the following data: * location * file_name * file_offset * timestamp :param str label: descriptive string to identify the backup :rtype: psycopg2.extras.DictRow """ try: conn = self.connect() # Rollback to release the transaction, as the pg_backup_start # invocation can last up to PostgreSQL's checkpoint_timeout conn.rollback() # Start an exclusive backup cur = conn.cursor(cursor_factory=DictCursor) if self.server_version >= 150000: raise PostgresObsoleteFeature("15") else: cur.execute( "SELECT location, " "({pg_walfile_name_offset}(location)).*, " "now() AS timestamp " "FROM {pg_backup_start}(%s,%s) AS location".format(**self.name_map), (label, self.immediate_checkpoint), ) start_row = cur.fetchone() # Rollback to release the transaction, as the connection # is to be retained until the end of backup conn.rollback() return start_row except (PostgresConnectionError, psycopg2.Error) as e: msg = ( "{pg_backup_start}(): %s".format(**self.name_map) % force_str(e).strip() ) _logger.debug(msg) raise PostgresException(msg) def start_concurrent_backup(self, label): """ Calls pg_backup_start on the PostgreSQL server using the API introduced with version 9.6 This method returns a dictionary containing the following data: * location * timeline * timestamp :param str label: descriptive string to identify the backup :rtype: psycopg2.extras.DictRow """ try: conn = self.connect() # Rollback to release the transaction, as the pg_backup_start # invocation can last up to PostgreSQL's checkpoint_timeout conn.rollback() # Start the backup using the api introduced in postgres 9.6 cur = conn.cursor(cursor_factory=DictCursor) if self.server_version >= 150000: pg_backup_args = "%s, %s" else: # PostgreSQLs below 15 have a boolean parameter to specify # not to use exclusive backup pg_backup_args = "%s, %s, FALSE" # pg_backup_start and pg_backup_stop need to be run in the # same session when taking concurrent backups, so we disable # idle_session_timeout to avoid failures when stopping the # backup if copy takes more than idle_session_timeout to complete if self.server_version >= 140000: cur.execute("SET idle_session_timeout TO 0") cur.execute( "SELECT location, " "(SELECT timeline_id " "FROM pg_control_checkpoint()) AS timeline, " "now() AS timestamp " "FROM {pg_backup_start}({pg_backup_args}) AS location".format( pg_backup_args=pg_backup_args, **self.name_map ), (label, self.immediate_checkpoint), ) start_row = cur.fetchone() # Rollback to release the transaction, as the connection # is to be retained until the end of backup conn.rollback() return start_row except (PostgresConnectionError, psycopg2.Error) as e: msg = "{pg_backup_start} command: %s".format(**self.name_map) % ( force_str(e).strip(), ) _logger.debug(msg) raise PostgresException(msg) def stop_exclusive_backup(self): """ Calls pg_backup_stop() on the PostgreSQL server This method returns a dictionary containing the following data: * location * file_name * file_offset * timestamp :rtype: psycopg2.extras.DictRow """ try: conn = self.connect() # Rollback to release the transaction, as the pg_backup_stop # invocation could will wait until the current WAL file is shipped conn.rollback() # Stop the backup cur = conn.cursor(cursor_factory=DictCursor) if self.server_version >= 150000: raise PostgresObsoleteFeature("15") cur.execute( "SELECT location, " "({pg_walfile_name_offset}(location)).*, " "now() AS timestamp " "FROM {pg_backup_stop}() AS location".format(**self.name_map) ) return cur.fetchone() except (PostgresConnectionError, psycopg2.Error) as e: msg = "Error issuing {pg_backup_stop} command: %s" % force_str(e).strip() _logger.debug(msg) raise PostgresException( "Cannot terminate exclusive backup. " "You might have to manually execute {pg_backup_stop} " "on your PostgreSQL server".format(**self.name_map) ) def stop_concurrent_backup(self): """ Calls pg_backup_stop on the PostgreSQL server using the API introduced with version 9.6 This method returns a dictionary containing the following data: * location * timeline * backup_label * timestamp :rtype: psycopg2.extras.DictRow """ try: conn = self.connect() # Rollback to release the transaction, as the pg_backup_stop # invocation could will wait until the current WAL file is shipped conn.rollback() if self.server_version >= 150000: # The pg_backup_stop function accepts one argument, a boolean # wait_for_archive indicating whether PostgreSQL should wait # until all required WALs are archived. This is not set so that # we get the default behaviour which is to wait for the wals. pg_backup_args = "" else: # For PostgreSQLs below 15 the function accepts two arguments - # a boolean to indicate exclusive or concurrent backup and the # wait_for_archive boolean. We set exclusive to FALSE and leave # wait_for_archive unset as with PG >= 15. pg_backup_args = "FALSE" # Stop the backup using the api introduced with version 9.6 cur = conn.cursor(cursor_factory=DictCursor) # As we are about to run pg_backup_stop we can now reset # idle_session_timeout to whatever the user had # originally configured in PostgreSQL if self.server_version >= 140000: cur.execute("RESET idle_session_timeout") cur.execute( "SELECT end_row.lsn AS location, " "(SELECT CASE WHEN pg_is_in_recovery() " "THEN min_recovery_end_timeline ELSE timeline_id END " "FROM pg_control_checkpoint(), pg_control_recovery()" ") AS timeline, " "end_row.labelfile AS backup_label, " "now() AS timestamp FROM {pg_backup_stop}({pg_backup_args}) AS end_row".format( pg_backup_args=pg_backup_args, **self.name_map ) ) return cur.fetchone() except (PostgresConnectionError, psycopg2.Error) as e: msg = ( "Error issuing {pg_backup_stop} command: %s".format(**self.name_map) % force_str(e).strip() ) _logger.debug(msg) raise PostgresException(msg) def switch_wal(self): """ Execute a pg_switch_wal() To be SURE of the switch of a xlog, we collect the xlogfile name before and after the switch. The method returns the just closed xlog file name if the current xlog file has changed, it returns an empty string otherwise. The method returns None if something went wrong during the execution of the pg_switch_wal command. :rtype: str|None """ try: conn = self.connect() if not self.has_backup_privileges: raise BackupFunctionsAccessRequired( "Postgres user '%s' is missing required privileges " '(see "Preliminary steps" in the Barman manual)' % self.conn_parameters.get("user") ) # If this server is in recovery there is nothing to do if self.is_in_recovery: raise PostgresIsInRecovery() cur = conn.cursor() # Collect the xlog file name before the switch cur.execute( "SELECT {pg_walfile_name}(" "{pg_current_wal_insert_lsn}())".format(**self.name_map) ) pre_switch = cur.fetchone()[0] # Switch cur.execute( "SELECT {pg_walfile_name}({pg_switch_wal}())".format(**self.name_map) ) # Collect the xlog file name after the switch cur.execute( "SELECT {pg_walfile_name}(" "{pg_current_wal_insert_lsn}())".format(**self.name_map) ) post_switch = cur.fetchone()[0] if pre_switch < post_switch: return pre_switch else: return "" except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error issuing {pg_switch_wal}() command: %s".format(**self.name_map), force_str(e).strip(), ) return None def checkpoint(self): """ Execute a checkpoint """ try: conn = self.connect() # Requires superuser privilege if not self.has_checkpoint_privileges: raise PostgresCheckpointPrivilegesRequired() cur = conn.cursor() cur.execute("CHECKPOINT") except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug("Error issuing CHECKPOINT: %s", force_str(e).strip()) def get_replication_stats(self, client_type=STANDBY): """ Returns streaming replication information """ try: cur = self._cursor(cursor_factory=NamedTupleCursor) if not self.has_monitoring_privileges: raise BackupFunctionsAccessRequired( "Postgres user '%s' is missing required privileges " '(see "Preliminary steps" in the Barman manual)' % self.conn_parameters.get("user") ) # pg_stat_replication is a system view that contains one # row per WAL sender process with information about the # replication status of a standby server. It has been # introduced in PostgreSQL 9.1. Current fields are: # # - pid (procpid in 9.1) # - usesysid # - usename # - application_name # - client_addr # - client_hostname # - client_port # - backend_start # - backend_xmin (9.4+) # - state # - sent_lsn (sent_location before 10) # - write_lsn (write_location before 10) # - flush_lsn (flush_location before 10) # - replay_lsn (replay_location before 10) # - sync_priority # - sync_state # from_repslot = "" where_clauses = [] if self.server_version >= 100000: # Current implementation (10+) what = "r.*, rs.slot_name" # Look for replication slot name from_repslot = ( "LEFT JOIN pg_replication_slots rs ON (r.pid = rs.active_pid) " ) where_clauses += ["(rs.slot_type IS NULL OR rs.slot_type = 'physical')"] else: # PostgreSQL 9.5/9.6 what = ( "pid, " "usesysid, " "usename, " "application_name, " "client_addr, " "client_hostname, " "client_port, " "backend_start, " "backend_xmin, " "state, " "sent_location AS sent_lsn, " "write_location AS write_lsn, " "flush_location AS flush_lsn, " "replay_location AS replay_lsn, " "sync_priority, " "sync_state, " "rs.slot_name" ) # Look for replication slot name from_repslot = ( "LEFT JOIN pg_replication_slots rs ON (r.pid = rs.active_pid) " ) where_clauses += ["(rs.slot_type IS NULL OR rs.slot_type = 'physical')"] # Streaming client if client_type == self.STANDBY: # Standby server where_clauses += ["{replay_lsn} IS NOT NULL".format(**self.name_map)] elif client_type == self.WALSTREAMER: # WAL streamer where_clauses += ["{replay_lsn} IS NULL".format(**self.name_map)] if where_clauses: where = "WHERE %s " % " AND ".join(where_clauses) else: where = "" # Execute the query cur.execute( "SELECT %s, " "pg_is_in_recovery() AS is_in_recovery, " "CASE WHEN pg_is_in_recovery() " " THEN {pg_last_wal_receive_lsn}() " " ELSE {pg_current_wal_lsn}() " "END AS current_lsn " "FROM pg_stat_replication r " "%s" "%s" "ORDER BY sync_state DESC, sync_priority".format(**self.name_map) % (what, from_repslot, where) ) # Generate a list of standby objects return cur.fetchall() except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error retrieving status of standby servers: %s", force_str(e).strip() ) return None def get_replication_slot(self, slot_name): """ Retrieve from the PostgreSQL server a physical replication slot with a specific slot_name. This method returns a dictionary containing the following data: * slot_name * active * restart_lsn :param str slot_name: the replication slot name :rtype: psycopg2.extras.DictRow """ if self.server_version < 90400: # Raise exception if replication slot are not supported # by PostgreSQL version raise PostgresUnsupportedFeature("9.4") else: cur = self._cursor(cursor_factory=NamedTupleCursor) try: cur.execute( "SELECT slot_name, " "active, " "restart_lsn " "FROM pg_replication_slots " "WHERE slot_type = 'physical' " "AND slot_name = '%s'" % slot_name ) # Retrieve the replication slot information return cur.fetchone() except (PostgresConnectionError, psycopg2.Error) as e: _logger.debug( "Error retrieving replication_slots: %s", force_str(e).strip() ) raise def get_synchronous_standby_names(self): """ Retrieve the list of named synchronous standby servers from PostgreSQL This method returns a list of names :return list: synchronous standby names """ if self.server_version < 90100: # Raise exception if synchronous replication is not supported raise PostgresUnsupportedFeature("9.1") else: synchronous_standby_names = self.get_setting("synchronous_standby_names") # Return empty list if not defined if synchronous_standby_names is None: return [] # Normalise the list of sync standby names # On PostgreSQL 9.6 it is possible to specify the number of # required synchronous standby using this format: # n (name1, name2, ... nameN). # We only need the name list, so we discard everything else. # The name list starts after the first parenthesis or at pos 0 names_start = synchronous_standby_names.find("(") + 1 names_end = synchronous_standby_names.rfind(")") if names_end < 0: names_end = len(synchronous_standby_names) names_list = synchronous_standby_names[names_start:names_end] # We can blindly strip double quotes because PostgreSQL enforces # the format of the synchronous_standby_names content return [x.strip().strip('"') for x in names_list.split(",")] def send_heartbeat_query(self): """ Sends a heartbeat query to the server with the already opened connection. :returns tuple[bool, Exception|None]: A tuple where the first value is a boolean indicating if the query executed successfully or not and the second is the exception raised by ``psycopg2`` in case it did not succeed. """ try: with self._conn.cursor() as cursor: cursor.execute(self.HEARTBEAT_QUERY) _logger.debug("Sent heartbeat query to maintain the current connection") return True, None except psycopg2.Error as ex: _logger.debug( "Failed to execute heartbeat query on the current connection: %s" % force_str(ex) ) return False, ex @property def name_map(self): """ Return a map with function and directory names according to the current PostgreSQL version. Each entry has the `current` name as key and the name for the specific version as value. :rtype: dict[str] """ # Avoid raising an error if the connection is not available try: server_version = self.server_version except PostgresConnectionError: _logger.debug( "Impossible to detect the PostgreSQL version, " "name_map will return names from latest version" ) server_version = None return function_name_map(server_version) class StandbyPostgreSQLConnection(PostgreSQLConnection): """ A specialised PostgreSQLConnection for standby servers. Works almost exactly like a regular PostgreSQLConnection except it requires a primary_conninfo option at creation time which is used to create a connection to the primary for the purposes of forcing a WAL switch during the stop backup process. This increases the likelihood that backups against standbys with `archive_mode = always` and low traffic on the primary are able to complete. """ def __init__( self, conninfo, primary_conninfo, immediate_checkpoint=False, slot_name=None, primary_checkpoint_timeout=0, application_name="barman", ): """ Standby PostgreSQL connection constructor. :param str conninfo: Connection information (aka DSN) for the standby. :param str primary_conninfo: Connection information (aka DSN) for the primary. :param bool immediate_checkpoint: Whether to do an immediate checkpoint when a backup is started. :param str|None slot_name: Replication slot name. :param str: The application_name to use for this connection. """ super(StandbyPostgreSQLConnection, self).__init__( conninfo, immediate_checkpoint=immediate_checkpoint, slot_name=slot_name, application_name=application_name, ) # The standby connection has its own connection object used to talk to the # primary when switching WALs. self.primary_conninfo = primary_conninfo # The standby needs a connection to the primary so that it can # perform WAL switches itself when calling pg_backup_stop. self.primary = PostgreSQLConnection(self.primary_conninfo) self.primary_checkpoint_timeout = primary_checkpoint_timeout def close(self): """Close the connection to PostgreSQL.""" super(StandbyPostgreSQLConnection, self).close() return self.primary.close() def switch_wal(self): """Perform a WAL switch on the primary PostgreSQL instance.""" # Instead of calling the superclass switch_wal, which would invoke # pg_switch_wal on the standby, we use our connection to the primary to # switch the WAL directly. return self.primary.switch_wal() def switch_wal_in_background(self, done_q, times=10, wait=10): """ Perform a pg_switch_wal in a background process. This function runs in a child process and is intended to keep calling pg_switch_wal() until it is told to stop or until `times` is exceeded. The parent process will use `done_q` to tell this process to stop. :param multiprocessing.Queue done_q: A Queue used by the parent process to communicate with the WAL switching process. A value of `True` on this queue indicates that this function should stop. :param int times: The maximum number of times a WAL switch should be performed. :param int wait: The number of seconds to wait between WAL switches. """ # Use a new connection to prevent undefined behaviour self.primary = PostgreSQLConnection(self.primary_conninfo) # The stop backup call on the standby may have already completed by this # point so check whether we have been told to stop. try: if done_q.get(timeout=1): return except Empty: pass try: # Start calling pg_switch_wal on the primary until we either read something # from the done queue or we exceed the number of WAL switches we are allowed. for _ in range(0, times): self.switch_wal() # See if we have been told to stop. We use the wait value as our timeout # so that we can exit immediately if we receive a stop message or proceed # to another WAL switch if the wait time is exceeded. try: if done_q.get(timeout=wait): return except Empty: # An empty queue just means we haven't yet been told to stop pass if self.primary_checkpoint_timeout: _logger.warning( "Barman attempted to switch WALs %s times on the primary " "server, but the backup has not yet completed. " "A checkpoint will be forced on the primary server " "in %s seconds to ensure the backup can complete." % (times, self.primary_checkpoint_timeout) ) sleep_time = datetime.datetime.now() + datetime.timedelta( seconds=self.primary_checkpoint_timeout ) while True: try: # Always check if the queue is empty, so we know to stop # before the checkpoint execution if done_q.get(timeout=wait): return except Empty: # If the queue is empty, we can proceed to the checkpoint # if enough time has passed if sleep_time < datetime.datetime.now(): self.primary.checkpoint() self.primary.switch_wal() break # break out of the loop after the checkpoint and wal switch # execution. The connection will be closed in the finally statement finally: # Close the connection since only this subprocess will ever use it self.primary.close() def _start_wal_switch(self): """Start switching WALs in a child process.""" # The child process will stop if it reads a value of `True` from this queue. self.done_q = Queue() # Create and start the child process before we stop the backup. self.switch_wal_proc = Process( target=self.switch_wal_in_background, args=(self.done_q,) ) self.switch_wal_proc.start() def _stop_wal_switch(self): """Stop the WAL switching process.""" # Stop the child process by adding a `True` to its queue self.done_q.put(True) # Make sure the child process closes before we return. self.switch_wal_proc.join() def _stop_backup(self, stop_backup_fun): """ Stop a backup while also calling pg_switch_wal(). Starts a child process to call pg_switch_wal() on the primary before attempting to stop the backup on the standby. The WAL switch is intended to allow the pg_backup_stop call to complete when running against a standby with `archive_mode = always`. Once the call to `stop_concurrent_backup` completes the child process is stopped as no further WAL switches are required. :param function stop_backup_fun: The function which should be called to stop the backup. This will be a reference to one of the superclass methods stop_concurrent_backup or stop_exclusive_backup. :rtype: psycopg2.extras.DictRow """ self._start_wal_switch() stop_info = stop_backup_fun() self._stop_wal_switch() return stop_info def stop_concurrent_backup(self): """ Stop a concurrent backup on a standby PostgreSQL instance. :rtype: psycopg2.extras.DictRow """ return self._stop_backup( super(StandbyPostgreSQLConnection, self).stop_concurrent_backup ) def stop_exclusive_backup(self): """ Stop an exclusive backup on a standby PostgreSQL instance. :rtype: psycopg2.extras.DictRow """ return self._stop_backup( super(StandbyPostgreSQLConnection, self).stop_exclusive_backup ) class PostgresKeepAlive: """ Context manager to maintain a Postgres connection alive. A child thread is spawned to execute heartbeat queries in the background at a specified interval during its living context. It does not open or close any connections on its own. Instead, it waits for the specified connection to be opened on the main thread before start sending any query. :cvar THREAD_NAME: The name identifying the keep-alive thread. """ THREAD_NAME = "barman_keepalive_thread" def __init__(self, postgres, interval, raise_exception=False): """ Constructor. :param barman.postgres.PostgreSQLConnection postgres: The Postgres connection to keep alive. :param int interval: An interval in seconds at which a heartbeat query will be sent to keep the connection alive. A value <= ``0`` won't start the keepalive. :param bool raise_exception: A boolean indicating if an exception should be raised in case the connection is lost. If ``True``, a ``PostgresConnectionLost`` exception will be raised as soon as it's noticed a connection failure. If ``False``, it will keep executing normally until the context exits. """ self.postgres = postgres self.interval = interval self.raise_exception = raise_exception self._stop_thread = threading.Event() self._thread = threading.Thread( target=self._run_keep_alive, name=self.THREAD_NAME, ) def _prepare_signal_handler(self): """ Set up a signal handler to raise an exception on the main thread when the keep-alive thread wishes to interrupt it. This method listens for a ``SIGUSR1`` signal and, when received, raises a ``PostgresConnectionLost`` exception. .. note:: This code is, and only works if, executed while on the main thread. We are not able to set a signal listener on a child thread, therefore this method must be executed before the keep-alive thread starts. """ def raise_exception(signum, frame): raise PostgresConnectionLost("Connection to Postgres server was lost.") signal.signal(signal.SIGUSR1, raise_exception) def _raise_exception_on_main(self): """ Trigger an exception on the main thread at whatever frame is being executed at the moment. This is done by sending a ``SIGUSR1`` signal to the process, which will be caught by the signal handler set previously in this class. .. note:: This is an alternative way of interrupting the main thread's work, since there is no direct way of killing or raising exceptions on the main thread from a child thread in Python. A handler for this signal has been set beforehand by the ``_prepare_signal_handler`` method in this class. """ os.kill(os.getpid(), signal.SIGUSR1) def _run_keep_alive(self): """Runs the keepalive until a stop-thread event is set""" while not self._stop_thread.is_set(): if not self.postgres.has_connection: # Wait for the connection to be opened on the main thread time.sleep(1) continue success, ex = self.postgres.send_heartbeat_query() if not success and self.raise_exception: # If one of the below exeptions was raised by psycopg2, it most likely # means that the connection (and consequently, the session) was lost. In # such cases, we can stop the keep-alive exection and raise the exception if isinstance(ex, (psycopg2.InterfaceError, psycopg2.OperationalError)): self._stop_thread.set() self._raise_exception_on_main() self._stop_thread.wait(self.interval) def __enter__(self): """Enters context. Starts the thread""" if self.interval > 0: if self.raise_exception: self._prepare_signal_handler() self._thread.start() def __exit__(self, exc_type, exc_val, exc_tb): """Exits context. Makes sure the thread is terminated""" if self.interval > 0: self._stop_thread.set() self._thread.join() barman-3.13.3/barman/xlog.py0000644000175100001660000004153715002177425014022 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module contains functions to retrieve information about xlog files """ import collections import os import re from functools import partial from tempfile import NamedTemporaryFile from barman.exceptions import ( BadHistoryFileContents, BadXlogPrefix, BadXlogSegmentName, CommandException, WalArchiveContentError, ) # xlog file segment name parser (regular expression) _xlog_re = re.compile( r""" ^ ([\dA-Fa-f]{8}) # everything has a timeline (?: ([\dA-Fa-f]{8})([\dA-Fa-f]{8}) # segment name, if a wal file (?: # and optional \.[\dA-Fa-f]{8}\.backup # offset, if a backup label | \.partial # partial, if a partial file )? | \.history # or only .history, if a history file ) $ """, re.VERBOSE, ) # xlog prefix parser (regular expression) _xlog_prefix_re = re.compile(r"^([\dA-Fa-f]{8})([\dA-Fa-f]{8})$") # xlog location parser for concurrent backup (regular expression) _location_re = re.compile(r"^([\dA-F]+)/([\dA-F]+)$") # Taken from xlog_internal.h from PostgreSQL sources #: XLOG_SEG_SIZE is the size of a single WAL file. This must be a power of 2 #: and larger than XLOG_BLCKSZ (preferably, a great deal larger than #: XLOG_BLCKSZ). DEFAULT_XLOG_SEG_SIZE = 1 << 24 #: This namedtuple is a container for the information #: contained inside history files HistoryFileData = collections.namedtuple( "HistoryFileData", "tli parent_tli switchpoint reason" ) def is_any_xlog_file(path): """ Return True if the xlog is either a WAL segment, a .backup file or a .history file, False otherwise. It supports either a full file path or a simple file name. :param str path: the file name to test :rtype: bool """ match = _xlog_re.match(os.path.basename(path)) if match: return True return False def is_history_file(path): """ Return True if the xlog is a .history file, False otherwise It supports either a full file path or a simple file name. :param str path: the file name to test :rtype: bool """ match = _xlog_re.search(os.path.basename(path)) if match and match.group(0).endswith(".history"): return True return False def is_backup_file(path): """ Return True if the xlog is a .backup file, False otherwise It supports either a full file path or a simple file name. :param str path: the file name to test :rtype: bool """ match = _xlog_re.search(os.path.basename(path)) if match and match.group(0).endswith(".backup"): return True return False def is_partial_file(path): """ Return True if the xlog is a .partial file, False otherwise It supports either a full file path or a simple file name. :param str path: the file name to test :rtype: bool """ match = _xlog_re.search(os.path.basename(path)) if match and match.group(0).endswith(".partial"): return True return False def is_wal_file(path): """ Return True if the xlog is a regular xlog file, False otherwise It supports either a full file path or a simple file name. :param str path: the file name to test :rtype: bool """ match = _xlog_re.search(os.path.basename(path)) if not match: return False ends_with_backup = match.group(0).endswith(".backup") ends_with_history = match.group(0).endswith(".history") ends_with_partial = match.group(0).endswith(".partial") if ends_with_backup: return False if ends_with_history: return False if ends_with_partial: return False return True def decode_segment_name(path): """ Retrieve the timeline, log ID and segment ID from the name of a xlog segment It can handle either a full file path or a simple file name. :param str path: the file name to decode :rtype: list[int] """ name = os.path.basename(path) match = _xlog_re.match(name) if not match: raise BadXlogSegmentName(name) return [int(x, 16) if x else None for x in match.groups()] def encode_segment_name(tli, log, seg): """ Build the xlog segment name based on timeline, log ID and segment ID :param int tli: timeline number :param int log: log number :param int seg: segment number :return str: segment file name """ return "%08X%08X%08X" % (tli, log, seg) def encode_history_file_name(tli): """ Build the history file name based on timeline :return str: history file name """ return "%08X.history" % (tli,) def xlog_segments_per_file(xlog_segment_size): """ Given that WAL files are named using the following pattern: this is the number of XLOG segments in an XLOG file. By XLOG file we don't mean an actual file on the filesystem, but the definition used in the PostgreSQL sources: meaning a set of files containing the same file number. :param int xlog_segment_size: The XLOG segment size in bytes :return int: The number of segments in an XLOG file """ return 0xFFFFFFFF // xlog_segment_size def xlog_segment_mask(xlog_segment_size): """ Given that WAL files are named using the following pattern: this is the bitmask of segment part of an XLOG file. See the documentation of `xlog_segments_per_file` for a commentary on the definition of `XLOG` file. :param int xlog_segment_size: The XLOG segment size in bytes :return int: The size of an XLOG file """ return xlog_segment_size * xlog_segments_per_file(xlog_segment_size) def generate_segment_names(begin, end=None, version=None, xlog_segment_size=None): """ Generate a sequence of XLOG segments starting from ``begin`` If an ``end`` segment is provided the sequence will terminate after returning it, otherwise the sequence will never terminate. If the XLOG segment size is known, this generator is precise, switching to the next file when required. It the XLOG segment size is unknown, this generator will generate all the possible XLOG file names. The size of an XLOG segment can be every power of 2 between the XLOG block size (8Kib) and the size of a log segment (4Gib) :param str begin: begin segment name :param str|None end: optional end segment name :param int|None version: optional postgres version as an integer (e.g. 90301 for 9.3.1) :param int xlog_segment_size: the size of a XLOG segment :rtype: collections.Iterable[str] :raise: BadXlogSegmentName """ begin_tli, begin_log, begin_seg = decode_segment_name(begin) end_tli, end_log, end_seg = None, None, None if end: end_tli, end_log, end_seg = decode_segment_name(end) # this method doesn't support timeline changes assert begin_tli == end_tli, ( "Begin segment (%s) and end segment (%s) " "must have the same timeline part" % (begin, end) ) # If version is less than 9.3 the last segment must be skipped skip_last_segment = version is not None and version < 90300 # This is the number of XLOG segments in an XLOG file. By XLOG file # we don't mean an actual file on the filesystem, but the definition # used in the PostgreSQL sources: a set of files containing the # same file number. if xlog_segment_size: # The generator is operating is precise and correct mode: # knowing exactly when a switch to the next file is required xlog_seg_per_file = xlog_segments_per_file(xlog_segment_size) else: # The generator is operating only in precise mode: generating every # possible XLOG file name. xlog_seg_per_file = 0x7FFFF # Start from the first xlog and generate the segments sequentially # If ``end`` has been provided, the while condition ensure the termination # otherwise this generator will never stop cur_log, cur_seg = begin_log, begin_seg while ( end is None or cur_log < end_log or (cur_log == end_log and cur_seg <= end_seg) ): yield encode_segment_name(begin_tli, cur_log, cur_seg) cur_seg += 1 if cur_seg > xlog_seg_per_file or ( skip_last_segment and cur_seg == xlog_seg_per_file ): cur_seg = 0 cur_log += 1 def hash_dir(path): """ Get the directory where the xlog segment will be stored It can handle either a full file path or a simple file name. :param str|unicode path: xlog file name :return str: directory name """ tli, log, _ = decode_segment_name(path) # tli is always not None if log is not None: return "%08X%08X" % (tli, log) else: return "" def decode_hash_dir(hash_dir): """ Get the timeline and log from a hash dir prefix. :param str hash_dir: A string representing the prefix used when determining the folder or object key prefix under which Barman will store a given WAL segment. This prefix is composed of the timeline and the higher 32-bit number of the WAL segment. :rtype: List[int] :return: A list of two elements where the first item is the timeline and the second is the higher 32-bit number of the WAL segment. """ match = _xlog_prefix_re.match(hash_dir) if not match: raise BadXlogPrefix(hash_dir) return [int(x, 16) if x else None for x in match.groups()] def parse_lsn(lsn_string): """ Transform a string XLOG location, formatted as %X/%X, in the corresponding numeric representation :param str lsn_string: the string XLOG location, i.e. '2/82000168' :rtype: int """ lsn_list = lsn_string.split("/") if len(lsn_list) != 2: raise ValueError("Invalid LSN: %s", lsn_string) return (int(lsn_list[0], 16) << 32) + int(lsn_list[1], 16) def diff_lsn(lsn_string1, lsn_string2): """ Calculate the difference in bytes between two string XLOG location, formatted as %X/%X Tis function is a Python implementation of the ``pg_xlog_location_diff(str, str)`` PostgreSQL function. :param str lsn_string1: the string XLOG location, i.e. '2/82000168' :param str lsn_string2: the string XLOG location, i.e. '2/82000168' :rtype: int """ # If one the input is None returns None if lsn_string1 is None or lsn_string2 is None: return None return parse_lsn(lsn_string1) - parse_lsn(lsn_string2) def format_lsn(lsn): """ Transform a numeric XLOG location, in the corresponding %X/%X string representation :param int lsn: numeric XLOG location :rtype: str """ return "%X/%X" % (lsn >> 32, lsn & 0xFFFFFFFF) def location_to_xlogfile_name_offset(location, timeline, xlog_segment_size): """ Convert transaction log location string to file_name and file_offset This is a reimplementation of pg_xlogfile_name_offset PostgreSQL function This method returns a dictionary containing the following data: * file_name * file_offset :param str location: XLOG location :param int timeline: timeline :param int xlog_segment_size: the size of a XLOG segment :rtype: dict """ lsn = parse_lsn(location) log = lsn >> 32 seg = (lsn & xlog_segment_mask(xlog_segment_size)) // xlog_segment_size offset = lsn & (xlog_segment_size - 1) return { "file_name": encode_segment_name(timeline, log, seg), "file_offset": offset, } def location_from_xlogfile_name_offset(file_name, file_offset, xlog_segment_size): """ Convert file_name and file_offset to a transaction log location. This is the inverted function of PostgreSQL's pg_xlogfile_name_offset function. :param str file_name: a WAL file name :param int file_offset: a numeric offset :param int xlog_segment_size: the size of a XLOG segment :rtype: str """ decoded_segment = decode_segment_name(file_name) location = decoded_segment[1] << 32 location += decoded_segment[2] * xlog_segment_size location += file_offset return format_lsn(location) def decode_history_file(wal_info, comp_manager): """ Read an history file and parse its contents. Each line in the file represents a timeline switch, each field is separated by tab, empty lines are ignored and lines starting with '#' are comments. Each line is composed by three fields: parentTLI, switchpoint and reason. "parentTLI" is the ID of the parent timeline. "switchpoint" is the WAL position where the switch happened "reason" is an human-readable explanation of why the timeline was changed The method requires a CompressionManager object to handle the eventual compression of the history file. :param barman.infofile.WalFileInfo wal_info: history file obj :param comp_manager: compression manager used in case of history file compression :return List[HistoryFileData]: information from the history file """ path = wal_info.orig_filename # Decompress the file if needed if wal_info.compression: # Use a NamedTemporaryFile to avoid explicit cleanup uncompressed_file = NamedTemporaryFile( dir=os.path.dirname(path), prefix=".%s." % wal_info.name, suffix=".uncompressed", ) path = uncompressed_file.name comp_manager.get_compressor(wal_info.compression).decompress( wal_info.orig_filename, path ) # Extract the timeline from history file name tli, _, _ = decode_segment_name(wal_info.name) lines = [] with open(path) as fp: for line in fp: line = line.strip() # Skip comments and empty lines if line.startswith("#"): continue # Skip comments and empty lines if len(line) == 0: continue # Use tab as separator contents = line.split("\t") if len(contents) != 3: # Invalid content of the line raise BadHistoryFileContents(path) history = HistoryFileData( tli=tli, parent_tli=int(contents[0]), switchpoint=parse_lsn(contents[1]), reason=contents[2], ) lines.append(history) # Empty history file or containing invalid content if len(lines) == 0: raise BadHistoryFileContents(path) else: return lines def _validate_timeline(timeline): """Check that timeline is a valid timeline value.""" try: # Explicitly check the type because python 2 will allow < to be used # between strings and ints if type(timeline) is not int or timeline < 1: raise ValueError() return True except Exception: raise CommandException( "Cannot check WAL archive with malformed timeline %s" % timeline ) def _wal_archive_filter_fun(timeline, wal): try: if not is_any_xlog_file(wal): raise ValueError() except Exception: raise WalArchiveContentError("Unexpected file %s found in WAL archive" % wal) wal_timeline, _, _ = decode_segment_name(wal) return timeline <= wal_timeline def check_archive_usable(existing_wals, timeline=None): """ Carry out pre-flight checks on the existing content of a WAL archive to determine if it is safe to archive WALs from the supplied timeline. """ if timeline is None: if len(existing_wals) > 0: raise WalArchiveContentError("Expected empty archive") else: _validate_timeline(timeline) filter_fun = partial(_wal_archive_filter_fun, timeline) unexpected_wals = [wal for wal in existing_wals if filter_fun(wal)] num_unexpected_wals = len(unexpected_wals) if num_unexpected_wals > 0: raise WalArchiveContentError( "Found %s file%s in WAL archive equal to or newer than " "timeline %s" % ( num_unexpected_wals, num_unexpected_wals > 1 and "s" or "", timeline, ) ) barman-3.13.3/barman/postgres_plumbing.py0000644000175100001660000001016715002177425016607 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ PostgreSQL Plumbing module This module contain low-level PostgreSQL related information, such as the on-disk structure and the name of the core functions in different PostgreSQL versions. """ PGDATA_EXCLUDE_LIST = [ # Exclude log files (pg_log was renamed to log in Postgres v10) "/pg_log/*", "/log/*", # Exclude WAL files (pg_xlog was renamed to pg_wal in Postgres v10) "/pg_xlog/*", "/pg_wal/*", # We handle this on a different step of the copy "/global/pg_control", ] EXCLUDE_LIST = [ # Files: see excludeFiles const in PostgreSQL source "pgsql_tmp*", "postgresql.auto.conf.tmp", "current_logfiles.tmp", "pg_internal.init", "postmaster.pid", "postmaster.opts", "recovery.conf", "standby.signal", # Directories: see excludeDirContents const in PostgreSQL source "pg_dynshmem/*", "pg_notify/*", "pg_replslot/*", "pg_serial/*", "pg_stat_tmp/*", "pg_snapshots/*", "pg_subtrans/*", ] def function_name_map(server_version): """ Return a map with function and directory names according to the current PostgreSQL version. Each entry has the `current` name as key and the name for the specific version as value. :param number|None server_version: Version of PostgreSQL as returned by psycopg2 (i.e. 90301 represent PostgreSQL 9.3.1). If the version is None, default to the latest PostgreSQL version :rtype: dict[str] """ # Start by defining the current names in name_map name_map = { "pg_backup_start": "pg_backup_start", "pg_backup_stop": "pg_backup_stop", "pg_switch_wal": "pg_switch_wal", "pg_walfile_name": "pg_walfile_name", "pg_wal": "pg_wal", "pg_walfile_name_offset": "pg_walfile_name_offset", "pg_last_wal_replay_lsn": "pg_last_wal_replay_lsn", "pg_current_wal_lsn": "pg_current_wal_lsn", "pg_current_wal_insert_lsn": "pg_current_wal_insert_lsn", "pg_last_wal_receive_lsn": "pg_last_wal_receive_lsn", "sent_lsn": "sent_lsn", "write_lsn": "write_lsn", "flush_lsn": "flush_lsn", "replay_lsn": "replay_lsn", } if server_version and server_version < 150000: # For versions below 15, pg_backup_start and pg_backup_stop are named # pg_start_backup and pg_stop_backup respectively name_map.update( { "pg_backup_start": "pg_start_backup", "pg_backup_stop": "pg_stop_backup", } ) if server_version and server_version < 100000: # For versions below 10, xlog is used in place of wal and location is # used in place of lsn name_map.update( { "pg_switch_wal": "pg_switch_xlog", "pg_walfile_name": "pg_xlogfile_name", "pg_wal": "pg_xlog", "pg_walfile_name_offset": "pg_xlogfile_name_offset", "pg_last_wal_replay_lsn": "pg_last_xlog_replay_location", "pg_current_wal_lsn": "pg_current_xlog_location", "pg_current_wal_insert_lsn": "pg_current_xlog_insert_location", "pg_last_wal_receive_lsn": "pg_last_xlog_receive_location", "sent_lsn": "sent_location", "write_lsn": "write_location", "flush_lsn": "flush_location", "replay_lsn": "replay_location", } ) return name_map barman-3.13.3/barman/cloud.py0000644000175100001660000030472415002177425014157 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2018-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import collections import copy import datetime import errno import json import logging import multiprocessing import operator import os import shutil import signal import tarfile import time from abc import ABCMeta, abstractmethod, abstractproperty from functools import partial from io import BytesIO, RawIOBase from tempfile import NamedTemporaryFile from barman import xlog from barman.annotations import KeepManagerMixinCloud from barman.backup_executor import ConcurrentBackupStrategy, SnapshotBackupExecutor from barman.clients import cloud_compression from barman.clients.cloud_cli import get_missing_attrs from barman.exceptions import ( BackupException, BackupPreconditionException, BarmanException, ConfigurationException, ) from barman.fs import UnixLocalCommand, path_allowed from barman.infofile import BackupInfo, WalFileInfo from barman.postgres_plumbing import EXCLUDE_LIST, PGDATA_EXCLUDE_LIST from barman.utils import ( BarmanEncoder, force_str, get_backup_info_from_name, human_readable_timedelta, is_backup_id, pretty_size, range_fun, total_seconds, with_metaclass, ) try: # Python 3.x from queue import Empty as EmptyQueue except ImportError: # Python 2.x from Queue import Empty as EmptyQueue BUFSIZE = 16 * 1024 LOGGING_FORMAT = "%(asctime)s [%(process)s] %(levelname)s: %(message)s" # Allowed compression algorithms ALLOWED_COMPRESSIONS = { ".gz": "gzip", ".bz2": "bzip2", ".xz": "xz", ".snappy": "snappy", ".zst": "zstd", ".lz4": "lz4", } DEFAULT_DELIMITER = "/" def configure_logging(config): """ Get a nicer output from the Python logging package """ verbosity = config.verbose - config.quiet log_level = max(logging.WARNING - verbosity * 10, logging.DEBUG) logging.basicConfig(format=LOGGING_FORMAT, level=log_level) def copyfileobj_pad_truncate(src, dst, length=None): """ Copy length bytes from fileobj src to fileobj dst. If length is None, copy the entire content. This method is used by the TarFileIgnoringTruncate.addfile(). """ if length == 0: return if length is None: shutil.copyfileobj(src, dst, BUFSIZE) return blocks, remainder = divmod(length, BUFSIZE) for _ in range(blocks): buf = src.read(BUFSIZE) dst.write(buf) if len(buf) < BUFSIZE: # End of file reached # The file must have been truncated, so pad with zeroes dst.write(tarfile.NUL * (BUFSIZE - len(buf))) if remainder != 0: buf = src.read(remainder) dst.write(buf) if len(buf) < remainder: # End of file reached # The file must have been truncated, so pad with zeroes dst.write(tarfile.NUL * (remainder - len(buf))) class CloudProviderError(BarmanException): """ This exception is raised when we get an error in the response from the cloud provider """ class CloudUploadingError(BarmanException): """ This exception is raised when there are upload errors """ class TarFileIgnoringTruncate(tarfile.TarFile): """ Custom TarFile class that ignore truncated or vanished files. """ format = tarfile.PAX_FORMAT # Use PAX format to better preserve metadata def addfile(self, tarinfo, fileobj=None): """ Add the provided fileobj to the tar ignoring truncated or vanished files. This method completely replaces TarFile.addfile() """ self._check("awx") tarinfo = copy.copy(tarinfo) buf = tarinfo.tobuf(self.format, self.encoding, self.errors) self.fileobj.write(buf) self.offset += len(buf) # If there's data to follow, append it. if fileobj is not None: copyfileobj_pad_truncate(fileobj, self.fileobj, tarinfo.size) blocks, remainder = divmod(tarinfo.size, tarfile.BLOCKSIZE) if remainder > 0: self.fileobj.write(tarfile.NUL * (tarfile.BLOCKSIZE - remainder)) blocks += 1 self.offset += blocks * tarfile.BLOCKSIZE self.members.append(tarinfo) class CloudTarUploader(object): # This is the method we use to create new buffers # We use named temporary files, so we can pass them by name to # other processes _buffer = partial( NamedTemporaryFile, delete=False, prefix="barman-upload-", suffix=".part" ) def __init__( self, cloud_interface, key, chunk_size, compression=None, max_bandwidth=None ): """ A tar archive that resides on cloud storage :param CloudInterface cloud_interface: cloud interface instance :param str key: path inside the bucket :param str compression: required compression :param int chunk_size: the upload chunk size :param int max_bandwidth: the maximum amount of data per second that should be uploaded by this tar uploader """ self.cloud_interface = cloud_interface self.key = key self.chunk_size = chunk_size self.max_bandwidth = max_bandwidth self.upload_metadata = None self.buffer = None self.counter = 0 self.compressor = None # Some supported compressions (e.g. snappy) require CloudTarUploader to apply # compression manually rather than relying on the tar file. self.compressor = cloud_compression.get_compressor(compression) # If the compression is supported by tar then it will be added to the filemode # passed to tar_mode. tar_mode = cloud_compression.get_streaming_tar_mode("w", compression) # The value of 65536 for the chunk size is based on comments in the python-snappy # library which suggest it should be good for almost every scenario. # See: https://github.com/andrix/python-snappy/blob/0.6.0/snappy/snappy.py#L282 self.tar = TarFileIgnoringTruncate.open( fileobj=self, mode=tar_mode, bufsize=64 << 10 ) self.size = 0 self.stats = None self.time_of_last_upload = None self.size_of_last_upload = None def write(self, buf): if self.buffer and self.buffer.tell() > self.chunk_size: self.flush() if not self.buffer: self.buffer = self._buffer() if self.compressor: # If we have a custom compressor we must use it here compressed_buf = self.compressor.add_chunk(buf) self.buffer.write(compressed_buf) self.size += len(compressed_buf) else: # If there is no custom compressor then we are either not using # compression or tar has already compressed it - in either case we # just write the data to the buffer self.buffer.write(buf) self.size += len(buf) def _throttle_upload(self, part_size): """ Throttles the upload according to the value of `self.max_bandwidth`. Waits until enough time has passed since the last upload that a new part can be uploaded without exceeding `self.max_bandwidth`. If sufficient time has already passed then this function will return without waiting. :param int part_size: Size in bytes of the part which is to be uplaoded. """ if (self.time_of_last_upload and self.size_of_last_upload) is not None: min_time_to_next_upload = self.size_of_last_upload / self.max_bandwidth seconds_since_last_upload = ( datetime.datetime.now() - self.time_of_last_upload ).total_seconds() if seconds_since_last_upload < min_time_to_next_upload: logging.info( f"Uploaded {self.size_of_last_upload} bytes " f"{seconds_since_last_upload} seconds ago which exceeds " f"limit of {self.max_bandwidth} bytes/s" ) time_to_wait = min_time_to_next_upload - seconds_since_last_upload logging.info(f"Throttling upload by waiting for {time_to_wait} seconds") time.sleep(time_to_wait) self.time_of_last_upload = datetime.datetime.now() self.size_of_last_upload = part_size def flush(self): if not self.upload_metadata: self.upload_metadata = self.cloud_interface.create_multipart_upload( self.key ) part_size = self.buffer.tell() self.buffer.flush() self.buffer.seek(0, os.SEEK_SET) self.counter += 1 if self.max_bandwidth: # Upload throttling is applied just before uploading the next part so that # compression and flushing have already happened before we start waiting. self._throttle_upload(part_size) self.cloud_interface.async_upload_part( upload_metadata=self.upload_metadata, key=self.key, body=self.buffer, part_number=self.counter, ) self.buffer.close() self.buffer = None def close(self): if self.tar: self.tar.close() self.flush() self.cloud_interface.async_complete_multipart_upload( upload_metadata=self.upload_metadata, key=self.key, parts_count=self.counter, ) self.stats = self.cloud_interface.wait_for_multipart_upload(self.key) class CloudUploadController(object): def __init__( self, cloud_interface, key_prefix, max_archive_size, compression, min_chunk_size=None, max_bandwidth=None, ): """ Create a new controller that upload the backup in cloud storage :param CloudInterface cloud_interface: cloud interface instance :param str|None key_prefix: path inside the bucket :param int max_archive_size: the maximum size of an archive :param str|None compression: required compression :param int|None min_chunk_size: the minimum size of a single upload part :param int|None max_bandwidth: the maximum amount of data per second that should be uploaded during the backup """ self.cloud_interface = cloud_interface if key_prefix and key_prefix[0] == "/": key_prefix = key_prefix[1:] self.key_prefix = key_prefix if max_archive_size < self.cloud_interface.MAX_ARCHIVE_SIZE: self.max_archive_size = max_archive_size else: logging.warning( "max-archive-size too big. Capping it to to %s", pretty_size(self.cloud_interface.MAX_ARCHIVE_SIZE), ) self.max_archive_size = self.cloud_interface.MAX_ARCHIVE_SIZE # We aim to a maximum of MAX_CHUNKS_PER_FILE / 2 chunks per file calculated_chunk_size = 2 * int( max_archive_size / self.cloud_interface.MAX_CHUNKS_PER_FILE ) # Use whichever is higher - the calculated chunk_size, the requested # min_chunk_size or the cloud interface MIN_CHUNK_SIZE. possible_min_chunk_sizes = [ calculated_chunk_size, cloud_interface.MIN_CHUNK_SIZE, ] if min_chunk_size is not None: possible_min_chunk_sizes.append(min_chunk_size) self.chunk_size = max(possible_min_chunk_sizes) self.compression = compression self.max_bandwidth = max_bandwidth self.tar_list = {} self.upload_stats = {} """Already finished uploads list""" self.copy_start_time = datetime.datetime.now() """Copy start time""" self.copy_end_time = None """Copy end time""" def _build_dest_name(self, name, count=0): """ Get the destination tar name :param str name: the name prefix :param int count: the part count :rtype: str """ components = [name] if count > 0: components.append("_%04d" % count) components.append(".tar") if self.compression == "gz": components.append(".gz") elif self.compression == "bz2": components.append(".bz2") elif self.compression == "snappy": components.append(".snappy") return "".join(components) def _get_tar(self, name): """ Get a named tar file from cloud storage. Subsequent call with the same name return the same name :param str name: tar name :rtype: tarfile.TarFile """ if name not in self.tar_list or not self.tar_list[name]: self.tar_list[name] = [ CloudTarUploader( cloud_interface=self.cloud_interface, key=os.path.join(self.key_prefix, self._build_dest_name(name)), chunk_size=self.chunk_size, compression=self.compression, max_bandwidth=self.max_bandwidth, ) ] # If the current uploading file size is over DEFAULT_MAX_TAR_SIZE # Close the current file and open the next part uploader = self.tar_list[name][-1] if uploader.size > self.max_archive_size: uploader.close() uploader = CloudTarUploader( cloud_interface=self.cloud_interface, key=os.path.join( self.key_prefix, self._build_dest_name(name, len(self.tar_list[name])), ), chunk_size=self.chunk_size, compression=self.compression, max_bandwidth=self.max_bandwidth, ) self.tar_list[name].append(uploader) return uploader.tar def upload_directory(self, label, src, dst, exclude=None, include=None): logging.info( "Uploading '%s' directory '%s' as '%s'", label, src, self._build_dest_name(dst), ) for root, dirs, files in os.walk(src): tar_root = os.path.relpath(root, src) if not path_allowed(exclude, include, tar_root, True): continue try: self._get_tar(dst).add(root, arcname=tar_root, recursive=False) except EnvironmentError as e: if e.errno == errno.ENOENT: # If a directory disappeared just skip it, # WAL reply will take care during recovery. continue else: raise for item in files: tar_item = os.path.join(tar_root, item) if not path_allowed(exclude, include, tar_item, False): continue logging.debug("Uploading %s", tar_item) try: self._get_tar(dst).add(os.path.join(root, item), arcname=tar_item) except EnvironmentError as e: if e.errno == errno.ENOENT: # If a file disappeared just skip it, # WAL reply will take care during recovery. continue else: raise def add_file(self, label, src, dst, path, optional=False): if optional and not os.path.exists(src): return logging.info( "Uploading '%s' file from '%s' to '%s' with path '%s'", label, src, self._build_dest_name(dst), path, ) tar = self._get_tar(dst) tar.add(src, arcname=path) def add_fileobj(self, label, fileobj, dst, path, mode=None, uid=None, gid=None): logging.info( "Uploading '%s' file to '%s' with path '%s'", label, self._build_dest_name(dst), path, ) tar = self._get_tar(dst) tarinfo = tar.tarinfo(path) fileobj.seek(0, os.SEEK_END) tarinfo.size = fileobj.tell() if mode is not None: tarinfo.mode = mode if uid is not None: tarinfo.gid = uid if gid is not None: tarinfo.gid = gid fileobj.seek(0, os.SEEK_SET) tar.addfile(tarinfo, fileobj) def close(self): logging.info("Marking all the uploaded archives as 'completed'") for name in self.tar_list: if self.tar_list[name]: # Tho only opened file is the last one, all the others # have been already closed self.tar_list[name][-1].close() self.upload_stats[name] = [tar.stats for tar in self.tar_list[name]] self.tar_list[name] = None # Store the end time self.copy_end_time = datetime.datetime.now() def statistics(self): """ Return statistics about the CloudUploadController object. :rtype: dict """ logging.info("Calculating backup statistics") # This method can only run at the end of a non empty copy assert self.copy_end_time assert self.upload_stats # Initialise the result calculating the total runtime stat = { "total_time": total_seconds(self.copy_end_time - self.copy_start_time), "number_of_workers": self.cloud_interface.worker_processes_count, # Cloud uploads have no analysis "analysis_time": 0, "analysis_time_per_item": {}, "copy_time_per_item": {}, "serialized_copy_time_per_item": {}, } # Calculate the time spent uploading upload_start = None upload_end = None serialized_time = datetime.timedelta(0) for name in self.upload_stats: name_start = None name_end = None total_time = datetime.timedelta(0) for index, data in enumerate(self.upload_stats[name]): logging.debug( "Calculating statistics for file %s, index %s, data: %s", name, index, json.dumps(data, indent=2, sort_keys=True, cls=BarmanEncoder), ) if upload_start is None or upload_start > data["start_time"]: upload_start = data["start_time"] if upload_end is None or upload_end < data["end_time"]: upload_end = data["end_time"] if name_start is None or name_start > data["start_time"]: name_start = data["start_time"] if name_end is None or name_end < data["end_time"]: name_end = data["end_time"] parts = data["parts"] for num in parts: part = parts[num] total_time += part["end_time"] - part["start_time"] stat["serialized_copy_time_per_item"][name] = total_seconds(total_time) serialized_time += total_time # Cloud uploads have no analysis stat["analysis_time_per_item"][name] = 0 stat["copy_time_per_item"][name] = total_seconds(name_end - name_start) # Store the total time spent by copying stat["copy_time"] = total_seconds(upload_end - upload_start) stat["serialized_copy_time"] = total_seconds(serialized_time) return stat class FileUploadStatistics(dict): def __init__(self, *args, **kwargs): super(FileUploadStatistics, self).__init__(*args, **kwargs) start_time = datetime.datetime.now() self.setdefault("status", "uploading") self.setdefault("start_time", start_time) self.setdefault("parts", {}) def set_part_end_time(self, part_number, end_time): part = self["parts"].setdefault(part_number, {"part_number": part_number}) part["end_time"] = end_time def set_part_start_time(self, part_number, start_time): part = self["parts"].setdefault(part_number, {"part_number": part_number}) part["start_time"] = start_time class DecompressingStreamingIO(RawIOBase): """ Provide an IOBase interface which decompresses streaming cloud responses. This is intended to wrap azure_blob_storage.StreamingBlobIO and aws_s3.StreamingBodyIO objects, transparently decompressing chunks while continuing to expose them via the read method of the IOBase interface. This allows TarFile to stream the uncompressed data directly from the cloud provider responses without requiring it to know anything about the compression. """ # The value of 65536 for the chunk size is based on comments in the python-snappy # library which suggest it should be good for almost every scenario. # See: https://github.com/andrix/python-snappy/blob/0.6.0/snappy/snappy.py#L300 COMPRESSED_CHUNK_SIZE = 65536 def __init__(self, streaming_response, decompressor): """ Create a new DecompressingStreamingIO object. A DecompressingStreamingIO object will be created which reads compressed bytes from streaming_response and decompresses them with the supplied decompressor. :param RawIOBase streaming_response: A file-like object which provides the data in the response streamed from the cloud provider. :param barman.clients.cloud_compression.ChunkedCompressor: A ChunkedCompressor object which provides a decompress(bytes) method to return the decompressed bytes. """ self.streaming_response = streaming_response self.decompressor = decompressor self.buffer = bytes() def _read_from_uncompressed_buffer(self, n): """ Read up to n bytes from the local buffer of uncompressed data. Removes up to n bytes from the local buffer and returns them. If n is greater than the length of the buffer then the entire buffer content is returned and the buffer is emptied. :param int n: The number of bytes to read :return: The bytes read from the local buffer :rtype: bytes """ if n <= len(self.buffer): return_bytes = self.buffer[:n] self.buffer = self.buffer[n:] return return_bytes else: return_bytes = self.buffer self.buffer = bytes() return return_bytes def read(self, n=-1): """ Read up to n bytes of uncompressed data from the wrapped IOBase. Bytes are initially read from the local buffer of uncompressed data. If more bytes are required then chunks of COMPRESSED_CHUNK_SIZE are read from the wrapped IOBase and decompressed in memory until >= n uncompressed bytes have been read. n bytes are then returned with any remaining bytes being stored in the local buffer for future requests. :param int n: The number of uncompressed bytes required :return: Up to n uncompressed bytes from the wrapped IOBase :rtype: bytes """ uncompressed_bytes = self._read_from_uncompressed_buffer(n) if len(uncompressed_bytes) == n: return uncompressed_bytes while len(uncompressed_bytes) < n: compressed_bytes = self.streaming_response.read(self.COMPRESSED_CHUNK_SIZE) uncompressed_bytes += self.decompressor.decompress(compressed_bytes) if len(compressed_bytes) < self.COMPRESSED_CHUNK_SIZE: # If we got fewer bytes than we asked for then we're done break return_bytes = uncompressed_bytes[:n] self.buffer = uncompressed_bytes[n:] return return_bytes class CloudInterface(with_metaclass(ABCMeta)): """ Abstract base class which provides the interface between barman and cloud storage providers. Support for individual cloud providers should be implemented by inheriting from this class and providing implementations for the abstract methods. This class provides generic boilerplate for the asynchronous and parallel upload of objects to cloud providers which support multipart uploads. These uploads are carried out by worker processes which are spawned by _ensure_async and consume upload jobs from a queue. The public async_upload_part and async_complete_multipart_upload methods add jobs to this queue. When the worker processes consume the jobs they execute the synchronous counterparts to the async_* methods (_upload_part and _complete_multipart_upload) which must be implemented in CloudInterface sub-classes. Additional boilerplate for creating buckets and streaming objects as tar files is also provided. """ @abstractproperty def MAX_CHUNKS_PER_FILE(self): """ Maximum number of chunks allowed in a single file in cloud storage. The exact definition of chunk depends on the cloud provider, for example in AWS S3 a chunk would be one part in a multipart upload. In Azure a chunk would be a single block of a block blob. :type: int """ pass @abstractproperty def MIN_CHUNK_SIZE(self): """ Minimum size in bytes of a single chunk. :type: int """ pass @abstractproperty def MAX_ARCHIVE_SIZE(self): """ Maximum size in bytes of a single file in cloud storage. :type: int """ pass @abstractproperty def MAX_DELETE_BATCH_SIZE(self): """ The maximum number of objects which can be deleted in a single batch. :type: int """ pass def __init__(self, url, jobs=2, tags=None, delete_batch_size=None): """ Base constructor :param str url: url for the cloud storage resource :param int jobs: How many sub-processes to use for asynchronous uploading, defaults to 2. :param List[tuple] tags: List of tags as k,v tuples to be added to all uploaded objects :param int|None delete_batch_size: the maximum number of objects to be deleted in a single request """ self.url = url self.tags = tags # We use the maximum allowed batch size by default. self.delete_batch_size = self.MAX_DELETE_BATCH_SIZE if delete_batch_size is not None: # If a specific batch size is requested we clamp it between 1 and the # maximum allowed batch size. self.delete_batch_size = max( 1, min(delete_batch_size, self.MAX_DELETE_BATCH_SIZE), ) # The worker process and the shared queue are created only when # needed self.queue = None self.result_queue = None self.errors_queue = None self.done_queue = None self.error = None self.abort_requested = False self.worker_processes_count = jobs self.worker_processes = [] # The parts DB is a dictionary mapping each bucket key name to a list # of uploaded parts. # This structure is updated by the _refresh_parts_db method call self.parts_db = collections.defaultdict(list) # Statistics about uploads self.upload_stats = collections.defaultdict(FileUploadStatistics) def close(self): """ Wait for all the asynchronous operations to be done """ if self.queue: for _ in self.worker_processes: self.queue.put(None) for process in self.worker_processes: process.join() def _abort(self): """ Abort all the operations """ if self.queue: for process in self.worker_processes: os.kill(process.pid, signal.SIGINT) self.close() def _ensure_async(self): """ Ensure that the asynchronous execution infrastructure is up and the worker process is running """ if self.queue: return manager = multiprocessing.Manager() self.queue = manager.JoinableQueue(maxsize=self.worker_processes_count) self.result_queue = manager.Queue() self.errors_queue = manager.Queue() self.done_queue = manager.Queue() # Delay assigning the worker_processes list to the object until we have # finished spawning the workers so they do not get pickled by multiprocessing # (pickling the worker process references will fail in Python >= 3.8) worker_processes = [] for process_number in range(self.worker_processes_count): process = multiprocessing.Process( target=self._worker_process_main, args=(process_number,) ) process.start() worker_processes.append(process) self.worker_processes = worker_processes def _retrieve_results(self): """ Receive the results from workers and update the local parts DB, making sure that each part list is sorted by part number """ # Wait for all the current jobs to be completed self.queue.join() touched_keys = [] while not self.result_queue.empty(): result = self.result_queue.get() touched_keys.append(result["key"]) self.parts_db[result["key"]].append(result["part"]) # Save the upload end time of the part stats = self.upload_stats[result["key"]] stats.set_part_end_time(result["part_number"], result["end_time"]) for key in touched_keys: self.parts_db[key] = sorted( self.parts_db[key], key=operator.itemgetter("PartNumber") ) # Read the results of completed uploads while not self.done_queue.empty(): result = self.done_queue.get() self.upload_stats[result["key"]].update(result) # Raise an error if a job failed self._handle_async_errors() def _handle_async_errors(self): """ If an upload error has been discovered, stop the upload process, stop all the workers and raise an exception :return: """ # If an error has already been reported, do nothing if self.error: return try: self.error = self.errors_queue.get_nowait() except EmptyQueue: return logging.error("Error received from upload worker: %s", self.error) self._abort() raise CloudUploadingError(self.error) def _worker_process_main(self, process_number): """ Repeatedly grab a task from the queue and execute it, until a task containing "None" is grabbed, indicating that the process must stop. :param int process_number: the process number, used in the logging output """ logging.info("Upload process started (worker %s)", process_number) # We create a new session instead of reusing the one # from the parent process to avoid any race condition self._reinit_session() while True: task = self.queue.get() if not task: self.queue.task_done() break try: self._worker_process_execute_job(task, process_number) except Exception as exc: logging.error( "Upload error: %s (worker %s)", force_str(exc), process_number ) logging.debug("Exception details:", exc_info=exc) self.errors_queue.put(force_str(exc)) except KeyboardInterrupt: if not self.abort_requested: logging.info( "Got abort request: upload cancelled (worker %s)", process_number, ) self.abort_requested = True finally: self.queue.task_done() logging.info("Upload process stopped (worker %s)", process_number) def _worker_process_execute_job(self, task, process_number): """ Exec a single task :param Dict task: task to execute :param int process_number: the process number, used in the logging output :return: """ if task["job_type"] == "upload_part": if self.abort_requested: logging.info( "Skipping '%s', part '%s' (worker %s)" % (task["key"], task["part_number"], process_number) ) os.unlink(task["body"]) return else: logging.info( "Uploading '%s', part '%s' (worker %s)" % (task["key"], task["part_number"], process_number) ) with open(task["body"], "rb") as fp: part = self._upload_part( task["upload_metadata"], task["key"], fp, task["part_number"] ) os.unlink(task["body"]) self.result_queue.put( { "key": task["key"], "part_number": task["part_number"], "end_time": datetime.datetime.now(), "part": part, } ) elif task["job_type"] == "complete_multipart_upload": if self.abort_requested: logging.info("Aborting %s (worker %s)" % (task["key"], process_number)) self._abort_multipart_upload(task["upload_metadata"], task["key"]) self.done_queue.put( { "key": task["key"], "end_time": datetime.datetime.now(), "status": "aborted", } ) else: logging.info( "Completing '%s' (worker %s)" % (task["key"], process_number) ) self._complete_multipart_upload( task["upload_metadata"], task["key"], task["parts_metadata"] ) self.done_queue.put( { "key": task["key"], "end_time": datetime.datetime.now(), "status": "done", } ) else: raise ValueError("Unknown task: %s", repr(task)) def async_upload_part(self, upload_metadata, key, body, part_number): """ Asynchronously upload a part into a multipart upload :param dict upload_metadata: Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3 :param str key: The key to use in the cloud service :param any body: A stream-like object to upload :param int part_number: Part number, starting from 1 """ # If an error has already been reported, do nothing if self.error: return self._ensure_async() self._handle_async_errors() # Save the upload start time of the part stats = self.upload_stats[key] stats.set_part_start_time(part_number, datetime.datetime.now()) # Pass the job to the uploader process self.queue.put( { "job_type": "upload_part", "upload_metadata": upload_metadata, "key": key, "body": body.name, "part_number": part_number, } ) def async_complete_multipart_upload(self, upload_metadata, key, parts_count): """ Asynchronously finish a certain multipart upload. This method grant that the final call to the cloud storage will happen after all the already scheduled parts have been uploaded. :param dict upload_metadata: Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3 :param str key: The key to use in the cloud service :param int parts_count: Number of parts """ # If an error has already been reported, do nothing if self.error: return self._ensure_async() self._handle_async_errors() # If parts_db has less then expected parts for this upload, # wait for the workers to send the missing metadata while len(self.parts_db[key]) < parts_count: # Wait for all the current jobs to be completed and # receive all available updates on worker status self._retrieve_results() # Finish the job in the uploader process self.queue.put( { "job_type": "complete_multipart_upload", "upload_metadata": upload_metadata, "key": key, "parts_metadata": self.parts_db[key], } ) del self.parts_db[key] def wait_for_multipart_upload(self, key): """ Wait for a multipart upload to be completed and return the result :param str key: The key to use in the cloud service """ # The upload must exist assert key in self.upload_stats # async_complete_multipart_upload must have been called assert key not in self.parts_db # If status is still uploading the upload has not finished yet while self.upload_stats[key]["status"] == "uploading": # Wait for all the current jobs to be completed and # receive all available updates on worker status self._retrieve_results() return self.upload_stats[key] def setup_bucket(self): """ Search for the target bucket. Create it if not exists """ if self.bucket_exists is None: self.bucket_exists = self._check_bucket_existence() # Create the bucket if it doesn't exist if not self.bucket_exists: self._create_bucket() self.bucket_exists = True def extract_tar(self, key, dst): """ Extract a tar archive from cloud to the local directory :param str key: The key identifying the tar archive :param str dst: Path of the directory into which the tar archive should be extracted """ extension = os.path.splitext(key)[-1] compression = "" if extension == ".tar" else extension[1:] tar_mode = cloud_compression.get_streaming_tar_mode("r", compression) fileobj = self.remote_open(key, cloud_compression.get_compressor(compression)) with tarfile.open(fileobj=fileobj, mode=tar_mode) as tf: tf.extractall(path=dst) @abstractmethod def _reinit_session(self): """ Reinitialises any resources used to maintain a session with a cloud provider. This is called by child processes in order to avoid any potential race conditions around re-using the same session as the parent process. """ @abstractmethod def test_connectivity(self): """ Test that the cloud provider is reachable :return: True if the cloud provider is reachable, False otherwise :rtype: bool """ @abstractmethod def _check_bucket_existence(self): """ Check cloud storage for the target bucket :return: True if the bucket exists, False otherwise :rtype: bool """ @abstractmethod def _create_bucket(self): """ Create the bucket in cloud storage """ @abstractmethod def list_bucket(self, prefix="", delimiter=DEFAULT_DELIMITER): """ List bucket content in a directory manner :param str prefix: :param str delimiter: :return: List of objects and dirs right under the prefix :rtype: List[str] """ @abstractmethod def download_file(self, key, dest_path, decompress): """ Download a file from cloud storage :param str key: The key identifying the file to download :param str dest_path: Where to put the destination file :param str|None decompress: Compression scheme to use for decompression """ @abstractmethod def remote_open(self, key, decompressor=None): """ Open a remote object in cloud storage and returns a readable stream :param str key: The key identifying the object to open :param barman.clients.cloud_compression.ChunkedCompressor decompressor: A ChunkedCompressor object which will be used to decompress chunks of bytes as they are read from the stream :return: A file-like object from which the stream can be read or None if the key does not exist """ @abstractmethod def upload_fileobj(self, fileobj, key, override_tags=None): """ Synchronously upload the content of a file-like object to a cloud key :param fileobj IOBase: File-like object to upload :param str key: The key to identify the uploaded object :param List[tuple] override_tags: List of k,v tuples which should override any tags already defined in the cloud interface """ @abstractmethod def create_multipart_upload(self, key): """ Create a new multipart upload and return any metadata returned by the cloud provider. This metadata is treated as an opaque blob by CloudInterface and will be passed into the _upload_part, _complete_multipart_upload and _abort_multipart_upload methods. The implementations of these methods will need to handle this metadata in the way expected by the cloud provider. Some cloud services do not require multipart uploads to be explicitly created. In such cases the implementation can be a no-op which just returns None. :param key: The key to use in the cloud service :return: The multipart upload metadata :rtype: dict[str, str]|None """ @abstractmethod def _upload_part(self, upload_metadata, key, body, part_number): """ Upload a part into this multipart upload and return a dict of part metadata. The part metadata must contain the key "PartNumber" and can optionally contain any other metadata available (for example the ETag returned by S3). The part metadata will included in a list of metadata for all parts of the upload which is passed to the _complete_multipart_upload method. :param dict upload_metadata: Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3 :param str key: The key to use in the cloud service :param object body: A stream-like object to upload :param int part_number: Part number, starting from 1 :return: The part metadata :rtype: dict[str, None|str] """ @abstractmethod def _complete_multipart_upload(self, upload_metadata, key, parts_metadata): """ Finish a certain multipart upload :param dict upload_metadata: Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3 :param str key: The key to use in the cloud service :param List[dict] parts_metadata: The list of metadata for the parts composing the multipart upload. Each part is guaranteed to provide a PartNumber and may optionally contain additional metadata returned by the cloud provider such as ETags. """ @abstractmethod def _abort_multipart_upload(self, upload_metadata, key): """ Abort a certain multipart upload The implementation of this method should clean up any dangling resources left by the incomplete upload. :param dict upload_metadata: Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3 :param str key: The key to use in the cloud service """ @abstractmethod def _delete_objects_batch(self, paths): """ Delete a single batch of objects :param List[str] paths: """ if len(paths) > self.MAX_DELETE_BATCH_SIZE: raise ValueError("Max batch size exceeded") def delete_objects(self, paths): """ Delete the objects at the specified paths Deletes the objects defined by the supplied list of paths in batches specified by either batch_size or MAX_DELETE_BATCH_SIZE, whichever is lowest. :param List[str] paths: """ errors = False for i in range_fun(0, len(paths), self.delete_batch_size): try: self._delete_objects_batch(paths[i : i + self.delete_batch_size]) except CloudProviderError: # Don't let one error stop us from trying to delete any remaining # batches. errors = True if errors: raise CloudProviderError( "Error from cloud provider while deleting objects - " "please check the command output." ) @abstractmethod def get_prefixes(self, prefix): """ Return only the common prefixes under the supplied prefix. :param str prefix: The object key prefix under which the common prefixes will be found. :rtype: Iterator[str] :return: A list of unique prefixes immediately under the supplied prefix. """ @abstractmethod def delete_under_prefix(self, prefix): """ Delete all objects under the specified prefix. :param str prefix: The object key prefix under which all objects should be deleted. """ class CloudBackup(with_metaclass(ABCMeta)): """ Abstract base class for taking cloud backups of PostgreSQL servers. This class handles the coordination of the physical backup copy with the PostgreSQL server via the PostgreSQL low-level backup API. This is handled by the _coordinate_backup method. Concrete classes will need to implement the following abstract methods which are called during the _coordinate_backup method: _take_backup _upload_backup_label _finalise_copy _add_stats_to_backup_info Implementations must also implement the public backup method which should carry out any prepartion and invoke _coordinate_backup. """ def __init__(self, server_name, cloud_interface, postgres, backup_name=None): """ :param str server_name: The name of the server being backed up. :param CloudInterface cloud_interface: The CloudInterface for interacting with the cloud object store. :param barman.postgres.PostgreSQLConnection|None postgres: A connection to the PostgreSQL instance being backed up. :param str|None backup_name: A friendly name which can be used to reference this backup in the future. """ self.server_name = server_name self.cloud_interface = cloud_interface self.postgres = postgres self.backup_name = backup_name # Stats self.copy_start_time = None self.copy_end_time = None # Object properties set at backup time self.backup_info = None # The following abstract methods are called when coordinating the backup. # They are all specific to the backup copy mechanism so the implementation must # happen in the subclass. @abstractmethod def _take_backup(self): """ Perform the actions necessary to create the backup. This method must be called between pg_backup_start and pg_backup_stop which is guaranteed to happen if the _coordinate_backup method is used. """ @abstractmethod def _upload_backup_label(self): """ Upload the backup label to cloud storage. """ @abstractmethod def _finalise_copy(self): """ Perform any finalisation required to complete the copy of backup data. """ @abstractmethod def _add_stats_to_backup_info(self): """ Add statistics about the backup to self.backup_info. """ # The public facing backup method must also be implemented in concrete classes. @abstractmethod def backup(self): """ External interface for performing a cloud backup of the postgres server. When providing an implementation of this method, concrete classes *must* set `self.backup_info` before coordinating the backup. Implementations *should* call `self._coordinate_backup` to carry out the backup process. """ # The following concrete methods are independent of backup copy mechanism. def _start_backup(self): """ Start the backup via the PostgreSQL backup API. """ self.strategy = ConcurrentBackupStrategy(self.postgres, self.server_name) logging.info("Starting backup '%s'", self.backup_info.backup_id) self.strategy.start_backup(self.backup_info) def _stop_backup(self): """ Stop the backup via the PostgreSQL backup API. """ logging.info("Stopping backup '%s'", self.backup_info.backup_id) self.strategy.stop_backup(self.backup_info) def _create_restore_point(self): """ Create a restore point named after this backup. """ target_name = "barman_%s" % self.backup_info.backup_id self.postgres.create_restore_point(target_name) def _get_backup_info(self, server_name): """ Create and return the backup_info for this CloudBackup. """ backup_info = BackupInfo( backup_id=datetime.datetime.now().strftime("%Y%m%dT%H%M%S"), server_name=server_name, ) backup_info.set_attribute("systemid", self.postgres.get_systemid()) return backup_info def _upload_backup_info(self): """ Upload the backup_info for this CloudBackup. """ with BytesIO() as backup_info_file: key = os.path.join( self.cloud_interface.path, self.server_name, "base", self.backup_info.backup_id, "backup.info", ) self.backup_info.save(file_object=backup_info_file) backup_info_file.seek(0, os.SEEK_SET) logging.info("Uploading '%s'", key) self.cloud_interface.upload_fileobj(backup_info_file, key) def _check_postgres_version(self): """ Verify we are running against a supported PostgreSQL version. """ if not self.postgres.is_minimal_postgres_version(): raise BackupException( "unsupported PostgresSQL version %s. Expecting %s or above." % ( self.postgres.server_major_version, self.postgres.minimal_txt_version, ) ) def _log_end_of_backup(self): """ Write log lines indicating end of backup. """ logging.info( "Backup end at LSN: %s (%s, %08X)", self.backup_info.end_xlog, self.backup_info.end_wal, self.backup_info.end_offset, ) logging.info( "Backup completed (start time: %s, elapsed time: %s)", self.copy_start_time, human_readable_timedelta(datetime.datetime.now() - self.copy_start_time), ) def _coordinate_backup(self): """ Coordinate taking the backup with the PostgreSQL server. """ try: # Store the start time self.copy_start_time = datetime.datetime.now() self._start_backup() self._take_backup() self._stop_backup() self._create_restore_point() self._upload_backup_label() self._finalise_copy() # Store the end time self.copy_end_time = datetime.datetime.now() # Store statistics about the copy self._add_stats_to_backup_info() # Set the backup status as DONE self.backup_info.set_attribute("status", BackupInfo.DONE) except BaseException as exc: # Mark the backup as failed and exit self.handle_backup_errors("uploading data", exc, self.backup_info) raise SystemExit(1) finally: # Add the name to the backup info if self.backup_name is not None: self.backup_info.set_attribute("backup_name", self.backup_name) try: self._upload_backup_info() except BaseException as exc: # Mark the backup as failed and exit self.handle_backup_errors( "uploading backup.info file", exc, self.backup_info ) raise SystemExit(1) self._log_end_of_backup() def handle_backup_errors(self, action, exc, backup_info): """ Mark the backup as failed and exit :param str action: the upload phase that has failed :param BaseException exc: the exception that caused the failure :param barman.infofile.BackupInfo backup_info: the backup info file """ msg_lines = force_str(exc).strip().splitlines() # If the exception has no attached message use the raw # type name if len(msg_lines) == 0: msg_lines = [type(exc).__name__] if backup_info: # Use only the first line of exception message # in backup_info error field backup_info.set_attribute("status", BackupInfo.FAILED) backup_info.set_attribute( "error", "failure %s (%s)" % (action, msg_lines[0]) ) logging.error("Backup failed %s (%s)", action, msg_lines[0]) logging.debug("Exception details:", exc_info=exc) class CloudBackupUploader(CloudBackup): """ Uploads backups from a PostgreSQL server to cloud object storage. """ def __init__( self, server_name, cloud_interface, max_archive_size, postgres, compression=None, backup_name=None, min_chunk_size=None, max_bandwidth=None, ): """ Base constructor. :param str server_name: The name of the server as configured in Barman :param CloudInterface cloud_interface: The interface to use to upload the backup :param int max_archive_size: the maximum size of an uploading archive :param barman.postgres.PostgreSQLConnection|None postgres: A connection to the PostgreSQL instance being backed up. :param str compression: Compression algorithm to use :param str|None backup_name: A friendly name which can be used to reference this backup in the future. :param int min_chunk_size: the minimum size of a single upload part :param int max_bandwidth: the maximum amount of data per second that should be uploaded during the backup """ super(CloudBackupUploader, self).__init__( server_name, cloud_interface, postgres, backup_name, ) self.compression = compression self.max_archive_size = max_archive_size self.min_chunk_size = min_chunk_size self.max_bandwidth = max_bandwidth # Object properties set at backup time self.controller = None # The following methods add specific functionality required to upload backups to # cloud object storage. def _get_tablespace_location(self, tablespace): """ Return the on-disk location of the supplied tablespace. This will usually just be the location of the tablespace however subclasses which run against Barman server will need to override this method. :param infofile.Tablespace tablespace: The tablespace whose location should be returned. :rtype: str :return: The path of the supplied tablespace. """ return tablespace.location def _create_upload_controller(self, backup_id): """ Create an upload controller from the specified backup_id :param str backup_id: The backup identifier :rtype: CloudUploadController :return: The upload controller """ key_prefix = os.path.join( self.cloud_interface.path, self.server_name, "base", backup_id, ) return CloudUploadController( self.cloud_interface, key_prefix, self.max_archive_size, self.compression, self.min_chunk_size, self.max_bandwidth, ) def _backup_data_files( self, controller, backup_info, pgdata_dir, server_major_version ): """ Perform the actual copy of the data files uploading it to cloud storage. First, it copies one tablespace at a time, then the PGDATA directory, then pg_control. Bandwidth limitation, according to configuration, is applied in the process. :param barman.cloud.CloudUploadController controller: upload controller :param barman.infofile.BackupInfo backup_info: backup information :param str pgdata_dir: Path to pgdata directory :param str server_major_version: Major version of the postgres server being backed up """ # List of paths to be excluded by the PGDATA copy exclude = [] # Process every tablespace if backup_info.tablespaces: for tablespace in backup_info.tablespaces: # If the tablespace location is inside the data directory, # exclude and protect it from being copied twice during # the data directory copy if tablespace.location.startswith(backup_info.pgdata + "/"): exclude += [tablespace.location[len(backup_info.pgdata) :]] # Exclude and protect the tablespace from being copied again # during the data directory copy exclude += ["/pg_tblspc/%s" % tablespace.oid] # Copy the tablespace directory. # NOTE: Barman should archive only the content of directory # "PG_" + PG_MAJORVERSION + "_" + CATALOG_VERSION_NO # but CATALOG_VERSION_NO is not easy to retrieve, so we copy # "PG_" + PG_MAJORVERSION + "_*" # It could select some spurious directory if a development or # a beta version have been used, but it's good enough for a # production system as it filters out other major versions. controller.upload_directory( label=tablespace.name, src=self._get_tablespace_location(tablespace), dst="%s" % tablespace.oid, exclude=["/*"] + EXCLUDE_LIST, include=["/PG_%s_*" % server_major_version], ) # Copy PGDATA directory (or if that is itself a symlink, just follow it # and copy whatever it points to; we won't store the symlink in the tar # file) if os.path.islink(pgdata_dir): pgdata_dir = os.path.realpath(pgdata_dir) controller.upload_directory( label="pgdata", src=pgdata_dir, dst="data", exclude=PGDATA_EXCLUDE_LIST + EXCLUDE_LIST + exclude, ) # At last copy pg_control controller.add_file( label="pg_control", src="%s/global/pg_control" % pgdata_dir, dst="data", path="global/pg_control", ) def _backup_config_files(self, controller, backup_info): """ Perform the backup of any external config files. :param barman.cloud.CloudUploadController controller: upload controller :param barman.infofile.BackupInfo backup_info: backup information """ # Copy configuration files (if not inside PGDATA) external_config_files = backup_info.get_external_config_files() included_config_files = [] for config_file in external_config_files: # Add included files to a list, they will be handled later if config_file.file_type == "include": included_config_files.append(config_file) continue # If the ident file is missing, it isn't an error condition # for PostgreSQL. # Barman is consistent with this behavior. optional = False if config_file.file_type == "ident_file": optional = True # Create the actual copy jobs in the controller controller.add_file( label=config_file.file_type, src=config_file.path, dst="data", path=os.path.basename(config_file.path), optional=optional, ) # Check for any include directives in PostgreSQL configuration # Currently, include directives are not supported for files that # reside outside PGDATA. These files must be manually backed up. # Barman will emit a warning and list those files if any(included_config_files): msg = ( "The usage of include directives is not supported " "for files that reside outside PGDATA.\n" "Please manually backup the following files:\n" "\t%s\n" % "\n\t".join(icf.path for icf in included_config_files) ) logging.warning(msg) @property def _pgdata_dir(self): """ The location of the PGDATA directory to be backed up. """ return self.backup_info.pgdata # The remaining methods are the concrete implementations of the abstract methods from # the parent class. def _take_backup(self): """ Make a backup by copying PGDATA, tablespaces and config to cloud storage. """ self._backup_data_files( self.controller, self.backup_info, self._pgdata_dir, self.postgres.server_major_version, ) self._backup_config_files(self.controller, self.backup_info) def _finalise_copy(self): """ Close the upload controller, forcing the flush of any buffered uploads. """ self.controller.close() def _upload_backup_label(self): """ Upload the backup label to cloud storage. Upload is via the upload controller so that the backup label is added to the data tarball. """ if self.backup_info.backup_label: pgdata_stat = os.stat(self.backup_info.pgdata) self.controller.add_fileobj( label="backup_label", fileobj=BytesIO(self.backup_info.backup_label.encode("UTF-8")), dst="data", path="backup_label", uid=pgdata_stat.st_uid, gid=pgdata_stat.st_gid, ) def _add_stats_to_backup_info(self): """ Adds statistics from the upload controller to the backup_info. """ self.backup_info.set_attribute("copy_stats", self.controller.statistics()) def backup(self): """ Upload a Backup to cloud storage directly from a live PostgreSQL server. """ server_name = "cloud" self.backup_info = self._get_backup_info(server_name) self.controller = self._create_upload_controller(self.backup_info.backup_id) self._check_postgres_version() self._coordinate_backup() class CloudBackupUploaderBarman(CloudBackupUploader): """ A cloud storage upload client for a preexisting backup on the Barman server. """ def __init__( self, server_name, cloud_interface, max_archive_size, backup_dir, backup_id, backup_info_path, compression=None, min_chunk_size=None, max_bandwidth=None, ): """ Create the cloud storage upload client for a backup in the specified location with the specified backup_id. :param str server_name: The name of the server as configured in Barman :param CloudInterface cloud_interface: The interface to use to upload the backup :param int max_archive_size: the maximum size of an uploading archive :param str backup_dir: Path to the directory containing the backup to be uploaded :param str backup_id: The id of the backup to upload :param str backup_info_path: Path of the ``backup.info`` file. :param str compression: Compression algorithm to use :param int min_chunk_size: the minimum size of a single upload part :param int max_bandwidth: the maximum amount of data per second that should be uploaded during the backup """ super(CloudBackupUploaderBarman, self).__init__( server_name, cloud_interface, max_archive_size, compression=compression, postgres=None, min_chunk_size=min_chunk_size, max_bandwidth=max_bandwidth, ) self.backup_dir = backup_dir self.backup_id = backup_id self.backup_info_path = backup_info_path def handle_backup_errors(self, action, exc): """ Log that the backup upload has failed and exit This differs from the function in the superclass because it does not update the backup.info metadata (this must be left untouched since it relates to the original backup made with Barman). :param str action: the upload phase that has failed :param BaseException exc: the exception that caused the failure """ msg_lines = force_str(exc).strip().splitlines() # If the exception has no attached message use the raw # type name if len(msg_lines) == 0: msg_lines = [type(exc).__name__] logging.error("Backup upload failed %s (%s)", action, msg_lines[0]) logging.debug("Exception details:", exc_info=exc) def _get_tablespace_location(self, tablespace): """ Return the on-disk location of the supplied tablespace. Combines the backup_dir and the tablespace OID to determine the location of the tablespace on the Barman server. :param infofile.Tablespace tablespace: The tablespace whose location should be returned. :rtype: str :return: The path of the supplied tablespace. """ return os.path.join(self.backup_dir, str(tablespace.oid)) @property def _pgdata_dir(self): """ The location of the PGDATA directory to be backed up. """ return os.path.join(self.backup_dir, "data") def _take_backup(self): """ Make a backup by copying PGDATA and tablespaces to cloud storage. """ self._backup_data_files( self.controller, self.backup_info, self._pgdata_dir, self.backup_info.pg_major_version(), ) def backup(self): """ Upload a Backup to cloud storage This deviates from other CloudBackup classes because it does not make use of the self._coordinate_backup function. This is because there is no need to coordinate the backup with a live PostgreSQL server, create a restore point or upload the backup label independently of the backup (it will already be in the base backup directoery). """ # Read the backup_info file from disk as the backup has already been created self.backup_info = BackupInfo(self.backup_id) self.backup_info.load(filename=self.backup_info_path) self.controller = self._create_upload_controller(self.backup_id) try: self.copy_start_time = datetime.datetime.now() self._take_backup() # Closing the controller will finalize all the running uploads self.controller.close() # Store the end time self.copy_end_time = datetime.datetime.now() # Manually add backup.info with open(self.backup_info_path, "rb") as backup_info_file: self.cloud_interface.upload_fileobj( backup_info_file, key=os.path.join(self.controller.key_prefix, "backup.info"), ) # Use BaseException instead of Exception to catch events like # KeyboardInterrupt (e.g.: CTRL-C) except BaseException as exc: # Mark the backup as failed and exit self.handle_backup_errors("uploading data", exc) raise SystemExit(1) logging.info( "Upload of backup completed (start time: %s, elapsed time: %s)", self.copy_start_time, human_readable_timedelta(datetime.datetime.now() - self.copy_start_time), ) class CloudBackupSnapshot(CloudBackup): """ A cloud backup client using disk snapshots to create the backup. """ def __init__( self, server_name, cloud_interface, snapshot_interface, postgres, snapshot_instance, snapshot_disks, backup_name=None, ): """ Create the backup client for snapshot backups :param str server_name: The name of the server as configured in Barman :param CloudInterface cloud_interface: The interface to use to upload the backup :param SnapshotInterface snapshot_interface: The interface to use for creating a backup using snapshots :param barman.postgres.PostgreSQLConnection|None postgres: A connection to the PostgreSQL instance being backed up. :param str snapshot_instance: The name of the VM instance to which the disks to be backed up are attached. :param list[str] snapshot_disks: A list containing the names of the disks for which snapshots should be taken at backup time. :param str|None backup_name: A friendly name which can be used to reference this backup in the future. """ super(CloudBackupSnapshot, self).__init__( server_name, cloud_interface, postgres, backup_name ) self.snapshot_interface = snapshot_interface self.snapshot_instance = snapshot_instance self.snapshot_disks = snapshot_disks # The remaining methods are the concrete implementations of the abstract methods from # the parent class. def _finalise_copy(self): """ Perform any finalisation required to complete the copy of backup data. This is a no-op for snapshot backups. """ pass def _add_stats_to_backup_info(self): """ Add statistics about the backup to self.backup_info. """ self.backup_info.set_attribute( "copy_stats", { "copy_time": total_seconds(self.copy_end_time - self.copy_start_time), "total_time": total_seconds(self.copy_end_time - self.copy_start_time), }, ) def _upload_backup_label(self): """ Upload the backup label to cloud storage. Snapshot backups just upload the backup label as a single object rather than adding it to a tar archive. """ backup_label_key = os.path.join( self.cloud_interface.path, self.server_name, "base", self.backup_info.backup_id, "backup_label", ) self.cloud_interface.upload_fileobj( BytesIO(self.backup_info.backup_label.encode("UTF-8")), backup_label_key, ) def _take_backup(self): """ Make a backup by creating snapshots of the specified disks. """ volumes_to_snapshot = self.snapshot_interface.get_attached_volumes( self.snapshot_instance, self.snapshot_disks ) cmd = UnixLocalCommand() SnapshotBackupExecutor.add_mount_data_to_volume_metadata( volumes_to_snapshot, cmd ) self.snapshot_interface.take_snapshot_backup( self.backup_info, self.snapshot_instance, volumes_to_snapshot, ) # The following method implements specific functionality for snapshot backups. def _check_backup_preconditions(self): """ Perform additional checks for snapshot backups, specifically: - check that the VM instance for which snapshots should be taken exists - check that the expected disks are attached to that instance - check that the attached disks are mounted on the filesystem Raises a BackupPreconditionException if any of the checks fail. """ if not self.snapshot_interface.instance_exists(self.snapshot_instance): raise BackupPreconditionException( "Cannot find compute instance %s" % self.snapshot_instance ) cmd = UnixLocalCommand() ( missing_disks, unmounted_disks, ) = SnapshotBackupExecutor.find_missing_and_unmounted_disks( cmd, self.snapshot_interface, self.snapshot_instance, self.snapshot_disks, ) if len(missing_disks) > 0: raise BackupPreconditionException( "Cannot find disks attached to compute instance %s: %s" % (self.snapshot_instance, ", ".join(missing_disks)) ) if len(unmounted_disks) > 0: raise BackupPreconditionException( "Cannot find disks mounted on compute instance %s: %s" % (self.snapshot_instance, ", ".join(unmounted_disks)) ) # Specific implementation of the public-facing backup method. def backup(self): """ Take a backup by creating snapshots of the specified disks. """ self._check_backup_preconditions() self.backup_info = self._get_backup_info(self.server_name) self._check_postgres_version() self._coordinate_backup() class BackupFileInfo(object): def __init__(self, oid=None, base=None, path=None, compression=None): self.oid = oid self.base = base self.path = path self.compression = compression self.additional_files = [] class CloudBackupCatalog(KeepManagerMixinCloud): """ Cloud storage backup catalog """ def __init__(self, cloud_interface, server_name): """ Object responsible for retrieving backup catalog from cloud storage :param CloudInterface cloud_interface: The interface to use to upload the backup :param str server_name: The name of the server as configured in Barman """ super(CloudBackupCatalog, self).__init__( cloud_interface=cloud_interface, server_name=server_name ) self.cloud_interface = cloud_interface self.server_name = server_name self.prefix = os.path.join(self.cloud_interface.path, self.server_name, "base") self.wal_prefix = os.path.join( self.cloud_interface.path, self.server_name, "wals" ) self._backup_list = None self._wal_paths = None self.unreadable_backups = [] def get_backup_list(self): """ Retrieve the list of available backup from cloud storage :rtype: Dict[str,BackupInfo] """ if self._backup_list is None: backup_list = {} # get backups metadata for backup_dir in self.cloud_interface.list_bucket(self.prefix + "/"): # We want only the directories if backup_dir[-1] != "/": continue backup_id = os.path.basename(backup_dir.rstrip("/")) try: backup_info = self.get_backup_info(backup_id) except Exception as exc: logging.warning( "Unable to open backup.info file for %s: %s" % (backup_id, exc) ) self.unreadable_backups.append(backup_id) continue if backup_info: backup_list[backup_id] = backup_info self._backup_list = backup_list return self._backup_list def remove_backup_from_cache(self, backup_id): """ Remove backup with backup_id from the cached list. This is intended for cases where we want to update the state without firing lots of requests at the bucket. """ if self._backup_list: self._backup_list.pop(backup_id) def get_wal_prefixes(self): """ Return only the common prefixes under the wals prefix. """ return self.cloud_interface.get_prefixes(self.wal_prefix) def get_wal_paths(self): """ Retrieve a dict of WAL paths keyed by the WAL name from cloud storage """ if self._wal_paths is None: wal_paths = {} for wal in self.cloud_interface.list_bucket( self.wal_prefix + "/", delimiter="" ): wal_basename = os.path.basename(wal) if xlog.is_any_xlog_file(wal_basename): # We have an uncompressed xlog of some kind wal_paths[wal_basename] = wal else: # Allow one suffix for compression and try again wal_name, suffix = os.path.splitext(wal_basename) if suffix in ALLOWED_COMPRESSIONS and xlog.is_any_xlog_file( wal_name ): wal_paths[wal_name] = wal else: # If it still doesn't look like an xlog file, ignore continue self._wal_paths = wal_paths return self._wal_paths def remove_wal_from_cache(self, wal_name): """ Remove named wal from the cached list. This is intended for cases where we want to update the state without firing lots of requests at the bucket. """ if self._wal_paths: self._wal_paths.pop(wal_name) def _get_backup_info_from_name(self, backup_name): """ Get the backup metadata for the named backup. :param str backup_name: The name of the backup for which the backup metadata should be retrieved :return BackupInfo|None: The backup metadata for the named backup """ available_backups = self.get_backup_list().values() return get_backup_info_from_name(available_backups, backup_name) def parse_backup_id(self, backup_id): """ Parse a backup identifier and return the matching backup ID. If the identifier is a backup ID it is returned, otherwise it is assumed to be a name. :param str backup_id: The backup identifier to be parsed :return str: The matching backup ID for the supplied identifier """ if not is_backup_id(backup_id): backup_info = self._get_backup_info_from_name(backup_id) if backup_info is not None: return backup_info.backup_id else: raise ValueError( "Unknown backup '%s' for server '%s'" % (backup_id, self.server_name) ) else: return backup_id def get_backup_info(self, backup_id): """ Load a BackupInfo from cloud storage :param str backup_id: The backup id to load :rtype: BackupInfo """ backup_info_path = os.path.join(self.prefix, backup_id, "backup.info") backup_info_file = self.cloud_interface.remote_open(backup_info_path) if backup_info_file is None: return None backup_info = BackupInfo(backup_id) backup_info.load(file_object=backup_info_file) return backup_info def get_backup_files(self, backup_info, allow_missing=False): """ Get the list of expected files part of a backup :param BackupInfo backup_info: the backup information :param bool allow_missing: True if missing backup files are allowed, False otherwise. A value of False will cause a SystemExit to be raised if any files expected due to the `backup_info` content cannot be found. :rtype: dict[int, BackupFileInfo] """ # Correctly format the source path source_dir = os.path.join(self.prefix, backup_info.backup_id) base_path = os.path.join(source_dir, "data") backup_files = {None: BackupFileInfo(None, base_path)} if backup_info.tablespaces: for tblspc in backup_info.tablespaces: base_path = os.path.join(source_dir, "%s" % tblspc.oid) backup_files[tblspc.oid] = BackupFileInfo(tblspc.oid, base_path) for item in self.cloud_interface.list_bucket(source_dir + "/"): for backup_file in backup_files.values(): if item.startswith(backup_file.base): # Automatically detect additional files suffix = item[len(backup_file.base) :] # Avoid to match items that are prefix of other items if not suffix or suffix[0] not in (".", "_"): logging.debug( "Skipping spurious prefix match: %s|%s", backup_file.base, suffix, ) continue # If this file have a suffix starting with `_`, # it is an additional file and we add it to the main # BackupFileInfo ... if suffix[0] == "_": info = BackupFileInfo(backup_file.oid, base_path) backup_file.additional_files.append(info) ext = suffix.split(".", 1)[-1] # ... otherwise this is the main file else: info = backup_file ext = suffix[1:] # Infer the compression from the file extension if ext == "tar": info.compression = None elif ext == "tar.gz": info.compression = "gzip" elif ext == "tar.bz2": info.compression = "bzip2" elif ext == "tar.snappy": info.compression = "snappy" else: logging.warning("Skipping unknown extension: %s", ext) continue info.path = item logging.info( "Found file from backup '%s' of server '%s': %s", backup_info.backup_id, self.server_name, info.path, ) break for backup_file in backup_files.values(): logging_fun = logging.warning if allow_missing else logging.error if backup_file.path is None and backup_info.snapshots_info is None: logging_fun( "Missing file %s.* for server %s", backup_file.base, self.server_name, ) if not allow_missing: raise SystemExit(1) return backup_files def get_latest_archived_wals_info(self): """ Return a dictionary of timelines associated with the WalFileInfo of the last WAL file in the archive, or an empty dict if the archive doesn't contain any WAL file. :rtype: dict[str, WalFileInfo] """ if not self.get_wal_paths(): return dict() timelines = {} for name in sorted(self.get_wal_paths(), reverse=True): # Extract the timeline. If it is not valid, skip this directory try: timeline = name[0:8] int(timeline, 16) except ValueError: continue # If this timeline already has a file, skip this directory if timeline in timelines: continue timelines[timeline] = WalFileInfo(name=name) break # Return the timeline map return timelines class CloudSnapshotInterface(with_metaclass(ABCMeta)): """Defines a common interface for handling cloud snapshots.""" _required_config_for_backup = ("snapshot_disks", "snapshot_instance") _required_config_for_restore = ("snapshot_recovery_instance",) @classmethod def validate_backup_config(cls, config): """ Additional validation for backup options. Raises a ConfigurationException if any required options are missing. :param argparse.Namespace config: The backup options provided at the command line. """ missing_options = get_missing_attrs(config, cls._required_config_for_backup) if len(missing_options) > 0: raise ConfigurationException( "Incomplete options for snapshot backup - missing: %s" % ", ".join(missing_options) ) @classmethod def validate_restore_config(cls, config): """ Additional validation for restore options. Raises a ConfigurationException if any required options are missing. :param argparse.Namespace config: The backup options provided at the command line. """ missing_options = get_missing_attrs(config, cls._required_config_for_restore) if len(missing_options) > 0: raise ConfigurationException( "Incomplete options for snapshot restore - missing: %s" % ", ".join(missing_options) ) @abstractmethod def take_snapshot_backup(self, backup_info, instance_name, volumes): """ Take a snapshot backup for the named instance. Implementations of this method must do the following: * Create a snapshot of the disk. * Set the snapshots_info field of the backup_info to a SnapshotsInfo implementation which contains the snapshot metadata required both by Barman and any third party tooling which needs to recover the snapshots. :param barman.infofile.LocalBackupInfo backup_info: Backup information. :param str instance_name: The name of the VM instance to which the disks to be backed up are attached. :param dict[str,barman.cloud.VolumeMetadata] volumes: Metadata for the volumes to be backed up. """ @abstractmethod def delete_snapshot_backup(self, backup_info): """ Delete all snapshots for the supplied backup. :param barman.infofile.LocalBackupInfo backup_info: Backup information. """ @abstractmethod def get_attached_volumes(self, instance_name, disks=None, fail_on_missing=True): """ Returns metadata for the volumes attached to this instance. Queries the cloud provider for metadata relating to the volumes attached to the named instance and returns a dict of `VolumeMetadata` objects, keyed by disk name. If the optional disks parameter is supplied then this method must return metadata for the disks in the supplied list only. A SnapshotBackupException must be raised if any of the supplied disks are not found to be attached to the instance. If the optional disks parameter is supplied then this method returns metadata for the disks in the supplied list only. If fail_on_missing is set to True then a SnapshotBackupException is raised if any of the supplied disks are not found to be attached to the instance. If the disks parameter is not supplied then this method must return a VolumeMetadata for all disks attached to this instance. :param str instance_name: The name of the VM instance to which the disks to be backed up are attached. :param list[str]|None disks: A list containing the names of disks to be backed up. :param bool fail_on_missing: Fail with a SnapshotBackupException if any specified disks are not attached to the instance. :rtype: dict[str, VolumeMetadata] :return: A dict of VolumeMetadata objects representing each volume attached to the instance, keyed by volume identifier. """ @abstractmethod def instance_exists(self, instance_name): """ Determine whether the named instance exists. :param str instance_name: The name of the VM instance to which the disks to be backed up are attached. :rtype: bool :return: True if the named instance exists, False otherwise. """ class VolumeMetadata(object): """ Represents metadata for a single volume attached to a cloud VM. The main purpose of this class is to allow calling code to determine the mount point and mount options for an attached volume without needing to know the details of how these are determined for a specific cloud provider. Implementations must therefore: - Store metadata obtained from the cloud provider which can be used to resolve this volume to an attached and mounted volume on the instance. This will typically be a device name or something which can be resolved to a device name. - Provide an implementation of `resolve_mounted_volume` which executes commands on the cloud VM via a supplied UnixLocalCommand object in order to set the _mount_point and _mount_options properties. If the volume was cloned from a snapshot then the source snapshot identifier must also be stored in this class so that calling code can determine if/how/where a volume cloned from a given snapshot is mounted. """ def __init__(self): self._mount_point = None self._mount_options = None @abstractmethod def resolve_mounted_volume(self, cmd): """ Resolve the mount point and mount options using shell commands. This method must use cmd together with any additional private properties available in the provider-specific implementation in order to resolve the mount point and mount options for this volume. :param UnixLocalCommand cmd: Wrapper for local/remote commands on the instance to which this volume is attached. """ @abstractproperty def source_snapshot(self): """ The source snapshot from which this volume was cloned. :rtype: str|None :return: A snapshot identifier. """ @property def mount_point(self): """ The mount point at which this volume is currently mounted. This must be resolved using metadata obtained from the cloud provider which describes how the volume is attached to the VM. """ return self._mount_point @property def mount_options(self): """ The mount options with which this device is currently mounted. This must be resolved using metadata obtained from the cloud provider which describes how the volume is attached to the VM. """ return self._mount_options class SnapshotMetadata(object): """ Represents metadata for a single snapshot. This class holds the snapshot metadata common to all snapshot providers. Currently this is the mount_options and the mount_point of the source disk for the snapshot at the time of the backup. The `identifier` and `device` properties are part of the public interface used within Barman so that the calling code can access the snapshot identifier and device path without having to worry about how these are composed from the snapshot metadata for each cloud provider. Specializations of this class must: 1. Add their provider-specific fields to `_provider_fields`. 2. Implement the `identifier` abstract property so that it returns a value which can identify the snapshot via the cloud provider API. An example would be the snapshot short name in GCP. 3. Implement the `device` abstract property so that it returns a full device path to the location at which the source disk was attached to the compute instance. """ _provider_fields = () def __init__(self, mount_options=None, mount_point=None): """ Constructor accepts properties generic to all snapshot providers. :param str mount_options: The mount options used for the source disk at the time of the backup. :param str mount_point: The mount point of the source disk at the time of the backup. """ self.mount_options = mount_options self.mount_point = mount_point @classmethod def from_dict(cls, info): """ Create a new SnapshotMetadata object from the raw metadata dict. This function will set the generic fields supported by SnapshotMetadata before iterating through fields listed in `cls._provider_fields`. This means subclasses do not need to override this method, they just need to add their fields to their own `_provider_fields`. :param dict[str,str] info: The raw snapshot metadata. :rtype: SnapshotMetadata """ snapshot_info = cls() if "mount" in info: for field in ("mount_options", "mount_point"): try: setattr(snapshot_info, field, info["mount"][field]) except KeyError: pass for field in cls._provider_fields: try: setattr(snapshot_info, field, info["provider"][field]) except KeyError: pass return snapshot_info def to_dict(self): """ Seralize this SnapshotMetadata object as a raw dict. This function will create a dict with the generic fields supported by SnapshotMetadata before iterating through fields listed in `self._provider_fields` and adding them to a special `provider` field. As long as they add their provider-specific fields to `_provider_fields` then subclasses do not need to override this method. :rtype: dict :return: A dict containing the metadata for this snapshot. """ info = { "mount": { "mount_options": self.mount_options, "mount_point": self.mount_point, }, } if len(self._provider_fields) > 0: info["provider"] = {} for field in self._provider_fields: info["provider"][field] = getattr(self, field) return info @abstractproperty def identifier(self): """ An identifier which can reference the snapshot via the cloud provider. Subclasses must ensure this returns a string which can be used by Barman to reference the snapshot when interacting with the cloud provider API. :rtype: str :return: A snapshot identifier. """ class SnapshotsInfo(object): """ Represents the snapshots_info field of backup metadata stored in BackupInfo. This class holds the metadata for a snapshot backup which is common to all snapshot providers. This is the list of SnapshotMetadata objects representing the individual snapshots. Specializations of this class must: 1. Add their provider-specific fields to `_provider_fields`. 2. Set their `_snapshot_metadata_cls` property to the required specialization of SnapshotMetadata. 3. Set the provider property to the required value. """ _provider_fields = () _snapshot_metadata_cls = SnapshotMetadata def __init__(self, snapshots=None): """ Constructor saves the list of snapshots if it is provided. :param list[SnapshotMetadata] snapshots: A list of metadata objects for each snapshot. """ if snapshots is None: snapshots = [] self.snapshots = snapshots self.provider = None @classmethod def from_dict(cls, info): """ Create a new SnapshotsInfo object from the raw metadata dict. This function will iterate through fields listed in `cls._provider_fields` and add them to the instantiated object. It will then create a new SnapshotMetadata object (of the type specified in `cls._snapshot_metadata_cls`) for each snapshot in the raw dict. Subclasses do not need to override this method, they just need to add their fields to their own `_provider_fields` and override `_snapshot_metadata_cls`. :param dict info: The raw snapshots_info dict. :rtype: SnapshotsInfo :return: The SnapshotsInfo object representing the raw dict. """ snapshots_info = cls() for field in cls._provider_fields: try: setattr(snapshots_info, field, info["provider_info"][field]) except KeyError: pass snapshots_info.snapshots = [ cls._snapshot_metadata_cls.from_dict(snapshot_info) for snapshot_info in info["snapshots"] ] return snapshots_info def to_dict(self): """ Seralize this SnapshotMetadata object as a raw dict. This function will create a dict with the generic fields supported by SnapshotMetadata before iterating through fields listed in `self._provider_fields` and adding them to a special `provider_info` field. The SnapshotMetadata objects in `self.snapshots` are serialized into the dict via their own `to_dict` function. As long as they add their provider-specific fields to `_provider_fields` then subclasses do not need to override this method. :rtype: dict :return: A dict containing the metadata for this snapshot. """ info = {"provider": self.provider} if len(self._provider_fields) > 0: info["provider_info"] = {} for field in self._provider_fields: info["provider_info"][field] = getattr(self, field) info["snapshots"] = [ snapshot_info.to_dict() for snapshot_info in self.snapshots ] return info barman-3.13.3/barman/retention_policies.py0000644000175100001660000005515715002177425016752 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module defines backup retention policies. A backup retention policy in Barman is a user-defined policy for determining how long backups and archived logs (WAL segments) need to be retained for media recovery. You can define a retention policy in terms of backup redundancy or a recovery window. Barman retains the periodical backups required to satisfy the current retention policy, and any archived WAL files required for complete recovery of those backups. """ import logging import re from abc import ABCMeta, abstractmethod from datetime import datetime, timedelta from dateutil import tz from barman.annotations import KeepManager from barman.exceptions import InvalidRetentionPolicy from barman.infofile import BackupInfo from barman.utils import with_metaclass _logger = logging.getLogger(__name__) class RetentionPolicy(with_metaclass(ABCMeta, object)): """Abstract base class for retention policies""" def __init__(self, mode, unit, value, context, server): """Constructor of the retention policy base class""" self.mode = mode self.unit = unit self.value = int(value) self.context = context self.server = server self._first_backup = None self._first_wal = None def report(self, source=None, context=None): """Report obsolete/valid objects according to the retention policy""" if context is None: context = self.context # Overrides the list of available backups if source is None: source = self.server.available_backups if context == "BASE": return self._backup_report(source) elif context == "WAL": return self._wal_report() else: raise ValueError("Invalid context %s", context) def backup_status(self, backup_id): """Report the status of a backup according to the retention policy""" source = self.server.available_backups if self.context == "BASE": return self._backup_report(source)[backup_id] else: return BackupInfo.NONE def first_backup(self): """Returns the first valid backup according to retention policies""" if not self._first_backup: self.report(context="BASE") return self._first_backup def first_wal(self): """Returns the first valid WAL according to retention policies""" if not self._first_wal: self.report(context="WAL") return self._first_wal @abstractmethod def __str__(self): """String representation""" pass @abstractmethod def debug(self): """Debug information""" pass @abstractmethod def _backup_report(self, source): """Report obsolete/valid backups according to the retention policy""" pass @abstractmethod def _wal_report(self): """Report obsolete/valid WALs according to the retention policy""" pass @classmethod def create(cls, server, option, value): """ If given option and value from the configuration file match, creates the retention policy object for the given server """ # using @abstractclassmethod from python3 would be better here raise NotImplementedError( "The class %s must override the create() class method", cls.__name__ ) def to_json(self): """ Output representation of the obj for JSON serialization """ return self.__str__() def _propagate_retention_status_to_children(self, backup_info, report, ret_status): """ Propagate retention status to all backups in the tree. .. note:: This has a side-effect. It modifies or add data to *report* dict. :param barman.infofile.BackupInfo backup_info: The object we want to propagate the RETENTION STATUS from. :param dict[str, str] report: The report data structure to be modified. Each key is the ID of a backup, and its value is the retention status of that backup. :param str ret_status: The status of the backup according to retention policies """ backup_tree = backup_info.walk_backups_tree(return_self=False) for backup in backup_tree: report[backup.backup_id] = ret_status _logger.debug( "Propagating %s retention status of backup %s to %s." % (ret_status, backup_info.backup_id, backup.backup_id) ) class RedundancyRetentionPolicy(RetentionPolicy): """ Retention policy based on redundancy, the setting that determines many periodical backups to keep. A redundancy-based retention policy is contrasted with retention policy that uses a recovery window. """ _re = re.compile(r"^\s*redundancy\s+(\d+)\s*$", re.IGNORECASE) def __init__(self, context, value, server): super(RedundancyRetentionPolicy, self).__init__( "redundancy", "b", value, "BASE", server ) assert value >= 0 def __str__(self): return "REDUNDANCY %s" % self.value def debug(self): return "Redundancy: %s (%s)" % (self.value, self.context) def _backup_report(self, source): """Report obsolete/valid backups according to the retention policy""" report = dict() backups = source # Normalise the redundancy value (according to minimum redundancy) redundancy = self.value if redundancy < self.server.minimum_redundancy: _logger.warning( "Retention policy redundancy (%s) is lower than " "the required minimum redundancy (%s). Enforce %s.", redundancy, self.server.minimum_redundancy, self.server.minimum_redundancy, ) redundancy = self.server.minimum_redundancy # Map the latest 'redundancy' DONE backups as VALID # The remaining DONE backups are classified as OBSOLETE # Non DONE backups are classified as NONE # NOTE: reverse key orders (simulate reverse chronology) i = 0 for bid in sorted(backups.keys(), reverse=True): if backups[bid].is_incremental: _logger.debug( "Ignoring incremental backup %s. The retention status will" " be propagated from %s." % (backups[bid], backups[bid].parent_backup_id) ) continue if backups[bid].status == BackupInfo.DONE: keep_target = self.server.get_keep_target(bid) if keep_target == KeepManager.TARGET_STANDALONE: report[bid] = BackupInfo.KEEP_STANDALONE elif keep_target: # Any other recovery target is treated as KEEP_FULL for safety report[bid] = BackupInfo.KEEP_FULL elif i < redundancy: report[bid] = BackupInfo.VALID self._first_backup = bid else: report[bid] = BackupInfo.OBSOLETE i = i + 1 else: report[bid] = BackupInfo.NONE if backups[bid].has_children: status = report[bid] # If the root backup retention status is KEEP:STANDALONE and the backup # is still VALID for retention policy, the incremental backups will have # the VALID retention status. But if this backup falls outside the # retention policy, it will be kept but the incremental backups will get # the status OBSOLETE. if status == BackupInfo.KEEP_STANDALONE: status = BackupInfo.VALID if i > redundancy: status = BackupInfo.OBSOLETE # If the root backup retention status is KEEP:FULL, the incremental # backups will have the VALID retention status. elif status == BackupInfo.KEEP_FULL: status = BackupInfo.VALID self._propagate_retention_status_to_children( backup_info=backups[bid], report=report, ret_status=status, ) return report def _wal_report(self): """Report obsolete/valid WALs according to the retention policy""" pass @classmethod def create(cls, server, context, optval): # Detect Redundancy retention type mtch = cls._re.match(optval) if not mtch: return None value = int(mtch.groups()[0]) return cls(context, value, server) class RecoveryWindowRetentionPolicy(RetentionPolicy): """ Retention policy based on recovery window. The DBA specifies a period of time and Barman ensures retention of backups and archived WAL files required for point-in-time recovery to any time during the recovery window. The interval always ends with the current time and extends back in time for the number of days specified by the user. For example, if the retention policy is set for a recovery window of seven days, and the current time is 9:30 AM on Friday, Barman retains the backups required to allow point-in-time recovery back to 9:30 AM on the previous Friday. """ _re = re.compile( r""" ^\s* recovery\s+window\s+of\s+ # recovery window of (\d+)\s+(day|month|week)s? # N (day|month|week) with optional 's' \s*$ """, re.IGNORECASE | re.VERBOSE, ) _kw = {"d": "DAYS", "m": "MONTHS", "w": "WEEKS"} def __init__(self, context, value, unit, server): super(RecoveryWindowRetentionPolicy, self).__init__( "window", unit, value, context, server ) assert value >= 0 assert unit == "d" or unit == "m" or unit == "w" assert context == "WAL" or context == "BASE" # Calculates the time delta if unit == "d": self.timedelta = timedelta(days=self.value) elif unit == "w": self.timedelta = timedelta(weeks=self.value) elif unit == "m": self.timedelta = timedelta(days=(31 * self.value)) def __str__(self): return "RECOVERY WINDOW OF %s %s" % (self.value, self._kw[self.unit]) def debug(self): return "Recovery Window: %s %s: %s (%s)" % ( self.value, self.unit, self.context, self._point_of_recoverability(), ) def _point_of_recoverability(self): """ Based on the current time and the window, calculate the point of recoverability, which will be then used to define the first backup or the first WAL """ return datetime.now(tz.tzlocal()) - self.timedelta def _backup_report(self, source): """Report obsolete/valid backups according to the retention policy""" report = dict() backups = source # Map as VALID all DONE backups having end time lower than # the point of recoverability. The older ones # are classified as OBSOLETE. # Non DONE backups are classified as NONE found = False valid = 0 # NOTE: reverse key orders (simulate reverse chronology) for bid in sorted(backups.keys(), reverse=True): if backups[bid].is_incremental: _logger.debug( "Ignoring incremental backup %s. The retention status will" " be propagated from %s." % (backups[bid], backups[bid].parent_backup_id) ) continue # We are interested in DONE backups only if backups[bid].status == BackupInfo.DONE: keep_target = self.server.get_keep_target(bid) if keep_target == KeepManager.TARGET_STANDALONE: keep_target = BackupInfo.KEEP_STANDALONE elif keep_target: # Any other recovery target is treated as KEEP_FULL for safety keep_target = BackupInfo.KEEP_FULL # By found, we mean "found the first backup outside the recovery # window" if that is the case then this bid is potentially obsolete. if found: # Check minimum redundancy requirements if valid < self.server.minimum_redundancy: if keep_target: _logger.info( "Keeping obsolete backup %s for server %s " "(older than %s) " "due to keep status: %s", bid, self.server.name, self._point_of_recoverability, keep_target, ) report[bid] = keep_target else: _logger.warning( "Keeping obsolete backup %s for server %s " "(older than %s) " "due to minimum redundancy requirements (%s)", bid, self.server.name, self._point_of_recoverability(), self.server.minimum_redundancy, ) # We mark the backup as potentially obsolete # as we must respect minimum redundancy requirements report[bid] = BackupInfo.POTENTIALLY_OBSOLETE self._first_backup = bid valid = valid + 1 else: if keep_target: _logger.info( "Keeping obsolete backup %s for server %s " "(older than %s) " "due to keep status: %s", bid, self.server.name, self._point_of_recoverability, keep_target, ) report[bid] = keep_target else: # We mark this backup as obsolete # (older than the first valid one) _logger.info( "Reporting backup %s for server %s as OBSOLETE " "(older than %s)", bid, self.server.name, self._point_of_recoverability(), ) report[bid] = BackupInfo.OBSOLETE else: _logger.debug( "Reporting backup %s for server %s as VALID (newer than %s)", bid, self.server.name, self._point_of_recoverability(), ) # Backup within the recovery window report[bid] = keep_target or BackupInfo.VALID self._first_backup = bid valid = valid + 1 # TODO: Currently we use the backup local end time # We need to make this more accurate if backups[bid].end_time < self._point_of_recoverability(): found = True else: report[bid] = BackupInfo.NONE if backups[bid].has_children: status = report[bid] # If the root backup retention status is KEEP:STANDALONE and the backup # is still VALID for retention policy, the incremental backups will have # the VALID retention status. But if this backup falls outside the # retention policy, it will be kept but the incremental backups will get # the status OBSOLETE. if status == BackupInfo.KEEP_STANDALONE: status = BackupInfo.VALID if found: status = BackupInfo.OBSOLETE # If the root backup retention status is KEEP:FULL, the incremental # backups will have the VALID retention status. elif status == BackupInfo.KEEP_FULL: status = BackupInfo.VALID self._propagate_retention_status_to_children( backup_info=backups[bid], report=report, ret_status=status, ) return report def _wal_report(self): """Report obsolete/valid WALs according to the retention policy""" pass @classmethod def create(cls, server, context, optval): # Detect Recovery Window retention type match = cls._re.match(optval) if not match: return None value = int(match.groups()[0]) unit = match.groups()[1][0].lower() return cls(context, value, unit, server) class SimpleWALRetentionPolicy(RetentionPolicy): """Simple retention policy for WAL files (identical to the main one)""" _re = re.compile(r"^\s*main\s*$", re.IGNORECASE) def __init__(self, context, policy, server): super(SimpleWALRetentionPolicy, self).__init__( "simple-wal", policy.unit, policy.value, context, server ) # The referred policy must be of type 'BASE' assert self.context == "WAL" and policy.context == "BASE" self.policy = policy def __str__(self): return "MAIN" def debug(self): return "Simple WAL Retention Policy (%s)" % self.policy def _backup_report(self, source): """Report obsolete/valid backups according to the retention policy""" pass def _wal_report(self): """Report obsolete/valid backups according to the retention policy""" self.policy.report(context="WAL") def first_wal(self): """Returns the first valid WAL according to retention policies""" return self.policy.first_wal() @classmethod def create(cls, server, context, optval): # Detect Redundancy retention type match = cls._re.match(optval) if not match: return None return cls(context, server.retention_policy, server) class ServerMetadata(object): """ Static retention metadata for a barman-managed server This will return the same values regardless of any changes in the state of the barman-managed server and associated backups. """ def __init__(self, server_name, backup_info_list, keep_manager, minimum_redundancy): self.name = server_name self.minimum_redundancy = minimum_redundancy self.retention_policy = None self.backup_info_list = backup_info_list self.keep_manager = keep_manager @property def available_backups(self): return self.backup_info_list def get_keep_target(self, backup_id): return self.keep_manager.get_keep_target(backup_id) class ServerMetadataLive(ServerMetadata): """ Live retention metadata for a barman-managed server This will always return the current values for the barman.Server passed in at construction time. """ def __init__(self, server, keep_manager): self.server = server self.keep_manager = keep_manager @property def name(self): return self.server.config.name @property def minimum_redundancy(self): return self.server.config.minimum_redundancy @property def retention_policy(self): return self.server.config.retention_policy @property def available_backups(self): return self.server.get_available_backups(BackupInfo.STATUS_NOT_EMPTY) def get_keep_target(self, backup_id): return self.keep_manager.get_keep_target(backup_id) class RetentionPolicyFactory(object): """Factory for retention policy objects""" # Available retention policy types policy_classes = [ RedundancyRetentionPolicy, RecoveryWindowRetentionPolicy, SimpleWALRetentionPolicy, ] @classmethod def create( cls, option, value, server=None, server_name=None, catalog=None, minimum_redundancy=0, ): """ Based on the given option and value from the configuration file, creates the appropriate retention policy object for the given server Either server *or* server_name and backup_info_list must be provided. If server (a `barman.Server`) is provided then the returned RetentionPolicy will update as the state of the `barman.Server` changes. If server_name and backup_info_list are provided then the RetentionPolicy will be a snapshot based on the backup_info_list passed at construction time. """ if option == "wal_retention_policy": context = "WAL" elif option == "retention_policy": context = "BASE" else: raise InvalidRetentionPolicy( "Unknown option for retention policy: %s" % option ) if server: server_metadata = ServerMetadataLive( server, keep_manager=server.backup_manager ) else: server_metadata = ServerMetadata( server_name, catalog.get_backup_list(), keep_manager=catalog, minimum_redundancy=minimum_redundancy, ) # Look for the matching rule for policy_class in cls.policy_classes: policy = policy_class.create(server_metadata, context, value) if policy: return policy raise InvalidRetentionPolicy("Cannot parse option %s: %s" % (option, value)) barman-3.13.3/barman/copy_controller.py0000644000175100001660000014076515002177425016271 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ Copy controller module A copy controller will handle the copy between a series of files and directory, and their final destination. """ import collections import datetime import logging import os.path import re import shutil import signal import tempfile import time from functools import partial from multiprocessing import Lock, Pool import dateutil.tz from barman.command_wrappers import RsyncPgData from barman.exceptions import CommandFailedException, RsyncListFilesFailure from barman.utils import human_readable_timedelta, total_seconds _logger = logging.getLogger(__name__) _worker_callable = None """ Global variable containing a callable used to execute the jobs. Initialized by `_init_worker` and used by `_run_worker` function. This variable must be None outside a multiprocessing worker Process. """ # Parallel copy bucket size (10GB) BUCKET_SIZE = 1024 * 1024 * 1024 * 10 def _init_worker(func): """ Store the callable used to execute jobs passed to `_run_worker` function :param callable func: the callable to invoke for every job """ global _worker_callable _worker_callable = func def _run_worker(job): """ Execute a job using the callable set using `_init_worker` function :param _RsyncJob job: the job to be executed """ global _worker_callable assert ( _worker_callable is not None ), "Worker has not been initialized with `_init_worker`" # This is the entrypoint of the worker process. Since the KeyboardInterrupt # exceptions is handled by the main process, let's forget about Ctrl-C # here. # When the parent process will receive a KeyboardInterrupt, it will ask # the pool to terminate its workers and then terminate itself. signal.signal(signal.SIGINT, signal.SIG_IGN) return _worker_callable(job) class _RsyncJob(object): """ A job to be executed by a worker Process """ def __init__(self, item_idx, description, id=None, file_list=None, checksum=None): """ :param int item_idx: The index of copy item containing this job :param str description: The description of the job, used for logging :param int id: Job ID (as in bucket) :param list[RsyncCopyController._FileItem] file_list: Path to the file containing the file list :param bool checksum: Whether to force the checksum verification """ self.id = id self.item_idx = item_idx self.description = description self.file_list = file_list self.checksum = checksum # Statistics self.copy_start_time = None self.copy_end_time = None class _FileItem(collections.namedtuple("_FileItem", "mode size date path")): """ This named tuple is used to store the content each line of the output of a "rsync --list-only" call """ class _RsyncCopyItem(object): """ Internal data object that contains the information about one of the items that have to be copied during a RsyncCopyController run. """ def __init__( self, label, src, dst, exclude=None, exclude_and_protect=None, include=None, is_directory=False, bwlimit=None, reuse=None, item_class=None, optional=False, ): """ The "label" parameter is meant to be used for error messages and logging. If "src" or "dst" content begin with a ':' character, it is a remote path. Only local paths are supported in "reuse" argument. If "reuse" parameter is provided and is not None, it is used to implement the incremental copy. This only works if "is_directory" is True :param str label: a symbolic name for this item :param str src: source directory. :param str dst: destination directory. :param list[str] exclude: list of patterns to be excluded from the copy. The destination will be deleted if present. :param list[str] exclude_and_protect: list of patterns to be excluded from the copy. The destination will be preserved if present. :param list[str] include: list of patterns to be included in the copy even if excluded. :param bool is_directory: Whether the item points to a directory. :param bwlimit: bandwidth limit to be enforced. (KiB) :param str|None reuse: the reference path for incremental mode. :param str|None item_class: If specified carries a meta information about what the object to be copied is. :param bool optional: Whether a failure copying this object should be treated as a fatal failure. This only works if "is_directory" is False """ self.label = label self.src = src self.dst = dst self.exclude = exclude self.exclude_and_protect = exclude_and_protect self.include = include self.is_directory = is_directory self.bwlimit = bwlimit self.reuse = reuse self.item_class = item_class self.optional = optional # Attributes that will e filled during the analysis self.temp_dir = None self.dir_file = None self.exclude_and_protect_file = None self.safe_list = None self.check_list = None # Statistics self.analysis_start_time = None self.analysis_end_time = None # Ensure that the user specified the item class, since it is mandatory # to correctly handle the item assert self.item_class def __str__(self): # Prepare strings for messages formatted_class = self.item_class formatted_name = self.src if self.src.startswith(":"): formatted_class = "remote " + self.item_class formatted_name = self.src[1:] formatted_class += " directory" if self.is_directory else " file" # Log the operation that is being executed if self.item_class in ( RsyncCopyController.PGDATA_CLASS, RsyncCopyController.PGCONTROL_CLASS, ): return "%s: %s" % (formatted_class, formatted_name) else: return "%s '%s': %s" % (formatted_class, self.label, formatted_name) class RsyncCopyController(object): """ Copy a list of files and directory to their final destination. """ # Constants to be used as "item_class" values PGDATA_CLASS = "PGDATA" TABLESPACE_CLASS = "tablespace" PGCONTROL_CLASS = "pg_control" CONFIG_CLASS = "config" # This regular expression is used to parse each line of the output # of a "rsync --list-only" call. This regexp has been tested with any known # version of upstream rsync that is supported (>= 3.0.4) LIST_ONLY_RE = re.compile( r""" ^ # start of the line # capture the mode (es. "-rw-------") (?P[-\w]+) \s+ # size is an integer (?P\d+) \s+ # The date field can have two different form (?P # "2014/06/05 18:00:00" if the sending rsync is compiled # with HAVE_STRFTIME [\d/]+\s+[\d:]+ | # "Thu Jun 5 18:00:00 2014" otherwise \w+\s+\w+\s+\d+\s+[\d:]+\s+\d+ ) \s+ # all the remaining characters are part of filename (?P.+) $ # end of the line """, re.VERBOSE, ) # This regular expression is used to ignore error messages regarding # vanished files that are not really an error. It is used because # in some cases rsync reports it with exit code 23 which could also mean # a fatal error VANISHED_RE = re.compile( r""" ^ # start of the line ( # files which vanished before rsync start rsync:\ link_stat\ ".+"\ failed:\ No\ such\ file\ or\ directory\ \(2\) | # files which vanished after rsync start file\ has\ vanished:\ ".+" | # files which have been truncated during transfer rsync:\ read\ errors\ mapping\ ".+":\ No\ data\ available\ \(61\) | # final summary rsync\ error:\ .* \(code\ 23\)\ at\ main\.c\(\d+\) \ \[(generator|receiver|sender)=[^\]]+\] ) $ # end of the line """, re.VERBOSE + re.IGNORECASE, ) def __init__( self, path=None, ssh_command=None, ssh_options=None, network_compression=False, reuse_backup=None, safe_horizon=None, exclude=None, retry_times=0, retry_sleep=0, workers=1, workers_start_batch_period=1, workers_start_batch_size=10, ): """ :param str|None path: the PATH where rsync executable will be searched :param str|None ssh_command: the ssh executable to be used to access remote paths :param list[str]|None ssh_options: list of ssh options to be used to access remote paths :param boolean network_compression: whether to use the network compression :param str|None reuse_backup: if "link" or "copy" enables the incremental copy feature :param datetime.datetime|None safe_horizon: if set, assumes that every files older than it are save to copy without checksum verification. :param list[str]|None exclude: list of patterns to be excluded from the copy :param int retry_times: The number of times to retry a failed operation :param int retry_sleep: Sleep time between two retry :param int workers: The number of parallel copy workers :param int workers_start_batch_period: The time period in seconds over which a single batch of workers will be started :param int workers_start_batch_size: The maximum number of parallel workers to start in a single batch """ super(RsyncCopyController, self).__init__() self.path = path self.ssh_command = ssh_command self.ssh_options = ssh_options self.network_compression = network_compression self.reuse_backup = reuse_backup self.safe_horizon = safe_horizon self.exclude = exclude self.retry_times = retry_times self.retry_sleep = retry_sleep self.workers = workers self.workers_start_batch_period = workers_start_batch_period self.workers_start_batch_size = workers_start_batch_size self._logger_lock = Lock() # Assume we are running with a recent rsync (>= 3.1) self.rsync_has_ignore_missing_args = True self.item_list = [] """List of items to be copied""" self.rsync_cache = {} """A cache of RsyncPgData objects""" # Attributes used for progress reporting self.total_steps = None """Total number of steps""" self.current_step = None """Current step number""" self.temp_dir = None """Temp dir used to store the status during the copy""" # Statistics self.jobs_done = None """Already finished jobs list""" self.copy_start_time = None """Copy start time""" self.copy_end_time = None """Copy end time""" def add_directory( self, label, src, dst, exclude=None, exclude_and_protect=None, include=None, bwlimit=None, reuse=None, item_class=None, ): """ Add a directory that we want to copy. If "src" or "dst" content begin with a ':' character, it is a remote path. Only local paths are supported in "reuse" argument. If "reuse" parameter is provided and is not None, it is used to implement the incremental copy. This only works if "is_directory" is True :param str label: symbolic name to be used for error messages and logging. :param str src: source directory. :param str dst: destination directory. :param list[str] exclude: list of patterns to be excluded from the copy. The destination will be deleted if present. :param list[str] exclude_and_protect: list of patterns to be excluded from the copy. The destination will be preserved if present. :param list[str] include: list of patterns to be included in the copy even if excluded. :param bwlimit: bandwidth limit to be enforced. (KiB) :param str|None reuse: the reference path for incremental mode. :param str item_class: If specified carries a meta information about what the object to be copied is. """ self.item_list.append( _RsyncCopyItem( label=label, src=src, dst=dst, is_directory=True, bwlimit=bwlimit, reuse=reuse, item_class=item_class, optional=False, exclude=exclude, exclude_and_protect=exclude_and_protect, include=include, ) ) def add_file(self, label, src, dst, item_class=None, optional=False, bwlimit=None): """ Add a file that we want to copy :param str label: symbolic name to be used for error messages and logging. :param str src: source directory. :param str dst: destination directory. :param str item_class: If specified carries a meta information about what the object to be copied is. :param bool optional: Whether a failure copying this object should be treated as a fatal failure. :param bwlimit: bandwidth limit to be enforced. (KiB) """ self.item_list.append( _RsyncCopyItem( label=label, src=src, dst=dst, is_directory=False, bwlimit=bwlimit, reuse=None, item_class=item_class, optional=optional, ) ) def _rsync_factory(self, item): """ Build the RsyncPgData object required for copying the provided item :param _RsyncCopyItem item: information about a copy operation :rtype: RsyncPgData """ # If the object already exists, use it if item in self.rsync_cache: return self.rsync_cache[item] # Prepare the command arguments args = self._reuse_args(item.reuse) # Merge the global exclude with the one into the item object if self.exclude and item.exclude: exclude = self.exclude + item.exclude else: exclude = self.exclude or item.exclude # Using `--ignore-missing-args` could fail in case # the local or the remote rsync is older than 3.1. # In that case we expect that during the analyze phase # we get an error. The analyze code must catch that error # and retry after flushing the rsync cache. if self.rsync_has_ignore_missing_args: args.append("--ignore-missing-args") # TODO: remove debug output or use it to progress tracking # By adding a double '--itemize-changes' option, the rsync # output will contain the full list of files that have been # touched, even those that have not changed args.append("--itemize-changes") args.append("--itemize-changes") # Build the rsync object that will execute the copy rsync = RsyncPgData( path=self.path, ssh=self.ssh_command, ssh_options=self.ssh_options, args=args, bwlimit=item.bwlimit, network_compression=self.network_compression, exclude=exclude, exclude_and_protect=item.exclude_and_protect, include=item.include, retry_times=self.retry_times, retry_sleep=self.retry_sleep, retry_handler=partial(self._retry_handler, item), ) self.rsync_cache[item] = rsync return rsync def _rsync_set_pre_31_mode(self): """ Stop using `--ignore-missing-args` and restore rsync < 3.1 compatibility """ _logger.info( "Detected rsync version less than 3.1. " "Stopping use of '--ignore-missing-args' argument." ) self.rsync_has_ignore_missing_args = False self.rsync_cache.clear() def copy(self): """ Execute the actual copy """ # Store the start time self.copy_start_time = datetime.datetime.now() # Create a temporary directory to hold the file lists. self.temp_dir = tempfile.mkdtemp(suffix="", prefix="barman-") # The following try block is to make sure the temporary directory # will be removed on exit and all the pool workers # have been terminated. pool = None try: # Initialize the counters used by progress reporting self._progress_init() _logger.info("Copy started (safe before %r)", self.safe_horizon) # Execute some preliminary steps for each item to be copied for item in self.item_list: # The initial preparation is necessary only for directories if not item.is_directory: continue # Store the analysis start time item.analysis_start_time = datetime.datetime.now() # Analyze the source and destination directory content _logger.info(self._progress_message("[global] analyze %s" % item)) self._analyze_directory(item) # Prepare the target directories, removing any unneeded file _logger.info( self._progress_message( "[global] create destination directories and delete " "unknown files for %s" % item ) ) self._create_dir_and_purge(item) # Store the analysis end time item.analysis_end_time = datetime.datetime.now() # Init the list of jobs done. Every job will be added to this list # once finished. The content will be used to calculate statistics # about the copy process. self.jobs_done = [] # The jobs are executed using a parallel processes pool # Each job is generated by `self._job_generator`, it is executed by # `_run_worker` using `self._execute_job`, which has been set # calling `_init_worker` function during the Pool initialization. pool = Pool( processes=self.workers, initializer=_init_worker, initargs=(self._execute_job,), ) for job in pool.imap_unordered( _run_worker, self._job_generator(exclude_classes=[self.PGCONTROL_CLASS]) ): # Store the finished job for further analysis self.jobs_done.append(job) # The PGCONTROL_CLASS items must always be copied last for job in pool.imap_unordered( _run_worker, self._job_generator(include_classes=[self.PGCONTROL_CLASS]) ): # Store the finished job for further analysis self.jobs_done.append(job) except KeyboardInterrupt: _logger.info( "Copy interrupted by the user (safe before %s)", self.safe_horizon ) raise except BaseException: _logger.info("Copy failed (safe before %s)", self.safe_horizon) raise else: _logger.info("Copy finished (safe before %s)", self.safe_horizon) finally: # The parent process may have finished naturally or have been # interrupted by an exception (i.e. due to a copy error or # the user pressing Ctrl-C). # At this point we must make sure that all the workers have been # correctly terminated before continuing. if pool: pool.terminate() pool.join() # Clean up the temp dir, any exception raised here is logged # and discarded to not clobber an eventual exception being handled. try: shutil.rmtree(self.temp_dir) except EnvironmentError as e: _logger.error("Error cleaning up '%s' (%s)", self.temp_dir, e) self.temp_dir = None # Store the end time self.copy_end_time = datetime.datetime.now() def _apply_rate_limit(self, generation_history): """ Apply the rate limit defined by `self.workers_start_batch_size` and `self.workers_start_batch_period`. Historic start times in `generation_history` are checked to determine whether more than `self.workers_start_batch_size` jobs have been started within the length of time defined by `self.workers_start_batch_period`. If the maximum has been reached then this function will wait until the oldest start time within the last `workers_start_batch_period` seconds is no longer within the time period. Once it has finished waiting, or simply determined it does not need to wait, it adds the current time to `generation_history` and returns it. :param list[int] generation_history: A list of the generation times of previous jobs. :return list[int]: An updated list of generation times including the current time (after completing any necessary waiting) and not including any times which were not within `self.workers_start_batch_period` when the function was called. """ # Job generation timestamps from before the start of the batch period are # removed from the history because they no longer affect the generation of new # jobs now = time.time() window_start_time = now - self.workers_start_batch_period new_history = [ timestamp for timestamp in generation_history if timestamp > window_start_time ] # If the number of jobs generated within the batch period is at capacity then we # wait until the oldest job is outside the batch period if len(new_history) >= self.workers_start_batch_size: wait_time = new_history[0] - window_start_time _logger.info( "%s jobs were started in the last %ss, waiting %ss" % (len(new_history), self.workers_start_batch_period, wait_time) ) time.sleep(wait_time) # Add the *current* time to the job generation history because this will be # newer than `now` if we had to wait new_history.append(time.time()) return new_history def _job_generator(self, include_classes=None, exclude_classes=None): """ Generate the jobs to be executed by the workers :param list[str]|None include_classes: If not none, copy only the items which have one of the specified classes. :param list[str]|None exclude_classes: If not none, skip all items which have one of the specified classes. :rtype: iter[_RsyncJob] """ # The generation time of each job is stored in a list so that we can limit the # rate at which jobs are generated. generation_history = [] for item_idx, item in enumerate(self.item_list): # Skip items of classes which are not required if include_classes and item.item_class not in include_classes: continue if exclude_classes and item.item_class in exclude_classes: continue # If the item is a directory then copy it in two stages, # otherwise copy it using a plain rsync if item.is_directory: # Copy the safe files using the default rsync algorithm msg = self._progress_message("[%%s] %%s copy safe files from %s" % item) phase_skipped = True for i, bucket in enumerate(self._fill_buckets(item.safe_list)): phase_skipped = False generation_history = self._apply_rate_limit(generation_history) yield _RsyncJob( item_idx, id=i, description=msg, file_list=bucket, checksum=False, ) if phase_skipped: _logger.info(msg, "global", "skipping") # Copy the check files forcing rsync to verify the checksum msg = self._progress_message( "[%%s] %%s copy files with checksum from %s" % item ) phase_skipped = True for i, bucket in enumerate(self._fill_buckets(item.check_list)): phase_skipped = False generation_history = self._apply_rate_limit(generation_history) yield _RsyncJob( item_idx, id=i, description=msg, file_list=bucket, checksum=True ) if phase_skipped: _logger.info(msg, "global", "skipping") else: # Copy the file using plain rsync msg = self._progress_message("[%%s] %%s copy %s" % item) generation_history = self._apply_rate_limit(generation_history) yield _RsyncJob(item_idx, description=msg) def _fill_buckets(self, file_list): """ Generate buckets for parallel copy :param list[_FileItem] file_list: list of file to transfer :rtype: iter[list[_FileItem]] """ # If there is only one worker, fall back to copying all file at once if self.workers < 2: yield file_list return # Create `self.workers` buckets buckets = [[] for _ in range(self.workers)] bucket_sizes = [0 for _ in range(self.workers)] pos = -1 # Sort the list by size for entry in sorted(file_list, key=lambda item: item.size): # Try to fill the file in a bucket for i in range(self.workers): pos = (pos + 1) % self.workers new_size = bucket_sizes[pos] + entry.size if new_size < BUCKET_SIZE: bucket_sizes[pos] = new_size buckets[pos].append(entry) break else: # All the buckets are filled, so return them all for i in range(self.workers): if len(buckets[i]) > 0: yield buckets[i] # Clear the bucket buckets[i] = [] bucket_sizes[i] = 0 # Put the current file in the first bucket bucket_sizes[0] = entry.size buckets[0].append(entry) pos = 0 # Send all the remaining buckets for i in range(self.workers): if len(buckets[i]) > 0: yield buckets[i] def _execute_job(self, job): """ Execute a `_RsyncJob` in a worker process :type job: _RsyncJob """ item = self.item_list[job.item_idx] if job.id is not None: bucket = "bucket %s" % job.id else: bucket = "global" # Build the rsync object required for the copy rsync = self._rsync_factory(item) # Store the start time job.copy_start_time = datetime.datetime.now() # Write in the log that the job is starting with self._logger_lock: _logger.info(job.description, bucket, "starting") if item.is_directory: # A directory item must always have checksum and file_list set assert ( job.file_list is not None ), "A directory item must not have a None `file_list` attribute" assert ( job.checksum is not None ), "A directory item must not have a None `checksum` attribute" # Generate a unique name for the file containing the list of files file_list_path = os.path.join( self.temp_dir, "%s_%s_%s.list" % (item.label, "check" if job.checksum else "safe", os.getpid()), ) # Write the list, one path per line with open(file_list_path, "w") as file_list: for entry in job.file_list: assert isinstance(entry, _FileItem), ( "expect %r to be a _FileItem" % entry ) file_list.write(entry.path + "\n") self._copy( rsync, item.src, item.dst, file_list=file_list_path, checksum=job.checksum, ) else: # A file must never have checksum and file_list set assert ( job.file_list is None ), "A file item must have a None `file_list` attribute" assert ( job.checksum is None ), "A file item must have a None `checksum` attribute" rsync(item.src, item.dst, allowed_retval=(0, 23, 24)) if rsync.ret == 23: if item.optional: _logger.warning("Ignoring error reading %s", item) else: raise CommandFailedException( dict(ret=rsync.ret, out=rsync.out, err=rsync.err) ) # Store the stop time job.copy_end_time = datetime.datetime.now() # Write in the log that the job is finished with self._logger_lock: _logger.info( job.description, bucket, "finished (duration: %s)" % human_readable_timedelta(job.copy_end_time - job.copy_start_time), ) # Return the job to the caller, for statistics purpose return job def _progress_init(self): """ Init counters used by progress logging """ self.total_steps = 0 for item in self.item_list: # Directories require 4 steps, files only one if item.is_directory: self.total_steps += 4 else: self.total_steps += 1 self.current_step = 0 def _progress_message(self, msg): """ Log a message containing the progress :param str msg: the message :return srt: message to log """ self.current_step += 1 return "Copy step %s of %s: %s" % (self.current_step, self.total_steps, msg) def _reuse_args(self, reuse_directory): """ If reuse_backup is 'copy' or 'link', build the rsync option to enable the reuse, otherwise returns an empty list :param str reuse_directory: the local path with data to be reused :rtype: list[str] """ if self.reuse_backup in ("copy", "link") and reuse_directory is not None: return ["--%s-dest=%s" % (self.reuse_backup, reuse_directory)] else: return [] def _retry_handler(self, item, command, args, kwargs, attempt, exc): """ :param _RsyncCopyItem item: The item that is being processed :param RsyncPgData command: Command object being executed :param list args: command args :param dict kwargs: command kwargs :param int attempt: attempt number (starting from 0) :param CommandFailedException exc: the exception which caused the failure """ _logger.warn("Failure executing rsync on %s (attempt %s)", item, attempt) _logger.warn("Retrying in %s seconds", self.retry_sleep) def _analyze_directory(self, item): """ Analyzes the status of source and destination directories identifying the files that are safe from the point of view of a PostgreSQL backup. The safe_horizon value is the timestamp of the beginning of the older backup involved in copy (as source or destination). Any files updated after that timestamp, must be checked as they could have been modified during the backup - and we do not reply WAL files to update them. The destination directory must exist. If the "safe_horizon" parameter is None, we cannot make any assumptions about what can be considered "safe", so we must check everything with checksums enabled. If "ref" parameter is provided and is not None, it is looked up instead of the "dst" dir. This is useful when we are copying files using '--link-dest' and '--copy-dest' rsync options. In this case, both the "dst" and "ref" dir must exist and the "dst" dir must be empty. If source or destination path begin with a ':' character, it is a remote path. Only local paths are supported in "ref" argument. :param _RsyncCopyItem item: information about a copy operation """ # If reference is not set we use dst as reference path ref = item.reuse if ref is None: ref = item.dst # Make sure the ref path ends with a '/' or rsync will add the # last path component to all the returned items during listing if ref[-1] != "/": ref += "/" # Build a hash containing all files present on reference directory. # Directories are not included try: ref_hash = {} ref_has_content = False for file_item in self._list_files(item, ref): if file_item.path != "." and not ( item.label == "pgdata" and file_item.path == "pg_tblspc" ): ref_has_content = True if file_item.mode[0] != "d": ref_hash[file_item.path] = file_item except (CommandFailedException, RsyncListFilesFailure) as e: # Here we set ref_hash to None, thus disable the code that marks as # "safe matching" those destination files with different time or # size, even if newer than "safe_horizon". As a result, all files # newer than "safe_horizon" will be checked through checksums. ref_hash = None _logger.error( "Unable to retrieve reference directory file list. " "Using only source file information to decide which files" " need to be copied with checksums enabled: %s" % e ) # The 'dir.list' file will contain every directory in the # source tree item.dir_file = os.path.join(self.temp_dir, "%s_dir.list" % item.label) dir_list = open(item.dir_file, "w+") # The 'protect.list' file will contain a filter rule to protect # each file present in the source tree. It will be used during # the first phase to delete all the extra files on destination. item.exclude_and_protect_file = os.path.join( self.temp_dir, "%s_exclude_and_protect.filter" % item.label ) exclude_and_protect_filter = open(item.exclude_and_protect_file, "w+") if not ref_has_content: # If the destination directory is empty then include all # directories and exclude all files. This stops the rsync # command which runs during the _create_dir_and_purge function # from copying the entire contents of the source directory and # ensures it only creates the directories. exclude_and_protect_filter.write("+ */\n") exclude_and_protect_filter.write("- *\n") # The `safe_list` will contain all items older than # safe_horizon, as well as files that we know rsync will # check anyway due to a difference in mtime or size item.safe_list = [] # The `check_list` will contain all items that need # to be copied with checksum option enabled item.check_list = [] for entry in self._list_files(item, item.src): # If item is a directory, we only need to save it in 'dir.list' if entry.mode[0] == "d": dir_list.write(entry.path + "\n") continue # Add every file in the source path to the list of files # to be protected from deletion ('exclude_and_protect.filter') # But only if we know the destination directory is non-empty if ref_has_content: exclude_and_protect_filter.write("P /" + entry.path + "\n") exclude_and_protect_filter.write("- /" + entry.path + "\n") # If source item is older than safe_horizon, # add it to 'safe.list' if self.safe_horizon and entry.date < self.safe_horizon: item.safe_list.append(entry) continue # If ref_hash is None, it means we failed to retrieve the # destination file list. We assume the only safe way is to # check every file that is older than safe_horizon if ref_hash is None: item.check_list.append(entry) continue # If source file differs by time or size from the matching # destination, rsync will discover the difference in any case. # It is then safe to skip checksum check here. dst_item = ref_hash.get(entry.path, None) if dst_item is None: item.safe_list.append(entry) continue different_size = dst_item.size != entry.size different_date = dst_item.date != entry.date if different_size or different_date: item.safe_list.append(entry) continue # All remaining files must be checked with checksums enabled item.check_list.append(entry) # Close all the control files dir_list.close() exclude_and_protect_filter.close() def _create_dir_and_purge(self, item): """ Create destination directories and delete any unknown file :param _RsyncCopyItem item: information about a copy operation """ # Build the rsync object required for the analysis rsync = self._rsync_factory(item) # Create directories and delete any unknown file self._rsync_ignore_vanished_files( rsync, "--recursive", "--delete", "--files-from=%s" % item.dir_file, "--filter", "merge %s" % item.exclude_and_protect_file, item.src, item.dst, check=True, ) def _copy(self, rsync, src, dst, file_list, checksum=False): """ The method execute the call to rsync, using as source a a list of files, and adding the checksum option if required by the caller. :param Rsync rsync: the Rsync object used to retrieve the list of files inside the directories for copy purposes :param str src: source directory :param str dst: destination directory :param str file_list: path to the file containing the sources for rsync :param bool checksum: if checksum argument for rsync is required """ # Build the rsync call args args = ["--files-from=%s" % file_list] if checksum: # Add checksum option if needed args.append("--checksum") self._rsync_ignore_vanished_files(rsync, src, dst, *args, check=True) def _list_files(self, item, path): """ This method recursively retrieves a list of files contained in a directory, either local or remote (if starts with ':') :param _RsyncCopyItem item: information about a copy operation :param str path: the path we want to inspect :except CommandFailedException: if rsync call fails :except RsyncListFilesFailure: if rsync output can't be parsed """ _logger.debug("list_files: %r", path) # Build the rsync object required for the analysis rsync = self._rsync_factory(item) try: # Use the --no-human-readable option to avoid digit groupings # in "size" field with rsync >= 3.1.0. # Ref: http://ftp.samba.org/pub/rsync/src/rsync-3.1.0-NEWS rsync.get_output( "--no-human-readable", "--list-only", "-r", path, check=True ) except CommandFailedException: # This could fail due to the local or the remote rsync # older than 3.1. IF so, fallback to pre 3.1 mode if self.rsync_has_ignore_missing_args and rsync.ret in ( 12, # Error in rsync protocol data stream (remote) 1, ): # Syntax or usage error (local) self._rsync_set_pre_31_mode() # Recursive call, uses the compatibility mode for item in self._list_files(item, path): yield item return else: raise # Cache tzlocal object we need to build dates tzinfo = dateutil.tz.tzlocal() for line in rsync.out.splitlines(): line = line.rstrip() match = self.LIST_ONLY_RE.match(line) if match: mode = match.group("mode") # no exceptions here: the regexp forces 'size' to be an integer size = int(match.group("size")) try: date_str = match.group("date") # The date format has been validated by LIST_ONLY_RE. # Use "2014/06/05 18:00:00" format if the sending rsync # is compiled with HAVE_STRFTIME, otherwise use # "Thu Jun 5 18:00:00 2014" format if date_str[0].isdigit(): date = datetime.datetime.strptime(date_str, "%Y/%m/%d %H:%M:%S") else: date = datetime.datetime.strptime( date_str, "%a %b %d %H:%M:%S %Y" ) date = date.replace(tzinfo=tzinfo) except (TypeError, ValueError): # This should not happen, due to the regexp msg = ( "Unable to parse rsync --list-only output line " "(date): '%s'" % line ) _logger.exception(msg) raise RsyncListFilesFailure(msg) path = match.group("path") yield _FileItem(mode, size, date, path) else: # This is a hard error, as we are unable to parse the output # of rsync. It can only happen with a modified or unknown # rsync version (perhaps newer than 3.1?) msg = "Unable to parse rsync --list-only output line: '%s'" % line _logger.error(msg) raise RsyncListFilesFailure(msg) def _rsync_ignore_vanished_files(self, rsync, *args, **kwargs): """ Wrap an Rsync.get_output() call and ignore missing args TODO: when rsync 3.1 will be widespread, replace this with --ignore-missing-args argument :param Rsync rsync: the Rsync object used to execute the copy """ kwargs["allowed_retval"] = (0, 23, 24) rsync.get_output(*args, **kwargs) # If return code is 23 and there is any error which doesn't match # the VANISHED_RE regexp raise an error if rsync.ret == 23 and rsync.err is not None: for line in rsync.err.splitlines(): match = self.VANISHED_RE.match(line.rstrip()) if match: continue else: _logger.error("First rsync error line: %s", line) raise CommandFailedException( dict(ret=rsync.ret, out=rsync.out, err=rsync.err) ) return rsync.out, rsync.err def statistics(self): """ Return statistics about the copy object. :rtype: dict """ # This method can only run at the end of a non empty copy assert self.copy_end_time assert self.item_list assert self.jobs_done # Initialise the result calculating the total runtime stat = { "total_time": total_seconds(self.copy_end_time - self.copy_start_time), "number_of_workers": self.workers, "analysis_time_per_item": {}, "copy_time_per_item": {}, "serialized_copy_time_per_item": {}, } # Calculate the time spent during the analysis of the items analysis_start = None analysis_end = None for item in self.item_list: # Some items don't require analysis if not item.analysis_end_time: continue # Build a human readable name to refer to an item in the output ident = item.label if not analysis_start: analysis_start = item.analysis_start_time elif analysis_start > item.analysis_start_time: analysis_start = item.analysis_start_time if not analysis_end: analysis_end = item.analysis_end_time elif analysis_end < item.analysis_end_time: analysis_end = item.analysis_end_time stat["analysis_time_per_item"][ident] = total_seconds( item.analysis_end_time - item.analysis_start_time ) stat["analysis_time"] = total_seconds(analysis_end - analysis_start) # Calculate the time spent per job # WARNING: this code assumes that every item is copied separately, # so it's strictly tied to the `_job_generator` method code item_data = {} for job in self.jobs_done: # WARNING: the item contained in the job is not the same object # contained in self.item_list, as it has gone through two # pickling/unpickling cycle # Build a human readable name to refer to an item in the output ident = self.item_list[job.item_idx].label # If this is the first time we see this item we just store the # values from the job if ident not in item_data: item_data[ident] = { "start": job.copy_start_time, "end": job.copy_end_time, "total_time": job.copy_end_time - job.copy_start_time, } else: data = item_data[ident] if data["start"] > job.copy_start_time: data["start"] = job.copy_start_time if data["end"] < job.copy_end_time: data["end"] = job.copy_end_time data["total_time"] += job.copy_end_time - job.copy_start_time # Calculate the time spent copying copy_start = None copy_end = None serialized_time = datetime.timedelta(0) for ident in item_data: data = item_data[ident] if copy_start is None or copy_start > data["start"]: copy_start = data["start"] if copy_end is None or copy_end < data["end"]: copy_end = data["end"] stat["copy_time_per_item"][ident] = total_seconds( data["end"] - data["start"] ) stat["serialized_copy_time_per_item"][ident] = total_seconds( data["total_time"] ) serialized_time += data["total_time"] # Store the total time spent by copying stat["copy_time"] = total_seconds(copy_end - copy_start) stat["serialized_copy_time"] = total_seconds(serialized_time) return stat barman-3.13.3/barman/output.py0000644000175100001660000024323615002177425014411 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2013-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module control how the output of Barman will be rendered """ from __future__ import print_function import datetime import inspect import json import logging import sys from dateutil import tz from barman.infofile import BackupInfo from barman.utils import ( BarmanEncoder, force_str, human_readable_timedelta, pretty_size, redact_passwords, timestamp, ) from barman.xlog import diff_lsn __all__ = [ "error_occurred", "debug", "info", "warning", "error", "exception", "result", "close_and_exit", "close", "set_output_writer", "AVAILABLE_WRITERS", "DEFAULT_WRITER", "ConsoleOutputWriter", "NagiosOutputWriter", "JsonOutputWriter", ] #: True if error or exception methods have been called error_occurred = False #: Exit code if error occurred error_exit_code = 1 #: Enable colors in the output ansi_colors_enabled = False def _ansi_color(command): """ Return the ansi sequence for the provided color """ return "\033[%sm" % command def _colored(message, color): """ Return a string formatted with the provided color. """ if ansi_colors_enabled: return _ansi_color(color) + message + _ansi_color("0") else: return message def _red(message): """ Format a red string """ return _colored(message, "31") def _green(message): """ Format a green string """ return _colored(message, "32") def _yellow(message): """ Format a yellow string """ return _colored(message, "33") def _format_message(message, args): """ Format a message using the args list. The result will be equivalent to message % args If args list contains a dictionary as its only element the result will be message % args[0] :param str message: the template string to be formatted :param tuple args: a list of arguments :return: the formatted message :rtype: str """ if len(args) == 1 and isinstance(args[0], dict): return message % args[0] elif len(args) > 0: return message % args else: return message def _put(level, message, *args, **kwargs): """ Send the message with all the remaining positional arguments to the configured output manager with the right output level. The message will be sent also to the logger unless explicitly disabled with log=False No checks are performed on level parameter as this method is meant to be called only by this module. If level == 'exception' the stack trace will be also logged :param str level: :param str message: the template string to be formatted :param tuple args: all remaining arguments are passed to the log formatter :key bool log: whether to log the message :key bool is_error: treat this message as an error """ # handle keyword-only parameters log = kwargs.pop("log", True) is_error = kwargs.pop("is_error", False) global error_exit_code error_exit_code = kwargs.pop("exit_code", error_exit_code) if len(kwargs): raise TypeError( "%s() got an unexpected keyword argument %r" % (inspect.stack()[1][3], kwargs.popitem()[0]) ) if is_error: global error_occurred error_occurred = True _writer.error_occurred() # Make sure the message is an unicode string if message: message = force_str(message) # dispatch the call to the output handler getattr(_writer, level)(message, *args) # log the message as originating from caller's caller module if log: exc_info = False if level == "exception": level = "error" exc_info = True frm = inspect.stack()[2] mod = inspect.getmodule(frm[0]) logger = logging.getLogger(mod.__name__) log_level = logging.getLevelName(level.upper()) logger.log(log_level, message, *args, **{"exc_info": exc_info}) def _dispatch(obj, prefix, name, *args, **kwargs): """ Dispatch the call to the %(prefix)s_%(name) method of the obj object :param obj: the target object :param str prefix: prefix of the method to be called :param str name: name of the method to be called :param tuple args: all remaining positional arguments will be sent to target :param dict kwargs: all remaining keyword arguments will be sent to target :return: the result of the invoked method :raise ValueError: if the target method is not present """ method_name = "%s_%s" % (prefix, name) handler = getattr(obj, method_name, None) if callable(handler): return handler(*args, **kwargs) else: raise ValueError( "The object %r does not have the %r method" % (obj, method_name) ) def is_quiet(): """ Calls the "is_quiet" method, accessing the protected parameter _quiet of the instanced OutputWriter :return bool: the _quiet parameter value """ return _writer.is_quiet() def is_debug(): """ Calls the "is_debug" method, accessing the protected parameter _debug of the instanced OutputWriter :return bool: the _debug parameter value """ return _writer.is_debug() def debug(message, *args, **kwargs): """ Output a message with severity 'DEBUG' :key bool log: whether to log the message """ _put("debug", message, *args, **kwargs) def info(message, *args, **kwargs): """ Output a message with severity 'INFO' :key bool log: whether to log the message """ _put("info", message, *args, **kwargs) def warning(message, *args, **kwargs): """ Output a message with severity 'WARNING' :key bool log: whether to log the message """ _put("warning", message, *args, **kwargs) def error(message, *args, **kwargs): """ Output a message with severity 'ERROR'. Also records that an error has occurred unless the ignore parameter is True. :key bool ignore: avoid setting an error exit status (default False) :key bool log: whether to log the message """ # ignore is a keyword-only parameter ignore = kwargs.pop("ignore", False) if not ignore: kwargs.setdefault("is_error", True) _put("error", message, *args, **kwargs) def exception(message, *args, **kwargs): """ Output a message with severity 'EXCEPTION' If raise_exception parameter doesn't evaluate to false raise and exception: - if raise_exception is callable raise the result of raise_exception() - if raise_exception is an exception raise it - else raise the last exception again :key bool ignore: avoid setting an error exit status :key raise_exception: raise an exception after the message has been processed :key bool log: whether to log the message """ # ignore and raise_exception are keyword-only parameters ignore = kwargs.pop("ignore", False) # noinspection PyNoneFunctionAssignment raise_exception = kwargs.pop("raise_exception", None) if not ignore: kwargs.setdefault("is_error", True) _put("exception", message, *args, **kwargs) if raise_exception: if callable(raise_exception): # noinspection PyCallingNonCallable raise raise_exception(message) elif isinstance(raise_exception, BaseException): raise raise_exception else: raise def init(command, *args, **kwargs): """ Initialize the output writer for a given command. :param str command: name of the command are being executed :param tuple args: all remaining positional arguments will be sent to the output processor :param dict kwargs: all keyword arguments will be sent to the output processor """ try: _dispatch(_writer, "init", command, *args, **kwargs) except ValueError: exception( 'The %s writer does not support the "%s" command', _writer.__class__.__name__, command, ) close_and_exit() def result(command, *args, **kwargs): """ Output the result of an operation. :param str command: name of the command are being executed :param tuple args: all remaining positional arguments will be sent to the output processor :param dict kwargs: all keyword arguments will be sent to the output processor """ try: _dispatch(_writer, "result", command, *args, **kwargs) except ValueError: exception( 'The %s writer does not support the "%s" command', _writer.__class__.__name__, command, ) close_and_exit() def close_and_exit(): """ Close the output writer and terminate the program. If an error has been emitted the program will report a non zero return value. """ close() if error_occurred: sys.exit(error_exit_code) else: sys.exit(0) def close(): """ Close the output writer. """ _writer.close() def set_output_writer(new_writer, *args, **kwargs): """ Replace the current output writer with a new one. The new_writer parameter can be a symbolic name or an OutputWriter object :param new_writer: the OutputWriter name or the actual OutputWriter :type: string or an OutputWriter :param tuple args: all remaining positional arguments will be passed to the OutputWriter constructor :param dict kwargs: all remaining keyword arguments will be passed to the OutputWriter constructor """ global _writer _writer.close() if new_writer in AVAILABLE_WRITERS: _writer = AVAILABLE_WRITERS[new_writer](*args, **kwargs) else: _writer = new_writer class ConsoleOutputWriter(object): SERVER_OUTPUT_PREFIX = "Server %s:" def __init__(self, debug=False, quiet=False): """ Default output writer that output everything on console. :param bool debug: print debug messages on standard error :param bool quiet: don't print info messages """ self._debug = debug self._quiet = quiet #: Used in check command to hold the check results self.result_check_list = [] #: The minimal flag. If set the command must output a single list of #: values. self.minimal = False #: The server is active self.active = True def _print(self, message, args, stream): """ Print an encoded message on the given output stream """ # Make sure to add a newline at the end of the message if message is None: message = "\n" else: message += "\n" # Format and encode the message, redacting eventual passwords encoded_msg = redact_passwords(_format_message(message, args)).encode("utf-8") try: # Python 3.x stream.buffer.write(encoded_msg) except AttributeError: # Python 2.x stream.write(encoded_msg) stream.flush() def _out(self, message, args): """ Print a message on standard output """ self._print(message, args, sys.stdout) def _err(self, message, args): """ Print a message on standard error """ self._print(message, args, sys.stderr) def is_quiet(self): """ Access the quiet property of the OutputWriter instance :return bool: if the writer is quiet or not """ return self._quiet def is_debug(self): """ Access the debug property of the OutputWriter instance :return bool: if the writer is in debug mode or not """ return self._debug def debug(self, message, *args): """ Emit debug. """ if self._debug: self._err("DEBUG: %s" % message, args) def info(self, message, *args): """ Normal messages are sent to standard output """ if not self._quiet: self._out(message, args) def warning(self, message, *args): """ Warning messages are sent to standard error """ self._err(_yellow("WARNING: %s" % message), args) def error(self, message, *args): """ Error messages are sent to standard error """ self._err(_red("ERROR: %s" % message), args) def exception(self, message, *args): """ Warning messages are sent to standard error """ self._err(_red("EXCEPTION: %s" % message), args) def error_occurred(self): """ Called immediately before any message method when the originating call has is_error=True """ def close(self): """ Close the output channel. Nothing to do for console. """ def result_backup(self, backup_info): """ Render the result of a backup. Nothing to do for console. """ # TODO: evaluate to display something useful here def result_recovery(self, results): """ Render the result of a recovery. """ if len(results["changes"]) > 0: self.info("") self.info("IMPORTANT") self.info("These settings have been modified to prevent data losses") self.info("") for assertion in results["changes"]: self.info( "%s line %s: %s = %s", assertion.filename, assertion.line, assertion.key, assertion.value, ) if len(results["warnings"]) > 0: self.info("") self.info("WARNING") self.info( "You are required to review the following options" " as potentially dangerous" ) self.info("") for assertion in results["warnings"]: self.info( "%s line %s: %s = %s", assertion.filename, assertion.line, assertion.key, assertion.value, ) if results["missing_files"]: # At least one file is missing, warn the user self.info("") self.info("WARNING") self.info( "The following configuration files have not been " "saved during backup, hence they have not been " "restored." ) self.info( "You need to manually restore them " "in order to start the restored PostgreSQL instance:" ) self.info("") for file_name in results["missing_files"]: self.info(" %s" % file_name) if results["get_wal"]: self.info("") self.info("WARNING: 'get-wal' is in the specified 'recovery_options'.") self.info( "Before you start up the PostgreSQL server, please " "review the %s file", results["recovery_configuration_file"], ) self.info( "inside the target directory. Make sure that " "'restore_command' can be executed by " "the PostgreSQL user." ) self.info("") self.info( "Restore operation completed (start time: %s, elapsed time: %s)", results["recovery_start_time"], human_readable_timedelta( datetime.datetime.now(tz.tzlocal()) - results["recovery_start_time"] ), ) self.info("Your PostgreSQL server has been successfully prepared for recovery!") def _record_check(self, server_name, check, status, hint, perfdata): """ Record the check line in result_check_map attribute This method is for subclass use :param str server_name: the server is being checked :param str check: the check name :param bool status: True if succeeded :param str,None hint: hint to print if not None :param str,None perfdata: additional performance data to print if not None """ self.result_check_list.append( dict( server_name=server_name, check=check, status=status, hint=hint, perfdata=perfdata, ) ) if not status and self.active: global error_occurred error_occurred = True def init_check(self, server_name, active, disabled): """ Init the check command :param str server_name: the server we are start listing :param boolean active: The server is active :param boolean disabled: The server is disabled """ display_name = server_name # If the server has been manually disabled if not active: display_name += " (inactive)" # If server has configuration errors elif disabled: display_name += " (WARNING: disabled)" self.info(self.SERVER_OUTPUT_PREFIX % display_name) self.active = active def result_check(self, server_name, check, status, hint=None, perfdata=None): """ Record a server result of a server check and output it as INFO :param str server_name: the server is being checked :param str check: the check name :param bool status: True if succeeded :param str,None hint: hint to print if not None :param str,None perfdata: additional performance data to print if not None """ self._record_check(server_name, check, status, hint, perfdata) if hint: self.info( "\t%s: %s (%s)" % (check, _green("OK") if status else _red("FAILED"), hint) ) else: self.info("\t%s: %s" % (check, _green("OK") if status else _red("FAILED"))) def init_list_backup(self, server_name, minimal=False): """ Init the list-backups command :param str server_name: the server we are start listing :param bool minimal: if true output only a list of backup id """ self.minimal = minimal def result_list_backup(self, backup_info, backup_size, wal_size, retention_status): """ Output a single backup in the list-backups command :param BackupInfo backup_info: backup we are displaying :param backup_size: size of base backup (with the required WAL files) :param wal_size: size of WAL files belonging to this backup (without the required WAL files) :param retention_status: retention policy status """ # If minimal is set only output the backup id if self.minimal: self.info(backup_info.backup_id) return out_list = ["%s %s" % (backup_info.server_name, backup_info.backup_id)] if backup_info.backup_name is not None: out_list.append(" '%s'" % backup_info.backup_name) # Set backup type label out_list.append(" - %s - " % backup_info.backup_type[0].upper()) if backup_info.status in BackupInfo.STATUS_COPY_DONE: end_time = backup_info.end_time.ctime() out_list.append( "%s - Size: %s - WAL Size: %s" % (end_time, pretty_size(backup_size), pretty_size(wal_size)) ) if backup_info.status == BackupInfo.WAITING_FOR_WALS: out_list.append(" - %s" % BackupInfo.WAITING_FOR_WALS) if retention_status and retention_status != BackupInfo.NONE: out_list.append(" - %s" % retention_status) else: out_list.append(backup_info.status) self.info("".join(out_list)) @staticmethod def render_show_backup_general(backup_info, output_fun, row): """ Render general backup metadata in plain text form. :param dict backup_info: a dictionary containing the backup metadata :param function output_fun: function which accepts a string and sends it to an output writer :param str row: format string which allows for `key: value` rows to be formatted """ backup_name = backup_info.get("backup_name") if backup_name: output_fun(row.format("Backup Name", backup_name)) output_fun(row.format("Server Name", backup_info["server_name"])) system_id = backup_info.get("systemid") if system_id: output_fun(row.format("System Id", system_id)) output_fun(row.format("Status", backup_info["status"])) if backup_info["status"] in BackupInfo.STATUS_COPY_DONE: output_fun(row.format("PostgreSQL Version", backup_info["version"])) output_fun(row.format("PGDATA directory", backup_info["pgdata"])) cluster_size = backup_info.get("cluster_size") if cluster_size: output_fun( row.format("Estimated Cluster Size", pretty_size(cluster_size)) ) output_fun("") @staticmethod def render_show_backup_server(backup_info, output_fun, header_row, nested_row): """ Render server metadata in plain text form. :param dict backup_info: a dictionary containing the backup metadata :param function output_fun: function which accepts a string and sends it to an output writer :param str header_row: format string which allows for single value header rows to be formatted :param str nested_row: format string which allows for `key: value` rows to be formatted """ data_checksums = backup_info.get("data_checksums") summarize_wal = backup_info.get("summarize_wal") if data_checksums or summarize_wal: output_fun(header_row.format("Server information")) if data_checksums: output_fun( nested_row.format("Checksums", backup_info["data_checksums"]) ) if summarize_wal: output_fun(nested_row.format("WAL summarizer", summarize_wal)) output_fun("") @staticmethod def render_show_backup_snapshots(backup_info, output_fun, header_row, nested_row): """ Render snapshot metadata in plain text form. :param dict backup_info: a dictionary containing the backup metadata :param function output_fun: function which accepts a string and sends it to an output writer :param str header_row: format string which allows for single value header rows to be formatted :param str nested_row: format string which allows for `key: value` rows to be formatted """ if ( "snapshots_info" in backup_info and backup_info["snapshots_info"] is not None ): output_fun(header_row.format("Snapshot information")) for key, value in backup_info["snapshots_info"].items(): if key != "snapshots" and key != "provider_info": output_fun(nested_row.format(key, value)) for key, value in backup_info["snapshots_info"]["provider_info"].items(): output_fun(nested_row.format(key, value)) output_fun("") for metadata in backup_info["snapshots_info"]["snapshots"]: for key, value in sorted(metadata["provider"].items()): output_fun(nested_row.format(key, value)) output_fun( nested_row.format("Mount point", metadata["mount"]["mount_point"]) ) output_fun( nested_row.format( "Mount options", metadata["mount"]["mount_options"] ) ) output_fun("") @staticmethod def render_show_backup_tablespaces(backup_info, output_fun, header_row, nested_row): """ Render tablespace metadata in plain text form. :param dict backup_info: a dictionary containing the backup metadata :param function output_fun: function which accepts a string and sends it to an output writer :param str header_row: format string which allows for single value header rows to be formatted :param str nested_row: format string which allows for `key: value` rows to be formatted """ if backup_info["tablespaces"]: output_fun(header_row.format("Tablespaces")) for item in backup_info["tablespaces"]: output = "{} (oid: {})".format(item.location, item.oid) output_fun(nested_row.format(item.name, output)) output_fun("") @staticmethod def render_show_backup_base(backup_info, output_fun, header_row, nested_row): """ Renders base backup metadata in plain text form. :param dict backup_info: a dictionary containing the backup metadata :param function output_fun: function which accepts a string and sends it to an output writer :param str header_row: format string which allows for single value header rows to be formatted :param str nested_row: format string which allows for `key: value` rows to be formatted """ output_fun(header_row.format("Base backup information")) backup_method = backup_info.get("mode") if backup_method: output_fun(nested_row.format("Backup Method", backup_method)) # The "show-backup" for the cloud takes the input of a backup_info, # not the result of a get_backup_ext_info() call, Instead, it # takes a backup_info.to_dict(). So in those cases, the following # fields will not exist: `backup_type`, `deduplication_ratio`, # "root_backup_id", "chain_size", "est_dedup_size", "copy_time", # "analysis_time" and "estimated_throughput". # Because of this, this ugly piece of code is a temporary workaround # so we do not break the show-backup for the "cloud-backup-show". backup_type = backup_info.get("backup_type") backup_size = backup_info.get("deduplicated_size") # Show only for postgres backups if backup_type and backup_method == "postgres": output_fun(nested_row.format("Backup Type", backup_type)) wal_size = backup_info.get("wal_size") if backup_size: backup_size_output = "{}".format(pretty_size(backup_size)) if wal_size: backup_size_output += " ({} with WALs)".format( pretty_size(backup_size + wal_size), ) output_fun(nested_row.format("Backup Size", backup_size_output)) if wal_size: output_fun(nested_row.format("WAL Size", pretty_size(wal_size))) # Show only for incremental and rsync backups est_dedup_size = backup_info.get("est_dedup_size") deduplication_ratio = backup_info.get("deduplication_ratio") cluster_size = backup_info.get("cluster_size") size = backup_info.get("size") if est_dedup_size is None: est_dedup_size = 0 deduplication_ratio = 0 sz = cluster_size if backup_method != "postgres": sz = size if sz and backup_size: deduplication_ratio = 1 - (backup_size / sz) # The following operation needs to use cluster_size o size # so we do not break backward compatibility when old backups # taken with barman < 3.11 are present in the backup catalog. # Old backups do not have the cluster_size field. if cluster_size or size: est_dedup_size = (cluster_size or size) * deduplication_ratio if backup_type and backup_type in {"rsync", "incremental"}: dedupe_output = "{} ({})".format( pretty_size(est_dedup_size), "{percent:.2%}".format(percent=deduplication_ratio), ) output_fun(nested_row.format("Resources saved", dedupe_output)) output_fun(nested_row.format("Timeline", backup_info["timeline"])) output_fun(nested_row.format("Begin WAL", backup_info["begin_wal"])) output_fun(nested_row.format("End WAL", backup_info["end_wal"])) # This is WAL stuff... wal_num = backup_info.get("wal_num") if wal_num: output_fun(nested_row.format("WAL number", wal_num)) wal_compression_ratio = backup_info.get("wal_compression_ratio", 0) # Output WAL compression ratio for basebackup WAL files if wal_compression_ratio > 0: wal_compression_output = "{percent:.2%}".format( percent=backup_info["wal_compression_ratio"] ) output_fun( nested_row.format("WAL compression ratio", wal_compression_output) ) # Back to regular stuff output_fun(nested_row.format("Begin time", backup_info["begin_time"])) output_fun(nested_row.format("End time", backup_info["end_time"])) # If copy statistics are available, show summary copy_time = backup_info.get("copy_time", 0) analysis_time = backup_info.get("analysis_time", 0) est_throughput = backup_info.get("estimated_throughput") number_of_workers = backup_info.get("number_of_workers", 1) copy_stats = backup_info.get("copy_stats", {}) if copy_stats and not copy_time: copy_time = copy_stats.get("copy_time", 0) analysis_time = copy_stats.get("analysis_time", 0) number_of_workers = copy_stats.get("number_of_workers", 1) if copy_time: copy_time_output = human_readable_timedelta( datetime.timedelta(seconds=copy_time) ) if analysis_time >= 1: copy_time_output += " + {} startup".format( human_readable_timedelta(datetime.timedelta(seconds=analysis_time)) ) output_fun(nested_row.format("Copy time", copy_time_output)) if est_throughput: est_througput_output = "{}/s".format(pretty_size(est_throughput)) if number_of_workers > 1: est_througput_output += " (%s jobs)" % number_of_workers output_fun( nested_row.format("Estimated throughput", est_througput_output) ) output_fun(nested_row.format("Begin Offset", backup_info["begin_offset"])) output_fun(nested_row.format("End Offset", backup_info["end_offset"])) output_fun(nested_row.format("Begin LSN", backup_info["begin_xlog"])) output_fun(nested_row.format("End LSN", backup_info["end_xlog"])) output_fun("") @staticmethod def render_show_backup_walinfo(backup_info, output_fun, header_row, nested_row): """ Renders WAL metadata in plain text form. :param dict backup_info: a dictionary containing the backup metadata :param function output_fun: function which accepts a string and sends it to an output writer :param str header_row: format string which allows for single value header rows to be formatted :param str nested_row: format string which allows for `key: value` rows to be formatted """ if any( key in backup_info for key in ( "wal_until_next_num", "wal_until_next_size", "wals_per_second", "wal_until_next_compression_ratio", "children_timelines", ) ): output_fun(header_row.format("WAL information")) output_fun( nested_row.format("No of files", backup_info["wal_until_next_num"]) ) output_fun( nested_row.format( "Disk usage", pretty_size(backup_info["wal_until_next_size"]) ) ) # Output WAL rate if backup_info["wals_per_second"] > 0: output_fun( nested_row.format( "WAL rate", "{:.2f}/hour".format(backup_info["wals_per_second"] * 3600), ) ) # Output WAL compression ratio for archived WAL files if backup_info["wal_until_next_compression_ratio"] > 0: output_fun( nested_row.format( "Compression ratio", "{percent:.2%}".format( percent=backup_info["wal_until_next_compression_ratio"] ), ), ) output_fun(nested_row.format("Last available", backup_info["wal_last"])) if backup_info["children_timelines"]: timelines = backup_info["children_timelines"] output_fun( nested_row.format( "Reachable timelines", ", ".join([str(history.tli) for history in timelines]), ), ) output_fun("") @staticmethod def render_show_backup_catalog_info( backup_info, output_fun, header_row, nested_row ): """ Renders catalog metadata in plain text form. :param dict backup_info: a dictionary containing the backup metadata :param function output_fun: function which accepts a string and sends it to an output writer :param str header_row: format string which allows for single value header rows to be formatted :param str nested_row: format string which allows for `key: value` rows to be formatted """ if "retention_policy_status" in backup_info: output_fun(header_row.format("Catalog information")) output_fun( nested_row.format( "Retention Policy", backup_info["retention_policy_status"] or "not enforced", ) ) previous_backup_id = backup_info.setdefault( "previous_backup_id", "not available" ) output_fun( nested_row.format( "Previous Backup", previous_backup_id or "- (this is the oldest base backup)", ) ) next_backup_id = backup_info.setdefault("next_backup_id", "not available") output_fun( nested_row.format( "Next Backup", next_backup_id or "- (this is the latest base backup)", ) ) if "children_timelines" in backup_info and backup_info["children_timelines"]: output_fun("") output_fun( "WARNING: WAL information is inaccurate due to " "multiple timelines interacting with this backup" ) backup_type = backup_info.get("backup_type") # Show only for incremental backups if backup_type == "incremental": output_fun(nested_row.format("Root Backup", backup_info["root_backup_id"])) output_fun( nested_row.format("Parent Backup", backup_info["parent_backup_id"]) ) output_fun( nested_row.format("Backup chain size", backup_info["chain_size"]) ) backup_method = backup_info.get("mode") # Show only for postgres backups if backup_method == "postgres": if backup_info["children_backup_ids"] is not None: output_fun( nested_row.format( "Children Backup(s)", backup_info["children_backup_ids"], ) ) @staticmethod def render_show_backup(backup_info, output_fun): """ Renders the output of a show backup command :param dict backup_info: a dictionary containing the backup metadata :param function output_fun: function which accepts a string and sends it to an output writer """ row = " {:<23}: {}" header_row = " {}:" nested_row = " {:<21}: {}" output_fun("Backup {}:".format(backup_info["backup_id"])) ConsoleOutputWriter.render_show_backup_general(backup_info, output_fun, row) ConsoleOutputWriter.render_show_backup_server( backup_info, output_fun, header_row, nested_row ) if backup_info["status"] in BackupInfo.STATUS_COPY_DONE: ConsoleOutputWriter.render_show_backup_snapshots( backup_info, output_fun, header_row, nested_row ) ConsoleOutputWriter.render_show_backup_tablespaces( backup_info, output_fun, header_row, nested_row ) ConsoleOutputWriter.render_show_backup_base( backup_info, output_fun, header_row, nested_row ) ConsoleOutputWriter.render_show_backup_walinfo( backup_info, output_fun, header_row, nested_row ) ConsoleOutputWriter.render_show_backup_catalog_info( backup_info, output_fun, header_row, nested_row ) else: if backup_info["error"]: output_fun(row.format("Error", backup_info["error"])) def result_show_backup(self, backup_ext_info): """ Output all available information about a backup in show-backup command The argument has to be the result of a Server.get_backup_ext_info() call :param dict backup_ext_info: a dictionary containing the info to display """ data = dict(backup_ext_info) self.render_show_backup(data, self.info) def init_status(self, server_name): """ Init the status command :param str server_name: the server we are start listing """ self.info(self.SERVER_OUTPUT_PREFIX, server_name) def result_status(self, server_name, status, description, message): """ Record a result line of a server status command and output it as INFO :param str server_name: the server is being checked :param str status: the returned status code :param str description: the returned status description :param str,object message: status message. It will be converted to str """ self.info("\t%s: %s", description, str(message)) def init_replication_status(self, server_name, minimal=False): """ Init the 'standby-status' command :param str server_name: the server we are start listing :param str minimal: minimal output """ self.minimal = minimal def result_replication_status(self, server_name, target, server_lsn, standby_info): """ Record a result line of a server status command and output it as INFO :param str server_name: the replication server :param str target: all|hot-standby|wal-streamer :param str server_lsn: server's current lsn :param StatReplication standby_info: status info of a standby """ if target == "hot-standby": title = "hot standby servers" elif target == "wal-streamer": title = "WAL streamers" else: title = "streaming clients" if self.minimal: # Minimal output if server_lsn: # current lsn from the master self.info( "%s for master '%s' (LSN @ %s):", title.capitalize(), server_name, server_lsn, ) else: # We are connected to a standby self.info("%s for slave '%s':", title.capitalize(), server_name) else: # Full output self.info("Status of %s for server '%s':", title, server_name) # current lsn from the master if server_lsn: self.info(" Current LSN on master: %s", server_lsn) if standby_info is not None and not len(standby_info): self.info(" No %s attached", title) return # Minimal output if self.minimal: n = 1 for standby in standby_info: if not standby.replay_lsn: # WAL streamer self.info( " %s. W) %s@%s S:%s W:%s P:%s AN:%s", n, standby.usename, standby.client_addr or "socket", standby.sent_lsn, standby.write_lsn, standby.sync_priority, standby.application_name, ) else: # Standby self.info( " %s. %s) %s@%s S:%s F:%s R:%s P:%s AN:%s", n, standby.sync_state[0].upper(), standby.usename, standby.client_addr or "socket", standby.sent_lsn, standby.flush_lsn, standby.replay_lsn, standby.sync_priority, standby.application_name, ) n += 1 else: n = 1 self.info(" Number of %s: %s", title, len(standby_info)) for standby in standby_info: self.info("") # Calculate differences in bytes sent_diff = diff_lsn(standby.sent_lsn, standby.current_lsn) write_diff = diff_lsn(standby.write_lsn, standby.current_lsn) flush_diff = diff_lsn(standby.flush_lsn, standby.current_lsn) replay_diff = diff_lsn(standby.replay_lsn, standby.current_lsn) # Determine the sync stage of the client sync_stage = None if not standby.replay_lsn: client_type = "WAL streamer" max_level = 3 else: client_type = "standby" max_level = 5 # Only standby can replay WAL info if replay_diff == 0: sync_stage = "5/5 Hot standby (max)" elif flush_diff == 0: sync_stage = "4/5 2-safe" # remote flush # If not yet done, set the sync stage if not sync_stage: if write_diff == 0: sync_stage = "3/%s Remote write" % max_level elif sent_diff == 0: sync_stage = "2/%s WAL Sent (min)" % max_level else: sync_stage = "1/%s 1-safe" % max_level # Synchronous standby if getattr(standby, "sync_priority", None) > 0: self.info( " %s. #%s %s %s", n, standby.sync_priority, standby.sync_state.capitalize(), client_type, ) # Asynchronous standby else: self.info( " %s. %s %s", n, standby.sync_state.capitalize(), client_type ) self.info(" Application name: %s", standby.application_name) self.info(" Sync stage : %s", sync_stage) if getattr(standby, "client_addr", None): self.info(" Communication : TCP/IP") self.info( " IP Address : %s / Port: %s / Host: %s", standby.client_addr, standby.client_port, standby.client_hostname or "-", ) else: self.info(" Communication : Unix domain socket") self.info(" User name : %s", standby.usename) self.info( " Current state : %s (%s)", standby.state, standby.sync_state ) if getattr(standby, "slot_name", None): self.info(" Replication slot: %s", standby.slot_name) self.info(" WAL sender PID : %s", standby.pid) self.info(" Started at : %s", standby.backend_start) if getattr(standby, "backend_xmin", None): self.info(" Standby's xmin : %s", standby.backend_xmin or "-") if getattr(standby, "sent_lsn", None): self.info( " Sent LSN : %s (diff: %s)", standby.sent_lsn, pretty_size(sent_diff), ) if getattr(standby, "write_lsn", None): self.info( " Write LSN : %s (diff: %s)", standby.write_lsn, pretty_size(write_diff), ) if getattr(standby, "flush_lsn", None): self.info( " Flush LSN : %s (diff: %s)", standby.flush_lsn, pretty_size(flush_diff), ) if getattr(standby, "replay_lsn", None): self.info( " Replay LSN : %s (diff: %s)", standby.replay_lsn, pretty_size(replay_diff), ) n += 1 def init_list_server(self, server_name, minimal=False): """ Init the list-servers command :param str server_name: the server we are start listing """ self.minimal = minimal def result_list_server(self, server_name, description=None): """ Output a result line of a list-servers command :param str server_name: the server is being checked :param str,None description: server description if applicable """ if self.minimal or not description: self.info("%s", server_name) else: self.info("%s - %s", server_name, description) def init_show_server(self, server_name, description=None): """ Init the show-servers command output method :param str server_name: the server we are displaying :param str,None description: server description if applicable """ if description: self.info(self.SERVER_OUTPUT_PREFIX % " ".join((server_name, description))) else: self.info(self.SERVER_OUTPUT_PREFIX % server_name) def result_show_server(self, server_name, server_info): """ Output the results of the show-servers command :param str server_name: the server we are displaying :param dict server_info: a dictionary containing the info to display """ for status, message in sorted(server_info.items()): self.info("\t%s: %s", status, message) def init_check_wal_archive(self, server_name): """ Init the check-wal-archive command output method :param str server_name: the server we are displaying """ self.info(self.SERVER_OUTPUT_PREFIX % server_name) def result_check_wal_archive(self, server_name): """ Output the results of the check-wal-archive command :param str server_name: the server we are displaying """ self.info(" - WAL archive check for server %s passed" % server_name) class JsonOutputWriter(ConsoleOutputWriter): def __init__(self, *args, **kwargs): """ Output writer that writes on standard output using JSON. When closed, it dumps all the collected results as a JSON object. """ super(JsonOutputWriter, self).__init__(*args, **kwargs) #: Store JSON data self.json_output = {} def _mangle_key(self, value): """ Mangle a generic description to be used as dict key :type value: str :rtype: str """ return value.lower().replace(" ", "_").replace("-", "_").replace(".", "") def _out_to_field(self, field, message, *args): """ Store a message in the required field """ if field not in self.json_output: self.json_output[field] = [] message = _format_message(message, args) self.json_output[field].append(message) def debug(self, message, *args): """ Add debug messages in _DEBUG list """ if not self._debug: return self._out_to_field("_DEBUG", message, *args) def info(self, message, *args): """ Add normal messages in _INFO list """ self._out_to_field("_INFO", message, *args) def warning(self, message, *args): """ Add warning messages in _WARNING list """ self._out_to_field("_WARNING", message, *args) def error(self, message, *args): """ Add error messages in _ERROR list """ self._out_to_field("_ERROR", message, *args) def exception(self, message, *args): """ Add exception messages in _EXCEPTION list """ self._out_to_field("_EXCEPTION", message, *args) def close(self): """ Close the output channel. Print JSON output """ if not self._quiet: json.dump(self.json_output, sys.stdout, sort_keys=True, cls=BarmanEncoder) self.json_output = {} def result_backup(self, backup_info): """ Save the result of a backup. """ self.json_output.update(backup_info.to_dict()) def result_recovery(self, results): """ Render the result of a recovery. """ changes_count = len(results["changes"]) self.json_output["changes_count"] = changes_count self.json_output["changes"] = results["changes"] if changes_count > 0: self.warning( "IMPORTANT! Some settings have been modified " "to prevent data losses. See 'changes' key." ) warnings_count = len(results["warnings"]) self.json_output["warnings_count"] = warnings_count self.json_output["warnings"] = results["warnings"] if warnings_count > 0: self.warning( "WARNING! You are required to review the options " "as potentially dangerous. See 'warnings' key." ) missing_files_count = len(results["missing_files"]) self.json_output["missing_files"] = results["missing_files"] if missing_files_count > 0: # At least one file is missing, warn the user self.warning( "WARNING! Some configuration files have not been " "saved during backup, hence they have not been " "restored. See 'missing_files' key." ) if results["get_wal"]: self.warning( "WARNING: 'get-wal' is in the specified " "'recovery_options'. Before you start up the " "PostgreSQL server, please review the recovery " "configuration inside the target directory. " "Make sure that 'restore_command' can be " "executed by the PostgreSQL user." ) self.json_output.update( { "recovery_start_time": results["recovery_start_time"].isoformat(" "), "recovery_start_time_timestamp": str( int(timestamp(results["recovery_start_time"])) ), "recovery_elapsed_time": human_readable_timedelta( datetime.datetime.now(tz.tzlocal()) - results["recovery_start_time"] ), "recovery_elapsed_time_seconds": ( datetime.datetime.now(tz.tzlocal()) - results["recovery_start_time"] ).total_seconds(), } ) def init_check(self, server_name, active, disabled): """ Init the check command :param str server_name: the server we are start listing :param boolean active: The server is active :param boolean disabled: The server is disabled """ self.json_output[server_name] = {} self.active = active def result_check(self, server_name, check, status, hint=None, perfdata=None): """ Record a server result of a server check and output it as INFO :param str server_name: the server is being checked :param str check: the check name :param bool status: True if succeeded :param str,None hint: hint to print if not None :param str,None perfdata: additional performance data to print if not None """ self._record_check(server_name, check, status, hint, perfdata) check_key = self._mangle_key(check) self.json_output[server_name][check_key] = dict( status="OK" if status else "FAILED", hint=hint or "" ) def init_list_backup(self, server_name, minimal=False): """ Init the list-backups command :param str server_name: the server we are listing :param bool minimal: if true output only a list of backup id """ self.minimal = minimal self.json_output[server_name] = [] def result_list_backup(self, backup_info, backup_size, wal_size, retention_status): """ Output a single backup in the list-backups command :param BackupInfo backup_info: backup we are displaying :param backup_size: size of base backup (with the required WAL files) :param wal_size: size of WAL files belonging to this backup (without the required WAL files) :param retention_status: retention policy status """ server_name = backup_info.server_name # If minimal is set only output the backup id if self.minimal: self.json_output[server_name].append(backup_info.backup_id) return output = dict(backup_id=backup_info.backup_id) if backup_info.backup_name is not None: output.update({"backup_name": backup_info.backup_name}) # Set backup type label output.update({"backup_type": backup_info.backup_type}) if backup_info.status in BackupInfo.STATUS_COPY_DONE: output.update( dict( end_time_timestamp=str(int(timestamp(backup_info.end_time))), end_time=backup_info.end_time.ctime(), size_bytes=backup_size, wal_size_bytes=wal_size, size=pretty_size(backup_size), wal_size=pretty_size(wal_size), status=backup_info.status, retention_status=retention_status or BackupInfo.NONE, ) ) else: output.update(dict(status=backup_info.status)) self.json_output[server_name].append(output) def result_show_backup(self, backup_ext_info): """ Output all available information about a backup in show-backup command in json format. The argument has to be the result of a :meth:`barman.server.Server.get_backup_ext_info` call :param dict backup_ext_info: a dictionary containing the info to display """ data = dict(backup_ext_info) server_name = data["server_name"] # General information output = self.json_output[server_name] = dict( backup_id=data["backup_id"], status=data["status"] ) backup_name = data.get("backup_name") if backup_name: output.update({"backup_name": backup_name}) system_id = data.get("systemid") if system_id: output.update({"system_id": system_id}) backup_type = data.get("backup_type") if backup_type: output.update({"backup_type": backup_type}) # Server information output["server_information"] = dict( data_checksums=data["data_checksums"], summarize_wal=data["summarize_wal"], ) cluster_size = data.get("cluster_size") if data["status"] in BackupInfo.STATUS_COPY_DONE: # This check is needed to keep backward compatibility between # barman versions <= 3.10.x, where cluster_size is not present. if cluster_size: output.update( dict( cluster_size=pretty_size(data["cluster_size"]), cluster_size_bytes=data["cluster_size"], ) ) # General information output.update( dict( postgresql_version=data["version"], pgdata_directory=data["pgdata"], tablespaces=[], ) ) # Base Backup information output["base_backup_information"] = dict( backup_method=data["mode"], backup_size=pretty_size(data["deduplicated_size"]), backup_size_bytes=data["deduplicated_size"], backup_size_with_wals=pretty_size( data["deduplicated_size"] + data["wal_size"] ), backup_size_with_wals_bytes=data["deduplicated_size"] + data["wal_size"], wal_size=pretty_size(data["wal_size"]), wal_size_bytes=data["wal_size"], timeline=data["timeline"], begin_wal=data["begin_wal"], end_wal=data["end_wal"], wal_num=data["wal_num"], begin_time_timestamp=str(int(timestamp(data["begin_time"]))), begin_time=data["begin_time"].isoformat(sep=" "), end_time_timestamp=str(int(timestamp(data["end_time"]))), end_time=data["end_time"].isoformat(sep=" "), begin_offset=data["begin_offset"], end_offset=data["end_offset"], begin_lsn=data["begin_xlog"], end_lsn=data["end_xlog"], ) if backup_type and backup_type in {"rsync", "incremental"}: output["base_backup_information"].update( dict( resources_saved=pretty_size(data["est_dedup_size"]), resources_saved_bytes=int(data["est_dedup_size"]), resources_saved_percentage="{percent:.2%}".format( percent=data["deduplication_ratio"] ), ) ) wal_comp_ratio = data.get("wal_compression_ratio", 0) if wal_comp_ratio > 0: output["base_backup_information"].update( dict( wal_compression_ratio="{percent:.2%}".format( percent=wal_comp_ratio ) ) ) cp_time = data.get("copy_time", 0) ans_time = data.get("analysis_time", 0) est_throughput = data.get("estimated_throughput") num_workers = data.get("number_of_workers", 1) copy_stats = data.get("copy_stats") or {} if copy_stats and not cp_time: cp_time = copy_stats.get("copy_time", 0) ans_time = copy_stats.get("analysis_time", 0) num_workers = copy_stats.get("number_of_workers", 1) if cp_time: output["base_backup_information"].update( dict( copy_time=human_readable_timedelta( datetime.timedelta(seconds=cp_time) ), copy_time_seconds=cp_time, ) ) if ans_time >= 1: output["base_backup_information"].update( dict( analysis_time=human_readable_timedelta( datetime.timedelta(seconds=ans_time) ), analysis_time_seconds=ans_time, ) ) if est_throughput is None: est_throughput = data["deduplicated_size"] / cp_time est_througput_output = "{}/s".format(pretty_size(est_throughput)) output["base_backup_information"].update( dict( throughput=est_througput_output, throughput_bytes=int(est_throughput), ) ) if num_workers: output["base_backup_information"].update( dict( number_of_workers=num_workers, ) ) # Tablespace information if data["tablespaces"]: for item in data["tablespaces"]: output["tablespaces"].append( dict(name=item.name, location=item.location, oid=item.oid) ) # Backups catalog information previous_backup_id = data.setdefault("previous_backup_id", "not available") next_backup_id = data.setdefault("next_backup_id", "not available") output["catalog_information"] = { "retention_policy": data["retention_policy_status"] or "not enforced", "previous_backup": previous_backup_id or "- (this is the oldest base backup)", "next_backup": next_backup_id or "- (this is the latest base backup)", } if backup_type == "incremental": output["catalog_information"].update( dict( root_backup_id=data["root_backup_id"], parent_backup_id=data["parent_backup_id"], chain_size=data["chain_size"], ) ) if data["mode"] == "postgres": children_bkp_ids = None if data["children_backup_ids"]: children_bkp_ids = data["children_backup_ids"].split(",") output["catalog_information"].update( dict( children_backup_ids=children_bkp_ids, ) ) # WAL information wal_output = output["wal_information"] = dict( no_of_files=data["wal_until_next_num"], disk_usage=pretty_size(data["wal_until_next_size"]), disk_usage_bytes=data["wal_until_next_size"], wal_rate=0, wal_rate_per_second=0, compression_ratio=0, last_available=data["wal_last"], timelines=[], ) # TODO: move the following calculations in a separate function # or upstream (backup_ext_info?) so that they are shared with # console output. if data["wals_per_second"] > 0: wal_output["wal_rate"] = "%0.2f/hour" % (data["wals_per_second"] * 3600) wal_output["wal_rate_per_second"] = data["wals_per_second"] if data["wal_until_next_compression_ratio"] > 0: wal_output["compression_ratio"] = "{percent:.2%}".format( percent=data["wal_until_next_compression_ratio"] ) if data["children_timelines"]: wal_output["_WARNING"] = ( "WAL information is inaccurate \ due to multiple timelines interacting with \ this backup" ) for history in data["children_timelines"]: wal_output["timelines"].append(str(history.tli)) # Snapshots information if "snapshots_info" in data and data["snapshots_info"]: output["snapshots_info"] = data["snapshots_info"] else: if data["error"]: output["error"] = data["error"] def init_status(self, server_name): """ Init the status command :param str server_name: the server we are start listing """ if not hasattr(self, "json_output"): self.json_output = {} self.json_output[server_name] = {} def result_status(self, server_name, status, description, message): """ Record a result line of a server status command and output it as INFO :param str server_name: the server is being checked :param str status: the returned status code :param str description: the returned status description :param str,object message: status message. It will be converted to str """ self.json_output[server_name][status] = dict( description=description, message=str(message) ) def init_replication_status(self, server_name, minimal=False): """ Init the 'standby-status' command :param str server_name: the server we are start listing :param str minimal: minimal output """ if not hasattr(self, "json_output"): self.json_output = {} self.json_output[server_name] = {} self.minimal = minimal def result_replication_status(self, server_name, target, server_lsn, standby_info): """ Record a result line of a server status command and output it as INFO :param str server_name: the replication server :param str target: all|hot-standby|wal-streamer :param str server_lsn: server's current lsn :param StatReplication standby_info: status info of a standby """ if target == "hot-standby": title = "hot standby servers" elif target == "wal-streamer": title = "WAL streamers" else: title = "streaming clients" title_key = self._mangle_key(title) if title_key not in self.json_output[server_name]: self.json_output[server_name][title_key] = [] self.json_output[server_name]["server_lsn"] = server_lsn if server_lsn else None if standby_info is not None and not len(standby_info): self.json_output[server_name]["standby_info"] = "No %s attached" % title return self.json_output[server_name][title_key] = [] # Minimal output if self.minimal: for idx, standby in enumerate(standby_info): if not standby.replay_lsn: # WAL streamer self.json_output[server_name][title_key].append( dict( user_name=standby.usename, client_addr=standby.client_addr or "socket", sent_lsn=standby.sent_lsn, write_lsn=standby.write_lsn, sync_priority=standby.sync_priority, application_name=standby.application_name, ) ) else: # Standby self.json_output[server_name][title_key].append( dict( sync_state=standby.sync_state[0].upper(), user_name=standby.usename, client_addr=standby.client_addr or "socket", sent_lsn=standby.sent_lsn, flush_lsn=standby.flush_lsn, replay_lsn=standby.replay_lsn, sync_priority=standby.sync_priority, application_name=standby.application_name, ) ) else: for idx, standby in enumerate(standby_info): self.json_output[server_name][title_key].append({}) json_output = self.json_output[server_name][title_key][idx] # Calculate differences in bytes lsn_diff = dict( sent=diff_lsn(standby.sent_lsn, standby.current_lsn), write=diff_lsn(standby.write_lsn, standby.current_lsn), flush=diff_lsn(standby.flush_lsn, standby.current_lsn), replay=diff_lsn(standby.replay_lsn, standby.current_lsn), ) # Determine the sync stage of the client sync_stage = None if not standby.replay_lsn: client_type = "WAL streamer" max_level = 3 else: client_type = "standby" max_level = 5 # Only standby can replay WAL info if lsn_diff["replay"] == 0: sync_stage = "5/5 Hot standby (max)" elif lsn_diff["flush"] == 0: sync_stage = "4/5 2-safe" # remote flush # If not yet done, set the sync stage if not sync_stage: if lsn_diff["write"] == 0: sync_stage = "3/%s Remote write" % max_level elif lsn_diff["sent"] == 0: sync_stage = "2/%s WAL Sent (min)" % max_level else: sync_stage = "1/%s 1-safe" % max_level # Synchronous standby if getattr(standby, "sync_priority", None) > 0: json_output["name"] = "#%s %s %s" % ( standby.sync_priority, standby.sync_state.capitalize(), client_type, ) # Asynchronous standby else: json_output["name"] = "%s %s" % ( standby.sync_state.capitalize(), client_type, ) json_output["application_name"] = standby.application_name json_output["sync_stage"] = sync_stage if getattr(standby, "client_addr", None): json_output.update( dict( communication="TCP/IP", ip_address=standby.client_addr, port=standby.client_port, host=standby.client_hostname or None, ) ) else: json_output["communication"] = "Unix domain socket" json_output.update( dict( user_name=standby.usename, current_state=standby.state, current_sync_state=standby.sync_state, ) ) if getattr(standby, "slot_name", None): json_output["replication_slot"] = standby.slot_name json_output.update( dict( wal_sender_pid=standby.pid, started_at=standby.backend_start.isoformat(sep=" "), ) ) if getattr(standby, "backend_xmin", None): json_output["standbys_xmin"] = standby.backend_xmin or None for lsn in lsn_diff.keys(): standby_key = lsn + "_lsn" if getattr(standby, standby_key, None): json_output.update( { lsn + "_lsn": getattr(standby, standby_key), lsn + "_lsn_diff": pretty_size(lsn_diff[lsn]), lsn + "_lsn_diff_bytes": lsn_diff[lsn], } ) def init_list_server(self, server_name, minimal=False): """ Init the list-servers command :param str server_name: the server we are listing """ self.json_output[server_name] = {} self.minimal = minimal def result_list_server(self, server_name, description=None): """ Output a result line of a list-servers command :param str server_name: the server is being checked :param str,None description: server description if applicable """ self.json_output[server_name] = dict(description=description) def init_show_server(self, server_name, description=None): """ Init the show-servers command output method :param str server_name: the server we are displaying :param str,None description: server description if applicable """ self.json_output[server_name] = dict(description=description) def result_show_server(self, server_name, server_info): """ Output the results of the show-servers command :param str server_name: the server we are displaying :param dict server_info: a dictionary containing the info to display """ for status, message in sorted(server_info.items()): if not isinstance(message, (int, str, bool, list, dict, type(None))): message = str(message) # Prevent null values overriding existing values if message is None and status in self.json_output[server_name]: continue self.json_output[server_name][status] = message def init_check_wal_archive(self, server_name): """ Init the check-wal-archive command output method :param str server_name: the server we are displaying """ self.json_output[server_name] = {} def result_check_wal_archive(self, server_name): """ Output the results of the check-wal-archive command :param str server_name: the server we are displaying """ self.json_output[server_name] = ( "WAL archive check for server %s passed" % server_name ) class NagiosOutputWriter(ConsoleOutputWriter): """ Nagios output writer. This writer doesn't output anything to console. On close it writes a nagios-plugin compatible status """ def _out(self, message, args): """ Do not print anything on standard output """ def _err(self, message, args): """ Do not print anything on standard error """ def _parse_check_results(self): """ Parse the check results and return the servers checked and any issues. :return tuple: a tuple containing a list of checked servers, a list of all issues found and a list of additional performance detail. """ # List of all servers that have been checked servers = [] # List of servers reporting issues issues = [] # Nagios performance data perf_detail = [] for item in self.result_check_list: # Keep track of all the checked servers if item["server_name"] not in servers: servers.append(item["server_name"]) # Keep track of the servers with issues if not item["status"] and item["server_name"] not in issues: issues.append(item["server_name"]) # Build the performance data list if item["check"] == "backup minimum size": perf_detail.append( "%s=%dB" % (item["server_name"], int(item["perfdata"])) ) if item["check"] == "wal size": perf_detail.append( "%s_wals=%dB" % (item["server_name"], int(item["perfdata"])) ) return servers, issues, perf_detail def _summarise_server_issues(self, issues): """ Converts the supplied list of issues into a printable summary. :return tuple: A tuple where the first element is a string summarising each server with issues and the second element is a string containing the details of all failures for each server. """ fail_summary = [] details = [] for server in issues: # Join all the issues for a server. Output format is in the # form: # " FAILED: , ... " # All strings will be concatenated into the $SERVICEOUTPUT$ # macro of the Nagios output server_fail = "%s FAILED: %s" % ( server, ", ".join( [ item["check"] for item in self.result_check_list if item["server_name"] == server and not item["status"] ] ), ) fail_summary.append(server_fail) # Prepare an array with the detailed output for # the $LONGSERVICEOUTPUT$ macro of the Nagios output # line format: # .: FAILED # .: FAILED (Hint if present) # : FAILED # ..... for issue in self.result_check_list: if issue["server_name"] == server and not issue["status"]: fail_detail = "%s.%s: FAILED" % (server, issue["check"]) if issue["hint"]: fail_detail += " (%s)" % issue["hint"] details.append(fail_detail) return fail_summary, details def _print_check_failure(self, servers, issues, perf_detail): """Prints the output for a failed check.""" # Generate the performance data message - blank string if no perf detail perf_detail_message = perf_detail and "|%s" % " ".join(perf_detail) or "" fail_summary, details = self._summarise_server_issues(issues) # Append the summary of failures to the first line of the output # using * as delimiter if len(servers) == 1: print( "BARMAN CRITICAL - server %s has issues * %s%s" % (servers[0], " * ".join(fail_summary), perf_detail_message) ) else: print( "BARMAN CRITICAL - %d server out of %d have issues * " "%s%s" % ( len(issues), len(servers), " * ".join(fail_summary), perf_detail_message, ) ) # add the detailed list to the output for issue in details: print(issue) def _print_check_success(self, servers, issues=None, perf_detail=None): """Prints the output for a successful check.""" if issues is None: issues = [] # Generate the issues message - blank string if no issues issues_message = "".join([" * IGNORING: %s" % issue for issue in issues]) # Generate the performance data message - blank string if no perf detail perf_detail_message = perf_detail and "|%s" % " ".join(perf_detail) or "" # Some issues, but only in skipped server good = [item for item in servers if item not in issues] # Display the output message for a single server check if len(good) == 0: print("BARMAN OK - No server configured%s" % issues_message) elif len(good) == 1: print( "BARMAN OK - Ready to serve the Espresso backup " "for %s%s%s" % (good[0], issues_message, perf_detail_message) ) else: # Display the output message for several servers, using # '*' as delimiter print( "BARMAN OK - Ready to serve the Espresso backup " "for %d servers * %s%s%s" % (len(good), " * ".join(good), issues_message, perf_detail_message) ) def close(self): """ Display the result of a check run as expected by Nagios. Also set the exit code as 2 (CRITICAL) in case of errors """ global error_occurred, error_exit_code servers, issues, perf_detail = self._parse_check_results() # Global error (detected at configuration level) if len(issues) == 0 and error_occurred: print("BARMAN CRITICAL - Global configuration errors") error_exit_code = 2 return if len(issues) > 0 and error_occurred: self._print_check_failure(servers, issues, perf_detail) error_exit_code = 2 else: self._print_check_success(servers, issues, perf_detail) #: This dictionary acts as a registry of available OutputWriters AVAILABLE_WRITERS = { "console": ConsoleOutputWriter, "json": JsonOutputWriter, # nagios is not registered as it isn't a general purpose output writer # 'nagios': NagiosOutputWriter, } #: The default OutputWriter DEFAULT_WRITER = "console" #: the current active writer. Initialized according DEFAULT_WRITER on load _writer = AVAILABLE_WRITERS[DEFAULT_WRITER]() barman-3.13.3/barman/wal_archiver.py0000644000175100001660000012173115002177425015512 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see import collections import filecmp import logging import os import shutil from abc import ABCMeta, abstractmethod from distutils.version import LooseVersion as Version from glob import glob from barman import output, xlog from barman.command_wrappers import CommandFailedException, PgReceiveXlog from barman.exceptions import ( AbortedRetryHookScript, ArchiverFailure, DuplicateWalFile, MatchingDuplicateWalFile, ) from barman.hooks import HookScriptRunner, RetryHookScriptRunner from barman.infofile import WalFileInfo from barman.remote_status import RemoteStatusMixin from barman.utils import fsync_dir, fsync_file, mkpath, with_metaclass from barman.xlog import is_partial_file _logger = logging.getLogger(__name__) class WalArchiverQueue(list): def __init__(self, items, errors=None, skip=None, batch_size=0): """ A WalArchiverQueue is a list of WalFileInfo which has two extra attribute list: * errors: containing a list of unrecognized files * skip: containing a list of skipped files. It also stores batch run size information in case it is requested by configuration, in order to limit the number of WAL files that are processed in a single run of the archive-wal command. :param items: iterable from which initialize the list :param batch_size: size of the current batch run (0=unlimited) :param errors: an optional list of unrecognized files :param skip: an optional list of skipped files """ super(WalArchiverQueue, self).__init__(items) self.skip = [] self.errors = [] if skip is not None: self.skip = skip if errors is not None: self.errors = errors # Normalises batch run size if batch_size > 0: self.batch_size = batch_size else: self.batch_size = 0 @property def size(self): """ Number of valid WAL segments waiting to be processed (in total) :return int: total number of valid WAL files """ return len(self) @property def run_size(self): """ Number of valid WAL files to be processed in this run - takes in consideration the batch size :return int: number of valid WAL files for this batch run """ # In case a batch size has been explicitly specified # (i.e. batch_size > 0), returns the minimum number between # batch size and the queue size. Otherwise, simply # returns the total queue size (unlimited batch size). if self.batch_size > 0: return min(self.size, self.batch_size) return self.size class WalArchiver(with_metaclass(ABCMeta, RemoteStatusMixin)): """ Base class for WAL archiver objects """ def __init__(self, backup_manager, name): """ Base class init method. :param backup_manager: The backup manager :param name: The name of this archiver :return: """ self.backup_manager = backup_manager self.server = backup_manager.server self.config = backup_manager.config self.name = name super(WalArchiver, self).__init__() def receive_wal(self, reset=False): """ Manage reception of WAL files. Does nothing by default. Some archiver classes, like the StreamingWalArchiver, have a full implementation. :param bool reset: When set, resets the status of receive-wal :raise ArchiverFailure: when something goes wrong """ def archive(self, verbose=True): """ Archive WAL files, discarding duplicates or those that are not valid. :param boolean verbose: Flag for verbose output """ compressor = self.backup_manager.compression_manager.get_default_compressor() processed = 0 header = "Processing xlog segments from %s for %s" % ( self.name, self.config.name, ) # Get the next batch of WAL files to be processed batch = self.get_next_batch() # Analyse the batch and properly log the information if batch.size: if batch.size > batch.run_size: # Batch mode enabled _logger.info( "Found %s xlog segments from %s for %s." " Archive a batch of %s segments in this run.", batch.size, self.name, self.config.name, batch.run_size, ) header += " (batch size: %s)" % batch.run_size else: # Single run mode (traditional) _logger.info( "Found %s xlog segments from %s for %s." " Archive all segments in one run.", batch.size, self.name, self.config.name, ) else: _logger.info( "No xlog segments found from %s for %s.", self.name, self.config.name ) # Print the header (verbose mode) if verbose: output.info(header, log=False) # Loop through all available WAL files for wal_info in batch: # Print the header (non verbose mode) if not processed and not verbose: output.info(header, log=False) # Exit when archive batch size is reached if processed >= batch.run_size: _logger.debug( "Batch size reached (%s) - Exit %s process for %s", batch.batch_size, self.name, self.config.name, ) break processed += 1 # Report to the user the WAL file we are archiving output.info("\t%s", wal_info.name, log=False) _logger.info( "Archiving segment %s of %s from %s: %s/%s", processed, batch.run_size, self.name, self.config.name, wal_info.name, ) # Archive the WAL file try: self.archive_wal(compressor, wal_info) except MatchingDuplicateWalFile: # We already have this file. Simply unlink the file. os.unlink(wal_info.orig_filename) continue except DuplicateWalFile: self.server.move_wal_file_to_errors_directory( wal_info.orig_filename, wal_info.name, "duplicate" ) output.info( "\tError: %s is already present in server %s. " "File moved to errors directory.", wal_info.name, self.config.name, ) continue except AbortedRetryHookScript as e: _logger.warning( "Archiving of %s/%s aborted by " "pre_archive_retry_script." "Reason: %s" % (self.config.name, wal_info.name, e) ) return if processed: _logger.debug( "Archived %s out of %s xlog segments from %s for %s", processed, batch.size, self.name, self.config.name, ) elif verbose: output.info("\tno file found", log=False) if batch.errors: output.info( "Some unknown objects have been found while " "processing xlog segments for %s. " "Objects moved to errors directory:", self.config.name, log=False, ) # Log unexpected files _logger.warning( "Archiver is about to move %s unexpected file(s) " "to errors directory for %s from %s", len(batch.errors), self.config.name, self.name, ) for error in batch.errors: basename = os.path.basename(error) output.info("\t%s", basename, log=False) # Print informative log line. _logger.warning( "Moving unexpected file for %s from %s: %s", self.config.name, self.name, basename, ) self.server.move_wal_file_to_errors_directory( error, basename, "unknown" ) def archive_wal(self, compressor, wal_info): """ Archive a WAL segment and update the wal_info object :param compressor: the compressor for the file (if any) :param WalFileInfo wal_info: the WAL file is being processed """ src_file = wal_info.orig_filename src_dir = os.path.dirname(src_file) dst_file = wal_info.fullpath(self.server) tmp_file = dst_file + ".tmp" dst_dir = os.path.dirname(dst_file) comp_manager = self.backup_manager.compression_manager error = None try: # Run the pre_archive_script if present. script = HookScriptRunner(self.backup_manager, "archive_script", "pre") script.env_from_wal_info(wal_info, src_file) script.run() # Run the pre_archive_retry_script if present. retry_script = RetryHookScriptRunner( self.backup_manager, "archive_retry_script", "pre" ) retry_script.env_from_wal_info(wal_info, src_file) retry_script.run() # Check if destination already exists if os.path.exists(dst_file): src_uncompressed = src_file dst_uncompressed = dst_file dst_info = comp_manager.get_wal_file_info(dst_file) try: if dst_info.compression is not None: dst_uncompressed = dst_file + ".uncompressed" comp_manager.get_compressor(dst_info.compression).decompress( dst_file, dst_uncompressed ) if wal_info.compression: src_uncompressed = src_file + ".uncompressed" comp_manager.get_compressor(wal_info.compression).decompress( src_file, src_uncompressed ) # Directly compare files. # When the files are identical # raise a MatchingDuplicateWalFile exception, # otherwise raise a DuplicateWalFile exception. if filecmp.cmp(dst_uncompressed, src_uncompressed): raise MatchingDuplicateWalFile(wal_info) else: raise DuplicateWalFile(wal_info) finally: if src_uncompressed != src_file: os.unlink(src_uncompressed) if dst_uncompressed != dst_file: os.unlink(dst_uncompressed) mkpath(dst_dir) # Compress the file only if not already compressed if compressor and not wal_info.compression: compressor.compress(src_file, tmp_file) # Perform the real filesystem operation with the xlogdb lock taken. # This makes the operation atomic from the xlogdb file POV with self.server.xlogdb("a") as fxlogdb: if compressor and not wal_info.compression: shutil.copystat(src_file, tmp_file) os.rename(tmp_file, dst_file) os.unlink(src_file) # Update wal_info stat = os.stat(dst_file) wal_info.size = stat.st_size wal_info.compression = compressor.compression else: # Try to atomically rename the file. If successful, # the renaming will be an atomic operation # (this is a POSIX requirement). try: os.rename(src_file, dst_file) except OSError: # Source and destination are probably on different # filesystems shutil.copy2(src_file, tmp_file) os.rename(tmp_file, dst_file) os.unlink(src_file) # At this point the original file has been removed wal_info.orig_filename = None # Execute fsync() on the archived WAL file fsync_file(dst_file) # Execute fsync() on the archived WAL containing directory fsync_dir(dst_dir) # Execute fsync() also on the incoming directory fsync_dir(src_dir) # Updates the information of the WAL archive with # the latest segments fxlogdb.write(wal_info.to_xlogdb_line()) # flush and fsync for every line fxlogdb.flush() os.fsync(fxlogdb.fileno()) except Exception as e: # In case of failure save the exception for the post scripts error = e raise # Ensure the execution of the post_archive_retry_script and # the post_archive_script finally: # Run the post_archive_retry_script if present. try: retry_script = RetryHookScriptRunner( self, "archive_retry_script", "post" ) retry_script.env_from_wal_info(wal_info, dst_file, error) retry_script.run() except AbortedRetryHookScript as e: # Ignore the ABORT_STOP as it is a post-hook operation _logger.warning( "Ignoring stop request after receiving " "abort (exit code %d) from post-archive " "retry hook script: %s", e.hook.exit_status, e.hook.script, ) # Run the post_archive_script if present. script = HookScriptRunner(self, "archive_script", "post", error) script.env_from_wal_info(wal_info, dst_file) script.run() @abstractmethod def get_next_batch(self): """ Return a WalArchiverQueue containing the WAL files to be archived. :rtype: WalArchiverQueue """ @abstractmethod def check(self, check_strategy): """ Perform specific checks for the archiver - invoked by server.check_postgres :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ @abstractmethod def status(self): """ Set additional status info - invoked by Server.status() """ @staticmethod def summarise_error_files(error_files): """ Summarise a error files list :param list[str] error_files: Error files list to summarise :return str: A summary, None if there are no error files """ if not error_files: return None # The default value for this dictionary will be 0 counters = collections.defaultdict(int) # Count the file types for name in error_files: if name.endswith(".error"): counters["not relevant"] += 1 elif name.endswith(".duplicate"): counters["duplicates"] += 1 elif name.endswith(".unknown"): counters["unknown"] += 1 else: counters["unknown failure"] += 1 # Return a summary list of the form: "item a: 2, item b: 5" return ", ".join("%s: %s" % entry for entry in counters.items()) class FileWalArchiver(WalArchiver): """ Manager of file-based WAL archiving operations (aka 'log shipping'). """ def __init__(self, backup_manager): super(FileWalArchiver, self).__init__(backup_manager, "file archival") def fetch_remote_status(self): """ Returns the status of the FileWalArchiver. This method does not raise any exception in case of errors, but set the missing values to None in the resulting dictionary. :rtype: dict[str, None|str] """ result = dict.fromkeys(["archive_mode", "archive_command"], None) postgres = self.server.postgres # If Postgres is not available we cannot detect anything if not postgres: return result # Query the database for 'archive_mode' and 'archive_command' result["archive_mode"] = postgres.get_setting("archive_mode") result["archive_command"] = postgres.get_setting("archive_command") # Add pg_stat_archiver statistics if the view is supported pg_stat_archiver = postgres.get_archiver_stats() if pg_stat_archiver is not None: result.update(pg_stat_archiver) return result def get_next_batch(self): """ Returns the next batch of WAL files that have been archived through a PostgreSQL's 'archive_command' (in the 'incoming' directory) :return: WalArchiverQueue: list of WAL files """ # Get the batch size from configuration (0 = unlimited) batch_size = self.config.archiver_batch_size # List and sort all files in the incoming directory # IMPORTANT: the list is sorted, and this allows us to know that the # WAL stream we have is monotonically increasing. That allows us to # verify that a backup has all the WALs required for the restore. file_names = glob(os.path.join(self.config.incoming_wals_directory, "*")) file_names.sort() # Process anything that looks like a valid WAL file. Anything # else is treated like an error/anomaly files = [] errors = [] for file_name in file_names: # Ignore temporary files if file_name.endswith(".tmp"): continue if xlog.is_any_xlog_file(file_name) and os.path.isfile(file_name): files.append(file_name) else: errors.append(file_name) # Build the list of WalFileInfo wal_files = [ WalFileInfo.from_file(f, self.backup_manager.compression_manager) for f in files ] return WalArchiverQueue(wal_files, batch_size=batch_size, errors=errors) def check(self, check_strategy): """ Perform additional checks for FileWalArchiver - invoked by server.check_postgres :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ check_strategy.init_check("archive_mode") remote_status = self.get_remote_status() # If archive_mode is None, there are issues connecting to PostgreSQL if remote_status["archive_mode"] is None: return # Check archive_mode parameter: must be on if remote_status["archive_mode"] in ("on", "always"): check_strategy.result(self.config.name, True) else: msg = "please set it to 'on'" if self.server.postgres.server_version >= 90500: msg += " or 'always'" check_strategy.result(self.config.name, False, hint=msg) check_strategy.init_check("archive_command") if ( remote_status["archive_command"] and remote_status["archive_command"] != "(disabled)" ): check_strategy.result(self.config.name, True, check="archive_command") # Report if the archiving process works without issues. # Skip if the archive_command check fails # It can be None if PostgreSQL is older than 9.4 if remote_status.get("is_archiving") is not None: check_strategy.result( self.config.name, remote_status["is_archiving"], check="continuous archiving", ) else: check_strategy.result( self.config.name, False, hint="please set it accordingly to documentation", ) def status(self): """ Set additional status info - invoked by Server.status() """ # We need to get full info here from the server remote_status = self.server.get_remote_status() # If archive_mode is None, there are issues connecting to PostgreSQL if remote_status["archive_mode"] is None: return output.result( "status", self.config.name, "archive_command", "PostgreSQL 'archive_command' setting", remote_status["archive_command"] or "FAILED (please set it accordingly to documentation)", ) last_wal = remote_status.get("last_archived_wal") # If PostgreSQL is >= 9.4 we have the last_archived_time if last_wal and remote_status.get("last_archived_time"): last_wal += ", at %s" % (remote_status["last_archived_time"].ctime()) output.result( "status", self.config.name, "last_archived_wal", "Last archived WAL", last_wal or "No WAL segment shipped yet", ) # Set output for WAL archive failures (PostgreSQL >= 9.4) if remote_status.get("failed_count") is not None: remote_fail = str(remote_status["failed_count"]) if int(remote_status["failed_count"]) > 0: remote_fail += " (%s at %s)" % ( remote_status["last_failed_wal"], remote_status["last_failed_time"].ctime(), ) output.result( "status", self.config.name, "failed_count", "Failures of WAL archiver", remote_fail, ) # Add hourly archive rate if available (PostgreSQL >= 9.4) and > 0 if remote_status.get("current_archived_wals_per_second"): output.result( "status", self.config.name, "server_archived_wals_per_hour", "Server WAL archiving rate", "%0.2f/hour" % (3600 * remote_status["current_archived_wals_per_second"]), ) class StreamingWalArchiver(WalArchiver): """ Object used for the management of streaming WAL archive operation. """ def __init__(self, backup_manager): super(StreamingWalArchiver, self).__init__(backup_manager, "streaming") def fetch_remote_status(self): """ Execute checks for replication-based wal archiving This method does not raise any exception in case of errors, but set the missing values to None in the resulting dictionary. :rtype: dict[str, None|str] """ remote_status = dict.fromkeys( ( "pg_receivexlog_compatible", "pg_receivexlog_installed", "pg_receivexlog_path", "pg_receivexlog_supports_slots", "pg_receivexlog_synchronous", "pg_receivexlog_version", ), None, ) # Test pg_receivexlog existence version_info = PgReceiveXlog.get_version_info(self.server.path) if version_info["full_path"]: remote_status["pg_receivexlog_installed"] = True remote_status["pg_receivexlog_path"] = version_info["full_path"] remote_status["pg_receivexlog_version"] = version_info["full_version"] pgreceivexlog_version = version_info["major_version"] else: remote_status["pg_receivexlog_installed"] = False return remote_status # Retrieve the PostgreSQL version pg_version = None if self.server.streaming is not None: pg_version = self.server.streaming.server_major_version # If one of the version is unknown we cannot compare them if pgreceivexlog_version is None or pg_version is None: return remote_status # pg_version is not None so transform into a Version object # for easier comparison between versions pg_version = Version(pg_version) # Set conservative default values (False) for modern features remote_status["pg_receivexlog_compatible"] = False remote_status["pg_receivexlog_supports_slots"] = False remote_status["pg_receivexlog_synchronous"] = False # pg_receivexlog 9.2 is compatible only with PostgreSQL 9.2. if "9.2" == pg_version == pgreceivexlog_version: remote_status["pg_receivexlog_compatible"] = True # other versions are compatible with lesser versions of PostgreSQL # WARNING: The development versions of `pg_receivexlog` are considered # higher than the stable versions here, but this is not an issue # because it accepts everything that is less than # the `pg_receivexlog` version(e.g. '9.6' is less than '9.6devel') elif "9.2" < pg_version <= pgreceivexlog_version: # At least PostgreSQL 9.3 is required here remote_status["pg_receivexlog_compatible"] = True # replication slots are supported starting from version 9.4 if "9.4" <= pg_version <= pgreceivexlog_version: remote_status["pg_receivexlog_supports_slots"] = True # Synchronous WAL streaming requires replication slots # and pg_receivexlog >= 9.5 if "9.4" <= pg_version and "9.5" <= pgreceivexlog_version: remote_status["pg_receivexlog_synchronous"] = self._is_synchronous() return remote_status def receive_wal(self, reset=False): """ Creates a PgReceiveXlog object and issues the pg_receivexlog command for a specific server :param bool reset: When set reset the status of receive-wal :raise ArchiverFailure: when something goes wrong """ # Ensure the presence of the destination directory mkpath(self.config.streaming_wals_directory) # Execute basic sanity checks on PostgreSQL connection streaming_status = self.server.streaming.get_remote_status() if streaming_status["streaming_supported"] is None: raise ArchiverFailure( "failed opening the PostgreSQL streaming connection " "for server %s" % (self.config.name) ) elif not streaming_status["streaming_supported"]: raise ArchiverFailure( "PostgreSQL version too old (%s < 9.2)" % self.server.streaming.server_txt_version ) # Execute basic sanity checks on pg_receivexlog command = "pg_receivewal" if self.server.streaming.server_version < 100000: command = "pg_receivexlog" remote_status = self.get_remote_status() if not remote_status["pg_receivexlog_installed"]: raise ArchiverFailure("%s not present in $PATH" % command) if not remote_status["pg_receivexlog_compatible"]: raise ArchiverFailure( "%s version not compatible with PostgreSQL server version" % command ) # Execute sanity check on replication slot usage postgres_status = self.server.postgres.get_remote_status() if self.config.slot_name: # Check if slots are supported if not remote_status["pg_receivexlog_supports_slots"]: raise ArchiverFailure( "Physical replication slot not supported by %s " "(9.4 or higher is required)" % self.server.streaming.server_txt_version ) # Check if the required slot exists if postgres_status["replication_slot"] is None: if self.config.create_slot == "auto": if not reset: output.info( "Creating replication slot '%s'", self.config.slot_name ) self.server.create_physical_repslot() else: raise ArchiverFailure( "replication slot '%s' doesn't exist. " "Please execute " "'barman receive-wal --create-slot %s'" % (self.config.slot_name, self.config.name) ) # Check if the required slot is available elif postgres_status["replication_slot"].active: raise ArchiverFailure( "replication slot '%s' is already in use" % (self.config.slot_name,) ) # Check if is a reset request if reset: self._reset_streaming_status(postgres_status, streaming_status) return # Check the size of the .partial WAL file and truncate it if needed self._truncate_partial_file_if_needed(postgres_status["xlog_segment_size"]) # Make sure we are not wasting precious PostgreSQL resources self.server.close() _logger.info("Activating WAL archiving through streaming protocol") try: output_handler = PgReceiveXlog.make_output_handler(self.config.name + ": ") receive = PgReceiveXlog( connection=self.server.streaming, destination=self.config.streaming_wals_directory, command=remote_status["pg_receivexlog_path"], version=remote_status["pg_receivexlog_version"], app_name=self.config.streaming_archiver_name, path=self.server.path, slot_name=self.config.slot_name, synchronous=remote_status["pg_receivexlog_synchronous"], out_handler=output_handler, err_handler=output_handler, ) # Finally execute the pg_receivexlog process receive.execute() except CommandFailedException as e: # Retrieve the return code from the exception ret_code = e.args[0]["ret"] if ret_code < 0: # If the return code is negative, then pg_receivexlog # was terminated by a signal msg = "%s terminated by signal: %s" % (command, abs(ret_code)) else: # Otherwise terminated with an error msg = "%s terminated with error code: %s" % (command, ret_code) raise ArchiverFailure(msg) except KeyboardInterrupt: # This is a normal termination, so there is nothing to do beside # informing the user. output.info("SIGINT received. Terminate gracefully.") def _reset_streaming_status(self, postgres_status, streaming_status): """ Reset the status of receive-wal by removing the .partial file that is marking the current position and creating one that is current with the PostgreSQL insert location """ current_wal = xlog.location_to_xlogfile_name_offset( postgres_status["current_lsn"], streaming_status["timeline"], postgres_status["xlog_segment_size"], )["file_name"] restart_wal = current_wal if ( postgres_status["replication_slot"] and postgres_status["replication_slot"].restart_lsn ): restart_wal = xlog.location_to_xlogfile_name_offset( postgres_status["replication_slot"].restart_lsn, streaming_status["timeline"], postgres_status["xlog_segment_size"], )["file_name"] restart_path = os.path.join(self.config.streaming_wals_directory, restart_wal) restart_partial_path = restart_path + ".partial" wal_files = sorted( glob(os.path.join(self.config.streaming_wals_directory, "*")), reverse=True ) # Pick the newer file last = None for last in wal_files: if xlog.is_wal_file(last) or xlog.is_partial_file(last): break # Check if the status is already up-to-date if not last or last == restart_partial_path or last == restart_path: output.info("Nothing to do. Position of receive-wal is aligned.") return if os.path.basename(last) > current_wal: output.error( "The receive-wal position is ahead of PostgreSQL " "current WAL lsn (%s > %s)", os.path.basename(last), postgres_status["current_xlog"], ) return output.info("Resetting receive-wal directory status") if xlog.is_partial_file(last): output.info("Removing status file %s" % last) os.unlink(last) output.info("Creating status file %s" % restart_partial_path) open(restart_partial_path, "w").close() def _truncate_partial_file_if_needed(self, xlog_segment_size): """ Truncate .partial WAL file if size is not 0 or xlog_segment_size :param int xlog_segment_size: """ # Retrieve the partial list (only one is expected) partial_files = glob( os.path.join(self.config.streaming_wals_directory, "*.partial") ) # Take the last partial file, ignoring wrongly formatted file names last_partial = None for partial in partial_files: if not is_partial_file(partial): continue if not last_partial or partial > last_partial: last_partial = partial # Skip further work if there is no good partial file if not last_partial: return # If size is either 0 or wal_segment_size everything is fine... partial_size = os.path.getsize(last_partial) if partial_size == 0 or partial_size == xlog_segment_size: return # otherwise truncate the file to be empty. This is safe because # pg_receivewal pads the file to the full size before start writing. output.info( "Truncating partial file %s that has wrong size %s " "while %s was expected." % (last_partial, partial_size, xlog_segment_size) ) open(last_partial, "wb").close() def get_next_batch(self): """ Returns the next batch of WAL files that have been archived via streaming replication (in the 'streaming' directory) This method always leaves one file in the "streaming" directory, because the 'pg_receivexlog' process needs at least one file to detect the current streaming position after a restart. :return: WalArchiverQueue: list of WAL files """ # Get the batch size from configuration (0 = unlimited) batch_size = self.config.streaming_archiver_batch_size # List and sort all files in the incoming directory. # IMPORTANT: the list is sorted, and this allows us to know that the # WAL stream we have is monotonically increasing. That allows us to # verify that a backup has all the WALs required for the restore. file_names = glob(os.path.join(self.config.streaming_wals_directory, "*")) file_names.sort() # Process anything that looks like a valid WAL file, # including partial ones and history files. # Anything else is treated like an error/anomaly files = [] skip = [] errors = [] for file_name in file_names: # Ignore temporary files if file_name.endswith(".tmp"): continue # If the file doesn't exist, it has been renamed/removed while # we were reading the directory. Ignore it. if not os.path.exists(file_name): continue if not os.path.isfile(file_name): errors.append(file_name) elif xlog.is_partial_file(file_name): skip.append(file_name) elif xlog.is_any_xlog_file(file_name): files.append(file_name) else: errors.append(file_name) # In case of more than a partial file, keep the last # and treat the rest as normal files if len(skip) > 1: partials = skip[:-1] _logger.info( "Archiving partial files for server %s: %s" % (self.config.name, ", ".join([os.path.basename(f) for f in partials])) ) files.extend(partials) skip = skip[-1:] # Keep the last full WAL file in case no partial file is present elif len(skip) == 0 and files: skip.append(files.pop()) # Build the list of WalFileInfo wal_files = [WalFileInfo.from_file(f, compression=None) for f in files] return WalArchiverQueue( wal_files, batch_size=batch_size, errors=errors, skip=skip ) def check(self, check_strategy): """ Perform additional checks for StreamingWalArchiver - invoked by server.check_postgres :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ check_strategy.init_check("pg_receivexlog") # Check the version of pg_receivexlog remote_status = self.get_remote_status() check_strategy.result( self.config.name, remote_status["pg_receivexlog_installed"] ) hint = None check_strategy.init_check("pg_receivexlog compatible") if not remote_status["pg_receivexlog_compatible"]: pg_version = "Unknown" if self.server.streaming is not None: pg_version = self.server.streaming.server_txt_version hint = "PostgreSQL version: %s, pg_receivexlog version: %s" % ( pg_version, remote_status["pg_receivexlog_version"], ) check_strategy.result( self.config.name, remote_status["pg_receivexlog_compatible"], hint=hint ) # Check if pg_receivexlog is running, by retrieving a list # of running 'receive-wal' processes from the process manager. receiver_list = self.server.process_manager.list("receive-wal") # If there's at least one 'receive-wal' process running for this # server, the test is passed check_strategy.init_check("receive-wal running") if receiver_list: check_strategy.result(self.config.name, True) else: check_strategy.result( self.config.name, False, hint="See the Barman log file for more details" ) def _is_synchronous(self): """ Check if receive-wal process is eligible for synchronous replication The receive-wal process is eligible for synchronous replication if `synchronous_standby_names` is configured and contains the value of `streaming_archiver_name` :rtype: bool """ # Nothing to do if postgres connection is not working postgres = self.server.postgres if postgres is None or postgres.server_txt_version is None: return None # Check if synchronous WAL streaming can be enabled # by peeking 'synchronous_standby_names' postgres_status = postgres.get_remote_status() syncnames = postgres_status["synchronous_standby_names"] _logger.debug( "Look for '%s' in 'synchronous_standby_names': %s", self.config.streaming_archiver_name, syncnames, ) # The receive-wal process is eligible for synchronous replication # if `synchronous_standby_names` is configured and contains # the value of `streaming_archiver_name` streaming_archiver_name = self.config.streaming_archiver_name synchronous = syncnames and ( "*" in syncnames or streaming_archiver_name in syncnames ) _logger.debug( "Synchronous WAL streaming for %s: %s", streaming_archiver_name, synchronous ) return synchronous def status(self): """ Set additional status info - invoked by Server.status() """ # TODO: Add status information for WAL streaming barman-3.13.3/barman/backup.py0000644000175100001660000022366315002177425014320 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module represents a backup. """ import datetime import logging import os import shutil import tempfile from collections import defaultdict from contextlib import closing from glob import glob import dateutil.parser import dateutil.tz from barman import output, xlog from barman.annotations import AnnotationManagerFile, KeepManager, KeepManagerMixin from barman.backup_executor import ( PassiveBackupExecutor, PostgresBackupExecutor, RsyncBackupExecutor, SnapshotBackupExecutor, ) from barman.backup_manifest import BackupManifest from barman.cloud_providers import get_snapshot_interface_from_backup_info from barman.command_wrappers import PgVerifyBackup from barman.compression import CompressionManager from barman.config import BackupOptions from barman.exceptions import ( AbortedRetryHookScript, BackupException, CommandFailedException, CompressionIncompatibility, LockFileBusy, SshCommandException, UnknownBackupIdException, ) from barman.fs import unix_command_factory from barman.hooks import HookScriptRunner, RetryHookScriptRunner from barman.infofile import BackupInfo, LocalBackupInfo, WalFileInfo from barman.lockfile import ServerBackupIdLock, ServerBackupSyncLock from barman.recovery_executor import recovery_executor_factory from barman.remote_status import RemoteStatusMixin from barman.storage.local_file_manager import LocalFileManager from barman.utils import ( SHA256, force_str, fsync_dir, fsync_file, get_backup_id_from_target_lsn, get_backup_id_from_target_time, get_backup_id_from_target_tli, get_backup_info_from_name, get_last_backup_id, human_readable_timedelta, pretty_size, ) _logger = logging.getLogger(__name__) class BackupManager(RemoteStatusMixin, KeepManagerMixin): """Manager of the backup archive for a server""" DEFAULT_STATUS_FILTER = BackupInfo.STATUS_COPY_DONE DELETE_ANNOTATION = "delete_this" def __init__(self, server): """ Constructor :param server: barman.server.Server """ super(BackupManager, self).__init__(server=server) self.server = server self.config = server.config self._backup_cache = None self.compression_manager = CompressionManager(self.config, server.path) self.annotation_manager = AnnotationManagerFile( self.server.meta_directory, self.server.config.basebackups_directory ) self.executor = None try: if server.passive_node: self.executor = PassiveBackupExecutor(self) elif self.config.backup_method == "postgres": self.executor = PostgresBackupExecutor(self) elif self.config.backup_method == "local-rsync": self.executor = RsyncBackupExecutor(self, local_mode=True) elif self.config.backup_method == "snapshot": self.executor = SnapshotBackupExecutor(self) else: self.executor = RsyncBackupExecutor(self) except SshCommandException as e: self.config.update_msg_list_and_disable_server(force_str(e).strip()) @property def mode(self): """ Property defining the BackupInfo mode content """ if self.executor: return self.executor.mode return None def get_available_backups(self, status_filter=DEFAULT_STATUS_FILTER): """ Get a list of available backups :param status_filter: default DEFAULT_STATUS_FILTER. The status of the backup list returned """ # If the filter is not a tuple, create a tuple using the filter if not isinstance(status_filter, tuple): status_filter = tuple( status_filter, ) # Load the cache if necessary if self._backup_cache is None: self._load_backup_cache() # Filter the cache using the status filter tuple backups = {} for key, value in self._backup_cache.items(): if value.status in status_filter: backups[key] = value return backups def _load_backup_cache(self): """ Populate the cache of the available backups, reading information from disk. """ self._backup_cache = {} # Previous to version 3.13.2, Barman used to store the backup.info file # alongside with the base backup. While that, in general, is not a problem, # when dealing with WORM environments that could cause issues as the # base backups are expected to be stored in an immutable storage. This # code is only maintained as a fallback mechanism during a transient state # in the backup catalog, where we will find backup.info files in both # locations because of backups taken with < 3.13.2 for filename in glob("%s/*/backup.info" % self.config.basebackups_directory): backup = LocalBackupInfo(self.server, filename) self._backup_cache[backup.backup_id] = backup # In version 3.13.2, Barman changed the location of backup.info files. # That was done so we have common location for the metadata, which # should always be in a mutable storage, independently if worm_mode # is enabled or not. So, this new approach takes precedence. for filename in glob("%s/*-backup.info" % self.server.meta_directory): backup = LocalBackupInfo(self.server, filename) self._backup_cache[backup.backup_id] = backup def backup_cache_add(self, backup_info): """ Register a BackupInfo object to the backup cache. NOTE: Initialise the cache - in case it has not been done yet :param barman.infofile.BackupInfo backup_info: the object we want to register in the cache """ # Load the cache if needed if self._backup_cache is None: self._load_backup_cache() # Insert the BackupInfo object into the cache self._backup_cache[backup_info.backup_id] = backup_info def backup_cache_remove(self, backup_info): """ Remove a BackupInfo object from the backup cache This method _must_ be called after removing the object from disk. :param barman.infofile.BackupInfo backup_info: the object we want to remove from the cache """ # Nothing to do if the cache is not loaded if self._backup_cache is None: return # Remove the BackupInfo object from the backups cache del self._backup_cache[backup_info.backup_id] def get_backup(self, backup_id): """ Return the backup information for the given backup id. If the backup_id is None or backup.info file doesn't exists, it returns None. :param str|None backup_id: the ID of the backup to return :rtype: BackupInfo|None """ if backup_id is not None: # Get all the available backups from the cache available_backups = self.get_available_backups(BackupInfo.STATUS_ALL) # Return the BackupInfo if present, or None return available_backups.get(backup_id) return None @staticmethod def find_previous_backup_in( available_backups, backup_id, status_filter=DEFAULT_STATUS_FILTER ): """ Find the next backup (if any) in the supplied dict of BackupInfo objects. """ ids = sorted(available_backups.keys()) try: current = ids.index(backup_id) while current > 0: res = available_backups[ids[current - 1]] if res.status in status_filter: return res current -= 1 return None except ValueError: raise UnknownBackupIdException("Could not find backup_id %s" % backup_id) def get_previous_backup(self, backup_id, status_filter=DEFAULT_STATUS_FILTER): """ Get the previous backup (if any) in the catalog :param status_filter: default DEFAULT_STATUS_FILTER. The status of the backup returned """ if not isinstance(status_filter, tuple): status_filter = tuple(status_filter) backup = LocalBackupInfo(self.server, backup_id=backup_id) available_backups = self.get_available_backups(status_filter + (backup.status,)) return self.find_previous_backup_in(available_backups, backup_id, status_filter) @staticmethod def should_remove_wals( backup, available_backups, keep_manager, skip_wal_cleanup_if_standalone, status_filter=DEFAULT_STATUS_FILTER, ): """ Determine whether we should remove the WALs for the specified backup. Returns the following tuple: - `(bool should_remove_wals, list wal_ranges_to_protect)` Where `should_remove_wals` is a boolean which is True if the WALs associated with this backup should be removed and False otherwise. `wal_ranges_to_protect` is a list of `(begin_wal, end_wal)` tuples which define *inclusive* ranges where any matching WAL should not be deleted. The rules for determining whether we should remove WALs are as follows: 1. If there is no previous backup then we can clean up the WALs. 2. If there is a previous backup and it has no keep annotation then do not clean up the WALs. We need to allow PITR from that older backup to the current time. 3. If there is a previous backup and it has a keep target of "full" then do nothing. We need to allow PITR from that keep:full backup to the current time. 4. If there is a previous backup and it has a keep target of "standalone": a. If that previous backup is the oldest backup then delete WALs up to the begin_wal of the next backup except for WALs which are >= begin_wal and <= end_wal of the keep:standalone backup - we can therefore add `(begin_wal, end_wal)` to `wal_ranges_to_protect` and return True. b. If that previous backup is not the oldest backup then we add the `(begin_wal, end_wal)` to `wal_ranges_to_protect` and go to 2 above. We will either end up returning False, because we hit a backup with keep:full or no keep annotation, or all backups to the oldest backup will be keep:standalone in which case we will delete up to the begin_wal of the next backup, preserving the WALs needed by each keep:standalone backups by adding them to `wal_ranges_to_protect`. This is a static method so it can be re-used by barman-cloud which will pass in its own dict of available_backups. :param BackupInfo backup_info: The backup for which we are determining whether we can clean up WALs. :param dict[str,BackupInfo] available_backups: A dict of BackupInfo objects keyed by backup_id which represent all available backups for the current server. :param KeepManagerMixin keep_manager: An object implementing the KeepManagerMixin interface. This will be either a BackupManager (in barman) or a CloudBackupCatalog (in barman-cloud). :param bool skip_wal_cleanup_if_standalone: If set to True then we should skip removing WALs for cases where all previous backups are standalone archival backups (i.e. they have a keep annotation of "standalone"). The default is True. It is only safe to set this to False if the backup is being deleted due to a retention policy rather than a `barman delete` command. :param status_filter: The status of the backups to check when determining if we should remove WALs. default to DEFAULT_STATUS_FILTER. """ previous_backup = BackupManager.find_previous_backup_in( available_backups, backup.backup_id, status_filter=status_filter ) wal_ranges_to_protect = [] while True: if previous_backup is None: # No previous backup so we should remove WALs and return any WAL ranges # we have found so far return True, wal_ranges_to_protect elif ( keep_manager.get_keep_target(previous_backup.backup_id) == KeepManager.TARGET_STANDALONE ): # A previous backup exists and it is a standalone backup - if we have # been asked to skip wal cleanup on standalone backups then we # should not remove wals if skip_wal_cleanup_if_standalone: return False, [] # Otherwise we add to the WAL ranges to protect wal_ranges_to_protect.append( (previous_backup.begin_wal, previous_backup.end_wal) ) # and continue iterating through previous backups until we find either # no previous backup or a non-standalone backup previous_backup = BackupManager.find_previous_backup_in( available_backups, previous_backup.backup_id, status_filter=status_filter, ) continue else: # A previous backup exists and it is not a standalone backup so we # must not remove any WALs and we can discard any wal_ranges_to_protect # since they are no longer relevant return False, [] @staticmethod def find_next_backup_in( available_backups, backup_id, status_filter=DEFAULT_STATUS_FILTER ): """ Find the next backup (if any) in the supplied dict of BackupInfo objects. """ ids = sorted(available_backups.keys()) try: current = ids.index(backup_id) while current < (len(ids) - 1): res = available_backups[ids[current + 1]] if res.status in status_filter: return res current += 1 return None except ValueError: raise UnknownBackupIdException("Could not find backup_id %s" % backup_id) def get_next_backup(self, backup_id, status_filter=DEFAULT_STATUS_FILTER): """ Get the next backup (if any) in the catalog :param status_filter: default DEFAULT_STATUS_FILTER. The status of the backup returned """ if not isinstance(status_filter, tuple): status_filter = tuple(status_filter) backup = LocalBackupInfo(self.server, backup_id=backup_id) available_backups = self.get_available_backups(status_filter + (backup.status,)) return self.find_next_backup_in(available_backups, backup_id, status_filter) def get_last_backup_id(self, status_filter=DEFAULT_STATUS_FILTER): """ Get the id of the latest/last backup in the catalog (if exists) :param status_filter: The status of the backup to return, default to :attr:`DEFAULT_STATUS_FILTER`. :return str|None: ID of the backup """ available_backups = self.get_available_backups(status_filter).values() backup_id = get_last_backup_id(available_backups) return backup_id def get_last_full_backup_id(self, status_filter=DEFAULT_STATUS_FILTER): """ Get the id of the latest/last FULL backup in the catalog (if exists) :param status_filter: The status of the backup to return, default to :attr:`DEFAULT_STATUS_FILTER`. :return str|None: ID of the backup """ available_full_backups = list( filter( lambda backup: backup.is_full, self.get_available_backups(status_filter).values(), ) ) if len(available_full_backups) == 0: return None backup_infos = sorted( available_full_backups, key=lambda backup_info: backup_info.backup_id ) return backup_infos[-1].backup_id def get_first_backup_id(self, status_filter=DEFAULT_STATUS_FILTER): """ Get the id of the oldest/first backup in the catalog (if exists) :param status_filter: The status of the backup to return, default to DEFAULT_STATUS_FILTER. :return string|None: ID of the backup """ available_backups = self.get_available_backups(status_filter) if len(available_backups) == 0: return None ids = sorted(available_backups.keys()) return ids[0] def get_backup_id_from_name(self, backup_name, status_filter=DEFAULT_STATUS_FILTER): """ Get the id of the named backup, if it exists. :param string backup_name: The name of the backup for which an ID should be returned :param tuple status_filter: The status of the backup to return. :return string|None: ID of the backup """ available_backups = self.get_available_backups(status_filter).values() backup_info = get_backup_info_from_name(available_backups, backup_name) if backup_info is not None: return backup_info.backup_id def get_closest_backup_id_from_target_time( self, target_time, target_tli, status_filter=DEFAULT_STATUS_FILTER ): """ Get the id of a backup according to the time passed as the recovery target *target_time*, and in the given *target_tli*, if specified. :param str target_time: The target value with timestamp format ``%Y-%m-%d %H:%M:%S`` with or without timezone. :param int|None target_tli: The target timeline, if a specific one is required. :param tuple[str, ...] status_filter: The status of the backup to return. :return str|None: ID of the backup. """ available_backups = self.get_available_backups(status_filter).values() backup_id = get_backup_id_from_target_time( available_backups, target_time, target_tli ) return backup_id def get_closest_backup_id_from_target_lsn( self, target_lsn, target_tli, status_filter=DEFAULT_STATUS_FILTER ): """ Get the id of a backup according to the lsn passed as the recovery target *target_lsn*, and in the given *target_tli*, if specified. :param str target_lsn: The target value with lsn format, e.g., ``3/64000000``. :param int|None target_tli: The target timeline, if a specific one is required. :param tuple[str, ...] status_filter: The status of the backup to return. :return str|None: ID of the backup. """ available_backups = self.get_available_backups(status_filter).values() backup_id = get_backup_id_from_target_lsn( available_backups, target_lsn, target_tli ) return backup_id def get_last_backup_id_from_target_tli( self, target_tli, status_filter=DEFAULT_STATUS_FILTER ): """ Get the id of a backup according to the timeline passed as the recovery target *target_tli*. :param int target_tli: The target timeline. :param tuple[str, ...] status_filter: The status of the backup to return. :return str|None: ID of the backup. """ available_backups = self.get_available_backups(status_filter).values() backup_id = get_backup_id_from_target_tli(available_backups, target_tli) return backup_id def put_delete_annotation(self, backup_id): """ Add a delete annotation to the specified backup. This method adds an annotation to the backup identified by *backup_id* to mark it for deletion. The annotation is stored using the annotation manager. :param str backup_id: The ID of the backup to annotate. """ self.annotation_manager.put_annotation( backup_id, self.DELETE_ANNOTATION, "delete" ) def check_delete_annotation(self, backup_id): """ Check if a delete annotation exists for the specified backup. This method checks if the backup identified by *backup_id* has a delete annotation. It returns ``True`` if the annotation exists, otherwise ``False``. :param str backup_id: The ID of the backup to check. :return bool: ``True`` if the delete annotation exists, ``False`` otherwise. """ return ( self.annotation_manager.get_annotation(backup_id, self.DELETE_ANNOTATION) is not None ) def release_delete_annotation(self, backup_id): """ Remove the delete annotation from the backup identified by *backup_id*. :param str backup_id: The ID of the backup to remove the annotation from. """ self.annotation_manager.delete_annotation(backup_id, self.DELETE_ANNOTATION) @staticmethod def get_timelines_to_protect(remove_until, deleted_backup, available_backups): """ Returns all timelines in available_backups which are not associated with the backup at remove_until. This is so that we do not delete WALs on any other timelines. """ timelines_to_protect = set() # If remove_until is not set there are no backup left if remove_until: # Retrieve the list of extra timelines that contains at least # a backup. On such timelines we don't want to delete any WAL for value in available_backups.values(): # Ignore the backup that is being deleted if value == deleted_backup: continue timelines_to_protect.add(value.timeline) # Remove the timeline of `remove_until` from the list. # We have enough information to safely delete unused WAL files # on it. timelines_to_protect -= set([remove_until.timeline]) return timelines_to_protect def delete_backup(self, backup, skip_wal_cleanup_if_standalone=True): """ Delete a backup :param backup: the backup to delete :param bool skip_wal_cleanup_if_standalone: By default we will skip removing WALs if the oldest backups are standalone archival backups (i.e. they have a keep annotation of "standalone"). If this function is being called in the context of a retention policy however, it is safe to set skip_wal_cleanup_if_standalone to False and clean up WALs associated with those backups. :return bool: True if deleted, False if could not delete the backup """ # Set the delete annotation self.put_delete_annotation(backup.backup_id) # Keep track of when the delete operation started. delete_start_time = datetime.datetime.now() # Run the pre_delete_script if present. script = HookScriptRunner(self, "delete_script", "pre") script.env_from_backup_info(backup) script.run() # Run the pre_delete_retry_script if present. retry_script = RetryHookScriptRunner(self, "delete_retry_script", "pre") retry_script.env_from_backup_info(backup) retry_script.run() output.info( "Deleting backup %s for server %s", backup.backup_id, self.config.name ) should_remove_wals, wal_ranges_to_protect = BackupManager.should_remove_wals( backup, self.get_available_backups( BackupManager.DEFAULT_STATUS_FILTER + (backup.status,) ), keep_manager=self, skip_wal_cleanup_if_standalone=skip_wal_cleanup_if_standalone, ) next_backup = self.get_next_backup(backup.backup_id) # Delete all the data contained in the backup try: self.delete_backup_data(backup) except OSError as e: output.error( "Failure deleting backup %s for server %s.\n%s", backup.backup_id, self.config.name, e, ) return False if should_remove_wals: # There is no previous backup or all previous backups are archival # standalone backups, so we can remove unused WALs (those WALs not # required by standalone archival backups). # If there is a next backup then all unused WALs up to the begin_wal # of the next backup can be removed. # If there is no next backup then there are no remaining backups so: # - In the case of exclusive backup, remove all unused WAL files. # - In the case of concurrent backup (the default), removes only # unused WAL files prior to the start of the backup being deleted, # as they might be useful to any concurrent backup started # immediately after. remove_until = None # means to remove all WAL files if next_backup: remove_until = next_backup elif BackupOptions.CONCURRENT_BACKUP in self.config.backup_options: remove_until = backup timelines_to_protect = self.get_timelines_to_protect( remove_until, backup, self.get_available_backups(BackupInfo.STATUS_ARCHIVING), ) output.info("Delete associated WAL segments:") for name in self.remove_wal_before_backup( remove_until, timelines_to_protect, wal_ranges_to_protect ): output.info("\t%s", name) # Remove the delete annotation self.release_delete_annotation(backup.backup_id) # Remove the base backup directory, try: self.delete_basebackup(backup) except OSError as e: output.error( "Failure deleting backup %s for server %s.\n%s\n" "Please manually remove the '%s' directory", backup.backup_id, self.config.name, e, backup.get_basebackup_directory(), ) return False # As a last action remove, remove the backup.info, ending the delete operation try: self.delete_backupinfo_file(backup) except OSError as e: output.error( "Failure deleting file %s for server %s.\n%s\n" "Please manually remove the file", backup.get_filename(), self.config.name, e, ) return False # Save the time of the complete removal of the backup delete_end_time = datetime.datetime.now() output.info( "Deleted backup %s (start time: %s, elapsed time: %s)", backup.backup_id, delete_start_time.ctime(), human_readable_timedelta(delete_end_time - delete_start_time), ) # remove its reference from its parent if it is an incremental backup parent_backup = backup.get_parent_backup_info() if parent_backup: parent_backup.children_backup_ids.remove(backup.backup_id) if not parent_backup.children_backup_ids: parent_backup.children_backup_ids = None parent_backup.save() # rsync backups can have deduplication at filesystem level by using # "reuse_backup = link". The deduplication size is calculated at the # time the backup is taken. If we remove a backup, it may be the case # that the next backup in the catalog is a rsync backup which was taken # with the "link" option. With that possibility in mind, we re-calculate the # deduplicated size of the next rsync backup because the removal of the # previous backup can impact on that number. # Note: we have no straight forward way of identifying if the next rsync # backup in the catalog was taken with "link" or not because # "reuse_backup" value is not stored in the "backup.info" file. In any # case, the "re-calculation" can still be performed even if "link" was not # used, and the only drawback is that we will waste some (small) amount # of CPU/disk usage. if next_backup and next_backup.backup_type == "rsync": self._set_backup_sizes(next_backup) # Remove the sync lockfile if exists sync_lock = ServerBackupSyncLock( self.config.barman_lock_directory, self.config.name, backup.backup_id ) if os.path.exists(sync_lock.filename): _logger.debug("Deleting backup sync lockfile: %s" % sync_lock.filename) os.unlink(sync_lock.filename) # Run the post_delete_retry_script if present. try: retry_script = RetryHookScriptRunner(self, "delete_retry_script", "post") retry_script.env_from_backup_info(backup) retry_script.run() except AbortedRetryHookScript as e: # Ignore the ABORT_STOP as it is a post-hook operation _logger.warning( "Ignoring stop request after receiving " "abort (exit code %d) from post-delete " "retry hook script: %s", e.hook.exit_status, e.hook.script, ) # Run the post_delete_script if present. script = HookScriptRunner(self, "delete_script", "post") script.env_from_backup_info(backup) script.run() self.backup_cache_remove(backup) return True def _set_backup_sizes(self, backup_info, fsync=False): """ Set the actual size on disk of a backup. Optionally fsync all files in the backup. :param LocalBackupInfo backup_info: the backup to update :param bool fsync: whether to fsync files to disk """ backup_size = 0 deduplicated_size = 0 backup_dest = backup_info.get_basebackup_directory() for dir_path, _, file_names in os.walk(backup_dest): if fsync: # If fsync, execute fsync() on the containing directory fsync_dir(dir_path) for filename in file_names: file_path = os.path.join(dir_path, filename) # If fsync, execute fsync() on all the contained files file_stat = fsync_file(file_path) if fsync else os.stat(file_path) backup_size += file_stat.st_size # Excludes hard links from real backup size and only counts # unique files for deduplicated size if file_stat.st_nlink == 1: deduplicated_size += file_stat.st_size # Save size into BackupInfo object backup_info.set_attribute("size", backup_size) backup_info.set_attribute("deduplicated_size", deduplicated_size) backup_info.save() def validate_backup_args(self, **kwargs): """ Validate backup arguments and Postgres configurations. Arguments might be syntactically correct but still be invalid if necessary Postgres configurations are not met. :kwparam str parent_backup_id: id of the parent backup when taking a Postgres incremental backup :raises BackupException: if a command argument is considered invalid """ if "parent_backup_id" in kwargs: self._validate_incremental_backup_configs(**kwargs) def _validate_incremental_backup_configs(self, **kwargs): """ Check required configurations for a Postgres incremental backup :raises BackupException: if a required configuration is missing """ if self.server.postgres.server_version < 170000: raise BackupException( "Postgres version 17 or greater is required for incremental backups " "using the Postgres backup method" ) if self.config.backup_method != "postgres": raise BackupException( "Backup using the `--incremental` flag is available only for " "'backup_method = postgres'. Check Barman's documentation for " "more help on this topic." ) summarize_wal = self.server.postgres.get_setting("summarize_wal") if summarize_wal != "on": raise BackupException( "'summarize_wal' option has to be enabled in the Postgres server " "to perform an incremental backup using the Postgres backup method" ) if self.config.backup_compression is not None: raise BackupException( "Incremental backups cannot be taken with " "'backup_compression' set in the configuration options." ) parent_backup_id = kwargs.get("parent_backup_id") parent_backup_info = self.get_backup(parent_backup_id) if parent_backup_info: if parent_backup_info.summarize_wal != "on": raise BackupException( "Backup ID %s is not eligible as a parent for an " "incremental backup because WAL summaries were not enabled " "when that backup was taken." % parent_backup_info.backup_id ) if parent_backup_info.compression is not None: raise BackupException( "The specified backup cannot be a parent for an " "incremental backup. Reason: " "Compressed backups are not eligible as parents of incremental backups." ) def backup(self, wait=False, wait_timeout=None, name=None, **kwargs): """ Performs a backup for the server :param bool wait: wait for all the required WAL files to be archived :param int|None wait_timeout: :param str|None name: the friendly name to be saved with this backup :kwparam str parent_backup_id: id of the parent backup when taking a Postgres incremental backup :return BackupInfo: the generated BackupInfo """ _logger.debug("initialising backup information") self.executor.init() backup_info = None try: # Create the BackupInfo object representing the backup backup_info = LocalBackupInfo( self.server, backup_id=datetime.datetime.now().strftime("%Y%m%dT%H%M%S"), backup_name=name, ) backup_info.set_attribute("systemid", self.server.systemid) backup_info.set_attribute( "parent_backup_id", kwargs.get("parent_backup_id"), ) backup_info.save() self.backup_cache_add(backup_info) output.info( "Starting backup using %s method for server %s in %s", self.mode, self.config.name, backup_info.get_basebackup_directory(), ) # Run the pre-backup-script if present. script = HookScriptRunner(self, "backup_script", "pre") script.env_from_backup_info(backup_info) script.run() # Run the pre-backup-retry-script if present. retry_script = RetryHookScriptRunner(self, "backup_retry_script", "pre") retry_script.env_from_backup_info(backup_info) retry_script.run() # Do the backup using the BackupExecutor self.executor.backup(backup_info) # Create a restore point after a backup target_name = "barman_%s" % backup_info.backup_id self.server.postgres.create_restore_point(target_name) # Free the Postgres connection self.server.postgres.close() # Compute backup size and fsync it on disk self.backup_fsync_and_set_sizes(backup_info) # Mark the backup as WAITING_FOR_WALS backup_info.set_attribute("status", BackupInfo.WAITING_FOR_WALS) # Use BaseException instead of Exception to catch events like # KeyboardInterrupt (e.g.: CTRL-C) except BaseException as e: msg_lines = force_str(e).strip().splitlines() # If the exception has no attached message use the raw # type name if len(msg_lines) == 0: msg_lines = [type(e).__name__] if backup_info: # Use only the first line of exception message # in backup_info error field backup_info.set_attribute("status", BackupInfo.FAILED) backup_info.set_attribute( "error", "failure %s (%s)" % (self.executor.current_action, msg_lines[0]), ) output.error( "Backup failed %s.\nDETAILS: %s", self.executor.current_action, "\n".join(msg_lines), ) else: output.info( "Backup end at LSN: %s (%s, %08X)", backup_info.end_xlog, backup_info.end_wal, backup_info.end_offset, ) executor = self.executor output.info( "Backup completed (start time: %s, elapsed time: %s)", self.executor.copy_start_time, human_readable_timedelta( datetime.datetime.now() - executor.copy_start_time ), ) # If requested, wait for end_wal to be archived if wait: try: self.server.wait_for_wal(backup_info.end_wal, wait_timeout) self.check_backup(backup_info) except KeyboardInterrupt: # Ignore CTRL-C pressed while waiting for WAL files output.info( "Got CTRL-C. Continuing without waiting for '%s' " "to be archived", backup_info.end_wal, ) finally: if backup_info: # IF is an incremental backup, we save here child backup info id # inside the parent list of children. no matter if the backup # is successful or not. This is needed to be able to retrieve # also failed incremental backups for removal or other operations # like show-backup. parent_backup_info = backup_info.get_parent_backup_info() if parent_backup_info: if parent_backup_info.children_backup_ids: parent_backup_info.children_backup_ids.append( # type: ignore backup_info.backup_id ) else: parent_backup_info.children_backup_ids = [backup_info.backup_id] parent_backup_info.save() backup_info.save() # Make sure we are not holding any PostgreSQL connection # during the post-backup scripts self.server.close() # Run the post-backup-retry-script if present. try: retry_script = RetryHookScriptRunner( self, "backup_retry_script", "post" ) retry_script.env_from_backup_info(backup_info) retry_script.run() except AbortedRetryHookScript as e: # Ignore the ABORT_STOP as it is a post-hook operation _logger.warning( "Ignoring stop request after receiving " "abort (exit code %d) from post-backup " "retry hook script: %s", e.hook.exit_status, e.hook.script, ) # Run the post-backup-script if present. script = HookScriptRunner(self, "backup_script", "post") script.env_from_backup_info(backup_info) script.run() # if the autogenerate_manifest functionality is active and the # backup files copy is successfully completed using the rsync method, # generate the backup manifest if ( isinstance(self.executor, RsyncBackupExecutor) and self.config.autogenerate_manifest and backup_info.status != BackupInfo.FAILED ): local_file_manager = LocalFileManager() backup_manifest = BackupManifest( backup_info.get_data_directory(), local_file_manager, SHA256() ) backup_manifest.create_backup_manifest() output.info( "Backup manifest for backup '%s' successfully " "generated for server %s", backup_info.backup_id, self.config.name, ) output.result("backup", backup_info) return backup_info def recover( self, backup_info, dest, wal_dest=None, tablespaces=None, remote_command=None, **kwargs ): """ Performs a recovery of a backup :param barman.infofile.LocalBackupInfo backup_info: the backup to recover :param str dest: the destination directory :param str|None wal_dest: the destination directory for WALs when doing PITR. See :meth:`~barman.recovery_executor.RecoveryExecutor._set_pitr_targets` for more details. :param dict[str,str]|None tablespaces: a tablespace name -> location map (for relocation) :param str|None remote_command: default None. The remote command to recover the base backup, in case of remote backup. :kwparam str|None target_tli: the target timeline :kwparam str|None target_time: the target time :kwparam str|None target_xid: the target xid :kwparam str|None target_lsn: the target LSN :kwparam str|None target_name: the target name created previously with pg_create_restore_point() function call :kwparam bool|None target_immediate: end recovery as soon as consistency is reached :kwparam bool exclusive: whether the recovery is exclusive or not :kwparam str|None target_action: default None. The recovery target action :kwparam bool|None standby_mode: the standby mode if needed :kwparam str|None recovery_conf_filename: filename for storing recovery configurations """ # Archive every WAL files in the incoming directory of the server self.server.archive_wal(verbose=False) # Delegate the recovery operation to a RecoveryExecutor object command = unix_command_factory(remote_command, self.server.path) executor = recovery_executor_factory(self, command, backup_info) # Run the pre_recovery_script if present. script = HookScriptRunner(self, "recovery_script", "pre") script.env_from_recover( backup_info, dest, tablespaces, remote_command, **kwargs ) script.run() # Run the pre_recovery_retry_script if present. retry_script = RetryHookScriptRunner(self, "recovery_retry_script", "pre") retry_script.env_from_recover( backup_info, dest, tablespaces, remote_command, **kwargs ) retry_script.run() # Execute the recovery. # We use a closing context to automatically remove # any resource eventually allocated during recovery. with closing(executor): recovery_info = executor.recover( backup_info, dest, wal_dest=wal_dest, tablespaces=tablespaces, remote_command=remote_command, **kwargs ) # Run the post_recovery_retry_script if present. try: retry_script = RetryHookScriptRunner(self, "recovery_retry_script", "post") retry_script.env_from_recover( backup_info, dest, tablespaces, remote_command, **kwargs ) retry_script.run() except AbortedRetryHookScript as e: # Ignore the ABORT_STOP as it is a post-hook operation _logger.warning( "Ignoring stop request after receiving " "abort (exit code %d) from post-recovery " "retry hook script: %s", e.hook.exit_status, e.hook.script, ) # Run the post-recovery-script if present. script = HookScriptRunner(self, "recovery_script", "post") script.env_from_recover( backup_info, dest, tablespaces, remote_command, **kwargs ) script.run() # Output recovery results output.result("recovery", recovery_info["results"]) def archive_wal(self, verbose=True): """ Executes WAL maintenance operations, such as archiving and compression If verbose is set to False, outputs something only if there is at least one file :param bool verbose: report even if no actions """ for archiver in self.server.archivers: archiver.archive(verbose) def cron_retention_policy(self): """ Retention policy management """ enforce_retention_policies = self.server.enforce_retention_policies retention_policy_mode = self.config.retention_policy_mode if enforce_retention_policies and retention_policy_mode == "auto": available_backups = self.get_available_backups(BackupInfo.STATUS_ALL) retention_status = self.config.retention_policy.report() # Find backups with the delete annotation and mark as obsolete for backup in available_backups.values(): if self.check_delete_annotation(backup.backup_id): retention_status[backup.backup_id] = BackupInfo.OBSOLETE self.release_delete_annotation(backup.backup_id) # Check if the backup path still exists while the delete annotation # was already removed elif backup.is_orphan: output.warning( "WARNING: Backup directory %s contains only a non-empty " "backup.info file which may indicate an incomplete delete operation. " "Please manually delete the directory.", backup.get_basebackup_directory(), ) for bid in sorted(retention_status.keys()): if retention_status[bid] == BackupInfo.OBSOLETE: try: # Lock acquisition: if you can acquire a ServerBackupLock # it means that no other processes like another delete operation # are running on that server for that backup id, # and the retention policy can be applied. with ServerBackupIdLock( self.config.barman_lock_directory, self.config.name, bid ): output.info( "Enforcing retention policy: removing backup %s for " "server %s" % (bid, self.config.name) ) self.delete_backup( available_backups[bid], skip_wal_cleanup_if_standalone=False, ) except LockFileBusy: # Another process is holding the backup lock, potentially # is being removed manually. Skip it and output a message output.warning( "Another action is in progress for the backup %s " "of server %s, skipping retention policy application" % (bid, self.config.name) ) def delete_basebackup(self, backup): """ Delete the basebackup dir of a given backup. :param barman.infofile.LocalBackupInfo backup: the backup to delete """ backup_dir = backup.get_basebackup_directory() _logger.debug("Deleting base backup directory: %s" % backup_dir) shutil.rmtree(backup_dir) def delete_backupinfo_file(self, backup): """ Delete the ``backup.info`` file of a given backup. :param barman.infofile.LocalBackupInfo backup: the backup to delete """ backup_info_path = backup.get_filename() if os.path.exists(backup_info_path): _logger.debug("Deleting backup.info file: %s" % backup_info_path) os.unlink(backup_info_path) def delete_backup_data(self, backup): """ Delete the data contained in a given backup. :param barman.infofile.LocalBackupInfo backup: the backup to delete """ # If this backup has snapshots then they should be deleted first. if backup.snapshots_info: _logger.debug( "Deleting the following snapshots: %s" % ", ".join( snapshot.identifier for snapshot in backup.snapshots_info.snapshots ) ) snapshot_interface = get_snapshot_interface_from_backup_info( backup, self.server.config ) snapshot_interface.delete_snapshot_backup(backup) # If this backup does *not* have snapshots then tablespaces are stored on the # barman server so must be deleted. elif backup.tablespaces: if backup.backup_version == 2: tbs_dir = backup.get_basebackup_directory() else: tbs_dir = os.path.join(backup.get_data_directory(), "pg_tblspc") for tablespace in backup.tablespaces: rm_dir = os.path.join(tbs_dir, str(tablespace.oid)) if os.path.exists(rm_dir): _logger.debug( "Deleting tablespace %s directory: %s" % (tablespace.name, rm_dir) ) shutil.rmtree(rm_dir) # Whether a backup has snapshots or not, the data directory will always be # present because this is where the backup_label is stored. It must therefore # be deleted here. pg_data = backup.get_data_directory() if os.path.exists(pg_data): _logger.debug("Deleting PGDATA directory: %s" % pg_data) shutil.rmtree(pg_data) def _run_pre_delete_wal_scripts(self, wal_info): """ Run the pre-delete hook-scripts, if any, on the given WAL. :param barman.infofile.WalFileInfo wal_info: WAL to run the script on. """ # Run the pre_wal_delete_script if present. script = HookScriptRunner(self, "wal_delete_script", "pre") script.env_from_wal_info(wal_info) script.run() # Run the pre_wal_delete_retry_script if present. retry_script = RetryHookScriptRunner(self, "wal_delete_retry_script", "pre") retry_script.env_from_wal_info(wal_info) retry_script.run() def _run_post_delete_wal_scripts(self, wal_info, error=None): """ Run the post-delete hook-scripts, if any, on the given WAL. :param barman.infofile.WalFileInfo wal_info: WAL to run the script on. :param None|str error: error message in case a failure happened. """ # Run the post_wal_delete_retry_script if present. try: retry_script = RetryHookScriptRunner( self, "wal_delete_retry_script", "post" ) retry_script.env_from_wal_info(wal_info, None, error) retry_script.run() except AbortedRetryHookScript as e: # Ignore the ABORT_STOP as it is a post-hook operation _logger.warning( "Ignoring stop request after receiving " "abort (exit code %d) from post-wal-delete " "retry hook script: %s", e.hook.exit_status, e.hook.script, ) # Run the post_wal_delete_script if present. script = HookScriptRunner(self, "wal_delete_script", "post") script.env_from_wal_info(wal_info, None, error) script.run() def delete_wal(self, wal_info): """ Delete a WAL segment, with the given WalFileInfo :param barman.infofile.WalFileInfo wal_info: the WAL to delete """ self._run_pre_delete_wal_scripts(wal_info) error = None try: os.unlink(wal_info.fullpath(self.server)) try: os.removedirs(os.path.dirname(wal_info.fullpath(self.server))) except OSError: # This is not an error condition # We always try to remove the trailing directories, # this means that hashdir is not empty. pass except OSError as e: error = "Ignoring deletion of WAL file %s for server %s: %s" % ( wal_info.name, self.config.name, e, ) output.warning(error) self._run_post_delete_wal_scripts(wal_info, error) def check(self, check_strategy): """ This function does some checks on the server. :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ check_strategy.init_check("compression settings") # Check compression_setting parameter if self.config.compression and not self.compression_manager.check(): check_strategy.result(self.config.name, False) else: status = True try: self.compression_manager.get_default_compressor() except CompressionIncompatibility as field: check_strategy.result(self.config.name, "%s setting" % field, False) status = False check_strategy.result(self.config.name, status) # Failed backups check check_strategy.init_check("failed backups") failed_backups = self.get_available_backups((BackupInfo.FAILED,)) status = len(failed_backups) == 0 check_strategy.result( self.config.name, status, hint="there are %s failed backups" % ( len( failed_backups, ) ), ) check_strategy.init_check("minimum redundancy requirements") # Minimum redundancy checks no_backups = len(self.get_available_backups(status_filter=(BackupInfo.DONE,))) # Check minimum_redundancy_requirements parameter if no_backups < int(self.config.minimum_redundancy): status = False else: status = True check_strategy.result( self.config.name, status, hint="have %s backups, expected at least %s" % (no_backups, self.config.minimum_redundancy), ) # TODO: Add a check for the existence of ssh and of rsync # Execute additional checks defined by the BackupExecutor if self.executor: self.executor.check(check_strategy) def status(self): """ This function show the server status """ # get number of backups no_backups = len(self.get_available_backups(status_filter=(BackupInfo.DONE,))) output.result( "status", self.config.name, "backups_number", "No. of available backups", no_backups, ) output.result( "status", self.config.name, "first_backup", "First available backup", self.get_first_backup_id(), ) output.result( "status", self.config.name, "last_backup", "Last available backup", self.get_last_backup_id(), ) # Minimum redundancy check. if number of backups minor than minimum # redundancy, fail. if no_backups < self.config.minimum_redundancy: output.result( "status", self.config.name, "minimum_redundancy", "Minimum redundancy requirements", "FAILED (%s/%s)" % (no_backups, self.config.minimum_redundancy), ) else: output.result( "status", self.config.name, "minimum_redundancy", "Minimum redundancy requirements", "satisfied (%s/%s)" % (no_backups, self.config.minimum_redundancy), ) # Output additional status defined by the BackupExecutor if self.executor: self.executor.status() def fetch_remote_status(self): """ Build additional remote status lines defined by the BackupManager. This method does not raise any exception in case of errors, but set the missing values to None in the resulting dictionary. :rtype: dict[str, None|str] """ if self.executor: return self.executor.get_remote_status() else: return {} def get_latest_archived_wals_info(self): """ Return a dictionary of timelines associated with the WalFileInfo of the last WAL file in the archive, or None if the archive doesn't contain any WAL file. :rtype: dict[str, WalFileInfo]|None """ from os.path import isdir, join root = self.config.wals_directory comp_manager = self.compression_manager # If the WAL archive directory doesn't exists the archive is empty if not isdir(root): return dict() # Traverse all the directory in the archive in reverse order, # returning the first WAL file found timelines = {} for name in sorted(os.listdir(root), reverse=True): fullname = join(root, name) # All relevant files are in subdirectories, so # we skip any non-directory entry if isdir(fullname): # Extract the timeline. If it is not valid, skip this directory try: timeline = name[0:8] int(timeline, 16) except ValueError: continue # If this timeline already has a file, skip this directory if timeline in timelines: continue hash_dir = fullname # Inspect contained files in reverse order for wal_name in sorted(os.listdir(hash_dir), reverse=True): fullname = join(hash_dir, wal_name) # Return the first file that has the correct name if not isdir(fullname) and xlog.is_wal_file(fullname): timelines[timeline] = comp_manager.get_wal_file_info(fullname) break # Return the timeline map return timelines def remove_wal_before_backup( self, backup_info, timelines_to_protect=None, wal_ranges_to_protect=[] ): """ Remove WAL files which have been archived before the start of the provided backup. If no backup_info is provided delete all available WAL files If timelines_to_protect list is passed, never remove a wal in one of these timelines. :param BackupInfo|None backup_info: the backup information structure :param set timelines_to_protect: optional list of timelines to protect :param list wal_ranges_to_protect: optional list of `(begin_wal, end_wal)` tuples which define inclusive ranges of WALs which must not be deleted. :return list: a list of removed WAL files """ # A dictionary where key is the WAL directory name and value is a list of # wal_info object representing the WALs to be deleted in that directory wals_to_remove = defaultdict(list) with self.server.xlogdb("r+") as fxlogdb: xlogdb_dir = os.path.dirname(fxlogdb.name) with tempfile.TemporaryFile(mode="w+", dir=xlogdb_dir) as fxlogdb_new: for line in fxlogdb: wal_info = WalFileInfo.from_xlogdb_line(line) if not xlog.is_any_xlog_file(wal_info.name): output.error( "invalid WAL segment name %r\n" 'HINT: Please run "barman rebuild-xlogdb %s" ' "to solve this issue", wal_info.name, self.config.name, ) continue # Keeps the WAL segment if it is a history file keep = xlog.is_history_file(wal_info.name) # Keeps the WAL segment if its timeline is in # `timelines_to_protect` if timelines_to_protect: tli, _, _ = xlog.decode_segment_name(wal_info.name) keep |= tli in timelines_to_protect # Keeps the WAL segment if it is within a protected range if xlog.is_backup_file(wal_info.name): # If we have a .backup file then truncate the name for the # range check wal_name = wal_info.name[:24] else: wal_name = wal_info.name for begin_wal, end_wal in wal_ranges_to_protect: keep |= wal_name >= begin_wal and wal_name <= end_wal # Keeps the WAL segment if it is a newer # than the given backup (the first available) if backup_info and backup_info.begin_wal is not None: keep |= wal_info.name >= backup_info.begin_wal # If the file has to be kept write it in the new xlogdb # otherwise add it to the removal list if keep: fxlogdb_new.write(wal_info.to_xlogdb_line()) else: wal_dir = os.path.dirname(wal_info.fullpath(self.server)) wals_to_remove[wal_dir].append(wal_info) wals_removed = self.delete_wals(wals_to_remove) fxlogdb_new.flush() fxlogdb_new.seek(0) fxlogdb.seek(0) shutil.copyfileobj(fxlogdb_new, fxlogdb) fxlogdb.truncate() return wals_removed def delete_wals(self, wals_to_delete): """ Delete the given WAL files. The entire WAL directory is deleted when possible. :param dict[str, list[WalFileInfo]] wals_to_delete: A dictionary where key is the WAL directory name and value is a list of wal_info objects representing the WALs to be deleted in that directory. :return list[str]: a list of deleted WAL names. """ wals_deleted = [] for wal_dir, wal_list in wals_to_delete.items(): delete_directory = False # Each directory can contain up to 256 WAL files. If the deletion list # contains 256 entries, the entire directory can be safely deleted # Otherwise, check if all WALs in the directory are in the deletion list if len(wal_list) >= 256: delete_directory = True else: wal_names_to_delete = {wal_info.name for wal_info in wal_list} wal_names_in_dir = os.listdir(wal_dir) if set(wal_names_in_dir).issubset(wal_names_to_delete): delete_directory = True # If the directory can be deleted, run the hook-scripts on each WAL file # before and after the rmtree. Otherwise, delete each WAL individually if delete_directory: for wal_info in wal_list: self._run_pre_delete_wal_scripts(wal_info) shutil.rmtree(wal_dir) for wal_info in wal_list: self._run_post_delete_wal_scripts(wal_info) wals_deleted.append(wal_info.name) else: for wal_info in wal_list: self.delete_wal(wal_info) wals_deleted.append(wal_info.name) return wals_deleted def validate_last_backup_maximum_age(self, last_backup_maximum_age): """ Evaluate the age of the last available backup in a catalogue. If the last backup is older than the specified time interval (age), the function returns False. If within the requested age interval, the function returns True. :param timedate.timedelta last_backup_maximum_age: time interval representing the maximum allowed age for the last backup in a server catalogue :return tuple: a tuple containing the boolean result of the check and auxiliary information about the last backup current age """ # Get the ID of the last available backup backup_id = self.get_last_backup_id() if backup_id: # Get the backup object backup = LocalBackupInfo(self.server, backup_id=backup_id) now = datetime.datetime.now(dateutil.tz.tzlocal()) # Evaluate the point of validity validity_time = now - last_backup_maximum_age # Pretty print of a time interval (age) msg = human_readable_timedelta(now - backup.end_time) # If the backup end time is older than the point of validity, # return False, otherwise return true if backup.end_time < validity_time: return False, msg else: return True, msg else: # If no backup is available return false return False, "No available backups" def validate_last_backup_min_size(self, last_backup_minimum_size): """ Evaluate the size of the last available backup in a catalogue. If the last backup is smaller than the specified size the function returns False. Otherwise, the function returns True. :param last_backup_minimum_size: size in bytes representing the maximum allowed age for the last backup in a server catalogue :return tuple: a tuple containing the boolean result of the check and auxiliary information about the last backup current age """ # Get the ID of the last available backup backup_id = self.get_last_backup_id() if backup_id: # Get the backup object backup = LocalBackupInfo(self.server, backup_id=backup_id) if backup.size < last_backup_minimum_size: return False, backup.size else: return True, backup.size else: # If no backup is available return false return False, 0 def backup_fsync_and_set_sizes(self, backup_info): """ Fsync all files in a backup and set the actual size on disk of a backup. Also evaluate the deduplication ratio and the deduplicated size if applicable. :param LocalBackupInfo backup_info: the backup to update """ # Calculate the base backup size self.executor.current_action = "calculating backup size" _logger.debug(self.executor.current_action) # Set backup sizes with fsync. We need to fsync files here to make sure # the backup files are persisted to disk, so we don't lose the backup in # the event of a system crash. self._set_backup_sizes(backup_info, fsync=True) if backup_info.size > 0: deduplication_ratio = 1 - ( float(backup_info.deduplicated_size) / backup_info.size ) else: deduplication_ratio = 0 if self.config.reuse_backup == "link": output.info( "Backup size: %s. Actual size on disk: %s" " (-%s deduplication ratio)." % ( pretty_size(backup_info.size), pretty_size(backup_info.deduplicated_size), "{percent:.2%}".format(percent=deduplication_ratio), ) ) else: output.info("Backup size: %s" % pretty_size(backup_info.size)) def check_backup(self, backup_info): """ Make sure that all the required WAL files to check the consistency of a physical backup (that is, from the beginning to the end of the full backup) are correctly archived. This command is automatically invoked by the cron command and at the end of every backup operation. :param backup_info: the target backup """ # Gather the list of the latest archived wals timelines = self.get_latest_archived_wals_info() # Get the basic info for the backup begin_wal = backup_info.begin_wal end_wal = backup_info.end_wal timeline = begin_wal[:8] # Case 0: there is nothing to check for this backup, as it is # currently in progress if not end_wal: return # Case 1: Barman still doesn't know about the timeline the backup # started with. We still haven't archived any WAL corresponding # to the backup, so we can't proceed with checking the existence # of the required WAL files if not timelines or timeline not in timelines: backup_info.status = BackupInfo.WAITING_FOR_WALS backup_info.save() return # Find the most recent archived WAL for this server in the timeline # where the backup was taken last_archived_wal = timelines[timeline].name # Case 2: the most recent WAL file archived is older than the # start of the backup. We must wait for the archiver to receive # and/or process the WAL files. if last_archived_wal < begin_wal: backup_info.status = BackupInfo.WAITING_FOR_WALS backup_info.save() return # Check the intersection between the required WALs and the archived # ones. They should all exist segments = backup_info.get_required_wal_segments() missing_wal = None for wal in segments: # Stop checking if we reach the last archived wal if wal > last_archived_wal: break wal_full_path = self.server.get_wal_full_path(wal) if not os.path.exists(wal_full_path): missing_wal = wal break if missing_wal: # Case 3: the most recent WAL file archived is more recent than # the one corresponding to the start of a backup. If WAL # file is missing, then we can't recover from the backup so we # must mark the backup as FAILED. # TODO: Verify if the error field is the right place # to store the error message backup_info.error = ( "At least one WAL file is missing. " "The first missing WAL file is %s" % missing_wal ) backup_info.status = BackupInfo.FAILED backup_info.save() output.error( "This backup has been marked as FAILED due to the " "following reason: %s" % backup_info.error ) return if end_wal <= last_archived_wal: # Case 4: if the most recent WAL file archived is more recent or # equal than the one corresponding to the end of the backup and # every WAL that will be required by the recovery is available, # we can mark the backup as DONE. backup_info.status = BackupInfo.DONE else: # Case 5: if the most recent WAL file archived is older than # the one corresponding to the end of the backup but # all the WAL files until that point are present. backup_info.status = BackupInfo.WAITING_FOR_WALS backup_info.save() def verify_backup(self, backup_info): """ This function should check if pg_verifybackup is installed and run it against backup path should test if pg_verifybackup is installed locally :param backup_info: barman.infofile.LocalBackupInfo instance """ output.info("Calling pg_verifybackup") # Test pg_verifybackup existence version_info = PgVerifyBackup.get_version_info(self.server.path) if version_info.get("full_path", None) is None: output.error("pg_verifybackup not found") return pg_verifybackup = PgVerifyBackup( data_path=backup_info.get_data_directory(), command=version_info["full_path"], version=version_info["full_version"], ) try: pg_verifybackup() except CommandFailedException as e: output.error( "verify backup failure on directory '%s'" % backup_info.get_data_directory() ) output.error(e.args[0]["err"]) return output.info(pg_verifybackup.get_output()[0].strip()) barman-3.13.3/barman/annotations.py0000644000175100001660000003522515002177425015403 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import errno import io import os from abc import ABCMeta, abstractmethod from barman.exceptions import ArchivalBackupException from barman.utils import with_metaclass class AnnotationManager(with_metaclass(ABCMeta)): """ This abstract base class defines the AnnotationManager interface which provides methods for read, write and delete of annotations for a given backup. """ @abstractmethod def put_annotation(self, backup_id, key, value): """Add an annotation""" @abstractmethod def get_annotation(self, backup_id, key): """Get the value of an annotation""" @abstractmethod def delete_annotation(self, backup_id, key): """Delete an annotation""" class AnnotationManagerFile(AnnotationManager): def __init__(self, path, old_path=None): """ Constructor for the file-based annotation manager. Should be initialised with the path to the barman base backup directory. :param str path: The path where the annotation file should be placed. :param str|None old_path: Optional path used to read annotations written before Barman 3.13.3. .. note: Starting from Barman 3.13.3, annotation files were moved out of the base backup directory and into a dedicated metadata directory. While this class is agnostic about what kind of annotation it stores, backwards compatibility is needed for users upgrading from previous versions who may have annotations stored in the legacy location. For that reason, the *old_path* parameter allows this class to read and migrate existing annotations from the old directory as to maintain backwards compatibility. """ self.path = path self.old_path = old_path def _get_old_annotation_path(self, backup_id, key): """ Builds the annotation path for the specified *backup_id* and annotation *key* for annotations created before Barman 3.13.3. Check the note on this class' constructor for more context. :param str backup_id: The backup ID. :param str key: The annotation file name. :returns str: The path to the annotation. """ return "%s/%s/annotations/%s" % (self.old_path, backup_id, key) def _get_annotation_path(self, backup_id, key): """ Builds the annotation path for the specified backup_id and annotation key. :param str backup_id: The backup ID. :param str key: The annotation file name. :returns str: The path to the annotation. """ return "%s/%s-%s" % (self.path, backup_id, key) def _check_and_relocate_old_annotation(self, backup_id, key): """ Check if the annotation exists in the old path, used before Barman 3.13.3, and relocate it to the new path as to maintain backwards compatibility. :param str backup_id: The backup ID. :param str key: The annotation file name. """ if not self.old_path: return old_path = self._get_old_annotation_path(backup_id, key) new_path = self._get_annotation_path(backup_id, key) if os.path.exists(old_path): os.rename(old_path, new_path) def delete_annotation(self, backup_id, key): """ Deletes an annotation from the filesystem for the specified backup_id and annotation key. """ self._check_and_relocate_old_annotation(backup_id, key) annotation_path = self._get_annotation_path(backup_id, key) try: os.remove(annotation_path) except EnvironmentError as e: # For Python 2 compatibility we must check the error code directly # If the annotation doesn't exist then the failure to delete it is not an # error condition and we should not proceed to remove the annotations # directory if e.errno == errno.ENOENT: return else: raise try: os.rmdir(os.path.dirname(annotation_path)) except EnvironmentError as e: # For Python 2 compatibility we must check the error code directly # If we couldn't remove the directory because it wasn't empty then we # do not consider it an error condition if e.errno != errno.ENOTEMPTY: raise def get_annotation(self, backup_id, key): """ Reads the annotation `key` for the specified backup_id from the filesystem and returns the value. """ self._check_and_relocate_old_annotation(backup_id, key) annotation_path = self._get_annotation_path(backup_id, key) try: with open(annotation_path, "r") as annotation_file: return annotation_file.read() except EnvironmentError as e: # For Python 2 compatibility we must check the error code directly # If the annotation doesn't exist then return None if e.errno != errno.ENOENT: raise def put_annotation(self, backup_id, key, value): """ Writes the specified value for annotation `key` for the specified backup_id to the filesystem. """ annotation_path = self._get_annotation_path(backup_id, key) try: os.makedirs(os.path.dirname(annotation_path)) except EnvironmentError as e: # For Python 2 compatibility we must check the error code directly # If the directory already exists then it is not an error condition if e.errno != errno.EEXIST: raise with open(annotation_path, "w") as annotation_file: if value: annotation_file.write(value) class AnnotationManagerCloud(AnnotationManager): def __init__(self, cloud_interface, server_name): """ Constructor for the cloud-based annotation manager. Should be initialised with the CloudInterface and name of the server which was used to create the backups. """ self.cloud_interface = cloud_interface self.server_name = server_name self.annotation_cache = None def _get_base_path(self): """ Returns the base path to the cloud storage, accounting for the fact that CloudInterface.path may be None. """ return self.cloud_interface.path and "%s/" % self.cloud_interface.path or "" def _get_annotation_path(self, backup_id, key): """ Builds the full key to the annotation in cloud storage for the specified backup_id and annotation key. """ return "%s%s/base/%s/annotations/%s" % ( self._get_base_path(), self.server_name, backup_id, key, ) def _populate_annotation_cache(self): """ Build a cache of which annotations actually exist by walking the bucket. This allows us to optimize get_annotation by just checking a (backup_id,key) tuple here which is cheaper (in time and money) than going to the cloud every time. """ self.annotation_cache = {} for object_key in self.cloud_interface.list_bucket( os.path.join(self._get_base_path(), self.server_name, "base") + "/", delimiter="", ): key_parts = object_key.split("/") if len(key_parts) > 3: if key_parts[-2] == "annotations": backup_id = key_parts[-3] annotation_key = key_parts[-1] self.annotation_cache[(backup_id, annotation_key)] = True def delete_annotation(self, backup_id, key): """ Deletes an annotation from cloud storage for the specified backup_id and annotation key. """ annotation_path = self._get_annotation_path(backup_id, key) self.cloud_interface.delete_objects([annotation_path]) def get_annotation(self, backup_id, key, use_cache=True): """ Reads the annotation `key` for the specified backup_id from cloud storage and returns the value. The default behaviour is that, when it is first run, it populates a cache of the annotations which exist for each backup by walking the bucket. Subsequent operations can check that cache and avoid having to call remote_open if an annotation is not found in the cache. This optimises for the case where annotations are sparse and assumes the cost of walking the bucket is less than the cost of the remote_open calls which would not return a value. In cases where we do not want to walk the bucket up front then the caching can be disabled. """ # Optimize for the most common case where there is no annotation if use_cache: if self.annotation_cache is None: self._populate_annotation_cache() if ( self.annotation_cache is not None and (backup_id, key) not in self.annotation_cache ): return None # We either know there's an annotation or we haven't used the cache so read # it from the cloud annotation_path = self._get_annotation_path(backup_id, key) annotation_fileobj = self.cloud_interface.remote_open(annotation_path) if annotation_fileobj: with annotation_fileobj: annotation_bytes = annotation_fileobj.readline() return annotation_bytes.decode("utf-8") else: # We intentionally return None if remote_open found nothing return None def put_annotation(self, backup_id, key, value): """ Writes the specified value for annotation `key` for the specified backup_id to cloud storage. """ annotation_path = self._get_annotation_path(backup_id, key) self.cloud_interface.upload_fileobj( io.BytesIO(value.encode("utf-8")), annotation_path ) class KeepManager(with_metaclass(ABCMeta, object)): """Abstract base class which defines the KeepManager interface""" ANNOTATION_KEY = "keep" TARGET_FULL = "full" TARGET_STANDALONE = "standalone" supported_targets = (TARGET_FULL, TARGET_STANDALONE) @abstractmethod def should_keep_backup(self, backup_id): pass @abstractmethod def keep_backup(self, backup_id, target): pass @abstractmethod def get_keep_target(self, backup_id): pass @abstractmethod def release_keep(self, backup_id): pass class KeepManagerMixin(KeepManager): """ A Mixin which adds KeepManager functionality to its subclasses. Keep management is built on top of annotations and consists of the following functionality: - Determine whether a given backup is intended to be kept beyond its retention period. - Determine the intended recovery target for the archival backup. - Add and remove the keep annotation. The functionality is implemented as a Mixin so that it can be used to add keep management to the backup management class in barman (BackupManager) as well as its closest analog in barman-cloud (CloudBackupCatalog). """ def __init__(self, *args, **kwargs): """ Base constructor (Mixin pattern). kwargs must contain *either*: - A barman.server.Server object with the key `server`, *or*: - A CloudInterface object and a server name, keys `cloud_interface` and `server_name` respectively. """ if "server" in kwargs: server = kwargs.pop("server") self.annotation_manager = AnnotationManagerFile( server.meta_directory, server.config.basebackups_directory ) elif "cloud_interface" in kwargs: self.annotation_manager = AnnotationManagerCloud( kwargs.pop("cloud_interface"), kwargs.pop("server_name") ) super(KeepManagerMixin, self).__init__(*args, **kwargs) def should_keep_backup(self, backup_id): """ Returns True if the specified backup_id for this server has a keep annotation. False otherwise. """ return ( self.annotation_manager.get_annotation(backup_id, type(self).ANNOTATION_KEY) is not None ) def keep_backup(self, backup_id, target): """ Add a keep annotation for backup with ID backup_id with the specified recovery target. """ if target not in KeepManagerMixin.supported_targets: raise ArchivalBackupException("Unsupported recovery target: %s" % target) self.annotation_manager.put_annotation( backup_id, type(self).ANNOTATION_KEY, target ) def get_keep_target(self, backup_id): """Retrieve the intended recovery target""" return self.annotation_manager.get_annotation( backup_id, type(self).ANNOTATION_KEY ) def release_keep(self, backup_id): """Release the keep annotation""" self.annotation_manager.delete_annotation(backup_id, type(self).ANNOTATION_KEY) class KeepManagerMixinCloud(KeepManagerMixin): """ A specialised KeepManager which allows the annotation caching optimization in the AnnotationManagerCloud backend to be optionally disabled. """ def should_keep_backup(self, backup_id, use_cache=True): """ Like KeepManagerMixinCloud.should_keep_backup but with the use_cache option. """ return ( self.annotation_manager.get_annotation( backup_id, type(self).ANNOTATION_KEY, use_cache=use_cache ) is not None ) def get_keep_target(self, backup_id, use_cache=True): """ Like KeepManagerMixinCloud.get_keep_target but with the use_cache option. """ return self.annotation_manager.get_annotation( backup_id, type(self).ANNOTATION_KEY, use_cache=use_cache ) barman-3.13.3/barman/backup_executor.py0000644000175100001660000027122315002177425016231 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ Backup Executor module A Backup Executor is a class responsible for the execution of a backup. Specific implementations of backups are defined by classes that derive from BackupExecutor (e.g.: backup with rsync through Ssh). A BackupExecutor is invoked by the BackupManager for backup operations. """ import datetime import logging import os import re import shutil from abc import ABCMeta, abstractmethod from contextlib import closing from distutils.version import LooseVersion as Version from functools import partial import dateutil.parser from barman import output, xlog from barman.cloud_providers import get_snapshot_interface_from_server_config from barman.command_wrappers import PgBaseBackup from barman.compression import get_pg_basebackup_compression from barman.config import BackupOptions from barman.copy_controller import RsyncCopyController from barman.exceptions import ( BackupException, CommandFailedException, DataTransferFailure, FileNotFoundException, FsOperationFailed, PostgresConnectionError, PostgresConnectionLost, PostgresIsInRecovery, SnapshotBackupException, SshCommandException, ) from barman.fs import UnixLocalCommand, UnixRemoteCommand, unix_command_factory from barman.infofile import BackupInfo from barman.postgres import PostgresKeepAlive from barman.postgres_plumbing import EXCLUDE_LIST, PGDATA_EXCLUDE_LIST from barman.remote_status import RemoteStatusMixin from barman.utils import ( check_aws_expiration_date_format, check_aws_snapshot_lock_cool_off_period_range, check_aws_snapshot_lock_duration_range, check_aws_snapshot_lock_mode, force_str, human_readable_timedelta, mkpath, total_seconds, with_metaclass, ) _logger = logging.getLogger(__name__) class BackupExecutor(with_metaclass(ABCMeta, RemoteStatusMixin)): """ Abstract base class for any backup executors. """ def __init__(self, backup_manager, mode=None): """ Base constructor :param barman.backup.BackupManager backup_manager: the BackupManager assigned to the executor :param str mode: The mode used by the executor for the backup. """ super(BackupExecutor, self).__init__() self.backup_manager = backup_manager self.server = backup_manager.server self.config = backup_manager.config self.strategy = None self._mode = mode self.copy_start_time = None self.copy_end_time = None # Holds the action being executed. Used for error messages. self.current_action = None def init(self): """ Initialise the internal state of the backup executor """ self.current_action = "starting backup" @property def mode(self): """ Property that defines the mode used for the backup. If a strategy is present, the returned string is a combination of the mode of the executor and the mode of the strategy (eg: rsync-exclusive) :return str: a string describing the mode used for the backup """ strategy_mode = self.strategy.mode if strategy_mode: return "%s-%s" % (self._mode, strategy_mode) else: return self._mode @abstractmethod def backup(self, backup_info): """ Perform a backup for the server - invoked by BackupManager.backup() :param barman.infofile.LocalBackupInfo backup_info: backup information """ def check(self, check_strategy): """ Perform additional checks - invoked by BackupManager.check() :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ def status(self): """ Set additional status info - invoked by BackupManager.status() """ def fetch_remote_status(self): """ Get additional remote status info - invoked by BackupManager.get_remote_status() This method does not raise any exception in case of errors, but set the missing values to None in the resulting dictionary. :rtype: dict[str, None|str] """ return {} def _purge_unused_wal_files(self, backup_info): """ If the provided backup is the first, purge unused WAL files before the backup start. .. note:: If ``worm_mode`` is enabled, then we don't remove those WAL files because they are (should be) stored in an immutable storage, and at this point the grace period might already have been expired. :param barman.infofile.LocalBackupInfo backup_info: The backup to check. """ if backup_info.begin_wal is None: return previous_backup = self.backup_manager.get_previous_backup(backup_info.backup_id) if not previous_backup: output.info("This is the first backup for server %s", self.config.name) if self.config.worm_mode is True: output.info("'worm_mode' is enabled, skip purging of unused WAL files.") return removed = self.backup_manager.remove_wal_before_backup(backup_info) if removed: # report the list of the removed WAL files output.info( "WAL segments preceding the current backup have been found:", log=False, ) for wal_name in removed: output.info( "\t%s from server %s has been removed", wal_name, self.config.name, ) def _start_backup_copy_message(self, backup_info): """ Output message for backup start :param barman.infofile.LocalBackupInfo backup_info: backup information """ output.info("Copying files for %s", backup_info.backup_id) def _stop_backup_copy_message(self, backup_info): """ Output message for backup end :param barman.infofile.LocalBackupInfo backup_info: backup information """ output.info( "Copy done (time: %s)", human_readable_timedelta( datetime.timedelta(seconds=backup_info.copy_stats["copy_time"]) ), ) def _parse_ssh_command(ssh_command): """ Parse a user provided ssh command to a single command and a list of arguments In case of error, the first member of the result (the command) will be None :param ssh_command: a ssh command provided by the user :return tuple[str,list[str]]: the command and a list of options """ try: ssh_options = ssh_command.split() except AttributeError: return None, [] ssh_command = ssh_options.pop(0) ssh_options.extend("-o BatchMode=yes -o StrictHostKeyChecking=no".split()) return ssh_command, ssh_options class PostgresBackupExecutor(BackupExecutor): """ Concrete class for backup via pg_basebackup (plain format). Relies on pg_basebackup command to copy data files from the PostgreSQL cluster using replication protocol. """ def __init__(self, backup_manager): """ Constructor :param barman.backup.BackupManager backup_manager: the BackupManager assigned to the executor """ super(PostgresBackupExecutor, self).__init__(backup_manager, "postgres") self.backup_compression = get_pg_basebackup_compression(self.server) self.validate_configuration() self.strategy = PostgresBackupStrategy( self.server.postgres, self.config.name, self.backup_compression ) def validate_configuration(self): """ Validate the configuration for this backup executor. If the configuration is not compatible this method will disable the server. """ # Check for the correct backup options if BackupOptions.EXCLUSIVE_BACKUP in self.config.backup_options: self.config.backup_options.remove(BackupOptions.EXCLUSIVE_BACKUP) output.warning( "'exclusive_backup' is not a valid backup_option " "using postgres backup_method. " "Overriding with 'concurrent_backup'." ) # Apply the default backup strategy if BackupOptions.CONCURRENT_BACKUP not in self.config.backup_options: self.config.backup_options.add(BackupOptions.CONCURRENT_BACKUP) output.debug( "The default backup strategy for " "postgres backup_method is: concurrent_backup" ) # Forbid tablespace_bandwidth_limit option. # It works only with rsync based backups. if self.config.tablespace_bandwidth_limit: # Report the error in the configuration errors message list self.server.config.update_msg_list_and_disable_server( "tablespace_bandwidth_limit option is not supported by " "postgres backup_method" ) # Forbid reuse_backup option. # It works only with rsync based backups. if self.config.reuse_backup in ("copy", "link"): # Report the error in the configuration errors message list self.server.config.update_msg_list_and_disable_server( "reuse_backup option is not supported by postgres backup_method" ) # Forbid network_compression option. # It works only with rsync based backups. if self.config.network_compression: # Report the error in the configuration errors message list self.server.config.update_msg_list_and_disable_server( "network_compression option is not supported by " "postgres backup_method" ) # The following checks require interactions with the PostgreSQL server # therefore they are carried out within a `closing` context manager to # ensure the connection is not left dangling in cases where no further # server interaction is required. remote_status = None with closing(self.server): if self.server.config.bandwidth_limit or self.backup_compression: # This method is invoked too early to have a working streaming # connection. So we avoid caching the result by directly # invoking fetch_remote_status() instead of get_remote_status() remote_status = self.fetch_remote_status() # bandwidth_limit option is supported by pg_basebackup executable # starting from Postgres 9.4 if ( self.server.config.bandwidth_limit and remote_status["pg_basebackup_bwlimit"] is False ): # If pg_basebackup is present and it doesn't support bwlimit # disable the server. # Report the error in the configuration errors message list self.server.config.update_msg_list_and_disable_server( "bandwidth_limit option is not supported by " "pg_basebackup version (current: %s, required: 9.4)" % remote_status["pg_basebackup_version"] ) # validate compression options if self.backup_compression: self._validate_compression(remote_status) def _validate_compression(self, remote_status): """ In charge of validating compression options. Note: Because this method requires a connection to the PostgreSQL server it should be called within the context of a closing context manager. :param remote_status: :return: """ try: issues = self.backup_compression.validate( self.server.postgres.server_version, remote_status ) if issues: self.server.config.update_msg_list_and_disable_server(issues) except PostgresConnectionError as exc: # If we can't validate the compression settings due to a connection error # it should not block whatever Barman is trying to do *unless* it is # doing a backup, in which case the pre-backup check will catch the # connection error and fail accordingly. # This is important because if the server is unavailable Barman # commands such as `recover` and `list-backups` must not break. _logger.warning( ( "Could not validate compression due to a problem " "with the PostgreSQL connection: %s" ), exc, ) def backup(self, backup_info): """ Perform a backup for the server - invoked by BackupManager.backup() through the generic interface of a BackupExecutor. This implementation is responsible for performing a backup through the streaming protocol. The connection must be made with a superuser or a user having REPLICATION permissions (see PostgreSQL documentation, Section 20.2), and pg_hba.conf must explicitly permit the replication connection. The server must also be configured with enough max_wal_senders to leave at least one session available for the backup. :param barman.infofile.LocalBackupInfo backup_info: backup information """ try: # Set data directory and server version self.strategy.start_backup(backup_info) backup_info.save() if backup_info.begin_wal is not None: output.info( "Backup start at LSN: %s (%s, %08X)", backup_info.begin_xlog, backup_info.begin_wal, backup_info.begin_offset, ) else: output.info("Backup start at LSN: %s", backup_info.begin_xlog) # Start the copy self.current_action = "copying files" self._start_backup_copy_message(backup_info) self.backup_copy(backup_info) self._stop_backup_copy_message(backup_info) self.strategy.stop_backup(backup_info) # If this is the first backup, purge eventually unused WAL files self._purge_unused_wal_files(backup_info) except CommandFailedException as e: _logger.exception(e) raise def check(self, check_strategy): """ Perform additional checks for PostgresBackupExecutor :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ check_strategy.init_check("pg_basebackup") remote_status = self.get_remote_status() # Check for the presence of pg_basebackup check_strategy.result( self.config.name, remote_status["pg_basebackup_installed"] ) # remote_status['pg_basebackup_compatible'] is None if # pg_basebackup cannot be executed and False if it is # not compatible. hint = None check_strategy.init_check("pg_basebackup compatible") if not remote_status["pg_basebackup_compatible"]: pg_version = "Unknown" basebackup_version = "Unknown" if self.server.streaming is not None: pg_version = self.server.streaming.server_txt_version if remote_status["pg_basebackup_version"] is not None: basebackup_version = remote_status["pg_basebackup_version"] hint = "PostgreSQL version: %s, pg_basebackup version: %s" % ( pg_version, basebackup_version, ) check_strategy.result( self.config.name, remote_status["pg_basebackup_compatible"], hint=hint ) # Skip further checks if the postgres connection doesn't work. # We assume that this error condition will be reported by # another check. postgres = self.server.postgres if postgres is None or postgres.server_txt_version is None: return check_strategy.init_check("pg_basebackup supports tablespaces mapping") # We can't backup a cluster with tablespaces if the tablespace # mapping option is not available in the installed version # of pg_basebackup. pg_version = Version(postgres.server_txt_version) tablespaces_list = postgres.get_tablespaces() # pg_basebackup supports the tablespace-mapping option, # so there are no problems in this case if remote_status["pg_basebackup_tbls_mapping"]: hint = None check_result = True # pg_basebackup doesn't support the tablespace-mapping option # and the data directory contains tablespaces, we can't correctly # backup it. elif tablespaces_list: check_result = False if pg_version < "9.3": hint = ( "pg_basebackup can't be used with tablespaces " "and PostgreSQL older than 9.3" ) else: hint = "pg_basebackup 9.4 or higher is required for tablespaces support" # Even if pg_basebackup doesn't support the tablespace-mapping # option, this location can be correctly backed up as doesn't # have any tablespaces else: check_result = True if pg_version < "9.3": hint = ( "pg_basebackup can be used as long as tablespaces " "support is not required" ) else: hint = "pg_basebackup 9.4 or higher is required for tablespaces support" check_strategy.result(self.config.name, check_result, hint=hint) def fetch_remote_status(self): """ Gather info from the remote server. This method does not raise any exception in case of errors, but set the missing values to None in the resulting dictionary. """ remote_status = dict.fromkeys( ( "pg_basebackup_compatible", "pg_basebackup_installed", "pg_basebackup_tbls_mapping", "pg_basebackup_path", "pg_basebackup_bwlimit", "pg_basebackup_version", ), None, ) # Test pg_basebackup existence version_info = PgBaseBackup.get_version_info(self.server.path) if version_info["full_path"]: remote_status["pg_basebackup_installed"] = True remote_status["pg_basebackup_path"] = version_info["full_path"] remote_status["pg_basebackup_version"] = version_info["full_version"] pgbasebackup_version = version_info["major_version"] else: remote_status["pg_basebackup_installed"] = False return remote_status # Is bandwidth limit supported? if ( remote_status["pg_basebackup_version"] is not None and remote_status["pg_basebackup_version"] < "9.4" ): remote_status["pg_basebackup_bwlimit"] = False else: remote_status["pg_basebackup_bwlimit"] = True # Is the tablespace mapping option supported? if pgbasebackup_version >= "9.4": remote_status["pg_basebackup_tbls_mapping"] = True else: remote_status["pg_basebackup_tbls_mapping"] = False # Retrieve the PostgreSQL version pg_version = None if self.server.streaming is not None: pg_version = self.server.streaming.server_major_version # If any of the two versions is unknown, we can't compare them if pgbasebackup_version is None or pg_version is None: # Return here. We are unable to retrieve # pg_basebackup or PostgreSQL versions return remote_status # pg_version is not None so transform into a Version object # for easier comparison between versions pg_version = Version(pg_version) # pg_basebackup 9.2 is compatible only with PostgreSQL 9.2. if "9.2" == pg_version == pgbasebackup_version: remote_status["pg_basebackup_compatible"] = True # other versions are compatible with lesser versions of PostgreSQL # WARNING: The development versions of `pg_basebackup` are considered # higher than the stable versions here, but this is not an issue # because it accepts everything that is less than # the `pg_basebackup` version(e.g. '9.6' is less than '9.6devel') elif "9.2" < pg_version <= pgbasebackup_version: remote_status["pg_basebackup_compatible"] = True else: remote_status["pg_basebackup_compatible"] = False return remote_status def backup_copy(self, backup_info): """ Perform the actual copy of the backup using pg_basebackup. First, manages tablespaces, then copies the base backup using the streaming protocol. In case of failure during the execution of the pg_basebackup command the method raises a DataTransferFailure, this trigger the retrying mechanism when necessary. :param barman.infofile.LocalBackupInfo backup_info: backup information """ # Make sure the destination directory exists, ensure the # right permissions to the destination dir backup_dest = backup_info.get_data_directory() dest_dirs = [backup_dest] # Store the start time self.copy_start_time = datetime.datetime.now() # Manage tablespaces, we need to handle them now in order to # be able to relocate them inside the # destination directory of the basebackup tbs_map = {} if backup_info.tablespaces: for tablespace in backup_info.tablespaces: source = tablespace.location destination = backup_info.get_data_directory(tablespace.oid) tbs_map[source] = destination dest_dirs.append(destination) # Prepare the destination directories for pgdata and tablespaces self._prepare_backup_destination(dest_dirs) # Retrieve pg_basebackup version information remote_status = self.get_remote_status() # If pg_basebackup supports --max-rate set the bandwidth_limit bandwidth_limit = None if remote_status["pg_basebackup_bwlimit"]: bandwidth_limit = self.config.bandwidth_limit # Make sure we are not wasting precious PostgreSQL resources # for the whole duration of the copy self.server.close() # Find the backup_manifest file path of the parent backup in case # it is an incremental backup parent_backup_info = backup_info.get_parent_backup_info() parent_backup_manifest_path = None if parent_backup_info: parent_backup_manifest_path = parent_backup_info.get_backup_manifest_path() pg_basebackup = PgBaseBackup( connection=self.server.streaming, destination=backup_dest, command=remote_status["pg_basebackup_path"], version=remote_status["pg_basebackup_version"], app_name=self.config.streaming_backup_name, tbs_mapping=tbs_map, bwlimit=bandwidth_limit, immediate=self.config.immediate_checkpoint, path=self.server.path, retry_times=self.config.basebackup_retry_times, retry_sleep=self.config.basebackup_retry_sleep, retry_handler=partial(self._retry_handler, dest_dirs), compression=self.backup_compression, err_handler=self._err_handler, out_handler=PgBaseBackup.make_logging_handler(logging.INFO), parent_backup_manifest_path=parent_backup_manifest_path, ) # Do the actual copy try: pg_basebackup() except CommandFailedException as e: msg = ( "data transfer failure on directory '%s'" % backup_info.get_data_directory() ) raise DataTransferFailure.from_command_error("pg_basebackup", e, msg) # Store the end time self.copy_end_time = datetime.datetime.now() # Store statistics about the copy copy_time = total_seconds(self.copy_end_time - self.copy_start_time) backup_info.copy_stats = { "copy_time": copy_time, "total_time": copy_time, } # Check for the presence of configuration files outside the PGDATA external_config = backup_info.get_external_config_files() if any(external_config): msg = ( "pg_basebackup does not copy the PostgreSQL " "configuration files that reside outside PGDATA. " "Please manually backup the following files:\n" "\t%s\n" % "\n\t".join(ecf.path for ecf in external_config) ) # Show the warning only if the EXTERNAL_CONFIGURATION option # is not specified in the backup_options. if BackupOptions.EXTERNAL_CONFIGURATION not in self.config.backup_options: output.warning(msg) else: _logger.debug(msg) def _retry_handler(self, dest_dirs, command, args, kwargs, attempt, exc): """ Handler invoked during a backup in case of retry. The method simply warn the user of the failure and remove the already existing directories of the backup. :param list[str] dest_dirs: destination directories :param RsyncPgData command: Command object being executed :param list args: command args :param dict kwargs: command kwargs :param int attempt: attempt number (starting from 0) :param CommandFailedException exc: the exception which caused the failure """ output.warning( "Failure executing a backup using pg_basebackup (attempt %s)", attempt ) output.warning( "The files copied so far will be removed and " "the backup process will restart in %s seconds", self.config.basebackup_retry_sleep, ) # Remove all the destination directories and reinit the backup self._prepare_backup_destination(dest_dirs) def _err_handler(self, line): """ Handler invoked during a backup when anything is sent to stderr. Used to perform a WAL switch on a primary server if pg_basebackup is running against a standby, otherwise just logs output at INFO level. :param str line: The error line to be handled. """ # Always log the line, since this handler will have overridden the # default command err_handler. # Although this is used as a stderr handler, the pg_basebackup lines # logged here are more appropriate at INFO level since they are just # describing regular behaviour. _logger.log(logging.INFO, "%s", line) if ( self.server.config.primary_conninfo is not None and "waiting for required WAL segments to be archived" in line ): # If pg_basebackup is waiting for WAL segments and primary_conninfo # is configured then we are backing up a standby and must manually # perform a WAL switch. self.server.postgres.switch_wal() def _prepare_backup_destination(self, dest_dirs): """ Prepare the destination of the backup, including tablespaces. This method is also responsible for removing a directory if it already exists and for ensuring the correct permissions for the created directories :param list[str] dest_dirs: destination directories """ for dest_dir in dest_dirs: # Remove a dir if exists. Ignore eventual errors shutil.rmtree(dest_dir, ignore_errors=True) # create the dir mkpath(dest_dir) # Ensure the right permissions to the destination directory # chmod 0700 octal os.chmod(dest_dir, 448) def _start_backup_copy_message(self, backup_info): output.info( "Starting backup copy via pg_basebackup for %s", backup_info.backup_id ) class ExternalBackupExecutor(with_metaclass(ABCMeta, BackupExecutor)): """ Abstract base class for non-postgres backup executors. An external backup executor is any backup executor which uses the PostgreSQL low-level backup API to coordinate the backup. Such executors can operate remotely via SSH or locally: - remote mode (default), operates via SSH - local mode, operates as the same user that Barman runs with It is also a factory for exclusive/concurrent backup strategy objects. Raises a SshCommandException if 'ssh_command' is not set and not operating in local mode. """ def __init__(self, backup_manager, mode, local_mode=False): """ Constructor of the abstract class for backups via Ssh :param barman.backup.BackupManager backup_manager: the BackupManager assigned to the executor :param str mode: The mode used by the executor for the backup. :param bool local_mode: if set to False (default), the class is able to operate on remote servers using SSH. Operates only locally if set to True. """ super(ExternalBackupExecutor, self).__init__(backup_manager, mode) # Set local/remote mode for copy self.local_mode = local_mode # Retrieve the ssh command and the options necessary for the # remote ssh access. self.ssh_command, self.ssh_options = _parse_ssh_command( backup_manager.config.ssh_command ) if not self.local_mode: # Remote copy requires ssh_command to be set if not self.ssh_command: raise SshCommandException( "Missing or invalid ssh_command in barman configuration " "for server %s" % backup_manager.config.name ) else: # Local copy requires ssh_command not to be set if self.ssh_command: raise SshCommandException( "Local copy requires ssh_command in barman configuration " "to be empty for server %s" % backup_manager.config.name ) # Apply the default backup strategy backup_options = self.config.backup_options concurrent_backup = BackupOptions.CONCURRENT_BACKUP in backup_options exclusive_backup = BackupOptions.EXCLUSIVE_BACKUP in backup_options if not concurrent_backup and not exclusive_backup: self.config.backup_options.add(BackupOptions.CONCURRENT_BACKUP) output.warning( "No backup strategy set for server '%s' " "(using default 'concurrent_backup').", self.config.name, ) # Depending on the backup options value, create the proper strategy if BackupOptions.CONCURRENT_BACKUP in self.config.backup_options: # Concurrent backup strategy self.strategy = LocalConcurrentBackupStrategy( self.server.postgres, self.config.name ) else: # Exclusive backup strategy self.strategy = ExclusiveBackupStrategy( self.server.postgres, self.config.name ) def _update_action_from_strategy(self): """ Update the executor's current action with the one of the strategy. This is used during exception handling to let the caller know where the failure occurred. """ action = getattr(self.strategy, "current_action", None) if action: self.current_action = action @abstractmethod def backup_copy(self, backup_info): """ Performs the actual copy of a backup for the server :param barman.infofile.LocalBackupInfo backup_info: backup information """ def backup(self, backup_info): """ Perform a backup for the server - invoked by BackupManager.backup() through the generic interface of a BackupExecutor. This implementation is responsible for performing a backup through a remote connection to the PostgreSQL server via Ssh. The specific set of instructions depends on both the specific class that derives from ExternalBackupExecutor and the selected strategy (e.g. exclusive backup through Rsync). :param barman.infofile.LocalBackupInfo backup_info: backup information """ # Start the backup, all the subsequent code must be wrapped in a # try except block which finally issues a stop_backup command try: self.strategy.start_backup(backup_info) except BaseException: self._update_action_from_strategy() raise connection_error = False try: # save any metadata changed by start_backup() call # This must be inside the try-except, because it could fail backup_info.save() if backup_info.begin_wal is not None: output.info( "Backup start at LSN: %s (%s, %08X)", backup_info.begin_xlog, backup_info.begin_wal, backup_info.begin_offset, ) else: output.info("Backup start at LSN: %s", backup_info.begin_xlog) # If this is the first backup, purge eventually unused WAL files self._purge_unused_wal_files(backup_info) # Start the copy self.current_action = "copying files" self._start_backup_copy_message(backup_info) self.backup_copy(backup_info) self._stop_backup_copy_message(backup_info) # Try again to purge eventually unused WAL files. At this point # the begin_wal value is surely known. Doing it twice is safe # because this function is useful only during the first backup. self._purge_unused_wal_files(backup_info) except PostgresConnectionLost: # This exception is most likely to be raised by the PostgresKeepAlive, # meaning that we lost the connection (and session) during the backup. connection_error = True raise except BaseException as ex: # we do not need to do anything here besides re-raising the # exception. It will be handled in the external try block. output.error("The backup has failed %s", self.current_action) # As we have found that in certain corner cases the exception # passing through this block is not logged or even hidden by # other exceptions happening in the finally block, we are adding a # debug log line to make sure that the exception is visible. _logger.debug("Backup failed: %s" % ex, exc_info=True) raise else: self.current_action = "issuing stop of the backup" finally: # If a connection error has been raised, it means we lost our session in the # Postgres server. In such cases, it's useless to try a backup stop command. if not connection_error: output.info("Asking PostgreSQL server to finalize the backup.") try: self.strategy.stop_backup(backup_info) except BaseException: self._update_action_from_strategy() raise def _local_check(self, check_strategy): """ Specific checks for local mode of ExternalBackupExecutor (same user) :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ cmd = UnixLocalCommand(path=self.server.path) pgdata = self.server.postgres.get_setting("data_directory") # Check that PGDATA is accessible check_strategy.init_check("local PGDATA") hint = "Access to local PGDATA" try: cmd.check_directory_exists(pgdata) except FsOperationFailed as e: hint = force_str(e).strip() # Output the result check_strategy.result(self.config.name, cmd is not None, hint=hint) def _remote_check(self, check_strategy): """ Specific checks for remote mode of ExternalBackupExecutor, via SSH. :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ # Check the SSH connection check_strategy.init_check("ssh") hint = "PostgreSQL server" cmd = None minimal_ssh_output = None try: cmd = UnixRemoteCommand( self.ssh_command, self.ssh_options, path=self.server.path ) minimal_ssh_output = "".join(cmd.get_last_output()) except FsOperationFailed as e: hint = force_str(e).strip() # Output the result check_strategy.result(self.config.name, cmd is not None, hint=hint) # Check that the communication channel is "clean" if minimal_ssh_output: check_strategy.init_check("ssh output clean") check_strategy.result( self.config.name, False, hint="the configured ssh_command must not add anything to " "the remote command output", ) # If SSH works but PostgreSQL is not responding server_txt_version = self.server.get_remote_status().get("server_txt_version") if cmd is not None and server_txt_version is None: # Check for 'backup_label' presence last_backup = self.server.get_backup( self.server.get_last_backup_id(BackupInfo.STATUS_NOT_EMPTY) ) # Look for the latest backup in the catalogue if last_backup: check_strategy.init_check("backup_label") # Get PGDATA and build path to 'backup_label' backup_label = os.path.join(last_backup.pgdata, "backup_label") # Verify that backup_label exists in the remote PGDATA. # If so, send an alert. Do not show anything if OK. exists = cmd.exists(backup_label) if exists: hint = ( "Check that the PostgreSQL server is up " "and no 'backup_label' file is in PGDATA." ) check_strategy.result(self.config.name, False, hint=hint) def check(self, check_strategy): """ Perform additional checks for ExternalBackupExecutor, including Ssh connection (executing a 'true' command on the remote server) and specific checks for the given backup strategy. :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ if self.local_mode: # Perform checks for the local case self._local_check(check_strategy) else: # Perform checks for the remote case self._remote_check(check_strategy) try: # Invoke specific checks for the backup strategy self.strategy.check(check_strategy) except BaseException: self._update_action_from_strategy() raise def status(self): """ Set additional status info for ExternalBackupExecutor using remote commands via Ssh, as well as those defined by the given backup strategy. """ try: # Invoke the status() method for the given strategy self.strategy.status() except BaseException: self._update_action_from_strategy() raise def fetch_remote_status(self): """ Get remote information on PostgreSQL using Ssh, such as last archived WAL file This method does not raise any exception in case of errors, but set the missing values to None in the resulting dictionary. :rtype: dict[str, None|str] """ remote_status = {} # Retrieve the last archived WAL using a Ssh connection on # the remote server and executing an 'ls' command. Only # for pre-9.4 versions of PostgreSQL. try: if self.server.postgres and self.server.postgres.server_version < 90400: remote_status["last_archived_wal"] = None if self.server.postgres.get_setting( "data_directory" ) and self.server.postgres.get_setting("archive_command"): if not self.local_mode: cmd = UnixRemoteCommand( self.ssh_command, self.ssh_options, path=self.server.path ) else: cmd = UnixLocalCommand(path=self.server.path) # Here the name of the PostgreSQL WALs directory is # hardcoded, but that doesn't represent a problem as # this code runs only for PostgreSQL < 9.4 archive_dir = os.path.join( self.server.postgres.get_setting("data_directory"), "pg_xlog", "archive_status", ) out = str(cmd.list_dir_content(archive_dir, ["-t"])) for line in out.splitlines(): if line.endswith(".done"): name = line[:-5] if xlog.is_any_xlog_file(name): remote_status["last_archived_wal"] = name break except (PostgresConnectionError, FsOperationFailed) as e: _logger.warning("Error retrieving PostgreSQL status: %s", e) return remote_status class PassiveBackupExecutor(BackupExecutor): """ Dummy backup executors for Passive servers. Raises a SshCommandException if 'primary_ssh_command' is not set. """ def __init__(self, backup_manager): """ Constructor of Dummy backup executors for Passive servers. :param barman.backup.BackupManager backup_manager: the BackupManager assigned to the executor """ super(PassiveBackupExecutor, self).__init__(backup_manager) # Retrieve the ssh command and the options necessary for the # remote ssh access. self.ssh_command, self.ssh_options = _parse_ssh_command( backup_manager.config.primary_ssh_command ) # Requires ssh_command to be set if not self.ssh_command: raise SshCommandException( "Invalid primary_ssh_command in barman configuration " "for server %s" % backup_manager.config.name ) def backup(self, backup_info): """ This method should never be called, because this is a passive server :param barman.infofile.LocalBackupInfo backup_info: backup information """ # The 'backup' command is not available on a passive node. # If we get here, there is a programming error assert False def check(self, check_strategy): """ Perform additional checks for PassiveBackupExecutor, including Ssh connection to the primary (executing a 'true' command on the remote server). :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ check_strategy.init_check("ssh") hint = "Barman primary node" cmd = None minimal_ssh_output = None try: cmd = UnixRemoteCommand( self.ssh_command, self.ssh_options, path=self.server.path ) minimal_ssh_output = "".join(cmd.get_last_output()) except FsOperationFailed as e: hint = force_str(e).strip() # Output the result check_strategy.result(self.config.name, cmd is not None, hint=hint) # Check if the communication channel is "clean" if minimal_ssh_output: check_strategy.init_check("ssh output clean") check_strategy.result( self.config.name, False, hint="the configured ssh_command must not add anything to " "the remote command output", ) def status(self): """ Set additional status info for PassiveBackupExecutor. """ # On passive nodes show the primary_ssh_command output.result( "status", self.config.name, "primary_ssh_command", "SSH command to primary server", self.config.primary_ssh_command, ) @property def mode(self): """ Property that defines the mode used for the backup. :return str: a string describing the mode used for the backup """ return "passive" class RsyncBackupExecutor(ExternalBackupExecutor): """ Concrete class for backup via Rsync+Ssh. It invokes PostgreSQL commands to start and stop the backup, depending on the defined strategy. Data files are copied using Rsync via Ssh. It heavily relies on methods defined in the ExternalBackupExecutor class from which it derives. """ def __init__(self, backup_manager, local_mode=False): """ Constructor :param barman.backup.BackupManager backup_manager: the BackupManager assigned to the strategy """ super(RsyncBackupExecutor, self).__init__(backup_manager, "rsync", local_mode) self.validate_configuration() def validate_configuration(self): # Verify that backup_compression is not set if self.server.config.backup_compression: self.server.config.update_msg_list_and_disable_server( "backup_compression option is not supported by rsync backup_method" ) def backup(self, *args, **kwargs): """ Perform an Rsync backup. .. note:: This method currently only calls the parent backup method but inside a keepalive context to ensure the connection does not become idle long enough to get dropped by a firewall, for instance. This is important to ensure that ``pg_backup_start()`` and ``pg_backup_stop()`` are called within the same session. """ try: with PostgresKeepAlive( self.server.postgres, self.config.keepalive_interval, True ): super(RsyncBackupExecutor, self).backup(*args, **kwargs) except PostgresConnectionLost: raise BackupException( "Connection to the Postgres server was lost during the backup." ) def backup_copy(self, backup_info): """ Perform the actual copy of the backup using Rsync. First, it copies one tablespace at a time, then the PGDATA directory, and finally configuration files (if outside PGDATA). Bandwidth limitation, according to configuration, is applied in the process. This method is the core of base backup copy using Rsync+Ssh. :param barman.infofile.LocalBackupInfo backup_info: backup information """ # Retrieve the previous backup metadata, then calculate safe_horizon previous_backup = self.backup_manager.get_previous_backup(backup_info.backup_id) safe_horizon = None reuse_backup = None # Store the start time self.copy_start_time = datetime.datetime.now() if previous_backup: # safe_horizon is a tz-aware timestamp because BackupInfo class # ensures that property reuse_backup = self.config.reuse_backup safe_horizon = previous_backup.begin_time # Create the copy controller object, specific for rsync, # which will drive all the copy operations. Items to be # copied are added before executing the copy() method controller = RsyncCopyController( path=self.server.path, ssh_command=self.ssh_command, ssh_options=self.ssh_options, network_compression=self.config.network_compression, reuse_backup=reuse_backup, safe_horizon=safe_horizon, retry_times=self.config.basebackup_retry_times, retry_sleep=self.config.basebackup_retry_sleep, workers=self.config.parallel_jobs, workers_start_batch_period=self.config.parallel_jobs_start_batch_period, workers_start_batch_size=self.config.parallel_jobs_start_batch_size, ) # List of paths to be excluded by the PGDATA copy exclude_and_protect = [] # Process every tablespace if backup_info.tablespaces: for tablespace in backup_info.tablespaces: # If the tablespace location is inside the data directory, # exclude and protect it from being copied twice during # the data directory copy if tablespace.location.startswith(backup_info.pgdata + "/"): exclude_and_protect += [ tablespace.location[len(backup_info.pgdata) :] ] # Exclude and protect the tablespace from being copied again # during the data directory copy exclude_and_protect += ["/pg_tblspc/%s" % tablespace.oid] # Make sure the destination directory exists in order for # smart copy to detect that no file is present there tablespace_dest = backup_info.get_data_directory(tablespace.oid) mkpath(tablespace_dest) # Add the tablespace directory to the list of objects # to be copied by the controller. # NOTE: Barman should archive only the content of directory # "PG_" + PG_MAJORVERSION + "_" + CATALOG_VERSION_NO # but CATALOG_VERSION_NO is not easy to retrieve, so we copy # "PG_" + PG_MAJORVERSION + "_*" # It could select some spurious directory if a development or # a beta version have been used, but it's good enough for a # production system as it filters out other major versions. controller.add_directory( label=tablespace.name, src="%s/" % self._format_src(tablespace.location), dst=tablespace_dest, exclude=["/*"] + EXCLUDE_LIST, include=["/PG_%s_*" % self.server.postgres.server_major_version], bwlimit=self.config.get_bwlimit(tablespace), reuse=self._reuse_path(previous_backup, tablespace), item_class=controller.TABLESPACE_CLASS, ) # Make sure the destination directory exists in order for smart copy # to detect that no file is present there backup_dest = backup_info.get_data_directory() mkpath(backup_dest) # Add the PGDATA directory to the list of objects to be copied # by the controller controller.add_directory( label="pgdata", src="%s/" % self._format_src(backup_info.pgdata), dst=backup_dest, exclude=PGDATA_EXCLUDE_LIST + EXCLUDE_LIST, exclude_and_protect=exclude_and_protect, bwlimit=self.config.get_bwlimit(), reuse=self._reuse_path(previous_backup), item_class=controller.PGDATA_CLASS, ) # At last copy pg_control controller.add_file( label="pg_control", src="%s/global/pg_control" % self._format_src(backup_info.pgdata), dst="%s/global/pg_control" % (backup_dest,), item_class=controller.PGCONTROL_CLASS, ) # Copy configuration files (if not inside PGDATA) external_config_files = backup_info.get_external_config_files() included_config_files = [] for config_file in external_config_files: # Add included files to a list, they will be handled later if config_file.file_type == "include": included_config_files.append(config_file) continue # If the ident file is missing, it isn't an error condition # for PostgreSQL. # Barman is consistent with this behavior. optional = False if config_file.file_type == "ident_file": optional = True # Create the actual copy jobs in the controller controller.add_file( label=config_file.file_type, src=self._format_src(config_file.path), dst=backup_dest, optional=optional, item_class=controller.CONFIG_CLASS, ) # Execute the copy try: controller.copy() # TODO: Improve the exception output except CommandFailedException as e: msg = "data transfer failure" raise DataTransferFailure.from_command_error("rsync", e, msg) # Store the end time self.copy_end_time = datetime.datetime.now() # Store statistics about the copy backup_info.copy_stats = controller.statistics() # Check for any include directives in PostgreSQL configuration # Currently, include directives are not supported for files that # reside outside PGDATA. These files must be manually backed up. # Barman will emit a warning and list those files if any(included_config_files): msg = ( "The usage of include directives is not supported " "for files that reside outside PGDATA.\n" "Please manually backup the following files:\n" "\t%s\n" % "\n\t".join(icf.path for icf in included_config_files) ) # Show the warning only if the EXTERNAL_CONFIGURATION option # is not specified in the backup_options. if BackupOptions.EXTERNAL_CONFIGURATION not in self.config.backup_options: output.warning(msg) else: _logger.debug(msg) def _reuse_path(self, previous_backup_info, tablespace=None): """ If reuse_backup is 'copy' or 'link', builds the path of the directory to reuse, otherwise always returns None. If oid is None, it returns the full path of PGDATA directory of the previous_backup otherwise it returns the path to the specified tablespace using it's oid. :param barman.infofile.LocalBackupInfo previous_backup_info: backup to be reused :param barman.infofile.Tablespace tablespace: the tablespace to copy :returns: a string containing the local path with data to be reused or None :rtype: str|None """ oid = None if tablespace: oid = tablespace.oid if ( self.config.reuse_backup in ("copy", "link") and previous_backup_info is not None ): try: return previous_backup_info.get_data_directory(oid) except ValueError: return None def _format_src(self, path): """ If the executor is operating in remote mode, add a `:` in front of the path for rsync to work via SSH. :param string path: the path to format :return str: the formatted path string """ if not self.local_mode: return ":%s" % path return path def _start_backup_copy_message(self, backup_info): """ Output message for backup start. :param barman.infofile.LocalBackupInfo backup_info: backup information """ number_of_workers = self.config.parallel_jobs via = "rsync/SSH" if self.local_mode: via = "local rsync" message = "Starting backup copy via %s for %s" % ( via, backup_info.backup_id, ) if number_of_workers > 1: message += " (%s jobs)" % number_of_workers output.info(message) class SnapshotBackupExecutor(ExternalBackupExecutor): """ Concrete class which uses cloud provider disk snapshots to create backups. It invokes PostgreSQL commands to start and stop the backup, depending on the defined strategy. It heavily relies on methods defined in the ExternalBackupExecutor class from which it derives. No data files are copied and instead snapshots are created of the requested disks using the cloud provider API (abstracted through a CloudSnapshotInterface). As well as ensuring the backup happens via snapshot copy, this class also: - Checks that the specified disks are attached to the named instance. - Checks that the specified disks are mounted on the named instance. - Records the mount points and options of each disk in the backup info. Barman will still store the following files in its backup directory: - The backup_label (for concurrent backups) which is written by the LocalConcurrentBackupStrategy. - The backup.info which is written by the BackupManager responsible for instantiating this class. """ def __init__(self, backup_manager): """ Constructor for the SnapshotBackupExecutor :param barman.backup.BackupManager backup_manager: the BackupManager assigned to the strategy """ super(SnapshotBackupExecutor, self).__init__(backup_manager, "snapshot") self.snapshot_instance = self.config.snapshot_instance self.snapshot_disks = self.config.snapshot_disks self.validate_configuration() try: self.snapshot_interface = get_snapshot_interface_from_server_config( self.config ) except Exception as exc: self.server.config.update_msg_list_and_disable_server( "Error initialising snapshot provider %s: %s" % (self.config.snapshot_provider, exc) ) def validate_configuration(self): """Verify configuration is valid for a snapshot backup.""" excluded_config = ( "backup_compression", "bandwidth_limit", "network_compression", "tablespace_bandwidth_limit", ) for config_var in excluded_config: if getattr(self.server.config, config_var): self.server.config.update_msg_list_and_disable_server( "%s option is not supported by snapshot backup_method" % config_var ) if self.config.reuse_backup in ("copy", "link"): self.server.config.update_msg_list_and_disable_server( "reuse_backup option is not supported by snapshot backup_method" ) required_config = ( "snapshot_disks", "snapshot_instance", "snapshot_provider", ) for config_var in required_config: if not getattr(self.server.config, config_var): self.server.config.update_msg_list_and_disable_server( "%s option is required by snapshot backup_method" % config_var ) # Check if aws_snapshot_lock_mode is set with snapshot_provider = aws. if ( getattr(self.server.config, "aws_snapshot_lock_mode") and getattr(self.server.config, "snapshot_provider") == "aws" ): self._validate_aws_lock_configuration() def _validate_aws_lock_configuration(self): """Verify configuration is valid for locking a snapshot backup using AWS provider. """ if not ( getattr(self.server.config, "aws_snapshot_lock_duration") or getattr(self.server.config, "aws_snapshot_lock_expiration_date") ): self.server.config.update_msg_list_and_disable_server( "'aws_snapshot_lock_mode' is set, you must specify " "'aws_snapshot_lock_duration' or 'aws_snapshot_lock_expiration_date'." ) if getattr(self.server.config, "aws_snapshot_lock_duration") and getattr( self.server.config, "aws_snapshot_lock_expiration_date" ): self.server.config.update_msg_list_and_disable_server( "You must specify either 'aws_snapshot_lock_duration' or " "'aws_snapshot_lock_expiration_date' in the configuration, but not both." ) if getattr( self.server.config, "aws_snapshot_lock_mode" ) == "governance" and getattr( self.server.config, "aws_snapshot_lock_cool_off_period" ): self.server.config.update_msg_list_and_disable_server( "'aws_snapshot_lock_cool_off_period' cannot be used with " "'aws_snapshot_lock_mode' = 'governance'." ) lock_args = { "aws_snapshot_lock_cool_off_period": check_aws_snapshot_lock_cool_off_period_range, "aws_snapshot_lock_duration": check_aws_snapshot_lock_duration_range, "aws_snapshot_lock_mode": check_aws_snapshot_lock_mode, "aws_snapshot_lock_expiration_date": check_aws_expiration_date_format, } for arg, parser in lock_args.items(): lock_arg = getattr(self.server.config, arg) if lock_arg: try: _ = parser(lock_arg) except Exception as e: error_message = str(e) self.server.config.update_msg_list_and_disable_server(error_message) @staticmethod def add_mount_data_to_volume_metadata(volumes, remote_cmd): """ Adds the mount point and mount options for each supplied volume. Calls `resolve_mounted_volume` on each supplied volume so that the volume metadata (which originated from the cloud provider) can be resolved to the mount point and mount options of the volume as mounted on a compute instance. This will set the current mount point and mount options of the volume so that they can be stored in the snapshot metadata for the backup when the backup is taken. :param dict[str,barman.cloud.VolumeMetadata] volumes: Metadata for the volumes attached to a specific compute instance. :param UnixLocalCommand remote_cmd: Wrapper for executing local/remote commands on the compute instance to which the volumes are attached. """ for volume in volumes.values(): volume.resolve_mounted_volume(remote_cmd) def backup_copy(self, backup_info): """ Perform the backup using cloud provider disk snapshots. :param barman.infofile.LocalBackupInfo backup_info: Backup information. """ # Create data dir so backup_label can be written cmd = UnixLocalCommand(path=self.server.path) cmd.create_dir_if_not_exists(backup_info.get_data_directory()) # Start the snapshot self.copy_start_time = datetime.datetime.now() # Get volume metadata for the disks to be backed up volumes_to_snapshot = self.snapshot_interface.get_attached_volumes( self.snapshot_instance, self.snapshot_disks ) # Resolve volume metadata to mount metadata using shell commands on the # compute instance to which the volumes are attached - this information # can then be added to the metadata for each snapshot when the backup is # taken. remote_cmd = UnixRemoteCommand(ssh_command=self.server.config.ssh_command) self.add_mount_data_to_volume_metadata(volumes_to_snapshot, remote_cmd) self.snapshot_interface.take_snapshot_backup( backup_info, self.snapshot_instance, volumes_to_snapshot, ) self.copy_end_time = datetime.datetime.now() # Store statistics about the copy copy_time = total_seconds(self.copy_end_time - self.copy_start_time) backup_info.copy_stats = { "copy_time": copy_time, "total_time": copy_time, } @staticmethod def find_missing_and_unmounted_disks( cmd, snapshot_interface, snapshot_instance, snapshot_disks ): """ Checks for any disks listed in snapshot_disks which are not correctly attached and mounted on the named instance and returns them as a tuple of two lists. This is used for checking that the disks which are to be used as sources for snapshots at backup time are attached and mounted on the instance to be backed up. :param UnixLocalCommand cmd: Wrapper for local/remote commands. :param barman.cloud.CloudSnapshotInterface snapshot_interface: Interface for taking snapshots and associated operations via cloud provider APIs. :param str snapshot_instance: The name of the VM instance to which the disks to be backed up are attached. :param list[str] snapshot_disks: A list containing the names of the disks for which snapshots should be taken at backup time. :rtype tuple[list[str],list[str]] :return: A tuple where the first element is a list of all disks which are not attached to the VM instance and the second element is a list of all disks which are attached but not mounted. """ attached_volumes = snapshot_interface.get_attached_volumes( snapshot_instance, snapshot_disks, fail_on_missing=False ) missing_disks = [] for disk in snapshot_disks: if disk not in attached_volumes.keys(): missing_disks.append(disk) unmounted_disks = [] for disk in snapshot_disks: try: attached_volumes[disk].resolve_mounted_volume(cmd) mount_point = attached_volumes[disk].mount_point except KeyError: # Ignore disks which were not attached continue except SnapshotBackupException as exc: logging.warn("Error resolving mount point: {}".format(exc)) mount_point = None if mount_point is None: unmounted_disks.append(disk) return missing_disks, unmounted_disks def check(self, check_strategy): """ Perform additional checks for SnapshotBackupExecutor, specifically: - check that the VM instance for which snapshots should be taken exists - check that the expected disks are attached to that instance - check that the attached disks are mounted on the filesystem :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ super(SnapshotBackupExecutor, self).check(check_strategy) if self.server.config.disabled: # Skip checks if the server is not active return check_strategy.init_check("snapshot instance exists") if not self.snapshot_interface.instance_exists(self.snapshot_instance): check_strategy.result( self.config.name, False, hint="cannot find compute instance %s" % self.snapshot_instance, ) return else: check_strategy.result(self.config.name, True) check_strategy.init_check("snapshot disks attached to instance") cmd = unix_command_factory(self.config.ssh_command, self.server.path) missing_disks, unmounted_disks = self.find_missing_and_unmounted_disks( cmd, self.snapshot_interface, self.snapshot_instance, self.snapshot_disks, ) if len(missing_disks) > 0: check_strategy.result( self.config.name, False, hint="cannot find snapshot disks attached to instance %s: %s" % (self.snapshot_instance, ", ".join(missing_disks)), ) else: check_strategy.result(self.config.name, True) check_strategy.init_check("snapshot disks mounted on instance") if len(unmounted_disks) > 0: check_strategy.result( self.config.name, False, hint="cannot find snapshot disks mounted on instance %s: %s" % (self.snapshot_instance, ", ".join(unmounted_disks)), ) else: check_strategy.result(self.config.name, True) def _start_backup_copy_message(self, backup_info): """ Output message for backup start. :param barman.infofile.LocalBackupInfo backup_info: Backup information. """ output.info("Starting backup with disk snapshots for %s", backup_info.backup_id) def _stop_backup_copy_message(self, backup_info): """ Output message for backup end. :param barman.infofile.LocalBackupInfo backup_info: Backup information. """ output.info( "Snapshot backup done (time: %s)", human_readable_timedelta( datetime.timedelta(seconds=backup_info.copy_stats["copy_time"]) ), ) class BackupStrategy(with_metaclass(ABCMeta, object)): """ Abstract base class for a strategy to be used by a backup executor. """ #: Regex for START WAL LOCATION info START_TIME_RE = re.compile(r"^START TIME: (.*)", re.MULTILINE) #: Regex for START TIME info WAL_RE = re.compile(r"^START WAL LOCATION: (.*) \(file (.*)\)", re.MULTILINE) def __init__(self, postgres, server_name, mode=None): """ Constructor :param barman.postgres.PostgreSQLConnection postgres: the PostgreSQL connection :param str server_name: The name of the server """ self.postgres = postgres self.server_name = server_name # Holds the action being executed. Used for error messages. self.current_action = None self.mode = mode def start_backup(self, backup_info): """ Issue a start of a backup - invoked by BackupExecutor.backup() :param barman.infofile.BackupInfo backup_info: backup information """ # Retrieve PostgreSQL server metadata self._pg_get_metadata(backup_info) # Record that we are about to start the backup self.current_action = "issuing start backup command" _logger.debug(self.current_action) @abstractmethod def stop_backup(self, backup_info): """ Issue a stop of a backup - invoked by BackupExecutor.backup() :param barman.infofile.LocalBackupInfo backup_info: backup information """ @abstractmethod def check(self, check_strategy): """ Perform additional checks - invoked by BackupExecutor.check() :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ # noinspection PyMethodMayBeStatic def status(self): """ Set additional status info - invoked by BackupExecutor.status() """ def _pg_get_metadata(self, backup_info): """ Load PostgreSQL metadata into the backup_info parameter :param barman.infofile.BackupInfo backup_info: backup information """ # Get the PostgreSQL data directory location self.current_action = "detecting data directory" output.debug(self.current_action) data_directory = self.postgres.get_setting("data_directory") backup_info.set_attribute("pgdata", data_directory) # Set server version backup_info.set_attribute("version", self.postgres.server_version) # Set XLOG segment size backup_info.set_attribute("xlog_segment_size", self.postgres.xlog_segment_size) # Set configuration files location cf = self.postgres.get_configuration_files() for key in cf: backup_info.set_attribute(key, cf[key]) # Get tablespaces information self.current_action = "detecting tablespaces" output.debug(self.current_action) tablespaces = self.postgres.get_tablespaces() if tablespaces and len(tablespaces) > 0: backup_info.set_attribute("tablespaces", tablespaces) for item in tablespaces: msg = "\t%s, %s, %s" % (item.oid, item.name, item.location) _logger.info(msg) # Set data_checksums state data_checksums = self.postgres.get_setting("data_checksums") backup_info.set_attribute("data_checksums", data_checksums) # Get summarize_wal information for incremental backups # Postgres major version should be >= 17 backup_info.set_attribute("summarize_wal", None) if self.postgres.server_version >= 170000: summarize_wal = self.postgres.get_setting("summarize_wal") backup_info.set_attribute("summarize_wal", summarize_wal) # Set total size of the PostgreSQL server backup_info.set_attribute("cluster_size", self.postgres.current_size) @staticmethod def _backup_info_from_start_location(backup_info, start_info): """ Fill a backup info with information from a start_backup :param barman.infofile.BackupInfo backup_info: object representing a backup :param DictCursor start_info: the result of the pg_backup_start command """ backup_info.set_attribute("status", BackupInfo.STARTED) backup_info.set_attribute("begin_time", start_info["timestamp"]) backup_info.set_attribute("begin_xlog", start_info["location"]) # PostgreSQL 9.6+ directly provides the timeline if start_info.get("timeline") is not None: backup_info.set_attribute("timeline", start_info["timeline"]) # Take a copy of stop_info because we are going to update it start_info = start_info.copy() start_info.update( xlog.location_to_xlogfile_name_offset( start_info["location"], start_info["timeline"], backup_info.xlog_segment_size, ) ) # If file_name and file_offset are available, use them file_name = start_info.get("file_name") file_offset = start_info.get("file_offset") if file_name is not None and file_offset is not None: backup_info.set_attribute("begin_wal", start_info["file_name"]) backup_info.set_attribute("begin_offset", start_info["file_offset"]) # If the timeline is still missing, extract it from the file_name if backup_info.timeline is None: backup_info.set_attribute( "timeline", int(start_info["file_name"][0:8], 16) ) @staticmethod def _backup_info_from_stop_location(backup_info, stop_info): """ Fill a backup info with information from a backup stop location :param barman.infofile.BackupInfo backup_info: object representing a backup :param DictCursor stop_info: location info of stop backup """ # If file_name or file_offset are missing build them using the stop # location and the timeline. file_name = stop_info.get("file_name") file_offset = stop_info.get("file_offset") if file_name is None or file_offset is None: # Take a copy of stop_info because we are going to update it stop_info = stop_info.copy() # Get the timeline from the stop_info if available, otherwise # Use the one from the backup_label timeline = stop_info.get("timeline") if timeline is None: timeline = backup_info.timeline stop_info.update( xlog.location_to_xlogfile_name_offset( stop_info["location"], timeline, backup_info.xlog_segment_size ) ) backup_info.set_attribute("end_time", stop_info["timestamp"]) backup_info.set_attribute("end_xlog", stop_info["location"]) backup_info.set_attribute("end_wal", stop_info["file_name"]) backup_info.set_attribute("end_offset", stop_info["file_offset"]) def _backup_info_from_backup_label(self, backup_info): """ Fill a backup info with information from the backup_label file :param barman.infofile.BackupInfo backup_info: object representing a backup """ # The backup_label must be already loaded assert backup_info.backup_label # Parse backup label wal_info = self.WAL_RE.search(backup_info.backup_label) start_time = self.START_TIME_RE.search(backup_info.backup_label) if wal_info is None or start_time is None: raise ValueError( "Failure parsing backup_label for backup %s" % backup_info.backup_id ) # Set data in backup_info from backup_label backup_info.set_attribute("timeline", int(wal_info.group(2)[0:8], 16)) backup_info.set_attribute("begin_xlog", wal_info.group(1)) backup_info.set_attribute("begin_wal", wal_info.group(2)) backup_info.set_attribute( "begin_offset", xlog.parse_lsn(wal_info.group(1)) % backup_info.xlog_segment_size, ) # If we have already obtained a begin_time then it takes precedence over the # begin time in the backup label if not backup_info.begin_time: backup_info.set_attribute( "begin_time", dateutil.parser.parse(start_time.group(1)) ) def _read_backup_label(self, backup_info): """ Read the backup_label file :param barman.infofile.LocalBackupInfo backup_info: backup information """ self.current_action = "reading the backup label" label_path = os.path.join(backup_info.get_data_directory(), "backup_label") output.debug("Reading backup label: %s" % label_path) with open(label_path, "r") as f: backup_label = f.read() backup_info.set_attribute("backup_label", backup_label) class PostgresBackupStrategy(BackupStrategy): """ Concrete class for postgres backup strategy. This strategy is for PostgresBackupExecutor only and is responsible for executing pre e post backup operations during a physical backup executed using pg_basebackup. """ def __init__(self, postgres, server_name, backup_compression=None): """ Constructor :param barman.postgres.PostgreSQLConnection postgres: the PostgreSQL connection :param str server_name: The name of the server :param barman.compression.PgBaseBackupCompression backup_compression: the pg_basebackup compression options used for this backup """ super(PostgresBackupStrategy, self).__init__(postgres, server_name) self.backup_compression = backup_compression def check(self, check_strategy): """ Perform additional checks for the Postgres backup strategy """ def start_backup(self, backup_info): """ Manage the start of an pg_basebackup backup The method performs all the preliminary operations required for a backup executed using pg_basebackup to start, gathering information from postgres and filling the backup_info. :param barman.infofile.LocalBackupInfo backup_info: backup information """ self.current_action = "initialising postgres backup_method" super(PostgresBackupStrategy, self).start_backup(backup_info) current_xlog_info = self.postgres.current_xlog_info self._backup_info_from_start_location(backup_info, current_xlog_info) def stop_backup(self, backup_info): """ Manage the stop of an pg_basebackup backup The method retrieves the information necessary for the backup.info file reading the backup_label file. Due of the nature of the pg_basebackup, information that are gathered during the start of a backup performed using rsync, are retrieved here :param barman.infofile.LocalBackupInfo backup_info: backup information """ if self.backup_compression and self.backup_compression.config.format != "plain": backup_info.set_attribute( "compression", self.backup_compression.config.type ) self._read_backup_label(backup_info) self._backup_info_from_backup_label(backup_info) # Set data in backup_info from current_xlog_info self.current_action = "stopping postgres backup_method" output.info("Finalising the backup.") # Get the current xlog position current_xlog_info = self.postgres.current_xlog_info if current_xlog_info: self._backup_info_from_stop_location(backup_info, current_xlog_info) # Ask PostgreSQL to switch to another WAL file. This is needed # to archive the transaction log file containing the backup # end position, which is required to recover from the backup. try: self.postgres.switch_wal() except PostgresIsInRecovery: # Skip switching XLOG if a standby server pass def _read_compressed_backup_label(self, backup_info): """ Read the contents of a backup_label file from a compressed archive. :param barman.infofile.LocalBackupInfo backup_info: backup information """ basename = os.path.join(backup_info.get_data_directory(), "base") try: return self.backup_compression.get_file_content("backup_label", basename) except FileNotFoundException: raise BackupException( "Could not find backup_label in %s" % self.backup_compression.with_suffix(basename) ) def _read_backup_label(self, backup_info): """ Read the backup_label file. Transparently handles the fact that the backup_label file may be in a compressed tarball. :param barman.infofile.LocalBackupInfo backup_info: backup information """ self.current_action = "reading the backup label" if backup_info.compression is not None: backup_label = self._read_compressed_backup_label(backup_info) backup_info.set_attribute("backup_label", backup_label) else: super(PostgresBackupStrategy, self)._read_backup_label(backup_info) class ExclusiveBackupStrategy(BackupStrategy): """ Concrete class for exclusive backup strategy. This strategy is for ExternalBackupExecutor only and is responsible for coordinating Barman with PostgreSQL on standard physical backup operations (known as 'exclusive' backup), such as invoking pg_start_backup() and pg_stop_backup() on the master server. """ def __init__(self, postgres, server_name): """ Constructor :param barman.postgres.PostgreSQLConnection postgres: the PostgreSQL connection :param str server_name: The name of the server """ super(ExclusiveBackupStrategy, self).__init__( postgres, server_name, "exclusive" ) def start_backup(self, backup_info): """ Manage the start of an exclusive backup The method performs all the preliminary operations required for an exclusive physical backup to start, as well as preparing the information on the backup for Barman. :param barman.infofile.LocalBackupInfo backup_info: backup information """ super(ExclusiveBackupStrategy, self).start_backup(backup_info) label = "Barman backup %s %s" % (backup_info.server_name, backup_info.backup_id) # Issue an exclusive start backup command _logger.debug("Start of exclusive backup") start_info = self.postgres.start_exclusive_backup(label) self._backup_info_from_start_location(backup_info, start_info) def stop_backup(self, backup_info): """ Manage the stop of an exclusive backup The method informs the PostgreSQL server that the physical exclusive backup is finished, as well as preparing the information returned by PostgreSQL for Barman. :param barman.infofile.LocalBackupInfo backup_info: backup information """ self.current_action = "issuing stop backup command" _logger.debug("Stop of exclusive backup") stop_info = self.postgres.stop_exclusive_backup() self._backup_info_from_stop_location(backup_info, stop_info) def check(self, check_strategy): """ Perform additional checks for ExclusiveBackupStrategy :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ # Make sure PostgreSQL is not in recovery (i.e. is a master) check_strategy.init_check("not in recovery") if self.postgres: is_in_recovery = self.postgres.is_in_recovery if not is_in_recovery: check_strategy.result(self.server_name, True) else: check_strategy.result( self.server_name, False, hint="cannot perform exclusive backup on a standby", ) check_strategy.init_check("exclusive backup supported") try: if self.postgres and self.postgres.server_version < 150000: check_strategy.result(self.server_name, True) else: check_strategy.result( self.server_name, False, hint="exclusive backups not supported " "on PostgreSQL %s" % self.postgres.server_major_version, ) except PostgresConnectionError: check_strategy.result( self.server_name, False, hint="unable to determine postgres version", ) class ConcurrentBackupStrategy(BackupStrategy): """ Concrete class for concurrent backup strategy. This strategy is responsible for coordinating Barman with PostgreSQL on concurrent physical backup operations through concurrent backup PostgreSQL api. """ def __init__(self, postgres, server_name): """ Constructor :param barman.postgres.PostgreSQLConnection postgres: the PostgreSQL connection :param str server_name: The name of the server """ super(ConcurrentBackupStrategy, self).__init__( postgres, server_name, "concurrent" ) def check(self, check_strategy): """ Checks that Postgres is at least minimal version :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ check_strategy.init_check("postgres minimal version") try: # We execute this check only if the postgres connection is not None # to validate the server version matches at least minimal version if self.postgres and not self.postgres.is_minimal_postgres_version(): check_strategy.result( self.server_name, False, hint="unsupported PostgresSQL version %s. Expecting %s or above." % ( self.postgres.server_major_version, self.postgres.minimal_txt_version, ), ) except PostgresConnectionError: # Skip the check if the postgres connection doesn't work. # We assume that this error condition will be reported by # another check. pass def start_backup(self, backup_info): """ Start of the backup. The method performs all the preliminary operations required for a backup to start. :param barman.infofile.BackupInfo backup_info: backup information """ super(ConcurrentBackupStrategy, self).start_backup(backup_info) label = "Barman backup %s %s" % (backup_info.server_name, backup_info.backup_id) if not self.postgres.is_minimal_postgres_version(): _logger.error("Postgres version not supported") raise BackupException("Postgres version not supported") # On 9.6+ execute native concurrent start backup _logger.debug("Start of native concurrent backup") self._concurrent_start_backup(backup_info, label) def stop_backup(self, backup_info): """ Stop backup wrapper :param barman.infofile.BackupInfo backup_info: backup information """ self.current_action = "issuing stop backup command (native concurrent)" if not self.postgres.is_minimal_postgres_version(): _logger.error( "Postgres version not supported. Minimal version is %s" % self.postgres.minimal_txt_version ) raise BackupException("Postgres version not supported") _logger.debug("Stop of native concurrent backup") self._concurrent_stop_backup(backup_info) # Update the current action in preparation for writing the backup label. # NOTE: The actual writing of the backup label happens either in the # specialization of this function in LocalConcurrentBackupStrategy # or out-of-band in a CloudBackupUploader (when ConcurrentBackupStrategy # is used directly when writing to an object store). self.current_action = "writing backup label" # Ask PostgreSQL to switch to another WAL file. This is needed # to archive the transaction log file containing the backup # end position, which is required to recover from the backup. try: self.postgres.switch_wal() except PostgresIsInRecovery: # Skip switching XLOG if a standby server pass def _concurrent_start_backup(self, backup_info, label): """ Start a concurrent backup using the PostgreSQL 9.6 concurrent backup api :param barman.infofile.BackupInfo backup_info: backup information :param str label: the backup label """ start_info = self.postgres.start_concurrent_backup(label) self.postgres.allow_reconnect = False self._backup_info_from_start_location(backup_info, start_info) def _concurrent_stop_backup(self, backup_info): """ Stop a concurrent backup using the PostgreSQL 9.6 concurrent backup api :param barman.infofile.BackupInfo backup_info: backup information """ stop_info = self.postgres.stop_concurrent_backup() self.postgres.allow_reconnect = True backup_info.set_attribute("backup_label", stop_info["backup_label"]) self._backup_info_from_stop_location(backup_info, stop_info) class LocalConcurrentBackupStrategy(ConcurrentBackupStrategy): """ Concrete class for concurrent backup strategy writing data locally. This strategy is for ExternalBackupExecutor only and is responsible for coordinating Barman with PostgreSQL on concurrent physical backup operations through concurrent backup PostgreSQL api. """ # noinspection PyMethodMayBeStatic def _write_backup_label(self, backup_info): """ Write the backup_label file inside local data directory :param barman.infofile.LocalBackupInfo backup_info: backup information """ label_file = os.path.join(backup_info.get_data_directory(), "backup_label") output.debug("Writing backup label: %s" % label_file) with open(label_file, "w") as f: f.write(backup_info.backup_label) def stop_backup(self, backup_info): """ Stop backup wrapper :param barman.infofile.LocalBackupInfo backup_info: backup information """ super(LocalConcurrentBackupStrategy, self).stop_backup(backup_info) self._write_backup_label(backup_info) barman-3.13.3/barman/process.py0000644000175100001660000001350715002177425014523 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see import errno import logging import os import signal import time from glob import glob from barman import output from barman.exceptions import LockFileParsingError from barman.lockfile import ServerWalReceiveLock _logger = logging.getLogger(__name__) class ProcessInfo(object): """ Barman process representation """ def __init__(self, pid, server_name, task): """ This object contains all the information required to identify a barman process :param int pid: Process ID :param string server_name: Name of the server owning the process :param string task: Task name (receive-wal, archive-wal...) """ self.pid = pid self.server_name = server_name self.task = task class ProcessManager(object): """ Class for the management of barman processes owned by a server """ # Map containing the tasks we want to retrieve (and eventually manage) TASKS = {"receive-wal": ServerWalReceiveLock} def __init__(self, config): """ Build a ProcessManager for the provided server :param config: configuration of the server owning the process manager """ self.config = config self.process_list = [] # Cycle over the lock files in the lock directory for this server for path in glob( os.path.join( self.config.barman_lock_directory, ".%s-*.lock" % self.config.name ) ): for task, lock_class in self.TASKS.items(): # Check the lock_name against the lock class lock = lock_class.build_if_matches(path) if lock: try: # Use the lock to get the owner pid pid = lock.get_owner_pid() except LockFileParsingError: _logger.warning( "Skipping the %s process for server %s: " "Error reading the PID from lock file '%s'", task, self.config.name, path, ) break # If there is a pid save it in the process list if pid: self.process_list.append(ProcessInfo(pid, config.name, task)) # In any case, we found a match, so we must stop iterating # over the task types and handle the next path break def list(self, task_filter=None): """ Returns a list of processes owned by this server If no filter is provided, all the processes are returned. :param str task_filter: Type of process we want to retrieve :return list[ProcessInfo]: List of processes for the server """ server_tasks = [] for process in self.process_list: # Filter the processes if necessary if task_filter and process.task != task_filter: continue server_tasks.append(process) return server_tasks def kill(self, process_info, retries=10): """ Kill a process Returns True if killed successfully False otherwise :param ProcessInfo process_info: representation of the process we want to kill :param int retries: number of times the method will check if the process is still alive :rtype: bool """ # Try to kill the process try: _logger.debug("Sending SIGINT to PID %s", process_info.pid) os.kill(process_info.pid, signal.SIGINT) _logger.debug("os.kill call succeeded") except OSError as e: _logger.debug("os.kill call failed: %s", e) # The process doesn't exists. It has probably just terminated. if e.errno == errno.ESRCH: return True # Something unexpected has happened output.error("%s", e) return False # Check if the process have been killed. the fastest (and maybe safest) # way is to send a kill with 0 as signal. # If the method returns an OSError exceptions, the process have been # killed successfully, otherwise is still alive. for counter in range(retries): try: _logger.debug( "Checking with SIG_DFL if PID %s is still alive", process_info.pid ) os.kill(process_info.pid, signal.SIG_DFL) _logger.debug("os.kill call succeeded") except OSError as e: _logger.debug("os.kill call failed: %s", e) # If the process doesn't exists, we are done. if e.errno == errno.ESRCH: return True # Something unexpected has happened output.error("%s", e) return False time.sleep(1) _logger.debug( "The PID %s has not been terminated after %s retries", process_info.pid, retries, ) return False barman-3.13.3/barman/diagnose.py0000644000175100001660000001175215002177425014636 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module represents the barman diagnostic tool. """ import datetime import json import logging from dateutil import tz import barman from barman import fs, output from barman.backup import BackupInfo from barman.exceptions import CommandFailedException, FsOperationFailed from barman.utils import BarmanEncoderV2 _logger = logging.getLogger(__name__) def exec_diagnose(servers, models, errors_list, show_config_source): """ Diagnostic command: gathers information from backup server and from all the configured servers. Gathered information should be used for support and problems detection :param dict(str,barman.server.Server) servers: list of configured servers :param models: list of configured models. :param list errors_list: list of global errors :param show_config_source: if we should include the configuration file that provides the effective value for each configuration option. """ # global section. info about barman server diagnosis = {"global": {}, "servers": {}, "models": {}} # barman global config diagnosis["global"]["config"] = dict( barman.__config__.global_config_to_json(show_config_source) ) diagnosis["global"]["config"]["errors_list"] = errors_list try: command = fs.UnixLocalCommand() # basic system info diagnosis["global"]["system_info"] = command.get_system_info() except CommandFailedException as e: diagnosis["global"]["system_info"] = {"error": repr(e)} diagnosis["global"]["system_info"]["barman_ver"] = barman.__version__ diagnosis["global"]["system_info"]["timestamp"] = datetime.datetime.now( tz=tz.tzlocal() ) # per server section for name in sorted(servers): server = servers[name] if server is None: output.error("Unknown server '%s'" % name) continue # server configuration diagnosis["servers"][name] = {} diagnosis["servers"][name]["config"] = server.config.to_json(show_config_source) # server model active_model = ( server.config.active_model.name if server.config.active_model is not None else None ) diagnosis["servers"][name]["active_model"] = active_model # server system info if server.config.ssh_command: try: command = fs.UnixRemoteCommand( ssh_command=server.config.ssh_command, path=server.path ) diagnosis["servers"][name]["system_info"] = command.get_system_info() except FsOperationFailed: pass # barman status information for the server diagnosis["servers"][name]["status"] = server.get_remote_status() # backup list backups = server.get_available_backups(BackupInfo.STATUS_ALL) # update date format for each backup begin_time and end_time and ensure local timezone. # This code is a duplicate from BackupInfo.to_json() # This should be temporary to keep original behavior for other usage. for key in backups.keys(): data = backups[key].to_dict() if data.get("tablespaces") is not None: data["tablespaces"] = [list(item) for item in data["tablespaces"]] if data.get("begin_time") is not None: data["begin_time"] = data["begin_time"].astimezone(tz=tz.tzlocal()) if data.get("end_time") is not None: data["end_time"] = data["end_time"].astimezone(tz=tz.tzlocal()) backups[key] = data diagnosis["servers"][name]["backups"] = backups # wal status diagnosis["servers"][name]["wals"] = { "last_archived_wal_per_timeline": server.backup_manager.get_latest_archived_wals_info(), } # Release any PostgreSQL resource server.close() # per model section for name in sorted(models): model = models[name] if model is None: output.error("Unknown model '%s'" % name) continue # model configuration diagnosis["models"][name] = {} diagnosis["models"][name]["config"] = model.to_json(show_config_source) output.info( json.dumps(diagnosis, cls=BarmanEncoderV2, indent=4, sort_keys=True), log=False ) barman-3.13.3/barman/lockfile.py0000644000175100001660000002731415002177425014636 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module is the lock manager for Barman """ import errno import fcntl import os import re from barman.exceptions import ( LockFileBusy, LockFileParsingError, LockFilePermissionDenied, ) class LockFile(object): """ Ensures that there is only one process which is running against a specified LockFile. It supports the Context Manager interface, allowing the use in with statements. with LockFile('file.lock') as locked: if not locked: print "failed" else: You can also use exceptions on failures try: with LockFile('file.lock', True): except LockFileBusy, e, file: print "failed to lock %s" % file """ LOCK_PATTERN = None r""" If defined in a subclass, it must be a compiled regular expression which matches the lock filename. It must provide named groups for the constructor parameters which produce the same lock name. I.e.: >>> ServerWalReceiveLock('/tmp', 'server-name').filename '/tmp/.server-name-receive-wal.lock' >>> ServerWalReceiveLock.LOCK_PATTERN = re.compile( r'\.(?P.+)-receive-wal\.lock') >>> m = ServerWalReceiveLock.LOCK_PATTERN.match( '.server-name-receive-wal.lock') >>> ServerWalReceiveLock('/tmp', **(m.groupdict())).filename '/tmp/.server-name-receive-wal.lock' """ @classmethod def build_if_matches(cls, path): """ Factory method that creates a lock instance if the path matches the lock filename created by the actual class :param path: the full path of a LockFile :return: """ # If LOCK_PATTERN is not defined always return None if not cls.LOCK_PATTERN: return None # Matches the provided path against LOCK_PATTERN lock_directory = os.path.abspath(os.path.dirname(path)) lock_name = os.path.basename(path) match = cls.LOCK_PATTERN.match(lock_name) if match: # Build the lock object for the provided path return cls(lock_directory, **(match.groupdict())) return None def __init__(self, filename, raise_if_fail=True, wait=False): self.filename = os.path.abspath(filename) self.fd = None self.raise_if_fail = raise_if_fail self.wait = wait def acquire(self, raise_if_fail=None, wait=None, update_pid=True): """ Creates and holds on to the lock file. When raise_if_fail, a LockFileBusy is raised if the lock is held by someone else and a LockFilePermissionDenied is raised when the user executing barman have insufficient rights for the creation of a LockFile. Returns True if lock has been successfully acquired, False otherwise. :param bool raise_if_fail: If True raise an exception on failure :param bool wait: If True issue a blocking request :param bool update_pid: Whether to write our pid in the lockfile :returns bool: whether the lock has been acquired """ if self.fd: return True fd = None # method arguments take precedence on class parameters raise_if_fail = ( raise_if_fail if raise_if_fail is not None else self.raise_if_fail ) wait = wait if wait is not None else self.wait try: # 384 is 0600 in octal, 'rw-------' fd = os.open(self.filename, os.O_CREAT | os.O_RDWR, 384) flags = fcntl.LOCK_EX if not wait: flags |= fcntl.LOCK_NB fcntl.flock(fd, flags) if update_pid: # Once locked, replace the content of the file os.lseek(fd, 0, os.SEEK_SET) os.write(fd, ("%s\n" % os.getpid()).encode("ascii")) # Truncate the file at the current position os.ftruncate(fd, os.lseek(fd, 0, os.SEEK_CUR)) self.fd = fd return True except (OSError, IOError) as e: if fd: os.close(fd) # let's not leak file descriptors if raise_if_fail: if e.errno in (errno.EAGAIN, errno.EWOULDBLOCK): raise LockFileBusy(self.filename) elif e.errno == errno.EACCES: raise LockFilePermissionDenied(self.filename) else: raise else: return False def release(self): """ Releases the lock. If the lock is not held by the current process it does nothing. """ if not self.fd: return try: fcntl.flock(self.fd, fcntl.LOCK_UN) os.close(self.fd) except (OSError, IOError): pass self.fd = None def __del__(self): """ Avoid stale lock files. """ self.release() # Contextmanager interface def __enter__(self): return self.acquire() def __exit__(self, exception_type, value, traceback): self.release() def get_owner_pid(self): """ Test whether a lock is already held by a process. Returns the PID of the owner process or None if the lock is available. :rtype: int|None :raises LockFileParsingError: when the lock content is garbled :raises LockFilePermissionDenied: when the lockfile is not accessible """ try: self.acquire(raise_if_fail=True, wait=False, update_pid=False) except LockFileBusy: try: # Read the lock content and parse the PID # NOTE: We cannot read it in the self.acquire method to avoid # reading the previous locker PID with open(self.filename, "r") as file_object: return int(file_object.readline().strip()) except ValueError as e: # This should not happen raise LockFileParsingError(e) # release the lock and return None self.release() return None class GlobalCronLock(LockFile): """ This lock protects cron from multiple executions. Creates a global '.cron.lock' lock file under the given lock_directory. """ def __init__(self, lock_directory): super(GlobalCronLock, self).__init__( os.path.join(lock_directory, ".cron.lock"), raise_if_fail=True ) class ServerBackupLock(LockFile): """ This lock protects a server from multiple executions of backup command Creates a '.-backup.lock' lock file under the given lock_directory for the named SERVER. """ def __init__(self, lock_directory, server_name): super(ServerBackupLock, self).__init__( os.path.join(lock_directory, ".%s-backup.lock" % server_name), raise_if_fail=True, ) class ServerCronLock(LockFile): """ This lock protects a server from multiple executions of cron command Creates a '.-cron.lock' lock file under the given lock_directory for the named SERVER. """ def __init__(self, lock_directory, server_name): super(ServerCronLock, self).__init__( os.path.join(lock_directory, ".%s-cron.lock" % server_name), raise_if_fail=True, wait=False, ) class ServerXLOGDBLock(LockFile): """ This lock protects a server's xlogdb access Creates a '.-xlogdb.lock' lock file under the given lock_directory for the named SERVER. """ def __init__(self, lock_directory, server_name): super(ServerXLOGDBLock, self).__init__( os.path.join(lock_directory, ".%s-xlogdb.lock" % server_name), raise_if_fail=True, wait=True, ) class ServerWalArchiveLock(LockFile): """ This lock protects a server from multiple executions of wal-archive command Creates a '.-archive-wal.lock' lock file under the given lock_directory for the named SERVER. """ def __init__(self, lock_directory, server_name): super(ServerWalArchiveLock, self).__init__( os.path.join(lock_directory, ".%s-archive-wal.lock" % server_name), raise_if_fail=True, wait=False, ) class ServerWalReceiveLock(LockFile): """ This lock protects a server from multiple executions of receive-wal command Creates a '.-receive-wal.lock' lock file under the given lock_directory for the named SERVER. """ # TODO: Implement on the other LockFile subclasses LOCK_PATTERN = re.compile(r"\.(?P.+)-receive-wal\.lock") def __init__(self, lock_directory, server_name): super(ServerWalReceiveLock, self).__init__( os.path.join(lock_directory, ".%s-receive-wal.lock" % server_name), raise_if_fail=True, wait=False, ) class ServerBackupIdLock(LockFile): """ This lock protects from changing a backup that is in use. Creates a '.-.lock' lock file under the given lock_directory for a BACKUP of a SERVER. """ def __init__(self, lock_directory, server_name, backup_id): super(ServerBackupIdLock, self).__init__( os.path.join(lock_directory, ".%s-%s.lock" % (server_name, backup_id)), raise_if_fail=True, wait=False, ) class ServerBackupSyncLock(LockFile): """ This lock protects from multiple executions of the sync command on the same backup. Creates a '.--sync-backup.lock' lock file under the given lock_directory for a BACKUP of a SERVER. """ def __init__(self, lock_directory, server_name, backup_id): super(ServerBackupSyncLock, self).__init__( os.path.join( lock_directory, ".%s-%s-sync-backup.lock" % (server_name, backup_id) ), raise_if_fail=True, wait=False, ) class ServerWalSyncLock(LockFile): """ This lock protects from multiple executions of the sync-wal command Creates a '.-sync-wal.lock' lock file under the given lock_directory for the named SERVER. """ def __init__(self, lock_directory, server_name): super(ServerWalSyncLock, self).__init__( os.path.join(lock_directory, ".%s-sync-wal.lock" % server_name), raise_if_fail=True, wait=True, ) class ConfigUpdateLock(LockFile): """ This lock protects barman from multiple executions of config-update command Creates a ``.config-update.lock`` lock file under the given ``lock_directory``. """ def __init__(self, lock_directory): """ Initialize a new :class:`ConfigUpdateLock` object. :param lock_directory str: where to create the ``.config-update.lock`` file. """ super(ConfigUpdateLock, self).__init__( os.path.join(lock_directory, ".config-update.lock"), raise_if_fail=True, wait=False, ) barman-3.13.3/barman/server.py0000644000175100001660000057272515002177425014367 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module represents a Server. Barman is able to manage multiple servers. """ import datetime import errno import json import logging import os import re import shutil import sys import tarfile import tempfile import time from collections import namedtuple from contextlib import closing, contextmanager from glob import glob from tempfile import NamedTemporaryFile import dateutil.tz import barman from barman import output, xlog from barman.backup import BackupManager from barman.command_wrappers import BarmanSubProcess, Command, Rsync from barman.compression import CustomCompressor from barman.copy_controller import RsyncCopyController from barman.exceptions import ( ArchiverFailure, BackupException, BadXlogSegmentName, CommandFailedException, ConninfoException, InvalidRetentionPolicy, LockFileBusy, LockFileException, LockFilePermissionDenied, PostgresCheckpointPrivilegesRequired, PostgresDuplicateReplicationSlot, PostgresException, PostgresInvalidReplicationSlot, PostgresIsInRecovery, PostgresObsoleteFeature, PostgresReplicationSlotInUse, PostgresReplicationSlotsFull, PostgresSuperuserRequired, PostgresUnsupportedFeature, SyncError, SyncNothingToDo, SyncToBeDeleted, TimeoutError, UnknownBackupIdException, ) from barman.infofile import BackupInfo, LocalBackupInfo, WalFileInfo from barman.lockfile import ( ServerBackupIdLock, ServerBackupLock, ServerBackupSyncLock, ServerCronLock, ServerWalArchiveLock, ServerWalReceiveLock, ServerWalSyncLock, ServerXLOGDBLock, ) from barman.postgres import ( PostgreSQL, PostgreSQLConnection, StandbyPostgreSQLConnection, StreamingConnection, ) from barman.process import ProcessManager from barman.remote_status import RemoteStatusMixin from barman.retention_policies import RetentionPolicy, RetentionPolicyFactory from barman.utils import ( BarmanEncoder, file_hash, force_str, fsync_dir, fsync_file, human_readable_timedelta, is_power_of_two, mkpath, parse_target_tli, pretty_size, timeout, ) from barman.wal_archiver import FileWalArchiver, StreamingWalArchiver, WalArchiver PARTIAL_EXTENSION = ".partial" PRIMARY_INFO_FILE = "primary.info" SYNC_WALS_INFO_FILE = "sync-wals.info" _logger = logging.getLogger(__name__) # NamedTuple for a better readability of SyncWalInfo SyncWalInfo = namedtuple("SyncWalInfo", "last_wal last_position") class CheckStrategy(object): """ This strategy for the 'check' collects the results of every check and does not print any message. This basic class is also responsible for immediately logging any performed check with an error in case of check failure and a debug message in case of success. """ # create a namedtuple object called CheckResult to manage check results CheckResult = namedtuple("CheckResult", "server_name check status") # Default list used as a filter to identify non-critical checks NON_CRITICAL_CHECKS = [ "minimum redundancy requirements", "backup maximum age", "backup minimum size", "failed backups", "archiver errors", "empty incoming directory", "empty streaming directory", "incoming WALs directory", "streaming WALs directory", "wal maximum age", "PostgreSQL server is standby", ] def __init__(self, ignore_checks=NON_CRITICAL_CHECKS): """ Silent Strategy constructor :param list ignore_checks: list of checks that can be ignored """ self.ignore_list = ignore_checks self.check_result = [] self.has_error = False self.running_check = None def init_check(self, check_name): """ Mark in the debug log when barman starts the execution of a check :param str check_name: the name of the check that is starting """ self.running_check = check_name _logger.debug("Starting check: '%s'" % check_name) def _check_name(self, check): if not check: check = self.running_check assert check return check def result(self, server_name, status, hint=None, check=None, perfdata=None): """ Store the result of a check (with no output). Log any check result (error or debug level). :param str server_name: the server is being checked :param bool status: True if succeeded :param str,None hint: hint to print if not None: :param str,None check: the check name :param str,None perfdata: additional performance data to print if not None """ check = self._check_name(check) if not status: # If the name of the check is not in the filter list, # treat it as a blocking error, then notify the error # and change the status of the strategy if check not in self.ignore_list: self.has_error = True _logger.error( "Check '%s' failed for server '%s'" % (check, server_name) ) else: # otherwise simply log the error (as info) _logger.info( "Ignoring failed check '%s' for server '%s'" % (check, server_name) ) else: _logger.debug("Check '%s' succeeded for server '%s'" % (check, server_name)) # Store the result and does not output anything result = self.CheckResult(server_name, check, status) self.check_result.append(result) self.running_check = None class CheckOutputStrategy(CheckStrategy): """ This strategy for the 'check' command immediately sends the result of a check to the designated output channel. This class derives from the basic CheckStrategy, reuses the same logic and adds output messages. """ def __init__(self): """ Output Strategy constructor """ super(CheckOutputStrategy, self).__init__(ignore_checks=()) def result(self, server_name, status, hint=None, check=None, perfdata=None): """ Store the result of a check. Log any check result (error or debug level). Output the result to the user :param str server_name: the server being checked :param str check: the check name :param bool status: True if succeeded :param str,None hint: hint to print if not None: :param str,None perfdata: additional performance data to print if not None """ check = self._check_name(check) super(CheckOutputStrategy, self).result( server_name, status, hint, check, perfdata ) # Send result to output output.result("check", server_name, check, status, hint, perfdata) class Server(RemoteStatusMixin): """ This class represents the PostgreSQL server to backup. """ XLOGDB_NAME = "{server}-xlog.db" # the strategy for the management of the results of the various checks __default_check_strategy = CheckOutputStrategy() def __init__(self, config): """ Server constructor. :param barman.config.ServerConfig config: the server configuration """ super(Server, self).__init__() self.config = config self.path = self._build_path(self.config.path_prefix) self.process_manager = ProcessManager(self.config) # If 'primary_ssh_command' is specified, the source of the backup # for this server is a Barman installation (not a Postgres server) self.passive_node = config.primary_ssh_command is not None self.enforce_retention_policies = False self.postgres = None self.streaming = None self.archivers = [] # Postgres configuration is available only if node is not passive if not self.passive_node: self._init_postgres(config) # Initialize the backup manager self.backup_manager = BackupManager(self) if not self.passive_node: self._init_archivers() # Set global and tablespace bandwidth limits self._init_bandwidth_limits() # Initialize minimum redundancy self._init_minimum_redundancy() # Initialise retention policies self._init_retention_policies() def _init_postgres(self, config): # Initialize the main PostgreSQL connection try: # Check that 'conninfo' option is properly set if config.conninfo is None: raise ConninfoException( "Missing 'conninfo' parameter for server '%s'" % config.name ) # If primary_conninfo is set then we're connecting to a standby if config.primary_conninfo is not None: self.postgres = StandbyPostgreSQLConnection( config.conninfo, config.primary_conninfo, config.immediate_checkpoint, config.slot_name, config.primary_checkpoint_timeout, ) # If primary_conninfo is set but conninfo does not point to a standby # it could be that a failover happend and the standby has been promoted. # In this case, don't set a standby connection and just warn the user. # A standard connection will be set further so that Barman keeps working. if self.postgres.is_in_recovery is False: self.postgres.close() self.postgres = None output.warning( "'primary_conninfo' is set but 'conninfo' does not point to a " "standby server. Ignoring 'primary_conninfo'." ) if self.postgres is None: self.postgres = PostgreSQLConnection( config.conninfo, config.immediate_checkpoint, config.slot_name ) # If the PostgreSQLConnection creation fails, disable the Server except ConninfoException as e: self.config.update_msg_list_and_disable_server( "PostgreSQL connection: " + force_str(e).strip() ) # Initialize the streaming PostgreSQL connection only when # backup_method is postgres or the streaming_archiver is in use if config.backup_method == "postgres" or config.streaming_archiver: try: if config.streaming_conninfo is None: raise ConninfoException( "Missing 'streaming_conninfo' parameter for " "server '%s'" % config.name ) self.streaming = StreamingConnection(config.streaming_conninfo) # If the StreamingConnection creation fails, disable the server except ConninfoException as e: self.config.update_msg_list_and_disable_server( "Streaming connection: " + force_str(e).strip() ) def _init_archivers(self): # Initialize the StreamingWalArchiver # WARNING: Order of items in self.archivers list is important! # The files will be archived in that order. if self.config.streaming_archiver: try: self.archivers.append(StreamingWalArchiver(self.backup_manager)) # If the StreamingWalArchiver creation fails, # disable the server except AttributeError as e: _logger.debug(e) self.config.update_msg_list_and_disable_server( "Unable to initialise the streaming archiver" ) # IMPORTANT: The following lines of code have been # temporarily commented in order to make the code # back-compatible after the introduction of 'archiver=off' # as default value in Barman 2.0. # When the back compatibility feature for archiver will be # removed, the following lines need to be decommented. # ARCHIVER_OFF_BACKCOMPATIBILITY - START OF CODE # # At least one of the available archive modes should be enabled # if len(self.archivers) < 1: # self.config.update_msg_list_and_disable_server( # "No archiver enabled for server '%s'. " # "Please turn on 'archiver', 'streaming_archiver' or both" # % config.name # ) # ARCHIVER_OFF_BACKCOMPATIBILITY - END OF CODE # Sanity check: if file based archiver is disabled, and only # WAL streaming is enabled, a replication slot name must be # configured. if ( not self.config.archiver and self.config.streaming_archiver and self.config.slot_name is None ): self.config.update_msg_list_and_disable_server( "Streaming-only archiver requires 'streaming_conninfo' " "and 'slot_name' options to be properly configured" ) # ARCHIVER_OFF_BACKCOMPATIBILITY - START OF CODE # IMPORTANT: This is a back-compatibility feature that has # been added in Barman 2.0. It highlights a deprecated # behaviour, and helps users during this transition phase. # It forces 'archiver=on' when both archiver and streaming_archiver # are set to 'off' (default values) and displays a warning, # requesting users to explicitly set the value in the # configuration. # When this back-compatibility feature will be removed from Barman # (in a couple of major releases), developers will need to remove # this block completely and reinstate the block of code you find # a few lines below (search for ARCHIVER_OFF_BACKCOMPATIBILITY # throughout the code). if self.config.archiver is False and self.config.streaming_archiver is False: output.warning( "No archiver enabled for server '%s'. " "Please turn on 'archiver', " "'streaming_archiver' or both", self.config.name, ) output.warning("Forcing 'archiver = on'") self.config.archiver = True # ARCHIVER_OFF_BACKCOMPATIBILITY - END OF CODE # Initialize the FileWalArchiver # WARNING: Order of items in self.archivers list is important! # The files will be archived in that order. if self.config.archiver: try: self.archivers.append(FileWalArchiver(self.backup_manager)) except AttributeError as e: _logger.debug(e) self.config.update_msg_list_and_disable_server( "Unable to initialise the file based archiver" ) def _init_bandwidth_limits(self): # Global bandwidth limits if self.config.bandwidth_limit: try: self.config.bandwidth_limit = int(self.config.bandwidth_limit) except ValueError: _logger.warning( 'Invalid bandwidth_limit "%s" for server "%s" ' '(fallback to "0")' % (self.config.bandwidth_limit, self.config.name) ) self.config.bandwidth_limit = None # Tablespace bandwidth limits if self.config.tablespace_bandwidth_limit: rules = {} for rule in self.config.tablespace_bandwidth_limit.split(): try: key, value = rule.split(":", 1) value = int(value) if value != self.config.bandwidth_limit: rules[key] = value except ValueError: _logger.warning( "Invalid tablespace_bandwidth_limit rule '%s'" % rule ) if len(rules) > 0: self.config.tablespace_bandwidth_limit = rules else: self.config.tablespace_bandwidth_limit = None def _init_minimum_redundancy(self): # Set minimum redundancy (default 0) try: self.config.minimum_redundancy = int(self.config.minimum_redundancy) if self.config.minimum_redundancy < 0: _logger.warning( 'Negative value of minimum_redundancy "%s" ' 'for server "%s" (fallback to "0")' % (self.config.minimum_redundancy, self.config.name) ) self.config.minimum_redundancy = 0 except ValueError: _logger.warning( 'Invalid minimum_redundancy "%s" for server "%s" ' '(fallback to "0")' % (self.config.minimum_redundancy, self.config.name) ) self.config.minimum_redundancy = 0 def _init_retention_policies(self): # Set retention policy mode if self.config.retention_policy_mode != "auto": _logger.warning( 'Unsupported retention_policy_mode "%s" for server "%s" ' '(fallback to "auto")' % (self.config.retention_policy_mode, self.config.name) ) self.config.retention_policy_mode = "auto" # If retention_policy is present, enforce them if self.config.retention_policy and not isinstance( self.config.retention_policy, RetentionPolicy ): # Check wal_retention_policy if self.config.wal_retention_policy != "main": _logger.warning( 'Unsupported wal_retention_policy value "%s" ' 'for server "%s" (fallback to "main")' % (self.config.wal_retention_policy, self.config.name) ) self.config.wal_retention_policy = "main" # Create retention policy objects try: rp = RetentionPolicyFactory.create( "retention_policy", self.config.retention_policy, server=self ) # Reassign the configuration value (we keep it in one place) self.config.retention_policy = rp _logger.debug( "Retention policy for server %s: %s" % (self.config.name, self.config.retention_policy) ) try: rp = RetentionPolicyFactory.create( "wal_retention_policy", self.config.wal_retention_policy, server=self, ) # Reassign the configuration value # (we keep it in one place) self.config.wal_retention_policy = rp _logger.debug( "WAL retention policy for server %s: %s" % (self.config.name, self.config.wal_retention_policy) ) except InvalidRetentionPolicy: _logger.exception( 'Invalid wal_retention_policy setting "%s" ' 'for server "%s" (fallback to "main")' % (self.config.wal_retention_policy, self.config.name) ) rp = RetentionPolicyFactory.create( "wal_retention_policy", "main", server=self ) self.config.wal_retention_policy = rp self.enforce_retention_policies = True except InvalidRetentionPolicy: _logger.exception( 'Invalid retention_policy setting "%s" for server "%s"' % (self.config.retention_policy, self.config.name) ) def get_identity_file_path(self): """ Get the path of the file that should contain the identity of the cluster :rtype: str """ return os.path.join(self.config.backup_directory, "identity.json") def write_identity_file(self): """ Store the identity of the server if it doesn't already exist. """ file_path = self.get_identity_file_path() # Do not write the identity if file already exists if os.path.exists(file_path): return systemid = self.systemid if systemid: try: with open(file_path, "w") as fp: json.dump( { "systemid": systemid, "version": self.postgres.server_major_version, }, fp, indent=4, sort_keys=True, ) fp.write("\n") except IOError: _logger.exception( 'Cannot write system Id file for server "%s"' % (self.config.name) ) def read_identity_file(self): """ Read the server identity :rtype: dict[str,str] """ file_path = self.get_identity_file_path() try: with open(file_path, "r") as fp: return json.load(fp) except IOError: _logger.exception( 'Cannot read system Id file for server "%s"' % (self.config.name) ) return {} def close(self): """ Close all the open connections to PostgreSQL """ if self.postgres: self.postgres.close() if self.streaming: self.streaming.close() def check(self, check_strategy=__default_check_strategy): """ Implements the 'server check' command and makes sure SSH and PostgreSQL connections work properly. It checks also that backup directories exist (and if not, it creates them). The check command will time out after a time interval defined by the check_timeout configuration value (default 30 seconds) :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ try: with timeout(self.config.check_timeout): # Check WAL archive self.check_archive(check_strategy) # Postgres configuration is not available on passive nodes if not self.passive_node: self.check_postgres(check_strategy) self.check_wal_streaming(check_strategy) # Check barman directories from barman configuration self.check_directories(check_strategy) # Check retention policies self.check_retention_policy_settings(check_strategy) # Check for backup validity self.check_backup_validity(check_strategy) # Check WAL archiving is happening self.check_wal_validity(check_strategy) # Executes the backup manager set of checks self.backup_manager.check(check_strategy) # Check if the msg_list of the server # contains messages and output eventual failures self.check_configuration(check_strategy) # Check the system Id coherence between # streaming and normal connections self.check_identity(check_strategy) # Executes check() for every archiver, passing # remote status information for efficiency for archiver in self.archivers: archiver.check(check_strategy) # Check archiver errors self.check_archiver_errors(check_strategy) except TimeoutError: # The check timed out. # Add a failed entry to the check strategy for this. _logger.info( "Check command timed out executing '%s' check" % check_strategy.running_check ) check_strategy.result( self.config.name, False, hint="barman check command timed out", check="check timeout", ) def check_archive(self, check_strategy): """ Checks WAL archive :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ check_strategy.init_check("WAL archive") # Make sure that WAL archiving has been setup # XLOG_DB needs to exist and its size must be > 0 # NOTE: we do not need to acquire a lock in this phase xlogdb_empty = True if os.path.exists(self.xlogdb_file_path): with open(self.xlogdb_file_path, "rb") as fxlogdb: if os.fstat(fxlogdb.fileno()).st_size > 0: xlogdb_empty = False # NOTE: This check needs to be only visible if it fails if xlogdb_empty: # Skip the error if we have a terminated backup # with status WAITING_FOR_WALS. # TODO: Improve this check backup_id = self.get_last_backup_id([BackupInfo.WAITING_FOR_WALS]) if not backup_id: check_strategy.result( self.config.name, False, hint="please make sure WAL shipping is setup", ) # Check the number of wals in the incoming directory self._check_wal_queue(check_strategy, "incoming", "archiver") # Check the number of wals in the streaming directory self._check_wal_queue(check_strategy, "streaming", "streaming_archiver") def _check_wal_queue(self, check_strategy, dir_name, archiver_name): """ Check if one of the wal queue directories beyond the max file threshold """ # Read the wal queue location from the configuration config_name = "%s_wals_directory" % dir_name assert hasattr(self.config, config_name) incoming_dir = getattr(self.config, config_name) # Check if the archiver is enabled assert hasattr(self.config, archiver_name) enabled = getattr(self.config, archiver_name) # Inspect the wal queue directory file_count = 0 for file_item in glob(os.path.join(incoming_dir, "*")): # Ignore temporary files if file_item.endswith(".tmp"): continue file_count += 1 max_incoming_wal = self.config.max_incoming_wals_queue # Subtract one from the count because of .partial file inside the # streaming directory if dir_name == "streaming": file_count -= 1 # If this archiver is disabled, check the number of files in the # corresponding directory. # If the directory is NOT empty, fail the check and warn the user. # NOTE: This check is visible only when it fails check_strategy.init_check("empty %s directory" % dir_name) if not enabled: if file_count > 0: check_strategy.result( self.config.name, False, hint="'%s' must be empty when %s=off" % (incoming_dir, archiver_name), ) # No more checks are required if the archiver # is not enabled return # At this point if max_wals_count is none, # means that no limit is set so we just need to return if max_incoming_wal is None: return check_strategy.init_check("%s WALs directory" % dir_name) if file_count > max_incoming_wal: msg = "there are too many WALs in queue: %s, max %s" % ( file_count, max_incoming_wal, ) check_strategy.result(self.config.name, False, hint=msg) def check_postgres(self, check_strategy): """ Checks PostgreSQL connection :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ check_strategy.init_check("PostgreSQL") # Take the status of the remote server remote_status = self.get_remote_status() if not remote_status.get("server_txt_version"): check_strategy.result(self.config.name, False) return # Now we know server version is accessible we can check if it is valid if remote_status.get("version_supported") is False: minimal_txt_version = PostgreSQL.int_version_to_string_version( PostgreSQL.MINIMAL_VERSION ) check_strategy.result( self.config.name, False, hint="unsupported version: PostgreSQL server " "is too old (%s < %s)" % (remote_status["server_txt_version"], minimal_txt_version), ) return else: check_strategy.result(self.config.name, True) # Check for superuser privileges or # privileges needed to perform backups if remote_status.get("has_backup_privileges") is not None: check_strategy.init_check( "superuser or standard user with backup privileges" ) if remote_status.get("has_backup_privileges"): check_strategy.result(self.config.name, True) else: check_strategy.result( self.config.name, False, hint="privileges for PostgreSQL backup functions are " "required (see documentation)", check="no access to backup functions", ) self._check_streaming_supported(check_strategy, remote_status) self._check_wal_level(check_strategy, remote_status) if self.config.primary_conninfo is not None: self._check_standby(check_strategy) def _check_streaming_supported(self, check_strategy, remote_status, suffix=None): """ Check whether the remote status indicates streaming is possible. :param CheckStrategy check_strategy: The strategy for the management of the result of this check :param dict[str, None|str] remote_status: Remote status information used by this check :param str|None suffix: A suffix to be appended to the check name """ if "streaming_supported" in remote_status: check_name = "PostgreSQL streaming" + ( "" if suffix is None else f" ({suffix})" ) check_strategy.init_check(check_name) hint = None # If a streaming connection is available, # add its status to the output of the check if remote_status["streaming_supported"] is None: hint = remote_status["connection_error"] check_strategy.result( self.config.name, remote_status.get("streaming"), hint=hint ) def _check_wal_level(self, check_strategy, remote_status, suffix=None): """ Check whether the remote status indicates ``wal_level`` is correct. :param CheckStrategy check_strategy: The strategy for the management of the result of this check :param dict[str, None|str] remote_status: Remote status information used by this check :param str|None suffix: A suffix to be appended to the check name """ # Check wal_level parameter: must be different from 'minimal' # the parameter has been introduced in postgres >= 9.0 if "wal_level" in remote_status: check_name = "wal_level" + ("" if suffix is None else f" ({suffix})") check_strategy.init_check(check_name) if remote_status["wal_level"] != "minimal": check_strategy.result(self.config.name, True) else: check_strategy.result( self.config.name, False, hint="please set it to a higher level than 'minimal'", ) def _check_has_monitoring_privileges( self, check_strategy, remote_status, suffix=None ): """ Check whether the remote status indicates monitoring information can be read. :param CheckStrategy check_strategy: The strategy for the management of the result of this check :param dict[str, None|str] remote_status: Remote status information used by this check :param str|None suffix: A suffix to be appended to the check name """ check_name = "has monitoring privileges" + ( "" if suffix is None else f" ({suffix})" ) check_strategy.init_check(check_name) if remote_status.get("has_monitoring_privileges"): check_strategy.result(self.config.name, True) else: check_strategy.result( self.config.name, False, hint="privileges for PostgreSQL monitoring functions are " "required (see documentation)", check="no access to monitoring functions", ) def check_wal_streaming(self, check_strategy): """ Perform checks related to the streaming of WALs only (not backups). If no WAL-specific connection information is defined then checks already performed on the default connection information will have verified their suitability for WAL streaming so this check will only call :meth:`_check_replication_slot` for the existing streaming connection as this is the only additional check required. If WAL-specific connection information *is* defined then we must verify that streaming is possible using that connection information *as well as* check the replication slot. This check will therefore: 1. Create these connections. 2. Fetch the remote status of these connections. 3. Pass the remote status information to :meth:`_check_wal_streaming_preflight` which will verify that the status information returned by these connections indicates they are suitable for WAL streaming. 4. Pass the remote status information to :meth:`_check_replication_slot` so that the status of the replication slot can be verified. :param CheckStrategy check_strategy: The strategy for the management of the result of this check """ # If we have wal-specific conninfo then we must use those to get # the remote status information for the check streaming_conninfo, conninfo = self.config.get_wal_conninfo() if conninfo != self.config.conninfo: with closing(StreamingConnection(streaming_conninfo)) as streaming, closing( PostgreSQLConnection(conninfo, slot_name=self.config.slot_name) ) as postgres: remote_status = postgres.get_remote_status() remote_status.update(streaming.get_remote_status()) self._check_wal_streaming_preflight(check_strategy, remote_status) self._check_replication_slot( check_strategy, remote_status, "WAL streaming" ) else: # Use the status for the existing postgres connections remote_status = self.get_remote_status() self._check_replication_slot(check_strategy, remote_status) def _check_wal_streaming_preflight(self, check_strategy, remote_status): """ Verify the supplied remote_status indicates WAL streaming is possible. Uses the remote status information to run the :meth:`_check_streaming_supported`, :meth:`_check_wal_level` and :meth:`check_identity` checks in order to verify that the connections can be used for WAL streaming. Also runs an additional :meth:`_has_monitoring_privileges` check, which validates the WAL-specific conninfo connects with a user than can read monitoring information. :param CheckStrategy check_strategy: The strategy for the management of the result of this check :param dict[str, None|str] remote_status: Remote status information used by this check """ self._check_has_monitoring_privileges( check_strategy, remote_status, "WAL streaming" ) self._check_streaming_supported(check_strategy, remote_status, "WAL streaming") self._check_wal_level(check_strategy, remote_status, "WAL streaming") self.check_identity(check_strategy, remote_status, "WAL streaming") def _check_replication_slot(self, check_strategy, remote_status, suffix=None): """ Check the replication slot used for WAL streaming. If ``streaming_archiver`` is enabled, checks that the replication slot specified in the configuration exists, is initialised and is active. If ``streaming_archiver`` is disabled, checks that the replication slot does not exist. :param CheckStrategy check_strategy: The strategy for the management of the result of this check :param dict[str, None|str] remote_status: Remote status information used by this check :param str|None suffix: A suffix to be appended to the check name """ # Check the presence and the status of the configured replication slot # This check will be skipped if `slot_name` is undefined if self.config.slot_name: check_name = "replication slot" + ("" if suffix is None else f" ({suffix})") check_strategy.init_check(check_name) slot = remote_status["replication_slot"] # The streaming_archiver is enabled if self.config.streaming_archiver is True: # Replication slots are supported # The slot is not present if slot is None: check_strategy.result( self.config.name, False, hint="replication slot '%s' doesn't exist. " "Please execute 'barman receive-wal " "--create-slot %s'" % (self.config.slot_name, self.config.name), ) else: # The slot is present but not initialised if slot.restart_lsn is None: check_strategy.result( self.config.name, False, hint="slot '%s' not initialised: is " "'receive-wal' running?" % self.config.slot_name, ) # The slot is present but not active elif slot.active is False: check_strategy.result( self.config.name, False, hint="slot '%s' not active: is " "'receive-wal' running?" % self.config.slot_name, ) else: check_strategy.result(self.config.name, True) else: # If the streaming_archiver is disabled and the slot_name # option is present in the configuration, we check that # a replication slot with the specified name is NOT present # and NOT active. # NOTE: This is not a failure, just a warning. if slot is not None: if slot.restart_lsn is not None: slot_status = "initialised" # Check if the slot is also active if slot.active: slot_status = "active" # Warn the user check_strategy.result( self.config.name, True, hint="WARNING: slot '%s' is %s but not required " "by the current config" % (self.config.slot_name, slot_status), ) def _check_standby(self, check_strategy): """ Perform checks specific to a primary/standby configuration. :param CheckStrategy check_strategy: The strategy for the management of the results of the various checks. """ is_standby_conn = isinstance(self.postgres, StandbyPostgreSQLConnection) # Check that standby is standby check_strategy.init_check("PostgreSQL server is standby") # The server only is in recovery if we have a standby connection and pg_is_in_recovery() is True is_in_recovery = is_standby_conn and self.postgres.is_in_recovery if is_in_recovery: check_strategy.result(self.config.name, True) else: check_strategy.result( self.config.name, False, hint=( "conninfo should point to a standby server if " "primary_conninfo is set" ), ) # if we don't have a standby connection object then we can't perform # any of the further checks as they require a primary reference if not is_standby_conn: return # Check that primary is not standby check_strategy.init_check("Primary server is not a standby") primary_is_in_recovery = self.postgres.primary.is_in_recovery if not primary_is_in_recovery: check_strategy.result(self.config.name, True) else: check_strategy.result( self.config.name, False, hint=( "primary_conninfo should point to a primary server, " "not a standby" ), ) # Check that system ID is the same for both check_strategy.init_check("Primary and standby have same system ID") standby_id = self.postgres.get_systemid() primary_id = self.postgres.primary.get_systemid() if standby_id == primary_id: check_strategy.result(self.config.name, True) else: check_strategy.result( self.config.name, False, hint=( "primary_conninfo and conninfo should point to primary and " "standby servers which share the same system identifier" ), ) def _make_directories(self): """ Make backup directories in case they do not exist """ for key in self.config.KEYS: if key.endswith("_directory") and hasattr(self.config, key): val = getattr(self.config, key) if val is not None and not os.path.isdir(val): # noinspection PyTypeChecker os.makedirs(val) def check_directories(self, check_strategy): """ Checks backup directories and creates them if they do not exist :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ check_strategy.init_check("directories") if not self.config.disabled: try: self._make_directories() except OSError as e: check_strategy.result( self.config.name, False, "%s: %s" % (e.filename, e.strerror) ) else: check_strategy.result(self.config.name, True) def check_configuration(self, check_strategy): """ Check for error messages in the message list of the server and output eventual errors :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ check_strategy.init_check("configuration") if len(self.config.msg_list): check_strategy.result(self.config.name, False) for conflict_paths in self.config.msg_list: output.info("\t\t%s" % conflict_paths) def check_retention_policy_settings(self, check_strategy): """ Checks retention policy setting :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ check_strategy.init_check("retention policy settings") config = self.config if config.retention_policy and not self.enforce_retention_policies: check_strategy.result(self.config.name, False, hint="see log") else: check_strategy.result(self.config.name, True) def check_backup_validity(self, check_strategy): """ Check if backup validity requirements are satisfied :param CheckStrategy check_strategy: the strategy for the management of the results of the various checks """ check_strategy.init_check("backup maximum age") # first check: check backup maximum age if self.config.last_backup_maximum_age is not None: # get maximum age information backup_age = self.backup_manager.validate_last_backup_maximum_age( self.config.last_backup_maximum_age ) # format the output check_strategy.result( self.config.name, backup_age[0], hint="interval provided: %s, latest backup age: %s" % ( human_readable_timedelta(self.config.last_backup_maximum_age), backup_age[1], ), ) else: # last_backup_maximum_age provided by the user check_strategy.result( self.config.name, True, hint="no last_backup_maximum_age provided" ) # second check: check backup minimum size check_strategy.init_check("backup minimum size") if self.config.last_backup_minimum_size is not None: backup_size = self.backup_manager.validate_last_backup_min_size( self.config.last_backup_minimum_size ) gtlt = ">" if backup_size[0] else "<" check_strategy.result( self.config.name, backup_size[0], hint="last backup size %s %s %s minimum" % ( pretty_size(backup_size[1]), gtlt, pretty_size(self.config.last_backup_minimum_size), ), perfdata=backup_size[1], ) else: # no last_backup_minimum_size provided by the user backup_size = self.backup_manager.validate_last_backup_min_size(0) check_strategy.result( self.config.name, True, hint=pretty_size(backup_size[1]), perfdata=backup_size[1], ) def _check_wal_info(self, wal_info, last_wal_maximum_age): """ Checks the supplied wal_info is within the last_wal_maximum_age. :param last_backup_minimum_age: timedelta representing the time from now during which a WAL is considered valid :return tuple: a tuple containing the boolean result of the check, a string with auxiliary information about the check, and an integer representing the size of the WAL in bytes """ wal_last = datetime.datetime.fromtimestamp( wal_info["wal_last_timestamp"], dateutil.tz.tzlocal() ) now = datetime.datetime.now(dateutil.tz.tzlocal()) wal_age = now - wal_last if wal_age <= last_wal_maximum_age: wal_age_isok = True else: wal_age_isok = False wal_message = "interval provided: %s, latest wal age: %s" % ( human_readable_timedelta(last_wal_maximum_age), human_readable_timedelta(wal_age), ) if wal_info["wal_until_next_size"] is None: wal_size = 0 else: wal_size = wal_info["wal_until_next_size"] return wal_age_isok, wal_message, wal_size def check_wal_validity(self, check_strategy): """ Check if wal archiving requirements are satisfied """ check_strategy.init_check("wal maximum age") backup_id = self.backup_manager.get_last_backup_id() backup_info = self.get_backup(backup_id) if backup_info is not None: wal_info = self.get_wal_info(backup_info) # first check: check wal maximum age if self.config.last_wal_maximum_age is not None: # get maximum age information if backup_info is None or wal_info["wal_last_timestamp"] is None: # No WAL files received # (we should have the .backup file, as a minimum) # This may also be an indication that 'barman cron' is not # running wal_age_isok = False wal_message = "No WAL files archived for last backup" wal_size = 0 else: wal_age_isok, wal_message, wal_size = self._check_wal_info( wal_info, self.config.last_wal_maximum_age ) # format the output check_strategy.result(self.config.name, wal_age_isok, hint=wal_message) else: # no last_wal_maximum_age provided by the user if backup_info is None or wal_info["wal_until_next_size"] is None: wal_size = 0 else: wal_size = wal_info["wal_until_next_size"] check_strategy.result( self.config.name, True, hint="no last_wal_maximum_age provided" ) check_strategy.init_check("wal size") check_strategy.result( self.config.name, True, pretty_size(wal_size), perfdata=wal_size ) def check_archiver_errors(self, check_strategy): """ Checks the presence of archiving errors :param CheckStrategy check_strategy: the strategy for the management of the results of the check """ check_strategy.init_check("archiver errors") if os.path.isdir(self.config.errors_directory): errors = os.listdir(self.config.errors_directory) else: errors = [] check_strategy.result( self.config.name, len(errors) == 0, hint=WalArchiver.summarise_error_files(errors), ) def check_identity(self, check_strategy, remote_status=None, suffix=None): """ Check the systemid retrieved from the streaming connection is the same that is retrieved from the standard connection, and then verifies it matches the one stored on disk. :param CheckStrategy check_strategy: The strategy for the management of the result of this check :param dict[str, None|str] remote_status: Remote status information used by this check :param str|None suffix: A suffix to be appended to the check name """ check_name = "systemid coherence" + ("" if suffix is None else f" ({suffix})") check_strategy.init_check(check_name) if remote_status is None: remote_status = self.get_remote_status() # Get system identifier from streaming and standard connections systemid_from_streaming = remote_status.get("streaming_systemid") systemid_from_postgres = remote_status.get("postgres_systemid") # If both available, makes sure they are coherent with each other if systemid_from_streaming and systemid_from_postgres: if systemid_from_streaming != systemid_from_postgres: check_strategy.result( self.config.name, systemid_from_streaming == systemid_from_postgres, hint="is the streaming DSN targeting the same server " "of the PostgreSQL connection string?", ) return systemid_from_server = systemid_from_streaming or systemid_from_postgres if not systemid_from_server: # Can't check without system Id information check_strategy.result(self.config.name, True, hint="no system Id available") return # Retrieves the content on disk and matches it with the live ID file_path = self.get_identity_file_path() if not os.path.exists(file_path): # We still don't have the systemid cached on disk, # so let's wait until we store it check_strategy.result( self.config.name, True, hint="no system Id stored on disk" ) return identity_from_file = self.read_identity_file() if systemid_from_server != identity_from_file.get("systemid"): check_strategy.result( self.config.name, False, hint="the system Id of the connected PostgreSQL server " 'changed, stored in "%s"' % file_path, ) else: check_strategy.result(self.config.name, True) def status_postgres(self): """ Status of PostgreSQL server """ remote_status = self.get_remote_status() if remote_status["server_txt_version"]: output.result( "status", self.config.name, "pg_version", "PostgreSQL version", remote_status["server_txt_version"], ) else: output.result( "status", self.config.name, "pg_version", "PostgreSQL version", "FAILED trying to get PostgreSQL version", ) return # Define the cluster state as pg_controldata do. if remote_status["is_in_recovery"]: output.result( "status", self.config.name, "is_in_recovery", "Cluster state", "in archive recovery", ) else: output.result( "status", self.config.name, "is_in_recovery", "Cluster state", "in production", ) if remote_status.get("current_size") is not None: output.result( "status", self.config.name, "current_size", "Current data size", pretty_size(remote_status["current_size"]), ) if remote_status["data_directory"]: output.result( "status", self.config.name, "data_directory", "PostgreSQL Data directory", remote_status["data_directory"], ) if remote_status["current_xlog"]: output.result( "status", self.config.name, "current_xlog", "Current WAL segment", remote_status["current_xlog"], ) def status_wal_archiver(self): """ Status of WAL archiver(s) """ for archiver in self.archivers: archiver.status() def status_retention_policies(self): """ Status of retention policies enforcement """ if self.enforce_retention_policies: output.result( "status", self.config.name, "retention_policies", "Retention policies", "enforced " "(mode: %s, retention: %s, WAL retention: %s)" % ( self.config.retention_policy_mode, self.config.retention_policy, self.config.wal_retention_policy, ), ) else: output.result( "status", self.config.name, "retention_policies", "Retention policies", "not enforced", ) def status(self): """ Implements the 'server-status' command. """ if self.config.description: output.result( "status", self.config.name, "description", "Description", self.config.description, ) output.result( "status", self.config.name, "active", "Active", self.config.active ) output.result( "status", self.config.name, "disabled", "Disabled", self.config.disabled ) # Show active configuration model information active_model = ( self.config.active_model.name if self.config.active_model else None ) output.result( "status", self.config.name, "active_model", "Active configuration model", active_model, ) # Postgres status is available only if node is not passive if not self.passive_node: self.status_postgres() self.status_wal_archiver() output.result( "status", self.config.name, "passive_node", "Passive node", self.passive_node, ) self.status_retention_policies() # Executes the backup manager status info method self.backup_manager.status() def fetch_remote_status(self): """ Get the status of the remote server This method does not raise any exception in case of errors, but set the missing values to None in the resulting dictionary. :rtype: dict[str, None|str] """ result = {} # Merge status for a postgres connection if self.postgres: result.update(self.postgres.get_remote_status()) # Merge status for a streaming connection if self.streaming: result.update(self.streaming.get_remote_status()) # Merge status for each archiver for archiver in self.archivers: result.update(archiver.get_remote_status()) # Merge status defined by the BackupManager result.update(self.backup_manager.get_remote_status()) return result def show(self): """ Shows the server configuration """ # Populate result map with all the required keys result = self.config.to_json() # Is the server a passive node? result["passive_node"] = self.passive_node # Skip remote status if the server is passive if not self.passive_node: remote_status = self.get_remote_status() result.update(remote_status) # Backup maximum age section if self.config.last_backup_maximum_age is not None: age = self.backup_manager.validate_last_backup_maximum_age( self.config.last_backup_maximum_age ) # If latest backup is between the limits of the # last_backup_maximum_age configuration, display how old is # the latest backup. if age[0]: msg = "%s (latest backup: %s )" % ( human_readable_timedelta(self.config.last_backup_maximum_age), age[1], ) else: # If latest backup is outside the limits of the # last_backup_maximum_age configuration (or the configuration # value is none), warn the user. msg = "%s (WARNING! latest backup is %s old)" % ( human_readable_timedelta(self.config.last_backup_maximum_age), age[1], ) result["last_backup_maximum_age"] = msg else: result["last_backup_maximum_age"] = "None" # Add active model information result["active_model"] = ( self.config.active_model.name if self.config.active_model else None ) output.result("show_server", self.config.name, result) def delete_backup(self, backup): """ Deletes a backup. Performs some checks to confirm that the backup can indeed be deleted and if so it is deleted along with all backups that depend on it, if any. :param barman.infofile.LocalBackupInfo backup: the backup to delete :return bool: True if deleted, False if could not delete the backup """ if self.backup_manager.should_keep_backup(backup.backup_id): output.warning( "Skipping delete of backup %s for server %s " "as it has a current keep request. If you really " "want to delete this backup please remove the keep " "and try again.", backup.backup_id, self.config.name, ) return False # Honour minimum required redundancy available_backups = self.get_available_backups(status_filter=(BackupInfo.DONE,)) minimum_redundancy = self.config.minimum_redundancy if backup.status == BackupInfo.DONE and minimum_redundancy >= len( available_backups ): output.warning( "Skipping delete of backup %s for server %s " "due to minimum redundancy requirements " "(minimum redundancy = %s, " "current redundancy = %s)", backup.backup_id, self.config.name, minimum_redundancy, len(available_backups), ) return False if backup.children_backup_ids: output.warning( "Backup %s has incremental backups which depend on it. " "Deleting all backups in the tree", backup.backup_id, ) try: # Lock acquisition: if you can acquire a ServerBackupLock it means # that no other processes like a backup or another delete is running with ServerBackupLock(self.config.barman_lock_directory, self.config.name): # Delete the backup along with all its descendants in the # backup tree i.e. all its subsequent incremental backups. # If it has no descendants or it is an rsync backup then # only the current backup is deleted. deleted = False backups_to_delete = backup.walk_backups_tree() for del_backup in backups_to_delete: deleted = self.perform_delete_backup(del_backup) if not deleted and del_backup.backup_id != backup.backup_id: output.error( "Failed to delete one of its incremental backups. Make sure " "all its dependent backups are deletable and try again." ) break return deleted except LockFileBusy: # Otherwise if the lockfile is busy, a backup process is actually running output.error( "Another process in running on server %s. " "Impossible to delete the backup." % self.config.name ) return False except LockFilePermissionDenied as e: # We cannot access the lockfile. # Exit without removing the backup. output.error("Permission denied, unable to access '%s'" % e) return False def perform_delete_backup(self, backup): """ Performs the deletion of a backup. Deletes a single backup, ensuring that no other process can access the backup simultaneously during its deletion. :param barman.infofile.LocalBackupInfo backup: the backup to delete :return bool: True if deleted, False if could not delete the backup """ try: # Take care of the backup lock. # Only one process can modify a backup at a time lock = ServerBackupIdLock( self.config.barman_lock_directory, self.config.name, backup.backup_id ) with lock: deleted = self.backup_manager.delete_backup(backup) # At this point no-one should try locking a backup that # doesn't exists, so we can remove the lock # WARNING: the previous statement is true only as long as # no-one wait on this lock if deleted: os.remove(lock.filename) return deleted except LockFileBusy: # If another process is holding the backup lock, # warn the user and terminate output.error( "Another process is holding the lock for " "backup %s of server %s." % (backup.backup_id, self.config.name) ) return False except LockFilePermissionDenied as e: # We cannot access the lockfile. # warn the user and terminate output.error("Permission denied, unable to access '%s'" % e) return False def backup(self, wait=False, wait_timeout=None, backup_name=None, **kwargs): """ Performs a backup for the server :param bool wait: wait for all the required WAL files to be archived :param int|None wait_timeout: the time, in seconds, the backup will wait for the required WAL files to be archived before timing out :param str|None backup_name: a friendly name by which this backup can be referenced in the future :kwparam str parent_backup_id: id of the parent backup when taking a Postgres incremental backup """ # The 'backup' command is not available on a passive node. # We assume that if we get here the node is not passive assert not self.passive_node try: # validate arguments, raise BackupException if any error is found self.backup_manager.validate_backup_args(**kwargs) # Default strategy for check in backup is CheckStrategy # This strategy does not print any output - it only logs checks strategy = CheckStrategy() self.check(strategy) if strategy.has_error: output.error( "Impossible to start the backup. Check the log " "for more details, or run 'barman check %s'" % self.config.name ) return # check required backup directories exist self._make_directories() except BackupException as e: output.error("failed to start backup: %s", force_str(e)) return except OSError as e: output.error("failed to create %s directory: %s", e.filename, e.strerror) return # Save the database identity self.write_identity_file() # Make sure we are not wasting an precious streaming PostgreSQL # connection that may have been opened by the self.check() call if self.streaming: self.streaming.close() try: # lock acquisition and backup execution with ServerBackupLock(self.config.barman_lock_directory, self.config.name): backup_info = self.backup_manager.backup( wait=wait, wait_timeout=wait_timeout, name=backup_name, **kwargs, ) # Archive incoming WALs and update WAL catalogue self.archive_wal(verbose=False) # Invoke sanity check of the backup if backup_info.status == BackupInfo.WAITING_FOR_WALS: self.check_backup(backup_info) # At this point is safe to remove any remaining WAL file before the # first backup. The only exception is when worm_mode is enabled, in # which case the storage is expected to be immutable and out of the # grace period, so we skip that. previous_backup = self.get_previous_backup(backup_info.backup_id) if not previous_backup and self.config.worm_mode is False: self.backup_manager.remove_wal_before_backup(backup_info) # check if the backup chain (in case it is a Postgres incremental) is consistent # with their checksums configurations if not backup_info.is_checksum_consistent(): output.warning( "This is an incremental backup taken with `data_checksums = on` whereas " "some previous backups in the chain were taken with `data_checksums = off`. " "This can lead to potential recovery issues. Consider taking a new full backup " "to avoid having inconsistent backup chains." ) if backup_info.status == BackupInfo.WAITING_FOR_WALS: output.warning( "IMPORTANT: this backup is classified as " "WAITING_FOR_WALS, meaning that Barman has not received " "yet all the required WAL files for the backup " "consistency.\n" "This is a common behaviour in concurrent backup " "scenarios, and Barman automatically set the backup as " "DONE once all the required WAL files have been " "archived.\n" "Hint: execute the backup command with '--wait'" ) except LockFileBusy: output.error("Another backup process is running") except LockFilePermissionDenied as e: output.error("Permission denied, unable to access '%s'" % e) def get_available_backups(self, status_filter=BackupManager.DEFAULT_STATUS_FILTER): """ Get a list of available backups param: status_filter: the status of backups to return, default to BackupManager.DEFAULT_STATUS_FILTER """ return self.backup_manager.get_available_backups(status_filter) def get_last_backup_id(self, status_filter=BackupManager.DEFAULT_STATUS_FILTER): """ Get the id of the latest/last backup in the catalog (if exists) :param status_filter: The status of the backup to return, default to :attr:`BackupManager.DEFAULT_STATUS_FILTER`. :return str|None: ID of the backup """ return self.backup_manager.get_last_backup_id(status_filter) def get_last_full_backup_id( self, status_filter=BackupManager.DEFAULT_STATUS_FILTER ): """ Get the id of the latest/last FULL backup in the catalog (if exists) :param status_filter: The status of the backup to return, default to DEFAULT_STATUS_FILTER. :return string|None: ID of the backup """ return self.backup_manager.get_last_full_backup_id(status_filter) def get_first_backup_id(self, status_filter=BackupManager.DEFAULT_STATUS_FILTER): """ Get the id of the oldest/first backup in the catalog (if exists) :param status_filter: The status of the backup to return, default to DEFAULT_STATUS_FILTER. :return string|None: ID of the backup """ return self.backup_manager.get_first_backup_id(status_filter) def get_backup_id_from_name( self, backup_name, status_filter=BackupManager.DEFAULT_STATUS_FILTER ): """ Get the id of the named backup, if it exists. :param string backup_name: The name of the backup for which an ID should be returned :param tuple status_filter: The status of the backup to return. :return string|None: ID of the backup """ # Iterate through backups and see if there is one which matches the name return self.backup_manager.get_backup_id_from_name(backup_name, status_filter) def get_closest_backup_id_from_target_lsn( self, target_lsn, target_tli, status_filter=BackupManager.DEFAULT_STATUS_FILTER, ): """ Get the id of a backup according to the *target_lsn* and *target_tli*. :param str target_lsn: The target value with lsn format, e.g., ``3/64000000``. :param int|None target_tli: The target timeline, if a specific one is required. :param tuple[str, ...] status_filter: The status of the backup to return. :return str|None: ID of the backup. """ return self.backup_manager.get_closest_backup_id_from_target_lsn( target_lsn, target_tli, status_filter ) def get_closest_backup_id_from_target_time( self, target_time, target_tli, status_filter=BackupManager.DEFAULT_STATUS_FILTER, ): """ Get the id of a backup according to the *target_time* and *target_tli*, if it exists. :param str target_time: The target value with timestamp format ``%Y-%m-%d %H:%M:%S`` with or without timezone. :param int|None target_tli: The target timeline, if a specific one is required. :param tuple[str, ...] status_filter: The status of the backup to return. :return str|None: ID of the backup. """ return self.backup_manager.get_closest_backup_id_from_target_time( target_time, target_tli, status_filter ) def get_last_backup_id_from_target_tli( self, target_tli, status_filter=BackupManager.DEFAULT_STATUS_FILTER, ): """ Get the id of a backup according to the *target_tli*. :param int target_tli: The recovery target timeline. :param tuple[str, ...] status_filter: The status of the backup to return. :return str|None: ID of the backup. """ return self.backup_manager.get_last_backup_id_from_target_tli( target_tli, status_filter ) def list_backups(self): """ Lists all the available backups for the server """ retention_status = self.report_backups() backups = self.get_available_backups(BackupInfo.STATUS_ALL) for key in sorted(backups.keys(), reverse=True): backup = backups[key] backup_size = backup.size or 0 wal_size = 0 rstatus = None if backup.status in BackupInfo.STATUS_COPY_DONE: try: wal_info = self.get_wal_info(backup) backup_size += wal_info["wal_size"] wal_size = wal_info["wal_until_next_size"] except BadXlogSegmentName as e: output.error( "invalid WAL segment name %r\n" 'HINT: Please run "barman rebuild-xlogdb %s" ' "to solve this issue", force_str(e), self.config.name, ) if ( self.enforce_retention_policies and retention_status[backup.backup_id] != BackupInfo.VALID ): rstatus = retention_status[backup.backup_id] output.result("list_backup", backup, backup_size, wal_size, rstatus) def get_backup(self, backup_id): """ Return the backup information for the given backup id. If the backup_id is None or backup.info file doesn't exists, it returns None. :param str|None backup_id: the ID of the backup to return :rtype: barman.infofile.LocalBackupInfo|None """ return self.backup_manager.get_backup(backup_id) def get_previous_backup(self, backup_id): """ Get the previous backup (if any) from the catalog :param backup_id: the backup id from which return the previous """ return self.backup_manager.get_previous_backup(backup_id) def get_next_backup(self, backup_id): """ Get the next backup (if any) from the catalog :param backup_id: the backup id from which return the next """ return self.backup_manager.get_next_backup(backup_id) def get_required_xlog_files( self, backup, target_tli=None, target_time=None, target_xid=None, target_lsn=None, target_immediate=False, ): """ Get the xlog files required for a recovery. .. note:: *target_time* and *target_xid* are ignored by this method. As it can be very expensive to parse WAL dumps to identify which WAL files are required to honor the specific targets, we simply copy all WAL files up to the calculated target timeline, so we make sure recovery will be able to finish successfully (assuming the archived WALs honor the specified targets). On the other hand, *target_tli*, *target_lsn* and *target_immediate* are easier to handle, so we only copy the WALs required to reach the requested targets. :param BackupInfo backup: a backup object :param target_tli : target timeline, either a timeline ID or one of the keywords supported by Postgres :param target_time: target time, in epoch :param target_xid: target transaction ID :param target_lsn: target LSN :param target_immediate: target that ends recovery as soon as consistency is reached. Defaults to ``False``. """ begin = backup.begin_wal end = backup.end_wal # Calculate the integer value of TLI if a keyword is provided calculated_target_tli = parse_target_tli( self.backup_manager, target_tli, backup ) # If timeline isn't specified, assume it is the same timeline # of the backup if not target_tli: target_tli, _, _ = xlog.decode_segment_name(end) calculated_target_tli = target_tli # If a target LSN was specified, get the name of the last WAL file that is # required for the recovery process if target_lsn: target_wal = xlog.location_to_xlogfile_name_offset( target_lsn, calculated_target_tli, backup.xlog_segment_size, )["file_name"] with self.xlogdb() as fxlogdb: for line in fxlogdb: wal_info = WalFileInfo.from_xlogdb_line(line) # Handle .history files: add all of them to the output, # regardless of their age if xlog.is_history_file(wal_info.name): yield wal_info continue if wal_info.name < begin: continue tli, _, _ = xlog.decode_segment_name(wal_info.name) if tli > calculated_target_tli: continue if wal_info.name > end: if target_immediate: break if target_lsn and wal_info.name > target_wal: break end = wal_info.name yield wal_info # return all the remaining history files for line in fxlogdb: wal_info = WalFileInfo.from_xlogdb_line(line) if xlog.is_history_file(wal_info.name): yield wal_info # TODO: merge with the previous def get_wal_until_next_backup(self, backup, include_history=False): """ Get the xlog files between backup and the next :param BackupInfo backup: a backup object, the starting point to retrieve WALs :param bool include_history: option for the inclusion of include_history files into the output """ begin = backup.begin_wal next_end = None if self.get_next_backup(backup.backup_id): next_end = self.get_next_backup(backup.backup_id).end_wal backup_tli, _, _ = xlog.decode_segment_name(begin) with self.xlogdb() as fxlogdb: for line in fxlogdb: wal_info = WalFileInfo.from_xlogdb_line(line) # Handle .history files: add all of them to the output, # regardless of their age, if requested (the 'include_history' # parameter is True) if xlog.is_history_file(wal_info.name): if include_history: yield wal_info continue if wal_info.name < begin: continue tli, _, _ = xlog.decode_segment_name(wal_info.name) if tli > backup_tli: continue if not xlog.is_wal_file(wal_info.name): continue if next_end and wal_info.name > next_end: break yield wal_info def get_wal_full_path(self, wal_name): """ Build the full path of a WAL for a server given the name :param wal_name: WAL file name """ # Build the path which contains the file hash_dir = os.path.join(self.config.wals_directory, xlog.hash_dir(wal_name)) # Build the WAL file full path full_path = os.path.join(hash_dir, wal_name) return full_path def get_wal_possible_paths(self, wal_name, partial=False): """ Build a list of possible positions of a WAL file :param str wal_name: WAL file name :param bool partial: add also the '.partial' paths """ paths = list() # Path in the archive hash_dir = os.path.join(self.config.wals_directory, xlog.hash_dir(wal_name)) full_path = os.path.join(hash_dir, wal_name) paths.append(full_path) # Path in incoming directory incoming_path = os.path.join(self.config.incoming_wals_directory, wal_name) paths.append(incoming_path) # Path in streaming directory streaming_path = os.path.join(self.config.streaming_wals_directory, wal_name) paths.append(streaming_path) # If partial files are required check also the '.partial' path if partial: paths.append(streaming_path + PARTIAL_EXTENSION) # Add the streaming_path again to handle races with pg_receivewal # completing the WAL file paths.append(streaming_path) # The following two path are only useful to retrieve the last # incomplete segment archived before a promotion. paths.append(full_path + PARTIAL_EXTENSION) paths.append(incoming_path + PARTIAL_EXTENSION) # Append the archive path again, to handle races with the archiver paths.append(full_path) return paths def get_wal_info(self, backup_info): """ Returns information about WALs for the given backup :param barman.infofile.LocalBackupInfo backup_info: the target backup """ begin = backup_info.begin_wal end = backup_info.end_wal # counters wal_info = dict.fromkeys( ( "wal_num", "wal_size", "wal_until_next_num", "wal_until_next_size", "wal_until_next_compression_ratio", "wal_compression_ratio", ), 0, ) # First WAL (always equal to begin_wal) and Last WAL names and ts wal_info["wal_first"] = None wal_info["wal_first_timestamp"] = None wal_info["wal_last"] = None wal_info["wal_last_timestamp"] = None # WAL rate (default 0.0 per second) wal_info["wals_per_second"] = 0.0 for item in self.get_wal_until_next_backup(backup_info): if item.name == begin: wal_info["wal_first"] = item.name wal_info["wal_first_timestamp"] = item.time if item.name <= end: wal_info["wal_num"] += 1 wal_info["wal_size"] += item.size else: wal_info["wal_until_next_num"] += 1 wal_info["wal_until_next_size"] += item.size wal_info["wal_last"] = item.name wal_info["wal_last_timestamp"] = item.time # Calculate statistics only for complete backups # If the cron is not running for any reason, the required # WAL files could be missing if wal_info["wal_first"] and wal_info["wal_last"]: # Estimate WAL ratio # Calculate the difference between the timestamps of # the first WAL (begin of backup) and the last WAL # associated to the current backup wal_last_timestamp = wal_info["wal_last_timestamp"] wal_first_timestamp = wal_info["wal_first_timestamp"] wal_info["wal_total_seconds"] = wal_last_timestamp - wal_first_timestamp if wal_info["wal_total_seconds"] > 0: wal_num = wal_info["wal_num"] wal_until_next_num = wal_info["wal_until_next_num"] wal_total_seconds = wal_info["wal_total_seconds"] wal_info["wals_per_second"] = ( float(wal_num + wal_until_next_num) / wal_total_seconds ) # evaluation of compression ratio for basebackup WAL files wal_info["wal_theoretical_size"] = wal_info["wal_num"] * float( backup_info.xlog_segment_size ) try: wal_size = wal_info["wal_size"] wal_info["wal_compression_ratio"] = 1 - ( wal_size / wal_info["wal_theoretical_size"] ) except ZeroDivisionError: wal_info["wal_compression_ratio"] = 0.0 # evaluation of compression ratio of WAL files wal_until_next_num = wal_info["wal_until_next_num"] wal_info["wal_until_next_theoretical_size"] = wal_until_next_num * float( backup_info.xlog_segment_size ) try: wal_until_next_size = wal_info["wal_until_next_size"] until_next_theoretical_size = wal_info[ "wal_until_next_theoretical_size" ] wal_info["wal_until_next_compression_ratio"] = 1 - ( wal_until_next_size / until_next_theoretical_size ) except ZeroDivisionError: wal_info["wal_until_next_compression_ratio"] = 0.0 return wal_info def recover( self, backup_info, dest, wal_dest=None, tablespaces=None, remote_command=None, **kwargs, ): """ Performs a recovery of a backup :param barman.infofile.LocalBackupInfo backup_info: the backup to recover :param str dest: the destination directory :param str|None wal_dest: the destination directory for WALs when doing PITR. See :meth:`~barman.recovery_executor.RecoveryExecutor._set_pitr_targets` for more details. :param dict[str,str]|None tablespaces: a tablespace name -> location map (for relocation) :param str|None remote_command: default None. The remote command to recover the base backup, in case of remote backup. :kwparam str|None target_tli: the target timeline :kwparam str|None target_time: the target time :kwparam str|None target_xid: the target xid :kwparam str|None target_lsn: the target LSN :kwparam str|None target_name: the target name created previously with pg_create_restore_point() function call :kwparam bool|None target_immediate: end recovery as soon as consistency is reached :kwparam bool exclusive: whether the recovery is exclusive or not :kwparam str|None target_action: the recovery target action :kwparam bool|None standby_mode: the standby mode :kwparam str|None recovery_conf_filename: filename for storing recovery configurations """ return self.backup_manager.recover( backup_info, dest, wal_dest, tablespaces, remote_command, **kwargs ) def get_wal( self, wal_name, compression=None, keep_compression=False, output_directory=None, peek=None, partial=False, ): """ Retrieve a WAL file from the archive :param str wal_name: id of the WAL file to find into the WAL archive :param str|None compression: compression format for the output :param bool keep_compression: if True, do not uncompress compressed WAL files :param str|None output_directory: directory where to deposit the WAL file :param int|None peek: if defined list the next N WAL file :param bool partial: retrieve also partial WAL files """ # If used through SSH identify the client to add it to logs source_suffix = "" ssh_connection = os.environ.get("SSH_CONNECTION") if ssh_connection: # The client IP is the first value contained in `SSH_CONNECTION` # which contains four space-separated values: client IP address, # client port number, server IP address, and server port number. source_suffix = " (SSH host: %s)" % (ssh_connection.split()[0],) # Sanity check if not xlog.is_any_xlog_file(wal_name): output.error( "'%s' is not a valid wal file name%s", wal_name, source_suffix, exit_code=3, ) return # If peek is requested we only output a list of files if peek: # Get the next ``peek`` files following the provided ``wal_name``. # If ``wal_name`` is not a simple wal file, # we cannot guess the names of the following WAL files. # So ``wal_name`` is the only possible result, if exists. if xlog.is_wal_file(wal_name): # We can't know what was the segment size of PostgreSQL WAL # files at backup time. Because of this, we generate all # the possible names for a WAL segment, and then we check # if the requested one is included. wal_peek_list = xlog.generate_segment_names(wal_name) else: wal_peek_list = iter([wal_name]) # Output the content of wal_peek_list until we have displayed # enough files or find a missing file count = 0 while count < peek: try: wal_peek_name = next(wal_peek_list) except StopIteration: # No more item in wal_peek_list break # Get list of possible location. We do not prefetch # partial files wal_peek_paths = self.get_wal_possible_paths( wal_peek_name, partial=False ) # If the next WAL file is found, output the name # and continue to the next one if any(os.path.exists(path) for path in wal_peek_paths): count += 1 output.info(wal_peek_name, log=False) continue # If ``wal_peek_file`` doesn't exist, check if we need to # look in the following segment tli, log, seg = xlog.decode_segment_name(wal_peek_name) # If `seg` is not a power of two, it is not possible that we # are at the end of a WAL group, so we are done if not is_power_of_two(seg): break # This is a possible WAL group boundary, let's try the # following group seg = 0 log += 1 # Install a new generator from the start of the next segment. # If the file doesn't exists we will terminate because # zero is not a power of two wal_peek_name = xlog.encode_segment_name(tli, log, seg) wal_peek_list = xlog.generate_segment_names(wal_peek_name) # Do not output anything else return # If an output directory was provided write the file inside it # otherwise we use standard output if output_directory is not None: destination_path = os.path.join(output_directory, wal_name) destination_description = "into '%s' file" % destination_path # Use the standard output for messages logger = output try: destination = open(destination_path, "wb") except IOError as e: output.error( "Unable to open '%s' file%s: %s", destination_path, source_suffix, e, exit_code=3, ) return else: destination_description = "to standard output" # Do not use the standard output for messages, otherwise we would # taint the output stream logger = _logger try: # Python 3.x destination = sys.stdout.buffer except AttributeError: # Python 2.x destination = sys.stdout # Get the list of WAL file possible paths wal_paths = self.get_wal_possible_paths(wal_name, partial) for wal_file in wal_paths: # Check for file existence if not os.path.exists(wal_file): continue logger.info( "Sending WAL '%s' for server '%s' %s%s", os.path.basename(wal_file), self.config.name, destination_description, source_suffix, ) try: # Try returning the wal_file to the client self.get_wal_sendfile( wal_file, compression, keep_compression, destination ) # We are done, return to the caller return except CommandFailedException: # If an external command fails we cannot really know why, # but if the WAL file disappeared, we assume # it has been moved in the archive so we ignore the error. # This file will be retrieved later, as the last entry # returned by get_wal_possible_paths() is the archive position if not os.path.exists(wal_file): pass else: raise except OSError as exc: # If the WAL file disappeared just ignore the error # This file will be retrieved later, as the last entry # returned by get_wal_possible_paths() is the archive # position if exc.errno == errno.ENOENT and exc.filename == wal_file: pass else: raise logger.info("Skipping vanished WAL file '%s'%s", wal_file, source_suffix) output.error( "WAL file '%s' not found in server '%s'%s", wal_name, self.config.name, source_suffix, ) def get_wal_sendfile(self, wal_file, compression, keep_compression, destination): """ Send a WAL file to the destination file, using the required compression :param str wal_file: WAL file path :param str compression: required compression :param bool keep_compression: if True, do not uncompress compressed WAL files :param destination: file stream to use to write the data """ # Identify the wal file wal_info = self.backup_manager.compression_manager.get_wal_file_info(wal_file) # Get a decompressor for the file (None if not compressed) wal_compressor = self.backup_manager.compression_manager.get_compressor( wal_info.compression ) # Get a compressor for the output (None if not compressed) out_compressor = self.backup_manager.compression_manager.get_compressor( compression ) # Initially our source is the stored WAL file and we do not have # any temporary file source_file = wal_file uncompressed_file = None compressed_file = None # Ignore compression/decompression when: # * The user wants to decompress on the client side; or # * It's a partial WAL file. In this case, the WAL file is still being written # by pg_receivewal, and surely has not yet been compressed by the Barman # archiver. if not keep_compression and not xlog.is_partial_file(wal_info.fullpath(self)): # If the required compression is different from the source we # decompress/compress it into the required format (getattr is # used here to gracefully handle None objects) if getattr(wal_compressor, "compression", None) != getattr( out_compressor, "compression", None ): # If source is compressed, decompress it into a temporary file if wal_compressor is not None: uncompressed_file = NamedTemporaryFile( dir=self.config.wals_directory, prefix=".%s." % os.path.basename(wal_file), suffix=".uncompressed", ) # If a custom decompression filter is set, we prioritize using it # instead of the compression guessed by Barman based on the magic # number. is_decompressed = False if ( self.config.custom_decompression_filter is not None and not isinstance(wal_compressor, CustomCompressor) ): try: self.backup_manager.compression_manager.get_compressor( "custom" ).decompress(source_file, uncompressed_file.name) except CommandFailedException as exc: output.debug("Error decompressing WAL: %s", str(exc)) else: is_decompressed = True # But if a custom decompression filter is not set, or if using the # custom decompression filter was not successful, then try using # the decompressor identified by the magic number if not is_decompressed: try: wal_compressor.decompress( source_file, uncompressed_file.name ) except CommandFailedException as exc: output.error("Error decompressing WAL: %s", str(exc)) return source_file = uncompressed_file.name # If output compression is required compress the source # into a temporary file if out_compressor is not None: compressed_file = NamedTemporaryFile( dir=self.config.wals_directory, prefix=".%s." % os.path.basename(wal_file), suffix=".compressed", ) out_compressor.compress(source_file, compressed_file.name) source_file = compressed_file.name # Copy the prepared source file to destination with open(source_file, "rb") as input_file: shutil.copyfileobj(input_file, destination) # Remove temp files if uncompressed_file is not None: uncompressed_file.close() if compressed_file is not None: compressed_file.close() def put_wal(self, fileobj): """ Receive a WAL file from SERVER_NAME and securely store it in the incoming directory. The file will be read from the fileobj passed as parameter. """ # If used through SSH identify the client to add it to logs source_suffix = "" ssh_connection = os.environ.get("SSH_CONNECTION") if ssh_connection: # The client IP is the first value contained in `SSH_CONNECTION` # which contains four space-separated values: client IP address, # client port number, server IP address, and server port number. source_suffix = " (SSH host: %s)" % (ssh_connection.split()[0],) # Incoming directory is where the files will be extracted dest_dir = self.config.incoming_wals_directory # Ensure the presence of the destination directory mkpath(dest_dir) incoming_file = namedtuple( "incoming_file", [ "name", "tmp_path", "path", "checksum", ], ) # Stream read tar from stdin, store content in incoming directory # The closing wrapper is needed only for Python 2.6 extracted_files = {} validated_files = {} hashsums = {} extracted_files_with_checksums = {} hash_algorithm = "sha256" try: with closing(tarfile.open(mode="r|", fileobj=fileobj)) as tar: for item in tar: name = item.name # Strip leading './' - tar has been manually created if name.startswith("./"): name = name[2:] # Requires a regular file as tar item if not item.isreg(): output.error( "Unsupported file type '%s' for file '%s' " "in put-wal for server '%s'%s", item.type, name, self.config.name, source_suffix, ) return # Subdirectories are not supported if "/" in name: output.error( "Unsupported filename '%s' in put-wal for server '%s'%s", name, self.config.name, source_suffix, ) return # Checksum file if name in ("MD5SUMS", "SHA256SUMS"): # Parse content and store it in md5sums dictionary for line in tar.extractfile(item).readlines(): line = line.decode().rstrip() try: # Split checksums and path info checksum, path = re.split(r" [* ]", line, 1) except ValueError: output.warning( "Bad checksum line '%s' found " "in put-wal for server '%s'%s", line, self.config.name, source_suffix, ) continue # Strip leading './' from path in the checksum file if path.startswith("./"): path = path[2:] hashsums[path] = checksum if name == "MD5SUMS": hash_algorithm = "md5" else: # Extract using a temp name (with PID) tmp_path = os.path.join( dest_dir, ".%s-%s" % (os.getpid(), name) ) path = os.path.join(dest_dir, name) tar.makefile(item, tmp_path) # Set the original timestamp tar.utime(item, tmp_path) # Add the tuple to the dictionary of extracted files extracted_files[name] = dict( name=name, tmp_path=tmp_path, path=path, ) validated_files[name] = False for name, _dict in extracted_files.items(): extracted_files_with_checksums[name] = incoming_file( _dict["name"], _dict["tmp_path"], _dict["path"], file_hash(_dict["tmp_path"], hash_algorithm=hash_algorithm), ) # For each received checksum verify the corresponding file for name in hashsums: # Check that file is present in the tar archive if name not in extracted_files_with_checksums: output.error( "Checksum without corresponding file '%s' " "in put-wal for server '%s'%s", name, self.config.name, source_suffix, ) return # Verify the checksum of the file if extracted_files_with_checksums[name].checksum != hashsums[name]: output.error( "Bad file checksum '%s' (should be %s) " "for file '%s' " "in put-wal for server '%s'%s", extracted_files_with_checksums[name].checksum, hashsums[name], name, self.config.name, source_suffix, ) return _logger.info( "Received file '%s' with checksum '%s' " "by put-wal for server '%s'%s", name, hashsums[name], self.config.name, source_suffix, ) validated_files[name] = True # Put the files in the final place, atomically and fsync all for item in extracted_files_with_checksums.values(): # Final verification of checksum presence for each file if not validated_files[item.name]: output.error( "Missing checksum for file '%s' " "in put-wal for server '%s'%s", item.name, self.config.name, source_suffix, ) return # If a file with the same name exists, checksums are compared. # If checksums mismatch, an error message is generated, the incoming # file is moved to the errors directory. # If checksums are identical, a debug message is generated and the file # is skipped. # In both cases the archiving process will exit with 0, avoiding # that WALs pile up on Postgres. if os.path.exists(item.path): incoming_dir_file_checksum = file_hash( file_path=item.path, hash_algorithm=hash_algorithm ) if item.checksum == incoming_dir_file_checksum: output.debug( "Duplicate Files with Identical Checksums. File %s already " "exists on server %s, and the checksums are identical. " "Skipping the file.", item.name, self.config.name, ) continue else: self.move_wal_file_to_errors_directory( item.tmp_path, item.name, "duplicate" ) output.info( "\tError: Duplicate Files Detected with Mismatched " "Checksums. File %s already exists on server %s with " "checksum %s, but the checksum of the incoming file is%s. " "The file has been moved to the errors directory.", item.name, self.config.name, incoming_dir_file_checksum, item.checksum, ) continue os.rename(item.tmp_path, item.path) fsync_file(item.path) fsync_dir(dest_dir) finally: # Cleanup of any remaining temp files (where applicable) for item in extracted_files_with_checksums.values(): if os.path.exists(item.tmp_path): os.unlink(item.tmp_path) def cron(self, wals=True, retention_policies=True, keep_descriptors=False): """ Maintenance operations :param bool wals: WAL archive maintenance :param bool retention_policies: retention policy maintenance :param bool keep_descriptors: whether to keep subprocess descriptors, defaults to False """ try: # Actually this is the highest level of locking in the cron, # this stops the execution of multiple cron on the same server with ServerCronLock(self.config.barman_lock_directory, self.config.name): # When passive call sync.cron() and never run # local WAL archival if self.passive_node: self.sync_cron(keep_descriptors) # WAL management and maintenance elif wals: # Execute the archive-wal sub-process self.cron_archive_wal(keep_descriptors) if self.config.streaming_archiver: # Spawn the receive-wal sub-process self.background_receive_wal(keep_descriptors) else: # Terminate the receive-wal sub-process if present self.kill("receive-wal", fail_if_not_present=False) # Verify backup self.cron_check_backup(keep_descriptors) # Retention policies execution if retention_policies: self.backup_manager.cron_retention_policy() except LockFileBusy: output.info( "Another cron process is already running on server %s. " "Skipping to the next server" % self.config.name ) except LockFilePermissionDenied as e: output.error("Permission denied, unable to access '%s'" % e) except (OSError, IOError) as e: output.error("%s", e) def cron_archive_wal(self, keep_descriptors): """ Method that handles the start of an 'archive-wal' sub-process. This method must be run protected by ServerCronLock :param bool keep_descriptors: whether to keep subprocess descriptors attached to this process. """ try: # Try to acquire ServerWalArchiveLock, if the lock is available, # no other 'archive-wal' processes are running on this server. # # There is a very little race condition window here because # even if we are protected by ServerCronLock, the user could run # another 'archive-wal' command manually. However, it would result # in one of the two commands failing on lock acquisition, # with no other consequence. with ServerWalArchiveLock( self.config.barman_lock_directory, self.config.name ): # Output and release the lock immediately output.info( "Starting WAL archiving for server %s", self.config.name, log=False ) # Init a Barman sub-process object archive_process = BarmanSubProcess( subcommand="archive-wal", config=barman.__config__.config_file, args=[self.config.name], keep_descriptors=keep_descriptors, ) # Launch the sub-process archive_process.execute() except LockFileBusy: # Another archive process is running for the server, # warn the user and skip to the next one. output.info( "Another archive-wal process is already running " "on server %s. Skipping to the next server" % self.config.name ) def background_receive_wal(self, keep_descriptors): """ Method that handles the start of a 'receive-wal' sub process, running in background. This method must be run protected by ServerCronLock :param bool keep_descriptors: whether to keep subprocess descriptors attached to this process. """ try: # Try to acquire ServerWalReceiveLock, if the lock is available, # no other 'receive-wal' processes are running on this server. # # There is a very little race condition window here because # even if we are protected by ServerCronLock, the user could run # another 'receive-wal' command manually. However, it would result # in one of the two commands failing on lock acquisition, # with no other consequence. with ServerWalReceiveLock( self.config.barman_lock_directory, self.config.name ): # Output and release the lock immediately output.info( "Starting streaming archiver for server %s", self.config.name, log=False, ) # Start a new receive-wal process receive_process = BarmanSubProcess( subcommand="receive-wal", config=barman.__config__.config_file, args=[self.config.name], keep_descriptors=keep_descriptors, ) # Launch the sub-process receive_process.execute() except LockFileBusy: # Another receive-wal process is running for the server # exit without message _logger.debug( "Another STREAMING ARCHIVER process is running for " "server %s" % self.config.name ) def cron_check_backup(self, keep_descriptors): """ Method that handles the start of a 'check-backup' sub process :param bool keep_descriptors: whether to keep subprocess descriptors attached to this process. """ backup_id = self.get_first_backup_id([BackupInfo.WAITING_FOR_WALS]) if not backup_id: # Nothing to be done for this server return try: # Try to acquire ServerBackupIdLock, if the lock is available, # no other 'check-backup' processes are running on this backup. # # There is a very little race condition window here because # even if we are protected by ServerCronLock, the user could run # another command that takes the lock. However, it would result # in one of the two commands failing on lock acquisition, # with no other consequence. with ServerBackupIdLock( self.config.barman_lock_directory, self.config.name, backup_id ): # Output and release the lock immediately output.info( "Starting check-backup for backup %s of server %s", backup_id, self.config.name, log=False, ) # Start a check-backup process check_process = BarmanSubProcess( subcommand="check-backup", config=barman.__config__.config_file, args=[self.config.name, backup_id], keep_descriptors=keep_descriptors, ) check_process.execute() except LockFileBusy: # Another process is holding the backup lock _logger.debug( "Another process is holding the backup lock for %s " "of server %s" % (backup_id, self.config.name) ) def archive_wal(self, verbose=True): """ Perform the WAL archiving operations. Usually run as subprocess of the barman cron command, but can be executed manually using the barman archive-wal command :param bool verbose: if false outputs something only if there is at least one file """ output.debug("Starting archive-wal for server %s", self.config.name) try: # Take care of the archive lock. # Only one archive job per server is admitted with ServerWalArchiveLock( self.config.barman_lock_directory, self.config.name ): self.backup_manager.archive_wal(verbose) except LockFileBusy: # If another process is running for this server, # warn the user and skip to the next server output.info( "Another archive-wal process is already running " "on server %s. Skipping to the next server" % self.config.name ) def create_physical_repslot(self): """ Create a physical replication slot using the streaming connection """ if not self.streaming: output.error( "Unable to create a physical replication slot: " "streaming connection not configured" ) return # Replication slots are not supported by PostgreSQL < 9.4 try: if self.streaming.server_version < 90400: output.error( "Unable to create a physical replication slot: " "not supported by '%s' " "(9.4 or higher is required)" % self.streaming.server_major_version ) return except PostgresException as exc: msg = "Cannot connect to server '%s'" % self.config.name output.error(msg, log=False) _logger.error("%s: %s", msg, force_str(exc).strip()) return if not self.config.slot_name: output.error( "Unable to create a physical replication slot: " "slot_name configuration option required" ) return output.info( "Creating physical replication slot '%s' on server '%s'", self.config.slot_name, self.config.name, ) try: self.streaming.create_physical_repslot(self.config.slot_name) output.info("Replication slot '%s' created", self.config.slot_name) except PostgresDuplicateReplicationSlot: output.error("Replication slot '%s' already exists", self.config.slot_name) except PostgresReplicationSlotsFull: output.error( "All replication slots for server '%s' are in use\n" "Free one or increase the max_replication_slots " "value on your PostgreSQL server.", self.config.name, ) except PostgresException as exc: output.error( "Cannot create replication slot '%s' on server '%s': %s", self.config.slot_name, self.config.name, force_str(exc).strip(), ) def drop_repslot(self): """ Drop a replication slot using the streaming connection """ if not self.streaming: output.error( "Unable to drop a physical replication slot: " "streaming connection not configured" ) return # Replication slots are not supported by PostgreSQL < 9.4 try: if self.streaming.server_version < 90400: output.error( "Unable to drop a physical replication slot: " "not supported by '%s' (9.4 or higher is " "required)" % self.streaming.server_major_version ) return except PostgresException as exc: msg = "Cannot connect to server '%s'" % self.config.name output.error(msg, log=False) _logger.error("%s: %s", msg, force_str(exc).strip()) return if not self.config.slot_name: output.error( "Unable to drop a physical replication slot: " "slot_name configuration option required" ) return output.info( "Dropping physical replication slot '%s' on server '%s'", self.config.slot_name, self.config.name, ) try: self.streaming.drop_repslot(self.config.slot_name) output.info("Replication slot '%s' dropped", self.config.slot_name) except PostgresInvalidReplicationSlot: output.error("Replication slot '%s' does not exist", self.config.slot_name) except PostgresReplicationSlotInUse: output.error( "Cannot drop replication slot '%s' on server '%s' " "because it is in use.", self.config.slot_name, self.config.name, ) except PostgresException as exc: output.error( "Cannot drop replication slot '%s' on server '%s': %s", self.config.slot_name, self.config.name, force_str(exc).strip(), ) def receive_wal(self, reset=False): """ Enable the reception of WAL files using streaming protocol. Usually started by barman cron command. Executing this manually, the barman process will not terminate but will continuously receive WAL files from the PostgreSQL server. :param reset: When set, resets the status of receive-wal """ # Execute the receive-wal command only if streaming_archiver # is enabled if not self.config.streaming_archiver: output.error( "Unable to start receive-wal process: " "streaming_archiver option set to 'off' in " "barman configuration file" ) return # Use the default CheckStrategy to silently check WAL streaming # conditions are met and write errors to the log file. strategy = CheckStrategy() self._check_wal_streaming_preflight(strategy, self.get_remote_status()) if strategy.has_error: output.error( "Impossible to start WAL streaming. Check the log " "for more details, or run 'barman check %s'" % self.config.name ) return if not reset: output.info("Starting receive-wal for server %s", self.config.name) try: # Take care of the receive-wal lock. # Only one receiving process per server is permitted with ServerWalReceiveLock( self.config.barman_lock_directory, self.config.name ): try: # Only the StreamingWalArchiver implementation # does something. # WARNING: This codes assumes that there is only one # StreamingWalArchiver in the archivers list. for archiver in self.archivers: archiver.receive_wal(reset) except ArchiverFailure as e: output.error(e) except LockFileBusy: # If another process is running for this server, if reset: output.error( "Unable to reset the status of receive-wal " "for server %s. Process is still running" % self.config.name ) else: output.error( "Another receive-wal process is already running " "for server %s." % self.config.name ) @property def meta_directory(self): """ Directory used to store server metadata files. """ return os.path.join(self.config.backup_directory, "meta") @property def systemid(self): """ Get the system identifier, as returned by the PostgreSQL server :return str: the system identifier """ status = self.get_remote_status() # Main PostgreSQL connection has higher priority if status.get("postgres_systemid"): return status.get("postgres_systemid") # Fallback: streaming connection return status.get("streaming_systemid") @property def xlogdb_directory(self): """ The base directory where the xlogdb file lives :return str: the directory that contains the xlogdb file """ return self.config.xlogdb_directory @property def xlogdb_file_name(self): """ The name of the xlogdb file. :return str: the dynamic name for the xlogdb file """ return self.XLOGDB_NAME.format(server=self.config.name) @property def xlogdb_file_path(self): """ The path of the xlogdb file :return str: the full path of the xlogdb file """ return os.path.join(self.xlogdb_directory, self.xlogdb_file_name) @contextmanager def xlogdb(self, mode="r"): """ Context manager to access the xlogdb file. This method uses locking to make sure only one process is accessing the database at a time. The database file will be created if it not exists. Usage example: with server.xlogdb('w') as file: file.write(new_line) :param str mode: open the file with the required mode (default read-only) """ xlogdb = self.xlogdb_file_path if not os.path.exists(xlogdb): self.rebuild_xlogdb(silent=True) with ServerXLOGDBLock(self.config.barman_lock_directory, self.config.name): with open(xlogdb, mode) as f: # execute the block nested in the with statement try: yield f finally: # we are exiting the context # if file is writable (mode contains w, a or +) # make sure the data is written to disk # http://docs.python.org/2/library/os.html#os.fsync if any((c in "wa+") for c in f.mode): f.flush() os.fsync(f.fileno()) def report_backups(self): if not self.enforce_retention_policies: return dict() else: return self.config.retention_policy.report() def rebuild_xlogdb(self, silent=False): """ Rebuild the whole xlog database guessing it from the archive content. :param bool silent: Supress output logs if ``True``. """ from os.path import isdir, join if not silent: output.info("Rebuilding xlogdb for server %s", self.config.name) # create xlogdb directory and xlogdb file if they do not exist yet if not os.path.exists(self.xlogdb_file_path): if not os.path.exists(self.xlogdb_directory): os.makedirs(self.xlogdb_directory) open(self.xlogdb_file_path, mode="a").close() # the xlogdb file was renamed in Barman 3.13. In case of a recent # migration, also attempt to delete the old file to clean up leftovers try: os.unlink(os.path.join(self.config.wals_directory, "xlog.db")) except FileNotFoundError: pass root = self.config.wals_directory comp_manager = self.backup_manager.compression_manager wal_count = label_count = history_count = 0 # lock the xlogdb as we are about replacing it completely with self.xlogdb("w") as fxlogdb: xlogdb_dir = os.path.dirname(fxlogdb.name) with tempfile.TemporaryFile(mode="w+", dir=xlogdb_dir) as fxlogdb_new: for name in sorted(os.listdir(root)): # ignore the xlogdb and its lockfile if name.startswith(self.xlogdb_file_name): continue fullname = join(root, name) if isdir(fullname): # all relevant files are in subdirectories hash_dir = fullname for wal_name in sorted(os.listdir(hash_dir)): fullname = join(hash_dir, wal_name) if isdir(fullname): _logger.warning( "unexpected directory " "rebuilding the wal database: %s", fullname, ) else: if xlog.is_wal_file(fullname): wal_count += 1 elif xlog.is_backup_file(fullname): label_count += 1 elif fullname.endswith(".tmp"): _logger.warning( "temporary file found " "rebuilding the wal database: %s", fullname, ) continue else: _logger.warning( "unexpected file " "rebuilding the wal database: %s", fullname, ) continue wal_info = comp_manager.get_wal_file_info(fullname) fxlogdb_new.write(wal_info.to_xlogdb_line()) else: # only history files are here if xlog.is_history_file(fullname): history_count += 1 wal_info = comp_manager.get_wal_file_info(fullname) fxlogdb_new.write(wal_info.to_xlogdb_line()) else: _logger.warning( "unexpected file rebuilding the wal database: %s", fullname, ) fxlogdb_new.flush() fxlogdb_new.seek(0) fxlogdb.seek(0) shutil.copyfileobj(fxlogdb_new, fxlogdb) fxlogdb.truncate() if not silent: output.info( "Done rebuilding xlogdb for server %s " "(history: %s, backup_labels: %s, wal_file: %s)", self.config.name, history_count, label_count, wal_count, ) def get_backup_ext_info(self, backup_info): """ Return a dictionary containing all available information about a backup The result is equivalent to the sum of information from * BackupInfo object * the Server.get_wal_info() return value * the context in the catalog (if available) * the retention policy status * the copy statistics * the incremental backups information * extra backup.info properties :param backup_info: the target backup :rtype dict: all information about a backup """ backup_ext_info = backup_info.to_dict() if backup_info.status in BackupInfo.STATUS_COPY_DONE: try: previous_backup = self.backup_manager.get_previous_backup( backup_ext_info["backup_id"] ) next_backup = self.backup_manager.get_next_backup( backup_ext_info["backup_id"] ) backup_ext_info["previous_backup_id"] = None backup_ext_info["next_backup_id"] = None if previous_backup: backup_ext_info["previous_backup_id"] = previous_backup.backup_id if next_backup: backup_ext_info["next_backup_id"] = next_backup.backup_id except UnknownBackupIdException: # no next_backup_id and previous_backup_id items # means "Not available" pass backup_ext_info.update(self.get_wal_info(backup_info)) backup_ext_info["retention_policy_status"] = None if self.enforce_retention_policies: policy = self.config.retention_policy backup_ext_info["retention_policy_status"] = policy.backup_status( backup_info.backup_id ) # Check any child timeline exists children_timelines = self.get_children_timelines( backup_ext_info["timeline"], forked_after=backup_info.end_xlog ) backup_ext_info["children_timelines"] = children_timelines # If copy statistics are available copy_stats = backup_ext_info.get("copy_stats") if copy_stats: analysis_time = copy_stats.get("analysis_time", 0) if analysis_time >= 1: backup_ext_info["analysis_time"] = analysis_time copy_time = copy_stats.get("copy_time", 0) if copy_time > 0: backup_ext_info["copy_time"] = copy_time dedup_size = backup_ext_info.get("deduplicated_size", 0) if dedup_size > 0: estimated_throughput = dedup_size / copy_time backup_ext_info["estimated_throughput"] = estimated_throughput number_of_workers = copy_stats.get("number_of_workers", 1) if number_of_workers > 1: backup_ext_info["number_of_workers"] = number_of_workers backup_chain = [backup for backup in backup_info.walk_to_root()] chain_size = len(backup_chain) # last is root root_backup_info = backup_chain[-1] # "Incremental" backups backup_ext_info["root_backup_id"] = root_backup_info.backup_id backup_ext_info["chain_size"] = chain_size # Properties added to the result dictionary backup_ext_info["backup_type"] = backup_info.backup_type backup_ext_info["deduplication_ratio"] = backup_info.deduplication_ratio # A new field "cluster_size" was added to backup.info to be # able to calculate the resource saved by "incremental" backups # introduced in Postgres 17. # To keep backward compatibility between versions, barman relies # on two possible values to calculate "est_dedup_size", # "size" being used for older versions when "cluster_size" # is non existent (None). backup_ext_info["est_dedup_size"] = ( backup_ext_info["cluster_size"] or backup_ext_info["size"] ) * backup_ext_info["deduplication_ratio"] return backup_ext_info def show_backup(self, backup_info): """ Output all available information about a backup :param backup_info: the target backup """ try: backup_ext_info = self.get_backup_ext_info(backup_info) output.result("show_backup", backup_ext_info) except BadXlogSegmentName as e: output.error( "invalid xlog segment name %r\n" 'HINT: Please run "barman rebuild-xlogdb %s" ' "to solve this issue", force_str(e), self.config.name, ) output.close_and_exit() @staticmethod def _build_path(path_prefix=None): """ If a path_prefix is provided build a string suitable to be used in PATH environment variable by joining the path_prefix with the current content of PATH environment variable. If the `path_prefix` is None returns None. :rtype: str|None """ if not path_prefix: return None sys_path = os.environ.get("PATH") return "%s%s%s" % (path_prefix, os.pathsep, sys_path) def kill(self, task, fail_if_not_present=True): """ Given the name of a barman sub-task type, attempts to stop all the processes :param string task: The task we want to stop :param bool fail_if_not_present: Display an error when the process is not present (default: True) """ process_list = self.process_manager.list(task) for process in process_list: if self.process_manager.kill(process): output.info("Stopped process %s(%s)", process.task, process.pid) return else: output.error( "Cannot terminate process %s(%s)", process.task, process.pid ) return if fail_if_not_present: output.error( "Termination of %s failed: no such process for server %s", task, self.config.name, ) def switch_wal(self, force=False, archive=None, archive_timeout=None): """ Execute the switch-wal command on the target server """ closed_wal = None try: if force: # If called with force, execute a checkpoint before the # switch_wal command _logger.info("Force a CHECKPOINT before pg_switch_wal()") self.postgres.checkpoint() # Perform the switch_wal. expect a WAL name only if the switch # has been successfully executed, False otherwise. closed_wal = self.postgres.switch_wal() if closed_wal is None: # Something went wrong during the execution of the # pg_switch_wal command output.error( "Unable to perform pg_switch_wal " "for server '%s'." % self.config.name ) return if closed_wal: # The switch_wal command have been executed successfully output.info( "The WAL file %s has been closed on server '%s'" % (closed_wal, self.config.name) ) else: # Is not necessary to perform a switch_wal output.info("No switch required for server '%s'" % self.config.name) except PostgresIsInRecovery: output.info( "No switch performed because server '%s' " "is a standby." % self.config.name ) except PostgresCheckpointPrivilegesRequired: # Superuser rights are required to perform the switch_wal output.error( "Barman switch-wal --force requires superuser rights or " "the 'pg_checkpoint' role" ) return # If the user has asked to wait for a WAL file to be archived, # wait until a new WAL file has been found # or the timeout has expired if archive: self.wait_for_wal(closed_wal, archive_timeout) def wait_for_wal(self, wal_file=None, archive_timeout=None): """ Wait for a WAL file to be archived on the server :param str|None wal_file: Name of the WAL file, or None if we should just wait for a new WAL file to be archived :param int|None archive_timeout: Timeout in seconds """ max_msg = "" if archive_timeout: max_msg = " (max: %s seconds)" % archive_timeout initial_wals = dict() if not wal_file: wals = self.backup_manager.get_latest_archived_wals_info() initial_wals = dict([(tli, wals[tli].name) for tli in wals]) if wal_file: output.info( "Waiting for the WAL file %s from server '%s'%s", wal_file, self.config.name, max_msg, ) else: output.info( "Waiting for a WAL file from server '%s' to be archived%s", self.config.name, max_msg, ) # Wait for a new file until end_time or forever if no archive_timeout end_time = None if archive_timeout: end_time = time.time() + archive_timeout while not end_time or time.time() < end_time: self.archive_wal(verbose=False) # Finish if the closed wal file is in the archive. if wal_file: if os.path.exists(self.get_wal_full_path(wal_file)): break else: # Check if any new file has been archived, on any timeline wals = self.backup_manager.get_latest_archived_wals_info() current_wals = dict([(tli, wals[tli].name) for tli in wals]) if current_wals != initial_wals: break # sleep a bit before retrying time.sleep(0.1) else: if wal_file: output.error( "The WAL file %s has not been received in %s seconds", wal_file, archive_timeout, ) else: output.info( "A WAL file has not been received in %s seconds", archive_timeout ) def replication_status(self, target="all"): """ Implements the 'replication-status' command. """ if target == "hot-standby": client_type = PostgreSQLConnection.STANDBY elif target == "wal-streamer": client_type = PostgreSQLConnection.WALSTREAMER else: client_type = PostgreSQLConnection.ANY_STREAMING_CLIENT try: standby_info = self.postgres.get_replication_stats(client_type) if standby_info is None: output.error("Unable to connect to server %s" % self.config.name) else: output.result( "replication_status", self.config.name, target, self.postgres.current_xlog_location, standby_info, ) except PostgresUnsupportedFeature as e: output.info(" Requires PostgreSQL %s or higher", e) except PostgresObsoleteFeature as e: output.info(" Requires PostgreSQL lower than %s", e) except PostgresSuperuserRequired: output.info(" Requires superuser rights") def get_children_timelines(self, tli, forked_after=None): """ Get a list of the children of the passed timeline :param int tli: Id of the timeline to check :param str forked_after: XLog location after which the timeline must have been created :return List[xlog.HistoryFileData]: the list of timelines that have the timeline with id 'tli' as parent """ comp_manager = self.backup_manager.compression_manager if forked_after: forked_after = xlog.parse_lsn(forked_after) children = [] # Search all the history files after the passed timeline children_tli = tli while True: children_tli += 1 history_path = os.path.join( self.config.wals_directory, "%08X.history" % children_tli ) # If the file doesn't exists, stop searching if not os.path.exists(history_path): break # Create the WalFileInfo object using the file wal_info = comp_manager.get_wal_file_info(history_path) # Get content of the file. We need to pass a compressor manager # here to handle an eventual compression of the history file history_info = xlog.decode_history_file( wal_info, self.backup_manager.compression_manager ) # Save the history only if is reachable from this timeline. for tinfo in history_info: # The history file contains the full genealogy # but we keep only the line with `tli` timeline as parent. if tinfo.parent_tli != tli: continue # We need to return this history info only if this timeline # has been forked after the passed LSN if forked_after and tinfo.switchpoint < forked_after: continue children.append(tinfo) return children def check_backup(self, backup_info): """ Make sure that we have all the WAL files required by a physical backup for consistency (from the first to the last WAL file) :param backup_info: the target backup """ output.debug( "Checking backup %s of server %s", backup_info.backup_id, self.config.name ) try: # No need to check a backup which is not waiting for WALs. # Doing that we could also mark as DONE backups which # were previously FAILED due to copy errors if backup_info.status == BackupInfo.FAILED: output.error("The validity of a failed backup cannot be checked") return # Take care of the backup lock. # Only one process can modify a backup a a time with ServerBackupIdLock( self.config.barman_lock_directory, self.config.name, backup_info.backup_id, ): orig_status = backup_info.status self.backup_manager.check_backup(backup_info) if orig_status == backup_info.status: output.debug( "Check finished: the status of backup %s of server %s " "remains %s", backup_info.backup_id, self.config.name, backup_info.status, ) else: output.debug( "Check finished: the status of backup %s of server %s " "changed from %s to %s", backup_info.backup_id, self.config.name, orig_status, backup_info.status, ) except LockFileBusy: # If another process is holding the backup lock, # notify the user and terminate. # This is not an error condition because it happens when # another process is validating the backup. output.info( "Another process is holding the lock for " "backup %s of server %s." % (backup_info.backup_id, self.config.name) ) return except LockFilePermissionDenied as e: # We cannot access the lockfile. # warn the user and terminate output.error("Permission denied, unable to access '%s'" % e) return def sync_status(self, last_wal=None, last_position=None): """ Return server status for sync purposes. The method outputs JSON, containing: * list of backups (with DONE status) * server configuration * last read position (in xlog.db) * last read wal * list of archived wal files If last_wal is provided, the method will discard all the wall files older than last_wal. If last_position is provided the method will try to read the xlog.db file using last_position as starting point. If the wal file at last_position does not match last_wal, read from the start and use last_wal as limit :param str|None last_wal: last read wal :param int|None last_position: last read position (in xlog.db) """ sync_status = {} wals = [] # Get all the backups using default filter for # get_available_backups method # (BackupInfo.DONE) backups = self.get_available_backups() # Retrieve the first wal associated to a backup, it will be useful # to filter our eventual WAL too old to be useful first_useful_wal = None if backups: first_useful_wal = backups[sorted(backups.keys())[0]].begin_wal # Read xlogdb file. with self.xlogdb() as fxlogdb: starting_point = self.set_sync_starting_point( fxlogdb, last_wal, last_position ) check_first_wal = starting_point == 0 and last_wal is not None # The wal_info and line variables are used after the loop. # We initialize them here to avoid errors with an empty xlogdb. line = None wal_info = None for line in fxlogdb: # Parse the line wal_info = WalFileInfo.from_xlogdb_line(line) # Check if user is requesting data that is not available. # TODO: probably the check should be something like # TODO: last_wal + 1 < wal_info.name if check_first_wal: if last_wal < wal_info.name: raise SyncError( "last_wal '%s' is older than the first" " available wal '%s'" % (last_wal, wal_info.name) ) else: check_first_wal = False # If last_wal is provided, discard any line older than last_wal if last_wal: if wal_info.name <= last_wal: continue # Else don't return any WAL older than first available backup elif first_useful_wal and wal_info.name < first_useful_wal: continue wals.append(wal_info) if wal_info is not None: # Check if user is requesting data that is not available. if last_wal is not None and last_wal > wal_info.name: raise SyncError( "last_wal '%s' is newer than the last available wal " " '%s'" % (last_wal, wal_info.name) ) # Set last_position with the current position - len(last_line) # (returning the beginning of the last line) sync_status["last_position"] = fxlogdb.tell() - len(line) # Set the name of the last wal of the file sync_status["last_name"] = wal_info.name else: # we started over sync_status["last_position"] = 0 sync_status["last_name"] = "" sync_status["backups"] = backups sync_status["wals"] = wals sync_status["version"] = barman.__version__ sync_status["config"] = self.config json.dump(sync_status, sys.stdout, cls=BarmanEncoder, indent=4) def sync_cron(self, keep_descriptors): """ Manage synchronisation operations between passive node and master node. The method recover information from the remote master server, evaluate if synchronisation with the master is required and spawn barman sub processes, syncing backups and WAL files :param bool keep_descriptors: whether to keep subprocess descriptors attached to this process. """ # Recover information from primary node sync_wal_info = self.load_sync_wals_info() # Use last_wal and last_position for the remote call to the # master server try: remote_info = self.primary_node_info( sync_wal_info.last_wal, sync_wal_info.last_position ) except SyncError as exc: output.error( "Failed to retrieve the primary node status: %s" % force_str(exc) ) return # Perform backup synchronisation if remote_info["backups"]: # Get the list of backups that need to be synced # with the local server local_backup_list = self.get_available_backups() # Subtract the list of the already # synchronised backups from the remote backup lists, # obtaining the list of backups still requiring synchronisation sync_backup_list = set(remote_info["backups"]) - set(local_backup_list) else: # No backup to synchronisation required output.info( "No backup synchronisation required for server %s", self.config.name, log=False, ) sync_backup_list = [] for backup_id in sorted(sync_backup_list): # Check if this backup_id needs to be synchronized by spawning a # sync-backup process. # The same set of checks will be executed by the spawned process. # This "double check" is necessary because we don't want the cron # to spawn unnecessary processes. try: local_backup_info = self.get_backup(backup_id) self.check_sync_required(backup_id, remote_info, local_backup_info) except SyncError as e: # It means that neither the local backup # nor the remote one exist. # This should not happen here. output.exception("Unexpected state: %s", e) break except SyncToBeDeleted: # The backup does not exist on primary server # and is FAILED here. # It must be removed by the sync-backup process. pass except SyncNothingToDo: # It could mean that the local backup is in DONE state or # that it is obsolete according to # the local retention policies. # In both cases, continue with the next backup. continue # Now that we are sure that a backup-sync subprocess is necessary, # we need to acquire the backup lock, to be sure that # there aren't other processes synchronising the backup. # If cannot acquire the lock, another synchronisation process # is running, so we give up. try: with ServerBackupSyncLock( self.config.barman_lock_directory, self.config.name, backup_id ): output.info( "Starting copy of backup %s for server %s", backup_id, self.config.name, ) except LockFileBusy: output.info( "A synchronisation process for backup %s" " on server %s is already in progress", backup_id, self.config.name, log=False, ) # Stop processing this server break # Init a Barman sub-process object sub_process = BarmanSubProcess( subcommand="sync-backup", config=barman.__config__.config_file, args=[self.config.name, backup_id], keep_descriptors=keep_descriptors, ) # Launch the sub-process sub_process.execute() # Stop processing this server break # Perform WAL synchronisation if remote_info["wals"]: # We need to acquire a sync-wal lock, to be sure that # there aren't other processes synchronising the WAL files. # If cannot acquire the lock, another synchronisation process # is running, so we give up. try: with ServerWalSyncLock( self.config.barman_lock_directory, self.config.name, ): output.info( "Started copy of WAL files for server %s", self.config.name ) except LockFileBusy: output.info( "WAL synchronisation already running for server %s", self.config.name, log=False, ) return # Init a Barman sub-process object sub_process = BarmanSubProcess( subcommand="sync-wals", config=barman.__config__.config_file, args=[self.config.name], keep_descriptors=keep_descriptors, ) # Launch the sub-process sub_process.execute() else: # no WAL synchronisation is required output.info( "No WAL synchronisation required for server %s", self.config.name, log=False, ) def check_sync_required(self, backup_name, primary_info, local_backup_info): """ Check if it is necessary to sync a backup. If the backup is present on the Primary node: * if it does not exist locally: continue (synchronise it) * if it exists and is DONE locally: raise SyncNothingToDo (nothing to do) * if it exists and is FAILED locally: continue (try to recover it) If the backup is not present on the Primary node: * if it does not exist locally: raise SyncError (wrong call) * if it exists and is DONE locally: raise SyncNothingToDo (nothing to do) * if it exists and is FAILED locally: raise SyncToBeDeleted (remove it) If a backup needs to be synchronised but it is obsolete according to local retention policies, raise SyncNothingToDo, else return to the caller. :param str backup_name: str name of the backup to sync :param dict primary_info: dict containing the Primary node status :param barman.infofile.BackupInfo local_backup_info: BackupInfo object representing the current backup state :raise SyncError: There is an error in the user request :raise SyncNothingToDo: Nothing to do for this request :raise SyncToBeDeleted: Backup is not recoverable and must be deleted """ backups = primary_info["backups"] # Backup not present on Primary node, and not present # locally. Raise exception. if backup_name not in backups and local_backup_info is None: raise SyncError( "Backup %s is absent on %s server" % (backup_name, self.config.name) ) # Backup not present on Primary node, but is # present locally with status FAILED: backup incomplete. # Remove the backup and warn the user if ( backup_name not in backups and local_backup_info is not None and local_backup_info.status == BackupInfo.FAILED ): raise SyncToBeDeleted( "Backup %s is absent on %s server and is incomplete locally" % (backup_name, self.config.name) ) # Backup not present on Primary node, but is # present locally with status DONE. Sync complete, local only. if ( backup_name not in backups and local_backup_info is not None and local_backup_info.status == BackupInfo.DONE ): raise SyncNothingToDo( "Backup %s is absent on %s server, but present locally " "(local copy only)" % (backup_name, self.config.name) ) # Backup present on Primary node, and present locally # with status DONE. Sync complete. if ( backup_name in backups and local_backup_info is not None and local_backup_info.status == BackupInfo.DONE ): raise SyncNothingToDo( "Backup %s is already synced with" " %s server" % (backup_name, self.config.name) ) # Retention Policy: if the local server has a Retention policy, # check that the remote backup is not obsolete. enforce_retention_policies = self.enforce_retention_policies retention_policy_mode = self.config.retention_policy_mode if enforce_retention_policies and retention_policy_mode == "auto": # All the checks regarding retention policies are in # this boolean method. if self.is_backup_locally_obsolete(backup_name, backups): # The remote backup is obsolete according to # local retention policies. # Nothing to do. raise SyncNothingToDo( "Remote backup %s/%s is obsolete for " "local retention policies." % (primary_info["config"]["name"], backup_name) ) def load_sync_wals_info(self): """ Load the content of SYNC_WALS_INFO_FILE for the given server :return collections.namedtuple: last read wal and position information """ sync_wals_info_file = os.path.join( self.config.wals_directory, SYNC_WALS_INFO_FILE ) if not os.path.exists(sync_wals_info_file): return SyncWalInfo(None, None) try: with open(sync_wals_info_file) as f: return SyncWalInfo._make(f.readline().split("\t")) except (OSError, IOError) as e: raise SyncError( "Cannot open %s file for server %s: %s" % (SYNC_WALS_INFO_FILE, self.config.name, e) ) def primary_node_info(self, last_wal=None, last_position=None): """ Invoke sync-info directly on the specified primary node The method issues a call to the sync-info method on the primary node through an SSH connection :param barman.server.Server self: the Server object :param str|None last_wal: last read wal :param int|None last_position: last read position (in xlog.db) :raise SyncError: if the ssh command fails """ # First we need to check if the server is in passive mode _logger.debug( "primary sync-info(%s, %s, %s)", self.config.name, last_wal, last_position ) if not self.passive_node: raise SyncError("server %s is not passive" % self.config.name) # Issue a call to 'barman sync-info' to the primary node, # using primary_ssh_command option to establish an # SSH connection. remote_command = Command( cmd=self.config.primary_ssh_command, shell=True, check=True, path=self.path ) # We run it in a loop to retry when the master issues error. while True: try: # Include the config path as an option if configured for this server if self.config.forward_config_path: base_cmd = "barman -c %s sync-info" % barman.__config__.config_file else: base_cmd = "barman sync-info" # Build the command string cmd_str = "%s %s" % (base_cmd, self.config.name) # If necessary we add last_wal and last_position # to the command string if last_wal is not None: cmd_str += " %s " % last_wal if last_position is not None: cmd_str += " %s " % last_position # Then issue the command remote_command(cmd_str) # All good, exit the retry loop with 'break' break except CommandFailedException as exc: # In case we requested synchronisation with a last WAL info, # we try again requesting the full current status, but only if # exit code is 1. A different exit code means that # the error is not from Barman (i.e. ssh failure) if exc.args[0]["ret"] == 1 and last_wal is not None: last_wal = None last_position = None output.warning( "sync-info is out of sync. " "Self-recovery procedure started: " "requesting full synchronisation from " "primary server %s" % self.config.name ) continue # Wrap the CommandFailed exception with a SyncError # for custom message and logging. raise SyncError( "sync-info execution on remote " "primary server %s failed: %s" % (self.config.name, exc.args[0]["err"]) ) # Save the result on disk primary_info_file = os.path.join( self.config.backup_directory, PRIMARY_INFO_FILE ) # parse the json output remote_info = json.loads(remote_command.out) try: # TODO: rename the method to make it public # noinspection PyProtectedMember self._make_directories() # Save remote info to disk # We do not use a LockFile here. Instead we write all data # in a new file (adding '.tmp' extension) then we rename it # replacing the old one. # It works while the renaming is an atomic operation # (this is a POSIX requirement) primary_info_file_tmp = primary_info_file + ".tmp" with open(primary_info_file_tmp, "w") as info_file: info_file.write(remote_command.out) os.rename(primary_info_file_tmp, primary_info_file) except (OSError, IOError) as e: # Wrap file access exceptions using SyncError raise SyncError( "Cannot open %s file for server %s: %s" % (PRIMARY_INFO_FILE, self.config.name, e) ) return remote_info def is_backup_locally_obsolete(self, backup_name, remote_backups): """ Check if a remote backup is obsolete according with the local retention policies. :param barman.server.Server self: Server object :param str backup_name: str name of the backup to sync :param dict remote_backups: dict containing the Primary node status :return bool: returns if the backup is obsolete or not """ # Get the local backups and add the remote backup info. This will # simulate the situation after the copy of the remote backup. local_backups = self.get_available_backups(BackupInfo.STATUS_NOT_EMPTY) backup = remote_backups[backup_name] local_backups[backup_name] = LocalBackupInfo.from_json(self, backup) # Execute the local retention policy on the modified list of backups report = self.config.retention_policy.report(source=local_backups) # If the added backup is obsolete return true. return report[backup_name] == BackupInfo.OBSOLETE def sync_backup(self, backup_name): """ Method for the synchronisation of a backup from a primary server. The Method checks that the server is passive, then if it is possible to sync with the Primary. Acquires a lock at backup level and copy the backup from the Primary node using rsync. During the sync process the backup on the Passive node is marked as SYNCING and if the sync fails (due to network failure, user interruption...) it is marked as FAILED. :param barman.server.Server self: the passive Server object to sync :param str backup_name: the name of the backup to sync. """ _logger.debug("sync_backup(%s, %s)", self.config.name, backup_name) if not self.passive_node: raise SyncError("server %s is not passive" % self.config.name) local_backup_info = self.get_backup(backup_name) # Step 1. Parse data from Primary server. _logger.info( "Synchronising with server %s backup %s: step 1/3: " "parse server information", self.config.name, backup_name, ) try: primary_info = self.load_primary_info() self.check_sync_required(backup_name, primary_info, local_backup_info) except SyncError as e: # Invocation error: exit with return code 1 output.error("%s", e) return except SyncToBeDeleted as e: # The required backup does not exist on primary, # therefore it should be deleted also on passive node, # as it's not in DONE status. output.warning("%s, purging local backup", e) self.delete_backup(local_backup_info) return except SyncNothingToDo as e: # Nothing to do. Log as info level and exit output.info("%s", e) return # If the backup is present on Primary node, and is not present at all # locally or is present with FAILED status, execute sync. # Retrieve info about the backup from PRIMARY_INFO_FILE remote_backup_info = primary_info["backups"][backup_name] remote_backup_dir = primary_info["config"]["basebackups_directory"] # Try to acquire the backup lock, if the lock is not available abort # the copy. try: with ServerBackupSyncLock( self.config.barman_lock_directory, self.config.name, backup_name ): try: backup_manager = self.backup_manager # Build a BackupInfo object local_backup_info = LocalBackupInfo.from_json( self, remote_backup_info ) local_backup_info.set_attribute("status", BackupInfo.SYNCING) local_backup_info.save() backup_manager.backup_cache_add(local_backup_info) # Activate incremental copy if requested # Calculate the safe_horizon as the start time of the older # backup involved in the copy # NOTE: safe_horizon is a tz-aware timestamp because # BackupInfo class ensures that property reuse_mode = self.config.reuse_backup safe_horizon = None reuse_dir = None if reuse_mode: prev_backup = backup_manager.get_previous_backup(backup_name) next_backup = backup_manager.get_next_backup(backup_name) # If a newer backup is present, using it is preferable # because that backup will remain valid longer if next_backup: safe_horizon = local_backup_info.begin_time reuse_dir = next_backup.get_basebackup_directory() elif prev_backup: safe_horizon = prev_backup.begin_time reuse_dir = prev_backup.get_basebackup_directory() else: reuse_mode = None # Try to copy from the Primary node the backup using # the copy controller. copy_controller = RsyncCopyController( ssh_command=self.config.primary_ssh_command, network_compression=self.config.network_compression, path=self.path, reuse_backup=reuse_mode, safe_horizon=safe_horizon, retry_times=self.config.basebackup_retry_times, retry_sleep=self.config.basebackup_retry_sleep, workers=self.config.parallel_jobs, workers_start_batch_period=self.config.parallel_jobs_start_batch_period, workers_start_batch_size=self.config.parallel_jobs_start_batch_size, ) # Exclude primary Barman metadata and state exclude_and_protect = ["/backup.info", "/.backup.lock"] # Exclude any tablespace symlinks created by pg_basebackup if local_backup_info.tablespaces is not None: for tablespace in local_backup_info.tablespaces: exclude_and_protect += [ "/data/pg_tblspc/%s" % tablespace.oid ] copy_controller.add_directory( "basebackup", ":%s/%s/" % (remote_backup_dir, backup_name), local_backup_info.get_basebackup_directory(), exclude_and_protect=exclude_and_protect, bwlimit=self.config.bandwidth_limit, reuse=reuse_dir, item_class=RsyncCopyController.PGDATA_CLASS, ) _logger.info( "Synchronising with server %s backup %s: step 2/3: " "file copy", self.config.name, backup_name, ) copy_controller.copy() # Save the backup state and exit _logger.info( "Synchronising with server %s backup %s: " "step 3/3: finalise sync", self.config.name, backup_name, ) local_backup_info.set_attribute("status", BackupInfo.DONE) local_backup_info.save() except CommandFailedException as e: # Report rsync errors msg = "failure syncing server %s backup %s: %s" % ( self.config.name, backup_name, e, ) output.error(msg) # Set the BackupInfo status to FAILED local_backup_info.set_attribute("status", BackupInfo.FAILED) local_backup_info.set_attribute("error", msg) local_backup_info.save() return # Catch KeyboardInterrupt (Ctrl+c) and all the exceptions except BaseException as e: msg_lines = force_str(e).strip().splitlines() if local_backup_info: # Use only the first line of exception message # in local_backup_info error field local_backup_info.set_attribute("status", BackupInfo.FAILED) # If the exception has no attached message # use the raw type name if not msg_lines: msg_lines = [type(e).__name__] local_backup_info.set_attribute( "error", "failure syncing server %s backup %s: %s" % (self.config.name, backup_name, msg_lines[0]), ) local_backup_info.save() output.error( "Backup failed syncing with %s: %s\n%s", self.config.name, msg_lines[0], "\n".join(msg_lines[1:]), ) except LockFileException: output.error( "Another synchronisation process for backup %s " "of server %s is already running.", backup_name, self.config.name, ) def sync_wals(self): """ Method for the synchronisation of WAL files on the passive node, by copying them from the primary server. The method checks if the server is passive, then tries to acquire a sync-wal lock. Recovers the id of the last locally archived WAL file from the status file ($wals_directory/sync-wals.info). Reads the primary.info file and parses it, then obtains the list of WAL files that have not yet been synchronised with the master. Rsync is used for file synchronisation with the primary server. Once the copy is finished, acquires a lock on xlog.db, updates it then releases the lock. Before exiting, the method updates the last_wal and last_position fields in the sync-wals.info file. :param barman.server.Server self: the Server object to synchronise """ _logger.debug("sync_wals(%s)", self.config.name) if not self.passive_node: raise SyncError("server %s is not passive" % self.config.name) # Try to acquire the sync-wal lock if the lock is not available, # abort the sync-wal operation try: with ServerWalSyncLock( self.config.barman_lock_directory, self.config.name, ): try: # Need to load data from status files: primary.info # and sync-wals.info sync_wals_info = self.load_sync_wals_info() primary_info = self.load_primary_info() # We want to exit if the compression on master is different # from the one on the local server if primary_info["config"]["compression"] != self.config.compression: raise SyncError( "Compression method on server %s " "(%s) does not match local " "compression method (%s) " % ( self.config.name, primary_info["config"]["compression"], self.config.compression, ) ) # If the first WAL that needs to be copied is older # than the begin WAL of the first locally available backup, # synchronisation is skipped. This means that we need # to copy a WAL file which won't be associated to any local # backup. Consider the following scenarios: # # bw: indicates the begin WAL of the first backup # sw: the first WAL to be sync-ed # # The following examples use truncated names for WAL files # (e.g. 1 instead of 000000010000000000000001) # # Case 1: bw = 10, sw = 9 - SKIP and wait for backup # Case 2: bw = 10, sw = 10 - SYNC # Case 3: bw = 10, sw = 15 - SYNC # # Search for the first WAL file (skip history, # backup and partial files) first_remote_wal = None for wal in primary_info["wals"]: if xlog.is_wal_file(wal["name"]): first_remote_wal = wal["name"] break first_backup_id = self.get_first_backup_id() first_backup = ( self.get_backup(first_backup_id) if first_backup_id else None ) # Also if there are not any backups on the local server # no wal synchronisation is required if not first_backup: output.warning( "No base backup for server %s" % self.config.name ) return if first_backup.begin_wal > first_remote_wal: output.warning( "Skipping WAL synchronisation for " "server %s: no available local backup " "for %s" % (self.config.name, first_remote_wal) ) return local_wals = [] wal_file_paths = [] for wal in primary_info["wals"]: # filter all the WALs that are smaller # or equal to the name of the latest synchronised WAL if ( sync_wals_info.last_wal and wal["name"] <= sync_wals_info.last_wal ): continue # Generate WalFileInfo Objects using remote WAL metas. # This list will be used for the update of the xlog.db wal_info_file = WalFileInfo(**wal) local_wals.append(wal_info_file) wal_file_paths.append(wal_info_file.relpath()) # Rsync Options: # recursive: recursive copy of subdirectories # perms: preserve permissions on synced files # times: preserve modification timestamps during # synchronisation # protect-args: force rsync to preserve the integrity of # rsync command arguments and filename. # inplace: for inplace file substitution # and update of files rsync = Rsync( args=[ "--recursive", "--perms", "--times", "--protect-args", "--inplace", ], ssh=self.config.primary_ssh_command, bwlimit=self.config.bandwidth_limit, allowed_retval=(0,), network_compression=self.config.network_compression, path=self.path, ) # Source and destination of the rsync operations src = ":%s/" % primary_info["config"]["wals_directory"] dest = "%s/" % self.config.wals_directory # Perform the rsync copy using the list of relative paths # obtained from the primary.info file rsync.from_file_list(wal_file_paths, src, dest) # If everything is synced without errors, # update xlog.db using the list of WalFileInfo object with self.xlogdb("a") as fxlogdb: for wal_info in local_wals: fxlogdb.write(wal_info.to_xlogdb_line()) # We need to update the sync-wals.info file with the latest # synchronised WAL and the latest read position. self.write_sync_wals_info_file(primary_info) except CommandFailedException as e: msg = "WAL synchronisation for server %s failed: %s" % ( self.config.name, e, ) output.error(msg) return except BaseException as e: msg_lines = force_str(e).strip().splitlines() # Use only the first line of exception message # If the exception has no attached message # use the raw type name if not msg_lines: msg_lines = [type(e).__name__] output.error( "WAL synchronisation for server %s failed with: %s\n%s", self.config.name, msg_lines[0], "\n".join(msg_lines[1:]), ) except LockFileException: output.error( "Another sync-wal operation is running for server %s ", self.config.name, ) @staticmethod def set_sync_starting_point(xlogdb_file, last_wal, last_position): """ Check if the xlog.db file has changed between two requests from the client and set the start point for reading the file :param file xlogdb_file: an open and readable xlog.db file object :param str|None last_wal: last read name :param int|None last_position: last read position :return int: the position has been set """ # If last_position is None start reading from the beginning of the file position = int(last_position) if last_position is not None else 0 # Seek to required position xlogdb_file.seek(position) # Read 24 char (the size of a wal name) wal_name = xlogdb_file.read(24) # If the WAL name is the requested one start from last_position if wal_name == last_wal: # Return to the line start xlogdb_file.seek(position) return position # If the file has been truncated, start over xlogdb_file.seek(0) return 0 def write_sync_wals_info_file(self, primary_info): """ Write the content of SYNC_WALS_INFO_FILE on disk :param dict primary_info: """ try: with open( os.path.join(self.config.wals_directory, SYNC_WALS_INFO_FILE), "w" ) as syncfile: syncfile.write( "%s\t%s" % (primary_info["last_name"], primary_info["last_position"]) ) except (OSError, IOError): # Wrap file access exceptions using SyncError raise SyncError( "Unable to write %s file for server %s" % (SYNC_WALS_INFO_FILE, self.config.name) ) def load_primary_info(self): """ Load the content of PRIMARY_INFO_FILE for the given server :return dict: primary server information """ primary_info_file = os.path.join( self.config.backup_directory, PRIMARY_INFO_FILE ) try: with open(primary_info_file) as f: return json.load(f) except (OSError, IOError) as e: # Wrap file access exceptions using SyncError raise SyncError( "Cannot open %s file for server %s: %s" % (PRIMARY_INFO_FILE, self.config.name, e) ) def restart_processes(self): """ Restart server subprocesses. """ # Terminate the receive-wal sub-process if present self.kill("receive-wal", fail_if_not_present=False) if self.config.streaming_archiver: # Spawn the receive-wal sub-process self.background_receive_wal(keep_descriptors=False) def move_wal_file_to_errors_directory(self, src, file_name, suffix): """ Move an unknown or (mismatching) duplicate WAL file to the ``errors`` directory. .. note: The issues can happen when: * Unknown WAL file: * The asynchronous WAL archiver detects a file in the ``incoming`` or ``streaming`` directory which is not an WAL file. * Duplicate WAL file: * ``barman-wal-archive`` attempts to write a file to the ``incoming`` directory which already exists there, but with a different content. * The asynchronous WAL archiver detects a file in the ``incoming`` or ``streaming`` which already exists in the ``wals`` directory, but with a different content. :param str src: Incoming file to be moved to the ``errors`` directory. :param str file_name: Name of the incoming file. :param str suffix: String which identifies the kind of the issue. * ``duplicate``: if *src* is a (mismatching) duplicate WAL file. * ``unknown``: if *src* is not an WAL file. """ stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%SZ") error_dst = os.path.join( self.config.errors_directory, "%s.%s.%s" % (file_name, stamp, suffix), ) # TODO: cover corner case of duplication (unlikely, # but theoretically possible) try: shutil.move(src, error_dst) except IOError as e: if e.errno == errno.ENOENT: _logger.warning("%s not found" % src) barman-3.13.3/barman/fs.py0000644000175100001660000004705015002177425013455 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2013-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import logging import re import shutil import sys from abc import ABCMeta, abstractmethod from barman import output from barman.command_wrappers import Command, full_command_quote from barman.exceptions import FsOperationFailed from barman.utils import with_metaclass _logger = logging.getLogger(__name__) class UnixLocalCommand(object): """ This class is a wrapper for local calls for file system operations """ def __init__(self, path=None): # initialize a shell self.internal_cmd = Command(cmd="sh", args=["-c"], path=path) def cmd(self, cmd_name, args=[]): """ Execute a command string, escaping it, if necessary """ return self.internal_cmd(full_command_quote(cmd_name, args)) def get_last_output(self): """ Return the output and the error strings from the last executed command :rtype: tuple[str,str] """ return self.internal_cmd.out, self.internal_cmd.err def move(self, source_path, dest_path): """ Move a file from source_path to dest_path. :param str source_path: full path to the source file. :param str dest_path: full path to the destination file. :returns bool: True if the move completed successfully, False otherwise. """ _logger.debug("Moving %s to %s" % (source_path, dest_path)) mv_ret = self.cmd("mv", args=[source_path, dest_path]) if mv_ret == 0: return True else: raise FsOperationFailed("mv execution failed") def create_dir_if_not_exists(self, dir_path, mode=None): """ This method recursively creates a directory if not exists If the path exists and is not a directory raise an exception. :param str dir_path: full path for the directory :param mode str|None: Specify the mode to use for creation. Not used if the directory already exists. :returns bool: False if the directory already exists True if the directory is created. """ _logger.debug("Create directory %s if it does not exists" % dir_path) if self.check_directory_exists(dir_path): return False else: # Make parent directories if needed args = ["-p", dir_path] if mode is not None: args.extend(["-m", mode]) mkdir_ret = self.cmd("mkdir", args=args) if mkdir_ret == 0: return True else: raise FsOperationFailed("mkdir execution failed") def delete_if_exists(self, path): """ This method check for the existence of a path. If it exists, then is removed using a rm -fr command, and returns True. If the command fails an exception is raised. If the path does not exists returns False :param path the full path for the directory """ _logger.debug("Delete path %s if exists" % path) exists = self.exists(path, False) if exists: rm_ret = self.cmd("rm", args=["-fr", path]) if rm_ret == 0: return True else: raise FsOperationFailed("rm execution failed") else: return False def check_directory_exists(self, dir_path): """ Check for the existence of a directory in path. if the directory exists returns true. if the directory does not exists returns false. if exists a file and is not a directory raises an exception :param dir_path full path for the directory """ _logger.debug("Check if directory %s exists" % dir_path) exists = self.exists(dir_path) if exists: is_dir = self.cmd("test", args=["-d", dir_path]) if is_dir != 0: raise FsOperationFailed( "A file with the same name exists, but is not a directory" ) else: return True else: return False def get_file_mode(self, path): """ Should check that :param dir_path: :param mode: :return: mode """ if not self.exists(path): raise FsOperationFailed("Following path does not exist: %s" % path) args = ["-c", "%a", path] if self.is_osx(): print("is osx") args = ["-f", "%Lp", path] cmd_ret = self.cmd("stat", args=args) if cmd_ret != 0: raise FsOperationFailed( "Failed to get file mode for %s: %s" % (path, self.internal_cmd.err) ) return self.internal_cmd.out.strip() def is_osx(self): """ Identify whether is is a Linux or Darwin system :return: True is it is osx os """ self.cmd("uname", args=["-s"]) if self.internal_cmd.out.strip() == "Darwin": return True return False def validate_file_mode(self, path, mode): """ Validate the file or dir has the expected mode. Raises an exception otherwise. :param path: str :param mode: str (700, 750, ...) :return: """ path_mode = self.get_file_mode(path) if path_mode != mode: FsOperationFailed( "Following file %s does not have expected access right %s. Got %s instead" % (path, mode, path_mode) ) def check_write_permission(self, dir_path): """ check write permission for barman on a given path. Creates a hidden file using touch, then remove the file. returns true if the file is written and removed without problems raise exception if the creation fails. raise exception if the removal fails. :param dir_path full dir_path for the directory to check """ _logger.debug("Check if directory %s is writable" % dir_path) exists = self.exists(dir_path) if exists: is_dir = self.cmd("test", args=["-d", dir_path]) if is_dir == 0: can_write = self.cmd( "touch", args=["%s/.barman_write_check" % dir_path] ) if can_write == 0: can_remove = self.cmd( "rm", args=["%s/.barman_write_check" % dir_path] ) if can_remove == 0: return True else: raise FsOperationFailed("Unable to remove file") else: raise FsOperationFailed("Unable to create write check file") else: raise FsOperationFailed("%s is not a directory" % dir_path) else: raise FsOperationFailed("%s does not exists" % dir_path) def create_symbolic_link(self, src, dst): """ Create a symlink pointing to src named dst. Check src exists, if so, checks that destination does not exists. if src is an invalid folder, raises an exception. if dst already exists, raises an exception. if ln -s command fails raises an exception :param src full path to the source of the symlink :param dst full path for the destination of the symlink """ _logger.debug("Create symbolic link %s -> %s" % (dst, src)) exists = self.exists(src) if exists: exists_dst = self.exists(dst) if not exists_dst: link = self.cmd("ln", args=["-s", src, dst]) if link == 0: return True else: raise FsOperationFailed("ln command failed") else: raise FsOperationFailed("ln destination already exists") else: raise FsOperationFailed("ln source does not exists") def get_system_info(self): """ Gather important system information for 'barman diagnose' command """ result = {} # self.internal_cmd.out can be None. The str() call will ensure it # will be translated to a literal 'None' release = "" if self.cmd("lsb_release", args=["-a"]) == 0: release = self.internal_cmd.out.rstrip() elif self.exists("/etc/lsb-release"): self.cmd("cat", args=["/etc/lsb-release"]) release = "Ubuntu Linux %s" % self.internal_cmd.out.rstrip() elif self.exists("/etc/debian_version"): self.cmd("cat", args=["/etc/debian_version"]) release = "Debian GNU/Linux %s" % self.internal_cmd.out.rstrip() elif self.exists("/etc/redhat-release"): self.cmd("cat", args=["/etc/redhat-release"]) release = "RedHat Linux %s" % self.internal_cmd.out.rstrip() elif self.cmd("sw_vers") == 0: release = self.internal_cmd.out.rstrip() result["release"] = release self.cmd("uname", args=["-a"]) result["kernel_ver"] = self.internal_cmd.out.rstrip() result["python_ver"] = "Python %s.%s.%s" % ( sys.version_info.major, sys.version_info.minor, sys.version_info.micro, ) result["python_executable"] = sys.executable self.cmd("rsync", args=["--version", "2>&1"]) try: result["rsync_ver"] = self.internal_cmd.out.splitlines(True)[0].rstrip() except IndexError: result["rsync_ver"] = "" self.cmd("ssh", args=["-V", "2>&1"]) result["ssh_ver"] = self.internal_cmd.out.rstrip() return result def get_file_content(self, path): """ Retrieve the content of a file If the file doesn't exist or isn't readable, it raises an exception. :param str path: full path to the file to read """ _logger.debug("Reading content of file %s" % path) result = self.exists(path) if not result: raise FsOperationFailed("The %s file does not exist" % path) result = self.cmd("test", args=["-r", path]) if result != 0: raise FsOperationFailed("The %s file is not readable" % path) result = self.cmd("cat", args=[path]) if result != 0: raise FsOperationFailed("Failed to execute \"cat '%s'\"" % path) return self.internal_cmd.out def exists(self, path, dereference=True): """ Check for the existence of a path. :param str path: full path to check :param bool dereference: whether dereference symlinks, defaults to True :return bool: if the file exists or not. """ _logger.debug("check for existence of: %s" % path) options = ["-e", path] if not dereference: options += ["-o", "-L", path] result = self.cmd("test", args=options) return result == 0 def ping(self): """ 'Ping' the server executing the `true` command. :return int: the true cmd result """ _logger.debug("execute the true command") result = self.cmd("true") return result def list_dir_content(self, dir_path, options=[]): """ List the contents of a given directory. :param str dir_path: the path where we want the ls to be executed :param list[str] options: a string containing the options for the ls command :return str: the ls cmd output """ _logger.debug("list the content of a directory") ls_options = [] if options: ls_options += options ls_options.append(dir_path) self.cmd("ls", args=ls_options) return self.internal_cmd.out def findmnt(self, device): """ Retrieve the mount point and mount options for the provided device. :param str device: The device for which the mount point and options should be found. :rtype: List[str|None, str|None] :return: The mount point and the mount options of the specified device or [None, None] if the device could not be found by findmnt. """ _logger.debug("finding mount point and options for device %s", device) self.cmd("findmnt", args=("-o", "TARGET,OPTIONS", "-n", device)) output = self.internal_cmd.out if output == "": # No output means we successfully ran the command but couldn't find # the mount point return [None, None] output_fields = output.split() if len(output_fields) != 2: raise FsOperationFailed( "Unexpected findmnt output: %s" % self.internal_cmd.out ) else: return output_fields class UnixRemoteCommand(UnixLocalCommand): """ This class is a wrapper for remote calls for file system operations """ # noinspection PyMissingConstructor def __init__(self, ssh_command, ssh_options=None, path=None): """ Uses the same commands as the UnixLocalCommand but the constructor is overridden and a remote shell is initialized using the ssh_command provided by the user :param str ssh_command: the ssh command provided by the user :param list[str] ssh_options: the options to be passed to SSH :param str path: the path to be used if provided, otherwise the PATH environment variable will be used """ # Ensure that ssh_option is iterable if ssh_options is None: ssh_options = [] if ssh_command is None: raise FsOperationFailed("No ssh command provided") self.internal_cmd = Command( ssh_command, args=ssh_options, path=path, shell=True ) try: ret = self.cmd("true") except OSError: raise FsOperationFailed("Unable to execute %s" % ssh_command) if ret != 0: raise FsOperationFailed( "Connection failed using '%s %s' return code %s" % (ssh_command, " ".join(ssh_options), ret) ) def unix_command_factory(remote_command=None, path=None): """ Function in charge of instantiating a Unix Command. :param remote_command: :param path: :return: UnixLocalCommand """ if remote_command: try: cmd = UnixRemoteCommand(remote_command, path=path) logging.debug("Created a UnixRemoteCommand") return cmd except FsOperationFailed: output.error( "Unable to connect to the target host using the command '%s'", remote_command, ) output.close_and_exit() else: cmd = UnixLocalCommand() logging.debug("Created a UnixLocalCommand") return cmd def path_allowed(exclude, include, path, is_dir): """ Filter files based on include/exclude lists. The rules are evaluated in steps: 1. if there are include rules and the proposed path match them, it is immediately accepted. 2. if there are exclude rules and the proposed path match them, it is immediately rejected. 3. the path is accepted. Look at the documentation for the "evaluate_path_matching_rules" function for more information about the syntax of the rules. :param list[str]|None exclude: The list of rules composing the exclude list :param list[str]|None include: The list of rules composing the include list :param str path: The patch to patch :param bool is_dir: True is the passed path is a directory :return bool: True is the patch is accepted, False otherwise """ if include and _match_path(include, path, is_dir): return True if exclude and _match_path(exclude, path, is_dir): return False return True def _match_path(rules, path, is_dir): """ Determine if a certain list of rules match a filesystem entry. The rule-checking algorithm also handles rsync-like anchoring of rules prefixed with '/'. If the rule is not anchored then it match every file whose suffix matches the rule. That means that a rule like 'a/b', will match 'a/b' and 'x/a/b' too. A rule like '/a/b' will match 'a/b' but not 'x/a/b'. If a rule ends with a slash (i.e. 'a/b/') if will be used only if the passed path is a directory. This function implements the basic wildcards. For more information about that, consult the documentation of the "translate_to_regexp" function. :param list[str] rules: match :param path: the path of the entity to match :param is_dir: True if the entity is a directory :return bool: """ for rule in rules: if rule[-1] == "/": if not is_dir: continue rule = rule[:-1] anchored = False if rule[0] == "/": rule = rule[1:] anchored = True if _wildcard_match_path(path, rule): return True if not anchored and _wildcard_match_path(path, "**/" + rule): return True return False def _wildcard_match_path(path, pattern): """ Check if the proposed shell pattern match the path passed. :param str path: :param str pattern: :rtype bool: True if it match, False otherwise """ regexp = re.compile(_translate_to_regexp(pattern)) return regexp.match(path) is not None def _translate_to_regexp(pattern): """ Translate a shell PATTERN to a regular expression. These wildcard characters you to use: - "?" to match every character - "*" to match zero or more characters, excluding "/" - "**" to match zero or more characters, including "/" There is no way to quote meta-characters. This implementation is based on the one in the Python fnmatch module :param str pattern: A string containing wildcards """ i, n = 0, len(pattern) res = "" while i < n: c = pattern[i] i = i + 1 if pattern[i - 1 :].startswith("**"): res = res + ".*" i = i + 1 elif c == "*": res = res + "[^/]*" elif c == "?": res = res + "." else: res = res + re.escape(c) return r"(?s)%s\Z" % res class PathDeletionCommand(with_metaclass(ABCMeta, object)): """ Stand-alone object that will execute delete operation on a self contained path """ @abstractmethod def delete(self): """ Will delete the actual path """ class LocalLibPathDeletionCommand(PathDeletionCommand): def __init__(self, path): """ :param path: str """ self.path = path def delete(self): shutil.rmtree(self.path, ignore_errors=True) class UnixCommandPathDeletionCommand(PathDeletionCommand): def __init__(self, path, unix_command): """ :param path: :param unix_command UnixLocalCommand: """ self.path = path self.command = unix_command def delete(self): self.command.delete_if_exists(self.path) barman-3.13.3/barman/exceptions.py0000644000175100001660000002351715002177425015230 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . class BarmanException(Exception): """ The base class of all other barman exceptions """ class ConfigurationException(BarmanException): """ Base exception for all the Configuration errors """ class CommandException(BarmanException): """ Base exception for all the errors related to the execution of a Command. """ class CompressionException(BarmanException): """ Base exception for all the errors related to the execution of a compression action. """ class PostgresException(BarmanException): """ Base exception for all the errors related to PostgreSQL. """ class BackupException(BarmanException): """ Base exception for all the errors related to the execution of a backup. """ class WALFileException(BarmanException): """ Base exception for all the errors related to WAL files. """ def __str__(self): """ Human readable string representation """ return "%s:%s" % (self.__class__.__name__, self.args[0] if self.args else None) class HookScriptException(BarmanException): """ Base exception for all the errors related to Hook Script execution. """ class LockFileException(BarmanException): """ Base exception for lock related errors """ class SyncException(BarmanException): """ Base Exception for synchronisation functions """ class DuplicateWalFile(WALFileException): """ A duplicate WAL file has been found """ class MatchingDuplicateWalFile(DuplicateWalFile): """ A duplicate WAL file has been found, but it's identical to the one we already have. """ class SshCommandException(CommandException): """ Error parsing ssh_command parameter """ class UnknownBackupIdException(BackupException): """ The searched backup_id doesn't exists """ class BackupInfoBadInitialisation(BackupException): """ Exception for a bad initialization error """ class BackupPreconditionException(BackupException): """ Exception for a backup precondition not being met """ class SnapshotBackupException(BackupException): """ Exception for snapshot backups """ class SnapshotInstanceNotFoundException(SnapshotBackupException): """ Raised when the VM instance related to a snapshot backup cannot be found """ class SyncError(SyncException): """ Synchronisation error """ class SyncNothingToDo(SyncException): """ Nothing to do during sync operations """ class SyncToBeDeleted(SyncException): """ An incomplete backup is to be deleted """ class CommandFailedException(CommandException): """ Exception representing a failed command """ class CommandMaxRetryExceeded(CommandFailedException): """ A command with retry_times > 0 has exceeded the number of available retry """ class RsyncListFilesFailure(CommandException): """ Failure parsing the output of a "rsync --list-only" command """ class DataTransferFailure(CommandException): """ Used to pass failure details from a data transfer Command """ @classmethod def from_command_error(cls, cmd, e, msg): """ This method build a DataTransferFailure exception and report the provided message to the user (both console and log file) along with the output of the failed command. :param str cmd: The command that failed the transfer :param CommandFailedException e: The exception we are handling :param str msg: a descriptive message on what we are trying to do :return DataTransferFailure: will contain the message provided in msg """ try: details = msg details += "\n%s error:\n" % cmd details += e.args[0]["out"] details += e.args[0]["err"] return cls(details) except (TypeError, NameError): # If it is not a dictionary just convert it to a string from barman.utils import force_str return cls(force_str(e.args)) class CompressionIncompatibility(CompressionException): """ Exception for compression incompatibility """ class FileNotFoundException(CompressionException): """ Exception for file not found in archive """ class FsOperationFailed(CommandException): """ Exception which represents a failed execution of a command on FS """ class LockFileBusy(LockFileException): """ Raised when a lock file is not free """ class LockFilePermissionDenied(LockFileException): """ Raised when a lock file is not accessible """ class LockFileParsingError(LockFileException): """ Raised when the content of the lockfile is unexpected """ class ConninfoException(ConfigurationException): """ Error for missing or failed parsing of the conninfo parameter (DSN) """ class PostgresConnectionError(PostgresException): """ Error connecting to the PostgreSQL server """ def __str__(self): # Returns the first line if self.args and self.args[0]: from barman.utils import force_str return force_str(self.args[0]).splitlines()[0].strip() else: return "" class PostgresConnectionLost(PostgresException): """ The Postgres connection was lost during an execution """ class PostgresAppNameError(PostgresConnectionError): """ Error setting application name with PostgreSQL server """ class PostgresSuperuserRequired(PostgresException): """ Superuser access is required """ class BackupFunctionsAccessRequired(PostgresException): """ Superuser or access to backup functions is required """ class PostgresCheckpointPrivilegesRequired(PostgresException): """ Superuser or role 'pg_checkpoint' is required """ class PostgresIsInRecovery(PostgresException): """ PostgreSQL is in recovery, so no write operations are allowed """ class PostgresUnsupportedFeature(PostgresException): """ Unsupported feature """ class PostgresObsoleteFeature(PostgresException): """ Obsolete feature, i.e. one which has been deprecated and since removed. """ class PostgresDuplicateReplicationSlot(PostgresException): """ The creation of a physical replication slot failed because the slot already exists """ class PostgresReplicationSlotsFull(PostgresException): """ The creation of a physical replication slot failed because the all the replication slots have been taken """ class PostgresReplicationSlotInUse(PostgresException): """ The drop of a physical replication slot failed because the replication slots is in use """ class PostgresInvalidReplicationSlot(PostgresException): """ Exception representing a failure during the deletion of a non existent replication slot """ class TimeoutError(CommandException): """ A timeout occurred. """ class ArchiverFailure(WALFileException): """ Exception representing a failure during the execution of the archive process """ class BadXlogSegmentName(WALFileException): """ Exception for a bad xlog name """ class BadXlogPrefix(WALFileException): """ Exception for a bad xlog prefix """ class BadHistoryFileContents(WALFileException): """ Exception for a corrupted history file """ class AbortedRetryHookScript(HookScriptException): """ Exception for handling abort of retry hook scripts """ def __init__(self, hook): """ Initialise the exception with hook script info """ self.hook = hook def __str__(self): """ String representation """ return "Abort '%s_%s' retry hook script (%s, exit code: %d)" % ( self.hook.phase, self.hook.name, self.hook.script, self.hook.exit_status, ) class RecoveryException(BarmanException): """ Exception for a recovery error """ class RecoveryPreconditionException(RecoveryException): """ Exception for a recovery precondition not being met """ class RecoveryTargetActionException(RecoveryException): """ Exception for a wrong recovery target action """ class RecoveryStandbyModeException(RecoveryException): """ Exception for a wrong recovery standby mode """ class RecoveryInvalidTargetException(RecoveryException): """ Exception for a wrong recovery target """ class UnrecoverableHookScriptError(BarmanException): """ Exception for hook script errors which mean the script should not be retried. """ class ArchivalBackupException(BarmanException): """ Exception for errors concerning archival backups. """ class WalArchiveContentError(BarmanException): """ Exception raised when unexpected content is detected in the WAL archive. """ class InvalidRetentionPolicy(BarmanException): """ Exception raised when a retention policy cannot be parsed. """ class BackupManifestException(BarmanException): """ Exception raised when there is a problem with the backup manifest. """ barman-3.13.3/barman/remote_status.py0000644000175100001660000000441315002177425015737 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ Remote Status module A Remote Status class implements a standard interface for retrieving and caching the results of a remote component (such as Postgres server, WAL archiver, etc.). It follows the Mixin pattern. """ from abc import ABCMeta, abstractmethod from barman.utils import with_metaclass class RemoteStatusMixin(with_metaclass(ABCMeta, object)): """ Abstract base class that implements remote status capabilities following the Mixin pattern. """ def __init__(self, *args, **kwargs): """ Base constructor (Mixin pattern) """ self._remote_status = None super(RemoteStatusMixin, self).__init__(*args, **kwargs) @abstractmethod def fetch_remote_status(self): """ Retrieve status information from the remote component The implementation of this method must not raise any exception in case of errors, but should set the missing values to None in the resulting dictionary. :rtype: dict[str, None|str] """ def get_remote_status(self): """ Get the status of the remote component This method does not raise any exception in case of errors, but set the missing values to None in the resulting dictionary. :rtype: dict[str, None|str] """ if self._remote_status is None: self._remote_status = self.fetch_remote_status() return self._remote_status def reset_remote_status(self): """ Reset the cached result """ self._remote_status = None barman-3.13.3/barman/utils.py0000644000175100001660000011144515002177425014205 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module contains utility functions used in Barman. """ import datetime import decimal import errno import grp import hashlib import json import logging import logging.handlers import os import pwd import re import signal import sys from abc import ABCMeta, abstractmethod from argparse import ArgumentTypeError from contextlib import contextmanager from distutils.version import Version from glob import glob from dateutil import tz from barman import lockfile, xlog from barman.exceptions import TimeoutError _logger = logging.getLogger(__name__) if sys.version_info[0] >= 3: _text_type = str _string_types = str else: _text_type = unicode # noqa _string_types = basestring # noqa RESERVED_BACKUP_IDS = ( "latest", "last", "oldest", "first", "last-failed", "latest-full", "last-full", "auto", ) def drop_privileges(user): """ Change the system user of the current python process. It will only work if called as root or as the target user. :param string user: target user :raise KeyError: if the target user doesn't exists :raise OSError: when the user change fails """ pw = pwd.getpwnam(user) if pw.pw_uid == os.getuid(): return groups = [e.gr_gid for e in grp.getgrall() if pw.pw_name in e.gr_mem] groups.append(pw.pw_gid) os.setgroups(groups) os.setgid(pw.pw_gid) os.setuid(pw.pw_uid) os.environ["HOME"] = pw.pw_dir def mkpath(directory): """ Recursively create a target directory. If the path already exists it does nothing. :param str directory: directory to be created """ if not os.path.isdir(directory): os.makedirs(directory) def configure_logging( log_file, log_level=logging.INFO, log_format="%(asctime)s %(name)s %(levelname)s: %(message)s", ): """ Configure the logging module :param str,None log_file: target file path. If None use standard error. :param int log_level: min log level to be reported in log file. Default to INFO :param str log_format: format string used for a log line. Default to "%(asctime)s %(name)s %(levelname)s: %(message)s" """ warn = None handler = logging.StreamHandler() if log_file: log_file = os.path.abspath(log_file) log_dir = os.path.dirname(log_file) try: mkpath(log_dir) handler = logging.handlers.WatchedFileHandler(log_file, encoding="utf-8") except (OSError, IOError): # fallback to standard error warn = ( "Failed opening the requested log file. " "Using standard error instead." ) formatter = logging.Formatter(log_format) handler.setFormatter(formatter) logging.root.addHandler(handler) if warn: # this will be always displayed because the default level is WARNING _logger.warn(warn) logging.root.setLevel(log_level) def parse_log_level(log_level): """ Convert a log level to its int representation as required by logging module. :param log_level: An integer or a string :return: an integer or None if an invalid argument is provided """ try: log_level_int = int(log_level) except ValueError: log_level_int = logging.getLevelName(str(log_level).upper()) if isinstance(log_level_int, int): return log_level_int return None # noinspection PyProtectedMember def get_log_levels(): """ Return a list of available log level names """ try: level_to_name = logging._levelToName except AttributeError: level_to_name = dict( [ (key, logging._levelNames[key]) for key in logging._levelNames if isinstance(key, int) ] ) for level in sorted(level_to_name): yield level_to_name[level] def pretty_size(size, unit=1024): """ This function returns a pretty representation of a size value :param int|long|float size: the number to to prettify :param int unit: 1000 or 1024 (the default) :rtype: str """ suffixes = ["B"] + [i + {1000: "B", 1024: "iB"}[unit] for i in "KMGTPEZY"] if unit == 1000: suffixes[1] = "kB" # special case kB instead of KB # cast to float to avoid losing decimals size = float(size) for suffix in suffixes: if abs(size) < unit or suffix == suffixes[-1]: if suffix == suffixes[0]: return "%d %s" % (size, suffix) else: return "%.1f %s" % (size, suffix) else: size /= unit def human_readable_timedelta(timedelta): """ Given a time interval, returns a human readable string :param timedelta: the timedelta to transform in a human readable form """ delta = abs(timedelta) # Calculate time units for the given interval time_map = { "day": int(delta.days), "hour": int(delta.seconds / 3600), "minute": int(delta.seconds / 60) % 60, "second": int(delta.seconds % 60), } # Build the resulting string time_list = [] # 'Day' part if time_map["day"] > 0: if time_map["day"] == 1: time_list.append("%s day" % time_map["day"]) else: time_list.append("%s days" % time_map["day"]) # 'Hour' part if time_map["hour"] > 0: if time_map["hour"] == 1: time_list.append("%s hour" % time_map["hour"]) else: time_list.append("%s hours" % time_map["hour"]) # 'Minute' part if time_map["minute"] > 0: if time_map["minute"] == 1: time_list.append("%s minute" % time_map["minute"]) else: time_list.append("%s minutes" % time_map["minute"]) # 'Second' part if time_map["second"] > 0: if time_map["second"] == 1: time_list.append("%s second" % time_map["second"]) else: time_list.append("%s seconds" % time_map["second"]) human = ", ".join(time_list) # Take care of timedelta when is shorter than a second if delta < datetime.timedelta(seconds=1): human = "less than one second" # If timedelta is negative append 'ago' suffix if delta != timedelta: human += " ago" return human def total_seconds(timedelta): """ Compatibility method because the total_seconds method has been introduced in Python 2.7 :param timedelta: a timedelta object :rtype: float """ if hasattr(timedelta, "total_seconds"): return timedelta.total_seconds() else: secs = (timedelta.seconds + timedelta.days * 24 * 3600) * 10**6 return (timedelta.microseconds + secs) / 10.0**6 def timestamp(datetime_value): """ Compatibility method because datetime.timestamp is not available in Python 2.7. :param datetime.datetime datetime_value: A datetime object to be converted into a timestamp. :rtype: float """ try: return datetime_value.timestamp() except AttributeError: return total_seconds( datetime_value - datetime.datetime(1970, 1, 1, tzinfo=tz.tzutc()) ) def range_fun(*args, **kwargs): """ Compatibility method required while we still support Python 2.7. This can be removed when Python 2.7 support is dropped and calling code can reference `range` directly. """ try: return xrange(*args, **kwargs) except NameError: return range(*args, **kwargs) def which(executable, path=None): """ This method is useful to find if a executable is present into the os PATH :param str executable: The name of the executable to find :param str|None path: An optional search path to override the current one. :return str|None: the path of the executable or None """ # Get the system path if needed if path is None: path = os.getenv("PATH") # If the path is None at this point we have nothing to search if path is None: return None # If executable is an absolute path, check if it exists and is executable # otherwise return failure. if os.path.isabs(executable): if os.path.exists(executable) and os.access(executable, os.X_OK): return executable else: return None # Search the requested executable in every directory present in path and # return the first occurrence that exists and is executable. for file_path in path.split(os.path.pathsep): file_path = os.path.join(file_path, executable) # If the file exists and is executable return the full path. if os.path.exists(file_path) and os.access(file_path, os.X_OK): return file_path # If no matching file is present on the system return None return None class BarmanEncoder(json.JSONEncoder): """ Custom JSON encoder used for BackupInfo encoding This encoder supports the following types: * dates and timestamps if they have a ctime() method. * objects that implement the 'to_json' method. * binary strings (python 3) """ method_list = [ "_to_json", "_datetime_to_str", "_timedelta_to_str", "_decimal_to_float", "binary_to_str", "version_to_str", ] def default(self, obj): # Go through all methods until one returns something for method in self.method_list: res = getattr(self, method)(obj) if res is not None: return res # Let the base class default method raise the TypeError return super(BarmanEncoder, self).default(obj) @staticmethod def _to_json(obj): """ # If the object implements to_json() method use it :param obj: :return: None|str """ if hasattr(obj, "to_json"): return obj.to_json() @staticmethod def _datetime_to_str(obj): """ Serialise date and datetime objects using ctime() method :param obj: :return: None|str """ if hasattr(obj, "ctime") and callable(obj.ctime): return obj.ctime() @staticmethod def _timedelta_to_str(obj): """ Serialise timedelta objects using human_readable_timedelta() :param obj: :return: None|str """ if isinstance(obj, datetime.timedelta): return human_readable_timedelta(obj) @staticmethod def _decimal_to_float(obj): """ Serialise Decimal objects using their string representation WARNING: When deserialized they will be treat as float values which have a lower precision :param obj: :return: None|float """ if isinstance(obj, decimal.Decimal): return float(obj) @staticmethod def binary_to_str(obj): """ Binary strings must be decoded before using them in an unicode string :param obj: :return: None|str """ if hasattr(obj, "decode") and callable(obj.decode): return obj.decode("utf-8", "replace") @staticmethod def version_to_str(obj): """ Manage (Loose|Strict)Version objects as strings. :param obj: :return: None|str """ if isinstance(obj, Version): return str(obj) class BarmanEncoderV2(BarmanEncoder): """ This class purpose is to replace default datetime encoding from ctime to isoformat (ISO 8601). Next major barman version will use this new format. So this class will be merged back to BarmanEncoder. """ @staticmethod def _datetime_to_str(obj): """ Try set output isoformat for this datetime. Date must have tzinfo set. :param obj: :return: None|str """ if isinstance(obj, datetime.datetime): if obj.tzinfo is None: raise ValueError( 'Got naive datetime. Expecting tzinfo for date: "{}"'.format(obj) ) return obj.isoformat() def fsync_dir(dir_path): """ Execute fsync on a directory ensuring it is synced to disk :param str dir_path: The directory to sync :raise OSError: If fail opening the directory """ dir_fd = os.open(dir_path, os.O_DIRECTORY) try: os.fsync(dir_fd) except OSError as e: # On some filesystem doing a fsync on a directory # raises an EINVAL error. Ignoring it is usually safe. if e.errno != errno.EINVAL: raise finally: os.close(dir_fd) def fsync_file(file_path): """ Execute fsync on a file ensuring it is synced to disk Returns the file stats :param str file_path: The file to sync :return: file stat :raise OSError: If something fails """ file_fd = os.open(file_path, os.O_RDONLY) file_stat = os.fstat(file_fd) try: os.fsync(file_fd) return file_stat except OSError as e: # On some filesystem doing a fsync on a O_RDONLY fd # raises an EACCES error. In that case we need to try again after # reopening as O_RDWR. if e.errno != errno.EACCES: raise finally: os.close(file_fd) file_fd = os.open(file_path, os.O_RDWR) try: os.fsync(file_fd) finally: os.close(file_fd) return file_stat def simplify_version(version_string): """ Simplify a version number by removing the patch level :param version_string: the version number to simplify :return str: the simplified version number """ if version_string is None: return None version = version_string.split(".") # If a development/beta/rc version, split out the string part unreleased = re.search(r"[^0-9.]", version[-1]) if unreleased: last_component = version.pop() number = last_component[: unreleased.start()] string = last_component[unreleased.start() :] version += [number, string] return ".".join(version[:-1]) def with_metaclass(meta, *bases): """ Function from jinja2/_compat.py. License: BSD. Create a base class with a metaclass. :param type meta: Metaclass to add to base class """ # This requires a bit of explanation: the basic idea is to make a # dummy metaclass for one level of class instantiation that replaces # itself with the actual metaclass. class Metaclass(type): def __new__(mcs, name, this_bases, d): return meta(name, bases, d) return type.__new__(Metaclass, "temporary_class", (), {}) @contextmanager def timeout(timeout_duration): """ ContextManager responsible for timing out the contained block of code after a defined time interval. """ # Define the handler for the alarm signal def handler(signum, frame): raise TimeoutError() # set the timeout handler previous_handler = signal.signal(signal.SIGALRM, handler) if previous_handler != signal.SIG_DFL and previous_handler != signal.SIG_IGN: signal.signal(signal.SIGALRM, previous_handler) raise AssertionError("Another timeout is already defined") # set the timeout duration signal.alarm(timeout_duration) try: # Execute the contained block of code yield finally: # Reset the signal signal.alarm(0) signal.signal(signal.SIGALRM, signal.SIG_DFL) def is_power_of_two(number): """ Check if a number is a power of two or not """ # Returns None if number is set to None. if number is None: return None # This is a fast method to check for a power of two. # # A power of two has this structure: 100000 (one or more zeroes) # This is the same number minus one: 011111 (composed by ones) # This is the bitwise and: 000000 # # This is true only for every power of two return number != 0 and (number & (number - 1)) == 0 def file_hash(file_path, buffer_size=1024 * 16, hash_algorithm="sha256"): """ Calculate the checksum for the provided file path with a specific hashing algorithm :param str file_path: path of the file to read :param int buffer_size: read buffer size, default 16k :param str hash_algorithm: sha256 | md5 :return str: Hexadecimal md5 string """ hash_func = hashlib.new(hash_algorithm) with open(file_path, "rb") as file_object: while 1: buf = file_object.read(buffer_size) if not buf: break hash_func.update(buf) return hash_func.hexdigest() # Might be better to use stream instead of full file content. As done in file_hash. # Might create performance issue for large files. class ChecksumAlgorithm(with_metaclass(ABCMeta)): @abstractmethod def checksum(self, value): """ Creates hash hexadecimal string from input byte :param value: Value to create checksum from :type value: byte :return: Return the digest value as a string of hexadecimal digits. :rtype: str """ def checksum_from_str(self, value, encoding="utf-8"): """ Creates hash hexadecimal string from input string :param value: Value to create checksum from :type value: str :param encoding: The encoding in which to encode the string. :type encoding: str :return: Return the digest value as a string of hexadecimal digits. :rtype: str """ return self.checksum(value.encode(encoding)) def get_name(self): return self.__class__.__name__ class SHA256(ChecksumAlgorithm): def checksum(self, value): """ Creates hash hexadecimal string from input byte :param value: Value to create checksum from :type value: byte :return: Return the digest value as a string of hexadecimal digits. :rtype: str """ sha = hashlib.sha256(value) return sha.hexdigest() def force_str(obj, encoding="utf-8", errors="replace"): """ Force any object to an unicode string. Code inspired by Django's force_text function """ # Handle the common case first for performance reasons. if issubclass(type(obj), _text_type): return obj try: if issubclass(type(obj), _string_types): obj = obj.decode(encoding, errors) else: if sys.version_info[0] >= 3: if isinstance(obj, bytes): obj = _text_type(obj, encoding, errors) else: obj = _text_type(obj) elif hasattr(obj, "__unicode__"): obj = _text_type(obj) else: obj = _text_type(bytes(obj), encoding, errors) except (UnicodeDecodeError, TypeError): if isinstance(obj, Exception): # If we get to here, the caller has passed in an Exception # subclass populated with non-ASCII bytestring data without a # working unicode method. Try to handle this without raising a # further exception by individually forcing the exception args # to unicode. obj = " ".join(force_str(arg, encoding, errors) for arg in obj.args) else: # As last resort, use a repr call to avoid any exception obj = repr(obj) return obj def redact_passwords(text): """ Redact passwords from the input text. Password are found in these two forms: Keyword/Value Connection Strings: - host=localhost port=5432 dbname=mydb password=SHAME_ON_ME Connection URIs: - postgresql://[user[:password]][netloc][:port][/dbname] :param str text: Input content :return: String with passwords removed """ # Remove passwords as found in key/value connection strings text = re.sub("password=('(\\'|[^'])+'|[^ '\"]*)", "password=*REDACTED*", text) # Remove passwords in connection URLs text = re.sub(r"(?<=postgresql:\/\/)([^ :@]+:)([^ :@]+)?@", r"\1*REDACTED*@", text) return text def check_non_negative(value): """ Check for a positive integer option :param value: str containing the value to check """ if value is None: return None try: int_value = int(value) except Exception: raise ArgumentTypeError("'%s' is not a valid non negative integer" % value) if int_value < 0: raise ArgumentTypeError("'%s' is not a valid non negative integer" % value) return int_value def check_positive(value): """ Check for a positive integer option :param value: str containing the value to check """ if value is None: return None try: int_value = int(value) except Exception: raise ArgumentTypeError("'%s' is not a valid input" % value) if int_value < 1: raise ArgumentTypeError("'%s' is not a valid positive integer" % value) return int_value def check_aws_expiration_date_format(value): """ Check user input for aws expiration date timestamp with a specific format. :param value: str containing the value to check. :raise ValueError: Fails with an invalid date. """ fmt = "%Y-%m-%dT%H:%M:%S.%fZ" try: # Attempt to parse the input date string into a datetime object return datetime.datetime.strptime(value, fmt) except ValueError: raise ArgumentTypeError( "Invalid date: '%s'. Expected format is '%s'." % (value, fmt) ) def check_aws_snapshot_lock_duration_range(value): """ Check for AWS Snapshot Lock duration range option :param value: str containing the value to check """ if value is None: return None try: int_value = int(value) except Exception: raise ArgumentTypeError("'%s' is not a valid input" % value) if not 1 <= int_value <= 36500: raise ValueError( "aws_snapshot_lock_duration must be between 1 and 36,500 days." ) return int_value def check_aws_snapshot_lock_cool_off_period_range(value): """ Check for AWS Snapshot Lock cool-off period range option :param value: str containing the value to check """ if value is None: return None try: int_value = int(value) except Exception: raise ArgumentTypeError("'%s' is not a valid input" % value) if not 1 <= int_value <= 72: raise ValueError( "aws_snapshot_lock_cool_off_period must be between 1 and 72 hours." ) return int_value def check_aws_snapshot_lock_mode(value): """ Replication slot names may only contain lower case letters, numbers, and the underscore character. This function parse a replication slot name :param str value: slot_name value :return: """ AWS_SNAPSHOT_LOCK_MODE = ["governance", "compliance"] if value is None: return None value = value.lower() if value not in AWS_SNAPSHOT_LOCK_MODE: raise ValueError( "Invalid AWS snapshot lock mode. " "Please specify either 'governance' or 'compliance'. " "Ensure that the mode you choose aligns with your snapshot locking " "requirements." ) return value def check_tli(value): """ Check for a positive integer option, and also make "current" and "latest" acceptable values :param value: str containing the value to check """ if value is None: return None if value in ["current", "latest"]: return value else: return check_positive(value) def check_size(value): """ Check user input for a human readable size :param value: str containing the value to check """ if value is None: return None # Ignore cases value = value.upper() try: # If value ends with `B` we try to parse the multiplier, # otherwise it is a plain integer if value[-1] == "B": # By default we use base=1024, if the value ends with `iB` # it is a SI value and we use base=1000 if value[-2] == "I": base = 1000 idx = 3 else: base = 1024 idx = 2 multiplier = base # Parse the multiplicative prefix for prefix in "KMGTPEZY": if value[-idx] == prefix: int_value = int(float(value[:-idx]) * multiplier) break multiplier *= base else: # If we do not find the prefix, remove the unit # and try to parse the remainder as an integer # (e.g. '1234B') int_value = int(value[: -idx + 1]) else: int_value = int(value) except ValueError: raise ArgumentTypeError("'%s' is not a valid size string" % value) if int_value is None or int_value < 1: raise ArgumentTypeError("'%s' is not a valid size string" % value) return int_value def check_backup_name(backup_name): """ Verify that a backup name is not a backup ID or reserved identifier. Returns the backup name if it is a valid backup name and raises an exception otherwise. A backup name is considered valid if it is not None, not empty, does not match the backup ID format and is not any other reserved backup identifier. :param str backup_name: The backup name to be checked. :return str: The backup name. """ if backup_name is None: raise ArgumentTypeError("Backup name cannot be None") if backup_name == "": raise ArgumentTypeError("Backup name cannot be empty") if is_backup_id(backup_name): raise ArgumentTypeError( "Backup name '%s' is not allowed: backup ID" % backup_name ) if backup_name in (RESERVED_BACKUP_IDS): raise ArgumentTypeError( "Backup name '%s' is not allowed: reserved word" % backup_name ) return backup_name def is_backup_id(backup_id): """ Checks whether the supplied identifier is a backup ID. :param str backup_id: The backup identifier to check. :return bool: True if the backup matches the backup ID regex, False otherwise. """ return bool(re.match(r"(\d{8})T\d{6}$", backup_id)) def get_backup_info_from_name(backups, backup_name): """ Get the backup metadata for the named backup. :param list[BackupInfo] backups: A list of BackupInfo objects which should be searched for the named backup. :param str backup_name: The name of the backup for which the backup metadata should be retrieved. :return BackupInfo|None: The backup metadata for the named backup. """ matching_backups = [ backup for backup in backups if backup.backup_name == backup_name ] if len(matching_backups) > 1: matching_backup_ids = " ".join( [backup.backup_id for backup in matching_backups] ) msg = ( "Multiple backups found matching name '%s' " "(try using backup ID instead): %s" ) % (backup_name, matching_backup_ids) raise ValueError(msg) elif len(matching_backups) == 1: return matching_backups[0] def get_backup_id_using_shortcut(server, shortcut, BackupInfo): """ Get backup ID from one of Barman shortcuts. :param str server: The obj where to look from. :param str shortcut: pattern to search. :param BackupInfo BackupInfo: Place where we keep some Barman constants. :return str backup_id|None: The backup ID for the provided shortcut. """ backup_id = None if shortcut in ("latest", "last"): backup_id = server.get_last_backup_id() elif shortcut in ("oldest", "first"): backup_id = server.get_first_backup_id() elif shortcut in ("last-failed"): backup_id = server.get_last_backup_id([BackupInfo.FAILED]) elif shortcut in ("latest-full", "last-full"): backup_id = server.get_last_full_backup_id() elif is_backup_id(shortcut): backup_id = shortcut return backup_id def lock_files_cleanup(lock_dir, lock_directory_cleanup): """ Get all the lock files in the lock directory and try to acquire every single one. If the file is not locked, remove it. This method is part of cron and should help keeping clean the lockfile directory. """ if not lock_directory_cleanup: # Auto cleanup of lockfile directory disabled. # Log for debug only and return _logger.debug("Auto-cleanup of '%s' directory disabled" % lock_dir) return _logger.info("Cleaning up lockfiles directory.") for filename in glob(os.path.join(lock_dir, ".*.lock")): lock = lockfile.LockFile(filename, raise_if_fail=False, wait=False) with lock as locked: # if we have the lock we can remove the file if locked: try: _logger.debug("deleting %s" % filename) os.unlink(filename) _logger.debug("%s deleted" % filename) except FileNotFoundError: # IF we are trying to remove an already removed file, is not # a big deal, just pass. pass else: _logger.debug( "%s file lock already acquired, skipping removal" % filename ) def edit_config(file, section, option, value, lines=None): """ Utility method that given a file and a config section allows to: - add a new section if at least a key-value content is provided - add a new key-value to a config section - change a section value :param file: the path to the file to edit :type file: str :param section: the config section to edit or to add :type section: str :param option: the config key to edit or add :type option: str :param value: the value for the config key to update or add :type value: str :param lines: optional parameter containing the set of lines of the file to update :type lines: list :return: the updated lines of the file """ conf_section = False idx = 0 if lines is None: try: with open(file, "r") as config: lines = config.readlines() except FileNotFoundError: lines = [] eof = len(lines) - 1 for idx, line in enumerate(lines): # next section if conf_section and line.strip().startswith("["): lines.insert(idx - 1, option + " = " + value) break # Option found, update value elif conf_section and line.strip().replace(" ", "").startswith(option + "="): lines.pop(idx) lines.insert(idx, option + " = " + value + "\n") break # End of file reached, append lines elif conf_section and idx == eof: lines.append(option + " = " + value + "\n") break # Section found if line.strip() == "[" + section + "]": conf_section = True # Section not found, create a new section and append option if not conf_section: # Note: we need to use 2 append, otherwise the section matching is not # going to work lines.append("[" + section + "]\n") lines.append(option + " = " + value + "\n") return lines def get_backup_id_from_target_time(available_backups, target_time, target_tli=None): """ Get backup ID from the catalog based on the recovery_target *target_time* and *target_tli*. :param list[BackupInfo] available_backups: List of BackupInfo objects. :param str target_time: The target value with timestamp format ``%Y-%m-%d %H:%M:%S`` with or without timezone. :param int|None target_tli: Target timeline value, if a specific one is required. :return str|None: ID of the backup. :raises: :exc:`ValueError`: If *target_time* is an invalid value. """ from barman.infofile import load_datetime_tz try: parsed_target = load_datetime_tz(target_time) except Exception as e: raise ValueError( "Unable to parse the target time parameter %r: %s" % (target_time, e) ) for candidate_backup in sorted( available_backups, key=lambda backup: backup.backup_id, reverse=True ): if candidate_backup.end_time <= parsed_target: if target_tli is not None and candidate_backup.timeline != target_tli: continue return candidate_backup.backup_id return None def get_backup_id_from_target_lsn(available_backups, target_lsn, target_tli=None): """ Get backup ID from the catalog based on the recovery_target *target_lsn* and *target_tli*. :param list[BackupInfo] available_backups: List of BackupInfo objects. :param str target_lsn: The target value with lsn format, e.g., ``3/64000000``. :param int|None target_tli: Target timeline value, if a specific one is required. :return str|None: ID of the backup. """ parsed_target = xlog.parse_lsn(target_lsn) for candidate_backup in sorted( available_backups, key=lambda backup: backup.backup_id, reverse=True ): if xlog.parse_lsn(candidate_backup.end_xlog) <= parsed_target: if target_tli is not None and candidate_backup.timeline != target_tli: continue return candidate_backup.backup_id return None def get_last_backup_id(available_backups): """ Get last backup ID from the catalog. :param list[BackupInfo] available_backups: List of BackupInfo objects. :return str|None: ID of the backup. """ if len(available_backups) == 0: return None backups = sorted(available_backups, key=lambda backup: backup.backup_id) return backups[-1].backup_id def get_backup_id_from_target_tli(available_backups, target_tli): """ Get backup ID from the catalog based on the recovery_target *target_tli*. :param list[BackupInfo] available_backups: Dict values of BackupInfo objects. :param int target_tli: Target timeline value. :return str|None: ID of the backup. """ if len(available_backups) == 0: return None for candidate_backup in sorted( available_backups, key=lambda backup: backup.backup_id, reverse=True ): if candidate_backup.timeline == target_tli: return candidate_backup.backup_id return None def parse_target_tli(obj, target_tli, backup_info=None): """ Parse target timeline shorcut, ``latest`` and ``current``. .. note:: This method is used in two other methods that are part of the recovery of a a backup (:meth:`_set_pitr_targets` and :meth:`get_required_xlog_files`). When called with a *backup_info*, it means that the recover operation uses a specific backup for recovery and ``current`` is allowed in this case because this backup is considered the ``current`` backup. When called without a *backup_info*, it means that the recover operation is going to fetch the best backup for recovery, so ``current`` is not allowed because there is no ``current`` backup. This method can also be used in the cloud script to retrieve the latest WAL from the cloud catalog when shortcut is ``latest``. :param BackupManager|CloudBackupCatalog obj: A BackupManager or a CloudBackupCatalog object. :param str|int target_tli: Target timeline value. Accepts both an integer representing the timeline, or keywords accepted by Postgres, such as ``current`` and ``latest``. :param None|BackupInfo backup_info: Backup info object. :return int|None: ID of the timeline. :raise ValueError: if *target_tli* is an invalid value. """ parsed_target_tli = target_tli if target_tli and type(target_tli) is str: if target_tli == "current": if backup_info is None: raise ValueError( "'%s' is not a valid timeline keyword when recovering" " without a backup_id" % target_tli ) else: parsed_target_tli = backup_info.timeline elif target_tli == "latest": valid_timelines = obj.get_latest_archived_wals_info() parsed_target_tli = int(max(valid_timelines.keys()), 16) elif target_tli.isdigit(): parsed_target_tli = int(target_tli) else: raise ValueError("'%s' is not a valid timeline keyword" % target_tli) return parsed_target_tli barman-3.13.3/barman/command_wrappers.py0000644000175100001660000013563315002177425016413 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ This module contains a wrapper for shell commands """ from __future__ import print_function import errno import inspect import logging import os import re import select import signal import subprocess import sys import time from distutils.version import LooseVersion as Version import barman.utils from barman.exceptions import CommandFailedException, CommandMaxRetryExceeded _logger = logging.getLogger(__name__) class Handler: def __init__(self, logger, level, prefix=None): self.class_logger = logger self.level = level self.prefix = prefix def run(self, line): if line: if self.prefix: self.class_logger.log(self.level, "%s%s", self.prefix, line) else: self.class_logger.log(self.level, "%s", line) __call__ = run class StreamLineProcessor(object): """ Class deputed to reading lines from a file object, using a buffered read. NOTE: This class never call os.read() twice in a row. And is designed to work with the select.select() method. """ def __init__(self, fobject, handler): """ :param file fobject: The file that is being read :param callable handler: The function (taking only one unicode string argument) which will be called for every line """ self._file = fobject self._handler = handler self._buf = "" def fileno(self): """ Method used by select.select() to get the underlying file descriptor. :rtype: the underlying file descriptor """ return self._file.fileno() def process(self): """ Read the ready data from the stream and for each line found invoke the handler. :return bool: True when End Of File has been reached """ data = os.read(self._file.fileno(), 4096) # If nothing has been read, we reached the EOF if not data: self._file.close() # Handle the last line (always incomplete, maybe empty) self._handler(self._buf) return True self._buf += data.decode("utf-8", "replace") # If no '\n' is present, we just read a part of a very long line. # Nothing to do at the moment. if "\n" not in self._buf: return False tmp = self._buf.split("\n") # Leave the remainder in self._buf self._buf = tmp[-1] # Call the handler for each complete line. lines = tmp[:-1] for line in lines: self._handler(line) return False class Command(object): """ Wrapper for a system command """ def __init__( self, cmd, args=None, env_append=None, path=None, shell=False, check=False, allowed_retval=(0,), close_fds=True, out_handler=None, err_handler=None, retry_times=0, retry_sleep=0, retry_handler=None, ): """ If the `args` argument is specified the arguments will be always added to the ones eventually passed with the actual invocation. If the `env_append` argument is present its content will be appended to the environment of every invocation. The subprocess output and error stream will be processed through the output and error handler, respectively defined through the `out_handler` and `err_handler` arguments. If not provided every line will be sent to the log respectively at INFO and WARNING level. The `out_handler` and the `err_handler` functions will be invoked with one single argument, which is a string containing the line that is being processed. If the `close_fds` argument is True, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. If the `check` argument is True, the exit code will be checked against the `allowed_retval` list, raising a CommandFailedException if not in the list. If `retry_times` is greater than 0, when the execution of a command terminates with an error, it will be retried for a maximum of `retry_times` times, waiting for `retry_sleep` seconds between every attempt. Every time a command is retried the `retry_handler` is executed before running the command again. The retry_handler must be a callable that accepts the following fields: * the Command object * the arguments list * the keyword arguments dictionary * the number of the failed attempt * the exception containing the error An example of such a function is: > def retry_handler(command, args, kwargs, attempt, exc): > print("Failed command!") Some of the keyword arguments can be specified both in the class constructor and during the method call. If specified in both places, the method arguments will take the precedence over the constructor arguments. :param str cmd: The command to execute :param list[str]|None args: List of additional arguments to append :param dict[str.str]|None env_append: additional environment variables :param str path: PATH to be used while searching for `cmd` :param bool shell: If true, use the shell instead of an "execve" call :param bool check: Raise a CommandFailedException if the exit code is not present in `allowed_retval` :param list[int] allowed_retval: List of exit codes considered as a successful termination. :param bool close_fds: If set, close all the extra file descriptors :param callable out_handler: handler for lines sent on stdout :param callable err_handler: handler for lines sent on stderr :param int retry_times: number of allowed retry attempts :param int retry_sleep: wait seconds between every retry :param callable retry_handler: handler invoked during a command retry """ self.pipe = None self.cmd = cmd self.args = args if args is not None else [] self.shell = shell self.close_fds = close_fds self.check = check self.allowed_retval = allowed_retval self.retry_times = retry_times self.retry_sleep = retry_sleep self.retry_handler = retry_handler self.path = path self.ret = None self.out = None self.err = None # If env_append has been provided use it or replace with an empty dict env_append = env_append or {} # If path has been provided, replace it in the environment if path: env_append["PATH"] = path # Find the absolute path to the command to execute if not self.shell: full_path = barman.utils.which(self.cmd, self.path) if not full_path: raise CommandFailedException("%s not in PATH" % self.cmd) self.cmd = full_path # If env_append contains anything, build an env dict to be used during # subprocess call, otherwise set it to None and let the subprocesses # inherit the parent environment if env_append: self.env = os.environ.copy() self.env.update(env_append) else: self.env = None # If an output handler has been provided use it, otherwise log the # stdout as INFO if out_handler: self.out_handler = out_handler else: self.out_handler = self.make_logging_handler(logging.DEBUG) # If an error handler has been provided use it, otherwise log the # stderr as WARNING if err_handler: self.err_handler = err_handler else: self.err_handler = self.make_logging_handler(logging.WARNING) @staticmethod def _restore_sigpipe(): """restore default signal handler (http://bugs.python.org/issue1652)""" signal.signal(signal.SIGPIPE, signal.SIG_DFL) # pragma: no cover def __call__(self, *args, **kwargs): """ Run the command and return the exit code. The output and error strings are not returned, but they can be accessed as attributes of the Command object, as well as the exit code. If `stdin` argument is specified, its content will be passed to the executed command through the standard input descriptor. If the `close_fds` argument is True, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. If the `check` argument is True, the exit code will be checked against the `allowed_retval` list, raising a CommandFailedException if not in the list. Every keyword argument can be specified both in the class constructor and during the method call. If specified in both places, the method arguments will take the precedence over the constructor arguments. :rtype: int :raise: CommandFailedException :raise: CommandMaxRetryExceeded """ self.get_output(*args, **kwargs) return self.ret def get_output(self, *args, **kwargs): """ Run the command and return the output and the error as a tuple. The return code is not returned, but it can be accessed as an attribute of the Command object, as well as the output and the error strings. If `stdin` argument is specified, its content will be passed to the executed command through the standard input descriptor. If the `close_fds` argument is True, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. If the `check` argument is True, the exit code will be checked against the `allowed_retval` list, raising a CommandFailedException if not in the list. Every keyword argument can be specified both in the class constructor and during the method call. If specified in both places, the method arguments will take the precedence over the constructor arguments. :rtype: tuple[str, str] :raise: CommandFailedException :raise: CommandMaxRetryExceeded """ attempt = 0 while True: try: return self._get_output_once(*args, **kwargs) except CommandFailedException as exc: # Try again if retry number is lower than the retry limit if attempt < self.retry_times: # If a retry_handler is defined, invoke it passing the # Command instance and the exception if self.retry_handler: self.retry_handler(self, args, kwargs, attempt, exc) # Sleep for configured time, then try again time.sleep(self.retry_sleep) attempt += 1 else: if attempt == 0: # No retry requested by the user # Raise the original exception raise else: # If the max number of attempts is reached and # there is still an error, exit raising # a CommandMaxRetryExceeded exception and wrap the # original one raise CommandMaxRetryExceeded(*exc.args) def _get_output_once(self, *args, **kwargs): """ Run the command and return the output and the error as a tuple. The return code is not returned, but it can be accessed as an attribute of the Command object, as well as the output and the error strings. If `stdin` argument is specified, its content will be passed to the executed command through the standard input descriptor. If the `close_fds` argument is True, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. If the `check` argument is True, the exit code will be checked against the `allowed_retval` list, raising a CommandFailedException if not in the list. Every keyword argument can be specified both in the class constructor and during the method call. If specified in both places, the method arguments will take the precedence over the constructor arguments. :rtype: tuple[str, str] :raises: CommandFailedException """ out = [] err = [] def out_handler(line): out.append(line) if self.out_handler is not None: self.out_handler(line) def err_handler(line): err.append(line) if self.err_handler is not None: self.err_handler(line) # If check is true, it must be handled here check = kwargs.pop("check", self.check) allowed_retval = kwargs.pop("allowed_retval", self.allowed_retval) self.execute( out_handler=out_handler, err_handler=err_handler, check=False, *args, **kwargs ) self.out = "\n".join(out) self.err = "\n".join(err) _logger.debug("Command stdout: %s", self.out) _logger.debug("Command stderr: %s", self.err) # Raise if check and the return code is not in the allowed list if check: self.check_return_value(allowed_retval) return self.out, self.err def check_return_value(self, allowed_retval): """ Check the current return code and raise CommandFailedException when it's not in the allowed_retval list :param list[int] allowed_retval: list of return values considered success :raises: CommandFailedException """ if self.ret not in allowed_retval: raise CommandFailedException(dict(ret=self.ret, out=self.out, err=self.err)) def execute(self, *args, **kwargs): """ Execute the command and pass the output to the configured handlers If `stdin` argument is specified, its content will be passed to the executed command through the standard input descriptor. The subprocess output and error stream will be processed through the output and error handler, respectively defined through the `out_handler` and `err_handler` arguments. If not provided every line will be sent to the log respectively at INFO and WARNING level. If the `close_fds` argument is True, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. If the `check` argument is True, the exit code will be checked against the `allowed_retval` list, raising a CommandFailedException if not in the list. Every keyword argument can be specified both in the class constructor and during the method call. If specified in both places, the method arguments will take the precedence over the constructor arguments. :rtype: int :raise: CommandFailedException """ # Check keyword arguments stdin = kwargs.pop("stdin", None) check = kwargs.pop("check", self.check) allowed_retval = kwargs.pop("allowed_retval", self.allowed_retval) close_fds = kwargs.pop("close_fds", self.close_fds) out_handler = kwargs.pop("out_handler", self.out_handler) err_handler = kwargs.pop("err_handler", self.err_handler) if len(kwargs): raise TypeError( "%s() got an unexpected keyword argument %r" % (inspect.stack()[1][3], kwargs.popitem()[0]) ) # Reset status self.ret = None self.out = None self.err = None # Create the subprocess and save it in the current object to be usable # by signal handlers pipe = self._build_pipe(args, close_fds) self.pipe = pipe # Send the provided input and close the stdin descriptor if stdin: pipe.stdin.write(stdin) pipe.stdin.close() # Prepare the list of processors processors = [ StreamLineProcessor(pipe.stdout, out_handler), StreamLineProcessor(pipe.stderr, err_handler), ] # Read the streams until the subprocess exits self.pipe_processor_loop(processors) # Reap the zombie and read the exit code pipe.wait() self.ret = pipe.returncode # Remove the closed pipe from the object self.pipe = None _logger.debug("Command return code: %s", self.ret) # Raise if check and the return code is not in the allowed list if check: self.check_return_value(allowed_retval) return self.ret def _build_pipe(self, args, close_fds): """ Build the Pipe object used by the Command The resulting command will be composed by: self.cmd + self.args + args :param args: extra arguments for the subprocess :param close_fds: if True all file descriptors except 0, 1 and 2 will be closed before the child process is executed. :rtype: subprocess.Popen """ # Append the argument provided to this method of the base argument list args = self.args + list(args) # If shell is True, properly quote the command if self.shell: cmd = full_command_quote(self.cmd, args) else: cmd = [self.cmd] + args # Log the command we are about to execute _logger.debug("Command: %r", cmd) return subprocess.Popen( cmd, shell=self.shell, env=self.env, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, preexec_fn=self._restore_sigpipe, close_fds=close_fds, ) @staticmethod def pipe_processor_loop(processors): """ Process the output received through the pipe until all the provided StreamLineProcessor reach the EOF. :param list[StreamLineProcessor] processors: a list of StreamLineProcessor """ # Loop until all the streams reaches the EOF while processors: try: ready = select.select(processors, [], [])[0] except select.error as e: # If the select call has been interrupted by a signal # just retry if e.args[0] == errno.EINTR: continue raise # For each ready StreamLineProcessor invoke the process() method for stream in ready: eof = stream.process() # Got EOF on this stream if eof: # Remove the stream from the list of valid processors processors.remove(stream) @classmethod def make_logging_handler(cls, level, prefix=None): """ Build a handler function that logs every line it receives. The resulting callable object logs its input at the specified level with an optional prefix. :param level: The log level to use :param prefix: An optional prefix to prepend to the line :return: handler function """ class_logger = logging.getLogger(cls.__name__) return Handler(class_logger, level, prefix) @staticmethod def make_output_handler(prefix=None): """ Build a handler function which prints every line it receives. The resulting function prints (and log it at INFO level) its input with an optional prefix. :param prefix: An optional prefix to prepend to the line :return: handler function """ # Import the output module inside the function to avoid circular # dependency from barman import output def handler(line): if line: if prefix: output.info("%s%s", prefix, line) else: output.info("%s", line) return handler def enable_signal_forwarding(self, signal_id): """ Enable signal forwarding to the subprocess for a specified signal_id :param signal_id: The signal id to be forwarded """ # Get the current signal handler old_handler = signal.getsignal(signal_id) def _handler(sig, frame): """ This signal handler forward the signal to the subprocess then execute the original handler. """ # Forward the signal to the subprocess if self.pipe: self.pipe.send_signal(signal_id) # If the old handler is callable if callable(old_handler): old_handler(sig, frame) # If we have got a SIGTERM, we must exit elif old_handler == signal.SIG_DFL and signal_id == signal.SIGTERM: sys.exit(128 + signal_id) # Set the signal handler signal.signal(signal_id, _handler) class Rsync(Command): """ This class is a wrapper for the rsync system command, which is used vastly by barman """ def __init__( self, rsync="rsync", args=None, ssh=None, ssh_options=None, bwlimit=None, exclude=None, exclude_and_protect=None, include=None, network_compression=None, path=None, **kwargs ): """ :param str rsync: rsync executable name :param list[str]|None args: List of additional argument to always append :param str ssh: the ssh executable to be used when building the `-e` argument :param list[str] ssh_options: the ssh options to be used when building the `-e` argument :param str bwlimit: optional bandwidth limit :param list[str] exclude: list of file to be excluded from the copy :param list[str] exclude_and_protect: list of file to be excluded from the copy, preserving the destination if exists :param list[str] include: list of files to be included in the copy even if excluded. :param bool network_compression: enable the network compression :param str path: PATH to be used while searching for `cmd` :param bool check: Raise a CommandFailedException if the exit code is not present in `allowed_retval` :param list[int] allowed_retval: List of exit codes considered as a successful termination. """ options = [] if ssh: options += ["-e", full_command_quote(ssh, ssh_options)] if network_compression: options += ["-z"] # Include patterns must be before the exclude ones, because the exclude # patterns actually short-circuit the directory traversal stage # when rsync finds the files to send. if include: for pattern in include: options += ["--include=%s" % (pattern,)] if exclude: for pattern in exclude: options += ["--exclude=%s" % (pattern,)] if exclude_and_protect: for pattern in exclude_and_protect: options += ["--exclude=%s" % (pattern,), "--filter=P_%s" % (pattern,)] if args: options += self._args_for_suse(args) if bwlimit is not None and bwlimit > 0: options += ["--bwlimit=%s" % bwlimit] # By default check is on and the allowed exit code are 0 and 24 if "check" not in kwargs: kwargs["check"] = True if "allowed_retval" not in kwargs: kwargs["allowed_retval"] = (0, 24) Command.__init__(self, rsync, args=options, path=path, **kwargs) def _args_for_suse(self, args): """ Mangle args for SUSE compatibility See https://bugzilla.opensuse.org/show_bug.cgi?id=898513 """ # Prepend any argument starting with ':' with a space # Workaround for SUSE rsync issue return [" " + a if a.startswith(":") else a for a in args] def get_output(self, *args, **kwargs): """ Run the command and return the output and the error (if present) """ # Prepares args for SUSE args = self._args_for_suse(args) # Invoke the base class method return super(Rsync, self).get_output(*args, **kwargs) def from_file_list(self, filelist, src, dst, *args, **kwargs): """ This method copies filelist from src to dst. Returns the return code of the rsync command """ if "stdin" in kwargs: raise TypeError("from_file_list() doesn't support 'stdin' keyword argument") # The input string for the rsync --files-from argument must have a # trailing newline for compatibility with certain versions of rsync. input_string = ("\n".join(filelist) + "\n").encode("UTF-8") _logger.debug("from_file_list: %r", filelist) kwargs["stdin"] = input_string self.get_output("--files-from=-", src, dst, *args, **kwargs) return self.ret class RsyncPgData(Rsync): """ This class is a wrapper for rsync, specialised in sync-ing the Postgres data directory """ def __init__(self, rsync="rsync", args=None, **kwargs): """ Constructor :param str rsync: command to run """ options = ["-rLKpts", "--delete-excluded", "--inplace"] if args: options += args Rsync.__init__(self, rsync, args=options, **kwargs) class PostgreSQLClient(Command): """ Superclass of all the PostgreSQL client commands. """ COMMAND_ALTERNATIVES = None """ Sometimes the name of a command has been changed during the PostgreSQL evolution. I.e. that happened with pg_receivexlog, that has been renamed to pg_receivewal. In that case, we should try using pg_receivewal (the newer alternative) and, if that command doesn't exist, we should try using `pg_receivexlog`. This is a list of command names to be used to find the installed command. """ def __init__( self, connection, command, version=None, app_name=None, path=None, **kwargs ): """ Constructor :param PostgreSQL connection: an object representing a database connection :param str command: the command to use :param Version version: the command version :param str app_name: the application name to use for the connection :param str path: additional path for executable retrieval """ Command.__init__(self, command, path=path, **kwargs) if not connection: self.enable_signal_forwarding(signal.SIGINT) self.enable_signal_forwarding(signal.SIGTERM) return if version and version >= Version("9.3"): # If version of the client is >= 9.3 we use the connection # string because allows the user to use all the parameters # supported by the libpq library to create a connection conn_string = connection.get_connection_string(app_name) self.args.append("--dbname=%s" % conn_string) else: # 9.2 version doesn't support # connection strings so the 'split' version of the conninfo # option is used instead. conn_params = connection.conn_parameters self.args.append("--host=%s" % conn_params.get("host", None)) self.args.append("--port=%s" % conn_params.get("port", None)) self.args.append("--username=%s" % conn_params.get("user", None)) self.enable_signal_forwarding(signal.SIGINT) self.enable_signal_forwarding(signal.SIGTERM) @classmethod def find_command(cls, path=None): """ Find the active command, given all the alternatives as set in the property named `COMMAND_ALTERNATIVES` in this class. :param str path: The path to use while searching for the command :rtype: Command """ # TODO: Unit tests of this one # To search for an available command, testing if the command # exists in PATH is not sufficient. Debian will install wrappers for # all commands, even if the real command doesn't work. # # I.e. we may have a wrapper for `pg_receivewal` even it PostgreSQL # 10 isn't installed. # # This is an example of what can happen in this case: # # ``` # $ pg_receivewal --version; echo $? # Error: pg_wrapper: pg_receivewal was not found in # /usr/lib/postgresql/9.6/bin # 1 # $ pg_receivexlog --version; echo $? # pg_receivexlog (PostgreSQL) 9.6.3 # 0 # ``` # # That means we should not only ensure the existence of the command, # but we also need to invoke the command to see if it is a shim # or not. # Get the system path if needed if path is None: path = os.getenv("PATH") # If the path is None at this point we have nothing to search if path is None: path = "" # Search the requested executable in every directory present # in path and return a Command object first occurrence that exists, # is executable and runs without errors. for path_entry in path.split(os.path.pathsep): for cmd in cls.COMMAND_ALTERNATIVES: full_path = barman.utils.which(cmd, path_entry) # It doesn't exist try another if not full_path: continue # It exists, let's try invoking it with `--version` to check if # it's real or not. try: command = Command(full_path, path=path, check=True) command("--version") return command except CommandFailedException: # It's only a inactive shim continue # We don't have such a command raise CommandFailedException( "command not in PATH, tried: %s" % " ".join(cls.COMMAND_ALTERNATIVES) ) @classmethod def get_version_info(cls, path=None): """ Return a dictionary containing all the info about the version of the PostgreSQL client :param str path: the PATH env """ if cls.COMMAND_ALTERNATIVES is None: raise NotImplementedError( "get_version_info cannot be invoked on %s" % cls.__name__ ) version_info = dict.fromkeys( ("full_path", "full_version", "major_version"), None ) # Get the version string try: command = cls.find_command(path) except CommandFailedException as e: _logger.debug("Error invoking %s: %s", cls.__name__, e) return version_info version_info["full_path"] = command.cmd # Parse the full text version try: full_version = command.out.strip() # Remove values inside parenthesis, they # carries additional information we do not need. full_version = re.sub(r"\s*\([^)]*\)", "", full_version) full_version = full_version.split()[1] except IndexError: _logger.debug("Error parsing %s version output", version_info["full_path"]) return version_info if not re.match(r"(\d+)(\.(\d+)|devel|beta|alpha|rc).*", full_version): _logger.debug("Error parsing %s version output", version_info["full_path"]) return version_info # Extract the major version version_info["full_version"] = Version(full_version) version_info["major_version"] = Version( barman.utils.simplify_version(full_version) ) return version_info class PgBaseBackup(PostgreSQLClient): """ Wrapper class for the pg_basebackup system command """ COMMAND_ALTERNATIVES = ["pg_basebackup"] def __init__( self, connection, destination, command, version=None, app_name=None, bwlimit=None, tbs_mapping=None, immediate=False, check=True, compression=None, parent_backup_manifest_path=None, args=None, **kwargs ): """ Constructor :param PostgreSQL connection: an object representing a database connection :param str destination: destination directory path :param str command: the command to use :param Version version: the command version :param str app_name: the application name to use for the connection :param str bwlimit: bandwidth limit for pg_basebackup :param Dict[str, str] tbs_mapping: used for tablespace :param bool immediate: fast checkpoint identifier for pg_basebackup :param bool check: check if the return value is in the list of allowed values of the Command obj :param barman.compression.PgBaseBackupCompression compression: the pg_basebackup compression options used for this backup :param str parent_backup_manifest_path: the path to a backup_manifest file from a previous backup which can be used to perform an incremental backup :param List[str] args: additional arguments """ PostgreSQLClient.__init__( self, connection=connection, command=command, version=version, app_name=app_name, check=check, **kwargs ) # Set the backup destination self.args += ["-v", "--no-password", "--pgdata=%s" % destination] if version and version >= Version("10"): # If version of the client is >= 10 it would use # a temporary replication slot by default to keep WALs. # We don't need it because Barman already stores the full # WAL stream, so we disable this feature to avoid wasting one slot. self.args += ["--no-slot"] # We also need to specify that we do not want to fetch any WAL file self.args += ["--wal-method=none"] # The tablespace mapping option is repeated once for each tablespace if tbs_mapping: for tbs_source, tbs_destination in tbs_mapping.items(): self.args.append( "--tablespace-mapping=%s=%s" % (tbs_source, tbs_destination) ) # Only global bandwidth limit is supported if bwlimit is not None and bwlimit > 0: self.args.append("--max-rate=%s" % bwlimit) # If it has a manifest file path it means it is an incremental backup if parent_backup_manifest_path: self.args.append("--incremental=%s" % parent_backup_manifest_path) # Immediate checkpoint if immediate: self.args.append("--checkpoint=fast") # Append compression arguments, the exact format of which are determined # in another function since they depend on the command version self.args.extend(self._get_compression_args(version, compression)) # Manage additional args if args: self.args += args def _get_compression_args(self, version, compression): """ Determine compression related arguments for pg_basebackup from the supplied compression options in the format required by the pg_basebackup version. :param Version version: The pg_basebackup version for which the arguments should be formatted. :param barman.compression.PgBaseBackupCompression compression: the pg_basebackup compression options used for this backup """ compression_args = [] if compression is not None: if compression.config.format is not None: compression_format = compression.config.format else: compression_format = "tar" compression_args.append("--format=%s" % compression_format) # For clients >= 15 we use the new --compress argument format if version and version >= Version("15"): compress_arg = "--compress=" detail = [] if compression.config.location is not None: compress_arg += "%s-" % compression.config.location compress_arg += compression.config.type if compression.config.level is not None: detail.append("level=%d" % compression.config.level) if compression.config.workers is not None: detail.append("workers=%d" % compression.config.workers) if detail: compress_arg += ":%s" % ",".join(detail) compression_args.append(compress_arg) # For clients < 15 we use the old style argument format else: if compression.config.type == "none": compression_args.append("--compress=0") else: if compression.config.level is not None: compression_args.append( "--compress=%d" % compression.config.level ) # --gzip must be positioned after --compress when compression level=0 # so `base.tar.gz` can be created. Otherwise `.gz` won't be added. compression_args.append("--%s" % compression.config.type) return compression_args class PgReceiveXlog(PostgreSQLClient): """ Wrapper class for pg_receivexlog """ COMMAND_ALTERNATIVES = ["pg_receivewal", "pg_receivexlog"] def __init__( self, connection, destination, command, version=None, app_name=None, synchronous=False, check=True, slot_name=None, args=None, **kwargs ): """ Constructor :param PostgreSQL connection: an object representing a database connection :param str destination: destination directory path :param str command: the command to use :param Version version: the command version :param str app_name: the application name to use for the connection :param bool synchronous: request synchronous WAL streaming :param bool check: check if the return value is in the list of allowed values of the Command obj :param str slot_name: the replication slot name to use for the connection :param List[str] args: additional arguments """ PostgreSQLClient.__init__( self, connection=connection, command=command, version=version, app_name=app_name, check=check, **kwargs ) self.args += [ "--verbose", "--no-loop", "--no-password", "--directory=%s" % destination, ] # Add the replication slot name if set in the configuration. if slot_name is not None: self.args.append("--slot=%s" % slot_name) # Request synchronous mode if synchronous: self.args.append("--synchronous") # Manage additional args if args: self.args += args class PgVerifyBackup(PostgreSQLClient): """ Wrapper class for the pg_verify system command """ COMMAND_ALTERNATIVES = ["pg_verifybackup"] def __init__( self, data_path, command, connection=None, version=None, app_name=None, check=True, args=None, **kwargs ): """ Constructor :param str data_path: backup data directory :param str command: the command to use :param PostgreSQL connection: an object representing a database connection :param Version version: the command version :param str app_name: the application name to use for the connection :param bool check: check if the return value is in the list of allowed values of the Command obj :param List[str] args: additional arguments """ PostgreSQLClient.__init__( self, connection=connection, command=command, version=version, app_name=app_name, check=check, **kwargs ) self.args = ["-n", data_path] if args: self.args += args class PgCombineBackup(PostgreSQLClient): """ Wrapper class for the ``pg_combinebackup`` system command """ COMMAND_ALTERNATIVES = ["pg_combinebackup"] def __init__( self, destination, command, tbs_mapping=None, connection=None, version=None, app_name=None, check=True, args=None, **kwargs ): """ Constructor :param str destination: destination directory path :param str command: the command to use :param None|Dict[str, str] tbs_mapping: used for tablespace :param PostgreSQL connection: an object representing a database connection :param Version version: the command version :param str app_name: the application name to use for the connection :param bool check: check if the return value is in the list of allowed values of the :class:`Command` obj :param None|List[str] args: additional arguments """ PostgreSQLClient.__init__( self, connection=connection, command=command, version=version, app_name=app_name, check=check, **kwargs ) # Set the backup destination self.args = ["--output=%s" % destination] # The tablespace mapping option is repeated once for each tablespace if tbs_mapping: for tbs_source, tbs_destination in tbs_mapping.items(): self.args.append( "--tablespace-mapping=%s=%s" % (tbs_source, tbs_destination) ) # Manage additional args if args: self.args += args class BarmanSubProcess(object): """ Wrapper class for barman sub instances """ def __init__( self, command=sys.argv[0], subcommand=None, config=None, args=None, keep_descriptors=False, ): """ Build a specific wrapper for all the barman sub-commands, providing a unified interface. :param str command: path to barman :param str subcommand: the barman sub-command :param str config: path to the barman configuration file. :param list[str] args: a list containing the sub-command args like the target server name :param bool keep_descriptors: whether to keep the subprocess stdin, stdout, stderr descriptors attached. Defaults to False """ # The config argument is needed when the user explicitly # passes a configuration file, as the child process # must know the configuration file to use. # # The configuration file must always be propagated, # even in case of the default one. if not config: raise CommandFailedException( "No configuration file passed to barman subprocess" ) # Build the sub-command: # * be sure to run it with the right python interpreter # * pass the current configuration file with -c # * set it quiet with -q self.command = [sys.executable, command, "-c", config, "-q", subcommand] self.keep_descriptors = keep_descriptors # Handle args for the sub-command (like the server name) if args: self.command += args def execute(self): """ Execute the command and pass the output to the configured handlers """ _logger.debug("BarmanSubProcess: %r", self.command) # Redirect all descriptors to /dev/null devnull = open(os.devnull, "a+") additional_arguments = {} if not self.keep_descriptors: additional_arguments = {"stdout": devnull, "stderr": devnull} proc = subprocess.Popen( self.command, preexec_fn=os.setsid, close_fds=True, stdin=devnull, **additional_arguments ) _logger.debug("BarmanSubProcess: subprocess started. pid: %s", proc.pid) def shell_quote(arg): """ Quote a string argument to be safely included in a shell command line. :param str arg: The script argument :return: The argument quoted """ # This is an excerpt of the Bash manual page, and the same applies for # every Posix compliant shell: # # A non-quoted backslash (\) is the escape character. It preserves # the literal value of the next character that follows, with the # exception of . If a \ pair appears, and the # backslash is not itself quoted, the \ is treated as a # line continuation (that is, it is removed from the input # stream and effectively ignored). # # Enclosing characters in single quotes preserves the literal value # of each character within the quotes. A single quote may not occur # between single quotes, even when pre-ceded by a backslash. # # This means that, as long as the original string doesn't contain any # apostrophe character, it can be safely included between single quotes. # # If a single quote is contained in the string, we must terminate the # string with a quote, insert an apostrophe character escaping it with # a backslash, and then start another string using a quote character. assert arg is not None if arg == "|": return arg return "'%s'" % arg.replace("'", "'\\''") def full_command_quote(command, args=None): """ Produce a command with quoted arguments :param str command: the command to be executed :param list[str] args: the command arguments :rtype: str """ if args is not None and len(args) > 0: return "%s %s" % (command, " ".join([shell_quote(arg) for arg in args])) else: return command barman-3.13.3/barman/__init__.py0000644000175100001660000000157215002177425014603 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2011-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . """ The main Barman module """ from __future__ import absolute_import from .version import __version__ __config__ = None __all__ = ["__version__", "__config__"] barman-3.13.3/barman/backup_manifest.py0000644000175100001660000001256315002177425016201 0ustar 00000000000000# -*- coding: utf-8 -*- # © Copyright EnterpriseDB UK Limited 2013-2025 # # This file is part of Barman. # # Barman is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Barman is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Barman. If not, see . import json import os from barman.exceptions import BackupManifestException class BackupManifest: name = "backup_manifest" def __init__(self, path, file_manager, checksum_algorithm): """ :param path: backup directory :type path: str :param file_manager: File manager :type file_manager: barman. """ self.files = [] self.path = path self.file_manager = file_manager self.checksum_algorithm = checksum_algorithm def create_backup_manifest(self): """ Will create a manifest file if it doesn't exists. :return: """ if self.file_manager.file_exist(self._get_manifest_file_path()): msg = "File %s already exists." % self._get_manifest_file_path() raise BackupManifestException(msg) self._create_files_metadata() str_manifest = self._get_manifest_str() # Create checksum from string without last '}' and ',' instead manifest_checksum = self.checksum_algorithm.checksum_from_str(str_manifest) last_line = '"Manifest-Checksum": "%s"}\n' % manifest_checksum full_manifest = str_manifest + last_line self.file_manager.save_content_to_file( self._get_manifest_file_path(), full_manifest.encode(), file_mode="wb" ) def _get_manifest_from_dict(self): """ Old version used to create manifest first section Could be used :return: str """ manifest = { "PostgreSQL-Backup-Manifest-Version": 1, "Files": self.files, } # Convert to text # sort_keys and separators are used for python compatibility str_manifest = json.dumps( manifest, indent=2, sort_keys=True, separators=(",", ": ") ) str_manifest = str_manifest[:-2] + ",\n" return str_manifest def _get_manifest_str(self): """ :return: """ manifest = '{"PostgreSQL-Backup-Manifest-Version": 1,\n"Files": [\n' for i in self.files: # sort_keys needed for python 2/3 compatibility manifest += json.dumps(i, sort_keys=True) + ",\n" manifest = manifest[:-2] + "],\n" return manifest def _create_files_metadata(self): """ Parse all files in backup directory and get file identity values for each one of them. """ file_list = self.file_manager.get_file_list(self.path) for filepath in file_list: # Create FileEntity identity = FileIdentity( filepath, self.path, self.file_manager, self.checksum_algorithm ) self.files.append(identity.get_value()) def _get_manifest_file_path(self): """ Generates backup-manifest file path :return: backup-manifest file path :rtype: str """ return os.path.join(self.path, self.name) class FileIdentity: """ This class purpose is to aggregate file information for backup-manifest. """ def __init__(self, file_path, dir_path, file_manager, checksum_algorithm): """ :param file_path: File path to analyse :type file_path: str :param dir_path: Backup directory path :type dir_path: str :param file_manager: :type file_manager: barman.storage.FileManager :param checksum_algorithm: Object that will create checksum from bytes :type checksum_algorithm: """ self.file_path = file_path self.dir_path = dir_path self.file_manager = file_manager self.checksum_algorithm = checksum_algorithm def get_value(self): """ Returns a dictionary containing FileIdentity values """ stats = self.file_manager.get_file_stats(self.file_path) return { "Size": stats.get_size(), "Last-Modified": stats.get_last_modified(), "Checksum-Algorithm": self.checksum_algorithm.get_name(), "Path": self._get_relative_path(), "Checksum": self._get_checksum(), } def _get_relative_path(self): """ :return: file path from directory path :rtype: string """ if not self.file_path.startswith(self.dir_path): msg = "Expecting %s to start with %s" % (self.file_path, self.dir_path) raise AttributeError(msg) return self.file_path.split(self.dir_path)[1].strip("/") def _get_checksum(self): """ :return: file checksum :rtype: str """ content = self.file_manager.get_file_content(self.file_path) return self.checksum_algorithm.checksum(content) barman-3.13.3/docs/0000755000175100001660000000000015002177454012157 5ustar 00000000000000barman-3.13.3/docs/_build/0000755000175100001660000000000015002177454013415 5ustar 00000000000000barman-3.13.3/docs/_build/man/0000755000175100001660000000000015002177454014170 5ustar 00000000000000barman-3.13.3/docs/_build/man/barman-verify.10000644000175100001660000000171115002177425017012 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-VERIFY" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-verify \- Barman Sub-Commands .SH DESCRIPTION .sp Alias for \fBverify\-backup\fP command. .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-cloud-backup-show.10000644000175100001660000001046215002177425021040 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CLOUD-BACKUP-SHOW" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-cloud-backup-show \- Barman-cloud Commands .sp \fBSynopsis\fP .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-cloud\-backup\-show [ { \-V | \-\-version } ] [ \-\-help ] [ { { \-v | \-\-verbose } | { \-q | \-\-quiet } } ] [ { \-t | \-\-test } ] [ \-\-cloud\-provider { aws\-s3 | azure\-blob\-storage | google\-cloud\-storage } ] [ \-\-endpoint\-url ENDPOINT_URL ] [ { \-P | \-\-aws\-profile } AWS_PROFILE ] [ \-\-profile AWS_PROFILE ] [ \-\-read\-timeout READ_TIMEOUT ] [ { \-\-azure\-credential | \-\-credential } { azure\-cli | managed\-identity | default } ] [ \-\-format FORMAT ] SOURCE_URL SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .sp \fBDescription\fP .sp This script displays detailed information about a specific backup created with the \fBbarman\-cloud\-backup\fP command. The output is similar to the \fBbarman show\-backup\fP from the \fI\%barman show\-backup\fP command reference, but it has fewer information. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 For GCP, only authentication with \fBGOOGLE_APPLICATION_CREDENTIALS\fP env is supported. .UNINDENT .UNINDENT .sp \fBParameters\fP .INDENT 0.0 .TP .B \fBBACKUP_ID\fP The ID of the backup. .TP .B \fBSERVER_NAME\fP Name of the server that holds the backup to be displayed. .TP .B \fBSOURCE_URL\fP URL of the cloud source, such as a bucket in AWS S3. For example: \fBs3://bucket/path/to/folder\fP\&. .TP .B \fB\-V\fP / \fB\-\-version\fP Show version and exit. .TP .B \fB\-\-help\fP show this help message and exit. .TP .B \fB\-v\fP / \fB\-\-verbose\fP Increase output verbosity (e.g., \fB\-vv\fP is more than \fB\-v\fP). .TP .B \fB\-q\fP / \fB\-\-quiet\fP Decrease output verbosity (e.g., \fB\-qq\fP is less than \fB\-q\fP). .TP .B \fB\-t\fP / \fB\-\-test\fP Test cloud connectivity and exit. .TP .B \fB\-\-cloud\-provider\fP The cloud provider to use as a storage backend. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBaws\-s3\fP\&. .IP \(bu 2 \fBazure\-blob\-storage\fP\&. .IP \(bu 2 \fBgoogle\-cloud\-storage\fP\&. .UNINDENT .TP .B \fB\-\-format\fP Output format (\fBconsole\fP or \fBjson\fP). Default \fBconsole\fP\&. .UNINDENT .sp \fBExtra options for the AWS cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-endpoint\-url\fP Override default S3 endpoint URL with the given one. .TP .B \fB\-P\fP / \fB\-\-aws\-profile\fP Profile name (e.g. \fBINI\fP section in AWS credentials file). .TP .B \fB\-\-profile\fP (deprecated) Profile name (e.g. \fBINI\fP section in AWS credentials file) \- replaced by \fB\-\-aws\-profile\fP\&. .TP .B \fB\-\-read\-timeout\fP The time in seconds until a timeout is raised when waiting to read from a connection (defaults to \fB60\fP seconds). .UNINDENT .sp \fBExtra options for the Azure cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-azure\-credential / \-\-credential\fP Optionally specify the type of credential to use when authenticating with Azure. If omitted then Azure Blob Storage credentials will be obtained from the environment and the default Azure authentication flow will be used for authenticating with all other Azure services. If no credentials can be found in the environment then the default Azure authentication flow will also be used for Azure Blob Storage. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBazure\-cli\fP\&. .IP \(bu 2 \fBmanaged\-identity\fP\&. .IP \(bu 2 \fBdefault\fP\&. .UNINDENT .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-list-files.10000644000175100001660000000500115002177425017555 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-LIST-FILES" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-list-files \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX list\-files [ { \-h | \-\-help } ] [ \-\-target { data | full | standalone | wal } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp List all files in a specific backup. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-target\fP Define specific files to be listed. The possible values are: .INDENT 7.0 .IP \(bu 2 \fBstandalone\fP (default): List the base backup files, including required WAL files. .IP \(bu 2 \fBdata\fP: List just the data files. .IP \(bu 2 \fBwal\fP: List all the WAL files between the start of the base backup and the end of the log or the start of the following base backup (depending on whether the specified base backup is the most recent one available). .IP \(bu 2 \fBfull\fP: same as \fBdata\fP + \fBwal\fP\&. .UNINDENT .UNINDENT .SH SHORTCUTS .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-config-switch.10000644000175100001660000000340415002177425020253 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CONFIG-SWITCH" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-config-switch \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX config\-switch [ { \-h | \-\-help } ] SERVER_NAME { \-\-reset | MODEL_NAME } .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Apply a set of configuration overrides from the model to a server in Barman. The final configuration will combine or override the server\(aqs existing settings with the ones specified in the model. You can reset the server configurations with the \fB\-\-reset\fP argument. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only one model can be active at a time for a given server. .UNINDENT .UNINDENT .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fBMODEL_NAME\fP Name of the model. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-reset\fP Reset the server\(aqs configurations. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-cloud-wal-restore.10000644000175100001660000001202415002177425021055 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CLOUD-WAL-RESTORE" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-cloud-wal-restore \- Barman-cloud Commands .sp \fBSynopsis\fP .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-cloud\-wal\-restore [ { \-V | \-\-version } ] [ \-\-help ] [ { { \-v | \-\-verbose } | { \-q | \-\-quiet } } ] [ { \-t | \-\-test } ] [ \-\-cloud\-provider { aws\-s3 | azure\-blob\-storage | google\-cloud\-storage } ] [ \-\-endpoint\-url ENDPOINT_URL ] [ { \-P | \-\-aws\-profile } AWS_PROFILE ] [ \-\-profile AWS_PROFILE ] [ \-\-read\-timeout READ_TIMEOUT ] [ { \-\-azure\-credential | \-\-credential } { azure\-cli | managed\-identity | default } ] [ \-\-no\-partial ] SOURCE_URL SERVER_NAME WAL_NAME WAL_DEST .EE .UNINDENT .UNINDENT .sp \fBDescription\fP .sp The \fBbarman\-cloud\-wal\-restore\fP script functions as the \fBrestore_command\fP for retrieving WAL files from cloud storage and placing them directly into a Postgres standby server, bypassing the Barman server. .sp This script is used to download WAL files that were previously archived with the \fBbarman\-cloud\-wal\-archive\fP command. Disable automatic download of \fB\&.partial\fP files by calling \fB\-\-no\-partial\fP option. .sp \fBIMPORTANT:\fP .INDENT 0.0 .INDENT 3.5 On the target Postgres node, when \fBpg_wal\fP and the spool directory are on the same filesystem, files are moved via renaming, which is faster than copying and deleting. This speeds up serving WAL files significantly. If the directories are on different filesystems, the process still involves copying and deleting, so there\(aqs no performance gain in that case. .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 For GCP, only authentication with \fBGOOGLE_APPLICATION_CREDENTIALS\fP env is supported. .UNINDENT .UNINDENT .sp \fBParameters\fP .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server that will have WALs restored. .TP .B \fBSOURCE_URL\fP URL of the cloud source, such as a bucket in AWS S3. For example: \fBs3://bucket/path/to/folder\fP\&. .TP .B \fBWAL_NAME\fP The value of the \(aq%f\(aq keyword (according to \fBrestore_command\fP). .TP .B \fBWAL_DEST\fP The value of the \(aq%p\(aq keyword (according to \fBrestore_command\fP). .TP .B \fB\-V\fP / \fB\-\-version\fP Show version and exit. .TP .B \fB\-\-help\fP show this help message and exit. .TP .B \fB\-v\fP / \fB\-\-verbose\fP Increase output verbosity (e.g., \fB\-vv\fP is more than \fB\-v\fP). .TP .B \fB\-q\fP / \fB\-\-quiet\fP Decrease output verbosity (e.g., \fB\-qq\fP is less than \fB\-q\fP). .TP .B \fB\-t\fP / \fB\-\-test\fP Test cloud connectivity and exit. .TP .B \fB\-\-cloud\-provider\fP The cloud provider to use as a storage backend. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBaws\-s3\fP\&. .IP \(bu 2 \fBazure\-blob\-storage\fP\&. .IP \(bu 2 \fBgoogle\-cloud\-storage\fP\&. .UNINDENT .TP .B \fB\-\-no\-partial\fP Do not download partial WAL files .UNINDENT .sp \fBExtra options for the AWS cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-endpoint\-url\fP Override default S3 endpoint URL with the given one. .TP .B \fB\-P\fP / \fB\-\-aws\-profile\fP Profile name (e.g. \fBINI\fP section in AWS credentials file). .TP .B \fB\-\-profile\fP (deprecated) Profile name (e.g. \fBINI\fP section in AWS credentials file) \- replaced by \fB\-\-aws\-profile\fP\&. .TP .B \fB\-\-read\-timeout\fP The time in seconds until a timeout is raised when waiting to read from a connection (defaults to \fB60\fP seconds). .UNINDENT .sp \fBExtra options for the Azure cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-azure\-credential / \-\-credential\fP Optionally specify the type of credential to use when authenticating with Azure. If omitted then Azure Blob Storage credentials will be obtained from the environment and the default Azure authentication flow will be used for authenticating with all other Azure services. If no credentials can be found in the environment then the default Azure authentication flow will also be used for Azure Blob Storage. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBazure\-cli\fP\&. .IP \(bu 2 \fBmanaged\-identity\fP\&. .IP \(bu 2 \fBdefault\fP\&. .UNINDENT .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-cloud-backup.10000644000175100001660000002707115002177425020066 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CLOUD-BACKUP" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-cloud-backup \- Barman-cloud Commands .sp \fBSynopsis\fP .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-cloud\-backup [ { \-V | \-\-version } ] [ \-\-help ] [ { { \-v | \-\-verbose } | { \-q | \-\-quiet } } ] [ { \-t | \-\-test } ] [ \-\-cloud\-provider { aws\-s3 | azure\-blob\-storage | google\-cloud\-storage } ] [ { { \-z | \-\-gzip } | { \-j | \-\-bzip2 } | \-\-snappy } ] [ { \-h | \-\-host } HOST ] [ { \-p | \-\-port } PORT ] [ { \-U | \-\-user } USER ] [ { \-d | \-\-dbname } DBNAME ] [ { \-n | \-\-name } BACKUP_NAME ] [ { \-J | \-\-jobs } JOBS ] [ { \-S | \-\-max\-archive\-size } MAX_ARCHIVE_SIZE ] [ \-\-immediate\-checkpoint ] [ \-\-min\-chunk\-size MIN_CHUNK_SIZE ] [ \-\-max\-bandwidth MAX_BANDWIDTH ] [ \-\-snapshot\-instance SNAPSHOT_INSTANCE ] [ \-\-snapshot\-disk NAME ] [ \-\-snapshot\-zone GCP_ZONE ] [ \-snapshot\-gcp\-project GCP_PROJECT ] [ \-\-tags TAG [ TAG ... ] ] [ \-\-endpoint\-url ENDPOINT_URL ] [ { \-P | \-\-aws\-profile } AWS_PROFILE ] [ \-\-profile AWS_PROFILE ] [ \-\-read\-timeout READ_TIMEOUT ] [ { \-e | \-\-encryption } { AES256 | aws:kms } ] [ \-\-sse\-kms\-key\-id SSE_KMS_KEY_ID ] [ \-\-aws\-region AWS_REGION ] [ \-\-aws\-await\-snapshots\-timeout AWS_AWAIT_SNAPSHOTS_TIMEOUT ] [ \-\-aws\-snapshot\-lock\-mode { compliance | governance } ] [ \-\-aws\-snapshot\-lock\-duration DAYS ] [ \-\-aws\-snapshot\-lock\-cool\-off\-period HOURS ] [ \-\-aws\-snapshot\-lock\-expiration\-date DATETIME ] [ { \-\-azure\-credential | \-\-credential } { azure\-cli | managed\-identity | default } ] [ \-\-encryption\-scope ENCRYPTION_SCOPE ] [ \-\-azure\-subscription\-id AZURE_SUBSCRIPTION_ID ] [ \-\-azure\-resource\-group AZURE_RESOURCE_GROUP ] [ \-\-gcp\-project GCP_PROJECT ] [ \-\-kms\-key\-name KMS_KEY_NAME ] [ \-\-gcp\-zone GCP_ZONE ] DESTINATION_URL SERVER_NAME .EE .UNINDENT .UNINDENT .sp \fBDescription\fP .sp The \fBbarman\-cloud\-backup\fP script is used to create a local backup of a Postgres server and transfer it to a supported cloud provider, bypassing the Barman server. It can also be utilized as a hook script for copying Barman backups from the Barman server to one of the supported clouds (post_backup_retry_script). .sp This script requires read access to PGDATA and tablespaces, typically run as the postgres user. When used on a Barman server, it requires read access to the directory where Barman backups are stored. If \fB\-\-snapshot\-\fP arguments are used and snapshots are supported by the selected cloud provider, the backup will be performed using snapshots of the specified disks (\fB\-\-snapshot\-disk\fP). The backup label and metadata will also be uploaded to the cloud. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 For GCP, only authentication with \fBGOOGLE_APPLICATION_CREDENTIALS\fP env is supported. .UNINDENT .UNINDENT .sp \fBIMPORTANT:\fP .INDENT 0.0 .INDENT 3.5 The cloud upload may fail if any file larger than the configured \fB\-\-max\-archive\-size\fP is present in the data directory or tablespaces. However, Postgres files up to \fB1GB\fP are always allowed, regardless of the \fB\-\-max\-archive\-size\fP setting. .UNINDENT .UNINDENT .sp \fBParameters\fP .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server to be backed up. .TP .B \fBDESTINATION_URL\fP URL of the cloud destination, such as a bucket in AWS S3. For example: \fBs3://bucket/path/to/folder\fP\&. .TP .B \fB\-V\fP / \fB\-\-version\fP Show version and exit. .TP .B \fB\-\-help\fP show this help message and exit. .TP .B \fB\-v\fP / \fB\-\-verbose\fP Increase output verbosity (e.g., \fB\-vv\fP is more than \fB\-v\fP). .TP .B \fB\-q\fP / \fB\-\-quiet\fP Decrease output verbosity (e.g., \fB\-qq\fP is less than \fB\-q\fP). .TP .B \fB\-t\fP / \fB\-\-test\fP Test cloud connectivity and exit. .TP .B \fB\-\-cloud\-provider\fP The cloud provider to use as a storage backend. .sp Allowed options: .INDENT 7.0 .IP \(bu 2 \fBaws\-s3\fP\&. .IP \(bu 2 \fBazure\-blob\-storage\fP\&. .IP \(bu 2 \fBgoogle\-cloud\-storage\fP\&. .UNINDENT .TP .B \fB\-z\fP / \fB\-\-gzip\fP gzip\-compress the backup while uploading to the cloud (should not be used with python < 3.2). .TP .B \fB\-j\fP / \fB\-\-bzip2\fP bzip2\-compress the backup while uploading to the cloud (should not be used with python < 3.3). .TP .B \fB\-\-snappy\fP snappy\-compress the backup while uploading to the cloud (requires optional \fBpython\-snappy\fP library). .TP .B \fB\-h\fP / \fB\-\-host\fP Host or Unix socket for Postgres connection (default: libpq settings). .TP .B \fB\-p\fP / \fB\-\-port\fP Port for Postgres connection (default: libpq settings). .TP .B \fB\-U\fP / \fB\-\-user\fP User name for Postgres connection (default: libpq settings). .TP .B \fB\-d\fP / \fB\-\-dbname\fP Database name or conninfo string for Postgres connection (default: \(dqpostgres\(dq). .TP .B \fB\-n\fP / \fB\-\-name\fP A name which can be used to reference this backup in commands such as \fBbarman\-cloud\-restore\fP and \fBbarman\-cloud\-backup\-delete\fP\&. .TP .B \fB\-J\fP / \fB\-\-jobs\fP Number of subprocesses to upload data to cloud storage (default: \fB2\fP). .TP .B \fB\-S\fP / \fB\-\-max\-archive\-size\fP Maximum size of an archive when uploading to cloud storage (default: \fB100GB\fP). .TP .B \fB\-\-immediate\-checkpoint\fP Forces the initial checkpoint to be done as quickly as possible. .TP .B \fB\-\-min\-chunk\-size\fP Minimum size of an individual chunk when uploading to cloud storage (default: \fB5MB\fP for \fBaws\-s3\fP, \fB64KB\fP for \fBazure\-blob\-storage\fP, not applicable for \fBgoogle\-cloud\-storage\fP). .TP .B \fB\-\-max\-bandwidth\fP The maximum amount of data to be uploaded per second when backing up to object storages (default: \fB0\fP \- no limit). .TP .B \fB\-\-snapshot\-instance\fP Instance where the disks to be backed up as snapshots are attached. .TP .B \fB\-\-snapshot\-disk\fP Name of a disk from which snapshots should be taken. .TP .B \fB\-\-tags\fP Tags to be added to all uploaded files in cloud storage, and/or to snapshots created, if snapshots are used. .UNINDENT .sp \fBExtra options for the AWS cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-endpoint\-url\fP Override default S3 endpoint URL with the given one. .TP .B \fB\-P\fP / \fB\-\-aws\-profile\fP Profile name (e.g. \fBINI\fP section in AWS credentials file). .TP .B \fB\-\-profile\fP (deprecated) Profile name (e.g. \fBINI\fP section in AWS credentials file) \- replaced by \fB\-\-aws\-profile\fP\&. .TP .B \fB\-\-read\-timeout\fP The time in seconds until a timeout is raised when waiting to read from a connection (defaults to \fB60\fP seconds). .TP .B \fB\-e\fP / \fB\-\-encryption\fP The encryption algorithm used when storing the uploaded data in S3. .sp Allowed options: .INDENT 7.0 .IP \(bu 2 \fBAES256\fP\&. .IP \(bu 2 \fBaws:kms\fP\&. .UNINDENT .TP .B \fB\-\-sse\-kms\-key\-id\fP The AWS KMS key ID that should be used for encrypting the uploaded data in S3. Can be specified using the key ID on its own or using the full ARN for the key. Only allowed if \fB\-e\fP / \fB\-\-encryption\fP is set to \fBaws:kms\fP\&. .TP .B \fB\-\-aws\-region\fP The name of the AWS region containing the EC2 VM and storage volumes defined by the \fB\-\-snapshot\-instance\fP and \fB\-\-snapshot\-disk\fP arguments. .TP .B \fB\-\-aws\-await\-snapshots\-timeout\fP The length of time in seconds to wait for snapshots to be created in AWS before timing out (default: 3600 seconds). .TP .B \fB\-\-aws\-snapshot\-lock\-mode\fP The lock mode for the snapshot. This is only valid if \fB\-\-snapshot\-instance\fP and \fB\-\-snapshot\-disk\fP are set. .sp Allowed options: .INDENT 7.0 .IP \(bu 2 \fBcompliance\fP\&. .IP \(bu 2 \fBgovernance\fP\&. .UNINDENT .TP .B \fB\-\-aws\-snapshot\-lock\-duration\fP The lock duration is the period of time (in days) for which the snapshot is to remain locked, ranging from 1 to 36,500. Set either the lock duration or the expiration date (not both). .TP .B \fB\-\-aws\-snapshot\-lock\-cool\-off\-period\fP The cooling\-off period is an optional period of time (in hours) that you can specify when you lock a snapshot in \fBcompliance\fP mode, ranging from 1 to 72. .TP .B \fB\-\-aws\-snapshot\-lock\-expiration\-date\fP The lock duration is determined by an expiration date in the future. It must be at least 1 day after the snapshot creation date and time, using the format \fBYYYY\-MM\-DDTHH:MM:SS.sssZ\fP\&. Set either the lock duration or the expiration date (not both). .UNINDENT .sp \fBExtra options for the Azure cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-azure\-credential / \-\-credential\fP Optionally specify the type of credential to use when authenticating with Azure. If omitted then Azure Blob Storage credentials will be obtained from the environment and the default Azure authentication flow will be used for authenticating with all other Azure services. If no credentials can be found in the environment then the default Azure authentication flow will also be used for Azure Blob Storage. .sp Allowed options: .INDENT 7.0 .IP \(bu 2 \fBazure\-cli\fP\&. .IP \(bu 2 \fBmanaged\-identity\fP\&. .IP \(bu 2 \fBdefault\fP\&. .UNINDENT .TP .B \fB\-\-encryption\-scope\fP The name of an encryption scope defined in the Azure Blob Storage service which is to be used to encrypt the data in Azure. .TP .B \fB\-\-azure\-subscription\-id\fP The ID of the Azure subscription which owns the instance and storage volumes defined by the \fB\-\-snapshot\-instance\fP and \fB\-\-snapshot\-disk\fP arguments. .TP .B \fB\-\-azure\-resource\-group\fP The name of the Azure resource group to which the compute instance and disks defined by the \fB\-\-snapshot\-instance\fP and \fB\-\-snapshot\-disk\fP arguments belong. .UNINDENT .sp \fBExtra options for GCP cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-gcp\-project\fP GCP project under which disk snapshots should be stored. .TP .B \fB\-\-snapshot\-gcp\-project\fP (deprecated) GCP project under which disk snapshots should be stored \- replaced by \fB\-\-gcp\-project\fP\&. .TP .B \fB\-\-kms\-key\-name\fP The name of the GCP KMS key which should be used for encrypting the uploaded data in GCS. .TP .B \fB\-\-gcp\-zone\fP Zone of the disks from which snapshots should be taken. .TP .B \fB\-\-snapshot\-zone\fP (deprecated) Zone of the disks from which snapshots should be taken \- replaced by \fB\-\-gcp\-zone\fP\&. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-list_backups.10000644000175100001660000000326315002177425020175 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-LIST_BACKUPS" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-list_backups \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX list\-backups [ { \-h | \-\-help } ] [ \-\-minimal ] SERVER_NAME .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Display the available backups for a server. This command is useful for retrieving both the backup ID and the backup type. You can find details about this command in \fI\%Catalog usage\fP\&. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-minimal\fP Machine readable output. .UNINDENT .SH SHORTCUTS .sp Use shortcuts instead of \fBSERVER_NAME\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-receive-wal.10000644000175100001660000000367015002177425017717 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-RECEIVE-WAL" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-receive-wal \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX receive\-wal [ \-\-create\-slot ] [ \-\-drop\-slot ] [ { \-h | \-\-help } ] [ \-\-reset ] [ \-\-stop ] SERVER_NAME .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Initiate the streaming of transaction logs for a server. This process uses \fBpg_receivewal\fP or \fBpg_receivexlog\fP to receive WAL files from Postgres servers via the streaming protocol. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fB\-\-create\-slot\fP Create the physical replication slot configured with the \fBslot_name\fP configuration parameter. .TP .B \fB\-\-drop\-slot\fP Drop the physical replication slot configured with the \fBslot_name\fP configuration parameter. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-reset\fP Reset the status of \fBreceive\-wal\fP, restarting the streaming from the current WAL file of the server. .TP .B \fB\-\-stop\fP Stop the process for the server. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-check-backup.10000644000175100001660000000413615002177425020032 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CHECK-BACKUP" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-check-backup \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX check\-backup [ { \-h | \-\-help } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Check that all necessary WAL files for verifying the consistency of a physical backup are properly archived. This command is automatically executed by the cron job and at the end of each backup operation. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH SHORTCUTS .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-wal-archive.10000644000175100001660000000566315002177425017722 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-WAL-ARCHIVE" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-wal-archive \- Barman-cli Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-wal\-archive [ { \-h | \-\-help } ] [ { \-V | \-\-version } ] [ { \-U | \-\-user } USER ] [ \-\-port PORT ] [ { \-c | \-\-config } CONFIG ] [ { \-t | \-\-test } ] [ \-\-md5 ] BARMAN_HOST SERVER_NAME WAL_PATH .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp This script can be utilized in the \fBarchive_command\fP of a Postgres server to transfer WAL files to a Barman host using the \fBput\-wal\fP command (introduced in Barman 2.6). It establishes an SSH connection to the Barman host, enabling seamless integration of Barman within Postgres clusters for improved business continuity. .sp \fBExit Statuses\fP are: .INDENT 0.0 .IP \(bu 2 \fB0\fP for \fBSUCCESS\fP\&. .IP \(bu 2 \fBnon\-zero\fP for \fBFAILURE\fP\&. .UNINDENT .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP The server name configured in Barman for the Postgres server from which the WAL file is retrieved. .TP .B \fBBARMAN_HOST\fP The host of the Barman server. .TP .B \fBWAL_PATH\fP The value of the \(aq%p\(aq keyword (according to \fBarchive_command\fP). .TP .B \fB\-h\fP / \fB\-\-help\fP Display a help message and exit. .TP .B \fB\-V\fP / \fB\-\-version\fP Display the program\(aqs version number and exit. .TP .B \fB\-U\fP / \fB\-\-user\fP Specify the user for the SSH connection to the Barman server (defaults to \fBbarman\fP). .TP .B \fB\-\-port\fP Define the port used for the SSH connection to the Barman server. .TP .B \fB\-c\fP / \fB\-\-config\fP Specify the configuration file on the Barman server. .TP .B \fB\-t\fP / \fB\-\-test\fP Test the connection and configuration of the specified Postgres server in Barman to ensure it is ready to receive WAL files. This option ignores the mandatory argument \fBWAL_PATH\fP\&. .TP .B \fB\-\-md5\fP Use MD5 instead of SHA256 as the hash algorithm to calculate the checksum of the WAL file when transmitting it to the Barman server. This is used to maintain compatibility with older server versions, as older versions of Barman server used to support only MD5. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-cloud-backup-list.10000644000175100001660000001014215002177425021026 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CLOUD-BACKUP-LIST" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-cloud-backup-list \- Barman-cloud Commands .sp \fBSynopsis\fP .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-cloud\-backup\-list [ { \-V | \-\-version } ] [ \-\-help ] [ { { \-v | \-\-verbose } | { \-q | \-\-quiet } } ] [ { \-t | \-\-test } ] [ \-\-cloud\-provider { aws\-s3 | azure\-blob\-storage | google\-cloud\-storage } ] [ \-\-endpoint\-url ENDPOINT_URL ] [ { \-P | \-\-aws\-profile } AWS_PROFILE ] [ \-\-profile AWS_PROFILE ] [ \-\-read\-timeout READ_TIMEOUT ] [ { \-\-azure\-credential | \-\-credential } { azure\-cli | managed\-identity | default } ] [ \-\-format FORMAT ] SOURCE_URL SERVER_NAME .EE .UNINDENT .UNINDENT .sp \fBDescription\fP .sp This script lists backups stored in the cloud that were created using the \fBbarman\-cloud\-backup\fP command. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 For GCP, only authentication with \fBGOOGLE_APPLICATION_CREDENTIALS\fP env is supported. .UNINDENT .UNINDENT .sp \fBParameters\fP .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server that holds the backup to be listed. .TP .B \fBSOURCE_URL\fP URL of the cloud source, such as a bucket in AWS S3. For example: \fBs3://bucket/path/to/folder\fP\&. .TP .B \fB\-V\fP / \fB\-\-version\fP Show version and exit. .TP .B \fB\-\-help\fP show this help message and exit. .TP .B \fB\-v\fP / \fB\-\-verbose\fP Increase output verbosity (e.g., \fB\-vv\fP is more than \fB\-v\fP). .TP .B \fB\-q\fP / \fB\-\-quiet\fP Decrease output verbosity (e.g., \fB\-qq\fP is less than \fB\-q\fP). .TP .B \fB\-t\fP / \fB\-\-test\fP Test cloud connectivity and exit. .TP .B \fB\-\-cloud\-provider\fP The cloud provider to use as a storage backend. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBaws\-s3\fP\&. .IP \(bu 2 \fBazure\-blob\-storage\fP\&. .IP \(bu 2 \fBgoogle\-cloud\-storage\fP\&. .UNINDENT .TP .B \fB\-\-format\fP Output format (\fBconsole\fP or \fBjson\fP). Default \fBconsole\fP\&. .UNINDENT .sp \fBExtra options for the AWS cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-endpoint\-url\fP Override default S3 endpoint URL with the given one. .TP .B \fB\-P\fP / \fB\-\-aws\-profile\fP Profile name (e.g. \fBINI\fP section in AWS credentials file). .TP .B \fB\-\-profile\fP (deprecated) Profile name (e.g. \fBINI\fP section in AWS credentials file) \- replaced by \fB\-\-aws\-profile\fP\&. .TP .B \fB\-\-read\-timeout\fP The time in seconds until a timeout is raised when waiting to read from a connection (defaults to \fB60\fP seconds). .UNINDENT .sp \fBExtra options for the Azure cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-azure\-credential / \-\-credential\fP Optionally specify the type of credential to use when authenticating with Azure. If omitted then Azure Blob Storage credentials will be obtained from the environment and the default Azure authentication flow will be used for authenticating with all other Azure services. If no credentials can be found in the environment then the default Azure authentication flow will also be used for Azure Blob Storage. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBazure\-cli\fP\&. .IP \(bu 2 \fBmanaged\-identity\fP\&. .IP \(bu 2 \fBdefault\fP\&. .UNINDENT .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-cloud-backup-delete.10000644000175100001660000001411615002177425021322 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CLOUD-BACKUP-DELETE" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-cloud-backup-delete \- Barman-cloud Commands .sp \fBSynopsis\fP .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-cloud\-backup\-delete [ { \-V | \-\-version } ] [ \-\-help ] [ { { \-v | \-\-verbose } | { \-q | \-\-quiet } } ] [ { \-t | \-\-test } ] [ \-\-cloud\-provider { aws\-s3 | azure\-blob\-storage | google\-cloud\-storage } ] [ \-\-endpoint\-url ENDPOINT_URL ] [ { \-r | \-\-retention\-policy } RETENTION_POLICY ] [ { \-m | \-\-minimum\-redundancy } MINIMUM_REDUNDANCY ] [ { \-b | \-\-backup\-id } BACKUP_ID] [ \-\-dry\-run ] [ { \-P | \-\-aws\-profile } AWS_PROFILE ] [ \-\-profile AWS_PROFILE ] [ \-\-read\-timeout READ_TIMEOUT ] [ { \-\-azure\-credential | \-\-credential } { azure\-cli | managed\-identity | default } ] [\-\-batch\-size DELETE_BATCH_SIZE] SOURCE_URL SERVER_NAME .EE .UNINDENT .UNINDENT .sp \fBDescription\fP .sp The \fBbarman\-cloud\-backup\-delete\fP script is used to delete one or more backups created with the \fBbarman\-cloud\-backup\fP command from cloud storage and to remove the associated WAL files. .sp Backups can be specified for deletion either by their backup ID (as obtained from \fBbarman\-cloud\-backup\-list\fP) or by a retention policy. Retention policies mirror those used by the Barman server, deleting all backups that are not required to meet the specified policy. When a backup is deleted, any unused WAL files associated with that backup are also removed. .sp WALs are considered unused if: .INDENT 0.0 .IP \(bu 2 The WALs predate the begin_wal value of the oldest remaining backup. .IP \(bu 2 The WALs are not required by any archival backups stored in the cloud. .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 For GCP, only authentication with \fBGOOGLE_APPLICATION_CREDENTIALS\fP env is supported. .UNINDENT .UNINDENT .sp \fBIMPORTANT:\fP .INDENT 0.0 .INDENT 3.5 Each backup deletion involves three separate requests to the cloud provider: one for the backup files, one for the \fBbackup.info\fP file, and one for the associated WALs. Deleting by retention policy may result in a high volume of delete requests if a large number of backups are accumulated in cloud storage. .UNINDENT .UNINDENT .sp \fBParameters\fP .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server that holds the backup to be deleted. .TP .B \fBSOURCE_URL\fP URL of the cloud source, such as a bucket in AWS S3. For example: \fBs3://bucket/path/to/folder\fP\&. .TP .B \fB\-V\fP / \fB\-\-version\fP Show version and exit. .TP .B \fB\-\-help\fP show this help message and exit. .TP .B \fB\-v\fP / \fB\-\-verbose\fP Increase output verbosity (e.g., \fB\-vv\fP is more than \fB\-v\fP). .TP .B \fB\-q\fP / \fB\-\-quiet\fP Decrease output verbosity (e.g., \fB\-qq\fP is less than \fB\-q\fP). .TP .B \fB\-t\fP / \fB\-\-test\fP Test cloud connectivity and exit. .TP .B \fB\-\-cloud\-provider\fP The cloud provider to use as a storage backend. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBaws\-s3\fP\&. .IP \(bu 2 \fBazure\-blob\-storage\fP\&. .IP \(bu 2 \fBgoogle\-cloud\-storage\fP\&. .UNINDENT .TP .B \fB\-b\fP / \fB\-\-backup\-id\fP ID of the backup to be deleted .TP .B \fB\-m\fP / \fB\-\-minimum\-redundancy\fP The minimum number of backups that should always be available. .TP .B \fB\-r\fP / \fB\-\-retention\-policy\fP If specified, delete all backups eligible for deletion according to the supplied retention policy. .sp Syntax: \fBREDUNDANCY value | RECOVERY WINDOW OF value { DAYS | WEEKS | MONTHS }\fP .TP .B \fB\-\-batch\-size\fP The maximum number of objects to be deleted in a single request to the cloud provider. If unset then the maximum allowed batch size for the specified cloud provider will be used (\fB1000\fP for aws\-s3, \fB256\fP for azure\-blob\-storage and \fB100\fP for google\-cloud\-storage). .TP .B \fB\-\-dry\-run\fP Find the objects which need to be deleted but do not delete them. .UNINDENT .sp \fBExtra options for the AWS cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-endpoint\-url\fP Override default S3 endpoint URL with the given one. .TP .B \fB\-P\fP / \fB\-\-aws\-profile\fP Profile name (e.g. \fBINI\fP section in AWS credentials file). .TP .B \fB\-\-profile\fP (deprecated) Profile name (e.g. \fBINI\fP section in AWS credentials file) \- replaced by \fB\-\-aws\-profile\fP\&. .TP .B \fB\-\-read\-timeout\fP The time in seconds until a timeout is raised when waiting to read from a connection (defaults to \fB60\fP seconds). .UNINDENT .sp \fBExtra options for the Azure cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-azure\-credential / \-\-credential\fP Optionally specify the type of credential to use when authenticating with Azure. If omitted then Azure Blob Storage credentials will be obtained from the environment and the default Azure authentication flow will be used for authenticating with all other Azure services. If no credentials can be found in the environment then the default Azure authentication flow will also be used for Azure Blob Storage. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBazure\-cli\fP\&. .IP \(bu 2 \fBmanaged\-identity\fP\&. .IP \(bu 2 \fBdefault\fP\&. .UNINDENT .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-cloud-check-wal-archive.10000644000175100001660000001043015002177425022065 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CLOUD-CHECK-WAL-ARCHIVE" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-cloud-check-wal-archive \- Barman-cloud Commands .sp \fBSynopsis\fP .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-cloud\-check\-wal\-archive [ { \-V | \-\-version } ] [ \-\-help ] [ { { \-v | \-\-verbose } | { \-q | \-\-quiet } } ] [ { \-t | \-\-test } ] [ \-\-cloud\-provider { aws\-s3 | azure\-blob\-storage | google\-cloud\-storage } ] [ \-\-endpoint\-url ENDPOINT_URL ] [ { \-P | \-\-aws\-profile } AWS_PROFILE ] [ \-\-profile AWS_PROFILE ] [ \-\-read\-timeout READ_TIMEOUT ] [ { \-\-azure\-credential | \-\-credential } { azure\-cli | managed\-identity | default } ] [ \-\-timeline TIMELINE ] DESTINATION_URL SERVER_NAME .EE .UNINDENT .UNINDENT .sp \fBDescription\fP .sp Verify that the WAL archive destination for a server is suitable for use with a new Postgres cluster. By default, the check will succeed if the WAL archive is empty or if the target bucket is not found. Any other conditions will result in a failure. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 For GCP, only authentication with \fBGOOGLE_APPLICATION_CREDENTIALS\fP env is supported. .UNINDENT .UNINDENT .sp \fBParameters\fP .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server that needs to be checked. .TP .B \fBDESTINATION_URL\fP URL of the cloud destination, such as a bucket in AWS S3. For example: \fBs3://bucket/path/to/folder\fP\&. .TP .B \fB\-V\fP / \fB\-\-version\fP Show version and exit. .TP .B \fB\-\-help\fP show this help message and exit. .TP .B \fB\-v\fP / \fB\-\-verbose\fP Increase output verbosity (e.g., \fB\-vv\fP is more than \fB\-v\fP). .TP .B \fB\-q\fP / \fB\-\-quiet\fP Decrease output verbosity (e.g., \fB\-qq\fP is less than \fB\-q\fP). .TP .B \fB\-t\fP / \fB\-\-test\fP Test cloud connectivity and exit. .TP .B \fB\-\-cloud\-provider\fP The cloud provider to use as a storage backend. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBaws\-s3\fP\&. .IP \(bu 2 \fBazure\-blob\-storage\fP\&. .IP \(bu 2 \fBgoogle\-cloud\-storage\fP\&. .UNINDENT .TP .B \fB\-\-timeline\fP The earliest timeline whose WALs should cause the check to fail. .UNINDENT .sp \fBExtra options for the AWS cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-endpoint\-url\fP Override default S3 endpoint URL with the given one. .TP .B \fB\-P\fP / \fB\-\-aws\-profile\fP Profile name (e.g. \fBINI\fP section in AWS credentials file). .TP .B \fB\-\-profile\fP (deprecated) Profile name (e.g. \fBINI\fP section in AWS credentials file) \- replaced by \fB\-\-aws\-profile\fP\&. .TP .B \fB\-\-read\-timeout\fP The time in seconds until a timeout is raised when waiting to read from a connection (defaults to \fB60\fP seconds). .UNINDENT .sp \fBExtra options for the Azure cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-azure\-credential / \-\-credential\fP Optionally specify the type of credential to use when authenticating with Azure. If omitted then Azure Blob Storage credentials will be obtained from the environment and the default Azure authentication flow will be used for authenticating with all other Azure services. If no credentials can be found in the environment then the default Azure authentication flow will also be used for Azure Blob Storage. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBazure\-cli\fP\&. .IP \(bu 2 \fBmanaged\-identity\fP\&. .IP \(bu 2 \fBdefault\fP\&. .UNINDENT .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-cloud-wal-archive.10000644000175100001660000001656415002177425021030 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CLOUD-WAL-ARCHIVE" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-cloud-wal-archive \- Barman-cloud Commands .sp \fBSynopsis\fP .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-cloud\-wal\-archive [ { \-V | \-\-version } ] [ \-\-help ] [ { { \-v | \-\-verbose } | { \-q | \-\-quiet } } ] [ { \-t | \-\-test } ] [ \-\-cloud\-provider { aws\-s3 | azure\-blob\-storage | google\-cloud\-storage } ] [ { { \-z | \-\-gzip } | { \-j | \-\-bzip2 } | \-\-xz | \-\-snappy | \-\-zstd | \-\-lz4 } ] [ \-\-tags TAG [ TAG ... ] ] [ \-\-history\-tags HISTORY_TAG [ HISTORY_TAG ... ] ] [ \-\-endpoint\-url ENDPOINT_URL ] [ { \-P | \-\-aws\-profile } AWS_PROFILE ] [ \-\-profile AWS_PROFILE ] [ \-\-read\-timeout READ_TIMEOUT ] [ { \-e | \-\-encryption } ENCRYPTION ] [ \-\-sse\-kms\-key\-id SSE_KMS_KEY_ID ] [ { \-\-azure\-credential | \-\-credential } { azure\-cli | managed\-identity | default } ] [ \-\-encryption\-scope ENCRYPTION_SCOPE ] [ \-\-max\-block\-size MAX_BLOCK_SIZE ] [ \-\-max\-concurrency MAX_CONCURRENCY ] [ \-\-max\-single\-put\-size MAX_SINGLE_PUT_SIZE ] [ \-\-kms\-key\-name KMS_KEY_NAME ] DESTINATION_URL SERVER_NAME [ WAL_PATH ] .EE .UNINDENT .UNINDENT .sp \fBDescription\fP .sp The \fBbarman\-cloud\-wal\-archive\fP command is designed to be used in the \fBarchive_command\fP of a Postgres server to directly ship WAL files to cloud storage. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 If you are using Python 2 or unsupported versions of Python 3, avoid using the compression options \fB\-\-gzip\fP or \fB\-\-bzip2\fP\&. The script cannot restore gzip\-compressed WALs on Python < 3.2 or bzip2\-compressed WALs on Python < 3.3. .UNINDENT .UNINDENT .sp This script enables the direct transfer of WAL files to cloud storage, bypassing the Barman server. Additionally, it can be utilized as a hook script for WAL archiving (pre_archive_retry_script). .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 For GCP, only authentication with \fBGOOGLE_APPLICATION_CREDENTIALS\fP env is supported. .UNINDENT .UNINDENT .sp \fBParameters\fP .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server that will have the WALs archived. .TP .B \fBDESTINATION_URL\fP URL of the cloud destination, such as a bucket in AWS S3. For example: \fBs3://bucket/path/to/folder\fP\&. .TP .B \fBWAL_PATH\fP The value of the \(aq%p\(aq keyword (according to \fBarchive_command\fP). .TP .B \fB\-V\fP / \fB\-\-version\fP Show version and exit. .TP .B \fB\-\-help\fP show this help message and exit. .TP .B \fB\-v\fP / \fB\-\-verbose\fP Increase output verbosity (e.g., \fB\-vv\fP is more than \fB\-v\fP). .TP .B \fB\-q\fP / \fB\-\-quiet\fP Decrease output verbosity (e.g., \fB\-qq\fP is less than \fB\-q\fP). .TP .B \fB\-t\fP / \fB\-\-test\fP Test cloud connectivity and exit. .TP .B \fB\-\-cloud\-provider\fP The cloud provider to use as a storage backend. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBaws\-s3\fP\&. .IP \(bu 2 \fBazure\-blob\-storage\fP\&. .IP \(bu 2 \fBgoogle\-cloud\-storage\fP\&. .UNINDENT .TP .B \fB\-z\fP / \fB\-\-gzip\fP gzip\-compress the WAL while uploading to the cloud. .TP .B \fB\-j\fP / \fB\-\-bzip2\fP bzip2\-compress the WAL while uploading to the cloud. .TP .B \fB\-\-xz\fP xz\-compress the WAL while uploading to the cloud. .TP .B \fB\-\-snappy\fP snappy\-compress the WAL while uploading to the cloud (requires optional \fBpython\-snappy\fP library). .TP .B \fB\-\-zstd\fP zstd\-compress the WAL while uploading to the cloud (requires optional \fBzstandard\fP library). .TP .B \fB\-\-lz4\fP lz4\-compress the WAL while uploading to the cloud (requires optional \fBlz4\fP library). .TP .B \fB\-\-tags\fP Tags to be added to archived WAL files in cloud storage. .TP .B \fB\-\-history\-tags\fP Tags to be added to archived history files in cloud storage. .UNINDENT .sp \fBExtra options for the AWS cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-endpoint\-url\fP Override default S3 endpoint URL with the given one. .TP .B \fB\-P\fP / \fB\-\-aws\-profile\fP Profile name (e.g. \fBINI\fP section in AWS credentials file). .TP .B \fB\-\-profile\fP (deprecated) Profile name (e.g. \fBINI\fP section in AWS credentials file) \- replaced by \fB\-\-aws\-profile\fP\&. .TP .B \fB\-\-read\-timeout\fP The time in seconds until a timeout is raised when waiting to read from a connection (defaults to \fB60\fP seconds). .TP .B \fB\-e\fP / \fB\-\-encryption\fP The encryption algorithm used when storing the uploaded data in S3. .sp Allowed options: .INDENT 7.0 .IP \(bu 2 \fBAES256\fP\&. .IP \(bu 2 \fBaws:kms\fP\&. .UNINDENT .TP .B \fB\-\-sse\-kms\-key\-id\fP The AWS KMS key ID that should be used for encrypting the uploaded data in S3. Can be specified using the key ID on its own or using the full ARN for the key. Only allowed if \fB\-e\fP / \fB\-\-encryption\fP is set to \fBaws:kms\fP\&. .UNINDENT .sp \fBExtra options for the Azure cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-azure\-credential / \-\-credential\fP Optionally specify the type of credential to use when authenticating with Azure. If omitted then Azure Blob Storage credentials will be obtained from the environment and the default Azure authentication flow will be used for authenticating with all other Azure services. If no credentials can be found in the environment then the default Azure authentication flow will also be used for Azure Blob Storage. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBazure\-cli\fP\&. .IP \(bu 2 \fBmanaged\-identity\fP\&. .IP \(bu 2 \fBdefault\fP\&. .UNINDENT .TP .B \fB\-\-encryption\-scope\fP The name of an encryption scope defined in the Azure Blob Storage service which is to be used to encrypt the data in Azure. .TP .B \fB\-\-max\-block\-size\fP The chunk size to be used when uploading an object via the concurrent chunk method (default: \fB4MB\fP). .TP .B \fB\-\-max\-concurrency\fP The maximum number of chunks to be uploaded concurrently (default: \fB1\fP). .TP .B \fB\-\-max\-single\-put\-size\fP Maximum size for which the Azure client will upload an object in a single request (default: \fB64MB\fP). If this is set lower than the Postgres WAL segment size after any applied compression then the concurrent chunk upload method for WAL archiving will be used. .UNINDENT .sp \fBExtra options for GCP cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-kms\-key\-name\fP The name of the GCP KMS key which should be used for encrypting the uploaded data in GCS. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-sync-backup.10000644000175100001660000000437415002177425017735 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-SYNC-BACKUP" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-sync-backup \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX sync\-backup [ { \-h | \-\-help } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp This command synchronizes a passive node with its primary by copying all files from a backup present on the server node. It is available only for passive nodes and uses the \fBprimary_ssh_command\fP option to establish a secure connection with the primary node. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH SHORTCUTS .sp For some commands, instead of using the timestamp backup ID, you can use the following shortcuts or aliases to identify a backup for a given server: .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-sync-info.10000644000175100001660000000363015002177425017415 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-SYNC-INFO" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-sync-info \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX sync\-info [ { \-h | \-\-help } ] [ \-\-primary ] SERVER_NAME [ LAST_WAL [ LAST_POS ] ] .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Gather information about the current status of a Barman server for synchronization purposes. .sp This command returns a JSON output for a server that includes: all successfully completed backups, all archived WAL files, the configuration, the last WAL file read from \fBxlog.db\fP, and its position within the file. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBLAST_WAL\fP Instructs sync\-info to skip any WAL files that precede the specified file (for incremental synchronization). .TP .B \fBLAST_POS\fP Hint for quickly positioning in the \fBxlog.db\fP file (for incremental synchronization). .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-primary\fP Execute the sync\-info on the primary node (if set). .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-switch-xlog.10000644000175100001660000000172415002177425017762 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-SWITCH-XLOG" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-switch-xlog \- Barman Sub-Commands .SH DESCRIPTION .sp Alias for the \fBswitch\-wal\fP command. .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-diagnose.10000644000175100001660000000316315002177425017302 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-DIAGNOSE" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-diagnose \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX diagnose [ { \-h | \-\-help } ] [ \-\-show\-config\-source ] .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Display diagnostic information about the Barman node, which is the server where Barman is installed, as well as all configured Postgres servers. This includes details such as global configuration, SSH version, Python version, rsync version, the current configuration and status of all servers, and many more. .SH PARAMETERS .INDENT 0.0 .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-show\-config\-source\fP Include the source file which provides the effective value for each configuration option. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman.50000644000175100001660000013304315002177425015520 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN" "5" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman \- Barman Configurations .sp Barman follows a convention over configuration approach, which simplifies configuration by allowing some options to be defined globally and overridden at the server level. This means you can set a default behavior for your servers and then customize specific servers as needed. This design reduces the need for excessive configuration while maintaining flexibility. .SH USAGE .sp Proper configuration is critical for its effective operation. Barman uses different types of configuration files to manage global settings, server\-specific settings, and model\-specific settings that is made up of three scopes: .sp 1. \fBGlobal Configuration\fP: It comprises one file with a set of configurations for the barman system, such as the main directory, system user, log file, and other general options. Default location is \fB/etc/barman.conf\fP and it can be overridden on a per\-user level by \fB~/.barman.conf\fP or by specifying a \fB\&.conf\fP file using the \fB\-c\fP / \fB\-\-config\fP with the \fI\%barman command\fP directly in the CLI. .sp 2. \fBServer Configuration\fP: It comprises one or more files with a set of configurations for a Postgres server that you want to keep track and interact for backup, recovery and/or replication. Default location is \fB/etc/barman.d\fP and must use the \fB\&.conf\fP suffix. You may have one or multiple files for servers. You can override the default location by setting the \fBconfiguration_files_directory\fP option in the global configuration file and placing the files in that particular location. .sp 3. \fBModel Configuration\fP: It comprises one or more files with a set of configurations overrides that can be applied to Barman servers within the same cluster as the model. These overrides can be implemented using the barman \fBconfig\-switch\fP command. Default location is \fB/etc/barman.d\fP and must use the \fB\&.conf\fP suffix. The same \fBconfiguration_files_directory\fP override option from the server configuration applies for models. You may have one or multiple files for models. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Historically, you could have a single configuration file containing global, server, and model options, but, for maintenance reasons, this approach is deprecated. .UNINDENT .UNINDENT .sp Configuration files follow the \fBINI\fP format and consist of sections denoted by headers in square brackets. Each section can include various options. .sp Models and servers must have unique identifiers, and reserved words cannot be used as names. .sp \fBReserved Words\fP .sp The following reserved words cannot be used as server or model names: .INDENT 0.0 .IP \(bu 2 \fBbarman\fP: Identifies the global section. .IP \(bu 2 \fBall\fP: A special shortcut for executing commands on all managed servers. .UNINDENT .sp \fBParameter Types\fP .sp Configuration options can be of the following types: .INDENT 0.0 .IP \(bu 2 \fBString\fP: Textual data (e.g., file paths, names). .IP \(bu 2 \fBEnum\fP: Enumerated values, often limited to predefined choices. .IP \(bu 2 \fBInteger\fP: Numeric values. .IP \(bu 2 \fBBoolean\fP: Can be \fBon\fP, \fBtrue\fP, \fB1\fP (true) or \fBoff\fP, \fBfalse\fP, \fB0\fP (false). .sp \fBNOTE:\fP .INDENT 2.0 .INDENT 3.5 Some enums allow \fBoff\fP, but not \fBfalse\fP\&. .UNINDENT .UNINDENT .UNINDENT .SH OPTIONS .sp Options in the configuration files can have specific or shared scopes. The following configuration options are used not only for configuring how Barman will execute backups and recoveries, but also for configuring various aspects of how Barman interacts with the configured Postgres servers to be able to apply your Backup and Recovery, and High\-Availability strategies. .SS General .sp These are general configurations options. .sp \fBactive\fP .sp When this option is set to \fBtrue\fP (default), the server operates fully. If set to \fBfalse\fP, the server is restricted to diagnostic use only, meaning that operational commands such as backup execution or WAL archiving are temporarily disabled. When incorporating a new server into Barman, we recommend initially setting \fBactive = false\fP\&. Verify that barman check shows no issues before activating the server. This approach helps prevent excessive error logging in Barman during the initial setup. .sp Scope: Server / Model. .sp \fBarchiver\fP .sp This option enables log file shipping through Postgres\(aq \fBarchive_command\fP for a server. When set to \fBtrue\fP, Barman expects continuous archiving to be configured and will manage WAL files that Postgres stores in the incoming directory (\fBincoming_wals_directory\fP), including their checks, handling, and compression. When set to \fBfalse\fP (default), continuous archiving is disabled. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 If neither \fBarchiver\fP nor \fBstreaming_archiver\fP is configured, Barman will automatically set this option to \fBtrue\fP to maintain compatibility with the previous default behavior where archiving was enabled by default. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBarchiver_batch_size\fP .sp This option enables batch processing of WAL files for the archiver process by setting it to a value greater than \fB0\fP\&. If not set, the archiver will use unlimited (default) processing mode for the WAL queue. With batch processing enabled, the archiver process will handle a maximum of \fBarchiver_batch_size\fP WAL segments per run. This value must be an integer. .sp Scope: Global / Server / Model. .sp \fBbandwidth_limit\fP .sp Specifies the maximum transfer rate in kilobytes per second for backup and recovery operations. A value of \fB0\fP indicates no limit (default). .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Applies only when \fBbackup_method = postgres | rsync\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBbarman_home\fP .sp Designates the main data directory for Barman. Defaults to \fB/var/lib/barman\fP\&. .sp Scope: Global. .sp \fBbarman_lock_directory\fP .sp Specifies the directory for lock files. The default is \fBbarman_home\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 The \fBbarman_lock_directory\fP should be on a non\-network local filesystem. .UNINDENT .UNINDENT .sp Scope: Global. .sp \fBcheck_timeout\fP .sp Sets the maximum execution time in seconds for a Barman check command per server. Set to \fB0\fP to disable the timeout. Default is \fB30\fP seconds. Must be a non\-negative integer. .sp Scope: Global / Server / Model. .sp \fBcluster\fP .sp Tag the server or model to an associated cluster name. Barman uses this association to override configuration for all servers/models in this cluster. If omitted for servers, it defaults to the server\(aqs name. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Must be specified for configuration models to group applicable servers. .UNINDENT .UNINDENT .sp Scope: Server / Model. .sp \fBconfig_changes_queue\fP .sp Designates the filesystem location for Barman\(aqs queue that handles configuration changes requested via the barman \fBconfig\-update\fP command. This queue manages the serialization and retry of configuration change requests. By default, Barman writes to a file named \fBcfg_changes.queue\fP under \fBbarman_home\fP\&. .sp Scope: Global. .sp \fBconfiguration_files_directory\fP .sp Designates the directory where server/model configuration files will be read by Barman. Defaults to \fB/etc/barman.d/\fP\&. .sp Scope: Global. .sp \fBconninfo\fP .sp Specifies the connection string used by Barman to connect to the Postgres server. This is a libpq connection string. Commonly used keys include: \fBhost\fP, \fBhostaddr\fP, \fBport\fP, \fBdbname\fP, \fBuser\fP and \fBpassword\fP\&. See the \X'tty: link https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING'\fI\%PostgreSQL documentation\fP\X'tty: link' for details. .sp Scope: Server / Model. .sp \fBcreate_slot\fP .sp Determines whether Barman should automatically create a replication slot if it\(aqs not already present for streaming WAL files. When set to \fBauto\fP and \fBslot_name\fP is defined, Barman will attempt to create the slot automatically. When set to \fBmanual\fP (default), the replication slot must be created manually. .sp Scope: Global / Server / Model. .sp \fBdescription\fP .sp Provides a human\-readable description of a server. .sp Scope: Server / Model. .sp \fBerrors_directory\fP .sp The directory where WAL files that were errored while being archived by Barman are stored. This includes duplicate WAL files (e.g., an archived WAL file that has already been streamed but have different hash) and unexpected files found in the WAL archive directory. .sp The purpose of placing the files in this directory is so someone can later review why they failed to be archived and take appropriate actions (dispose of, store somewhere else, replace the duplicate file archived before, etc.) .sp Scope: Server. .sp \fBforward_config_path\fP .sp Determines whether a passive node should forward its configuration file path to its primary node during \fBcron\fP or \fBsync\-info\fP commands. Set to \fBtrue\fP if Barman is invoked with the \fB\-c\fP / \fB\-\-config\fP option and the configuration paths are identical on both passive and primary Barman servers. Defaults to \fBfalse\fP\&. .sp Scope: Global / Server / Model. .sp \fBimmediate_checkpoint\fP .sp Controls how Postgres handles checkpoints at the start of a backup. Set to \fBfalse\fP (default) to allow the checkpoint to complete according to \fBcheckpoint_completion_target\fP\&. Set to \fBtrue\fP for an immediate checkpoint, where Postgres completes the checkpoint as quickly as possible. .sp Scope: Global / Server / Model. .sp \fBkeepalive_interval\fP .sp Sets the interval in seconds for sending a heartbeat query to keep the libpq connection active during an rsync backup. Default is \fB60\fP seconds. Setting this to \fB0\fP disables the heartbeat. .sp Scope: Global / Server / Model. .sp \fBlock_directory_cleanup\fP .sp Enables automatic cleanup of unused lock files in the \fBbarman_lock_directory\fP\&. .sp Scope: Global. .sp \fBlog_file\fP .sp Specifies the location of Barman\(aqs log file. Defaults to \fB/var/log/barman/barman.log\fP\&. .sp Scope: Global. .sp \fBlog_level\fP .sp Sets the level of logging. Options include: \fBDEBUG\fP, \fBINFO\fP, \fBWARNING\fP, \fBERROR\fP and \fBCRITICAL\fP\&. .sp Scope: Global. .sp \fBminimum_redundancy\fP .sp Specifies the minimum number of backups to retain. Default is \fB0\fP\&. .sp Scope: Global / Server / Model. .sp \fBmodel\fP .sp When set to \fBtrue\fP, turns a server section from a configuration file into a model for a cluster. There is no \fBfalse\fP option in this case. If you want to simulate a \fBfalse\fP option, comment out (\fB#model=true\fP) or remove the option in the configuration. Defaults to the server name. .sp Scope: Model. .sp \fBnetwork_compression\fP .sp Enables or disables data compression for network transfers. Set to \fBfalse\fP (default) to disable compression, or \fBtrue\fP to enable it and reduce network usage. .sp Scope: Global / Server / Model. .sp \fBparallel_jobs\fP .sp Controls the number of parallel workers used to copy files during backup or recovery. It must be a positive integer. Default is \fB1\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Applies only when \fBbackup_method = rsync\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBparallel_jobs_start_batch_period\fP .sp Specifies the time interval in seconds over which a single batch of parallel jobs will start. Default is \fB1\fP second. This means that if \fBparallel_jobs_start_batch_size\fP is \fB10\fP and \fBparallel_jobs_start_batch_period\fP is \fB1\fP, this will yield an effective rate limit of \fB10\fP jobs per second, because there is a maximum of \fB10\fP jobs that can be started within \fB1\fP second. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Applies only when \fBbackup_method = rsync\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBparallel_jobs_start_batch_size\fP .sp Defines the maximum number of parallel jobs to start in a single batch. Default is \fB10\fP jobs. This means that if \fBparallel_jobs_start_batch_size\fP is \fB10\fP and \fBparallel_jobs_start_batch_period\fP is \fB2\fP, this will yield a maximum of \fB10\fP jobs that can be started within \fB2\fP seconds. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Applies only when \fBbackup_method = rsync\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBpath_prefix\fP .sp Lists one or more absolute paths, separated by colons, where Barman looks for executable files. These paths are checked before the \fBPATH\fP environment variable. This option can be set for each server and needs to point to the \fBbin\fP directory for the appropriate \fBPG_MAJOR_VERSION\fP\&. .sp Scope: Global / Server / Model. .sp \fBprimary_checkpoint_timeout\fP .sp Time to wait for new WAL files before forcing a checkpoint on the primary server. Defaults to \fB0\fP\&. .sp Scope: Server / Model. .sp \fBprimary_conninfo\fP .sp Connection string for Barman to connect to the primary Postgres server during a standby backup. .sp Scope: Server / Model. .sp \fBprimary_ssh_command\fP .sp SSH command for connecting to the primary Barman server if Barman is passive. .sp Scope: Global / Server / Model. .sp \fBslot_name\fP .sp Replication slot name for the \fBreceive\-wal\fP command when \fBstreaming_archiver\fP is enabled. .sp Scope: Global / Server / Model. .sp \fBssh_command\fP .sp SSH command used by Barman to connect to the Postgres server for rsync backups. .sp Scope: Server / Model. .sp \fBstreaming_archiver\fP .sp Enables Postgres\(aq streaming protocol for WAL files. Defaults to \fBfalse\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 If neither \fBarchiver\fP nor \fBstreaming_archiver\fP is configured, Barman will automatically set \fBarchiver\fP option to \fBtrue\fP to maintain compatibility with the previous default behavior where archiving was enabled by default. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBstreaming_archiver_batch_size\fP .sp Batch size for processing WAL files in streaming archiver. Defaults to \fB0\fP\&. .sp Scope: Global / Server / Model. .sp \fBstreaming_archiver_name\fP .sp Application name for the \fBreceive\-wal\fP command. Defaults to \fBbarman_receive_wal\fP\&. .sp Scope: Global / Server / Model. .sp \fBstreaming_backup_name\fP .sp Application name for the \fBpg_basebackup\fP command. Defaults to \fBbarman_streaming_backup\fP\&. .sp Scope: Global / Server / Model. .sp \fBstreaming_conninfo\fP .sp Connection string for streaming replication protocol. Defaults to \fBconninfo\fP\&. .sp Scope: Server / Model. .sp \fBtablespace_bandwidth_limit\fP .sp Maximum transfer rate for specific tablespaces for backup and recovery operations. A value of \fB0\fP indicates no limit (default). .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Applies only when \fBbackup_method = rsync\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .SS Backups .sp These configurations options are related to how Barman will execute backups. .sp \fBautogenerate_manifest\fP .sp This is a boolean option that allows for the automatic creation of backup manifest files. The manifest file, which is a JSON document, lists all files included in the backup. It is generated upon completion of the backup and saved in the backup directory. The format of the manifest file adheres to the specifications outlined in the \X'tty: link https://www.postgresql.org/docs/current/backup-manifest-format.html'\fI\%PostgreSQL documentation\fP\X'tty: link' and is compatible with the \fBpg_verifybackup\fP tool. Default is \fBfalse\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 This option is ignored if the \fBbackup_method\fP is not \fBrsync\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBbackup_compression\fP .sp Specifies the compression method for the backup process. It can be set to \fBgzip\fP, \fBlz4\fP, \fBzstd\fP, or \fBnone\fP\&. Ensure that the CLI tool for the chosen compression method is available on both the Barman and Postgres servers. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Note that \fBlz4\fP and \fBzstd\fP require Postgres version 15 or later. Unsetting this option or using \fBnone\fP results in an uncompressed archive (default). Only supported when \fBbackup_method = postgres\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBbackup_compression_format\fP .sp Determines the format \fBpg_basebackup\fP should use when saving compressed backups. Options are \fBplain\fP or \fBtar\fP, with \fBtar\fP as the default if unset. The \fBplain\fP format is available only if Postgres version 15 or later is in use and \fBbackup_compression_location\fP is set to \fBserver\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = postgres\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBbackup_compression_level\fP .sp Defines the level of compression for backups as an integer. The permissible values depend on the compression method specified in \fBbackup_compression\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = postgres\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBbackup_compression_location\fP .sp Specifies where compression should occur during the backup: either \fBclient\fP or \fBserver\fP\&. The \fBserver\fP option is available only if Postgres version 15 or later is being used. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = postgres\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBbackup_compression_workers\fP .sp Sets the number of threads used for compression during the backup process. This is applicable only when \fBbackup_compression=zstd\fP\&. The default value is 0, which uses the standard compression behavior. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = postgres\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBbackup_directory\fP .sp Specifies the directory where backup data for a server will be stored. Defaults to \fB/\fP\&. .sp Scope: Server. .sp \fBbackup_method\fP .sp Defines the method Barman uses to perform backups. Options include: .INDENT 0.0 .IP \(bu 2 \fBrsync\fP (default): Executes backups using the rsync command over SSH (requires \fBssh_command\fP). .IP \(bu 2 \fBpostgres\fP: Uses the \fBpg_basebackup\fP command for backups. .IP \(bu 2 \fBlocal\-rsync\fP: Assumes Barman runs on the same server and as the same user as the Postgres database, performing an rsync file system copy. .IP \(bu 2 \fBsnapshot\fP: Utilizes the API of the cloud provider specified in the \fBsnapshot_provider\fP option to create disk snapshots as defined in \fBsnapshot_disks\fP and saves only the backup label and metadata to its own storage. .UNINDENT .sp Scope: Global / Server / Model. .sp \fBbackup_options\fP .sp Controls how Barman interacts with Postgres during backups. This is a comma\-separated list that can include: .INDENT 0.0 .IP \(bu 2 \fBconcurrent_backup\fP (default): Uses concurrent backup, recommended for Postgres versions 9.6 and later, and supports backups from standby servers. .IP \(bu 2 \fBexclusive_backup\fP: Uses the deprecated exclusive backup method. Only for Postgres versions older than 15. This option will be removed in the future. .IP \(bu 2 \fBexternal_configuration\fP: Suppresses warnings about external configuration files during backup execution. .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 \fBexclusive_backup\fP and \fBconcurrent_backup\fP cannot be used together. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBbasebackups_directory\fP .sp Specifies the directory where base backups are stored. Defaults to \fB/base\fP\&. .sp Scope: Server. .sp \fBbasebackup_retry_sleep\fP .sp Sets the number of seconds to wait after a failed base backup copy before retrying. Default is \fB30\fP seconds. Must be a non\-negative integer. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 This applies to both backup and recovery operations. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBbasebackup_retry_times\fP .sp Defines the number of retry attempts for a base backup copy after an error occurs. Default is \fB0\fP (no retries). Must be a non\-negative integer. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 This applies to both backup and recovery operations. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBreuse_backup\fP .sp Controls incremental backup support when using \fBbackup_method=rsync\fP by reusing the last available backup. The options are: .INDENT 0.0 .IP \(bu 2 \fBoff\fP (default): Standard full backup. .IP \(bu 2 \fBcopy\fP: File\-level incremental backup, by reusing the last backup for a server and creating a copy of the unchanged files (just for backup time reduction) .IP \(bu 2 \fBlink\fP: File\-level incremental backup, by reusing the last backup for a server and creating a hard link of the unchanged files (for backup space and time reduction) .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 This option will be ignored when \fBbackup_method=postgres\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBworm_mode\fP .sp If set to \fBon\fP, enables support for WORM (Write Once Read Many) storage, allowing Barman to handle backups on immutable storage correctly. Default is \fBoff\fP\&. .sp Scope: Global / Server / Model. .SS Cloud Backups .sp These configuration options are related to how Barman will execute backups in the cloud. .sp \fBaws_await_snapshots_timeout\fP .sp Specifies the duration in seconds to wait for AWS snapshots to be created before a timeout occurs. The default value is \fB3600\fP seconds. This must be a positive integer. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = snapshot\fP and \fBsnapshot_provider = aws\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBaws_profile\fP .sp The name of the AWS profile to use when authenticating with AWS (e.g. \fBINI\fP section in AWS credentials file). .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = snapshot\fP and \fBsnapshot_provider = aws\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBaws_region\fP .sp Indicates the AWS region where the EC2 VM and storage volumes, as defined by \fBsnapshot_instance\fP and \fBsnapshot_disks\fP, are located. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = snapshot\fP and \fBsnapshot_provider = aws\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBaws_snapshot_lock_mode\fP .sp The lock mode for the snapshot. This is only valid if \fBsnapshot_instance\fP and \fBsnapshot_disk\fP are set. .sp Allowed options: .INDENT 0.0 .IP \(bu 2 \fBcompliance\fP\&. .IP \(bu 2 \fBgovernance\fP\&. .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = snapshot\fP and \fBsnapshot_provider = aws\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBaws_snapshot_lock_duration\fP .sp The lock duration is the period of time (in days) for which the snapshot is to remain locked, ranging from 1 to 36,500. Set either the lock duration or the expiration date (not both). .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = snapshot\fP and \fBsnapshot_provider = aws\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBaws_snapshot_lock_cool_off_period\fP .sp The cooling\-off period is an optional period of time (in hours) that you can specify when you lock a snapshot in \fBcompliance\fP mode, ranging from 1 to 72. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = snapshot\fP and \fBsnapshot_provider = aws\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBaws_snapshot_lock_expiration_date\fP .sp The lock duration is determined by an expiration date in the future. It must be at least 1 day after the snapshot creation date and time, using the format \fBYYYY\-MM\-DDTHH:MM:SS.sssZ\fP\&. Set either the lock duration or the expiration date (not both). .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = snapshot\fP and \fBsnapshot_provider = aws\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBazure_credential\fP .sp Specifies the type of Azure credential to use for authentication, either \fBazure\-cli\fP, \fBmanaged\-identity\fP or \fBdefault\fP\&. If not provided, the default Azure authentication method will be used. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = snapshot\fP and \fBsnapshot_provider = azure\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBazure_resource_group\fP .sp Specifies the name of the Azure resource group containing the compute instance and disks defined by \fBsnapshot_instance\fP and \fBsnapshot_disks\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = snapshot\fP and \fBsnapshot_provider = azure\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBazure_subscription_id\fP .sp Identifies the Azure subscription that owns the instance and storage volumes defined by \fBsnapshot_instance\fP and \fBsnapshot_disks\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = snapshot\fP and \fBsnapshot_provider = azure\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBgcp_project\fP .sp Specifies the ID of the GCP project that owns the instance and storage volumes defined by \fBsnapshot_instance\fP and \fBsnapshot_disks\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = snapshot\fP and \fBsnapshot_provider = gcp\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBgcp_zone\fP .sp Indicates the availability zone where the compute instance and disks are located for snapshot backups. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only supported when \fBbackup_method = snapshot\fP and \fBsnapshot_provider = gcp\fP\&. .UNINDENT .UNINDENT .sp Scope: Server / Model. .sp \fBsnapshot_disks\fP .sp This option is a comma\-separated list of disks to include in cloud snapshot backups. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Required when \fBbackup_method = snapshot\fP\&. .sp Ensure that the \fBsnapshot_disks\fP list includes all disks that store Postgres data, as any data not on these listed disks will not be included in the backup and will be unavailable during recovery. .UNINDENT .UNINDENT .sp Scope: Server / Model. .sp \fBsnapshot_instance\fP .sp The name of the VM or compute instance where the storage volumes are attached. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Required when \fBbackup_method = snapshot\fP\&. .UNINDENT .UNINDENT .sp Scope: Server / Model. .sp \fBsnapshot_provider\fP .sp The name of the cloud provider to use for creating snapshots. Supported value: \fBaws\fP, \fBazure\fP and \fBgcp\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Required when \fBbackup_method = snapshot\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .SS Hook Scripts .sp These configuration options are related to the pre or post execution of hook scripts. .sp \fBpost_archive_retry_script\fP .sp Specifies a hook script to run after a WAL file is archived. Barman will retry this script until it returns \fBSUCCESS\fP (0), \fBABORT_CONTINUE\fP (62), or \fBABORT_STOP\fP (63). In a post\-archive scenario, \fBABORT_STOP\fP has the same effect as \fBABORT_CONTINUE\fP\&. .sp Scope: Global / Server. .sp \fBpost_archive_script\fP .sp Specifies a hook script to run after a WAL file is archived, following the \fBpost_archive_retry_script\fP\&. .sp Scope: Global / Server. .sp \fBpost_backup_retry_script\fP .sp Specifies a hook script to run after a base backup. Barman will retry this script until it returns \fBSUCCESS\fP (0), \fBABORT_CONTINUE\fP (62), or \fBABORT_STOP\fP (63). In a post\-backup scenario, \fBABORT_STOP\fP has the same effect as \fBABORT_CONTINUE\fP\&. .sp Scope: Global / Server. .sp \fBpost_backup_script\fP .sp Specifies a hook script to run after a base backup, following the \fBpost_backup_retry_script\fP\&. .sp Scope: Global / Server. .sp \fBpost_delete_retry_script\fP .sp Specifies a hook script to run after deleting a backup. Barman will retry this script until it returns \fBSUCCESS\fP (0), \fBABORT_CONTINUE\fP (62), or \fBABORT_STOP\fP (63). In a post\-delete scenario, \fBABORT_STOP\fP has the same effect as \fBABORT_CONTINUE\fP\&. .sp Scope: Global / Server. .sp \fBpost_delete_script\fP .sp Specifies a hook script to run after deleting a backup, following the \fBpost_delete_retry_script\fP\&. .sp Scope: Global / Server. .sp \fBpost_recovery_retry_script\fP .sp Specifies a hook script to run after a recovery. Barman will retry this script until it returns \fBSUCCESS\fP (0), \fBABORT_CONTINUE\fP (62), or \fBABORT_STOP\fP (63). In a post\-recovery scenario, \fBABORT_STOP\fP has the same effect as \fBABORT_CONTINUE\fP\&. .sp Scope: Global / Server. .sp \fBpost_recovery_script\fP .sp Specifies a hook script to run after a recovery, following the \fBpost_recovery_retry_script\fP\&. .sp Scope: Global / Server. .sp \fBpost_wal_delete_retry_script\fP .sp Specifies a hook script to run after deleting a WAL file. Barman will retry this script until it returns \fBSUCCESS\fP (0), \fBABORT_CONTINUE\fP (62), or \fBABORT_STOP\fP (63). In a post\-WAL\-delete scenario, \fBABORT_STOP\fP has the same effect as \fBABORT_CONTINUE\fP\&. .sp Scope: Global / Server. .sp \fBpost_wal_delete_script\fP .sp Specifies a hook script to run after deleting a WAL file, following the \fBpost_wal_delete_retry_script\fP\&. .sp Scope: Global / Server. .sp \fBpre_archive_retry_script\fP .sp Specifies a hook script that runs before a WAL file is archived during maintenance, following the \fBpre_archive_script\fP\&. As a retry hook script, Barman will repeatedly execute the script until it returns either \fBSUCCESS\fP (0), \fBABORT_CONTINUE\fP (62), or \fBABORT_STOP\fP (63). Returning \fBABORT_STOP\fP will escalate the failure and halt the WAL archiving process. .sp Scope: Global / Server. .sp \fBpre_archive_script\fP .sp Specifies a hook script launched before a WAL file is archived by maintenance. .sp Scope: Global / Server. .sp \fBpre_backup_retry_script\fP .sp Specifies a hook script that runs before a base backup, following the \fBpre_backup_script\fP\&. As a retry hook script, Barman will attempt to execute the script repeatedly until it returns \fBSUCCESS\fP (0), \fBABORT_CONTINUE\fP (62), or \fBABORT_STOP\fP (63). Returning \fBABORT_STOP\fP will escalate the failure and interrupt the backup process. .sp Scope: Global / Server. .sp \fBpre_backup_script\fP .sp Specifies a hook script to run before starting a base backup. .sp Scope: Global / Server. .sp \fBpre_delete_retry_script\fP .sp Specifies a retry hook script to run before backup deletion, following the \fBpre_delete_script\fP\&. As a retry hook script, Barman will attempt to execute the script repeatedly until it returns \fBSUCCESS\fP (0), \fBABORT_CONTINUE\fP (62), or \fBABORT_STOP\fP (63). Returning \fBABORT_STOP\fP will escalate the failure and interrupt the backup deletion. .sp Scope: Global / Server. .sp \fBpre_delete_script\fP .sp Specifies a hook script run before deleting a backup. .sp Scope: Global / Server. .sp \fBpre_recovery_retry_script\fP .sp Specifies a retry hook script to run before recovery, following the \fBpre_recovery_script\fP\&. As a retry hook script, Barman will attempt to execute the script repeatedly until it returns \fBSUCCESS\fP (0), \fBABORT_CONTINUE\fP (62), or \fBABORT_STOP\fP (63). Returning \fBABORT_STOP\fP will escalate the failure and interrupt the recover process. .sp Scope: Global / Server. .sp \fBpre_recovery_script\fP .sp Specifies a hook script run before starting a recovery. .sp Scope: Global / Server. .sp \fBpre_wal_delete_retry_script\fP .sp Specifies a retry hook script for WAL file deletion, executed before \fBpre_wal_delete_script\fP\&. As a retry hook script, Barman will attempt to execute the script repeatedly until it returns \fBSUCCESS\fP (0), \fBABORT_CONTINUE\fP (62), or \fBABORT_STOP\fP (63). Returning \fBABORT_STOP\fP will escalate the failure and interrupt the WAL file deletion. .sp Scope: Global / Server. .sp \fBpre_wal_delete_script\fP .sp Specifies a hook script run before deleting a WAL file. .sp Scope: Global / Server. .SS Write\-Ahead Logs (WAL) .sp These configuration options are related to how Barman will manage the Write\-Ahead Logs (WALs) of the PostreSQL servers. .sp \fBcompression\fP .sp Specifies the standard compression algorithm for WAL files. Options include: \fBlz4\fP, \fBxz\fP, \fBzstd\fP, \fBgzip\fP, \fBpygzip\fP, \fBpigz\fP, \fBbzip2\fP, \fBpybzip2\fP and \fBcustom\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 All of these options require the module to be installed in the location where the compression will occur. .sp The \fBcustom\fP option is for custom compression, which requires you to set the following options as well: .INDENT 0.0 .IP \(bu 2 \fBcustom_compression_filter\fP: a compression filter. .IP \(bu 2 \fBcustom_decompression_filter\fP: a decompression filter .IP \(bu 2 \fBcustom_compression_magic\fP: a hex string to identify a custom compressed wal file. .UNINDENT .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBcustom_compression_filter\fP .sp Specifies a custom compression algorithm for WAL files. It must be a \fBstring\fP that will be used internally to create a bash command and it will prefix to the following string \fB> \(dq$2\(dq < \(dq$1\(dq;\fP\&. Write to standard output and do not delete input files. .sp \fBTIP:\fP .INDENT 0.0 .INDENT 3.5 \fBcustom_compression_filter = \(dqxz \-c\(dq\fP .sp This is the same as running \fBxz \-c > \(dq$2\(dq < \(dq$1\(dq;\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBcustom_compression_magic\fP .sp Defines a custom magic value to identify the custom compression algorithm used in WAL files. If this is set, Barman will avoid applying custom compression to WALs that have already been compressed with the specified algorithm. If not configured, Barman will apply custom compression to all WAL files, even those pre\-compressed. .sp \fBTIP:\fP .INDENT 0.0 .INDENT 3.5 For example, in the \fBxz\fP compression algorithm, the magic number is used to detect the format of \fB\&.xz\fP files. .INDENT 0.0 .TP .B For xz files, the magic number is the following sequence of bytes: Magic Number: \fBFD 37 7A 58 5A 00\fP .TP .B In hexadecimal representation, this can be expressed as: Hex String: \fBfd377a585a00\fP .UNINDENT .sp As Barman expects the value of \fBcustom_compression_magic\fP to be prefixed with \fB0x\fP, you would need to set that config option like this: .INDENT 0.0 .INDENT 3.5 .sp .EX custom_compression_magic = 0xfd377a585a00 .EE .UNINDENT .UNINDENT .sp Reference: \X'tty: link https://tukaani.org/xz/xz-file-format-1.0.4.txt'\fI\%xz\-file\-format\fP\X'tty: link' .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBcustom_decompression_filter\fP .sp Specifies a custom decompression algorithm for compressed WAL files. It must be a \fBstring\fP that will be used internally to create a bash command and it will prefix to the following string \fB> \(dq$2\(dq < \(dq$1\(dq;\fP\&. It must correspond with the compression algorithm used. .sp \fBTIP:\fP .INDENT 0.0 .INDENT 3.5 \fBcustom_compression_filter = \(dqxz \-c \-d\(dq\fP .sp This is the same as running \fBxz \-c \-d > \(dq$2\(dq < \(dq$1\(dq;\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBincoming_wals_directory\fP .sp Specifies the directory where incoming WAL files are archived. Requires \fBarchiver\fP to be enabled. Defaults to \fB/incoming\fP\&. .sp Scope: Server. .sp \fBlast_wal_maximum_age\fP .sp Defines the time frame within which the latest archived WAL file must fall. If the latest WAL file is older than this period, the barman check command will report an error. If left empty (default), the age of the WAL files is not checked. Format is the same as \fBlast_backup_maximum_age\fP\&. .sp Scope: Global / Server / Model. .sp \fBmax_incoming_wals_queue\fP .sp Defines the maximum number of WAL files allowed in the incoming queue (including both streaming and archiving pools) before the barman check command returns an error. Default is \fBNone\fP (disabled). .sp Scope: Global / Server / Model. .sp \fBstreaming_wals_directory\fP .sp Directory for streaming WAL files. Defaults to \fB/streaming\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 This option is applicable when \fBstreaming_archiver\fP is activated. .UNINDENT .UNINDENT .sp Scope: Server. .sp \fBwal_conninfo\fP .sp The \fBwal_conninfo\fP connection string is used by Barman for monitoring the status of the replication slot receiving WALs. If specified, it takes precedence over \fBwal_streaming_conninfo\fP for these checks. If \fBwal_conninfo\fP is not set but \fBwal_streaming_conninfo\fP is, \fBwal_conninfo\fP will fall back to \fBwal_streaming_conninfo\fP\&. If neither \fBwal_conninfo\fP nor \fBwal_streaming_conninfo\fP is set, \fBwal_conninfo\fP will fall back to \fBconninfo\fP\&. Both connection strings must access a Postgres instance within the same cluster as defined by \fBstreaming_conninfo\fP and \fBconninfo\fP\&. If both \fBwal_conninfo\fP and \fBwal_streaming_conninfo\fP are set, only \fBwal_conninfo\fP needs the appropriate permissions to read settings and check the replication slot status. However, if only \fBwal_streaming_conninfo\fP is set, it must have the necessary permissions to perform these tasks. The required permissions include roles such as \fBpg_monitor\fP, both \fBpg_read_all_settings\fP and \fBpg_read_all_stats\fP, or superuser privileges. .sp Scope: Server / Model. .sp \fBwal_streaming_conninfo\fP .sp This connection string is used by Barman to connect to the Postgres server for receiving WAL segments via streaming replication and checking the replication slot status, if \fBwal_conninfo\fP is not set. If not specified, Barman defaults to using \fBstreaming_conninfo\fP for these tasks. \fBwal_streaming_conninfo\fP must connect to a Postgres instance within the same cluster as defined by \fBstreaming_conninfo\fP, and it must support streaming replication. If both \fBwal_streaming_conninfo\fP and \fBwal_conninfo\fP are set, only \fBwal_conninfo\fP needs the required permissions to read settings and check the replication slot status. If only \fBwal_streaming_conninfo\fP is specified, it must have these permissions. The necessary permissions include roles such as \fBpg_monitor\fP, both \fBpg_read_all_settings\fP and \fBpg_read_all_stats\fP, or superuser privileges. .sp Scope: Server / Model. .sp \fBwals_directory\fP .sp Directory containing WAL files. Defaults to \fB/wals\fP\&. .sp Scope: Server. .sp \fBxlogdb_directory\fP .sp A custom directory for the \fBSERVER\-xlog.db\fP file, \fBSERVER\fP being the server name. This file stores metadata of archived WAL files and is used internally by Barman. If unset, it defaults to the value of \fBwals_directory\fP\&. .sp Scope: Global / Server. .SS Restore .sp These configuration options are related to how Barman manages restoration backups. .sp \fBlocal_staging_path\fP .sp Specifies the local path for combining block\-level incremental backups during recovery. This location must have sufficient space to temporarily store the new synthetic backup. Required for recovery from a block\-level incremental backup. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Applies only when \fBbackup_method = postgres\fP\&. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .sp \fBrecovery_options\fP .sp Options for recovery operations. Currently, only \fBget\-wal\fP is supported. This option enables the creation of a basic \fBrestore_command\fP in the recovery configuration, which uses the barman \fBget\-wal\fP command to retrieve WAL files directly from Barman\(aqs WAL archive. This setting accepts a comma\-separated list of values and defaults to empty. .sp Scope: Global / Server / Model. .sp \fBrecovery_staging_path\fP .sp Specifies the path on the recovery host for staging files from compressed backups. This location must have sufficient space to temporarily store the compressed backup. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Applies only for commpressed backups. .UNINDENT .UNINDENT .sp Scope: Global / Server / Model. .SS Retention Policies .sp These configuration options are related to how Barman manages retention policies of the backups. .sp \fBlast_backup_maximum_age\fP .sp Defines the time frame within which the latest backup must fall. If the latest backup is older than this period, the barman check command will report an error. If left empty (default), the latest backup is always considered valid. The accepted format is \fB\(dqn {DAYS|WEEKS|MONTHS|HOURS}\(dq\fP, where \fBn\fP is an integer greater than zero. .sp Scope: Global / Server / Model. .sp \fBlast_backup_minimum_size\fP .sp Specifies the minimum acceptable size for the latest successful backup. If the latest backup is smaller than this size, the barman check command will report an error. If left empty (default), the latest backup is always considered valid. The accepted format is \fB\(dqn {k|Ki|M|Mi|G|Gi|T|Ti}\(dq\fP and case\-sensitive, where \fBn\fP is an integer greater than zero, with an optional SI or IEC suffix. k stands for kilo with k = 1000, while Ki stands for kilobytes Ki = 1024. The rest of the options have the same reasoning for greater units of measure. .sp Scope: Global / Server / Model. .sp \fBretention_policy\fP .sp Defines how long backups and WAL files should be retained. If this option is left blank, no retention policies will be applied. Options include redundancy and recovery window policies. .INDENT 0.0 .INDENT 3.5 .sp .EX retention_policy = {REDUNDANCY value | RECOVERY WINDOW OF value {DAYS | WEEKS | MONTHS}} .EE .UNINDENT .UNINDENT .INDENT 0.0 .IP \(bu 2 \fBretention_policy = REDUNDANCY 2\fP will keep only 2 backups in the backup catalog automatically deleting the older one as new backups are created. The number must be a positive integer. .IP \(bu 2 \fBretention_policy = RECOVERY WINDOW OF 2 DAYS\fP will only keep backups needed to recover to any point in time in the last two days, automatically deleting backups that are older. The period number must be a positive integer, and the following options can be applied to it: \fBDAYS\fP, \fBWEEKS\fP, \fBMONTHS\fP\&. .UNINDENT .sp Scope: Global / Server / Model. .sp \fBretention_policy_mode\fP .sp Mode for enforcing retention policies. Currently only supports \fBauto\fP\&. .sp Scope: Global / Server / Model. .sp \fBwal_retention_policy\fP .sp Policy for retaining WAL files. Currently only \fBmain\fP is available. .sp Scope: Global / Server / Model. .SH CONFIGURATION MODELS .sp Configuration models provide a systematic approach to manage and apply configuration overrides for Postgres servers by organizing them under a specific \fBcluster\fP name. .SS Purpose .sp The primary goal of a configuration model is to simplify the management of configuration settings for Postgres servers grouped by the same \fBcluster\fP\&. By using a model, you can apply a set of common configuration overrides, enhancing operational efficiency. They are especially beneficial in clustered environments, allowing you to create various configuration models that can be utilized during failover events. .SS Application .sp The configurations defined in a model file can be applied to Postgres servers that share the same \fBcluster\fP name specified in the model. Consequently, any server utilizing that model can inherit these settings, promoting a consistent and adaptable configuration across all servers. .SS Usage .sp Model options can only be defined within a model section, which is identified in the same way as a server section. It is important to ensure that there are no conflicts between the identifiers of server sections and model sections. .sp To apply a configuration model, execute the \fBbarman config\-switch SERVER_NAME MODEL_NAME\fP\&. This command facilitates the application of the model\(aqs overrides to the relevant Barman server associated with the specified cluster name. .sp If you wish to remove the overrides, the deletion of the model configuration file alone will not have any effect, so you can do so by using the \fB\-\-reset\fP argument with the command, as follows: \fBbarman config\-switch SERVER_NAME \-\-reset\fP\&. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 The \fBconfig\-switch\fP command will only succeed if model name exists and is associated with the same \fBcluster\fP as the server. Additionally, there can be only one active model at a time; if you execute the command multiple times with different models, only the overrides defined in the last model will be applied. .sp Not all options can be configured through models. Please review the scope of the available configurations to determine which settings apply to models. .UNINDENT .UNINDENT .SS Benefits .INDENT 0.0 .IP \(bu 2 Consistency: Ensures uniform configuration across multiple Barman servers within a cluster. .IP \(bu 2 Efficiency: Simplifies configuration management by allowing centralized updates and overrides. .IP \(bu 2 Flexibility: Allows the use of multiple model files, providing the ability to define various sets of overrides as necessary. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-cloud-restore.10000644000175100001660000001404515002177425020301 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CLOUD-RESTORE" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-cloud-restore \- Barman-cloud Commands .sp \fBSynopsis\fP .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-cloud\-restore [ { \-V | \-\-version } ] [ \-\-help ] [ { { \-v | \-\-verbose } | { \-q | \-\-quiet } } ] [ { \-t | \-\-test } ] [ \-\-cloud\-provider { aws\-s3 | azure\-blob\-storage | google\-cloud\-storage } ] [ \-\-endpoint\-url ENDPOINT_URL ] [ { \-P | \-\-aws\-profile } AWS_PROFILE ] [ \-\-profile AWS_PROFILE ] [ \-\-read\-timeout READ_TIMEOUT ] [ { \-\-azure\-credential | \-\-credential } { azure\-cli | managed\-identity | default } ] [ \-\-snapshot\-recovery\-instance SNAPSHOT_RECOVERY_INSTANCE ] [ \-\-snapshot\-recovery\-zone GCP_ZONE ] [ \-\-aws\-region AWS_REGION ] [ \-\-gcp\-zone GCP_ZONE ] [ \-\-azure\-resource\-group AZURE_RESOURCE_GROUP ] [ \-\-tablespace NAME:LOCATION [ \-\-tablespace NAME:LOCATION ... ] ] [ \-\-target\-lsn LSN ] [ \-\-target\-time TIMESTAMP ] [ \-\-target\-tli TLI ] SOURCE_URL SERVER_NAME BACKUP_ID RECOVERY_DESTINATION .EE .UNINDENT .UNINDENT .sp \fBDescription\fP .sp Use this script to restore a backup directly from cloud storage that was created with the \fBbarman\-cloud\-backup\fP command. Additionally, this script can prepare for recovery from a snapshot backup by verifying that attached disks were cloned from the correct snapshots and by downloading the backup label from object storage. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 For GCP, only authentication with \fBGOOGLE_APPLICATION_CREDENTIALS\fP env is supported. .UNINDENT .UNINDENT .sp \fBParameters\fP .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server that holds the backup to be restored. .TP .B \fBSOURCE_URL\fP URL of the cloud source, such as a bucket in AWS S3. For example: \fBs3://bucket/path/to/folder\fP\&. .TP .B \fBBACKUP_ID\fP The ID of the backup to be restored. Use \fBauto\fP to have Barman automatically find the most suitable backup for the restore operation. .TP .B \fBRECOVERY_DESTINATION\fP The path to a directory for recovery. .TP .B \fB\-V\fP / \fB\-\-version\fP Show version and exit. .TP .B \fB\-\-help\fP show this help message and exit. .TP .B \fB\-v\fP / \fB\-\-verbose\fP Increase output verbosity (e.g., \fB\-vv\fP is more than \fB\-v\fP). .TP .B \fB\-q\fP / \fB\-\-quiet\fP Decrease output verbosity (e.g., \fB\-qq\fP is less than \fB\-q\fP). .TP .B \fB\-t\fP / \fB\-\-test\fP Test cloud connectivity and exit. .TP .B \fB\-\-cloud\-provider\fP The cloud provider to use as a storage backend. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBaws\-s3\fP\&. .IP \(bu 2 \fBazure\-blob\-storage\fP\&. .IP \(bu 2 \fBgoogle\-cloud\-storage\fP\&. .UNINDENT .TP .B \fB\-\-snapshot\-recovery\-instance\fP Instance where the disks recovered from the snapshots are attached. .TP .B \fB\-\-tablespace\fP Tablespace relocation rule. .TP .B \fB\-\-target\-lsn\fP The recovery target lsn, e.g., \fB3/64000000\fP\&. .TP .B \fB\-\-target\-time\fP The recovery target timestamp with or without timezone, in the format \fB%Y\-%m\-%d %H:%M:%S\fP\&. .TP .B \fB\-\-target\-tli\fP The recovery target timeline. .UNINDENT .sp \fBExtra options for the AWS cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-endpoint\-url\fP Override default S3 endpoint URL with the given one. .TP .B \fB\-P\fP / \fB\-\-aws\-profile\fP Profile name (e.g. \fBINI\fP section in AWS credentials file). .TP .B \fB\-\-profile\fP (deprecated) Profile name (e.g. \fBINI\fP section in AWS credentials file) \- replaced by \fB\-\-aws\-profile\fP\&. .TP .B \fB\-\-read\-timeout\fP The time in seconds until a timeout is raised when waiting to read from a connection (defaults to \fB60\fP seconds). .TP .B \fB\-\-aws\-region\fP The name of the AWS region containing the EC2 VM and storage volumes defined by the \fB\-\-snapshot\-instance\fP and \fB\-\-snapshot\-disk\fP arguments. .UNINDENT .sp \fBExtra options for the Azure cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-azure\-credential / \-\-credential\fP Optionally specify the type of credential to use when authenticating with Azure. If omitted then Azure Blob Storage credentials will be obtained from the environment and the default Azure authentication flow will be used for authenticating with all other Azure services. If no credentials can be found in the environment then the default Azure authentication flow will also be used for Azure Blob Storage. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBazure\-cli\fP\&. .IP \(bu 2 \fBmanaged\-identity\fP\&. .IP \(bu 2 \fBdefault\fP\&. .UNINDENT .TP .B \fB\-\-azure\-resource\-group\fP The name of the Azure resource group to which the compute instance and disks defined by the \fB\-\-snapshot\-instance\fP and \fB\-\-snapshot\-disk\fP arguments belong. .UNINDENT .sp \fBExtra options for GCP cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-gcp\-zone\fP Zone of the disks from which snapshots should be taken. .TP .B \fB\-\-snapshot\-recovery\-zone\fP (deprecated) Zone containing the instance and disks for the snapshot recovery \- replaced by \fB\-\-gcp\-zone\fP\&. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-list-servers.10000644000175100001660000000245615002177425020157 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-LIST-SERVERS" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-list-servers \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX list\-servers [ { \-h | \-\-help } ] [ \-\-minimal ] .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Display all configured servers along with their descriptions. .SH PARAMETERS .INDENT 0.0 .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-minimal\fP Machine readable output. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-restore.10000644000175100001660000002450515002177425017177 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-RESTORE" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-restore \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX restore [ \-\-aws\-region AWS_REGION } ] [ \-\-azure\-resource\-group AZURE_RESOURCE_GRP ] [ \-\-bwlimit KBPS ] [ \-\-exclusive ] [ \-\-gcp\-zone GCP_ZONE ] [ { \-\-get\-wal | \-\-no\-get\-wal } ] [ { \-h | \-\-help } ] [ { \-j | \-\-jobs } PARALLEL_WORKERS ] [ \-\-jobs\-start\-batch\-period SECONDS ] [ \-\-jobs\-start\-batch\-size NUMBER ] [ \-\-local\-staging\-path PATH ] [ { \-\-network\-compression | \-\-no\-network\-compression } ] [ \-\-no\-retry ] [ \-\-recovery\-conf\-filename FILENAME ] [ \-\-recovery\-staging\-path PATH ] [ \-\-remote\-ssh\-command STRING ] [ \-\-retry\-sleep SECONDS ] [ \-\-retry\-times NUMBER ] [ \-\-snapshot\-recovery\-instance INSTANCE_NAME ] [ \-\-snapshot\-recovery\-zone GCP_ZONE ] [ \-\-standby\-mode ] [ \-\-tablespace NAME:LOCATION [ \-\-tablespace NAME:LOCATION ... ] ] [ \-\-target\-action { pause | shutdown | promote } ] [ \-\-target\-immediate ] [ \-\-target\-lsn LSN ] [ \-\-target\-name RESTORE_POINT_NAME ] [ \-\-target\-time TIMESTAMP ] [ \-\-target\-tli TLI ] [ \-\-target\-xid XID ] [ \-\-staging\-wal\-directory ] SERVER_NAME BACKUP_ID DESTINATION_DIR .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Execute a PostreSQL server restore operation. Barman will restore the backup from a server in the destination directory. The restoration can be performed locally (on the barman node itself) or remotely (on another machine accessible via SSH). The location is determined by whether or not the \fB\-\-remote\-ssh\-command\fP option is used. More information on this command can be found in the \fI\%Recovery\fP section. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in the barman catalog. Use \fBauto\fP to have Barman automatically find the most suitable backup for the restore operation. .TP .B \fBDESTINATION_DIR\fP Destination directory to restore the backup. .TP .B \fB\-\-aws\-region\fP Specify the AWS region where the instance and disks for snapshot recovery are located. This option allows you to override the \fBaws_region\fP value in the Barman configuration. .TP .B \fB\-\-azure\-resource\-group\fP Specify the Azure resource group containing the instance and disks for snapshot recovery. This option allows you to override the \fBazure_resource_group\fP value in the Barman configuration. .TP .B \fB\-\-bwlimit\fP Specify the maximum transfer rate in kilobytes per second. A value of \fB0\fP indicates no limit. This setting overrides the \fBbandwidth_limit\fP configuration option. .TP .B \fB\-\-exclusive\fP Set target (time, XID or LSN) to be non inclusive. .TP .B \fB\-\-gcp\-zone\fP Specify the GCP zone where the instance and disks for snapshot recovery are located. This option allows you to override the \fBgcp_zone\fP value in the Barman configuration. .TP .B \fB\-\-get\-wal\fP / \fB\-\-no\-get\-wal\fP Enable/disable usage of \fBget\-wal\fP for WAL fetching during recovery. Default is based on \fBrecovery_options\fP setting. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-j\fP / \fB\-\-jobs\fP Specify the number of parallel workers to use for copying files during the backup. This setting overrides the \fBparallel_jobs\fP parameter if it is specified in the configuration file. .TP .B \fB\-\-jobs\-start\-batch\-period\fP Specify the time period, in seconds, for starting a single batch of jobs. This value overrides the \fBparallel_jobs_start_batch_period\fP parameter if it is set in the configuration file. The default is \fB1\fP second. .TP .B \fB\-\-jobs\-start\-batch\-size\fP Specify the maximum number of parallel workers to initiate in a single batch. This value overrides the \fBparallel_jobs_start_batch_size\fP parameter if it is defined in the configuration file. The default is \fB10\fP workers. .TP .B \fB\-\-local\-staging\-path\fP Specify path on the Barman host where the chain of backups will be combined before being copied to the destination directory. The contents created within the staging path will be removed upon completion of the restore process. This option is necessary for restoring from block\-level incremental backups and has no effect otherwise. .TP .B \fB\-\-network\-compression\fP / \fB\-\-no\-network\-compression\fP Enable/disable network compression during remote restore. Default is based on \fBnetwork_compression\fP configuration setting. .TP .B \fB\-\-no\-retry\fP There will be no retry in case of an error. It is the same as setting \fB\-\-retry\-times 0\fP\&. .TP .B \fB\-\-recovery\-conf\-filename\fP Specify the name of the file where Barman should write recovery options when recovering backups for Postgres versions 12 and later. By default, this is set to \fBpostgresql.auto.conf\fP\&. However, if \fB\-\-recovery\-conf\-filename\fP is specified, recovery options will be written to the specified value instead. While the default value is suitable for most Postgres installations, this option allows you to specify an alternative location if Postgres is managed by tools that alter the configuration mechanism (for example, if \fBpostgresql.auto.conf\fP is symlinked to \fB/dev/null\fP). .TP .B \fB\-\-recovery\-staging\-path\fP Specify a path on the recovery host where files for a compressed backup will be staged before being uncompressed to the destination directory. Backups will be staged in their own directory within the staging path, following the naming convention: \fBbarman\-staging\-SERVER_NAME\-BACKUP_ID\fP\&. This staging directory will be removed after the restore process is complete. This option is mandatory for restoring from compressed backups and has no effect otherwise. .TP .B \fB\-\-remote\-ssh\-command\fP This option enables remote restore by specifying the secure shell command to execute on a remote host. It functions similarly to the \fBssh_command\fP server option in the configuration file for remote restore, that is, \fB\(aqssh USER@SERVER\(aq\fP\&. .TP .B \fB\-\-retry\-sleep\fP Specify the number of seconds to wait after a failed copy before retrying. This setting applies to both backup and restore operations and overrides the \fBbasebackup_retry_sleep\fP parameter if it is defined in the configuration file. .TP .B \fB\-\-retry\-times\fP Specify the number of times to retry the base backup copy in case of an error. This applies to both backup and restore operations and overrides the \fBbasebackup_retry_times\fP parameter if it is set in the configuration file. .TP .B \fB\-\-snapshot\-recovery\-instance\fP Specify the name of the instance where the disks recovered from the snapshots are attached. This option is necessary when recovering backups created with \fBbackup_method=snapshot\fP\&. .TP .B \fB\-\-snapshot\-recovery\-zone\fP (deprecated) Zone containing the instance and disks for the snapshot recovery (deprecated: replaced by \fB\-\-gcp\-zone\fP) .TP .B \fB\-\-standby\-mode\fP Whether to start the Postgres server as a standby. .TP .B \fB\-\-tablespace\fP Specify tablespace relocation rule. \fBNAME\fP is the tablespace name and \fBLOCATION\fP is the recovery host destination path to restore the tablespace. .TP .B \fB\-\-target\-action\fP Trigger the specified action when the recovery target is reached. This option requires defining a target along with one of these actions. The possible values are: .INDENT 7.0 .IP \(bu 2 \fBpause\fP: Once recovery target is reached, the server is started in pause state, allowing users to inspect the instance .IP \(bu 2 \fBpromote\fP: Once recovery target is reached, the server will exit the recovery operation and is promoted as a master. .IP \(bu 2 \fBshutdown\fP: Once recovery target is reached, the server is shut down. .UNINDENT .TP .B \fB\-\-target\-immediate\fP Recovery is completed when a consistent state is reached (end of the base backup). .TP .B \fB\-\-target\-lsn\fP Recover to the specified LSN (Log Sequence Number). Requires Postgres 10 or above. .TP .B \fB\-\-target\-name\fP Recover to the specified name of a restore point previously created with the \fBpg_create_restore_point(name)\fP\&. .TP .B \fB\-\-target\-time\fP Recover to the specified time. Use the format \fBYYYY\-MM\-DD HH:MM:SS.mmm\fP\&. .TP .B \fB\-\-target\-tli\fP Recover the specified timeline. You can use the special values \fBcurrent\fP and \fBlatest\fP in addition to a numeric timeline ID. For Postgres versions 12 and above, the default is to recover to the latest timeline in the WAL archive. For Postgres versions below 12, the default is to recover to the timeline that was current at the time the backup was taken. .TP .B \fB\-\-target\-xid\fP Recover to the specified transaction ID. .TP .B \fB\-\-staging\-wal\-directory\fP A staging directory on the destination host for WAL files when performing PITR. If unspecified, it uses a \fBbarman_wal\fP directory inside the destination directory. .UNINDENT .SH SHORTCUTS .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-replication-status.10000644000175100001660000000456215002177425021347 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-REPLICATION-STATUS" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-replication-status \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX replication\-status [ { \-h | \-\-help } ] [ \-\-minimal ] [ \-\-source { backup\-host | wal\-host } ] [ \-\-target { hot\-standby | wal\-streamer | all } ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Display real\-time information and status of any streaming clients connected to the specified server. Specify \fBall\fP shortcut to diplay information for all configured servers. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-minimal\fP Machine readable output. .TP .B \fB\-\-source\fP The possible values are: .INDENT 7.0 .IP \(bu 2 \fBbackup\-host\fP (default): List clients using the backup connection information for a server. .IP \(bu 2 \fBwal\-host\fP: List clients using the WAL streaming connection information for a server. .UNINDENT .TP .B \fB\-\-target\fP The possible values are: .INDENT 7.0 .IP \(bu 2 \fBhot\-standby\fP: List only hot standby servers. .IP \(bu 2 \fBwal\-streamer\fP: List only WAL streaming clients, such as \fBpg_receivewal\fP\&. .IP \(bu 2 \fBall\fP (default): List all streaming clients. .UNINDENT .UNINDENT .SH SHORTCUTS .sp Use shortcuts instead of \fBSERVER_NAME\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-get-wal.10000644000175100001660000000573115002177425017054 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-GET-WAL" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-get-wal \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX get\-wal [ { \-\-bzip | \-j } ] [ { \-\-gzip | \-z | \-x } ] [ { \-h | \-\-help } ] [ \-\-keep\-compression ] [ { \-\-output\-directory | \-o } OUTPUT_DIRECTORY ] [ { \-\-peek | \-p } VALUE ] [ { \-P | \-\-partial } ] [ { \-t | \-\-test } ] SERVER_NAME WAL_NAME .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Retrieve a WAL file from the xlog archive of a specified server. By default, if the requested WAL file is found, it is returned as uncompressed content to \fBSTDOUT\fP\&. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBWAL_NAME\fP Id of the backup in barman catalog. .TP .B \fB\-\-bzip2\fP / \fB\-j\fP Output will be compressed using bzip2. .TP .B \fB\-\-gzip\fP / \fB\-z\fP / \fB\-x\fP Output will be compressed using gzip. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-keep\-compression\fP Do not uncompress the file content. The output will be the original compressed file. .TP .B \fB\-\-output\-directory\fP / \fB\-o\fP Destination directory where barman will store the WAL file. .TP .B \fB\-\-peek\fP / \fB\-p\fP Specify an integer value greater than or equal to 1 to retrieve WAL files from the specified WAL file up to the value specified by this parameter. When using this option, \fBget\-wal\fP returns a list of zero to the specified WAL segment names, with one name per row. .TP .B \fB\-P\fP / \fB\-\-partial\fP Additionally, collect partial WAL files (.partial). .TP .B \fB\-t\fP / \fB\-\-test\fP Test both the connection and configuration of the specified Postgres server in Barman for WAL retrieval. When this option is used, the required \fBWAL_NAME\fP argument is disregarded. .UNINDENT .sp \fBWARNING:\fP .INDENT 0.0 .INDENT 3.5 \fB\-z\fP / \fB\-\-gzip\fP and \fB\-j\fP / \fB\-\-bzip2\fP options are deprecated and will be removed in the future. For WAL compression, please make sure to enable it directly on the Barman server via the \fBcompression\fP configuration option. .UNINDENT .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-cloud-backup-keep.10000644000175100001660000001141315002177425021001 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CLOUD-BACKUP-KEEP" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-cloud-backup-keep \- Barman-cloud Commands .sp \fBSynopsis\fP .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-cloud\-backup\-keep [ { \-V | \-\-version } ] [ \-\-help ] [ { { \-v | \-\-verbose } | { \-q | \-\-quiet } } ] [ { \-t | \-\-test } ] [ \-\-cloud\-provider { aws\-s3 | azure\-blob\-storage | google\-cloud\-storage } ] [ \-\-endpoint\-url ENDPOINT_URL ] [ { \-P | \-\-aws\-profile } AWS_PROFILE ] [ \-\-profile AWS_PROFILE ] [ \-\-read\-timeout READ_TIMEOUT ] [ { \-\-azure\-credential | \-\-credential } { azure\-cli | managed\-identity | default } ] [ { { \-r | \-\-release } | { \-s | \-\-status } | \-\-target { full | standalone } } ] SOURCE_URL SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .sp \fBDescription\fP .sp Use this script to designate backups in cloud storage as archival backups, ensuring their indefinite retention regardless of retention policies. .sp This script allows you to mark backups previously created with \fBbarman\-cloud\-backup\fP as archival backups. Once flagged as archival, these backups are preserved indefinitely and are not subject to standard retention policies. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 For GCP, only authentication with \fBGOOGLE_APPLICATION_CREDENTIALS\fP env is supported. .UNINDENT .UNINDENT .sp \fBParameters\fP .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server that holds the backup to be kept. .TP .B \fBSOURCE_URL\fP URL of the cloud source, such as a bucket in AWS S3. For example: \fBs3://bucket/path/to/folder\fP\&. .TP .B \fBBACKUP_ID\fP The ID of the backup to be kept. .TP .B \fB\-V\fP / \fB\-\-version\fP Show version and exit. .TP .B \fB\-\-help\fP show this help message and exit. .TP .B \fB\-v\fP / \fB\-\-verbose\fP Increase output verbosity (e.g., \fB\-vv\fP is more than \fB\-v\fP). .TP .B \fB\-q\fP / \fB\-\-quiet\fP Decrease output verbosity (e.g., \fB\-qq\fP is less than \fB\-q\fP). .TP .B \fB\-t\fP / \fB\-\-test\fP Test cloud connectivity and exit. .TP .B \fB\-\-cloud\-provider\fP The cloud provider to use as a storage backend. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBaws\-s3\fP\&. .IP \(bu 2 \fBazure\-blob\-storage\fP\&. .IP \(bu 2 \fBgoogle\-cloud\-storage\fP\&. .UNINDENT .TP .B \fB\-r\fP / \fB\-\-release\fP If specified, the command will remove the keep annotation and the backup will be eligible for deletion. .TP .B \fB\-s\fP / \fB\-\-status\fP Print the keep status of the backup. .TP .B \fB\-\-target\fP Specify the recovery target for this backup. Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBfull\fP .IP \(bu 2 \fBstandalone\fP .UNINDENT .UNINDENT .sp \fBExtra options for the AWS cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-endpoint\-url\fP Override default S3 endpoint URL with the given one. .TP .B \fB\-P\fP / \fB\-\-aws\-profile\fP Profile name (e.g. \fBINI\fP section in AWS credentials file). .TP .B \fB\-\-profile\fP (deprecated) Profile name (e.g. \fBINI\fP section in AWS credentials file) \- replaced by \fB\-\-aws\-profile\fP\&. .TP .B \fB\-\-read\-timeout\fP The time in seconds until a timeout is raised when waiting to read from a connection (defaults to \fB60\fP seconds). .UNINDENT .sp \fBExtra options for the Azure cloud provider\fP .INDENT 0.0 .TP .B \fB\-\-azure\-credential / \-\-credential\fP Optionally specify the type of credential to use when authenticating with Azure. If omitted then Azure Blob Storage credentials will be obtained from the environment and the default Azure authentication flow will be used for authenticating with all other Azure services. If no credentials can be found in the environment then the default Azure authentication flow will also be used for Azure Blob Storage. .sp Allowed options are: .INDENT 7.0 .IP \(bu 2 \fBazure\-cli\fP\&. .IP \(bu 2 \fBmanaged\-identity\fP\&. .IP \(bu 2 \fBdefault\fP\&. .UNINDENT .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-show-backup.10000644000175100001660000000400715002177425017732 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-SHOW-BACKUP" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-show-backup \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX show\-backup [ { \-h | \-\-help } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Display detailed information about a specific backup. You can find details about this command in \fI\%Catalog usage\fP\&. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH SHORTCUTS .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-archive-wal.10000644000175100001660000000303015002177425017704 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-ARCHIVE-WAL" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-archive-wal \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX archive\-wal [ { \-h | \-\-help } ] SERVER_NAME .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Fetch WAL files received from either the standard \fBarchive_command\fP or streaming replication with \fBpg_receivewal\fP and store them in the server\(aqs WAL archive. If you have enabled \fBcompression\fP in the configuration file, the WAL files will be compressed before they are archived. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-check.10000644000175100001660000000341215002177425016563 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CHECK" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-check \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX check [ { \-h | \-\-help } ] [ \-\-nagios ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Display status information about a server, such as SSH connection, Postgres version, configuration and backup directories, archiving and streaming processes, replication slots, and more. Use \fBall\fP as shortcut to show diagnostic information for all configured servers. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-nagios\fP Nagios plugin compatible output. .UNINDENT .SH SHORTCUTS .sp Use shortcuts instead of \fBSERVER_NAME\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-switch-wal.10000644000175100001660000000426215002177425017574 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-SWITCH-WAL" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-switch-wal \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX switch\-wal [ \-\-archive ] [ \-\-archive\-timeout ] [ \-\-force ] [ { \-h | \-\-help } ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Execute \fBpg_switch_wal()\fP on the target server (Postgres versions 10 and later) or \fBpg_switch_xlog()\fP (for Postgres versions 8.3 to 9.6). .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-\-archive\fP Waits for one WAL file to be archived. If no WAL file is archived within a specified time (default: \fB30\fP seconds), Barman will terminate with a failure exit code. This option is also available on standby servers. .TP .B \fB\-\-archive\-timeout\fP Specify the amount of time in seconds (default: \fB30\fP seconds) that the archiver will wait for a new WAL file to be archived before timing out. This option is also available on standby servers. .TP .B \fB\-\-force\fP Forces the switch by executing a CHECKPOINT before \fBpg_switch_wal()\fP\&. .sp \fBNOTE:\fP .INDENT 7.0 .INDENT 3.5 Running a CHECKPOINT may increase I/O load on the Postgres server, so use this option cautiously. .UNINDENT .UNINDENT .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-config-update.10000644000175100001660000000461015002177425020234 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CONFIG-UPDATE" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-config-update \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX config\-update [ { \-h | \-\-help } ] STRING .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Create or update the configurations for servers and/or models in Barman. The parameter should be a JSON string containing an array of documents. Each document must include a \fBscope\fP key, which can be either server or model, and either a \fBserver_name\fP or \fBmodel_name\fP key, depending on the scope value. Additionally, the document should include other keys representing Barman configuration options and their desired values. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 The barman \fBconfig\-update\fP command writes configuration options to a file named \fB\&.barman.auto.conf\fP, located in the \fBbarman_home\fP directory. This configuration file has higher precedence and will override values from the global Barman configuration file (usually \fB/etc/barman.conf\fP) and from any included files specified in \fBconfiguration_files_directory\fP (typically files in \fB/etc/barman.d\fP). Be aware of this if you decide to manually modify configuration options in those files later. .UNINDENT .UNINDENT .SH PARAMETERS .INDENT 0.0 .TP .B \fBSTRING\fP List of JSON formatted string. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH EXAMPLE .sp \fBJSON_STRING=\(aq[{“scope”: “server”, “server_name”: “my_server”, “archiver”: “on”, “streaming_archiver”: “off”}]\(aq\fP .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-sync-wals.10000644000175100001660000000275715002177425017441 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-SYNC-WALS" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-sync-wals \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX sync\-wals [ { \-h | \-\-help } ] SERVER_NAME .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp This command synchronizes a passive node with its primary by copying all archived WAL files from the server node. It is available only for passive nodes and utilizes the \fBprimary_ssh_command\fP option to establish a secure connection with the primary node. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-cron.10000644000175100001660000000270715002177425016455 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-CRON" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-cron \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX cron [ { \-h | \-\-help } ] [ \-\-keep\-descriptors ] .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Carry out maintenance tasks, such as enforcing retention policies or managing WAL files. .SH PARAMETERS .INDENT 0.0 .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-keep\-descriptors\fP Keep the ^stdout^ and ^stderr^ streams of the Barman subprocesses connected to the main process. This is especially useful for Docker\-based installations. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-delete.10000644000175100001660000000363215002177425016754 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-DELETE" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-delete \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX delete [ { \-h | \-\-help } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Delete the specified backup. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH SHORTCUTS .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-show-servers.10000644000175100001660000000323115002177425020154 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-SHOW-SERVERS" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-show-servers \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX show\-servers [ { \-h | \-\-help } ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Display detailed information about a server, including \fBconninfo\fP, \fBbackup_directory\fP, \fBwals_directory\fP, \fBarchive_command\fP, and many more. To view information about all configured servers, specify the \fBall\fP shortcut instead of the server name. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH SHORTCUTS .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-status.10000644000175100001660000000302215002177425017026 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-STATUS" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-status \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX status [ { \-h | \-\-help } ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Display information about a server\(aqs status, including details such as the state, Postgres version, WAL information, available backups and more. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH SHORTCUTS .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman.10000644000175100001660000016011515002177425015514 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman \- Barman Commands .sp Barman has a command\-line interface named \fBbarman\fP, which is used basically to interact with Barman\(aqs backend. .sp Before jumping into each of the sub\-commands of \fBbarman\fP, be aware that \fBbarman\fP has global options available for all of the sub\-commands. These options can modify the behavior of the sub\-commands and can be used as follows: .SH BARMAN .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX barman [ { \-c | \-\-config } CONFIG ] [ { \-\-color | \-\-colour } { never | always | auto } ] [ { \-d | \-\-debug } ] [ { \-f | \-\-format } { json | console } ] [ { \-h | \-\-help } ] [ \-\-log\-level { NOTSET | DEBUG | INFO | WARNING | ERROR | CRITICAL } ] [ { \-q | \-\-quiet } ] [ { \-v | \-\-version } ] [ SUBCOMMAND ] .EE .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 This is the syntax for the synopsis: .INDENT 0.0 .IP \(bu 2 Options between square brackets are optional. .IP \(bu 2 Options between curly brackets represent a choose one of set operation. .IP \(bu 2 Options with \fB[ ... ]\fP can be specified multiple times. .IP \(bu 2 Things written in uppercase represent a literal that should be given a value to. .UNINDENT .sp We will use this same syntax when describing \fBbarman\fP sub\-commands in the following sections. .sp Also, when describing sub\-commands in the following sections, the commands\(aq synopsis should be seen as a replacement for the \fBSUBCOMMAND\fP\&. .UNINDENT .UNINDENT .SS Parameters .INDENT 0.0 .TP .B \fB\-c\fP / \fB\-\-config CONFIG\fP Specify the configuration file to be used. Defaults to \fB/etc/barman.conf\fP if not provided. .TP .B \fB\-\-color\fP / \fB\-\-colour { never | always | auto }\fP Control whether to use colors in the output. The default is \fBauto\fP\&. Options are: .INDENT 7.0 .IP \(bu 2 \fBnever\fP: Do not use color. .IP \(bu 2 \fBalways\fP: Always use color. .IP \(bu 2 \fBauto\fP: Use color if the output is to a terminal. .UNINDENT .TP .B \fB\-d\fP / \fB\-\-debug\fP Enable debug output. Default is \fBfalse\fP\&. Provides detailed logging information for troubleshooting. .TP .B \fB\-f\fP / \fB\-\-format { json | console }\fP Specify the output format. Options are: .INDENT 7.0 .IP \(bu 2 \fBjson\fP: Output in JSON format. .IP \(bu 2 \fBconsole\fP: Output in human\-readable format (default). .UNINDENT .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-log\-level { NOTSET | DEBUG | INFO | WARNING | ERROR | CRITICAL }\fP Override the default logging level. Options are: .INDENT 7.0 .IP \(bu 2 \fBNOTSET\fP: This is the default level when no specific logging level is set. It essentially means \(dqno filtering\(dq of log messages, allowing all messages to be processed according to the levels that are set in the configuration. .IP \(bu 2 \fBDEBUG\fP: This level is used for detailed, diagnostic information, often useful for developers when diagnosing problems. It includes messages that are more granular and detailed, intended to help trace the execution of the program. .IP \(bu 2 \fBINFO\fP: This level provides general information about the application\(aqs normal operation. It\(aqs used for messages that indicate the progress of the application or highlight key points in the execution flow that are useful but not indicative of any issues. .IP \(bu 2 \fBWARNING\fP: This level indicates that something unexpected happened or that there might be a potential problem. It\(aqs used for messages that are not critical but could be of concern, signaling that attention might be needed. .IP \(bu 2 \fBERROR\fP: This level is used when an error occurs that prevents a particular operation from completing successfully. It\(aqs used to indicate significant issues that need to be addressed but do not necessarily stop the application from running. .IP \(bu 2 \fBCRITICAL\fP: This is the highest level of severity, indicating a serious error that has likely caused the application to terminate or will have severe consequences if not addressed immediately. It\(aqs used for critical issues that demand urgent attention. .UNINDENT .TP .B \fB\-q\fP / \fB\-\-quiet\fP Suppress all output. Useful for cron jobs or automated scripts. .TP .B \fB\-v\fP / \fB\-\-version\fP Show the program version number and exit. .UNINDENT .SH SHORTCUTS .sp For some commands, you can use the following shortcuts or aliases to identify a backup for a given server. Specifically, the \fBall\fP shortcut can be used to identify all servers: .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SH EXIT STATUSES .sp Status code \fB0\fP means \fBsuccess\fP, while status code \fBNon\-Zero\fP means \fBfailure\fP\&. .SH SUB-COMMANDS .sp \fBbarman\fP exposes several handy operations. This section is intended to describe each of them. .sp In the following sections you can find a description of each command implemented by \fBbarman\fP\&. Some of these commands may have more detailed information in another main section in this documentation. If that is the case, a reference is provided to help you quickly navigate to it. .SS \fBbarman archive\-wal\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX archive\-wal [ { \-h | \-\-help } ] SERVER_NAME .EE .UNINDENT .UNINDENT .SS Description .sp Fetch WAL files received from either the standard \fBarchive_command\fP or streaming replication with \fBpg_receivewal\fP and store them in the server\(aqs WAL archive. If you have enabled \fBcompression\fP in the configuration file, the WAL files will be compressed before they are archived. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS \fBbarman backup\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX backup [ \-\-bwlimit KBPS ] [ { \-h | \-\-help } ] [ \-\-incremental BACKUP_ID ] [ \-\-immediate\-checkpoint ] [ { \-j | \-\-jobs } PARALLEL_WORKERS ] [ \-\-jobs\-start\-batch\-period PERIOD ] [ \-\-jobs\-start\-batch\-size SIZE ] [ \-\-keepalive\-interval SECONDS ] [ \-\-manifest ] [ \-\-name NAME ] [ \-\-no\-immediate\-checkpoint ] [ \-\-no\-manifest ] [ \-\-no\-retry ] [ \-\-retry\-sleep SECONDS ] [ \-\-retry\-times NUMBER ] [ \-\-reuse\-backup { off | copy | link } ] [ { \-\-wait | \-w } ] [ \-\-wait\-timeout SECONDS ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SS Description .sp Execute a PostreSQL server backup. Barman will use the parameters specified in the Global and Server configuration files. Specify \fBall\fP shortcut instead of the server name to execute backups from all servers configured in the Barman node. You can also specify multiple server names in sequence to execute backups for specific servers. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fB\-\-bwlimit\fP Specify the maximum transfer rate in kilobytes per second. A value of 0 indicates no limit. This setting overrides the \fBbandwidth_limit\fP configuration option. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-incremental\fP Execute a block\-level incremental backup. You must provide a \fBBACKUP_ID\fP or a shortcut to a previous backup, which will serve as the parent backup for the incremental backup. .sp \fBNOTE:\fP .INDENT 7.0 .INDENT 3.5 The backup to be and the parent backup must have \fBbackup_method=postgres\fP\&. .UNINDENT .UNINDENT .TP .B \fB\-\-immediate\-checkpoint\fP Forces the initial checkpoint to be executed as soon as possible, overriding any value set for the \fBimmediate_checkpoint\fP parameter in the configuration file. .TP .B \fB\-j\fP / \fB\-\-jobs\fP Specify the number of parallel workers to use for copying files during the backup. This setting overrides the \fBparallel_jobs\fP parameter if it\(aqs specified in the configuration file. .TP .B \fB\-\-jobs\-start\-batch\-period\fP Specify the time period, in seconds, for starting a single batch of jobs. This value overrides the \fBparallel_jobs_start_batch_period\fP parameter if it is set in the configuration file. The default is \fB1\fP second. .TP .B \fB\-\-jobs\-start\-batch\-size\fP Specify the maximum number of parallel workers to initiate in a single batch. This value overrides the \fBparallel_jobs_start_batch_size\fP parameter if it is defined in the configuration file. The default is \fB10\fP workers. .TP .B \fB\-\-keepalive\-interval\fP Specify an interval, in seconds, for sending a heartbeat query to the server to keep the libpq connection active during a Rsync backup. The default is \fB60\fP seconds. A value of \fB0\fP disables the heartbeat. .TP .B \fB\-\-manifest\fP Forces the creation of a backup manifest file upon completing a backup. Overrides the \fBautogenerate_manifest\fP parameter from the configuration file. Applicable only to rsync backup strategy. .TP .B \fB\-\-name\fP Specify a friendly name for this backup which can be used in place of the backup ID in barman commands. .TP .B \fB\-\-no\-immediate\-checkpoint\fP Forces the backup to wait for the checkpoint to be executed overriding any value set for the \fBimmediate_checkpoint\fP parameter in the configuration file. .TP .B \fB\-\-no\-manifest\fP Disables the automatic creation of a backup manifest file upon completing a backup. This setting overrides the \fBautogenerate_manifest\fP parameter from the configuration file and applies only to rsync backup strategy. .TP .B \fB\-\-no\-retry\fP There will be no retry in case of an error. It is the same as setting \fB\-\-retry\-times 0\fP\&. .TP .B \fB\-\-retry\-sleep\fP Specify the number of seconds to wait after a failed copy before retrying. This setting applies to both backup and recovery operations and overrides the \fBbasebackup_retry_sleep\fP parameter if it is defined in the configuration file. .TP .B \fB\-\-retry\-times\fP Specify the number of times to retry the base backup copy in case of an error. This applies to both backup and recovery operations and overrides the \fBbasebackup_retry_times\fP parameter if it is set in the configuration file. .TP .B \fB\-\-reuse\-backup\fP Overrides the behavior of the \fBreuse_backup\fP option configured in the configuration file. The possible values are: .INDENT 7.0 .IP \(bu 2 \fBoff\fP: Do not reuse the last available backup. .IP \(bu 2 \fBcopy\fP: Reuse the last available backup for a server and create copies of unchanged files (reduces backup time). .IP \(bu 2 \fBlink\fP (default): Reuse the last available backup for a server and create hard links to unchanged files (saves both backup time and space). .UNINDENT .sp \fBNOTE:\fP .INDENT 7.0 .INDENT 3.5 This will only have any effect if the last available backup was executed with \fBbackup_method=rsync\fP\&. .UNINDENT .UNINDENT .TP .B \fB\-\-wait\fP / \fB\-w\fP Wait for all necessary WAL files required by the base backup to be archived. .TP .B \fB\-\-wait\-timeout\fP Specify the duration, in seconds, to wait for the required WAL files to be archived before timing out. .UNINDENT .SS Shortcuts .sp Use shortcuts instead of \fBSERVER_NAME\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SS \fBbarman check\-backup\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX check\-backup [ { \-h | \-\-help } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SS Description .sp Check that all necessary WAL files for verifying the consistency of a physical backup are properly archived. This command is automatically executed by the cron job and at the end of each backup operation. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS Shortcuts .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SS \fBbarman check\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX check [ { \-h | \-\-help } ] [ \-\-nagios ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SS Description .sp Display status information about a server, such as SSH connection, Postgres version, configuration and backup directories, archiving and streaming processes, replication slots, and more. Use \fBall\fP as shortcut to show diagnostic information for all configured servers. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-nagios\fP Nagios plugin compatible output. .UNINDENT .SS Shortcuts .sp Use shortcuts instead of \fBSERVER_NAME\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SS \fBbarman config\-switch\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX config\-switch [ { \-h | \-\-help } ] SERVER_NAME { \-\-reset | MODEL_NAME } .EE .UNINDENT .UNINDENT .SS Description .sp Apply a set of configuration overrides from the model to a server in Barman. The final configuration will combine or override the server\(aqs existing settings with the ones specified in the model. You can reset the server configurations with the \fB\-\-reset\fP argument. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 Only one model can be active at a time for a given server. .UNINDENT .UNINDENT .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fBMODEL_NAME\fP Name of the model. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-reset\fP Reset the server\(aqs configurations. .UNINDENT .SS \fBbarman config\-update\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX config\-update [ { \-h | \-\-help } ] STRING .EE .UNINDENT .UNINDENT .SS Description .sp Create or update the configurations for servers and/or models in Barman. The parameter should be a JSON string containing an array of documents. Each document must include a \fBscope\fP key, which can be either server or model, and either a \fBserver_name\fP or \fBmodel_name\fP key, depending on the scope value. Additionally, the document should include other keys representing Barman configuration options and their desired values. .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 The barman \fBconfig\-update\fP command writes configuration options to a file named \fB\&.barman.auto.conf\fP, located in the \fBbarman_home\fP directory. This configuration file has higher precedence and will override values from the global Barman configuration file (usually \fB/etc/barman.conf\fP) and from any included files specified in \fBconfiguration_files_directory\fP (typically files in \fB/etc/barman.d\fP). Be aware of this if you decide to manually modify configuration options in those files later. .UNINDENT .UNINDENT .SS Parameters .INDENT 0.0 .TP .B \fBSTRING\fP List of JSON formatted string. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS Example .sp \fBJSON_STRING=\(aq[{“scope”: “server”, “server_name”: “my_server”, “archiver”: “on”, “streaming_archiver”: “off”}]\(aq\fP .SS \fBbarman cron\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX cron [ { \-h | \-\-help } ] [ \-\-keep\-descriptors ] .EE .UNINDENT .UNINDENT .SS Description .sp Carry out maintenance tasks, such as enforcing retention policies or managing WAL files. .SS Parameters .INDENT 0.0 .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-keep\-descriptors\fP Keep the ^stdout^ and ^stderr^ streams of the Barman subprocesses connected to the main process. This is especially useful for Docker\-based installations. .UNINDENT .SS \fBbarman delete\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX delete [ { \-h | \-\-help } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SS Description .sp Delete the specified backup. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS Shortcuts .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SS \fBbarman diagnose\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX diagnose [ { \-h | \-\-help } ] [ \-\-show\-config\-source ] .EE .UNINDENT .UNINDENT .SS Description .sp Display diagnostic information about the Barman node, which is the server where Barman is installed, as well as all configured Postgres servers. This includes details such as global configuration, SSH version, Python version, rsync version, the current configuration and status of all servers, and many more. .SS Parameters .INDENT 0.0 .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-show\-config\-source\fP Include the source file which provides the effective value for each configuration option. .UNINDENT .SS \fBbarman generate\-manifest\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX generate\-manifest [ { \-h | \-\-help } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SS Description .sp Generates a \fBbackup_manifest\fP file for a backup. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS Shortcuts .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SS \fBbarman get\-wal\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX get\-wal [ { \-\-bzip | \-j } ] [ { \-\-gzip | \-z | \-x } ] [ { \-h | \-\-help } ] [ \-\-keep\-compression ] [ { \-\-output\-directory | \-o } OUTPUT_DIRECTORY ] [ { \-\-peek | \-p } VALUE ] [ { \-P | \-\-partial } ] [ { \-t | \-\-test } ] SERVER_NAME WAL_NAME .EE .UNINDENT .UNINDENT .SS Description .sp Retrieve a WAL file from the xlog archive of a specified server. By default, if the requested WAL file is found, it is returned as uncompressed content to \fBSTDOUT\fP\&. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBWAL_NAME\fP Id of the backup in barman catalog. .TP .B \fB\-\-bzip2\fP / \fB\-j\fP Output will be compressed using bzip2. .TP .B \fB\-\-gzip\fP / \fB\-z\fP / \fB\-x\fP Output will be compressed using gzip. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-keep\-compression\fP Do not uncompress the file content. The output will be the original compressed file. .TP .B \fB\-\-output\-directory\fP / \fB\-o\fP Destination directory where barman will store the WAL file. .TP .B \fB\-\-peek\fP / \fB\-p\fP Specify an integer value greater than or equal to 1 to retrieve WAL files from the specified WAL file up to the value specified by this parameter. When using this option, \fBget\-wal\fP returns a list of zero to the specified WAL segment names, with one name per row. .TP .B \fB\-P\fP / \fB\-\-partial\fP Additionally, collect partial WAL files (.partial). .TP .B \fB\-t\fP / \fB\-\-test\fP Test both the connection and configuration of the specified Postgres server in Barman for WAL retrieval. When this option is used, the required \fBWAL_NAME\fP argument is disregarded. .UNINDENT .sp \fBWARNING:\fP .INDENT 0.0 .INDENT 3.5 \fB\-z\fP / \fB\-\-gzip\fP and \fB\-j\fP / \fB\-\-bzip2\fP options are deprecated and will be removed in the future. For WAL compression, please make sure to enable it directly on the Barman server via the \fBcompression\fP configuration option. .UNINDENT .UNINDENT .SS \fBbarman keep\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX keep [ { \-h | \-\-help } ] { { \-r | \-\-release } | { \-s | \-\-status } | \-\-target { full | standalone } } SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SS Description .sp Mark the specified backup with a \fBtarget\fP as an archival backup to be retained indefinitely, overriding any active retention policies. You can also check the keep \fBstatus\fP of a backup and \fBrelease\fP the keep mark from a backup. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-r\fP / \fB\-\-release\fP Release the keep mark from this backup. This will remove its archival status and make it available for deletion, either directly or by retention policy. .TP .B \fB\-s\fP / \fB\-\-status\fP Report the archival status of the backup. The status will be either \fBfull\fP or \fBstandalone\fP for archival backups, or \fBnokeep\fP for backups that have not been designated as archival. .TP .B \fB\-\-target\fP Define the recovery target for the archival backup. The possible values are: .INDENT 7.0 .IP \(bu 2 \fBfull\fP: The backup can be used to recover to the most recent point in time. To support this, Barman will keep all necessary WALs to maintain the backup\(aqs consistency as well as any subsequent WALs. .IP \(bu 2 \fBstandalone\fP: The backup can only be used to restore the server to its state at the time of the backup. Barman will retain only the WALs required to ensure the backup\(aqs consistency. .UNINDENT .UNINDENT .SS Shortcuts .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SS \fBbarman list\-backups\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX list\-backups [ { \-h | \-\-help } ] [ \-\-minimal ] SERVER_NAME .EE .UNINDENT .UNINDENT .SS Description .sp Display the available backups for a server. This command is useful for retrieving both the backup ID and the backup type. You can find details about this command in \fI\%Catalog usage\fP\&. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-minimal\fP Machine readable output. .UNINDENT .SS Shortcuts .sp Use shortcuts instead of \fBSERVER_NAME\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SS \fBbarman list\-files\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX list\-files [ { \-h | \-\-help } ] [ \-\-target { data | full | standalone | wal } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SS Description .sp List all files in a specific backup. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-target\fP Define specific files to be listed. The possible values are: .INDENT 7.0 .IP \(bu 2 \fBstandalone\fP (default): List the base backup files, including required WAL files. .IP \(bu 2 \fBdata\fP: List just the data files. .IP \(bu 2 \fBwal\fP: List all the WAL files between the start of the base backup and the end of the log or the start of the following base backup (depending on whether the specified base backup is the most recent one available). .IP \(bu 2 \fBfull\fP: same as \fBdata\fP + \fBwal\fP\&. .UNINDENT .UNINDENT .SS Shortcuts .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SS \fBbarman list\-servers\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX list\-servers [ { \-h | \-\-help } ] [ \-\-minimal ] .EE .UNINDENT .UNINDENT .SS Description .sp Display all configured servers along with their descriptions. .SS Parameters .INDENT 0.0 .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-minimal\fP Machine readable output. .UNINDENT .SS \fBbarman lock\-directory\-cleanup\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX lock\-directory\-cleanup [ { \-h | \-\-help } ] .EE .UNINDENT .UNINDENT .SS Description .sp Automatically removes unused lock files from the \fBbarman_lock_directory\fP\&. .SS Parameters .INDENT 0.0 .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS \fBbarman put\-wal\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX put\-wal [ { \-h | \-\-help } ] [ { \-t | \-\-test } ] SERVER_NAME .EE .UNINDENT .UNINDENT .SS Description .sp Receive a WAL file from a remote server and securely save it into the server incoming directory. The WAL file should be provided via \fBSTDIN\fP, encapsulated in a tar stream along with a \fBSHA256SUMS\fP or \fBMD5SUMS\fP file for validation (\fBsha256\fP is the default hash algorithm, but the user can choose \fBmd5\fP when setting the \fBarchive\-command\fP via \fBbarman\-wal\-archive\fP). This command is intended to be executed via SSH from a remote \fBbarman\-wal\-archive\fP utility (included in the barman\-cli package). Avoid using this command directly unless you fully manage the content of the files. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-t\fP / \fB\-\-test\fP Test both the connection and configuration of the specified Postgres server in Barman for WAL retrieval. .UNINDENT .SS \fBbarman rebuild\-xlogdb\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX rebuild\-xlogdb [ { \-h | \-\-help } ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SS Description .sp Rebuild the WAL file metadata for a server (or for all servers using the \fBall\fP shortcut) based on the disk content. The WAL archive metadata is stored in the \fBxlog.db\fP file, with each Barman server maintaining its own copy. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS Shortcuts .sp Use shortcuts instead of \fBSERVER_NAME\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SS \fBbarman recover\fP .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 This command is deprecated. Use the \fI\%barman restore\fP command instead. .UNINDENT .UNINDENT .SS \fBbarman receive\-wal\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX receive\-wal [ \-\-create\-slot ] [ \-\-drop\-slot ] [ { \-h | \-\-help } ] [ \-\-reset ] [ \-\-stop ] SERVER_NAME .EE .UNINDENT .UNINDENT .SS Description .sp Initiate the streaming of transaction logs for a server. This process uses \fBpg_receivewal\fP or \fBpg_receivexlog\fP to receive WAL files from Postgres servers via the streaming protocol. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fB\-\-create\-slot\fP Create the physical replication slot configured with the \fBslot_name\fP configuration parameter. .TP .B \fB\-\-drop\-slot\fP Drop the physical replication slot configured with the \fBslot_name\fP configuration parameter. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-reset\fP Reset the status of \fBreceive\-wal\fP, restarting the streaming from the current WAL file of the server. .TP .B \fB\-\-stop\fP Stop the process for the server. .UNINDENT .SS \fBbarman restore\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX restore [ \-\-aws\-region AWS_REGION } ] [ \-\-azure\-resource\-group AZURE_RESOURCE_GRP ] [ \-\-bwlimit KBPS ] [ \-\-exclusive ] [ \-\-gcp\-zone GCP_ZONE ] [ { \-\-get\-wal | \-\-no\-get\-wal } ] [ { \-h | \-\-help } ] [ { \-j | \-\-jobs } PARALLEL_WORKERS ] [ \-\-jobs\-start\-batch\-period SECONDS ] [ \-\-jobs\-start\-batch\-size NUMBER ] [ \-\-local\-staging\-path PATH ] [ { \-\-network\-compression | \-\-no\-network\-compression } ] [ \-\-no\-retry ] [ \-\-recovery\-conf\-filename FILENAME ] [ \-\-recovery\-staging\-path PATH ] [ \-\-remote\-ssh\-command STRING ] [ \-\-retry\-sleep SECONDS ] [ \-\-retry\-times NUMBER ] [ \-\-snapshot\-recovery\-instance INSTANCE_NAME ] [ \-\-snapshot\-recovery\-zone GCP_ZONE ] [ \-\-standby\-mode ] [ \-\-tablespace NAME:LOCATION [ \-\-tablespace NAME:LOCATION ... ] ] [ \-\-target\-action { pause | shutdown | promote } ] [ \-\-target\-immediate ] [ \-\-target\-lsn LSN ] [ \-\-target\-name RESTORE_POINT_NAME ] [ \-\-target\-time TIMESTAMP ] [ \-\-target\-tli TLI ] [ \-\-target\-xid XID ] [ \-\-staging\-wal\-directory ] SERVER_NAME BACKUP_ID DESTINATION_DIR .EE .UNINDENT .UNINDENT .SS Description .sp Execute a PostreSQL server restore operation. Barman will restore the backup from a server in the destination directory. The restoration can be performed locally (on the barman node itself) or remotely (on another machine accessible via SSH). The location is determined by whether or not the \fB\-\-remote\-ssh\-command\fP option is used. More information on this command can be found in the \fI\%Recovery\fP section. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in the barman catalog. Use \fBauto\fP to have Barman automatically find the most suitable backup for the restore operation. .TP .B \fBDESTINATION_DIR\fP Destination directory to restore the backup. .TP .B \fB\-\-aws\-region\fP Specify the AWS region where the instance and disks for snapshot recovery are located. This option allows you to override the \fBaws_region\fP value in the Barman configuration. .TP .B \fB\-\-azure\-resource\-group\fP Specify the Azure resource group containing the instance and disks for snapshot recovery. This option allows you to override the \fBazure_resource_group\fP value in the Barman configuration. .TP .B \fB\-\-bwlimit\fP Specify the maximum transfer rate in kilobytes per second. A value of \fB0\fP indicates no limit. This setting overrides the \fBbandwidth_limit\fP configuration option. .TP .B \fB\-\-exclusive\fP Set target (time, XID or LSN) to be non inclusive. .TP .B \fB\-\-gcp\-zone\fP Specify the GCP zone where the instance and disks for snapshot recovery are located. This option allows you to override the \fBgcp_zone\fP value in the Barman configuration. .TP .B \fB\-\-get\-wal\fP / \fB\-\-no\-get\-wal\fP Enable/disable usage of \fBget\-wal\fP for WAL fetching during recovery. Default is based on \fBrecovery_options\fP setting. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-j\fP / \fB\-\-jobs\fP Specify the number of parallel workers to use for copying files during the backup. This setting overrides the \fBparallel_jobs\fP parameter if it is specified in the configuration file. .TP .B \fB\-\-jobs\-start\-batch\-period\fP Specify the time period, in seconds, for starting a single batch of jobs. This value overrides the \fBparallel_jobs_start_batch_period\fP parameter if it is set in the configuration file. The default is \fB1\fP second. .TP .B \fB\-\-jobs\-start\-batch\-size\fP Specify the maximum number of parallel workers to initiate in a single batch. This value overrides the \fBparallel_jobs_start_batch_size\fP parameter if it is defined in the configuration file. The default is \fB10\fP workers. .TP .B \fB\-\-local\-staging\-path\fP Specify path on the Barman host where the chain of backups will be combined before being copied to the destination directory. The contents created within the staging path will be removed upon completion of the restore process. This option is necessary for restoring from block\-level incremental backups and has no effect otherwise. .TP .B \fB\-\-network\-compression\fP / \fB\-\-no\-network\-compression\fP Enable/disable network compression during remote restore. Default is based on \fBnetwork_compression\fP configuration setting. .TP .B \fB\-\-no\-retry\fP There will be no retry in case of an error. It is the same as setting \fB\-\-retry\-times 0\fP\&. .TP .B \fB\-\-recovery\-conf\-filename\fP Specify the name of the file where Barman should write recovery options when recovering backups for Postgres versions 12 and later. By default, this is set to \fBpostgresql.auto.conf\fP\&. However, if \fB\-\-recovery\-conf\-filename\fP is specified, recovery options will be written to the specified value instead. While the default value is suitable for most Postgres installations, this option allows you to specify an alternative location if Postgres is managed by tools that alter the configuration mechanism (for example, if \fBpostgresql.auto.conf\fP is symlinked to \fB/dev/null\fP). .TP .B \fB\-\-recovery\-staging\-path\fP Specify a path on the recovery host where files for a compressed backup will be staged before being uncompressed to the destination directory. Backups will be staged in their own directory within the staging path, following the naming convention: \fBbarman\-staging\-SERVER_NAME\-BACKUP_ID\fP\&. This staging directory will be removed after the restore process is complete. This option is mandatory for restoring from compressed backups and has no effect otherwise. .TP .B \fB\-\-remote\-ssh\-command\fP This option enables remote restore by specifying the secure shell command to execute on a remote host. It functions similarly to the \fBssh_command\fP server option in the configuration file for remote restore, that is, \fB\(aqssh USER@SERVER\(aq\fP\&. .TP .B \fB\-\-retry\-sleep\fP Specify the number of seconds to wait after a failed copy before retrying. This setting applies to both backup and restore operations and overrides the \fBbasebackup_retry_sleep\fP parameter if it is defined in the configuration file. .TP .B \fB\-\-retry\-times\fP Specify the number of times to retry the base backup copy in case of an error. This applies to both backup and restore operations and overrides the \fBbasebackup_retry_times\fP parameter if it is set in the configuration file. .TP .B \fB\-\-snapshot\-recovery\-instance\fP Specify the name of the instance where the disks recovered from the snapshots are attached. This option is necessary when recovering backups created with \fBbackup_method=snapshot\fP\&. .TP .B \fB\-\-snapshot\-recovery\-zone\fP (deprecated) Zone containing the instance and disks for the snapshot recovery (deprecated: replaced by \fB\-\-gcp\-zone\fP) .TP .B \fB\-\-standby\-mode\fP Whether to start the Postgres server as a standby. .TP .B \fB\-\-tablespace\fP Specify tablespace relocation rule. \fBNAME\fP is the tablespace name and \fBLOCATION\fP is the recovery host destination path to restore the tablespace. .TP .B \fB\-\-target\-action\fP Trigger the specified action when the recovery target is reached. This option requires defining a target along with one of these actions. The possible values are: .INDENT 7.0 .IP \(bu 2 \fBpause\fP: Once recovery target is reached, the server is started in pause state, allowing users to inspect the instance .IP \(bu 2 \fBpromote\fP: Once recovery target is reached, the server will exit the recovery operation and is promoted as a master. .IP \(bu 2 \fBshutdown\fP: Once recovery target is reached, the server is shut down. .UNINDENT .TP .B \fB\-\-target\-immediate\fP Recovery is completed when a consistent state is reached (end of the base backup). .TP .B \fB\-\-target\-lsn\fP Recover to the specified LSN (Log Sequence Number). Requires Postgres 10 or above. .TP .B \fB\-\-target\-name\fP Recover to the specified name of a restore point previously created with the \fBpg_create_restore_point(name)\fP\&. .TP .B \fB\-\-target\-time\fP Recover to the specified time. Use the format \fBYYYY\-MM\-DD HH:MM:SS.mmm\fP\&. .TP .B \fB\-\-target\-tli\fP Recover the specified timeline. You can use the special values \fBcurrent\fP and \fBlatest\fP in addition to a numeric timeline ID. For Postgres versions 12 and above, the default is to recover to the latest timeline in the WAL archive. For Postgres versions below 12, the default is to recover to the timeline that was current at the time the backup was taken. .TP .B \fB\-\-target\-xid\fP Recover to the specified transaction ID. .TP .B \fB\-\-staging\-wal\-directory\fP A staging directory on the destination host for WAL files when performing PITR. If unspecified, it uses a \fBbarman_wal\fP directory inside the destination directory. .UNINDENT .SS Shortcuts .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SS \fBbarman replication\-status\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX replication\-status [ { \-h | \-\-help } ] [ \-\-minimal ] [ \-\-source { backup\-host | wal\-host } ] [ \-\-target { hot\-standby | wal\-streamer | all } ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SS Description .sp Display real\-time information and status of any streaming clients connected to the specified server. Specify \fBall\fP shortcut to diplay information for all configured servers. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-minimal\fP Machine readable output. .TP .B \fB\-\-source\fP The possible values are: .INDENT 7.0 .IP \(bu 2 \fBbackup\-host\fP (default): List clients using the backup connection information for a server. .IP \(bu 2 \fBwal\-host\fP: List clients using the WAL streaming connection information for a server. .UNINDENT .TP .B \fB\-\-target\fP The possible values are: .INDENT 7.0 .IP \(bu 2 \fBhot\-standby\fP: List only hot standby servers. .IP \(bu 2 \fBwal\-streamer\fP: List only WAL streaming clients, such as \fBpg_receivewal\fP\&. .IP \(bu 2 \fBall\fP (default): List all streaming clients. .UNINDENT .UNINDENT .SS Shortcuts .sp Use shortcuts instead of \fBSERVER_NAME\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SS \fBbarman show\-backup\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX show\-backup [ { \-h | \-\-help } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SS Description .sp Display detailed information about a specific backup. You can find details about this command in \fI\%Catalog usage\fP\&. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS Shortcuts .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SS \fBbarman show\-servers\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX show\-servers [ { \-h | \-\-help } ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SS Description .sp Display detailed information about a server, including \fBconninfo\fP, \fBbackup_directory\fP, \fBwals_directory\fP, \fBarchive_command\fP, and many more. To view information about all configured servers, specify the \fBall\fP shortcut instead of the server name. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS Shortcuts .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SS \fBbarman status\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX status [ { \-h | \-\-help } ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SS Description .sp Display information about a server\(aqs status, including details such as the state, Postgres version, WAL information, available backups and more. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS Shortcuts .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SS \fBbarman switch\-wal\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX switch\-wal [ \-\-archive ] [ \-\-archive\-timeout ] [ \-\-force ] [ { \-h | \-\-help } ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SS Description .sp Execute \fBpg_switch_wal()\fP on the target server (Postgres versions 10 and later) or \fBpg_switch_xlog()\fP (for Postgres versions 8.3 to 9.6). .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-\-archive\fP Waits for one WAL file to be archived. If no WAL file is archived within a specified time (default: \fB30\fP seconds), Barman will terminate with a failure exit code. This option is also available on standby servers. .TP .B \fB\-\-archive\-timeout\fP Specify the amount of time in seconds (default: \fB30\fP seconds) that the archiver will wait for a new WAL file to be archived before timing out. This option is also available on standby servers. .TP .B \fB\-\-force\fP Forces the switch by executing a CHECKPOINT before \fBpg_switch_wal()\fP\&. .sp \fBNOTE:\fP .INDENT 7.0 .INDENT 3.5 Running a CHECKPOINT may increase I/O load on the Postgres server, so use this option cautiously. .UNINDENT .UNINDENT .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS \fBbarman switch\-xlog\fP .SS Description .sp Alias for the \fBswitch\-wal\fP command. .SS \fBbarman sync\-backup\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX sync\-backup [ { \-h | \-\-help } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SS Description .sp This command synchronizes a passive node with its primary by copying all files from a backup present on the server node. It is available only for passive nodes and uses the \fBprimary_ssh_command\fP option to establish a secure connection with the primary node. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS Shortcuts .sp For some commands, instead of using the timestamp backup ID, you can use the following shortcuts or aliases to identify a backup for a given server: .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SS \fBbarman sync\-info\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX sync\-info [ { \-h | \-\-help } ] [ \-\-primary ] SERVER_NAME [ LAST_WAL [ LAST_POS ] ] .EE .UNINDENT .UNINDENT .SS Description .sp Gather information about the current status of a Barman server for synchronization purposes. .sp This command returns a JSON output for a server that includes: all successfully completed backups, all archived WAL files, the configuration, the last WAL file read from \fBxlog.db\fP, and its position within the file. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBLAST_WAL\fP Instructs sync\-info to skip any WAL files that precede the specified file (for incremental synchronization). .TP .B \fBLAST_POS\fP Hint for quickly positioning in the \fBxlog.db\fP file (for incremental synchronization). .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-primary\fP Execute the sync\-info on the primary node (if set). .UNINDENT .SS \fBbarman sync\-wals\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX sync\-wals [ { \-h | \-\-help } ] SERVER_NAME .EE .UNINDENT .UNINDENT .SS Description .sp This command synchronizes a passive node with its primary by copying all archived WAL files from the server node. It is available only for passive nodes and utilizes the \fBprimary_ssh_command\fP option to establish a secure connection with the primary node. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS \fBbarman verify\-backup\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX verify\-backup [ { \-h | \-\-help } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SS Description .sp Runs \fBpg_verifybackup\fP on a backup manifest file (available since Postgres version 13). For rsync backups, it can be used after creating a manifest file using the \fBgenerate\-manifest\fP command. Requires \fBpg_verifybackup\fP to be installed on the backup server. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SS Shortcuts .sp For some commands, instead of using the timestamp backup ID, you can use the following shortcuts or aliases to identify a backup for a given server: .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SS \fBbarman verify\fP .SS Description .sp Alias for \fBverify\-backup\fP command. .SH BARMAN-CLI COMMANDS .sp The \fBbarman\-cli\fP package includes a collection of recommended client utilities that should be installed alongside the Postgres server. Here are the command references for both utilities. .SS \fBbarman\-wal\-archive\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-wal\-archive [ { \-h | \-\-help } ] [ { \-V | \-\-version } ] [ { \-U | \-\-user } USER ] [ \-\-port PORT ] [ { \-c | \-\-config } CONFIG ] [ { \-t | \-\-test } ] [ \-\-md5 ] BARMAN_HOST SERVER_NAME WAL_PATH .EE .UNINDENT .UNINDENT .SS Description .sp This script can be utilized in the \fBarchive_command\fP of a Postgres server to transfer WAL files to a Barman host using the \fBput\-wal\fP command (introduced in Barman 2.6). It establishes an SSH connection to the Barman host, enabling seamless integration of Barman within Postgres clusters for improved business continuity. .sp \fBExit Statuses\fP are: .INDENT 0.0 .IP \(bu 2 \fB0\fP for \fBSUCCESS\fP\&. .IP \(bu 2 \fBnon\-zero\fP for \fBFAILURE\fP\&. .UNINDENT .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP The server name configured in Barman for the Postgres server from which the WAL file is retrieved. .TP .B \fBBARMAN_HOST\fP The host of the Barman server. .TP .B \fBWAL_PATH\fP The value of the \(aq%p\(aq keyword (according to \fBarchive_command\fP). .TP .B \fB\-h\fP / \fB\-\-help\fP Display a help message and exit. .TP .B \fB\-V\fP / \fB\-\-version\fP Display the program\(aqs version number and exit. .TP .B \fB\-U\fP / \fB\-\-user\fP Specify the user for the SSH connection to the Barman server (defaults to \fBbarman\fP). .TP .B \fB\-\-port\fP Define the port used for the SSH connection to the Barman server. .TP .B \fB\-c\fP / \fB\-\-config\fP Specify the configuration file on the Barman server. .TP .B \fB\-t\fP / \fB\-\-test\fP Test the connection and configuration of the specified Postgres server in Barman to ensure it is ready to receive WAL files. This option ignores the mandatory argument \fBWAL_PATH\fP\&. .TP .B \fB\-\-md5\fP Use MD5 instead of SHA256 as the hash algorithm to calculate the checksum of the WAL file when transmitting it to the Barman server. This is used to maintain compatibility with older server versions, as older versions of Barman server used to support only MD5. .UNINDENT .SS \fBbarman\-wal\-restore\fP .SS Synopsis .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-wal\-restore [ { \-h | \-\-help } ] [ { \-V | \-\-version } ] [ { \-U | \-\-user } USER ] [ \-\-port PORT ] [ { \-s | \-\-sleep } SECONDS ] [ { \-p | \-\-parallel } JOBS ] [ \-\-spool\-dir SPOOL_DIR ] [ { \-P | \-\-partial } ] [ { { \-z | \-\-gzip } | { \-j | \-\-bzip2 } | \-\-keep\-compression } ] [ { \-c | \-\-config } CONFIG ] [ { \-t | \-\-test } ] BARMAN_HOST SERVER_NAME WAL_NAME WAL_DEST .EE .UNINDENT .UNINDENT .SS Description .sp This script serves as a \fBrestore_command\fP for Postgres servers, enabling the retrieval of WAL files through Barman\(aqs \fBget\-wal\fP feature. It establishes an SSH connection to the Barman host and facilitates the integration of Barman within Postgres clusters, enhancing business continuity. .sp \fBExit Statuses\fP are: .INDENT 0.0 .IP \(bu 2 \fB0\fP for \fBSUCCESS\fP\&. .IP \(bu 2 \fB1\fP for remote get\-wal command \fBFAILURE\fP, likely because the requested WAL could not be found. .IP \(bu 2 \fB2\fP for ssh connection \fBFAILURE\fP\&. .IP \(bu 2 Any other \fBnon\-zero\fP for \fBFAILURE\fP\&. .UNINDENT .SS Parameters .INDENT 0.0 .TP .B \fBSERVER_NAME\fP The server name configured in Barman for the Postgres server from which the WAL file is retrieved. .TP .B \fBBARMAN_HOST\fP The host of the Barman server. .TP .B \fBWAL_NAME\fP The value of the \(aq%f\(aq keyword (according to \fBrestore_command\fP). .TP .B \fBWAL_DEST\fP The value of the \(aq%p\(aq keyword (according to \fBrestore_command\fP). .TP .B \fB\-h\fP / \fB\-\-help\fP Display a help message and exit. .TP .B \fB\-V\fP / \fB\-\-version\fP Display the program\(aqs version number and exit. .TP .B \fB\-U\fP / \fB\-\-user\fP Specify the user for the SSH connection to the Barman server (defaults to \fBbarman\fP). .TP .B \fB\-\-port\fP Define the port used for the SSH connection to the Barman server. .TP .B \fB\-s\fP / \fB\-\-sleep\fP Pause for \fBSECONDS\fP after a failed \fBget\-wal\fP request (defaults to \fB0\fP \- no wait). .TP .B \fB\-p\fP / \fB\-\-parallel\fP Indicate the number of files to \fBpeek\fP and transfer simultaneously (defaults to \fB0\fP \- disabled). .TP .B \fB\-\-spool\-dir\fP Specify the spool directory for WAL files (defaults to \fB/var/tmp/walrestore\fP). .TP .B \fB\-P\fP / \fB\-\-partial\fP Include partial WAL files (.partial) in the retrieval. .TP .B \fB\-z\fP / \fB\-\-gzip\fP Transfer WAL files compressed with \fBgzip\fP\&. .TP .B \fB\-j\fP / \fB\-\-bzip2\fP Transfer WAL files compressed with \fBbzip2\fP\&. .TP .B \fB\-\-keep\-compression\fP If specified, compressed files will be trasnfered as\-is and decompressed on arrival on the client\-side. .TP .B \fB\-c\fP / \fB\-\-config\fP Specify the configuration file on the Barman server. .TP .B \fB\-t\fP / \fB\-\-test\fP Test the connection and configuration of the specified Postgres server in Barman to ensure it is ready to receive WAL files. This option ignores the mandatory arguments \fBWAL_NAME\fP and \fBWAL_DEST\fP\&. .UNINDENT .sp \fBWARNING:\fP .INDENT 0.0 .INDENT 3.5 \fB\-z\fP / \fB\-\-gzip\fP and \fB\-j\fP / \fB\-\-bzip2\fP options are deprecated and will be removed in the future. For WAL compression, please make sure to enable it directly on the Barman server via the \fBcompression\fP configuration option. .UNINDENT .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-backup.10000644000175100001660000001452115002177425016756 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-BACKUP" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-backup \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX backup [ \-\-bwlimit KBPS ] [ { \-h | \-\-help } ] [ \-\-incremental BACKUP_ID ] [ \-\-immediate\-checkpoint ] [ { \-j | \-\-jobs } PARALLEL_WORKERS ] [ \-\-jobs\-start\-batch\-period PERIOD ] [ \-\-jobs\-start\-batch\-size SIZE ] [ \-\-keepalive\-interval SECONDS ] [ \-\-manifest ] [ \-\-name NAME ] [ \-\-no\-immediate\-checkpoint ] [ \-\-no\-manifest ] [ \-\-no\-retry ] [ \-\-retry\-sleep SECONDS ] [ \-\-retry\-times NUMBER ] [ \-\-reuse\-backup { off | copy | link } ] [ { \-\-wait | \-w } ] [ \-\-wait\-timeout SECONDS ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Execute a PostreSQL server backup. Barman will use the parameters specified in the Global and Server configuration files. Specify \fBall\fP shortcut instead of the server name to execute backups from all servers configured in the Barman node. You can also specify multiple server names in sequence to execute backups for specific servers. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fB\-\-bwlimit\fP Specify the maximum transfer rate in kilobytes per second. A value of 0 indicates no limit. This setting overrides the \fBbandwidth_limit\fP configuration option. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-\-incremental\fP Execute a block\-level incremental backup. You must provide a \fBBACKUP_ID\fP or a shortcut to a previous backup, which will serve as the parent backup for the incremental backup. .sp \fBNOTE:\fP .INDENT 7.0 .INDENT 3.5 The backup to be and the parent backup must have \fBbackup_method=postgres\fP\&. .UNINDENT .UNINDENT .TP .B \fB\-\-immediate\-checkpoint\fP Forces the initial checkpoint to be executed as soon as possible, overriding any value set for the \fBimmediate_checkpoint\fP parameter in the configuration file. .TP .B \fB\-j\fP / \fB\-\-jobs\fP Specify the number of parallel workers to use for copying files during the backup. This setting overrides the \fBparallel_jobs\fP parameter if it\(aqs specified in the configuration file. .TP .B \fB\-\-jobs\-start\-batch\-period\fP Specify the time period, in seconds, for starting a single batch of jobs. This value overrides the \fBparallel_jobs_start_batch_period\fP parameter if it is set in the configuration file. The default is \fB1\fP second. .TP .B \fB\-\-jobs\-start\-batch\-size\fP Specify the maximum number of parallel workers to initiate in a single batch. This value overrides the \fBparallel_jobs_start_batch_size\fP parameter if it is defined in the configuration file. The default is \fB10\fP workers. .TP .B \fB\-\-keepalive\-interval\fP Specify an interval, in seconds, for sending a heartbeat query to the server to keep the libpq connection active during a Rsync backup. The default is \fB60\fP seconds. A value of \fB0\fP disables the heartbeat. .TP .B \fB\-\-manifest\fP Forces the creation of a backup manifest file upon completing a backup. Overrides the \fBautogenerate_manifest\fP parameter from the configuration file. Applicable only to rsync backup strategy. .TP .B \fB\-\-name\fP Specify a friendly name for this backup which can be used in place of the backup ID in barman commands. .TP .B \fB\-\-no\-immediate\-checkpoint\fP Forces the backup to wait for the checkpoint to be executed overriding any value set for the \fBimmediate_checkpoint\fP parameter in the configuration file. .TP .B \fB\-\-no\-manifest\fP Disables the automatic creation of a backup manifest file upon completing a backup. This setting overrides the \fBautogenerate_manifest\fP parameter from the configuration file and applies only to rsync backup strategy. .TP .B \fB\-\-no\-retry\fP There will be no retry in case of an error. It is the same as setting \fB\-\-retry\-times 0\fP\&. .TP .B \fB\-\-retry\-sleep\fP Specify the number of seconds to wait after a failed copy before retrying. This setting applies to both backup and recovery operations and overrides the \fBbasebackup_retry_sleep\fP parameter if it is defined in the configuration file. .TP .B \fB\-\-retry\-times\fP Specify the number of times to retry the base backup copy in case of an error. This applies to both backup and recovery operations and overrides the \fBbasebackup_retry_times\fP parameter if it is set in the configuration file. .TP .B \fB\-\-reuse\-backup\fP Overrides the behavior of the \fBreuse_backup\fP option configured in the configuration file. The possible values are: .INDENT 7.0 .IP \(bu 2 \fBoff\fP: Do not reuse the last available backup. .IP \(bu 2 \fBcopy\fP: Reuse the last available backup for a server and create copies of unchanged files (reduces backup time). .IP \(bu 2 \fBlink\fP (default): Reuse the last available backup for a server and create hard links to unchanged files (saves both backup time and space). .UNINDENT .sp \fBNOTE:\fP .INDENT 7.0 .INDENT 3.5 This will only have any effect if the last available backup was executed with \fBbackup_method=rsync\fP\&. .UNINDENT .UNINDENT .TP .B \fB\-\-wait\fP / \fB\-w\fP Wait for all necessary WAL files required by the base backup to be archived. .TP .B \fB\-\-wait\-timeout\fP Specify the duration, in seconds, to wait for the required WAL files to be archived before timing out. .UNINDENT .SH SHORTCUTS .sp Use shortcuts instead of \fBSERVER_NAME\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-generate-manifest.10000644000175100001660000000372415002177425021112 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-GENERATE-MANIFEST" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-generate-manifest \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX generate\-manifest [ { \-h | \-\-help } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Generates a \fBbackup_manifest\fP file for a backup. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH SHORTCUTS .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-keep.10000644000175100001660000000617315002177425016441 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-KEEP" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-keep \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX keep [ { \-h | \-\-help } ] { { \-r | \-\-release } | { \-s | \-\-status } | \-\-target { full | standalone } } SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Mark the specified backup with a \fBtarget\fP as an archival backup to be retained indefinitely, overriding any active retention policies. You can also check the keep \fBstatus\fP of a backup and \fBrelease\fP the keep mark from a backup. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-r\fP / \fB\-\-release\fP Release the keep mark from this backup. This will remove its archival status and make it available for deletion, either directly or by retention policy. .TP .B \fB\-s\fP / \fB\-\-status\fP Report the archival status of the backup. The status will be either \fBfull\fP or \fBstandalone\fP for archival backups, or \fBnokeep\fP for backups that have not been designated as archival. .TP .B \fB\-\-target\fP Define the recovery target for the archival backup. The possible values are: .INDENT 7.0 .IP \(bu 2 \fBfull\fP: The backup can be used to recover to the most recent point in time. To support this, Barman will keep all necessary WALs to maintain the backup\(aqs consistency as well as any subsequent WALs. .IP \(bu 2 \fBstandalone\fP: The backup can only be used to restore the server to its state at the time of the backup. Barman will retain only the WALs required to ensure the backup\(aqs consistency. .UNINDENT .UNINDENT .SH SHORTCUTS .sp Use shortcuts instead of \fBBACKUP_ID\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-wal-restore.10000644000175100001660000001027315002177425017755 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-WAL-RESTORE" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-wal-restore \- Barman-cli Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX barman\-wal\-restore [ { \-h | \-\-help } ] [ { \-V | \-\-version } ] [ { \-U | \-\-user } USER ] [ \-\-port PORT ] [ { \-s | \-\-sleep } SECONDS ] [ { \-p | \-\-parallel } JOBS ] [ \-\-spool\-dir SPOOL_DIR ] [ { \-P | \-\-partial } ] [ { { \-z | \-\-gzip } | { \-j | \-\-bzip2 } | \-\-keep\-compression } ] [ { \-c | \-\-config } CONFIG ] [ { \-t | \-\-test } ] BARMAN_HOST SERVER_NAME WAL_NAME WAL_DEST .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp This script serves as a \fBrestore_command\fP for Postgres servers, enabling the retrieval of WAL files through Barman\(aqs \fBget\-wal\fP feature. It establishes an SSH connection to the Barman host and facilitates the integration of Barman within Postgres clusters, enhancing business continuity. .sp \fBExit Statuses\fP are: .INDENT 0.0 .IP \(bu 2 \fB0\fP for \fBSUCCESS\fP\&. .IP \(bu 2 \fB1\fP for remote get\-wal command \fBFAILURE\fP, likely because the requested WAL could not be found. .IP \(bu 2 \fB2\fP for ssh connection \fBFAILURE\fP\&. .IP \(bu 2 Any other \fBnon\-zero\fP for \fBFAILURE\fP\&. .UNINDENT .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP The server name configured in Barman for the Postgres server from which the WAL file is retrieved. .TP .B \fBBARMAN_HOST\fP The host of the Barman server. .TP .B \fBWAL_NAME\fP The value of the \(aq%f\(aq keyword (according to \fBrestore_command\fP). .TP .B \fBWAL_DEST\fP The value of the \(aq%p\(aq keyword (according to \fBrestore_command\fP). .TP .B \fB\-h\fP / \fB\-\-help\fP Display a help message and exit. .TP .B \fB\-V\fP / \fB\-\-version\fP Display the program\(aqs version number and exit. .TP .B \fB\-U\fP / \fB\-\-user\fP Specify the user for the SSH connection to the Barman server (defaults to \fBbarman\fP). .TP .B \fB\-\-port\fP Define the port used for the SSH connection to the Barman server. .TP .B \fB\-s\fP / \fB\-\-sleep\fP Pause for \fBSECONDS\fP after a failed \fBget\-wal\fP request (defaults to \fB0\fP \- no wait). .TP .B \fB\-p\fP / \fB\-\-parallel\fP Indicate the number of files to \fBpeek\fP and transfer simultaneously (defaults to \fB0\fP \- disabled). .TP .B \fB\-\-spool\-dir\fP Specify the spool directory for WAL files (defaults to \fB/var/tmp/walrestore\fP). .TP .B \fB\-P\fP / \fB\-\-partial\fP Include partial WAL files (.partial) in the retrieval. .TP .B \fB\-z\fP / \fB\-\-gzip\fP Transfer WAL files compressed with \fBgzip\fP\&. .TP .B \fB\-j\fP / \fB\-\-bzip2\fP Transfer WAL files compressed with \fBbzip2\fP\&. .TP .B \fB\-\-keep\-compression\fP If specified, compressed files will be trasnfered as\-is and decompressed on arrival on the client\-side. .TP .B \fB\-c\fP / \fB\-\-config\fP Specify the configuration file on the Barman server. .TP .B \fB\-t\fP / \fB\-\-test\fP Test the connection and configuration of the specified Postgres server in Barman to ensure it is ready to receive WAL files. This option ignores the mandatory arguments \fBWAL_NAME\fP and \fBWAL_DEST\fP\&. .UNINDENT .sp \fBWARNING:\fP .INDENT 0.0 .INDENT 3.5 \fB\-z\fP / \fB\-\-gzip\fP and \fB\-j\fP / \fB\-\-bzip2\fP options are deprecated and will be removed in the future. For WAL compression, please make sure to enable it directly on the Barman server via the \fBcompression\fP configuration option. .UNINDENT .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-lock-directory-cleanup.10000644000175100001660000000243115002177425022065 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-LOCK-DIRECTORY-CLEANUP" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-lock-directory-cleanup \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX lock\-directory\-cleanup [ { \-h | \-\-help } ] .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Automatically removes unused lock files from the \fBbarman_lock_directory\fP\&. .SH PARAMETERS .INDENT 0.0 .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-put-wal.10000644000175100001660000000376215002177425017107 0ustar 00000000000000.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-PUT-WAL" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-put-wal \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX put\-wal [ { \-h | \-\-help } ] [ { \-t | \-\-test } ] SERVER_NAME .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Receive a WAL file from a remote server and securely save it into the server incoming directory. The WAL file should be provided via \fBSTDIN\fP, encapsulated in a tar stream along with a \fBSHA256SUMS\fP or \fBMD5SUMS\fP file for validation (\fBsha256\fP is the default hash algorithm, but the user can choose \fBmd5\fP when setting the \fBarchive\-command\fP via \fBbarman\-wal\-archive\fP). This command is intended to be executed via SSH from a remote \fBbarman\-wal\-archive\fP utility (included in the barman\-cli package). Avoid using this command directly unless you fully manage the content of the files. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .TP .B \fB\-t\fP / \fB\-\-test\fP Test both the connection and configuration of the specified Postgres server in Barman for WAL retrieval. .UNINDENT .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-rebuild-xlogdb.10000644000175100001660000000326215002177425020414 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-REBUILD-XLOGDB" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-rebuild-xlogdb \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX rebuild\-xlogdb [ { \-h | \-\-help } ] SERVER_NAME [ SERVER_NAME ... ] .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Rebuild the WAL file metadata for a server (or for all servers using the \fBall\fP shortcut) based on the disk content. The WAL archive metadata is stored in the \fBxlog.db\fP file, with each Barman server maintaining its own copy. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH SHORTCUTS .sp Use shortcuts instead of \fBSERVER_NAME\fP\&. .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBall\fP T} T{ All available servers T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/_build/man/barman-verify-backup.10000644000175100001660000000441215002177425020256 0ustar 00000000000000'\" t .\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "BARMAN-VERIFY-BACKUP" "1" "Apr 24, 2025" "3.13" "Barman" .SH NAME barman-verify-backup \- Barman Sub-Commands .SH SYNOPSIS .INDENT 0.0 .INDENT 3.5 .sp .EX verify\-backup [ { \-h | \-\-help } ] SERVER_NAME BACKUP_ID .EE .UNINDENT .UNINDENT .SH DESCRIPTION .sp Runs \fBpg_verifybackup\fP on a backup manifest file (available since Postgres version 13). For rsync backups, it can be used after creating a manifest file using the \fBgenerate\-manifest\fP command. Requires \fBpg_verifybackup\fP to be installed on the backup server. You can use a shortcut instead of \fBBACKUP_ID\fP\&. .SH PARAMETERS .INDENT 0.0 .TP .B \fBSERVER_NAME\fP Name of the server in barman node .TP .B \fBBACKUP_ID\fP Id of the backup in barman catalog. .TP .B \fB\-h\fP / \fB\-\-help\fP Show a help message and exit. Provides information about command usage. .UNINDENT .SH SHORTCUTS .sp For some commands, instead of using the timestamp backup ID, you can use the following shortcuts or aliases to identify a backup for a given server: .TS box center; l|l. T{ \fBShortcut\fP T} T{ \fBDescription\fP T} _ T{ \fBfirst/oldest\fP T} T{ Oldest available backup for the server, in chronological order. T} _ T{ \fBlast/latest\fP T} T{ Most recent available backup for the server, in chronological order. T} _ T{ \fBlast\-full/latest\-full\fP T} T{ Most recent full backup taken with methods \fBrsync\fP or \fBpostgres\fP\&. T} _ T{ \fBlast\-failed\fP T} T{ Most recent backup that failed, in chronological order. T} .TE .SH AUTHOR EnterpriseDB .SH COPYRIGHT © Copyright EnterpriseDB UK Limited 2011-2025 .\" Generated by docutils manpage writer. . barman-3.13.3/docs/barman.conf0000644000175100001660000000665515002177425014300 0ustar 00000000000000; Barman, Backup and Recovery Manager for PostgreSQL ; http://www.pgbarman.org/ - http://www.enterprisedb.com/ ; ; Main configuration file [barman] ; System user barman_user = barman ; Directory of configuration files. Place your sections in separate files with .conf extension ; For example place the 'main' server section in /etc/barman.d/main.conf configuration_files_directory = /etc/barman.d ; Main directory barman_home = /var/lib/barman ; Locks directory - default: %(barman_home)s ;barman_lock_directory = /var/run/barman ; Log location log_file = /var/log/barman/barman.log ; Log level (see https://docs.python.org/3/library/logging.html#levels) log_level = INFO ; Default compression level: possible values are None (default), bzip2, gzip, pigz, pygzip or pybzip2 ;compression = gzip ; Pre/post backup hook scripts ;pre_backup_script = env | grep ^BARMAN ;pre_backup_retry_script = env | grep ^BARMAN ;post_backup_retry_script = env | grep ^BARMAN ;post_backup_script = env | grep ^BARMAN ; Pre/post archive hook scripts ;pre_archive_script = env | grep ^BARMAN ;pre_archive_retry_script = env | grep ^BARMAN ;post_archive_retry_script = env | grep ^BARMAN ;post_archive_script = env | grep ^BARMAN ; Pre/post delete scripts ;pre_delete_script = env | grep ^BARMAN ;pre_delete_retry_script = env | grep ^BARMAN ;post_delete_retry_script = env | grep ^BARMAN ;post_delete_script = env | grep ^BARMAN ; Pre/post wal delete scripts ;pre_wal_delete_script = env | grep ^BARMAN ;pre_wal_delete_retry_script = env | grep ^BARMAN ;post_wal_delete_retry_script = env | grep ^BARMAN ;post_wal_delete_script = env | grep ^BARMAN ; Global bandwidth limit in kilobytes per second - default 0 (meaning no limit) ;bandwidth_limit = 4000 ; Number of parallel jobs for backup and recovery via rsync (default 1) ;parallel_jobs = 1 ; Immediate checkpoint for backup command - default false ;immediate_checkpoint = false ; Enable network compression for data transfers - default false ;network_compression = false ; Number of retries of data copy during base backup after an error - default 0 ;basebackup_retry_times = 0 ; Number of seconds of wait after a failed copy, before retrying - default 30 ;basebackup_retry_sleep = 30 ; Maximum execution time, in seconds, per server ; for a barman check command - default 30 ;check_timeout = 30 ; Time frame that must contain the latest backup date. ; If the latest backup is older than the time frame, barman check ; command will report an error to the user. ; If empty, the latest backup is always considered valid. ; Syntax for this option is: "i (DAYS | WEEKS | MONTHS | HOURS)" where i is an ; integer > 0 which identifies the number of days | weeks | months of ; validity of the latest backup for this check. Also known as 'smelly backup'. ;last_backup_maximum_age = ; Time frame that must contain the latest WAL file ; If the latest WAL file is older than the time frame, barman check ; command will report an error to the user. ; Syntax for this option is: "i (DAYS | WEEKS | MONTHS | HOURS)" where i is an ; integer > 0 ;last_wal_maximum_age = ; Minimum number of required backups (redundancy) ;minimum_redundancy = 1 ; Global retention policy (REDUNDANCY or RECOVERY WINDOW) ; Examples of retention policies ; Retention policy (disabled, default) ;retention_policy = ; Retention policy (based on redundancy) ;retention_policy = REDUNDANCY 2 ; Retention policy (based on recovery window) ;retention_policy = RECOVERY WINDOW OF 4 WEEKS barman-3.13.3/docs/barman.d/0000755000175100001660000000000015002177454013641 5ustar 00000000000000barman-3.13.3/docs/barman.d/ssh-server.conf-template0000644000175100001660000000301215002177425020414 0ustar 00000000000000; Barman, Backup and Recovery Manager for PostgreSQL ; https://www.pgbarman.org/ - https://www.enterprisedb.com/ ; ; Template configuration file for a server using ; SSH connections and rsync for copy. ; [ssh] ; Human readable description description = "Example of PostgreSQL Database (via SSH)" ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; SSH options (mandatory) ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ssh_command = ssh postgres@pg ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; PostgreSQL connection string (mandatory) ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; conninfo = host=pg user=barman dbname=postgres ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; Backup settings (via rsync over SSH) ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; backup_method = rsync ; Incremental backup support: possible values are None (default), link or copy ;reuse_backup = link ; Identify the standard behavior for backup operations: possible values are ; exclusive_backup, concurrent_backup (default) ; concurrent_backup is the preferred method backup_options = concurrent_backup ; Number of parallel workers to perform file copy during backup and recover ;parallel_jobs = 1 ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; Continuous WAL archiving (via 'archive_command') ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; archiver = on ;archiver_batch_size = 50 ; PATH setting for this server ;path_prefix = "/usr/pgsql-12/bin" barman-3.13.3/docs/barman.d/streaming-server.conf-template0000644000175100001660000000315415002177425021617 0ustar 00000000000000; Barman, Backup and Recovery Manager for PostgreSQL ; https://www.pgbarman.org/ - https://www.enterprisedb.com/ ; ; Template configuration file for a server using ; only streaming replication protocol ; [streaming-server] ; Human readable description description = "Example of PostgreSQL Database (Streaming-Only)" ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; PostgreSQL connection string (mandatory) ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; conninfo = host=pg user=barman dbname=postgres ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; PostgreSQL streaming connection string ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; To be used by pg_basebackup for backup and pg_receivewal for WAL streaming ; NOTE: streaming_barman is a regular user with REPLICATION privilege streaming_conninfo = host=pg user=streaming_barman ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; Backup settings (via pg_basebackup) ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; backup_method = postgres ;streaming_backup_name = barman_streaming_backup ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; WAL streaming settings (via pg_receivewal) ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; streaming_archiver = on slot_name = barman ;create_slot = auto ;streaming_archiver_name = barman_receive_wal ;streaming_archiver_batch_size = 50 ; Uncomment the following line if you are also using archive_command ; otherwise the "empty incoming directory" check will fail ;archiver = on ; PATH setting for this server ;path_prefix = "/usr/pgsql-12/bin" barman-3.13.3/docs/barman.d/passive-server.conf-template0000644000175100001660000000166615002177425021306 0ustar 00000000000000; Barman, Backup and Recovery Manager for PostgreSQL ; https://www.pgbarman.org/ - https://www.enterprisedb.com/ ; ; Template configuration file for a server using ; SSH connections and rsync for copy. ; [passive] ; Human readable description description = "Example of a Barman passive server" ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; Passive server configuration ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; Local parameter that identifies a barman server as 'passive'. ; A passive node uses as source for backups another barman server ; instead of a PostgreSQL cluster. ; If a primary ssh command is specified, barman will use it to establish a ; connection with the barman "master" server. ; Empty by default it can be also set as global value. primary_ssh_command = ssh barman@backup ; Incremental backup settings ;reuse_backup = link ; Compression: must be identical to the source ;compression = gzip barman-3.13.3/setup.py0000755000175100001660000001575115002177425012753 0ustar 00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- # # barman - Backup and Recovery Manager for PostgreSQL # # © Copyright EnterpriseDB UK Limited 2011-2025 # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . """Backup and Recovery Manager for PostgreSQL Barman (Backup and Recovery Manager) is an open-source administration tool for disaster recovery of PostgreSQL servers written in Python. It allows your organisation to perform remote backups of multiple servers in business critical environments to reduce risk and help DBAs during the recovery phase. Barman is distributed under GNU GPL 3 and maintained by EnterpriseDB. """ import sys from setuptools import find_packages, setup if sys.version_info < (3, 6): raise SystemExit("ERROR: Barman needs at least python 3.6 to work") # Depend on pytest_runner only when the tests are actually invoked needs_pytest = set(["pytest", "test"]).intersection(sys.argv) pytest_runner = ["pytest_runner"] if needs_pytest else [] setup_requires = pytest_runner install_requires = [ "psycopg2 >= 2.4.2", "python-dateutil", ] barman = {} with open("barman/version.py", "r", encoding="utf-8") as fversion: exec(fversion.read(), barman) setup( name="barman", version=barman["__version__"], author="EnterpriseDB", author_email="barman@enterprisedb.com", url="https://www.pgbarman.org/", packages=find_packages(exclude=["tests"]), data_files=[ ( "share/man/man1", [ "docs/_build/man/barman.1", "docs/_build/man/barman-archive-wal.1", "docs/_build/man/barman-backup.1", "docs/_build/man/barman-check.1", "docs/_build/man/barman-check-backup.1", "docs/_build/man/barman-cloud-backup.1", "docs/_build/man/barman-cloud-backup-delete.1", "docs/_build/man/barman-cloud-backup-keep.1", "docs/_build/man/barman-cloud-backup-list.1", "docs/_build/man/barman-cloud-backup-show.1", "docs/_build/man/barman-cloud-check-wal-archive.1", "docs/_build/man/barman-cloud-restore.1", "docs/_build/man/barman-cloud-wal-archive.1", "docs/_build/man/barman-cloud-wal-restore.1", "docs/_build/man/barman-config-switch.1", "docs/_build/man/barman-config-update.1", "docs/_build/man/barman-cron.1", "docs/_build/man/barman-delete.1", "docs/_build/man/barman-diagnose.1", "docs/_build/man/barman-generate-manifest.1", "docs/_build/man/barman-get-wal.1", "docs/_build/man/barman-keep.1", "docs/_build/man/barman-list_backups.1", "docs/_build/man/barman-list-files.1", "docs/_build/man/barman-list-servers.1", "docs/_build/man/barman-lock-directory-cleanup.1", "docs/_build/man/barman-put-wal.1", "docs/_build/man/barman-rebuild-xlogdb.1", "docs/_build/man/barman-receive-wal.1", "docs/_build/man/barman-restore.1", "docs/_build/man/barman-replication-status.1", "docs/_build/man/barman-show-backup.1", "docs/_build/man/barman-show-servers.1", "docs/_build/man/barman-status.1", "docs/_build/man/barman-switch-wal.1", "docs/_build/man/barman-switch-xlog.1", "docs/_build/man/barman-sync-backup.1", "docs/_build/man/barman-sync-info.1", "docs/_build/man/barman-sync-wals.1", "docs/_build/man/barman-verify.1", "docs/_build/man/barman-verify-backup.1", "docs/_build/man/barman-wal-restore.1", "docs/_build/man/barman-wal-archive.1", ], ), ("share/man/man5", ["docs/_build/man/barman.5"]), ], entry_points={ "console_scripts": [ "barman=barman.cli:main", "barman-cloud-backup=barman.clients.cloud_backup:main", "barman-cloud-wal-archive=barman.clients.cloud_walarchive:main", "barman-cloud-restore=barman.clients.cloud_restore:main", "barman-cloud-wal-restore=barman.clients.cloud_walrestore:main", "barman-cloud-backup-delete=barman.clients.cloud_backup_delete:main", "barman-cloud-backup-keep=barman.clients.cloud_backup_keep:main", "barman-cloud-backup-list=barman.clients.cloud_backup_list:main", "barman-cloud-backup-show=barman.clients.cloud_backup_show:main", "barman-cloud-check-wal-archive=barman.clients.cloud_check_wal_archive:main", "barman-wal-archive=barman.clients.walarchive:main", "barman-wal-restore=barman.clients.walrestore:main", ], }, license="GPL-3.0", description=__doc__.split("\n")[0], long_description="\n".join(__doc__.split("\n")[2:]), install_requires=install_requires, extras_require={ "argcomplete": ["argcomplete"], "aws-snapshots": ["boto3<=1.35.99"], "azure": ["azure-identity", "azure-storage-blob"], "azure-snapshots": ["azure-identity", "azure-mgmt-compute"], "cloud": ["boto3<=1.35.99"], "google": [ "google-cloud-storage", ], "google-snapshots": [ "grpcio", "google-cloud-compute", # requires minimum python3.7 ], "snappy": [ 'python-snappy==0.6.1; python_version<"3.7"', 'python-snappy; python_version>="3.7"', 'cramjam >= 2.7.0; python_version>="3.7"', ], "zstandard": ["zstandard"], "lz4": ["lz4"], }, platforms=["Linux", "Mac OS X"], classifiers=[ "Environment :: Console", "Development Status :: 5 - Production/Stable", "Topic :: System :: Archiving :: Backup", "Topic :: Database", "Topic :: System :: Recovery Tools", "Intended Audience :: System Administrators", "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)", "Programming Language :: Python", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", ], setup_requires=setup_requires, ) barman-3.13.3/barman.egg-info/0000755000175100001660000000000015002177454014161 5ustar 00000000000000barman-3.13.3/barman.egg-info/top_level.txt0000644000175100001660000000000715002177454016710 0ustar 00000000000000barman barman-3.13.3/barman.egg-info/SOURCES.txt0000644000175100001660000000651415002177454016053 0ustar 00000000000000AUTHORS LICENSE MANIFEST.in README.rst RELNOTES.md setup.cfg setup.py barman/__init__.py barman/annotations.py barman/backup.py barman/backup_executor.py barman/backup_manifest.py barman/cli.py barman/cloud.py barman/command_wrappers.py barman/compression.py barman/config.py barman/copy_controller.py barman/diagnose.py barman/exceptions.py barman/fs.py barman/hooks.py barman/infofile.py barman/lockfile.py barman/output.py barman/postgres.py barman/postgres_plumbing.py barman/process.py barman/recovery_executor.py barman/remote_status.py barman/retention_policies.py barman/server.py barman/utils.py barman/version.py barman/wal_archiver.py barman/xlog.py barman.egg-info/PKG-INFO barman.egg-info/SOURCES.txt barman.egg-info/dependency_links.txt barman.egg-info/entry_points.txt barman.egg-info/requires.txt barman.egg-info/top_level.txt barman/clients/__init__.py barman/clients/cloud_backup.py barman/clients/cloud_backup_delete.py barman/clients/cloud_backup_keep.py barman/clients/cloud_backup_list.py barman/clients/cloud_backup_show.py barman/clients/cloud_check_wal_archive.py barman/clients/cloud_cli.py barman/clients/cloud_compression.py barman/clients/cloud_restore.py barman/clients/cloud_walarchive.py barman/clients/cloud_walrestore.py barman/clients/walarchive.py barman/clients/walrestore.py barman/cloud_providers/__init__.py barman/cloud_providers/aws_s3.py barman/cloud_providers/azure_blob_storage.py barman/cloud_providers/google_cloud_storage.py barman/storage/__init__.py barman/storage/file_manager.py barman/storage/file_stats.py barman/storage/local_file_manager.py docs/barman.conf docs/_build/man/barman-archive-wal.1 docs/_build/man/barman-backup.1 docs/_build/man/barman-check-backup.1 docs/_build/man/barman-check.1 docs/_build/man/barman-cloud-backup-delete.1 docs/_build/man/barman-cloud-backup-keep.1 docs/_build/man/barman-cloud-backup-list.1 docs/_build/man/barman-cloud-backup-show.1 docs/_build/man/barman-cloud-backup.1 docs/_build/man/barman-cloud-check-wal-archive.1 docs/_build/man/barman-cloud-restore.1 docs/_build/man/barman-cloud-wal-archive.1 docs/_build/man/barman-cloud-wal-restore.1 docs/_build/man/barman-config-switch.1 docs/_build/man/barman-config-update.1 docs/_build/man/barman-cron.1 docs/_build/man/barman-delete.1 docs/_build/man/barman-diagnose.1 docs/_build/man/barman-generate-manifest.1 docs/_build/man/barman-get-wal.1 docs/_build/man/barman-keep.1 docs/_build/man/barman-list-files.1 docs/_build/man/barman-list-servers.1 docs/_build/man/barman-list_backups.1 docs/_build/man/barman-lock-directory-cleanup.1 docs/_build/man/barman-put-wal.1 docs/_build/man/barman-rebuild-xlogdb.1 docs/_build/man/barman-receive-wal.1 docs/_build/man/barman-replication-status.1 docs/_build/man/barman-restore.1 docs/_build/man/barman-show-backup.1 docs/_build/man/barman-show-servers.1 docs/_build/man/barman-status.1 docs/_build/man/barman-switch-wal.1 docs/_build/man/barman-switch-xlog.1 docs/_build/man/barman-sync-backup.1 docs/_build/man/barman-sync-info.1 docs/_build/man/barman-sync-wals.1 docs/_build/man/barman-verify-backup.1 docs/_build/man/barman-verify.1 docs/_build/man/barman-wal-archive.1 docs/_build/man/barman-wal-restore.1 docs/_build/man/barman.1 docs/_build/man/barman.5 docs/barman.d/passive-server.conf-template docs/barman.d/ssh-server.conf-template docs/barman.d/streaming-server.conf-template scripts/barman.bash_completionbarman-3.13.3/barman.egg-info/PKG-INFO0000644000175100001660000000306015002177454015255 0ustar 00000000000000Metadata-Version: 2.1 Name: barman Version: 3.13.3 Summary: Backup and Recovery Manager for PostgreSQL Home-page: https://www.pgbarman.org/ Author: EnterpriseDB Author-email: barman@enterprisedb.com License: GPL-3.0 Platform: Linux Platform: Mac OS X Classifier: Environment :: Console Classifier: Development Status :: 5 - Production/Stable Classifier: Topic :: System :: Archiving :: Backup Classifier: Topic :: Database Classifier: Topic :: System :: Recovery Tools Classifier: Intended Audience :: System Administrators Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+) Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3.6 Classifier: Programming Language :: Python :: 3.7 Classifier: Programming Language :: Python :: 3.8 Classifier: Programming Language :: Python :: 3.9 Classifier: Programming Language :: Python :: 3.10 Provides-Extra: argcomplete Provides-Extra: aws-snapshots Provides-Extra: azure Provides-Extra: azure-snapshots Provides-Extra: cloud Provides-Extra: google Provides-Extra: google-snapshots Provides-Extra: snappy Provides-Extra: zstandard Provides-Extra: lz4 License-File: LICENSE License-File: AUTHORS Barman (Backup and Recovery Manager) is an open-source administration tool for disaster recovery of PostgreSQL servers written in Python. It allows your organisation to perform remote backups of multiple servers in business critical environments to reduce risk and help DBAs during the recovery phase. Barman is distributed under GNU GPL 3 and maintained by EnterpriseDB. barman-3.13.3/barman.egg-info/entry_points.txt0000644000175100001660000000133115002177454017455 0ustar 00000000000000[console_scripts] barman = barman.cli:main barman-cloud-backup = barman.clients.cloud_backup:main barman-cloud-backup-delete = barman.clients.cloud_backup_delete:main barman-cloud-backup-keep = barman.clients.cloud_backup_keep:main barman-cloud-backup-list = barman.clients.cloud_backup_list:main barman-cloud-backup-show = barman.clients.cloud_backup_show:main barman-cloud-check-wal-archive = barman.clients.cloud_check_wal_archive:main barman-cloud-restore = barman.clients.cloud_restore:main barman-cloud-wal-archive = barman.clients.cloud_walarchive:main barman-cloud-wal-restore = barman.clients.cloud_walrestore:main barman-wal-archive = barman.clients.walarchive:main barman-wal-restore = barman.clients.walrestore:main barman-3.13.3/barman.egg-info/dependency_links.txt0000644000175100001660000000000115002177454020227 0ustar 00000000000000 barman-3.13.3/barman.egg-info/requires.txt0000644000175100001660000000070315002177454016561 0ustar 00000000000000psycopg2>=2.4.2 python-dateutil [argcomplete] argcomplete [aws-snapshots] boto3<=1.35.99 [azure] azure-identity azure-storage-blob [azure-snapshots] azure-identity azure-mgmt-compute [cloud] boto3<=1.35.99 [google] google-cloud-storage [google-snapshots] grpcio google-cloud-compute [lz4] lz4 [snappy] [snappy:python_version < "3.7"] python-snappy==0.6.1 [snappy:python_version >= "3.7"] python-snappy cramjam>=2.7.0 [zstandard] zstandard barman-3.13.3/RELNOTES.md0000644000175100001660000027014115002177425013007 0ustar 00000000000000# Barman release notes © Copyright EnterpriseDB UK Limited 2025 - All rights reserved. ## 3.13.3 (2025-04-24) ### Bugfixes - Fix local restore of block-level incremental backups When performing a local restore of block-level incremental backups, Barman was facing errors like the following: ```text ERROR: Destination directory '/home/vagrant/restore/internal_no_get_wal' must be empty ``` That error was caused by a regression when the option `--staging-wals-directory` was introduced in version 3.13.0. Along with it came a new check to ensure the WAL destination directory was empty before proceeding. However, when restoring block-level incremental backups locally, Barman was setting up the WAL destination directory before performing this check, triggering the error above. References: BAR-655. - Fix regression when running `barman-cloud-backup` as a hook Barman 3.13.2 changed the location of the `backup.info` metadata file as part of the work delivered to fix issues in WORM environments. However, those changes introduced a regression when using `barman-cloud-backup` as a backup hook in the Barman server: the hook was not aware of the new location of the metadata file. This update fixes that issue, so `barman-cloud-backup` becomes aware of the new folder structure, and properly locates the `backup.info` file, avoiding runtime failures. References: BAR-696. - Avoid decompressing partial WAL files when custom compression is configured Fixed an issue where Barman attempted to decompress partial WAL files when custom compression was configured. Partial WAL files are never compressed, so any attempt to decompress them is incorrect and caused errors when using the `--partial` flag with `barman-wal-restore` or `barman get-wal`. References: BAR-697. - Fixed `barman-cloud-backup` not recycling temporary part files This fixes a `barman-cloud-backup` problem where temporary part files were not deleted after being uploaded to the cloud, leading to disk space exhaustion. The issue happened only when using Python >= 3.12 and it was due to a change in Python that removed the `delete` attribute of named-temporary file objects, which Barman used to rely on when performing internal checks. References: BAR-674. - Fixed backup annotations usage in WORM environments Barman previously stored backup annotation files, used to track operations like `barman keep` and `barman delete`, inside the backup directory itself. These annotations help determine whether a backup should be kept or marked for deletion. However, in WORM environments, files in the backup directory cannot be modified or deleted after a certain period, which caused issues with managing backup states. This fix relocates annotation files to a dedicated metadata directory, as to ensure that such operations function correctly in WORM environments. References: BAR-663. ## 3.13.2 (2025-03-27) ### Minor changes - Fix errors when using an immutable storage Added a new `worm_mode` configuration to enable WORM (Write Once Read Many) handling in Barman, allowing it to support backups on immutable storage. This fix also provides automatic relocation of the backup.info file in a new directory `meta` inside `backup_directory`. This will let Barman update it in future when needed. Barman will also _not_ purge the wals directory for WAL files that are not needed when running the first backup. This will add some extra space which will be reclaimed when this first backup is obsolete and removed (by that time, the backups and the WALs will be outside the retention policy window). Added additional notes to the documentation explaining limitations when running with an immutable storage for backups. In particular the need for a grace period in the immutability of files and the fact that `barman keep` is not supported in these environments. References: BAR-649, BAR-645, BAR-650, BAR-651, BAR-652. ## 3.13.1 (2025-03-20) ### Minor changes - Improve behavior of the backup shortcuts `last-full` / `latest-full` The shortcuts `last-full` / `latest-full` were retrieving not the last full backup of the server, but the last full backup of the server which was eligible as the parent for an incremental backup. While this was the expected behavior, the feedback from the community has shown that it was confusing for the users. From now on, the shortcuts `last-full` / `latest-full` will retrieve the last full backup of the Barman server, independently if that backup is eligible as the parent for an incremental backup or not. The eligibility of the full backup as the parent of an incremental backup will still be validated by Barman in a later step, and a proper message will be displayed in case it doesn't suit as a parent. References: BAR-555. ### Bugfixes - Fix error message when parsing invalid `--target-time` in `barman restore` When using the `barman restore` command, the error message when parsing invalid `--target-time` string was: ```text EXCEPTION: local variable 'parsed_target' referenced before assignment ``` That exception was replaced with an understandable error message. References: BAR-627. - Fix mutual exclusive arguments in the cloud restore command In the `barman-cloud-restore` command, we were checking that `target_tli` and `target_lsn` were mutually exclusive arguments, where the correct pair to check would be `target_time` and `target_lsn`. References: BAR-624. - Fix Barman not honoring `custom_decompression_filter` Fixed an issue where Barman was not honoring the configured `custom_decompression_filter` if the compression algorithm specified was natively supported by Barman. Custom filters now take priority over native handlers when decompressing WAL files. References: BAR-584. - Fix barman restore with --no-get-wal and --standby Fixed an issue where Barman was removing the `pg_wal` directory during recovery if `--no-get-wal` and `--standby-mode` were specified together. The issue happened due to Barman incorrectly filling the recovery parameters referencing `pg_wal`, including `recovery_end_command`, which led to this issue. Barman will now ignore filling such parameters as they are not required for this specific case. References: BAR-630. - Fix argument parsing issue in `barman restore` and `barman-cloud-restore` In Barman 3.13.0, a regression was introduced causing errors when using `barman restore` and `barman-cloud-restore` commands. Specifically, the `backup_id` positional argument, which was made optional in that version, conflicted with other arguments, causing unrecognized arguments and errors. For example, running `barman-cloud-restore` like this: ```text barman-cloud-restore source_url server_name backup_id --cloud-provider aws-s3 recovery_dir ``` Would trigger an error like this: ```text barman-cloud-restore: error: unrecognized arguments: recovery_dir ``` This fix resolves the issue by making `backup_id` a required argument again. Additionally, a new "auto" value is now accepted as a `backup_id`, allowing Barman to automatically choose the best backup for restoration without needing a specific `backup_id`. This update fixes argument handling and still allows a smooth and flexible restoration process for the user. References: BAR-596. ## 3.13.0 (2025-02-20) ### Notable changes - Add new xlogdb_directory configuration Introduces a new `xlogdb_directory` configuration option. This parameter can be set either globally or per-server, and allows you to specify a custom directory for the `xlog.db` file. This file stores metadata of archived WAL files and is used internally by Barman in various scenarios. If unset, it defaults to the value of `wals_directory`. Additionally, the file was also renamed to contain the server name as a prefix. References: BAR-483. - Make "backup_id" optional when restoring a backup Historically, Barman always required a "backup_id" to restore a backup, and would use that backup as the source for the restore. This feature removes the need for specifying which backup to use as a source for restore, making it optional. This change applies to both Barman and the barman-cloud scripts. Now the user is able to restore a backup in the following ways: 1. Provide a "backup_id" 2. Do not provide a "backup_id". It will retrieve the most recent backup 3. Do not provide a "backup_id", but provide a recovery target, such as: - "target_time" (mutually exclusive with target_lsn) Will get the closest backup prior to the "target_time" - "target_lsn" (mutually exclusive with "target_time") Will get the closest backup prior to the "target_lsn" - "target_tli" (can be used combined with "target_time" or "target_lsn") Will get the most recent backup that matches the timeline. If combined with other recovery targets, it will get the most recent backup prior to the target_time or target_lsn that matches the timeline The recovery targets `--target-xid`, `--target-name` and `--target-immediate` are not supported, and will error out with a message if used. This feature will provide flexibility and ease when restoring a postgres cluster. References: BAR-541, BAR-473. ### Minor changes - Add current active model to `barman show-server` and `barman status` Previously, after applying a configuration model, the only way to check which model is currently active for a server was via the `barman diagnose` command. With this update, the `barman status` and `barman show-server` commands now also display the current active configuration model for a server, if any. References: BAR-524, BAR-400. - Add `--staging-wal-directory` option to `barman restore` command to allow alternative WAL directory on PITR A new command line option `--staging-wal-directory` was added to the `restore`/`recover` command to allow an alternative destination directory for WAL files when performing PITR. Previously, WAL files were copied to a `barman_wal` directory within the restore destination directory. This enhancement provides greater flexibility, such as storing WALs on separate partitions during recovery. References: BAR-224. - Pin boto3 version to any version <= 1.35.99 Boto3 version 1.36 has changed the way S3 integrity is checked making this version incompatible with the current Barman code, generating the following error: An error occurred (MissingContentLength) when calling the PutObject operation As a temporary workaround, the version for boto3 is pinned to any version <= 1.35.99 until support for 1.36 is implemented in Barman. References: BAR-535. - Make barman-wal-archive smarter when dealing with duplicate WAL files Under some corner cases, Postgres could attempt to archive the same WAL twice. For example: if `barman-wal-archive` copies the WAL file over to the Barman host, but the script is interrupted before reporting success to Postgres. New executions of `barman-wal-archive` could fail when trying to archive the same file again because the WAL was already copied from Postgres to Barman, but not yet processed by the asynchronous Barman WAL archiver. This minor change deals with this situation by verifying the checksum of the existing and the incoming file. If the checksums match the incoming file is ignored, otherwise an output info message is sent and the incoming file is moved to the errors directory. The code will exit with 0 in both situations, avoiding WALs piling up in the Postgres host due to a failing `archive_command`. References: BAR-225. - Document procedure to clear WAL archive failure check While redesigning the Barman docs we missed adding a note advising users to run a `switch-wal` command if the server is idle and `barman check` returns a failure on "WAL archiving". This addresses the gap left from the previous documentation. References: BAR-521. - Delete WALs by deleting the entire directory at once, when possible Previously, when WAL files needed to be deleted (e.g., due to deletion of a backup), Barman would iterate over every WAL file and delete them individually. This could cause performance issues, mainly in systems which use ZFS filesystem. With this change, the entire directory will be deleted whenever noticed that all files in the directory are no longer needed by Barman. References: BAR-511. - Add support for `DefaultAzureCredential` option on Azure authentication Users can now explicitly use Azure's `DefaultAzureCredential` for authentication by using the `default` option for `azure_credential` in the server configuration or the `--azure-credential default` option in the case of `barman-cloud-*`. Previously, that could only be set as a fallback when no credential was provided and no environment variables were set. References: BAR-539. - Improve diagnose output for retention policy info Improves the output of the barman diagnose command to display a more user-friendly string representations. Specifically, "REDUNDANCY 2" is shown instead of "redundancy 2 b" for the 'retention_policy' attribute, and "MAIN" is shown instead of "simple-wal 2 b" for the 'wal_retention_policy' attribute. References: BAR-100. ### Bugfixes - Fix PITR when using `barman restore` with `--target-tli` Barman was not creating the `recovery.signal` nor filling `recovery_target_timeline` in `postgresql.auto.conf` in these cases: - The only recovery target passed to `barman restore` was `--target-tli`; or - `--target-tli` was specified with some other `--target-*` option, but the specified target timeline was the same as the timeline of the chosen backup. Now, if any `--target-*` option is passed to `barman restore`, that will be correctly treated as PITR. References: BAR-543. - Fix bug when AWS 'profile' variable is referenced before assignment An issue was introduced by BAR-242 as part of the Barman 3.12.0 release. The issue was causing `barman-cloud-backup-delete` (and possibly other commands) to fail with errors like this when `--aws-profile` argument or `aws_profile` configuration were not set: ```bash ERROR: Barman cloud backup delete exception: local variable 'profile' referenced before assignment` ``` References: BAR-518. - Fix --zstd flag on barman-cloud-wal-archive Fixed a bug with the `--zstd` flag on `barman-cloud-wal-archive` where it was essentially being ignored and not really compressing the WAL file before upload. References: BAR-567. ## 3.12.1 (2024-12-09) ### Bugfixes - Add isoformat fields for backup start and end times in json output This patch modifies the json output of the infofile object adding two new fields: `begin_time_iso` and `end_time_iso`. The new fields allow the use of a more standard and timezone aware time format, preserving compatibility with previous versions. It is worth noting that in the future the iso format for dates will be the standard used by barman for storing dates and will be used everywhere non human readable output is requested. As part of the work, this patch reverts BAR-316, which was introduced on Barman 3.12.0. References: BAR-494. ## 3.12.0 (2024-11-21) ### Minor changes - Add FIPS support to Barman The `md5` hash algorithm is not FIPS compliant, so it is going to be replaced by `sha256`. `sha256` is FIPS compliant, vastly used, and is considered secure for most practical purposes. Up until this release, Barman's WAL archive client used `hashlib.md5` to generate checksums for tar files before they were sent to the Barman server. Here, a tar file is a file format used for bundling multiple files together with a `MD5SUMS` file that lists the checksums and their corresponding paths. In this release, the `md5` hashing algorithm is replaced by `sha256` as the default. As a result, checksums for the tar files will be calculated using `sha256`, and the `MD5SUMS` file will be named `SHA256SUMS`. Barman still has the ability to use the nondefault `md5` algorithm and the `MD5SUMS` file from the client if there is a use case for it. The user just needs to add the `--md5` flag to the `barman-wal-archive` `archive_command`. References: BAR-155, CP-34954, CP-34391. - Removed el7, debian10, and ubuntu1804 support; updated Debian and SLES. Support for el7, debian10, and ubuntu1804 has been removed. Additionally, version 12 and version name "bookworm" has been added for Debian, addressing a previously missing entry. The SLES image version has also been updated from sp4 to sp5. References: BAR-389. - Add support for Postgres Extended 17 (PGE) and Postgres Advanced Server 17 (EPAS) Tests were conducted on Postgres Extended 17 (PGE) and Postgres Advanced Server 17 (EPAS), confirming full compatibility with the latest features in Barman. This validation ensures that users of the latest version of PGE and EPAS can leverage all the new capabilities of Barman with confidence. References: BAR-331. - Improve WAL compression with `zstd`, `lz4` and `xz` algorithms Introduced support for xz compression on WAL files. It can be enabled by specifying `xz` in the `compression` server parameter. WALs will be compressed when entering the Barman's WAL archive. For the cloud, it can be enabled by specifying `--xz` when running `barman-cloud-wal-archive`. Introduced support for zstandard compression on WAL files. It can be enabled by specifying `zstd` in the `compression` server parameter. WALs will be compressed when entering the Barman's WAL archive. For the cloud, it can be enabled by specifying `--zstd` when running `barman-cloud-wal-archive`. Introduced support for lz4 compression on WAL files. It can be enabled by specifying `lz4` in the `compression` server parameter. WALs will be compressed when entering the Barman's WAL archive. For the cloud, it can be enabled by specifying `--lz4` when running `barman-cloud-wal-archive`. References: BAR-265, BAR-423, BAR-264. - Improve WAL upload performance on S3 buckets by avoiding multipart uploads Previously, WAL files were being uploaded to S3 buckets using multipart uploads provided by the boto3 library via the `upload_fileobj` method. It was noticed that multipart upload is slower when used for small files, such as WAL segments, compared to when uploading it in a single PUT request. This has been improved by avoiding multipart uploads for files smaller than 100MB. The average upload time of each WAL file is expected to be reduced by around 15% with this change. References: BAR-374. - Modify behavior when enforcing retention policy for `KEEP:STANDALONE` full backups When enforcing the retention policy on full backups created with `backup_method = postgres`, Barman was previously marking all dependent (child) incremental backups as `VALID`, regardless of the KEEP target used. However, this approach is incorrect: - For backups labeled `KEEP:STANDALONE`, Barman only retains the WAL files needed to restore the server to the exact state of that backup. Because these backups are self-contained, any dependent child backups are no longer needed once the root backup is outside the retention policy. - In contrast, backups marked `KEEP:FULL` are intended for point-in-time recovery. To support this, Barman retains all WALs, as well as any child backups, to ensure the backup's consistency and allow recovery to the latest possible point. This distinction ensures that `KEEP:STANDALONE` backups serve as snapshots of a specific moment, while `KEEP:FULL` backups retain everything needed for full point-in-time recovery. References: BAR-366. - Update documentation and user-facing features for Barman's recovery process. Barman docs and the tool itself used to use the terms "recover"/"recovery" both for referencing: - The Postgres recovery process; - The process of restoring a backup and preparing it for recovery. Both the code and documentation have been revised to accurately reflect the usage of the terms "restore" and "recover"/"recovery". Also, the `barman recover` command was renamed to `barman restore`. The old name is still kept as an alias for backward compatibility. References: BAR-337. - Add --keep-compression flag to barman-wal-restore and get-wal A new `--keep-compression` option has been added to both `barman-wal-restore` and `get-wal`. This option controls whether compressed WAL files should be decompressed on the Barman server before being fetched. When specified with `get-wal`, default decompression is skipped, and the output is the WAL file content in its original state. When specified with `barman-wal-restore`, the WAL file is fetched as-is and, if compressed, decompressed on the client side. References: BAR-435. - Ransomware protection - Add AWS Snapshot Lock Support Barman now supports AWS EBS Snapshot Lock, a new integrated feature to prevent accidental or malicious deletions of Amazon EBS snapshots. When a snapshot is locked, it can't be deleted by any user but remains fully accessible for use. This feature enables you to store snapshots in WORM (Write-Once-Read-Many) format for a specified duration, helping to meet regulatory requirements by keeping the data secure and tamper-proof until the lock expires. Special thanks to Rui Marinho, our community contributor who started this feature. References: BAR-242. - Prevent orphan files from being left from a crash while deleting a backup This commit fixes an issue where backups could leave behind files if the system crashed during the deletion of a backup. Now, when a backup is deleted, it will get a "delete marker" at the start. If a crash happens while the backup is being deleted, the marker will help recognize incomplete backup removals when the server restarts. The Barman cron job has been updated to look for these deleted markers. If it finds a backup with a "delete marker", it will complete the process. References: BAR-244. - Add support for using tags with snapshots Barman now supports tagging the snapshots when creating backups using the barman-cloud-backup script command. A new argument called --tags was added. Special thanks to Rui Marinho, our community contributor who started this feature. References: BAR-417. - Use ISO format instead of ctime when producing JSON output of Barman cloud commands The ctime format has no information about the time zone associated with the timestamp. Besides that, that format is better suited for human consumption. For machine consumption the ISO format is better suited. References: BAR-316. ### Bugfixes - Fix barman check which returns wrong results for Replication Slot Previously, when using architectures which backup from a standby node and stream WALs from the primary, Barman would incorrectly use `conninfo` (pointing to a standby server) for replication checks, leading to errors such as: `replication slot (WAL streaming): FAILED (replication slot 'barman' doesn't exist. Please execute 'barman receive-wal --create-slot pg17')` This fixes the following issue [#1024](https://github.com/EnterpriseDB/barman/issues/1024) by ensuring `wal_conninfo` is used for WAL replication checks if it's set. `wal_conninfo` takes precedence over `wal_streaming_conninfo`, when both are set. With this change, if only `wal_conninfo` is set, it will be used and will not fall back to `conninfo`. Also, in the documentation, changes were made so it is explicit that when `conninfo` points to a standby server, `wal_conninfo` must be set and used for accurate replication status checks. References: BAR-409. - Fix missing options for `barman keep` The error message that the Barman CLI emitted when running `barman keep` without any options suggested there were shortcut aliases for status and release. These aliases, -s and -r, do not exist, so the error message was misleading. This fixes the issue by including these short options in the Barman CLI, aligning it with other tools like `barman-cloud-backup-keep`, where these shortcuts already exist. References: BAR-356. - Lighten standby checks related to conninfo and primary_conninfo When backing up a standby server, Barman performs some checks to assert that `conninfo` is really pointing to a standby (in recovery mode) and that `primary_conninfo` is pointing to a primary (not in recovery). The problem, as reported in the issues #704 and #744, is that when a failover occurs, the `conninfo` will now be pointing to a primary instead and the checks will start failing, requiring the user to change Barman configs manually whenever a failover occurs. This fix solved the issue by making such checks non-critical, which means they will still fail but Barman will keep operating regardless. Essentially, Barman will ignore `primary_conninfo` if `conninfo` does not point to a standby. Warnings about this misconfiguration will also be emitted whenever running any Barman command so the user can be aware. References: BAR-348. - Check for USAGE instead of MEMBER when calling pg_has_role in Barman To work correctly Barman database user needs to be included in some roles. Barman was verifying the conditions was satisfied by calling `pg_has_role` in Postgres. However, it was check for the `MEMBER` privilege instead of `USAGE`. This oversight was fixed. This change is a contribution from @RealGreenDragon. References: BAR-489. ## 3.11.1 (2024-08-22) ### Bugfixes - Fix failures in `barman-cloud-backup-delete`. This command was failing when applying retention policies due to a bug introduced by the previous release. ## 3.11.0 (2024-08-22) ### Notable changes - Add support for Postgres 17+ incremental backups. This major feature is composed of several small changes: - Add `--incremental` command-line option to `barman backup` command. This is used to specify the parent backup when taking an incremental backup. The parent can be either a full backup or another incremental backup. - Add `latest-full` shortcut backup ID. Along with `latest`, this can be used as a shortcut to select the parent backup for an incremental backup. While `latest` takes the latest backup independently if it is full or incremental, `latest-full` takes the latest full backup. - `barman keep` command can only be applied to full backups when `backup_method = postgres`. - Retention policies do not take incremental backups into consideration. As incremental backups cannot be recovered without having the complete chain of backups available up to the full backup, only full backups account for retention policies. If a full backup has dependent incremental backups and the retention policy is applied, the full backup will propagate its status to the associated incremental backups. When the full backup is flagged with any `KEEP` target, Barman will set the status of all related incremental backups to `VALID`. - When deleting a backup all the incremental backups depending on it, if any, are also removed. - `barman recover` needs to combine the full backup with the chain of incremental backups when recovering. The new CLI option `--local-staging-path`, and the corresponding `local_staging_path` configuration option, are used to specify the path in the Barman host where the backups will be combined when recovering an incremental backup. - Changes to `barman show-backup` output: - Add the “Estimated cluster size” field. It's useful to have an estimation of the data directory size of a cluster when restoring a backup. It’s particularly useful when recovering compressed backups or incremental backups, situations where the size of the backup doesn’t reflect the size of the data directory in Postgres. In JSON format, this is stored as `cluster_size`. - Add the “WAL summarizer” field. This field shows if `summarize_wal` was enabled in Postgres at the time the backup was taken. In JSON format, this is stored as `server_information.summarize_wal`. This field is omitted for Postgres 16 and older. - Add “Data checksums” field. This shows if `data_checkums` was enabled in Postgres at the time the backup was taken. In JSON format, this is stored as `server_information.data_checksums`. - Add the “Backup method” field. This shows the backup method used for this backup. In JSON format, this is stored as `base_backup_information.backup_method`. - Rename the field “Disk Usage” as “Backup Size”. The latter provides a more comprehensive name which represents the size of the backup in the Barman host. The JSON field under `base_backup_information` was also renamed from `disk_usage` to `backup_size`. - Add the “WAL size” field. This shows the size of the WALs required by the backup. In JSON format, this is stored as `base_backup_information.wal_size`. - Refactor the field “Incremental size”. It is now named “Resources saving” and it now shows an estimation of resources saved when taking incremental backups with `rsync` or `pg_basebackup`. It compares the backup size with the estimated cluster size to estimate the amount of disk and network resources that were saved by taking an incremental backup. In JSON format, the field was renamed from `incremental_size` to `resource_savings` under `base_backup_information`. - Add the `system_id` field to the JSON document. This field contains the system identifier of Postgres. It was present in console format, but was missing in JSON format. - Add fields related with Postgres incremental backups: - “Backup type”: indicates if the Postgres backup is full or incremental. In JSON format, this is stored as `backup_type` under `base_backup_information`. - “Root backup”: the ID of the full backup that is the root of a chain of one or more incremental backups. In JSON format, this is stored as `catalog_information.root_backup_id`. - “Parent backup”: the ID of the full or incremental backup from which this incremental backup was taken. In JSON format, this is stored as `catalog_information.parent_backup_id`. - “Children Backup(s)”: the IDs of the incremental backups that were taken with this backup as the parent. In JSON format, this is stored as `catalog_information.children_backup_ids`. - “Backup chain size”: the number of backups in the chain from this incremental backup up to the root backup. In JSON format, this is stored as `catalog_information.chain_size`. - Changes to `barman list-backup` output: - It now includes the backup type in the JSON output, which can be either `rsync` for backups taken with rsync, `full` or `incremental` for backups taken with `pg_basebackup`, or `snapshot` for cloud snapshots. When printing to the console the backup type is represented by the corresponding labels `R`, `F`, `I` or `S`. - Remove tablespaces information from the output. That was bloating the output. Tablespaces information can still be found in the output of `barman show-backup`. - Always set a timestamp with a time zone when configuring `recovery_target_time` through `barman recover`. Previously, if no time zone was explicitly set through `--target-time`, Barman would configure `recovery_target_time` without a time zone in Postgres. Without a time zone, Postgres would assume whatever is configured through `timezone` GUC in Postgres. From now on Barman will issue a warning and configure `recovery_target_time` with the time zone of the Barman host if no time zone is set by the user through `--target-time` option. - When recovering a backup with the “no get wal” approach and `--target-lsn` is set, copy only the WAL files required to reach the configured target. Previously Barman would copy all the WAL files from its archive to Postgres. - When recovering a backup with the “no get wal” approach and `--target-immediate` is set, copy only the WAL files required to reach the consistent point. Previously Barman would copy all the WAL files from its archive to Postgres. - `barman-wal-restore` now moves WALs from the spool directory to `pg_wal` instead of copying them. This can improve performance if the spool directory and the `pg_wal` directory are in the same partition. - `barman check-backup` now shows the reason why a backup was marked as `FAILED` in the output and logs. Previously for a user to know why the backup was marked as `FAILED`, they would need to run `barman show-backup` command. - Add configuration option `aws_await_snapshots_timeout` and the corresponding `--aws-await-snapshots-timeout` command-line option on `barman-cloud-backup`. This specifies the timeout in seconds to wait for snapshot backups to reach the completed state. - Add a keep-alive mechanism to rsync-based backups. Previously the Postgres session created by Barman to run `pg_backup_start()` and `pg_backup_stop()` would stay idle for as long as the base backup copy would take. That could lead to a firewall or router dropping the connection because it was idle for a long time. The keep-alive mechanism sends heartbeat queries to Postgres through that connection, thus reducing the likelihood of a connection getting dropped. The interval between heartbeats can be controlled through the new configuration option `keepalive_interval` and the corresponding CLI option `--keepalive-interval` of the `barman backup` command. ### Bugfixes - When recovering a backup with the “no get wal” approach and `--target-time` set, copy all WAL files. Previously Barman would attempt to “guess” the WAL files required by Postgres to reach the configured target time. However, the mechanism was not robust enough as it was based on the stats of the WAL file in the Barman host (more specifically the creation time). For example: if there were archiving or streaming lag between Postgres and Barman, that could be enough for recovery to fail because Barman would miss to copy all the required WAL files due to the weak check based on file stats. - Pin `python-snappy` to `0.6.1` when running Barman through Python 3.6 or older. Newer versions of `python-snappy` require `cramjam` version `2.7.0` or newer, and these are only available for Python 3.7 or newer. - `barman receive-wal` now exits with code `1` instead of `0` in the following cases: - Being unable to run with `--reset` flag because `pg_receivewal` is running. - Being unable to start `pg_receivewal` process because it is already running. - Fix and improve information about Python in `barman diagnose` output: - The command now makes sure to use the same Python interpreter under which Barman is installed when outputting the Python version through `python_ver` JSON key. Previously, if an environment had multiple Python installations and/or virtual environments, the output could eventually be misleading, as it could be fetched from a different Python interpreter. - Added a `python_executable` key to the JSON output. That contains the path to the exact Python interpreter being used by Barman. ## 3.10.1 (2024-06-12) ### Bugfixes - Make `argcomplete` optional to avoid installation issues on some platforms. - Load `barman.auto.conf` only when the file exists. - Emit a warning when the `cfg_changes.queue` file is malformed. - Correct in documentation the postgresql version where `pg_checkpoint` is available. - Add `--no-partial` option to `barman-cloud-wal-restore`. ## 3.10.0 (2024-01-24) ### Notable changes - Limit the average bandwidth used by `barman-cloud-backup` when backing up to either AWS S3 or Azure Blob Storage according to the value set by a new CLI option `--max-bandwidth`. - Add the new configuration option `lock_directory_cleanup` That enables cron to automatically clean up the barman_lock_directory from unused lock files. - Add support for a new type of configuration called `model`. The model acts as a set of overrides for configuration options for a given Barman server. - Add a new barman command `barman config-update` that allows the creation and the update of configurations using JSON ### Bugfixes - Fix a bug that caused `--min-chunk-size` to be ignored when using barman-cloud-backup as hook script in Barman. ## 3.9.0 (2023-10-03) ### Notable changes - Allow `barman switch-wal --force` to be run against PG>=14 if the user has the `pg_checkpoint` role (thanks to toydarian for this patch). - Log the current check at `info` level when a check timeout occurs. - The minimum size of an upload chunk when using `barman-cloud-backup` with either S3 or Azure Blob Storage can now be specified using the `--min-chunk-size` option. - `backup_compression = none` is supported when using `pg_basebackup`. - For PostgreSQL 15 and later: the allowed `backup_compression_level` values for `zstd` and `lz4` have been updated to match those allowed by `pg_basebackup`. - For PostgreSQL versions earlier than 15: `backup_compression_level = 0` can now be used with `backup_compression = gzip`. ### Bugfixes - Fix `barman recover` on platforms where Multiprocessing uses spawn by default when starting new processes. ## 3.8.0 (2023-08-31) ### Notable changes - Clarify package installation. barman is packaged with default python version for each operating system. - The `minimum-redundancy` option is added to `barman-cloud-backup-delete`. It allows to set the minimum number of backups that should always be available. - Add a new `primary_checkpoint_timeout` configuration option. Allows define the amount of seconds that Barman will wait at the end of a backup if no new WAL files are produced, before forcing a checkpoint on the primary server. ### Bugfixes - Fix race condition in barman retention policies application. Backup deletions will now raise a warning if another deletion is in progress for the requested backup. - Fix `barman-cloud-backup-show` man page installation. ## 3.7.0 (2023-07-25) ### Notable changes - Support is added for snapshot backups on AWS using EBS volumes. - The `--profile` option in the `barman-cloud-*` scripts is renamed `--aws-profile`. The old name is deprecated and will be removed in a future release. - Backup manifests can now be generated automatically on completion of a backup made with `backup_method = rsync`. This is enabled by setting the `autogenerate_manifest` configuration variable and can be overridden using the `--manifest` and `--no-manifest` CLI options. ### Bugfixes - The `barman-cloud-*` scripts now correctly use continuation tokens to page through objects in AWS S3-compatible object stores. This fixes a bug where `barman-cloud-backup-delete` would only delete the oldest 1000 eligible WALs after backup deletion. - Minor documentation fixes. ## 3.6.0 (2023-06-15) ### Notable changes - PostgreSQL version 10 is no longer supported. - Support is added for snapshot backups on Microsoft Azure using Managed Disks. - The `--snapshot-recovery-zone` option is renamed `--gcp-zone` for consistency with other provider-specific options. The old name is deprecated and will be removed in a future release. - The `snapshot_zone` option and `--snapshot-zone` argument are renamed `gcp_zone` and `--gcp-zone` respectively. The old names are deprecated and will be removed in a future release. - The `snapshot_gcp_project` option and `--snapshot-gcp-project` argument are renamed to `gcp_project` and `--gcp-project`. The old names are deprecated and will be removed in a future release. ### Bugfixes - Barman will no longer attempt to execute the `replication-status` command for a passive node. - The `backup_label` is deleted from cloud storage when a snapshot backup is deleted with `barman-cloud-backup-delete`. - Man pages for the `generate-manifest` and `verify-backup` commands are added. - Minor documentation fixes. ## 3.5.0 (2023-03-29) ### Notable changes - Python 2.7 is no longer supported. The earliest Python version supported is now 3.6. - The `barman`, `barman-cli` and `barman-cli-cloud` packages for EL7 now require python 3.6 instead of python 2.7. For other supported platforms, Barman packages already require python versions 3.6 or later so packaging is unaffected. - Support for PostgreSQL 10 will be discontinued in future Barman releases; 3.5.x is the last version of Barman with support for PostgreSQL 10. - Backups and WALs uploaded to Google Cloud Storage can now be encrypted using a specific KMS key by using the `--kms-key-name` option with `barman-cloud-backup` or `barman-cloud-wal-archive`. - Backups and WALs uploaded to AWS S3 can now be encrypted using a specific KMS key by using the `--sse-kms-key-id` option with `barman-cloud-backup` or `barman-cloud-wal-archive` along with `--encryption=aws:kms`. - Two new configuration options are provided which make it possible to limit the rate at which parallel workers are started during backups with `backup_method = rsync` and recoveries. `parallel_jobs_start_batch_size` can be set to limit the amount of parallel workers which will be started in a single batch, and `parallel_jobs_start_batch_period` can be set to define the time in seconds over which a single batch of workers will be started. These can be overridden using the arguments `--jobs-start-batch-size` and `--jobs-start-batch-period` with the `barman backup` and `barman recover` commands. - A new option `--recovery-conf-filename` is added to `barman recover`. This can be used to change the file to which Barman should write the PostgreSQL recovery options from the default `postgresql.auto.conf` to an alternative location. ### Bugfixes - Fix a bug which prevented `barman-cloud-backup-show` from displaying the backup metadata for backups made with `barman backup` and uploaded by `barman-cloud-backup` as a post-backup hook script. - Fix a bug where the PostgreSQL connection used to validate backup compression settings was left open until termination of the Barman command. - Fix an issue which caused rsync-concurrent backups to fail when running for a duration greater than `idle_session_timeout`. - Fix a bug where the backup name was not saved in the backup metadata if the `--wait` flag was used with `barman backup`. - Thanks to mojtabash78, mhkarimi1383, epolkerman, barthisrael and hzetters for their contributions. ## 3.4.0 (2023-01-26) ### Notable changes - This is the last release of Barman which will support Python 2 and new features will henceforth require Python 3.6 or later. - A new `backup_method` named `snapshot` is added. This will create backups by taking snapshots of cloud storage volumes. Currently only Google Cloud Platform is supported however support for AWS and Azure will follow in future Barman releases. Note that this feature requires a minimum Python version of 3.7. Please see the Barman manual for more information. - Support for snapshot backups is also added to `barman-cloud-backup`, with minimal support for restoring a snapshot backup added to `barman-cloud-restore`. - A new command `barman-cloud-backup-show` is added which displays backup metadata stored in cloud object storage and is analogous to `barman show-backup`. This is provided so that snapshot metadata can be easily retrieved at restore time however it is also a convenient way of inspecting metadata for any backup made with `barman-cloud-backup`. - The instructions for installing Barman from RPMs in the docs are updated. - The formatting of NFS requirements in the docs is fixed. - Supported PostgreSQL versions are updated in the docs (this is a documentation fix only - the minimum supported major version is still 10). ## 3.3.0 (2022-12-14) ### Notable changes - A backup can now be given a name at backup time using the new `--name` option supported by the `barman backup` and `barman-cloud-backup` commands. The backup name can then be used in place of the backup ID when running commands to interact with backups. Additionally, the commands to list and show backups have been been updated to include the backup name in the plain text and JSON output formats. - Stricter checking of PostgreSQL version to verify that Barman is running against a supported version of PostgreSQL. ### Bugfixes - Fix inconsistencies between the barman cloud command docs and the help output for those commands. - Use a new PostgreSQL connection when switching WALs on the primary during the backup of a standby to avoid undefined behaviour such as `SSL error` messages and failed connections. - Reduce log volume by changing the default log level of stdout for commands executed in child processes to `DEBUG` (with the exception of `pg_basebackup` which is deliberately logged at `INFO` level due to it being a long-running process where it is frequently useful to see the output during the execution of the command). ## 3.2.0 (2022-10-20) ### Notable changes - `barman-cloud-backup-delete` now accepts a `--batch-size` option which determines the maximum number of objects deleted in a single request. - All `barman-cloud-*` commands now accept a `--read-timeout` option which, when used with the `aws-s3` cloud provider, determines the read timeout used by the boto3 library when making requests to S3. ### Bugfixes - Fix the failure of `barman recover` in cases where `backup_compression` is set in the Barman configuration but the PostgreSQL server is unavailable. ## 3.1.0 (2022-09-14) ### Notable changes - Backups taken with `backup_method = postgres` can now be compressed using lz4 and zstd compression by setting `backup_compression = lz4` or `backup_compression = zstd` respectively. These options are only supported with PostgreSQL 15 (beta) or later. - A new option `backup_compression_workers` is available which sets the number of threads used for parallel compression. This is currently only available with `backup_method = postgres` and `backup_compression = zstd`. - A new option `primary_conninfo` can be set to avoid the need for backups of standbys to wait for a WAL switch to occur on the primary when finalizing the backup. Barman will use the connection string in `primary_conninfo` to perform WAL switches on the primary when stopping the backup. - Support for certain Rsync versions patched for CVE-2022-29154 which require a trailing newline in the `--files-from` argument. - Allow `barman receive-wal` maintenance options (`--stop`, `--reset`, `--drop-slot` and `--create-slot`) to run against inactive servers. - Add `--port` option to `barman-wal-archive` and `barman-wal-restore` commands so that a custom SSH port can be used without requiring any SSH configuration. - Various documentation improvements. - Python 3.5 is no longer supported. ### Bugfixes - Ensure PostgreSQL connections are closed cleanly during the execution of `barman cron`. - `barman generate-manifest` now treats pre-existing backup_manifest files as an error condition. - backup_manifest files are renamed by appending the backup ID during recovery operations to prevent future backups including an old backup_manifest file. - Fix epoch timestamps in json output which were not timezone-aware. - The output of `pg_basebackup` is now written to the Barman log file while the backup is in progress. - We thank barthisrael, elhananjair, kraynopp, lucianobotti, and mxey for their contributions to this release. ## 3.0.1 (2022-06-27) ### Bugfixes - Fix package signing issue in PyPI (same sources as 3.0.0) ## 3.0.0 (2022-06-23) ### Breaking changes - PostgreSQL versions 9.6 and earlier are no longer supported. If you are using one of these versions you will need to use an earlier version of Barman. - The default backup mode for Rsync backups is now concurrent rather than exclusive. Exclusive backups have been deprecated since PostgreSQL 9.6 and have been removed in PostgreSQL 15. If you are running Barman against PostgreSQL versions earlier than 15 and want to use exclusive backups you will now need to set `exclusive_backup` in `backup_options`. - The backup metadata stored in the `backup.info` file for each backup has an extra field. This means that earlier versions of Barman will not work in the presence of any backups taken with 3.0.0. Additionally, users of pg-backup-api will need to upgrade it to version 0.2.0 so that pg-backup-api can work with the updated metadata. ### Notable changes - Backups taken with `backup_method = postgres` can now be compressed by pg_basebackup by setting the `backup_compression` config option. Additional options are provided to control the compression level, the backup format and whether the pg_basebackup client or the PostgreSQL server applies the compression. NOTE: Recovery of these backups requires Barman to stage the compressed files on the recovery server in a location specified by the `recovery_staging_path` option. - Add support for PostgreSQL 15. Exclusive backups are not supported by PostgreSQL 15 therefore Barman configurations for PostgreSQL 15 servers are not allowed to specify `exclusive_backup` in `backup_options`. - Use custom_compression_magic, if set, when identifying compressed WAL files. This allows Barman to correctly identify uncompressed WALs (such as `*.partial` files in the `streaming` directory) and return them instead of attempting to decompress them. ### Minor changes - Various documentation improvements. ### Bugfixes - Fix an ordering bug which caused Barman to log the message "Backup failed issuing start backup command." while handling a failure in the stop backup command. - Fix a bug which prevented recovery using `--target-tli` when timelines greater than 9 were present, due to hexadecimal values from WAL segment names being parsed as base 10 integers. - Fix an import error which occurs when using barman cloud with certain python2 installations due to issues with the enum34 dependency. - Fix a bug where Barman would not read more than three bytes from a compressed WAL when attempting to identify the magic bytes. This means that any custom compressed WALs using magic longer than three bytes are now decompressed correctly. - Fix a bug which caused the `--immediate-checkpoint` flag to be ignored during backups with `backup_method = rsync`. ## 2.19 (2022-03-09) ### Notable changes - Change `barman diagnose` output date format to ISO8601. - Add Google Cloud Storage (GCS) support to barman cloud. - Support `current` and `latest` recovery targets for the `--target-tli` option of `barman recover`. - Add documentation for installation on SLES. ### Bugfixes - `barman-wal-archive --test` now returns a non-zero exit code when an error occurs. - Fix `barman-cloud-check-wal-archive` behaviour when `-t` option is used so that it exits after connectivity test. - `barman recover` now continues when `--no-get-wal` is used and `"get-wal"` is not set in `recovery_options`. - Fix `barman show-servers --format=json ${server}` output for inactive server. - Check for presence of `barman_home` in configuration file. - Passive barman servers will no longer store two copies of the tablespace data when syncing backups taken with `backup_method = postgres`. - We thank richyen for his contributions to this release. ## 2.18 (2022-01-21) ### Notable changes - Add snappy compression algorithm support in barman cloud (requires the optional python-snappy dependency). - Allow Azure client concurrency parameters to be set when uploading WALs with barman-cloud-wal-archive. - Add `--tags` option in barman cloud so that backup files and archived WALs can be tagged in cloud storage (aws and azure). - Update the barman cloud exit status codes so that there is a dedicated code (2) for connectivity errors. - Add the commands `barman verify-backup` and `barman generate-manifest` to check if a backup is valid. - Add support for Azure Managed Identity auth in barman cloud which can be enabled with the `--credential` option. ### Bugfixes - Change `barman-cloud-check-wal-archive` behavior when bucket does not exist. - Ensure `list-files` output is always sorted regardless of the underlying filesystem. - Man pages for barman-cloud-backup-keep, barman-cloud-backup-delete and barman-cloud-check-wal-archive added to Python packaging. - We thank richyen and stratakis for their contributions to this release. ## 2.17 (2021-12-01) ### Notable changes - Resolves a performance regression introduced in version 2.14 which increased copy times for `barman backup` or `barman recover` commands when using the `--jobs` flag. - Ignore rsync partial transfer errors for `sender` processes so that such errors do not cause the backup to fail (thanks to barthisrael). ## 2.16 (2021-11-17) ### Notable changes - Add the commands `barman-check-wal-archive` and `barman-cloud-check-wal-archive` to validate if a proposed archive location is safe to use for a new PostgreSQL server. - Allow Barman to identify WAL that's already compressed using a custom compression scheme to avoid compressing it again. - Add `last_backup_minimum_size` and `last_wal_maximum_age` options to `barman check`. ### Bugfixes - Use argparse for command line parsing instead of the unmaintained argh module. - Make timezones consistent for `begin_time` and `end_time`. - We thank chtitux, George Hansper, stratakis, Thoro, and vrms for their contributions to this release. ## 2.15 (2021-10-12) ### Notable changes - Add plural forms for the `list-backup`, `list-server` and `show-server` commands which are now `list-backups`, `list-servers` and `show-servers`. The singular forms are retained for backward compatibility. - Add the `last-failed` backup shortcut which references the newest failed backup in the catalog so that you can do: - `barman delete last-failed` ### Bugfixes - Tablespaces will no longer be omitted from backups of EPAS versions 9.6 and 10 due to an issue detecting the correct version string on older versions of EPAS. ## 2.14 (2021-09-22) ### Notable changes - Add the `barman-cloud-backup-delete` command which allows backups in cloud storage to be deleted by specifying either a backup ID or a retention policy. - Allow backups to be retained beyond any retention policies in force by introducing the ability to tag existing backups as archival backups using `barman keep` and `barman-cloud-backup-keep`. - Allow the use of SAS authentication tokens created at the restricted blob container level (instead of the wider storage account level) for Azure blob storage - Significantly speed up `barman restore` into an empty directory for backups that contain hundreds of thousands of files. ### Bugfixes - The backup privileges check will no longer fail if the user lacks "userepl" permissions and will return better error messages if any required permissions are missing (#318 and #319). ## 2.13 (2021-07-26) ### Notable changes - Add Azure blob storage support to barman-cloud - Support tablespace remapping in barman-cloud-restore via `--tablespace name:location` - Allow barman-cloud-backup and barman-cloud-wal-archive to run as Barman hook scripts, to allow data to be relayed to cloud storage from the Barman server ### Bugfixes - Stop backups failing due to idle_in_transaction_session_timeout - Fix a race condition between backup and archive-wal in updating xlog.db entries (#328) - Handle PGDATA being a symlink in barman-cloud-backup, which led to "seeking backwards is not allowed" errors on restore (#351) - Recreate pg_wal on restore if the original was a symlink (#327) - Recreate pg_tblspc symlinks for tablespaces on restore (#343) - Make barman-cloud-backup-list skip backups it cannot read, e.g., because they are in Glacier storage (#332) - Add `-d database` option to barman-cloud-backup to specify which database to connect to initially (#307) - Fix "Backup failed uploading data" errors from barman-cloud-backup on Python 3.8 and above, caused by attempting to pickle the boto3 client (#361) - Correctly enable server-side encryption in S3 for buckets that do not have encryption enabled by default. In Barman 2.12, barman-cloud-backup's `--encryption` option did not correctly enable encryption for the contents of the backup if the backup was stored in an S3 bucket that did not have encryption enabled. If this is the case for you, please consider deleting your old backups and taking new backups with Barman 2.13. If your S3 buckets already have encryption enabled by default (which we recommend), this does not affect you. ## 2.12.1 (2021-06-30) ### Bugfixes - Allow specifying target-tli with other `target-*` recovery options. - Fix incorrect NAME in barman-cloud-backup-list manpage. - Don't raise an error if SIGALRM is ignored. - Fetch wal_keep_size, not wal_keep_segments, from Postgres 13. ## 2.12 (2020-11-05) ### Notable changes - Introduce a new backup_method option called local-rsync which targets those cases where Barman is installed on the same server where PostgreSQL is and directly uses rsync to take base backups, bypassing the SSH layer. ### Bugfixes - Avoid corrupting boto connection in worker processes. - Avoid connection attempts to PostgreSQL during tests. ## 2.11 (2020-07-09) ### Notable changes - Introduction of the barman-cli-cloud package that contains all cloud related utilities. - Add barman-cloud-wal-restore to restore a WAL file previously archived with barman-cloud-wal-archive from an object store. - Add barman-cloud-restore to restore a backup previously taken with barman-cloud-backup from an object store. - Add barman-cloud-backup-list to list backups taken with barman-cloud-backup in an object store. - Add support for arbitrary archive size for barman-cloud-backup. - Add support for --endpoint-url option to cloud utilities. - Remove strict superuser requirement for PG 10+ (by Kaarel Moppel). - Add --log-level runtime option for barman to override default log level for a specific command. - Support for PostgreSQL 13 ### Bugfixes - Suppress messages and warning with SSH connections in barman-cli (GH-257). - Fix a race condition when retrieving uploaded parts in barman-cloud-backup (GH-259). - Close the PostgreSQL connection after a backup (GH-258). - Check for uninitialized replication slots in receive-wal --reset (GH-260). - Ensure that begin_wal is valorised before acting on it (GH-262). - Fix bug in XLOG/WAL arithmetic with custom segment size (GH-287). - Fix rsync compatibility error with recent rsync. - Fix PostgreSQLClient version parsing. - Fix PostgreSQL exception handling with non ASCII messages. - Ensure each postgres connection has an empty search_path. - Avoid connecting to PostgreSQL while reading a backup.info file. If you are using already `barman-cloud-wal-archive` or `barman-cloud-backup` installed via RPM/Apt package and you are upgrading your system, you must install the barman-cli-cloud package. All cloud related tools are now part of the barman-cli-cloud package, including `barman-cloud-wal-archive` and `barman-cloud-backup` that were previously shipped with `barman-cli`. The reason is complex dependency management of the boto3 library, which is a requirement for the cloud utilities. ## 2.10 (2019-12-05) ### Notable changes - Pull .partial WAL files with get-wal and barman-wal-restore, allowing restore_command in a recovery scenario to fetch a partial WAL file's content from the Barman server. This feature simplifies and enhances RPO=0 recovery operations. - Store the PostgreSQL system identifier in the server directory and inside the backup information file. Improve check command to verify the consistency of the system identifier with active connections (standard and replication) and data on disk. - A new script called barman-cloud-wal-archive has been added to the barman-cli package to directly ship WAL files from PostgreSQL (using archive_command) to cloud object storage services that are compatible with AWS S3. It supports encryption and compression. - A new script called barman-cloud-backup has been added to the barman-cli package to directly ship base backups from a local PostgreSQL server to cloud object storage services that are compatible with AWS S3. It supports encryption, parallel upload, compression. - Automated creation of replication slots through the server/global option create_slot. When set to auto, Barman creates the replication slot, in case streaming_archiver is enabled and slot_name is defined. The default value is manual for back-compatibility. - Add '-w/--wait' option to backup command, making Barman wait for all required WAL files to be archived before considering the backup completed. Add also the --wait-timeout option (default 0, no timeout). - Redact passwords from Barman output, in particular from barman diagnose (InfoSec) - Improve robustness of receive-wal --reset command, by verifying that the last partial file is aligned with the current location or, if present, with replication slot's. - Documentation improvements ### Bugfixes - Wrong string matching operation when excluding tablespaces inside PGDATA (GH-245). - Minor fixes in WAL delete hook scripts (GH-240). - Fix PostgreSQL connection aliveness check (GH-239). ## 2.9 (2019-08-01) ### Notable changes - Transparently support PostgreSQL 12, by supporting the new way of managing recovery and standby settings through GUC options and signal files (recovery.signal and standby.signal) - Add --bwlimit command line option to set bandwidth limitation for backup and recover commands - Ignore WAL archive failure for check command in case the latest backup is WAITING_FOR_WALS - Add --target-lsn option to set recovery target Log Sequence Number for recover command with PostgreSQL 10 or higher - Add --spool-dir option to barman-wal-restore so that users can change the spool directory location from the default, avoiding conflicts in case of multiple PostgreSQL instances on the same server (thanks to Drazen Kacar). - Rename barman_xlog directory to barman_wal - JSON output writer to export command output as JSON objects and facilitate integration with external tools and systems (thanks to Marcin Onufry Hlybin). Experimental in this release. ### Bugfixes - `replication-status` doesn’t show streamers with no slot (GH-222) - When checking that a connection is alive (“SELECT 1” query), preserve the status of the PostgreSQL connection (GH-149). This fixes those cases of connections that were terminated due to idle-in-transaction timeout, causing concurrent backups to fail. ## 2.8 (2019-05-17) ### Notable changes - Add support for reuse_backup in geo-redundancy for incremental backup copy in passive nodes - Improve performance of rsync based copy by using strptime instead of the more generic dateutil.parser (#210) - Add ‘--test’ option to barman-wal-archive and barman-wal-restore to verify the connection with the Barman server - Complain if backup_options is not explicitly set, as the future default value will change from exclusive_backup to concurrent_backup when PostgreSQL 9.5 will be declared EOL by the PGDG - Display additional settings in the show-server and diagnose commands: archive_timeout, data_checksums, hot_standby, max_wal_senders, max_replication_slots and wal_compression. - Merge the barman-cli project in Barman ### Minor changes - Improve messaging of check --nagios for inactive servers. - Log remote SSH command with recover command. - Hide logical decoding connections in replication-status command. This release officially supports Python 3 and deprecates Python 2 (which might be discontinued in future releases). PostgreSQL 9.3 and older is deprecated from this release of Barman. Support for backup from standby is now limited to PostgreSQL 9.4 or higher and to WAL shipping from the standby (please refer to the documentation for details). ### Bugfixes - Fix encoding error in get-wal on Python 3 (Jeff Janes, #221). - Fix exclude_and_protect_filter (Jeff Janes, #217). - Remove spurious message when resetting WAL (Jeff Janes, #215). - Fix sync-wals error if primary has WALs older than the first backup. - Support for double quotes in synchronous_standby_names setting. ## 2.7 (2019-03-12) ### Notable changes - Fix error handling during the parallel backup. Previously an unrecoverable error during the copy could have corrupted the barman internal state, requiring a manual kill of barman process with SIGTERM and a manual cleanup of the running backup in PostgreSQL. (GH#199). - Fix support of UTF-8 characters in input and output (GH#194 and GH#196). - Ignore history/backup/partial files for first sync of geo-redundancy (GH#198). - Fix network failure with geo-redundancy causing cron to break (GH#202). - Fix backup validation in PostgreSQL older than 9.2. - Various documentation fixes. ## 2.6 (2019-02-04) ### Notable changes - Add support for Geographical redundancy, introducing 3 new commands: sync-info, sync-backup and sync-wals. Geo-redundancy allows a Barman server to use another Barman server as data source instead of a PostgreSQL server. - Add put-wal command that allows Barman to safely receive WAL files via PostgreSQL's archive_command using the barman-wal-archive script included in barman-cli. - Add ANSI colour support to check command. ### Bugfixes - Fix switch-wal on standby with an empty WAL directory. - Honour archiver locking in wait_for_wal method. - Fix WAL compression detection algorithm. - Fix current_action in concurrent stop backup errors. - Do not treat lock file busy as an error when validating a backup. ## 2.5 (2018-10-23) ### Notable changes - Add support for PostgreSQL 11 - Add check-backup command to verify that WAL files required for consistency of a base backup are present in the archive. Barman now adds a new state (WAITING_FOR_WALS) after completing a base backup, and sets it to DONE once it has verified that all WAL files from start to the end of the backup exist. This command is included in the regular cron maintenance job. Barman now notifies users attempting to recover a backup that is in WAITING_FOR_WALS state. - Allow switch-xlog --archive to work on a standby (just for the archive part) ### Bugfixes - Fix decoding errors reading external commands output (issue #174). - Fix documentation regarding WAL streaming and backup from standby. ## 2.4 (2018-05-25) ### Notable changes - Add standard and retry hook scripts for backup deletion (pre/post). - Add standard and retry hook scripts for recovery (pre/post). - Add standard and retry hook scripts for WAL deletion (pre/post). - Add --standby-mode option to barman recover to add standby_mode = on in pre-generated recovery.conf. - Add --target-action option to barman recover, allowing users to add shutdown, pause or promote to the pre-generated recovery.conf file. - Improve usability of point-in-time recovery with consistency checks (e.g. recovery time is after end time of backup). - Minor documentation improvements. - Drop support for Python 3.3. ### Bugfixes - Fix remote get_file_content method (GitHub #151), preventing incremental recovery from happening. - Unicode issues with command (GitHub #143 and #150). - Add --wal-method=none when pg_basebackup >= 10 (GitHub #133). - Stop process manager module from overwriting lock files content - Relax the rules for rsync output parsing - Ignore vanished files in streaming directory - Case insensitive slot names (GitHub #170) - Make DataTransferFailure.from_command_error() more resilient (GitHub #86) - Rename command() to barman_command() (GitHub #118) - Initialise synchronous standby names list if not set (GitHub #111) - Correct placeholders ordering (GitHub #138) - Force datestyle to iso for replication connections - Returns error if delete command does not remove the backup - Fix exception when calling is_power_of_two(None) - Downgraded sync standby names messages to debug (GitHub #89) ## 2.3 (2017-09-05) ### Notable changes - Add support to PostgreSQL 10 - Follow naming changes in PostgreSQL 10: - The switch-xlog command has been renamed to switch-wal. - In commands output, the xlog word has been changed to WAL and location has been changed to LSN when appropriate. - Add the --network-compression/--no-network-compression options to barman recover to enable or disable network compression at run-time - Add --target-immediate option to recover command, in order to exit recovery when a consistent state is reached (end of the backup, available from PostgreSQL 9.4) - Show cluster state (master or standby) with barman status command - Documentation improvements ### Bugfixes - Fix high memory usage with parallel_jobs > 1 (#116) - Better handling of errors using parallel copy (#114) - Make barman diagnose more robust with system exceptions - Let archive-wal ignore files with .tmp extension ## 2.2 (2017-07-17) ### Notable changes - Implement parallel copy for backup/recovery through the parallel_jobs global/server option to be overridden by the --jobs or -j runtime option for the backup and recover command. Parallel backup is available only for the rsync copy method. By default, it is set to 1 (for behaviour compatibility with previous versions). - Support custom WAL size for PostgreSQL 8.4 and newer. At backup time, Barman retrieves from PostgreSQL wal_segment_size and wal_block_size values and computes the necessary calculations. - Improve check command to ensure that incoming directory is empty when archiver=off, and streaming directory is empty when streaming_archiver=off (#80). - Add external_configuration to backup_options so that users can instruct Barman to ignore backup of configuration files when they are not inside PGDATA (default for Debian/Ubuntu installations). In this case, Barman does not display a warning anymore. - Add --get-wal and --no-get-wal options to barman recover - Add max_incoming_wals_queue global/server option for the check command so that a non blocking error is returned in case incoming WAL directories for both archiver and the streaming_archiver contain more files than the specified value. - Documentation improvements - File format changes: - The format of backup.info file has changed. For this reason a backup taken with Barman 2.2 cannot be read by a previous version of Barman. But, backups taken by previous versions can be read by Barman 2.2. ### Bugfixes - Allow replication-status to work against a standby - Close any PostgreSQL connection before starting pg_basebackup (#104, #108) - Safely handle paths containing special characters - Archive .partial files after promotion of streaming source - Recursively create directories during recovery (SF#44) - Improve xlog.db locking (#99) - Remove tablespace_map file during recover (#95) - Reconnect to PostgreSQL if connection drops (SF#82) ## 2.1 (2017-01-05) ### Notable changes - Add --archive and --archive-timeout options to switch-xlog command. - Preliminary support for PostgreSQL 10 (#73). - Minor additions: - Add last archived WAL info to diagnose output. - Add start time and execution time to the output of delete command. ### Bugfixes - Return failure for get-wal command on inactive server - Make streaming_archiver_names and streaming_backup_name options global (#57) - Fix rsync failures due to files truncated during transfer (#64) - Correctly handle compressed history files (#66) - Avoid de-referencing symlinks in pg_tblspc when preparing recovery (#55) - Fix comparison of last archiving failure (#40, #58) - Avoid failing recovery if postgresql.conf is not writable (#68) - Fix output of replication-status command (#56) - Exclude files from backups like pg_basebackup (#65, #72) - Exclude directories from other Postgres versions while copying tablespaces (#74) - Make retry hook script options global ## 2.0 (2016-09-27) ### Notable changes - Support for pg_basebackup and base backups over the PostgreSQL streaming replication protocol with backup_method=postgres (PostgreSQL 9.1 or higher required) - Support for physical replication slots through the slot_name configuration option as well as the --create-slot and --drop-slot options for the receive-wal command (PostgreSQL 9.4 or higher required). When slot_name is specified and streaming_archiver is enabled, receive-wal transparently integrates with pg_receivexlog, and check makes sure that slots exist and are actively used - Support for the new backup API introduced in PostgreSQL 9.6, which transparently enables concurrent backups and backups from standby servers using the standard rsync method of backup. Concurrent backup was only possible for PostgreSQL 9.2 to 9.5 versions through the pgespresso extension. The new backup API will make pgespresso redundant in the future - If properly configured, Barman can function as a synchronous standby in terms of WAL streaming. By properly setting the streaming_archiver_name in the synchronous_standby_names priority list on the master, and enabling replication slot support, the receive-wal command can now be part of a PostgreSQL synchronous replication cluster, bringing RPO=0 (PostgreSQL 9.5.5 or higher required) - Introduce barman-wal-restore, a standard and robust script written in Python that can be used as restore_command in recovery.conf files of any standby server of a cluster. It supports remote parallel fetching of WAL files by efficiently invoking get-wal through SSH. Currently available as a separate project called barman-cli. The barman-cli package is required for remote recovery when get-wal is listed in recovery_options - Control the maximum execution time of the check command through the check_timeout global/server configuration option (30 seconds by default) - Limit the number of WAL segments that are processed by an archive-wal run, through the archiver_batch_size and streaming_archiver_batch_size global/server options which control archiving of WAL segments coming from, respectively, the standard archiver and receive-wal - Removed locking of the XLOG database during check operations - The show-backup command is now aware of timelines and properly displays which timelines can be used as recovery targets for a given base backup. Internally, Barman is now capable of parsing .history files - Improved the logic behind the retry mechanism when copy operations experience problems. This involves backup (rsync and postgres) as well as remote recovery (rsync) - Code refactoring involving remote command and physical copy interfaces ### Bugfixes - Correctly handle .history files from streaming - Fix replication-status on PostgreSQL 9.1 - Fix replication-status when sent and write locations are not available - Fix misleading message on pg_receivexlog termination ## 1.6.1 (2016-05-23) ### Minor changes - Add --peek option to get-wal command to discover existing WAL files from the Barman's archive - Add replication-status command for monitoring the status of any streaming replication clients connected to the PostgreSQL server. The --target option allows users to limit the request to only hot standby servers or WAL streaming clients - Add the switch-xlog command to request a switch of a WAL file to the PostgreSQL server. Through the '--force' it issues a CHECKPOINT beforehand - Add streaming_archiver_name option, which sets a proper application_name to pg_receivexlog when streaming_archiver is enabled (only for PostgreSQL 9.3 and above) - Check for _superuser_ privileges with PostgreSQL's standard connections (#30) - Check the WAL archive is never empty - Check for 'backup_label' on the master when server is down - Improve barman-wal-restore contrib script ### Bugfixes - Treat the "failed backups" check as non-fatal - Rename '-x' option for get-wal as '-z' - Add archive_mode=always support for PostgreSQL 9.5 (#32) - Properly close PostgreSQL connections when necessary - Fix receive-wal for pg_receive_xlog version 9.2 ## 1.6.0 (2016-02-29) ### Notable changes - Support for streaming replication connection through the streaming_conninfo server option - Support for the streaming_archiver option that allows Barman to receive WAL files through PostgreSQL's native streaming protocol. When set to 'on', it relies on pg_receivexlog to receive WAL data, reducing Recovery Point Objective. Currently, WAL streaming is an additional feature (standard log archiving is still required) - Implement the receive-wal command that, when streaming_archiver is on, wraps pg_receivexlog for WAL streaming. Add --stop option to stop receiving WAL files via streaming protocol. Add --reset option to reset the streaming status and restart from the current xlog in Postgres. - Automatic management (startup and stop) of receive-wal command via cron command - Support for the path_prefix configuration option - Introduction of the archiver option (currently fixed to on) which enables continuous WAL archiving for a specific server, through log shipping via PostgreSQL's archive_command - Support for streaming_wals_directory and errors_directory options - Management of WAL duplicates in archive-wal command and integration with check command - Verify if pg_receivexlog is running in check command when streaming_archiver is enabled - Verify if failed backups are present in check command - Accept compressed WAL files in incoming directory - Add support for the pigz compressor (thanks to Stefano Zacchiroli ) - Implement pygzip and pybzip2 compressors (based on an initial idea of Christoph Moench-Tegeder ) - Creation of an implicit restore point at the end of a backup - Current size of the PostgreSQL data files in barman status - Permit archive_mode=always for PostgreSQL 9.5 servers (thanks to Christoph Moench-Tegeder ) - Complete refactoring of the code responsible for connecting to PostgreSQL - Improve messaging of cron command regarding sub-processes - Native support for Python >= 3.3 - Changes of behaviour: - Stop trashing WAL files during archive-wal (commit:e3a1d16) ### Bugfixes - Atomic WAL file archiving (#9 and #12) - Propagate "-c" option to any Barman subprocess (#19) - Fix management of backup ID during backup deletion (#22) - Improve archive-wal robustness and log messages (#24) - Improve error handling in case of missing parameters ## 1.5.1 (2015-11-16) ### Minor changes - Add support for the 'archive-wal' command which performs WAL maintenance operations on a given server - Add support for "per-server" concurrency of the 'cron' command - Improved management of xlog.db errors - Add support for mixed compression types in WAL files (SF.net#61) ### Bugfixes - Avoid retention policy checks during the recovery - Avoid 'wal_level' check on PostgreSQL version < 9.0 (#3) - Fix backup size calculation (#5) ## 1.5.0 (2015-09-28) ### Notable changes - Add support for the get-wal command which allows users to fetch any WAL file from the archive of a specific server - Add support for retry hook scripts, a special kind of hook scripts that Barman tries to run until they succeed - Add active configuration option for a server to temporarily disable the server by setting it to False - Add barman_lock_directory global option to change the location of lock files (by default: 'barman_home') - Execute the full suite of checks before starting a backup, and skip it in case one or more checks fail - Forbid to delete a running backup - Analyse include directives of a PostgreSQL server during backup and recover operations - Add check for conflicting paths in the configuration of Barman, both intra (by temporarily disabling a server) and inter-server (by refusing any command, to any server). - Add check for wal_level - Add barman-wal-restore script to be used as restore_command on a standby server, in conjunction with barman get-wal - Implement a standard and consistent policy for error management - Improved cache management of backups - Improved management of configuration in unit tests - Tutorial and man page sources have been converted to Markdown format - Add code documentation through Sphinx - Complete refactor of the code responsible for managing the backup and the recover commands - Changed internal directory structure of a backup - Introduce copy_method option (currently fixed to rsync) ### Bugfixes - Manage options without '=' in PostgreSQL configuration files - Preserve Timeline history files (Fixes: #70) - Workaround for rsync on SUSE Linux (Closes: #13 and #26) - Disables dangerous settings in postgresql.auto.conf (Closes: #68) ## 1.4.1 (2015-05-05) ### Minor changes - Improved management of xlogdb file, which is now correctly fsynced when updated. Also, the rebuild-xlogdb command now operates on a temporary new file, which overwrites the main one when finished. - Add unit tests for dateutil module compatibility - Modified Barman version following PEP 440 rules and added support of tests in Python 3.4 ### Bugfixes - Fix for WAL archival stop working if first backup is EMPTY (Closes: #64) - Fix exception during error handling in Barman recovery (Closes: #65) - After a backup, limit cron activity to WAL archiving only (Closes: #62) - Improved robustness and error reporting of the backup delete command (Closes: #63) - Fix computation of WAL production ratio as reported in the show-backup command ## 1.4.0 (2015-01-26) ### Notable changes - Incremental base backup implementation through the reuse_backup global/server option. Possible values are off (disabled, default), copy (preventing unmodified files from being transferred) and link (allowing for deduplication through hard links). - Store and show deduplication effects when using reuse_backup= link. - Added transparent support of pg_stat_archiver (PostgreSQL 9.4) in check, show-server and status commands. - Improved administration by invoking WAL maintenance at the end of a successful backup. - Changed the way unused WAL files are trashed, by differentiating between concurrent and exclusive backup cases. - Improved performance of WAL statistics calculation. - Treat a missing pg_ident.conf as a WARNING rather than an error. - Refactored output layer by removing remaining yield calls. - Check that rsync is in the system path. - Include history files in WAL management. - Improved robustness through more unit tests. ### Bugfixes - Fixed bug #55: Ignore fsync EINVAL errors on directories. - Fixed bug #58: retention policies delete. ## 1.3.3 (2014-08-21) ### Notable changes - Added "last_backup_max_age", a new global/server option that allows administrators to set the max age of the last backup in a catalogue, making it easier to detect any issues with periodical backup execution - Improved robustness of "barman backup" by introducing two global/ server options: "basebackup_retry_times" and "basebackup_retry_sleep". These options allow an administrator to specify, respectively, the number of attempts for a copy operation after a failure, and the number of seconds of wait before retrying - Improved the recovery process via rsync on an existing directory (incremental recovery), by splitting the previous rsync call into several ones - invoking checksum control only when necessary - Added support for PostgreSQL 8.3 ### Minor changes - Support for comma separated list values configuration options - Improved backup durability by calling fsync() on backup and WAL files during "barman backup" and "barman cron" - Improved Nagios output for "barman check --nagios" - Display compression ratio for WALs in "barman show-backup" - Correctly handled keyboard interruption (CTRL-C) while performing barman backup - Improved error messages of failures regarding the stop of a backup - Wider coverage of unit tests ### Bugfixes - Copies "recovery.conf" on the remote server during "barman recover" (#45) - Correctly detect pre/post archive hook scripts (#41) ## 1.3.2 (2014-04-15) ### Bugfixes - Fixed incompatibility with PostgreSQL 8.4 (Closes #40, bug introduced in version 1.3.1) ## 1.3.1 (2014-04-14) ### Minor changes - Added support for concurrent backup of PostgreSQL 9.2 and 9.3 servers that use the "pgespresso" extension. This feature is controlled by the "backup_options" configuration option (global/ server) and activated when set to "concurrent_backup". Concurrent backup allows DBAs to perform full backup operations from a streaming replicated standby. - Added the "barman diagnose" command which prints important information about the Barman system (extremely useful for support and problem solving) - Improved error messages and exception handling interface ### Bugfixes - Fixed bug in recovery of tablespaces that are created inside the PGDATA directory (bug introduced in version 1.3.0) - Fixed minor bug of unhandled -q option, for quiet mode of commands to be used in cron jobs (bug introduced in version 1.3.0) - Minor bug fixes and code refactoring ## 1.3.0 (2014-02-03) ### Notable changes - Refactored BackupInfo class for backup metadata to use the new FieldListFile class (infofile module) - Refactored output layer to use a dedicated module, in order to facilitate integration with Nagios (NagiosOutputWriter class) - Refactored subprocess handling in order to isolate stdin/stderr/ stdout channels (command_wrappers module) - Refactored hook scripts management - Extracted logging configuration and userid enforcement from the configuration class. - Support for hook scripts to be executed before and after a WAL file is archived, through the 'pre_archive_script' and 'post_archive_script' configuration options. - Implemented immediate checkpoint capability with --immediate-checkpoint command option and 'immediate_checkpoint' configuration option - Implemented network compression for remote backup and recovery through the 'network_compression' configuration option (#19) - Implemented the 'rebuild-xlogdb' command (Closes #27 and #28) - Added deduplication of tablespaces located inside the PGDATA directory - Refactored remote recovery code to work the same way local recovery does, by performing remote directory preparation (assuming the remote user has the right permissions on the remote server) - 'barman backup' now tries and create server directories before attempting to execute a full backup (#14) ### Bugfixes - Fixed bug #22: improved documentation for tablespaces relocation - Fixed bug #31: 'barman cron' checks directory permissions for lock file - Fixed bug #32: xlog.db read access during cron activities ## 1.2.3 (2013-09-05) ### Minor changes - Added support for PostgreSQL 9.3 - Added support for the "--target-name" recovery option, which allows to restore to a named point previously specified with pg_create_restore_point (only for PostgreSQL 9.1 and above users) - Introduced Python 3 compatibility ### Bugfixes - Fixed bug #27 about flock() usage with barman.lockfile (many thanks to Damon Snyder ) ## 1.2.2 (2013-06-24) ### Bugfixes - Fix python 2.6 compatibility ## 1.2.1 (2013-06-17) ### Minor changes - Added the "bandwidth_limit" global/server option which allows to limit the I/O bandwidth (in KBPS) for backup and recovery operations - Added the "tablespace_bandwidth_limit" global/server option which allows to limit the I/O bandwidth (in KBPS) for backup and recovery operations on a per tablespace basis - Added /etc/barman/barman.conf as default location ### Bugfixes - Avoid triggering the minimum_redundancy check on FAILED backups (thanks to Jérôme Vanandruel) ## 1.2.0 (2013-01-31) ### Notable changes - Added the "retention_policy_mode" global/server option which defines the method for enforcing retention policies (currently only "auto") - Added the "minimum_redundancy" global/server option which defines the minimum number of backups to be kept for a server - Added the "retention_policy" global/server option which defines retention policies management based on redundancy (e.g. REDUNDANCY 4) or recovery window (e.g. RECOVERY WINDOW OF 3 MONTHS) - Added retention policy support to the logging infrastructure, the "check" and the "status" commands - The "check" command now integrates minimum redundancy control - Added retention policy states (valid, obsolete and potentially obsolete) to "show-backup" and "list-backup" commands - The 'all' keyword is now forbidden as server name - Added basic support for Nagios plugin output to the 'check' command through the --nagios option - Barman now requires argh => 0.21.2 and argcomplete- - Minor bug fixes ## 1.1.2 (2012-11-29) ### Minor changes - Added "configuration_files_directory" option that allows to include multiple server configuration files from a directory - Support for special backup IDs: latest, last, oldest, first - Management of multiple servers to the 'list-backup' command. 'barman list-backup all' now list backups for all the configured servers. - Added "application_name" management for PostgreSQL >= 9.0 ### Bugfixes - Fixed bug #18: ignore missing WAL files if not found during delete ## 1.1.1 (2012-10-16) ### Bugfixes - Fix regressions in recover command. ## 1.1.0 (2012-10-12) ### Notable changes - Support for hook scripts to be executed before and after a 'backup' command through the 'pre_backup_script' and 'post_backup_script' configuration options. - Management of multiple servers to the 'backup' command. 'barman backup all' now iteratively backs up all the configured servers. - Add warning in recovery when file location options have been defined in the postgresql.conf file (issue #10) - Fail fast on recover command if the destination directory contains the ':' character (Closes: #4) or if an invalid tablespace relocation rule is passed - Report an informative message when pg_start_backup() invocation fails because an exclusive backup is already running (Closes: #8) ### Bugfixes - Fixed bug #9: "9.2 issue with pg_tablespace_location()" ## 1.0.0 (2012-07-06) ### Notable changes - Backup of multiple PostgreSQL servers, with different versions. Versions from PostgreSQL 8.4+ are supported. - Support for secure remote backup (through SSH) - Management of a catalog of backups for every server, allowing users to easily create new backups, delete old ones or restore them - Compression of WAL files that can be configured on a per server basis using compression/decompression filters, both predefined (gzip and bzip2) or custom - Support for INI configuration file with global and per-server directives. Default location for configuration files are /etc/barman.conf or ~/.barman.conf. The '-c' option allows users to specify a different one - Simple indexing of base backups and WAL segments that does not require a local database - Maintenance mode (invoked through the 'cron' command) which performs ordinary operations such as WAL archival and compression, catalog updates, etc. - Added the 'backup' command which takes a full physical base backup of the given PostgreSQL server configured in Barman - Added the 'recover' command which performs local recovery of a given backup, allowing DBAs to specify a point in time. The 'recover' command supports relocation of both the PGDATA directory and, where applicable, the tablespaces - Added the '--remote-ssh-command' option to the 'recover' command for remote recovery of a backup. Remote recovery does not currently support relocation of tablespaces - Added the 'list-server' command that lists all the active servers that have been configured in barman - Added the 'show-server' command that shows the relevant information for a given server, including all configuration options - Added the 'status' command which shows information about the current state of a server, including Postgres version, current transaction ID, archive command, etc. - Added the 'check' command which returns 0 if everything Barman needs is functioning correctly - Added the 'list-backup' command that lists all the available backups for a given server, including size of the base backup and total size of the related WAL segments - Added the 'show-backup' command that shows the relevant information for a given backup, including time of start, size, number of related WAL segments and their size, etc. - Added the 'delete' command which removes a backup from the catalog - Added the 'list-files' command which lists all the files for a single backup - RPM Package for RHEL 5/6 barman-3.13.3/AUTHORS0000644000175100001660000000260715002177425012302 0ustar 00000000000000Barman maintainers (in alphabetical order): * Andre Marchesini * Barbara Leidens * Giulio Calacoci * Gustavo Oliveira * Israel Barth * Martín Marqués Past contributors (in alphabetical order): * Abhijit Menon-Sen (architect) * Anna Bellandi (QA/testing) * Britt Cole (documentation reviewer) * Carlo Ascani (developer) * Didier Michel (developer) * Francesco Canovai (QA/testing) * Gabriele Bartolini (architect) * Gianni Ciolli (QA/testing) * Giulio Calacoci (developer) * Giuseppe Broccolo (developer) * Jane Threefoot (developer) * Jonathan Battiato (QA/testing) * Leonardo Cecchi (developer) * Marco Nenciarini (project leader) * Michael Wallace (developer) * Niccolò Fei (QA/testing) * Rubens Souza (QA/testing) * Stefano Bianucci (developer) Many thanks go to our sponsors (in alphabetical order): * 4Caast - http://4caast.morfeo-project.org/ (Founding sponsor) * Adyen - http://www.adyen.com/ * Agile Business Group - http://www.agilebg.com/ * BIJ12 - http://www.bij12.nl/ * CSI Piemonte - http://www.csipiemonte.it/ (Founding sponsor) * Ecometer - http://www.ecometer.it/ * GestionaleAuto - http://www.gestionaleauto.com/ (Founding sponsor) * Jobrapido - http://www.jobrapido.com/ * Navionics - http://www.navionics.com/ (Founding sponsor) * Sovon Vogelonderzoek Nederland - https://www.sovon.nl/ * Subito.it - http://www.subito.it/ * XCon Internet Services - http://www.xcon.it/ (Founding sponsor) barman-3.13.3/README.rst0000644000175100001660000000402215002177425012712 0ustar 00000000000000Barman, Backup and Recovery Manager for PostgreSQL ================================================== This is the new (starting with version 2.13) home of Barman. It replaces the legacy sourceforge repository. Barman (Backup and Recovery Manager) is an open-source administration tool for disaster recovery of PostgreSQL servers written in Python. It allows your organisation to perform remote backups of multiple servers in business critical environments to reduce risk and help DBAs during the recovery phase. Barman is distributed under GNU GPL 3 and maintained by EnterpriseDB. For further information, look at the "Web resources" section below. Source content -------------- Here you can find a description of files and directory distributed with Barman: - AUTHORS : development team of Barman - NEWS : release notes - ChangeLog : log of changes - LICENSE : GNU GPL3 details - TODO : our wishlist for Barman - barman : sources in Python - docs : tutorial and man pages - scripts : auxiliary scripts - tests : unit tests Web resources ------------- - Website : http://www.pgbarman.org/ - Download : http://github.com/EnterpriseDB/barman - Documentation : http://www.pgbarman.org/documentation/ - Community support : http://www.pgbarman.org/support/ - Professional support : https://www.enterprisedb.com/ - pre barman 2.13 versions : https://sourceforge.net/projects/pgbarman/files/ Licence ------- © Copyright 2011-2025 EnterpriseDB UK Limited Barman is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. Barman is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with Barman. If not, see http://www.gnu.org/licenses/.