././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4964876 virtnbdbackup-2.30/0000755000000000000000000000000015033267565013046 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/Changelog0000644000000000000000000012113015033267156014652 0ustar00rootrootVersion 2.30 --------- * virtnbdbackup: handle lost CBT case (#272): fallback to full backup in case stored dirty bitmaps in the qcow images are not sync with libvirt checkpoints. (Denis V. Lunev ) Version 2.29 --------- * virtnbdrestore: wrong vm config XML used if until option is used (#270) Version 2.28 --------- * virtnbdbackup: [Denis V. Lunev ] - improve overlap() performance. The call should be O(L) complexity (#265) Version 2.27 --------- * virtnbdbackup: - Skip saving checksum information for raw images included with --raw option: The checksum calculated does not actually match during verify because certain parts of the images may be seeked instead of actually written. (#269) Version 2.26 --------- * virtnbdbackup: - Add info log message before and after extent query has finished. - Enhance function to detect sparse/fstrimmed regions (#263). - Enhance CI tests Version 2.25 --------- * virtnbdbackup: - Detect QCOW2 formatted direct attached RAW Volumes (#264): Some solutions such as OpenNebula will use direct attached LVM volumes in virtual machines. These LVM volumes were treated like a RAW device, but allow all required operations for full/incremental backup. Changes introduced in v2.18 would exclude these devices from backup even if `--raw` option is passed. Now during disk detection, use the disk format in the driver type setting to correctly detect a lvm volume backed by qcow2, and handle them just like regular qcow based images. Version 2.24 --------- * virnbdbackup: - Add option --no-sparse-detection (#263): The feature implemented as with version 2.21 requires to query the complete base:allocation bitmap during incremental backup. Depending on the Hypervisor hardware and the virtual machine disk size, querying the base:allocation bitmap may take longer than to simply treat fstrimmed blocks the same way as changed blocks and include them in the backup. The option introduced can be used to disable the new behavior and can enhance backup times for specific users. Version 2.23 --------- * virtnbdbackup: - Log created nbd socket information in URI format, so it can be reused with external tools more easily. - Add human readable backup sizes in log output. Version 2.22 --------- * Test enhancements: - check btrfs filesystem stats in restored VM. - run checksum verify after all backups. * virtnbdbackup: - If NBD connection drops, catch correct exception. (#254) - Provide better error message if checkpoint inconsistency is detected. Version 2.21 --------- * virtnbdbackup: Detect dirty regions containing only zeroes during incremental backup (fstrim) (#250) Before, any dirty block being part of the incremental/differential bitmap would be backed up, even if not required. Thanks to the input and enhancements of Roman Grigoriev and Denis V. Lunev the zeroed block regions are now compared to the base:allocation bitmap and detected as such. This reduces the amount of data backed up if virtual machines have disks configured with discard=unmap option and use fstrim to free unused space. Version 2.20 --------- * virtnbdmap: add option -H for better debugging, code cleanup * virtnbdmap: fix situations where multiple blocks in row are required to map the correct data. Version 2.19 --------- * virtnbdmap sometimes fails to correctly map disk images due to block offsets spanning multiple frames in the stream format, resulting in Input/Output errors accessing the disk partition data (#249) Version 2.18 --------- * If auto backup mode is switched to incremental and --raw option is passed, backup would fails instead of showing an warning that raw devices are excluded (#239) Version 2.17 --------- * Add rpm package build for fedora 41 (#234) * virtnbdbackup: Remote backup fails if spaces exist in image path (#235) * virtnbdrestore: Remote restore would fail if target path contains spaces. Version 2.16 --------- * virtnbdrestore: If domain with multiple disks is restored and config adjustment is requested, only the path to the last disk is updated. (#232). Thanks Richard Stephens for report and fix. * virtnbdrestore: fix logic for adjusting the vm name Version 2.15 --------- * Update Dockerfile and README * virtnbdrestore: use absolute path to disk files during local restore. * virtnbdrestore: make absolute path mandatory for remote restore. Version 2.14 --------- * virtnbdrestore: Add option -A (--preallocate) to create thick, full pre-allocated target images during restore. Version 2.13 --------- * virtnbdbackup: If libvirt daemon is unreachable, it would fail with an misleading error code due exception not raised. * virtnbdbackup: setup callback function using registerCloseCallback, if libvirt connection drops during backup, re-establish the connection in order to stop leftover backup tasks * virtnbdbackup: catch exceptions if nbd read fails. * virtnbdrestore: add option -C to adjust config file name during restore. Version 2.12 --------- * Update README * virtnbdrestore: fix exception if no $HOME environment variable is set (#197) Version 2.11 --------- * Update README * Raise NBD connection error if `--tls` option is set but installed libnbd bindings for python do not support required features. * Set TLS connection options to TLS_REQUIRE instead of TLS_ALLOW which would fallback to non-encrypted data transfer. * If LIBNBD_DEBUG is set, add NBD trace messages to logfile. Version 2.10 --------- * Add --ssh-port option. Version 2.9 --------- * Fix: backup with compression enabled fails: unsupported operand type(s) for +: 'int' and 'dict' (#177) Version 2.8 --------- * Add packages compatible to fedora 39 to package build (#174) * Show total saved disk size in human readable output (#173) Version 2.7 --------- * Update README * End backup with warning if software emulated TPM device is attached (#169) * Detect remote connection based on libvirt URI, checking hostnames could lead to situation where local backup is detected as remote backup (#170) Version 2.6 --------- * Fix IndexError exception if auth file is used in qemu uri (#167) * Credential function for libvirt must return integer: fix NoneType exception if actual libvirt authentication is required. * Simplify libvirt authentication code: attempt to use SASL based mechanism only if --user and --password options are set. * If authentication fails because of missing SASL mechs, ajdust error message, provide hint for --user and --password options. * Update README regards OVIRT/RHEV/OLVM (no mechanism available: No worthy mechs found) Version 2.5 --------- * Move some log messages from info to debug loglevel * Log information about libnbd version only once * Catch command not found error during remote backup if qemu-img is missing: change loglevel to warning. * Catch command not found error during remote restore: fail with proper error message. * If no qcow image info has been created during backup, issue warning during restore that default options are used. * Update README: add note about scratch files. * Do not attempt to freeze filesystems if virtual machine is in paused state (#166) Version 2.4 --------- * Update RADME * Add qemu-utils to package dependencies: if installed on a system without libvirt/qemu, backup fails because of missing qemu-img executable. * Add openssh-client to package dependencies: required for remote libvirtd connection. * Catch exception if executed commands such as qemu-img are missing on system. * Code cleanup Version 2.3 --------- * Update README * Add option -S (--start-domain): if specified and virtual domain is offline during backup, domain will be started in pause mode, allowing to execute full/diff/inc backups. Domain is destroyed as soon as operation finished by using libvirt's AUTODESTROY flag. (#164) * Move code for preflight tests to separate module. Version 2.2 --------- * Fix Progressbar during restore: wrong values used. (#160) * Catch exception if during restore connection to NBD server fails (#163) * Provide better info message what NBD connection is waiting for. * Call flush() on NBD connection during restore: restore of domains with multiple disks could fail with NbdConnectionTimeout due to race condition (#163) * Pass pidFile to qemu-nbd process for local NBD server during restore, report PID of forked process instead of parent. Version 2.1 --------- * Fix Progressbar: Since change for issue #133 the Progressbar was updated with wrong values and as result progressed too fast for the amount of data actually written. (#160) Version 2.0 --------- * Update README * Fail early during incremental or differential backup if no full backup is found in target directory. Version 1.9.55 --------- * virtnbdmap: pass listen address argument to qemu-nbd command. Version 1.9.54 --------- * Exit with error if both compress option and raw output format is specified: options are mutually exclusive. * Update progressbar during chunked data read/write. If big portions of the disk are backed up, it would appear as if no progress is made (#133) * If an active remote backup is aborted (ctrl+c), stopping the libvirt backup job failed because the signal is sent to the used ssh connection, too. Now the signal handler attempts to re establish the connection to the remote system in order to correctly stop the backup operation. * Add close() function to libvirt client class: disconnect libvirt after backup has finished. * virtnbdmap: code reorg. Version 1.9.53 --------- * Move checksum related code to own function * Set a shorter thread name, makes logging messages are easier to read. * Add opensuse/leap:15.5 to release build. Version 1.9.52 --------- * Code cleanup * Build release packages using github actions. Version 1.9.51 --------- * Slight change in log message wording. * virtnbdrestore: if starting NBD service on remote system fails, catch exceptions accordingly. * virtnbdrestore: if --nbd-ip parameter is set, pass -b option to qemu-nbd command so the NDB service binds to specified IP address. * virtnbdrestore: Set SSH session mode to UPLOAD, otherwise uploading files such as UEFI images and vm config fails. * virtnbdrestore: Catch exception if connection to libvirt fails * virtnbdrestore: Create target directory on remote system only if it does not exist, Thanks @draggeta * CI: enhance remote backup check: test if UEFI loader files are copied correctly during remote backup. Version 1.9.50 --------- * Add openssh-client to docker image required for remote backup functionality (#151) * Check if specified ip address in `--nbd-ip` parameter is an ipv6 address and if so, use ipv6 ip notation for NBD client connection (#150) * Fix NBD connection timeout: current implementation only waited until NBD server on socket was reachable. Now it also attempts to retry connection for remote TCP NBD servers (#150) * Fix exception in ssh client during raise Version 1.9.49 --------- * Logging: Use blue color definition instead of light-blue which might not be supported by all colorlog versions (#148) Version 1.9.48 --------- * Rework some error messages / exception handling. Version 1.9.47 --------- * Remote backup: catch more exceptions accordingly, provide better error message if ssh connection fails. Version 1.9.46 --------- * Update package dependencies in docker file. * Spelling fixes Version 1.9.45 --------- * virtnbdbackup: Add more info output during checkpoint handling, move error message. * Add colorful log messages so warnings and and errors are easier to spot: can be disabled using the --nocolor option. Version 1.9.44 --------- * virtnbdbackup: disable progressbar if quiet option is enabled. Version 1.9.43 --------- * virtnbdbackup: add --quiet option: disable log output to stderr, log into logfile only. (#137) Version 1.9.42 --------- * virtnbdbackup: skip checksum output if backup is redirected to stdout Version 1.9.41 --------- * virtnbdrestore: add Option -B to allow changing default buffer size during verify to speed up process. Version 1.9.40 --------- * Update README * New feature: compute adler32 checksum for written data and add verify option to virtnbdrestore, so its possible to check for corrupt backups without having to restore. Version 1.9.39 --------- * Update README Version 1.9.38 --------- * Release packages compatible with RHEL9 (#130) + add required vagrant file + adjust requirements during package build + put dist files into separate directories Version 1.9.37 --------- * Update README: add example how to convert qcow images to the required format to support all backup features. Version 1.9.36 --------- * Fail with better understandable error message if libvirt version requires the virtual machine configuration to be adjusted for backup to work. Version 1.9.35 --------- * Fix some pylint warnings, add ignores. Version 1.9.34 --------- * Update README Version 1.9.33 --------- * Update README Version 1.9.32 --------- * Fix TypeError during backup of transient virtual machines: do not pass flags to checkpointCreateXML as list. (#122) * Add simple test for transient virtual machine handling. Version 1.9.31 --------- * setup.cfg: fix obsolete description-file warning * In case domain has an already active block job, add note about option -k to the error message (#120). Version 1.9.30 --------- * Update README Version 1.9.29 --------- * Update README Version 1.9.28 --------- * Remote backup: enhance logging if executing remote command via SSH fails. Version 1.9.27 --------- * Enhance debug logging for remote backup. * Update README regarding remote backup and its requirements. (#117) Version 1.9.26 --------- * virtnbdbackup: transient vm backup: validate the state of the checkpoint during redefine: this can help to detect situations where the bitmap is broken. * virtnbdmap: Fix typo in log message. * Update README regarding port usage during remote backups. * Add notes regarding stability of the features, thanks @pipo Version 1.9.25 --------- * Fix typo in log messages. Version 1.9.24 --------- * Remove leftover pid files in /var/tmp during offline backup (#114) Version 1.9.23 --------- * Code cleanups * Spelling fixes, run codespell during CI Version 1.9.22 --------- * Code cleanups, pylint warning * Update copyright stanca Version 1.9.21 --------- * Code cleanups Version 1.9.20 --------- * More code cleanup and reorg * Module directory is now pylint clean (except for some ignores): run pylint using github actions. Version 1.9.19 --------- * Code cleanup: move some functions to separate files. Version 1.9.18 --------- * virtnbdbackup: treat direct attached lun and block devices just like regular raw disks: backup will work but only in backup mode copy and of course full provisioned backup of these block devices is created if option `--raw` is enabled. (#111) Version 1.9.17 --------- * virtnbdrestore: fix restore fails with IndexError if CDROM or LUN devices appear in the list of devices before the first disk. (#110). Version 1.9.16 --------- * virtnbdrestore: use disk size from the latest backup during restore: handle situations where disk size has changed between full and incremental or differential backups (#109) Version 1.9.15 --------- * Add install_requires to setup.py * Backup and restore autostart setting for domain if set. (#106) Version 1.9.14 --------- * Adjust log messages during disk parsing. * Ignore direct attached block devices which use disk type notation (#98) Version 1.9.13 --------- * Fallback to backup mode copy if virtual machine has only raw disks attached (#94) * Some fixes in regard on how attached 'raw' disks are handled: + fail correctly if backup to stdout is attempted + override --raw option during attempted incremental or differential backup: backup modes will only be attempted for disks supporting the required features. * If no disks suitable for backup are detected or backup fails, save at least virtual machine configuration and related kernel/uefi files (#93) Version 1.9.12 --------- * Rework logging facilities: use local log instance of global logging object to have saner logging information. Version 1.9.11 --------- * Log traceback for unknown exceptions for easier analysis of issues. (#92) Version 1.9.10 --------- * Remote incremental backup fails: checkpoint virtnbdbackup.1 already exists: check for checkpoint file only on local system, not on the remote system (#89) Version 1.9.9 --------- * Fix exception handling/raise in case bitmap already exists. (#85) Version 1.9.8 --------- * Code cleanup Version 1.9.7 --------- * Update README * Code cleanup, use shorter class names, change some error handling. * virtnbdrestore: in case option -D is passed, set option to adjust virtual machine config automatically. * virtnbdbackup: amount of ports used for offline backup depends on how many workers are used, not how much disks are attached, log correct port range use during offline backup. Development: * Github workflow now saves all files created by testsuite for better analysis, extend some testcases. Version 1.9.6 --------- * virtnbdrestore: add detection for disks with volume notation, reset type to file during restore so virtual machine config is adjusted to new restore paths. (#81) Version 1.9.5 --------- * virtnbdrestore and virtnbdmap: code cleanups, add type annotations. Version 1.9.4 --------- * Code cleanup: more type annotations, correctly catch ssh exceptions. * Better logging information in case of remote backup * Remote backup of offline virtual machines with multiple disks may fail: (#82) port used by remote NBD Service must be unique. Also report used ports in logfile. Version 1.9.3 --------- * Move function call for backing up qemu disk information outside worker threads: should fix occasional "Fatal Python error: _enter_buffered_busy" error during backup. Version 1.9.2 --------- * Return function if gathering qemu disk info fails for some reason. * If remote backup is saved to local zip file, backup may fail because absolute path to additional loader/nvram files is wrong. * Fix naming if copy backup is saved to zip file output: dont append checkpoint information to created files. Version 1.9.1 --------- * Code cleanup: fix pylint warning, remove obsolete p.wait() calls with hardcoded time limit, start adding type annotations, reworked imports. * Fix disk detection with volume notation. (#81) * Backup and restore qcow image related settings: during backup qemu-img info is called on qcow based image devices and qcow specific options are stored as json dump. If information is existent, on restore the original settings of the qcow image (such as cluster size or other options) are now applied during image creation. (#80) * Fix some race conditions during restore of multiple disks and sequences: NBD connection handle was not closed, following qemu-nbd calls would fail due to locked files or already in use tcp ports. Development: * Change github workflows: now use ubuntu 22.04 and add test for remote backup via localhost. Version 1.9 --------- * Update README * Make --compress option configurable, it is now possible to set compression level (--compress=X) (#77) * Fix offline backup of virtual machines with multiple disks attached: nbd connection would use wrong socket file. (#76) * Fix offline backup of remote vms: set missing socket file parameter. Version 1.8.2 --------- * Code cleanup * Fix Issue (#74): Remote backup fails to copy [loader] file Version 1.8.1 --------- * Update manpages to be more debian compliant * Add manpages to manifest file, too Version 1.8 --------- * Add manpages Version 1.7 --------- * Code cleanup Version 1.6 --------- * virtnbdbackup: + add --syslog option: enable log output to system syslog facility. Version 1.5 --------- * virtnbdrestore: + add --logfile option (defaults to $HOME) + print actual json if dumping saveset contents * virtnbdmap: + add --logfile option (defaults to $HOME) Version 1.4 --------- * If backup/restore is executed as regular user, set default uri to qemu:///session. Version 1.3 --------- * Code cleanup * Update README Version 1.2 --------- * Code cleanup Version 1.1 --------- * Code cleanup * Add warning if reading checkpoint information with size fails. Version 1.0 --------- * Code cleanup * (#69) Rework --printonly option: use checkpoint size as reported by libvirt for estimating next incremental or differential backup size. * (#70) Add --threshold option for incremental / differential backup Version 0.99 --------- * Add python3-paramiko to debian build dependencies (#68) Version 0.98 --------- * Add missing license/copyright headers Version 0.97 --------- * Update README Version 0.96 --------- * Code cleanup * Add MANIFEST.in: add additional files like dockerfile to source dist package. * Add LICENSE and Changelog file to rpm/debian package distribution configs. Version 0.95 --------- * virtnbdrestore: code cleanup * Add updated dockerfile based on work by Adrián Parilli to the repository. * Update README Version 0.94 --------- * virtnbdmap: use outputhelper for replay Version 0.93 --------- * Code cleanup Version 0.92 --------- * Code cleanup * virtnbdbackup: Relax check for empty target directory: now only fails if already an backup (partial or not) exists within the target directory. Version 0.91 --------- * Code cleanup Version 0.90 --------- * Code cleanup Version 0.89 --------- * virtnbdrestore: check if additional boot files such as nvram/kernel images exist and if not, restore them to the original path instead of just warning the user about manual steps being required. Version 0.88 --------- * virtnbdrestore: Allow restore into non-empty directories, so one can restore the disk files into an existing libvirt managed volume directory. If the restore target path is an libvirt managed pool, refresh the pool contents. (#67) * virtnbdrestore: dont continue with restore if -o dump is specified. Version 0.87 --------- * Code cleanup * Add support for remote backup via NBD+TLS (#66) * Update README Version 0.86 --------- * libvirt uri: both user and password parameters are mandatory if no qemu+ssh session is specified. Version 0.85 --------- * virtnbdrestore: check if target files exist during remote operation and fail accordingly. * Remote backup: use paramikos built in sftp client to copy files instead of third party scp module. Version 0.84 --------- * Add --nbd-ip option: can be used to bind nbd service for backup task to specific remote IP. Version 0.83 --------- * Code reorg * Fix remote restore: UnboundLocalError: local variable 'pid' referenced before assignment Version 0.82 --------- * Move common arguments to separate file. Version 0.81 --------- * Report PID and errors of commands executed on remote system. * Initiate ssh session for backup of boot config only if required. * Update README Version 0.80 --------- * Fix debug output * Add remote backup functionality (#65). Version 0.79 --------- * Add -U/--uri option: can be used to specify libvirt connection URI, if authfile is specified, use openAuth to authenticate against libvirt daemon. * Add --user and --password options: can be used to authenticate against libvirt daemon. * Update README, add notes about ovirt/rhev etc. Version 0.78 --------- * Code cleanup * Update README Version 0.77 --------- * Code cleanup * virtnbdbackup + detect if there is an active backup operation running and fail accordingly instead of running into "Cannot acquire state change lock" timeout exception. Version 0.76 --------- * virtnbdbackup: + Change some log messages: more detailed report on skipped devices during backup. + Add option --freeze-mountpoint: during backup, only filesystems of specified mountpoints are freezed and thawed. (#60) * virtnbdrestore: + Option -c would only adjust virtual machine config for the first disk. + Option -c now correctly removes excluded disks from the adjusted virtual machine configuration. * Add --version option for all utilities. * Vagrant scripts: test installation/execution after creating rpm packages. * Update README Version 0.75 --------- * virtnbdbackup: report amount of thawed/freezed filesystems during backup. * Update README Version 0.74 --------- * virtnbdbackup: add some more debug messages around freezing/thwaing filesystems and backup job operation. Version 0.73 --------- * virtnbdbackup: limit the amount of concurrent workers to the amount of disks detected, so users cannot specify an higher amount of workers than disks are attached to the virtual machine. Version 0.72 --------- * Code cleanup * Update README Version 0.71 --------- * Code cleanup * Update README Version 0.70 --------- * Code cleanup * Exit gracefully if setting up the logfile already fails. * virtnbdrestore: make option `-a restore` default, if output directory is named "dump" or `-a dump` is specified, stream information is dumped. * Update README, examples and tests accordingly Version 0.69 --------- * Code cleanup * virtnbdbackup: exit early if removing checkpoints fails during full backup. * virtnbdrestore: do not write zeroes during restore of image, as the resulting zeroed regions are then reported as being "data": this would result in further backups of the virtual machine to be thick provisioned, as even zeroed regions are saved. (#56) * virtnbdrestore: dont adjust vm config if option -c is missing. * virtnbdrestore: now removes existent backing stores if adjust setting is enabled: usually the case if virtual machine operated on snapshot during backup. Version 0.68 --------- * Code cleanup at different places. Make the output helper wrapping. * Use Elementree from lxml for easier access via xpath, adjust dependencies * Adjust pylintrc * virtnbdrestore: add option -c: adjusts required paths within restored virtual machine config so it can be defined using virsh. (#54) * virtnbdrestore: add option -D: in combination with option -c can be used to register virtual machine on libvirt daemon after restore. * virtnbdrestore: add option -N: redefine domain with specified name, if not passed, prefix "restore_" is added. Version 0.67 --------- * Disable xml based check if incremental backup is enabled for libvirt versions >= 7.6.0, the feature is enabled by default now: https://github.com/libvirt/libvirt/blob/master/NEWS.rst#v760-2021-08-02 * Fix dependencies for rpm package: nbdkit-plugin-python3 -> nbdkit-plugin-python * Fix documentation regards build on almalinux * virtnbdbackup: introduce backup mode "auto": automatically execute full backup if target folder is empty. If full backup exists in target folder, switch to incremental backup mode automatically. (#52) Version 0.66 --------- * virtnbdmap: use absolute path of files specified. * nbdkit plugin: add debug flag, be less verbose (#47) * Fix pylint warnings * Update README Version 0.65 --------- * virtnbdmap: open target device with O_DIRECT during replay of incremental data to speedup the process: no need to sync then. * virtnbdmap: remove nbdkit logfile after exiting successfully * virtnbdmap: use pidfile and send signal to correct process. (#46) * virtnbdmap: check if the passed device starts with /dev/nbd * virtnbdmap: code cleanup Version 0.64 --------- * virtnbdmap: minor code cleanup * virtnbdmap: add progressbar during replay of incremental backups * virtnbdmap: pass listen port to qemu-nbd command too. Version 0.63 --------- * virtnbdmap: update epilog examples, add logfile option * virtnbdmap: add --readonly option. * virtnbdmap: add --listen-port option: useful if one wants to map multiple backups on the same system concurrently. * virtnbdmap: better error handling if nbdkit process fails to start. * Update README Version 0.62 --------- * Add script to create virtualenv with required dependencies. * virtnbdmap now supports mounting of full->incremental (or differential) chains: the current approach is to use the nbdkit COW (copy on write) filter to replay the data blocks against the mapped nbd device. Version 0.61 --------- * Add vagrant scripts * Print libvirt library version. * Disable xml based check if incremental backup capability is enabled for libvirt versions > 8002000: feature is enabled by default. (#4) Version 0.60 --------- * Add python3-dataclasses to RPM dependencies. * Update README Version 0.59 --------- * Add epilog to help output with some example commands * Update README Version 0.58 --------- * Slight code improvements * Skip devices with type "floppy", just as "cdrom" devices. Version 0.57 --------- * Backup configured kernel/initrd images used for direct boot too. Version 0.56 --------- * Fix for Issue (#40): redefining checkpoints fails Version 0.55 --------- * Correct some wrongly caught exceptions * Update README Version 0.54 --------- * (#38) Backup virtual machines loader bios and nvram if defined, provide notice to user during restore that files must be copied manually. * virtnbdrestore: Fixes for --until option: + would not stop processing further files since --sequence option was introduced + would stop before actually processing specified checkpoint, now function stops after restoring data of checkpoint to be reached. + extend tests * Update README Version 0.53 --------- * Cleanup codebase * Fix differential backup for offline domains: apply same logic as during incremental backup. Version 0.52 --------- * Update help regarding --qemu option: works with diff/inc now too * Group possible options in help output * Introduce more exceptions, cleanup codebase. Version 0.51 --------- * Cleanup codebase, no functional changes. Version 0.50 --------- * Fix sequence restore: function returned too early. * Parameter for --sequence option does not require absolute path anymore. Version 0.49 --------- * virtnbdrestore now support restoring a manual sequence of data files passed by the --sequence option. Version 0.48 --------- * Internal code changes: start using proper exceptions if stream format is not parseable * virtnbdbrestore: add --sequence option, allows to specify a comma separated list of backup files whose information should be dumped. Next step is to implement restore of a specified sequence. Version 0.47 --------- * Fix issue #37: If an backup directory contains multiple differential or incremental backups, its not possible to use the --until option for point in time restore, because the differential backups defaulted to the first checkpoint name in the metadata header. Recover would stop too early in such cases.. Now the checkpoint name in the metadata header contains the same timestamp as the target files, which make it possible to use the --until option too. Version 0.46 --------- * Add changelog file Version 0.45 --------- * Adds differential backup option: backup option -l diff now saves the delta since the last incremental or full backup. Option -q now uses nbdinfo instead of qemu-img map, which supports reading specific bitmap information: now option -q can be used during incremental/differential backup too,which might be good for debugging issues. * If incremental backup for offline Domain is attempted, backup would save all data, not just dirty extents: now saves only dirty extents. * Extends the testsuite with tests for the implemented new features * Fixes various pylint warnings, better error handling in some situations. * Update documentation and executable dependencies for the debian package build. **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.44...v0.45 Version 0.44 --------- Ensure checkpoint chain consistency: As discussed during issue #33, the "parent" option to a checkpoint created is readonly currently. Thus, virtnbdbackup can not force an parent checkpoint during creation of the actual checkpoints. This means we cannot ensure that the complete checkpoint chain we are based on for the current incremental backup is entirely created by us, like so: > virsh checkpoint-list vm1 --tree > virtnbdbackup.0 r > | > +- virtnbdbackup.1 > | > +- virtnbdbackup.2 > | > +- not-our-checkpoint The delta for "not our checkpoint" would never be saved because we dont know about this checkpoint. Now virtnbdbackup checks for checkpoints which might have been created by users or third party applications and exits with error if incremental or full backup is attempted. **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.43...v0.44 Version 0.43 --------- Remove --checkpoint option: has never behaved the way it should. The parent option to a checkpoint XML definition is, according to the documentation read only. So currently its not supported to force a specific checkpoint using this option. **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.42...v0.43 Version 0.42 --------- * Fixes some pylint/flake8 warnings. **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.41...v0.42 Version 0.41 --------- Minor code changes regards checkpoint handling, fix some pylint warnings. **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.40...v0.41 Version 0.40 --------- Introduce custom handler for logging facility to track number of warnings during backup operation. Recent changes introduce a new logging facility handler that counts the number of warnings during backup (for example if qemu agent is not reachable etc..) If option "--strict" is set during backup, the exit code will now set to 2. This allows calling applications or scripts to have a more detailed error handling in regards to warnings and/or errors. **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.39...v0.40 Version 0.39 --------- Adds full support for offline domains: * Allows for incremental backup of offline domains: changed data for last checkpoint will be saved until a new checkpoint is created. * Adds proper error handling for cases where starting the NBD server for offline domains causes error **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.38...v0.39 Version 0.38 --------- Add support for backup of shutdown domains (#27) Usually virtnbdbackup makes most sense while operating on running domains. Only if a virtual domain is running, checkpoints can be defined via libvirt API and the required backup operations can be executed. The qemu Process will then start the NBD backend in order to receive the backup data. With the recent changes, virtnbdbackup now detects if the backup source is offline and then: * Forces the backup mode to "copy": no full or incremental backup supported because we can't create an checkpoint via libvirt API. Copy type backup is like a full backup just without checkpoint. * Forces the amount of workers to 1 * Starts an NBD Process via qemu-nbd for each disk, operating directly on the Virtual Disk files as detected from the VM configuration. * Queries required extents and data from the NBD Service and saves it accordingly. Full changeset: **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.36...v0.38 Version 0.36 --------- Fix for issue #28 Fix for issue #26 Only minor changes regards logging messages. **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.35...v0.36 Version 0.35 --------- **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.34...v0.35 Version 0.34 --------- Slight improvements to `virtnbdmap`. Show argument parameter default values in help outputs where useful. Add nbdkit and python plugin to rpm/debian package dependencies. **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.33...v0.34 Version 0.33 --------- Slight improvements for `virtnbdmap`: * add option to pass listening socket for nbdkit process * add option to specify export name, might be required on older versions Tests: * add test for `virtndbmap` (currently skipped on github because access to nbd devices not possible within docker containers.) Version 0.32 --------- Introduce new `virtnbdmap` utility to allow simple single file and instant recovery, see: https://github.com/abbbi/virtnbdbackup#single-file-restore-and-instant-recovery Version 0.31 --------- Add documentation and update for nbdkit plugin: Its now possible to create single file restores or do instant recovery by booting the virtual machine from the thin provisioned full backups. Version 0.30 --------- Various bugfixes, added some small features in `virtnbdrestore`: it now attempts to use the original disks name for the disk files and copies the virtual machine config file. Fixed a few pylint warnings as well. Updated readme and improved testsuite. **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.29...v0.30 Version 0.29 --------- Version adds support for concurrent backup of multiple disks: https://github.com/abbbi/virtnbdbackup#backup-concurrency **Full Changelog**: https://github.com/abbbi/virtnbdbackup/compare/v0.28...v0.29 Version 0.28 --------- Minor changes: dont attempt to re-create checkpoints during copy backup, changes in error handling. Version 0.27 --------- Minor bugfix: exit gracefully if backup to stdout is attempted in raw format: does not work with the new zip archive stream. Version 0.26 --------- Implement feature from issue #16: it is now possible to dump complete backup output to stdout. Version 0.25 --------- Fixes issue #17: Fails to restore missing checkpoints after the 11th backup Version 0.24 --------- * Small change for log format output, enclose date. * Code reformat. Version 0.23 --------- No real functional changes: * Reformat python code with python-black to have a common code format Version 0.22 --------- * Support compression with `--compress` option: blocks saved in stream are lz4 compressed, add test cases. * Fix checkpoint creation with `--include` option: do not create checkpoints for disks which are not part of include list * Various other small Bugfixes related to error handling * If backup is done to stdout, save virtual machine config to current folder. * Update README Version 0.21 --------- * Add support for backup of raw disk images during full backup using option `--raw`: Version 0.20 --------- * Add Support for backup of transient domains in cluster environments. * Update README Version 0.19 --------- * If libvirthelper failed to setup backup job because of exception, an already freezed guest filesystem was not thawed. * Fix tests. Version 0.18 --------- * During backup to stdout it was attempted to close non opened file handle resulting in backup error * Include used libnbd version in backup log * Fix some pylint warnings * Update README Version 0.17 --------- * Remove sh module from requirements, obsoleted by subprocess module Version 0.16 --------- * Code cleanup, fix pylint warnings * Update README Version 0.15 --------- * Minor code changes, move some functions to common class * Use makedirs for targetfolder to support nested target paths. * Updated README with more information about backup workflow. Version 0.14 --------- Code cleanups Change shebang so executables appear in process list with real name instead of python3 executable Version 0.13 --------- Write backup to partial file first In case backup utility receives signal or fails otherwise, the backup data is written to the regular target file and it is assumed everything is OK. Now virtnbdbackup writes the data to a partial file first and renames the file as last step. During incremental backup the target directory is checked for possible existence of partial backups and backup is aborted with error. Version 0.11 --------- Mostly code cleanup and pylint related warning fixes. Version 0.10 --------- * Allow multiple concurrent backups as NBD server is now connected via local unix domain socket instead of TCP, allowing unique socket file names * Remove dependency on sh module Version 0.8 --------- * Fix name in setup.py * Provide RPM package for download Version 0.7 --------- * Minor code changes, improved error and signal handling * Update README with common backup errors * Introduce and use __version__, show version in log and command output Version 0.5 --------- * Show progress during restore, be less verbose. Version 0.4 --------- * Add per disk progress bar Version 0.3 --------- * backup: now calls fsFreeze() and fsThaw() functions to ensure consistent filesystems during backup. Version 0.2 --------- * Fix exception in virtnbdrestore due to missing arguments * Scratchfile target file name is now more unique, not causing issues if multiple domains are backed up at the same time * Minor tweaks and improvements Version 0.1 --------- First release version with following features: * Supports Full/copy/inc backup of virtual machines * Skips disks and direct attached disks which do not support backup via changed block tracking * Creates logfile for each executed backup * Allows to manually exclude certain disks for backup ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/LICENSE0000644000000000000000000010451515033267156014055 0ustar00rootroot GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read . ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/MANIFEST.in0000644000000000000000000000013215033267156014574 0ustar00rootrootinclude LICENSE include Changelog include docker/* include man/* include requirements.txt ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4964876 virtnbdbackup-2.30/PKG-INFO0000644000000000000000000000062615033267565014147 0ustar00rootrootMetadata-Version: 2.1 Name: virtnbdbackup Version: 2.30 Summary: Backup utility for libvirt Home-page: https://github.com/abbbi/virtnbdbackup/ Author: Michael Ablassmeier Author-email: abi@grinser.de License: GPL Keywords: libnbd backup libvirt Provides-Extra: dev Provides-Extra: docs Provides-Extra: testing License-File: LICENSE Backup utility for libvirt, using latest changed block tracking features ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/README.md0000644000000000000000000014144215033267156014327 0ustar00rootroot![ci](https://github.com/abbbi/virtnbdbackup/actions/workflows/ci-ubuntu-latest.yml/badge.svg) [![package-build](https://github.com/abbbi/virtnbdbackup/actions/workflows/build.yml/badge.svg)](https://github.com/abbbi/virtnbdbackup/actions/workflows/build.yml) # virtnbdbackup Backup utility for `libvirt`, using the latest changed block tracking features. Create online, thin provisioned full and incremental or differential backups of your `kvm/qemu` virtual machines. ![Alt text](screenshot.jpg?raw=true "Title") - [About](#about) - [Prerequisites](#prerequisites) - [Libvirt versions <= 7.6.0 (Debian Bullseye, Ubuntu 20.x)](#libvirt-versions--760-debian-bullseye-ubuntu-20x) - [RHEL/Centos Stream, Alma, Rocky Linux](#rhelcentos-stream-alma-rocky-linux) - [Version <= 8.5](#version--85) - [Version >= 8.6](#version--86) - [Environment dependencies](#environment-dependencies) - [Installation](#installation) - [Python package](#python-package) - [RPM package](#rpm-package) - [Debian package](#debian-package) - [Virtualenv](#virtualenv) - [Docker images](#docker-images) - [Backup modes and concept](#backup-modes-and-concept) - [Supported disk formats / raw disks](#supported-disk-formats--raw-disks) - [Backup Examples](#backup-examples) - [Local full/incremental backup](#local-fullincremental-backup) - [Backing up offline virtual domains](#backing-up-offline-virtual-domains) - [Application consistent backups](#application-consistent-backups) - [Rotating backups](#rotating-backups) - [Excluding disks](#excluding-disks) - [Estimating differential/incremental backup size](#estimating-differentialincremental-backup-size) - [Backup threshold](#backup-threshold) - [Backup concurrency](#backup-concurrency) - [Compression](#compression) - [Remote Backup](#remote-backup) - [QEMU Sessions](#qemu-sessions) - [NBD with TLS (NBDSSL)](#nbd-with-tls-nbdssl) - [Using a separate network for data transfer](#using-a-separate-network-for-data-transfer) - [Piping data to other hosts](#piping-data-to-other-hosts) - [Kernel/initrd and additional files](#kernelinitrd-and-additional-files) - [Restore examples](#restore-examples) - [Dumping backup information](#dumping-backup-information) - [Verifying created backups](#verifying-created-backups) - [Complete restore](#complete-restore) - [Process only specific disks during restore](#process-only-specific-disks-during-restore) - [Point in time recovery](#point-in-time-recovery) - [Restoring with modified virtual machine config](#restoring-with-modified-virtual-machine-config) - [Remote Restore](#remote-restore) - [Post restore steps and considerations](#post-restore-steps-and-considerations) - [Single file restore and instant recovery](#single-file-restore-and-instant-recovery) - [Transient virtual machines: checkpoint persistency on clusters](#transient-virtual-machines-checkpoint-persistency-on-clusters) - [Supported Hypervisors](#supported-hypervisors) - [Ovirt, RHEV or OLVM](#ovirt-rhev-or-olvm) - [OpenNebula](#opennebula) - [Authentication](#authentication) - [Internals](#internals) - [Backup Format](#backup-format) - [Extents](#extents) - [Backup I/O and performance: scratch files](#backup-io-and-performance-scratch-files) - [Debugging](#debugging) - [FAQ](#faq) - [The thin provisioned backups are bigger than the original qcow images](#the-thin-provisioned-backups-are-bigger-than-the-original-qcow-images) - [Backup fails with "Cannot store dirty bitmaps in qcow2 v2 files"](#backup-fails-with-cannot-store-dirty-bitmaps-in-qcow2-v2-files) - [Backup fails with "unable to execute QEMU command 'transaction': Bitmap already exists"](#backup-fails-with-unable-to-execute-qemu-command-transaction-bitmap-already-exists) - [Backup fails with "Bitmap inconsistency detected: please cleanup checkpoints using virsh and execute new full backup"](#backup-fails-with-bitmap-inconsistency-detected-please-cleanup-checkpoints-using-virsh-and-execute-new-full-backup) - [Backup fails with "Error during checkpoint removal: [internal error: bitmap 'XX' not found in backing chain of 'XX']"](#backup-fails-with-error-during-checkpoint-removal-internal-error-bitmap-xx-not-found-in-backing-chain-of-xx) - [Backup fails with "Virtual machine does not support required backup features, please adjust virtual machine configuration."](#backup-fails-with-virtual-machine-does-not-support-required-backup-features-please-adjust-virtual-machine-configuration) - [Backup fails with "Timed out during operation: cannot acquire state change lock"](#backup-fails-with-timed-out-during-operation-cannot-acquire-state-change-lock) - [Backup fails with "Failed to bind socket to /var/tmp/virtnbdbackup.XX: Permission denied"](#backup-fails-with-failed-to-bind-socket-to-vartmpvirtnbdbackupxx-permission-denied) - [High memory usage during backup](#high-memory-usage-during-backup) - [fstrim and (incremental) backup sizes](#fstrim-and-incremental-backup-sizes) - [Test your backups!](#test-your-backups) - [Links](#links) # About Existing backup solutions or scripts for `libvirt/kvm` usually depend on the external snapshot feature to create backups, sometimes even require to shutdown or pause the virtual machine. Recent additions to both the `libvirt` and `qemu` projects have introduced new capabilities that allow to create online (full and incremental) backups, by using so called `dirty bitmaps` (or changed block tracking). `virtnbdbackup` uses these features to create online full and incremental or differential backups. `virtnbdrestore` can be used to re-construct the complete image from the thin provisioned backups. `virtnbdmap` can be used to map an thin provisioned backup image into a block device on-the-fly, for easy single file restore or even instant boot from an backup image. For backing up standalone qemu virtual machines not managed by libvirt, see this project: [qmpbackup](https://github.com/abbbi/qmpbackup) # Prerequisites Obviously you require a libvirt/qemu version that supports the incremental backup features. Since libvirt v7.6.0 and qemu-6.1 the required features are [enabled by default](https://libvirt.org/news.html#v7-6-0-2021-08-02) and are considered production ready: everything will work out of the box. Following, you will find a short overview which older libvirt versions may require further adjustments to the virtual machine config. ## Libvirt versions <= 7.6.0 (Debian Bullseye, Ubuntu 20.x) If you are using Debian Bullseye or Ubuntu 20.x, the included libvirt version already has an almost complete support for incremental backup, although it doesn't work properly with migration or some block jobs. If you don't want to use migration or other blockjobs you can enable the incremental backup feature on these libvirt versions. Change the virtual machine config using `virsh edit ` like so: (the first line must be changed, too!): ``` [..] [..] ``` `Note`: > You must power cycle the virtual machine after enabling the feature! > Upstream libvirt strongly discourages enabling the feature on production > systems for these libvirt versions. ## RHEL/Centos Stream, Alma, Rocky Linux ### Version <= 8.5 Up to RHEL/Centos8/Almalinux 8.5, libvirt packages from the advanced virtualization stream support all required features. To install libvirt from the stream use: ``` yum install centos-release-advanced-virtualization yum makecache yum module install virt ``` and enable the feature by adjusting the virtual machine config. ### Version >= 8.6 As of RHEL 8.6, the advanced virtualization stream has been deprecated, and all components supporting the new feature are included in the virt:rhel module, the feature is enabled by default. [(Details)](https://access.redhat.com/solutions/6959344) ## Environment dependencies * python libvirt module version >= 6.0.0 (yum install python3-libvirt) * python libnbd bindings (https://github.com/libguestfs/libnbd) version >= `1.5.5` (yum install python3-libnbd) * The virtual machine should use qcow version 3 images to support the full feature set. # Installation There are several ways to install the utility, below you will find an short description for each of them. For Debian and RHEL/SuSE based derivates see [releases](https://github.com/abbbi/virtnbdbackup/releases) for pre-built packages. `Note`: > Please consider to check [past issues related to > installation](https://github.com/abbbi/virtnbdbackup/issues?q=is%3Aissue+is%3Aclosed+label%3Ainstallation) > if you face any troubles before opening a new issue. ## Python package ``` git clone https://github.com/abbbi/virtnbdbackup && cd virtnbdbackup pip install . ``` `Note`: > Do not install the "nbd" package available on PyPI, it does not provide the > required nbd bindings (unfortunately has the same name). You have to > additionally install the provided python3-libnbd packages by your > distribution, or compile the libnbd bindings by yourself. ## RPM package Packages for RHEL/Fedora and OpenSUSE are available via [releases](https://github.com/abbbi/virtnbdbackup/releases). To create an RPM package from source by yourself you can follow the steps from the github [build workflow](https://github.com/abbbi/virtnbdbackup/actions/workflows/build.yml). ## Debian package Official packages are available: [https://packages.debian.org/virtnbdbackup](http://packages.debian.org/virtnbdbackup) and are maintained on the [Debian salsa codespace](https://salsa.debian.org/debian/virtnbdbackup). For the latest packages available check [releases](https://github.com/abbbi/virtnbdbackup/releases). To create an Debian package from source by yourself you can follow the steps from the github [build workflow](https://github.com/abbbi/virtnbdbackup/actions/workflows/build.yml). ## Virtualenv For setup within an virtualenv see [venv scripts](venv/). ## Docker images You can build an docker image using the existing [Dockerfile README](docker/) All released versions and master branch are published via github container registry, too. Example: ``` docker run -it ghcr.io/abbbi/virtnbdbackup:master virtnbdbackup ``` See [packages](https://github.com/abbbi/virtnbdbackup/pkgs/container/virtnbdbackup). # Backup modes and concept Following backup modes can be used: * `auto`: If the target folder is empty, attempt to execute full backup, otherwise switch to backup mode incremental: allows rotation of backup into monthly folders. * `full`: Full, thin provisioned backup of the virtual machine, a new checkpoint named `virtnbdbackup` will be created, all existent checkpoints from prior backups matching this name will be removed: a new backup chain is created. The Virtual machine must be online and running for this backup mode to work. * `copy`: Full, thin provisioned backup of the virtual machine disks, no checkpoint is created for further incremental backups, existing checkpoints will be left untouched. This is the default mode and works with qcow images not supporting persistent bitmaps. * `inc`: Perform incremental backup, based on the last full or incremental backup. A checkpoint for each incremental backup is created and saved. * `diff`: Perform differential backup: saves the current delta to the last incremental or full backup. All required information for restore is stored to the same directory, including the latest virtual machine configuration, checkpoint information, disk data and logfiles. The target directory must be rotated if a new backup set is created. If the virtual domain is active and running, a backup job operation via `libvirt api` is started, which in turn initializes a new nbd server backend listening on a local unix socket. This nbd backend provides consistent access to the virtual machines, disk data and dirty blocks. After the backup process finishes, the job is stopped and the nbd server quits operation. It is possible to backup multiple virtual machines on the same host system at the same time, using separate calls to the application with a different target directory to store the data. # Supported disk formats / raw disks `libvirt/qemu` supports dirty bitmaps, required for incremental backups only with qcow(v3) based disk images. If you are using older image versions, you can only create `copy` backups, or consider converting the images to a newer format using `qemu-img`: > qemu-img convert -O qcow2 -o compat=1.1 disk-old.qcow2 disk.qcow2 By default `virtnbdbackup` will exclude all disks with format `raw` as well as direct attached (passthrough) disks such as LVM or ZVOL and ISCSI volumes. These type of virtual disks do not support storing checkpoint/bitmap metadata and do not support incremental/differential backup. [(more info)](https://patchew.org/QEMU/20210320093235.461485-1-pj@patrikjanousek.cz/) This behavior can be changed if option `--raw` is specified, raw disks will then be included during a `full` backup. This of course means that no thin provisioned backup is created for these particular disks. During restore, these files can be copied "as is" from the backup folder and must not be processed using `virtnbdrestore`. `Note:` > The backup data for raw disks will only be crash consistent, be aware > that this might result in inconsistent filesystems after restoring! # Backup Examples Each backup for a virtual machine must be saved to an individual target directory. Once the target directory includes an full backup, it can be used as base for further incremental or differential backups. ## Local full/incremental backup Start full backup of domain `vm1`, save data to `/tmp/backupset/vm1`: ``` virtnbdbackup -d vm1 -l full -o /tmp/backupset/vm1 ``` Start incremental backup for domain `vm1`, backup only changed blocks to the last full backup, the same directory is used as backup target: ``` virtnbdbackup -d vm1 -l inc -o /tmp/backupset/vm1 ``` The resulting directory will contain both backups and all other files required to restore the virtual machine. Created logfiles can be used for analyzing backup issues: ``` /tmp/backupset/vm1 ├── backup.full.05102021161752.log ├── backup.inc.05102021161813.log ├── checkpoints │   ├── virtnbdbackup.0.xml │   ├── virtnbdbackup.1.xml ├── sda.full.data ├── sda.inc.virtnbdbackup.1.data ├── vm1.cpt ├── vmconfig.virtnbdbackup.0.xml ├── vmconfig.virtnbdbackup.1.xml ``` ## Backing up offline virtual domains If the virtual domain is not in running state (powered off) `virtnbdbackup` supports `copy` and `inc/diff` backup modes. Incremental and differential backups will then save the changed blocks since last created checkpoint. Backup mode `full` is changed to mode `copy`, because libvirt does not allow to create checkpoints for offline domains. This behavior can be changed using the `-S` (`--start-domain`) option: prior to executing the backup, the virtual domain will then be started in `paused` state for the time the backup is created: The virtual machines CPU's are halted, but the running QEMU Process will allow all operations required to execute backups. Using this option will allow for all range of backup types (full/diff/inc) and makes most sense if used with the backup mode `auto`. Also, the option won't alter the virtual domain state if it is already online, thus it can be used for backing up virtual machines whose state is unknown prior to backup. ## Application consistent backups During backup `virtnbdbackup` attempts to freeze all file systems within the domain using the qemu guest agent filesystem freeze and thaw functions. In case no qemu agent is installed or filesystem freeze fails, a warning is shown during backup: ``` WARNING [..] Guest agent is not responding: QEMU guest agent is not connected ``` In case you receive this warning, check if the qemu agent is installed and running in the domain. It is also possible to specify one or multiple mountpoints used within the virtual machine to freeze only specific filesystems, like so: `virtnbdbackup -d vm1 -l inc -o /tmp/backupset/vm1 -F /mnt,/var` this way only the underlying filesystems on */mnt* and */var* are frozen and thawed. `Note:` > It is highly recommended to have an qemu agent running in the virtual > domain to ensure file system consistency during backup! ## Rotating backups With backup mode `auto` it is possible to have a monthly rotation/retention. If the target folder is empty, backup mode auto will create an full backup. On the following executions, it will automatically switch to backup mode incremental, if the target folder already includes an full backup. Example: ``` virtnbdbackup -d vm1 -l auto -o /tmp/2022-06 -> creates full backup virtnbdbackup -d vm1 -l auto -o /tmp/2022-06 -> creates inc backup virtnbdbackup -d vm1 -l auto -o /tmp/2022-06 -> creates inc backup virtnbdbackup -d vm1 -l auto -o /tmp/2022-07 -> creates full backup virtnbdbackup -d vm1 -l auto -o /tmp/2022-07 -> creates inc backup ``` ## Excluding disks Option `-x` can be used to exclude certain disks from the backup. The name of the disk to be excluded must match the disks target device name as configured in the domains xml definition, for example: ``` virtnbdbackup -d vm1 -l full -o /tmp/backupset/vm1 -x sda ``` Special devices such as `cdrom/floppy` or `direct attached luns` are excluded by default, as they are not supported by the changed block tracking layer. It is also possible to only backup specific disks using the include option (`--include`, or `-i`): ``` virtnbdbackup -d vm1 -l full -o /tmp/backupset/vm1 -i sdf ``` ## Estimating differential/incremental backup size Sometimes it can be useful to estimate the data size prior to executing the next `incremental` or `differential` backup. This can be done by using the option `-p` which will query the virtual machine checkpoint information for the current size: ``` virtnbdbackup -d vm1 -l inc -o /tmp/backupset/vm1 -p [..] [..] INFO virtnbdbackup - handleCheckpoints [MainThread]: Using checkpoint name: [virtnbdbackup.1]. [..] INFO virtnbdbackup - main [MainThread]: Estimated checkpoint backup size: [24248320] Bytes ``` `Note:` > Not all libvirt versions support the flag required to read the checkpoint > size. If the estimated checkpoint size is always 0, your libvirt version > might miss the required features. ## Backup threshold If an `incremental` or `differential` backup is attempted and the virtual machine is active, it is possible to specify an threshold for executing the backup using the `--threshold` option. The backup will then only be executed if the amount of data changed meets the specified threshold (in bytes): ``` virtnbdbackup -d vm1 -l inc -o /tmp/backupset/vm1 --threshold 3311264 [..] [..] INFO virtnbdbackup - handleCheckpoints [MainThread]: Using checkpoint name: [virtnbdbackup.1]. [..] ]virtnbdbackup - main [MainThread]: Backup size [3211264] does not meet required threshold [3311264], skipping backup. ``` ## Backup concurrency If `virtnbdbackup` saves data to a regular target directory, it starts one thread for each disk it detects to speed up the backup operation. This behavior can be changed using the `--worker` option to define an amount of threads to be used for backup. Depending on how many disks your virtual machine has attached, it might make sense to try a different amount of workers to see which amount your hardware can handle best. If standard output (`-`) is defined as backup target, the amount of workers is always limited to 1, to ensure a valid Zip file format. ## Compression It is possible to enable compression for the `stream` format via `lz4` algorithm by using the `--compress` option. The saved data is compressed inline and the saveset file is appended with compression trailer including information about the compressed block offsets. By default compression level `2` is set if no parameter is applied. Higher compression levels can be set via: `--compress=16` During the restore, `virtnbdrestore` will automatically detect such compressed backup streams and attempts to decompress saved blocks accordingly. Using compression will come with some CPU overhead, both lz4 checksums for block and original data are enabled. ## Remote Backup It is also possible to backup remote libvirt systems. The most convenient way is to use ssh for initiating the libvirt connection (key authentication mandatory). Before attempting an remote backup, please validate your environment meets the following criteria: * DNS resolution (forward and reverse) must work on all involved systems. * SSH Login to the remote system via ssh key authentication (using ssh agent or passwordless ssh key) should work without issues. * Unique hostnames must be set on all systems involved. ([background](https://github.com/abbbi/virtnbdbackup/issues/117)) * Firewall must allow connection on all ports involved. If the virtual machine has additional files configured, as described in [Kernel/initrd and additional files](#kernelinitrd-and-additional-files), these files will be copied from the remote system via SSH(SFTP). ### QEMU Sessions In order to backup virtual machines from a remote host, you must specify an [libvirt URI](https://libvirt.org/uri.html) to the remote system. The following example saves the virtual machine `vm1` from the remote libvirt host `hypervisor` to the local directory `/tmp/backupset/vm1`, it uses the `root` user for both the libvirt and ssh authentication: ``` virtnbdbackup -U qemu+ssh://root@hypervisor/system --ssh-user root -d vm1 -o /tmp/backupset/vm1 ``` See also: [Authentication](#authentication) `Note`: > If you want to run multiple remote backups at the same time you need to pass > an unique port for the NBD service used for data transfer via --nbd-port > option for each backup session. ### NBD with TLS (NBDSSL) By default disk data received from a remote system will be transferred via regular NBD protocol. You can enable TLS for this connection, using the `--tls` option. Before being able to use TLS, you *must* configure the required certificates on both sides. [See this script](https://github.com/abbbi/virtnbdbackup/blob/master/scripts/create-cert.sh). See the following documentation by the libvirt project for detailed instructions how setup: https://wiki.libvirt.org/page/TLSCreateCACert `Note:` > You should have installed at least version 1.12.6 of the libnbd library > which makes the transfer via NBDS more stable [full background](https://github.com/abbbi/virtnbdbackup/issues/66#issuecomment-1196813750) ### Using a separate network for data transfer In case you want to use a dedicated network for the data transfer via NBD, you can specify an specific IP address to bind the remote NBD service to via `--nbd-ip` option. ### Piping data to other hosts If the output target points to standard out (`-`), `virtnbdbackup` puts the resulting backup data into an uncompessed zip archive. A such, it is possible to transfer the backup data to different hosts, or pipe it to other programs. However, keep in mind that in case you want to perform incremental backups, you must keep the checkpoint files on the host you are executing the backup utility from, until you create another full backup. If output is set to standard out, `virtnbdbackup` will create the required checkpoint files in the directory it is executed from. Here is an example: ``` # mkdir backup-weekly; cd backup-weekly # virtnbdbackup -d vm1 -l full -o - | ssh root@remotehost 'cat > backup-full.zip' # [..] # INFO outputhelper - __init__: Writing zip file stream to stdout # [..] # INFO virtnbdbackup - main: Finished # INFO virtnbdbackup - main: Adding vm config to zipfile # [..] ``` Any subsequent incremental backup operations must be called from within this directory: ``` # cd backup-weekly # virtnbdbackup -d vm1 -l inc -o - | ssh root@remotehost 'cat > backup-inc1.zip' [..] ``` You may consider adding the created checkpoint files to some VCS system, like git, to have some kind of central backup history tracking. During restore unzip the data from both zip files into a single directory: (use `virtnbdrestore` to reconstruct the virtual machine images): ``` # unzip -o -d restoredata backup-full.zip # unzip -o -d restoredata backup-inc1.zip ``` ## Kernel/initrd and additional files If an domain has configured custom kernel, initrd, loader or nvram images (usually the case if the domain boots from OVM UEFI BIOS), these files will be saved to the backup folder as well. # Restore examples For restoring, `virtnbdrestore` can be used. It reconstructs the streamed backup format back into a usable qemu qcow image. The restore process will create a qcow image with the original virtual size. In a second step, the qcow image is then mapped to a ndb server instance where all data blocks are sent to and are applied accordingly. The resulting image can be mounted (using `guestmount`) or attached to a running virtual machine in order to recover required files. ## Dumping backup information As a first start, the `dump` parameter can be used to dump the saveset information of an existing backup: ``` virtnbdrestore -i /tmp/backupset/vm1 -o dump INFO:root:Dumping saveset meta information {'checkpointName': 'virtnbdbackup', 'dataSize': 704643072, 'date': '2020-11-15T20:50:36.448938', 'diskName': 'sda', 'incremental': False, 'parentCheckpoint': False, 'streamVersion': 1, 'virtualSize': 32212254720} [..] ``` The output includes information about the thick and thin provisioned disk space that is required for recovery, date of the backup and checkpoint chain. ## Verifying created backups As with version >= 1.9.40 `virtnbdbackup` creates an check sum for each created data file. Using `virtnbdrestore` you can check the integrity for the created data files without having to restore: ``` virtnbdrestore -i /tmp/backup/vm1 -o verify [..] INFO lib common - printVersion [MainThread]: Version: 1.9.39 Arguments: ./virtnbdrestore -i /tmp/backup/vm1 -o verify [..] INFO root virtnbdrestore - verify [MainThread]: Computing checksum for: /tmp/backup/vm1/sda.full.data [..] INFO root virtnbdrestore - verify [MainThread]: Checksum result: 541406837 [..] INFO root virtnbdrestore - verify [MainThread]: Comparing checksum with stored information [..] INFO root virtnbdrestore - verify [MainThread]: OK ``` this makes it easier to spot corrupted backup files due to storage issues. ([background](https://github.com/abbbi/virtnbdbackup/issues/134)) ## Complete restore To restore all disks within the backupset into a usable qcow image use command: ``` virtnbdrestore -i /tmp/backupset/vm1 -o /tmp/restore ``` All incremental backups found will be applied to the target images in the output directory `/tmp/restore` `Note`: > The restore utility will copy the latest virtual machine config to the > target directory, but won't alter its contents. You have to adjust the config > file for the new paths and/or excluded disks to be able to define and run it. `Note`: > Created disk images will be thin provisioned by default, you can change this > behavior using option `--preallocate` to create thick provisioned images. ## Process only specific disks during restore A single disk can be restored by using the option `-d`, the disk name has to match the virtual disks target name, for example: ``` virtnbdrestore -i /tmp/backupset/vm1 -o /tmp/restore -d sda ``` ## Point in time recovery Option `--until` allows to perform a point in time restore up to the desired checkpoint. The checkpoint name has to be specified as reported by the dump output (field `checkpointName`), for example: ``` virtnbdrestore -i /tmp/backupset/vm1 -o /tmp/restore --until virtnbdbackup.2 ``` It is also possible to specify the source data files specifically used for the rollback via `--sequence` option, but beware: you must be sure the sequence you apply has the right order, otherwise the restored image might be errnous, example: ``` virtnbdrestore -i /tmp/backupset/vm1 -o /tmp/restore --sequence vdb.full.data,vdb.inc.virtnbdbackup.1.data ``` ## Restoring with modified virtual machine config Option `-c` can be used to adjust the virtual machine configuration during restore accordingly, the following changes are done: * UUID of the virtual machine is removed from the config file * Name of the virtual machine is prefixed with "restore_" (use option `--name` to specify desired vm name) * The disk paths to the virtual machine are changed to the new target directory. * If virtual machine was operating on snapshots/backing store images, the references to the configured backing stores will be removed. * Raw devices are removed from VM config if `--raw` is not specified, as well as floppy or cdrom devices (which aren't part of the backup). `Note:` > If missing, Kernel, UEFI or NVRAM files are restored to their original > location as set in the virtual machine configuration. A restored virtual machine can then be defined and started right from the restored directory (or use option `-D` to define automatically): ``` virtnbdrestore -c -i /tmp/backupset/vm1 -o /tmp/restore [..] [..] INFO virtnbdrestore - restoreConfig [MainThread]: Adjusted config placed in: [/tmp/restore/vmconfig.virtnbdbackup.0.xml] [..] INFO virtnbdrestore - restoreConfig [MainThread]: Use 'virsh define /tmp/restore/vmconfig.virtnbdbackup.0.xml' to define VM ``` ## Remote Restore Restoring to a remote host is possible too, same options as during backup apply. The following example will restore the virtual machine from the local directory `/tmp/backupset` to the remote system "hypervisor", alter its configuration and register the virtual machine: ``` virtnbdrestore -U qemu+ssh://root@hypervisor/system --ssh-user root -cD -i /tmp/backupset/vm1 -o /remote/target ``` # Post restore steps and considerations If you restore the virtual machine with its original name on the same hypervisor, you may have to cleanup checkpoint information, otherwise backing up the restored virtual machine may fail, see [this discussion](https://github.com/abbbi/virtnbdbackup/discussions/48) # Single file restore and instant recovery The `virtnbdmap` utility can be used to map uncompressed backup images from the stream format into an accessible block device on the fly. This way, you can restore single files or even boot from an existing backup image without having to restore the complete dataset. The utility requires `nbdkit with the python plugin` to be installed on the system along with required qemu tools (`qemu-nbd`) and an loaded nbd kernel module. It must be executed with superuser (root) rights or via sudo. The following example maps an existing backup image to the network block device `/dev/nbd0`: ``` # modprobe nbd max_partitions=15 # virtnbdmap -f /backupset/vm1/sda.full.data [..] INFO virtnbdmap - [MainThread]: Done mapping backup image to [/dev/nbd0] [..] INFO virtnbdmap - [MainThread]: Press CTRL+C to disconnect ``` While the process is running, you can access the backup image like a regular block device: ``` fdisk -l /dev/nbd0 Disk /dev/nbd0: 2 GiB, 2147483648 bytes, 4194304 sectors ``` You can also create an mapped "point in time" recovery image by passing a sequence of full and incremental backups as parameter. The changes from the incremental backups will then be replayed to the block device on the fly and the image will represent the latest state: ``` virtnbdmap -f /backupset/vm1/sda.full.data,/backupset/vm1/sda.inc.virtnbdbackup.1.data,/backupset/vm1/sda.inc.virtnbdbackup.2.data [..] [..] INFO virtnbdmap - main [MainThread]: Need to replay incremental backups [..] INFO virtnbdmap - main [MainThread]: Replaying offset 420 from /backup/sda.inc.virtnbdbackup.1.data [..] INFO virtnbdmap - main [MainThread]: Replaying offset 131534 from /backup/sda.inc.virtnbdbackup.1.data [..] [..] INFO virtnbdmap - main [MainThread]: Replaying offset 33534 from /backup/sda.inc.virtnbdbackup.2.data [..] INFO virtnbdmap - [MainThread]: Done mapping backup image to [/dev/nbd0] [..] INFO virtnbdmap - [MainThread]: Press CTRL+C to disconnect [..] ``` The original image will be left untouched as nbdkits copy on write filter is used to replay the changes. Further you can create an overlay image via `qemu-img` and boot from it right away (or boot directly from the /dev/nbd0 device). ``` qemu-img create -b /dev/nbd0 -f qcow2 bootme.qcow2 qemu-system-x86_64 -enable-kvm -m 2000 -hda bootme.qcow2 ``` To remove the mappings, stop the utility via "CTRL-C" `Note`: > If the virtual machine includes volume groups, the system will attempt to > set them online as you create the mapping, because the copy on write device > is writable by default. > If your host system is using the same volume group names this could lead to > issues (check `dmesg` or `journalctl` then). > In case the volume groups are online, it is recommended to change them to > offline just before you remove the mapping, to free all references to the > mapped nbd device (`vgchange -a n `) `Note`: > If you map the image device with the `--readonly` option you may need to pass > certain options to the mount command (-o norecovery,ro) in order to be able > to mount the filesystems. This may also be the case if no qemu agent was > installed within the virtual machine during backup. # Transient virtual machines: checkpoint persistency on clusters In case virtual machines are started in transient environments, such as using cluster solutions like `pacemaker` situations can appear where the checkpoints for the virtual machine defined by libvirt are not in sync with the bitmap information in the qcow files. In case libvirt creates a checkpoint, the checkpoint information is stored in two places: * var/lib/libvirt/qemu/checkpoint/ * In the bitmap file of the virtual machines qcow image. Depending on the cluster solution, in case virtual machines are destroyed on host A and are re-defined on host B, libvirt loses the information about those checkpoints. Unfortunately `libvirtd` scans the checkpoint only once during startup. This can result in a situation, where the bitmap is still defined in the qcow image, but libvirt doesn't know about the checkpoint, backup then fails with: `Unable to execute QEMU command 'transaction': Bitmap already exists` By default `virtnbdbackup` attempts to store the checkpoint information in the default backup directory, in situations where it detects a checkpoint is missing, it attempts to redefine them from the prior backups. In order to store the checkpoint information at some central place the option `--checkpointdir` can be used, this allows having persistent checkpoints stored across multiple nodes: As example: 1) Create backup on host A, store checkpoints in a shared directory between hosts in `/mnt/shared/vm1`: `virtnbdbackup -d vm1 -l full -o /tmp/backup_hosta --checkpointdir /mnt/shared/vm1` 2) After backup, the virtual machine is relocated to host B and loses its information about checkpoints and bitmaps, thus, the next full backup usually fails with: ``` virtnbdbackup -d vm1 -l full -o /tmp/backup_hostb [..] unable to execute QEMU command 'transaction': Bitmap already exists: virtnbdbackup.0 ``` 3) Now pass the checkpoint dir and files written from host A, and virtnbdbackup will redefine missing checkpoints and execute a new full backup. As the new full backup removes all prior checkpoints the bitmap information is in sync after this operation and backup succeeds: ``` virtnbdbackup -d vm1 -l full -o /tmp/backup_hostb --checkpointdir /mnt/shared/vm1 [..] redefineCheckpoints: Redefine missing checkpoint virtnbdbackup.0 [..] ``` See also: https://github.com/abbbi/virtnbdbackup/pull/10 # Supported Hypervisors `virtnbdbackup` uses the lowest layer on top of libvirt to allow its functionality, you can also use it with more advanced hypervisors solutions such as [ovirt](https://www.ovirt.org/), RHEV or OpenNebula, but please bear in mind that it was not developed to target all of those solutions specifically! ## Ovirt, RHEV or OLVM If you are using the ovirt node based hypervisor hosts you should consider creating a virtualenv via the [venv scripts](venv/) and transferring it to the node system. On regular centos/alma/rhel based nodes, installation via RPM package should be preferred. The incremental backup functionality can be enabled via ovirt management interface. Usually ovirt restricts access to the libvirt daemon via different authentication methods. Use the `-U` parameter in order to specify an authentication file, if you chose to run the utility locally on the hypervisor: ``` virtnbdbackup -U qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf -d vm1 -o /tmp/backupset/vm1 ``` You can also use remote backup functionality: * System must be reachable via ssh public key auth as described in the [Remote Backup](#remote-backup) section. * Some OVIRT based setups may deny SASL based authentication if the hostname used to connect to does not match the hostname from the libvirt certificate. [more info](https://github.com/abbbi/virtnbdbackup/issues/167#issuecomment-2028467071) * Firewall port for NBD must be open: ``` root@hv-node~# firewall-cmd --zone=public --add-port=10809/tcp ``` and then backup via: ``` virtnbdbackup -U qemu+ssh://root@hv-node/session -d vm -o /backup --password password --user root --ssh-user root ``` `Note:` > `virtnbdrestore` has not been adopted to cope with the ovirt specific > domain xml format, so redefining and virtual machine on the node might not > work. ## OpenNebula See [past issues](https://github.com/abbbi/virtnbdbackup/issues?q=label%3Aopennebula) # Authentication Both `virtnbdbackup` and `virtnbdrestore` commands support authenticating against libvirtd with the usual URIs. Consider using the following options: `-U`: Specify an arbitrary connection URI to use against libvirt `--user`: Username to use for the specified connection URI `--password`: Password to use for the specified connection URI. It is also possible to specify the credentials stored as authentication file like it would be possible using the `virsh -c` option: ``` -U qemu:///system?authfile=/etc/virsh_auth.conf .. ``` `Note:` > The default connection URI used is `qemu:///system` which is usually the > case if virtual machines operate as root user. Use the `qemu:///session` URI > to backup virtual machines as regular user. # Internals ## Backup Format Currently, there are two output formats implemented: * `stream`: the resulting backup image is saved in a streamlined format, where the backup file consists of metadata about offsets and lengths of zeroed or allocated contents of the virtual machines disk. This is the default. The resulting backup image is thin provisioned. * `raw`: The resulting backup image will be a full provisioned raw image, this should mostly be used for debugging any problems with the extent handler, it won't work with incremental backups. ## Extents In order to save only used data from the images, dirty blocks are queried from the NBD server. The behavior can be changed by using the option `-q` to use common qemu tools (nbdinfo). By default `virtnbdbackup` uses a custom implemented extent handler. ## Backup I/O and performance: scratch files If virtual domains handle heavy I/O load during backup (such as writing or deleting lots of data while the backup is active) you might consider using the `--scratchdir` option to change the default scratch file location. During the backup operation qemu will use the created scratch files for fleecing, thus it is recommended to store these files on storage that meets the same I/O performance requirements as the backup target. The free space on the default scratch directory (`/var/tmp`) must be enough to be able to keep all fleecing data while the backup is active. ## Debugging To get more detailed debug output use `--verbose` option. To enable NBD specific debugging output export LIBNBD_DEBUG environment variable prior to executing the backup or restore: ``` export LIBNBD_DEBUG=1 virtnbdbackup [..] --verbose ``` # FAQ ## The thin provisioned backups are bigger than the original qcow images Virtual machines using the qcow format do compress data. During backup, the image contents are exposed as NBD device which is a RAW device. The backup data will be at least as big as the used data within the virtual machine. You can use the `--compress` option or other tools to compress the backup images in order to save storage space or consider using a deduplication capable target file system. ## Backup fails with "Cannot store dirty bitmaps in qcow2 v2 files" If the backup fails with error: ``` ERROR [..] internal error: unable to execute QEMU command dirty bitmaps in qcow2 v2 files ``` consider migrating your qcow files to version 3 format. QEMU qcow image version 2 does not support storing advanced bitmap information, as such only backup mode `copy` is supported. ## Backup fails with "unable to execute QEMU command 'transaction': Bitmap already exists" During backup `virtnbdbackup` creates a so called "checkpoint" using the libvirt API. This checkpoint represents an "bitmap" that is saved in the virtual machines disk image. If you receive this error during backup, there is an inconsistency between the checkpoints that the libvirt daemon thinks exist, and the bitmaps that are stored in the disk image. This inconsistency can be caused by several situations: 1) A virtual machine is operated on a cluster and is migrated between host systems (See also: [Transient virtual machines: checkpoint persistency on clusters](#transient-virtual-machines-checkpoint-persistency-on-clusters)) 2) A change to the libvirt environment between backups (such as re-installing the libvirt daemon) caused the system to lose track of the existing checkpoints, but the bitmaps are still existent in the disk files. 3) Between backups, the disk image contents were reset and now the image has already defined bitmaps (if disk was restored from a storage snapshot, for example ) 4) `virtnbdbackup` is started on an backup target directory with an old state and starts from a wrong checkpoint count, now attempting to create an checkpoint whose bitmap already exists (might happen if you rotate backup directories and pick the wrong target directory with an older state for some reason) To troubleshoot this situation, use virsh to list the checkpoints that libvirt thinks are existent using: ``` virsh checkpoint-list Name Creation Time ---------------------------------------------- virtnbdbackup.0 2024-08-01 20:45:44 +0200 ``` You can also check which disks were included in the checkpoint: ``` virsh checkpoint-dumpxml vm1 virtnbdbackup.0 | grep " ``` The example command shows one existing checkpoint for disk "sda". An bitmap with the same name should be listed using the `qemu-img` tool for each checkpoint. Note: > Bitmap information is written into the qcow2 metadata only once qemu will > close the image. As such you need to turn off the virtual machine prior to > checking the bitmaps. To list the bitmaps use: ``` virsh destroy vm1 # shutdown vm virsh domblklist vm1 | grep sda sda /tmp/tmp.Y2PskFFeVv/vm1-sda.qcow2 qemu-img info /tmp/tmp.Y2PskFFeVv/vm1-sda.qcow2 [..] bitmaps: [0]: flags: [0]: auto name: virtnbdbackup.1 granularity: 65536 [..] ``` Now, compare which checkpoints are listed and which bitmaps exist in the qcow image. In this example `virsh` only lists the checkpoint "virtnbdbackup.0" but the bitmap is called "virtnbdbackup.1", indicating there is an inconsistency. Remove the dangling bitmap(s) via: ``` qemu-img bitmap /tmp/tmp.Y2PskFFeVv/vm1-sda.qcow2 --remove virtnbdbackup.1 ``` Start with an new full backup to a fresh directory. ## Backup fails with "Bitmap inconsistency detected: please cleanup checkpoints using virsh and execute new full backup" If an qemu process for a virtual machine is forcefully shutdown after a backup (for example due to power outage or qemu process killed/crashed) the bitmaps required for further backups may not have yet been synced to the qcow image. In these cases, you need to delete the existent checkpoints using: ``` virsh checkpoint-delete --checkpointname --metadata ``` and start with a fresh full backup. `Note`: > Since version 2.30 virtnbdbackup will handle these situations automatically > and falls back to a full backup in case it detects a broken bitmap chain. ## Backup fails with "Error during checkpoint removal: [internal error: bitmap 'XX' not found in backing chain of 'XX']" In this situation the checkpoints are still defined in the libvirt ecosystem but the required bitmaps in the qcow files do not exist anymore. This is a situation that is not automatically cleaned up by the backup process because it may require manual intervention and should usually not happen. You can cleanup this situation by removing the reported checkpoints via: ``` virsh checkpoint-delete --checkpointname --metadata ``` `Note`: > Since version 2.30 virtnbdbackup will handle these situations automatically > and falls back to a full backup in case it detects a broken bitmap chain. ## Backup fails with "Virtual machine does not support required backup features, please adjust virtual machine configuration." The libvirt version you are using does by default not expose the functionality required for creating full or incremental backups. You can either use the backup mode `copy` or enable the backup features as described [here](#libvirt-versions--760-debian-bullseye-ubuntu-20x) ## Backup fails with "Timed out during operation: cannot acquire state change lock" If backups fail with error: ``` ERROR [..] Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainBackupBegin) ``` there is still some block jobs operation active on the running domain, for example a live migration or another backup job. It may also happen that `virtnbdbackup` crashes abnormally or is forcibly killed during backup operation, unable to stop its own backup job. You can use option `-k` to forcibly kill any running active block jobs for the domain, but use with care. It is better to check which operation is active with the `virsh domjobinfo` command first. ``` virtnbdbackup -d vm2 -l copy -k -o - [..] INFO virtnbdbackup - main: Stopping domain jobs ``` ## Backup fails with "Failed to bind socket to /var/tmp/virtnbdbackup.XX: Permission denied" The issue is most likely an active `apparmor` profile that prevents the qemu daemon from creating its socket file for the nbd server. Try to disable apparmor using the **aa-teardown** command for the current session you are executing a backup or restore. You can also add the following lines: ``` /var/tmp/virtnbdbackup.* rw, /var/tmp/backup.* rw, ``` to the configuration files (might not exist by default): ``` /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/local/abstractions/libvirt-qemu /etc/apparmor.d/local/usr.sbin.libvirtd ``` or, on newer versions: ``` sudo mkdir -p /etc/apparmor.d/abstractions/libvirt-qemu.d cat < # Uses parent directory as context: # git clone https://github.com/abbbi/virtnbdbackup # cd virtnbdbackup # docker build -f docker/Dockerfile . FROM debian:bookworm-slim ARG DEBIAN_FRONTEND="noninteractive" ARG source="https://github.com/abbbi/virtnbdbackup" LABEL container.name="virtnbdbackup-docker" LABEL container.source.description="Backup utiliy for Libvirt kvm / qemu with Incremental backup support via NBD" LABEL container.description="virtnbdbackup and virtnbdrestore (plus dependencies) to run on hosts with libvirt >= 6.0.0" LABEL container.source=$source LABEL container.version="1.1" LABEL maintainer="Michael Ablassmeier " COPY . /tmp/build/ # Deploys dependencies and pulls sources, installing virtnbdbackup and removing unnecessary content: RUN \ apt-get update && \ apt-get install -y --no-install-recommends \ ca-certificates openssh-client python3-all python3-libnbd python3-libvirt python3-lz4 python3-setuptools python3-tqdm qemu-utils python3-lxml python3-paramiko python3-colorlog && \ cd /tmp/build/ && python3 setup.py install && cd .. && \ apt-get purge -y ca-certificates && apt-get -y autoremove --purge && apt-get clean && \ rm -rf /var/lib/apt/lists/* /tmp/* # Default folder: WORKDIR / ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/docker/README.md0000644000000000000000000001270115033267156015571 0ustar00rootroot## Overview This dockerfile is intended for scenarios where isn't viable to provide the necessary environment, such as dependencies or tools, due to system limitations; such as an old OS version, inmutable or embedded rootfs, live distros, docker oriented OSes, etc. Originally was created to be used on Unraid OS (tested since v6.9.2), and should work equally fine on any other GNU/Linux distro as much as [requirements](#requirements) are accomplished. Includes `virtnbdbackup`, `virtnbdrestore` and similar utils, installed along with their required dependecies. Other utilities, such as latest Qemu Utils and OpenSSH Client, are also included to leverage all available features. Currently, is being built from latest `debian:bookworm-slim` official image. ## Requirements - Docker Engine on the host server. See [Docker Documentation](https://docs.docker.com/get-docker/) for further instructions - Libvirt >=v6.0.0. on the host server (minimal). A version >=7.6.0 is necessary to avoid [patching XML VM definitions](../README.md#libvirt-versions--760-debian-bullseye-ubuntu-20x) - Qemu Guest Agent installed and running inside the guest OS. For *NIX guests, use the latest available version according the distro (installed by default on Debian 12 when provisioned via ISO). For Windows guests, install latest [VirtIO drivers](https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/) ## Bind mounts: *All the trick consists into set the right bind mounts for your host OS case* - Virtnbdbackup needs to access libvirt's socket in order to work correctly, and attempts this via `/var/run/libvirt` path. In basically all mainstream distros of today (Debian, RedHat, Archlinux and the countless distros based on these) as in this image, `/var/run` is a symlink to `/run` and `/var/lock` a symlink to `run/lock`. Therefore, for the vast majority of scenarios the correct bind mount is: `-v /run:/run` But in some operating systems, `/run` and `/var/run` are still separated folders. Under this scenario you need to bind mount with `-v /var/run:/run` And most likely, you will need to mount with either `-v /var/lock:/run/lock` or `-v /var/run/lock:/run/lock` in order to run this container correctly. If you're in trouble with this, read [Main FAQ](../README.md#faq) first, and identify the error you're getting in order to set the correct bind mounts that work for the specific host that serves Docker. - Virtnbdbackup and virtnbdrestore create sockets for backup/restoration jobs tasks at `/var/tmp`. Ensure to *always* add a bind mount with `-v /var/tmp:/var/tmp` - When working with VMs that require to boot with UEFI emulation (e.g. Windows 10 and up), addiitonal bind mounts are needed: Path to `/etc/libvirt/qemu/nvram` is required to backup/restore nvram files per VM (which seems to be the same on Qemu implementations tested so far) Path to your distro correspondent OVMF files. This is `/usr/share/OVMF` on Debian based, and `/usr/share/qemu/ovmf-x64` on Unraid (feel free to report this path on other distributions) - Finally, using identical *host:container* bind mounts for virtual disk locations (as well nvram & ovmf binaries, when applies), is necessary to allow backup/restore commands to find out the files at the expected locations, in concordance with VM definitions at the host side. ## Usage Examples For detailed info about options, also see [backup](../README.md#backup-examples) and [restoration](../README.md#restoration-examples) examples ### Full or incremental backup: ``` docker run --rm \ -v /run:/run \ -v /var/tmp:/var/tmp \ -v /etc/libvirt/qemu/nvram:/etc/libvirt/qemu/nvram \ -v /usr/share/OVMF:/usr/share/OVMF \ -v /:/backups \ ghcr.io/abbbi/virtnbdbackup:master \ virtnbdbackup -d -l auto -o /backups/ ``` Where `` is an example of the actual master backups folder where VM sub-folders are being stored in your system, and `` the VM name (actual path to disk images is not required.) ### Full Backup Restoration to an existing VM: ``` docker run --rm \ -v /run:/run \ -v /var/tmp:/var/tmp \ -v /etc/libvirt/qemu/nvram:/etc/libvirt/qemu/nvram \ -v /usr/share/OVMF:/usr/share/OVMF \ -v /mnt/backups:/backups \ -v /:/ \ ghcr.io/abbbi/virtnbdbackup:master \ bash -c \ "mkdir -p //.old && \ mv ///* //.old/ && \ virtnbdrestore -i /backups/ -o //" ``` Where `//` is the actual folder where the specific disk image(s) of the VM to restore, are stored on the host system. In this case, bind mounts should be identical. On this example, any existing files are being moved to a folder named `.old`, because restore would fail if it finds the same image(s) that is attempting to restore onto the destination. For instance, you might opt to operate with existing images according your needs, e.g. deleting it before to restore from backup. ## Interactive Mode: This starts a session inside a (volatile) container, provisioning all bind mounts and allowing to do manual backups and restores, as well testing/troubleshooting: ``` docker run -it --rm \ -v /run:/run \ -v /var/tmp:/var/tmp \ -v /etc/libvirt/qemu/nvram:/etc/libvirt/qemu/nvram \ -v /usr/share/OVMF:/usr/share/OVMF \ -v /mnt/backups:/backups \ -v /:/ \ ghcr.io/abbbi/virtnbdbackup \ bash ``` ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4844875 virtnbdbackup-2.30/libvirtnbdbackup/0000755000000000000000000000000015033267565016373 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/__init__.py0000644000000000000000000000127515033267156020505 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ __version__ = "2.30" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/argopt.py0000644000000000000000000000776315033267156020252 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os from getpass import getuser from argparse import _ArgumentGroup from libvirtnbdbackup import __version__ def addRemoteArgs(opt: _ArgumentGroup) -> None: """Common remote backup arguments""" user = getuser() or None session = "qemu:///system" if user != "root": session = "qemu:///session" opt.add_argument( "-U", "--uri", default=session, required=False, type=str, help="Libvirt connection URI. (default: %(default)s)", ) opt.add_argument( "--user", default=None, required=False, type=str, help="User to authenticate against libvirtd. (default: %(default)s)", ) opt.add_argument( "--ssh-user", default=user, required=False, type=str, help=( "User to authenticate against remote sshd: " "used for remote copy of files. (default: %(default)s)" ), ) opt.add_argument( "--ssh-port", default=22, required=False, type=int, help=( "Port to connect to remote sshd: " "used for remote copy of files. (default: %(default)s)" ), ) opt.add_argument( "--password", default=None, required=False, type=str, help="Password to authenticate against libvirtd. (default: %(default)s)", ) opt.add_argument( "-P", "--nbd-port", type=int, default=10809, required=False, help=( "Port used by remote NBD Service, should be unique for each" " started backup. (default: %(default)s)" ), ) opt.add_argument( "-I", "--nbd-ip", type=str, default="", required=False, help=( "IP used to bind remote NBD service on" " (default: hostname returned by libvirtd)" ), ) opt.add_argument( "--tls", action="store_true", required=False, help="Enable and use TLS for NBD connection. (default: %(default)s)", ) opt.add_argument( "--tls-cert", type=str, default="/etc/pki/qemu/", required=False, help=( "Path to TLS certificates used during offline backup" " and restore. (default: %(default)s)" ), ) def addDebugArgs(opt: _ArgumentGroup) -> None: """Common debug arguments""" opt.add_argument( "-v", "--verbose", default=False, help="Enable debug output", action="store_true", ) opt.add_argument( "-V", "--version", default=False, help="Show version and exit", action="version", version=__version__, ) def addLogArgs(opt, prog): """Logging related arguments""" try: HOME = os.environ["HOME"] except KeyError: HOME = "/tmp" opt.add_argument( "-L", "--logfile", default=f"{HOME}/{prog}.log", type=str, help="Path to Logfile (default: %(default)s)", ) def addLogColorArgs(opt): """Option to enable or disable colored output""" opt.add_argument( "--nocolor", default=False, help="Disable colored output (default: %(default)s)", action="store_true", ) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4854875 virtnbdbackup-2.30/libvirtnbdbackup/backup/0000755000000000000000000000000015033267565017640 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/backup/__init__.py0000644000000000000000000000132115033267156021742 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ __title__ = "backup" __version__ = "0.1" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/backup/check.py0000644000000000000000000001214715033267156021270 0ustar00rootroot""" Copyright (C) 2024 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging from typing import List, Any from argparse import Namespace from libvirt import virDomain from libvirtnbdbackup import virt from libvirtnbdbackup import common as lib from libvirtnbdbackup import exceptions log = logging.getLogger() def arguments(args: Namespace) -> None: """Check passed arguments for validity""" if args.compress is not False and args.type == "raw": raise exceptions.BackupException("Compression not supported with raw output.") if args.stdout is True and args.type == "raw": raise exceptions.BackupException("Output type raw not supported to stdout.") if args.stdout is True and args.raw is True: raise exceptions.BackupException( "Saving raw images to stdout is not supported." ) if args.type == "raw" and args.level in ("inc", "diff"): raise exceptions.BackupException( "Stream format raw does not support incremental or differential backup." ) def targetDir(args: Namespace) -> None: """Check if target directory backup is started to meets all requirements based on the backup level executed""" if ( args.level not in ("copy", "full", "auto") and not lib.hasFullBackup(args) and not args.stdout ): raise exceptions.BackupException( f"Unable to execute [{args.level}] backup: " f"No full backup found in target directory: [{args.output}]" ) if lib.targetIsEmpty(args) and args.level == "auto": log.info("Backup mode auto, target folder is empty: executing full backup.") args.level = "full" elif not lib.targetIsEmpty(args) and args.level == "auto": if not lib.hasFullBackup(args): raise exceptions.BackupException( "Can't execute switch to auto incremental backup: " f"specified target folder [{args.output}] does not contain full backup.", ) log.info("Backup mode auto: executing incremental backup.") args.level = "inc" elif not args.stdout and not args.startonly and not args.killonly: if not lib.targetIsEmpty(args): raise exceptions.BackupException( "Target directory already contains full or copy backup." ) def vmstate(args, virtClient: virt.client, domObj: virDomain) -> None: """Check virtual machine state before executing backup and based on situation, either fallback to regular copy backup or attempt to bring VM into paused state""" if domObj.isActive() == 0: args.offline = True if args.start_domain is True: log.info("Starting domain in paused state") if virtClient.startDomain(domObj) == 0: args.offline = False else: log.info("Failed to start VM in paused mode.") if args.level == "full" and args.offline is True: log.warning("Domain is offline, resetting backup options.") args.level = "copy" log.warning("New Backup level: [%s].", args.level) if args.offline is True and args.startonly is True: raise exceptions.BackupException( "Domain is offline: must be active for this function." ) def vmfeature(virtClient: virt.client, domObj: virDomain) -> None: """Check if required features are enabled in domain config""" if virtClient.hasIncrementalEnabled(domObj) is False: raise exceptions.BackupException( "Virtual machine does not support required backup features, " "please adjust virtual machine configuration." ) def diskformat(args: Namespace, disks: List[Any]) -> None: """Check if disks meet requirements for backup mode, if not, switch backup job to type copy.""" if args.level != "copy" and lib.hasQcowDisks(disks) is False: args.level = "copy" raise exceptions.BackupException( "Only raw disks attached, switching to backup mode copy." ) def blockjobs( args, virtClient: virt.client, domObj: virDomain, disks: List[Any] ) -> None: """Check if there is an already active backup operation on the domain disks. If so, fail accordingly""" if ( not args.killonly and not args.offline and virtClient.blockJobActive(domObj, disks) ): raise exceptions.BackupException( "Active block job for running domain:" f" Check with [virsh domjobinfo {args.domain}] or use option -k to kill the active job." ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/backup/disk.py0000644000000000000000000002010215033267156021133 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging from argparse import Namespace from typing import List, Any, Tuple from libvirtnbdbackup import nbdcli from libvirtnbdbackup import virt from libvirtnbdbackup.virt.client import DomainDisk from libvirtnbdbackup.objects import processInfo from libvirtnbdbackup.sparsestream import streamer from libvirtnbdbackup.sparsestream import types from libvirtnbdbackup import exceptions from libvirtnbdbackup import chunk from libvirtnbdbackup import block from libvirtnbdbackup.backup import partialfile from libvirtnbdbackup.backup import server from libvirtnbdbackup.backup import target from libvirtnbdbackup.backup.metadata import backupChecksum from libvirtnbdbackup import extenthandler from libvirtnbdbackup.qemu import util as qemu from libvirtnbdbackup.qemu.exceptions import ProcessError from libvirtnbdbackup import common as lib from libvirtnbdbackup import output from libvirtnbdbackup.output import stream def _setStreamType(args: Namespace, disk: DomainDisk) -> str: """Set target stream type based on disk format""" streamType = "raw" if disk.format != streamType: streamType = args.type return streamType def _getExtentHandler(args: Namespace, nbdClient): """Query dirty blocks either via qemu client or self implemented extend handler""" if args.qemu: logging.info("Using qemu tools to query extents") extentHandler = extenthandler.ExtentHandler( qemu.util(nbdClient.cType.exportName), nbdClient.cType, args.no_sparse_detection, ) else: extentHandler = extenthandler.ExtentHandler( nbdClient, nbdClient.cType, args.no_sparse_detection ) return extentHandler def backup( # pylint: disable=too-many-arguments,too-many-branches, too-many-locals, too-many-statements args: Namespace, disk: DomainDisk, count: int, fileStream, virtClient: virt.client, ) -> Tuple[int, bool]: """Backup domain disk data.""" dStream = streamer.SparseStream(types) sTypes = types.SparseStreamTypes() lib.setThreadName(disk.target) streamType = _setStreamType(args, disk) metaContext = nbdcli.context.get(args, disk) nbdProc: processInfo remoteIP: str = virtClient.remoteHost port: int = args.nbd_port if args.nbd_ip != "": remoteIP = args.nbd_ip if args.offline is True: port = args.nbd_port + count try: nbdProc = server.setup(args, disk, virtClient.remoteHost, port) except ProcessError as errmsg: logging.error(errmsg) raise exceptions.DiskBackupFailed("Failed to start NBD server.") if disk.discardOption is not None: logging.info("Virtual disk discard option: [%s]", disk.discardOption) connection = server.connect(args, disk, metaContext, remoteIP, port, virtClient) extentHandler = _getExtentHandler(args, connection) extents = extentHandler.queryBlockStatus() diskSize = connection.nbd.get_size() if extents is None: logging.error("No extents returned by NBD server.") return 0, False thinBackupSize = sum(extent.length for extent in extents if extent.data is True) logging.info("Got %s extents to backup.", len(extents)) logging.debug("%s", lib.dumpExtentJson(extents)) logging.info("%s bytes [%s] virtual disk size", diskSize, lib.humanize(diskSize)) logging.info( "%s bytes [%s] of data extents to backup", thinBackupSize, lib.humanize(thinBackupSize), ) if args.level in ("inc", "diff") and thinBackupSize == 0: logging.info("No dirty blocks found") args.noprogress = True targetFile, targetFilePartial = target.Set(args, disk) # if writing to regular files we want instantiate an new # handle for each file otherwise multiple threads collid # during file close # in case of zip file output we want to use the existing # opened output channel if not args.stdout: fileStream = stream.get(args, output.target()) writer = target.get(args, fileStream, targetFile, targetFilePartial) if streamType == "raw": logging.info("Creating full provisioned raw backup image") writer.truncate(diskSize) else: logging.info("Creating thin provisioned stream backup image") header = dStream.dumpMetadata( args, diskSize, thinBackupSize, disk, ) dStream.writeFrame(writer, sTypes.META, 0, len(header)) writer.write(header) writer.write(sTypes.TERM) progressBar = lib.progressBar( thinBackupSize, f"saving disk {disk.target}", args, count=count ) compressedSizes: List[Any] = [] backupSize: int = 0 for save in extents: if save.data is True: if streamType == "stream": dStream.writeFrame(writer, sTypes.DATA, save.offset, save.length) logging.debug( "Read data from: start %s, length: %s", save.offset, save.length ) cSizes = None if save.length >= connection.maxRequestSize: logging.debug( "Chunked data read from: start %s, length: %s", save.offset, save.length, ) size, cSizes = chunk.write( writer, save, connection, streamType, args.compress, progressBar ) else: size = block.write( writer, save, connection, streamType, args.compress, ) if streamType == "raw": size = writer.seek(save.offset) progressBar.update(save.length) if streamType == "stream": writer.write(sTypes.TERM) if args.compress: logging.debug("Compressed size: %s", size) backupSize += size if cSizes: blockList = {} blockList[size] = cSizes compressedSizes.append(blockList) else: compressedSizes.append(size) else: assert size == save.length backupSize += save.length else: if streamType == "raw": writer.seek(save.offset) backupSize += save.length elif streamType == "stream" and args.level not in ("inc", "diff"): dStream.writeFrame(writer, sTypes.ZERO, save.offset, save.length) if streamType == "stream": dStream.writeFrame(writer, sTypes.STOP, 0, 0) if args.compress: dStream.writeCompressionTrailer(writer, compressedSizes) progressBar.close() writer.close() connection.disconnect() if args.offline is True and virtClient.remoteHost == "": logging.info("Stopping NBD Service.") lib.killProc(nbdProc.pid) if args.offline is True: lib.remove(args, nbdProc.pidFile) if not args.stdout: if args.noprogress is True: logging.info( "Backup of disk [%s] finished, file: [%s]", disk.target, targetFile ) partialfile.rename(targetFilePartial, targetFile) if streamType != "raw": backupChecksum(fileStream, targetFile) return backupSize, True ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/backup/job.py0000644000000000000000000000260315033267156020761 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging from argparse import Namespace from typing import List from libvirt import virDomain from libvirtnbdbackup import virt from libvirtnbdbackup.virt.client import DomainDisk from libvirtnbdbackup.virt.exceptions import startBackupFailed def start( args: Namespace, virtClient: virt.client, domObj: virDomain, disks: List[DomainDisk], ) -> bool: """Start backup job via libvirt API""" try: logging.info("Starting backup job.") virtClient.startBackup( args, domObj, disks, ) logging.debug("Backup job started.") return True except startBackupFailed as e: logging.error(e) return False ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/backup/metadata.py0000644000000000000000000001265415033267156021776 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import logging from argparse import Namespace from typing import List, Union from libvirtnbdbackup import output from libvirtnbdbackup.virt.client import DomainDisk from libvirtnbdbackup import common as lib from libvirtnbdbackup.qemu import util as qemu from libvirtnbdbackup.qemu.exceptions import ProcessError from libvirtnbdbackup.ssh.exceptions import sshError from libvirtnbdbackup.output.exceptions import OutputException log = logging.getLogger() def backupChecksum(fileStream, targetFile): """Save the calculated adler32 checksum, it can be verified by virtnbdbrestore's verify function.'""" checksum = fileStream.checksum() logging.info("Checksum for file: [%s]:[%s]", targetFile, checksum) chksumfile = f"{targetFile}.chksum" logging.info("Saving checksum to: [%s]", chksumfile) with output.openfile(chksumfile, "w") as cf: cf.write(f"{checksum}") def backupConfig(args: Namespace, vmConfig: str) -> Union[str, None]: """Save domain XML config file""" configFile = f"{args.output}/vmconfig.{lib.getIdent(args)}.xml" log.info("Saving VM config to: [%s]", configFile) try: with output.openfile(configFile, "wb") as fh: fh.write(vmConfig.encode()) return configFile except OutputException as e: log.error("Failed to save VM config: [%s]", e) return None def backupDiskInfo(args: Namespace, disk: DomainDisk): """Save information about qcow image, used to reconstruct the qemu image with the same settings during restore""" try: info = qemu.util("").info(disk.path, args.sshClient) except ( ProcessError, sshError, ) as errmsg: log.warning("Failed to read qcow image info: [%s]", errmsg) return configFile = f"{args.output}/{disk.target}.{lib.getIdent(args)}.qcow.json" try: with output.openfile(configFile, "wb") as fh: fh.write(info.out.encode()) log.info("Saved qcow image config to: [%s]", configFile) if args.stdout is True: args.diskInfo.append(configFile) except OutputException as e: log.warning("Failed to save qcow image config: [%s]", e) def backupBootConfig(args: Namespace) -> None: """Save domain uefi/nvram/kernel and loader if configured.""" for setting, val in args.info.items(): if args.level != "copy": tFile = f"{args.output}/{os.path.basename(val)}.{lib.getIdent(args)}" else: tFile = f"{args.output}/{os.path.basename(val)}" log.info("Save additional boot config [%s] to: [%s]", setting, tFile) lib.copy(args, val, tFile) args.info[setting] = tFile def backupAutoStart(args: Namespace) -> None: """Save information if virtual machine was marked for autostart during system boot""" log.info("Autostart setting configured for virtual machine.") autoStartFile = f"{args.output}/autostart.{lib.getIdent(args)}" try: with output.openfile(autoStartFile, "wb") as fh: fh.write(b"True") except OutputException as e: log.warning("Failed to save autostart information: [%s]", e) def saveFiles( args: Namespace, vmConfig: str, disks: List[DomainDisk], fileStream: Union[output.target.Directory, output.target.Zip], logFile: str, ): """Save additional files such as virtual machine configuration and UEFI / kernel images""" configFile = backupConfig(args, vmConfig) backupBootConfig(args) for disk in disks: if disk.format.startswith("qcow"): backupDiskInfo(args, disk) if args.stdout is True: addFiles(args, configFile, fileStream, logFile) def addFiles(args: Namespace, configFile: Union[str, None], zipStream, logFile: str): """Add backup log and other files to zip archive""" if configFile is not None: log.info("Adding vm config to zipfile") zipStream.zipStream.write(configFile, configFile) if args.level in ("full", "inc"): log.info("Adding checkpoint info to zipfile") zipStream.zipStream.write(args.cpt.file, args.cpt.file) for dirname, _, files in os.walk(args.checkpointdir): zipStream.zipStream.write(dirname) for filename in files: zipStream.zipStream.write(os.path.join(dirname, filename)) for setting, val in args.info.items(): log.info("Adding additional [%s] setting file [%s] to zipfile", setting, val) zipStream.zipStream.write(val, os.path.basename(val)) for diskInfo in args.diskInfo: log.info("Adding QCOW image format file [%s] to zipfile", diskInfo) zipStream.zipStream.write(diskInfo, os.path.basename(diskInfo)) log.info("Adding backup log [%s] to zipfile", logFile) zipStream.zipStream.write(logFile, logFile) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/backup/partialfile.py0000644000000000000000000000340515033267156022504 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import glob import logging from argparse import Namespace from libvirtnbdbackup import exceptions log = logging.getLogger() def _exists(args: Namespace) -> int: """Check for possible partial backup files""" partialFiles = glob.glob(f"{args.output}/*.partial") return len(partialFiles) > 0 def exists(args: Namespace) -> bool: """Check if target directory has an partial backup, makes backup utility exit errnous in case backup type is full or inc""" if args.level in ("inc", "diff") and args.stdout is False and _exists(args) is True: log.error("Partial backup found in target directory: [%s]", args.output) log.error("One of the last backups seems to have failed.") log.error("Consider re-executing full backup.") return True return False def rename(targetFilePartial: str, targetFile: str) -> None: """After backup, move .partial file to real target file""" try: os.rename(targetFilePartial, targetFile) except OSError as e: raise exceptions.DiskBackupFailed(f"Failed to rename file: [{e}]") from e ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/backup/server.py0000644000000000000000000000574515033267156021527 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging from argparse import Namespace from typing import Union from libvirtnbdbackup import nbdcli from libvirtnbdbackup import virt from libvirtnbdbackup.virt.client import DomainDisk from libvirtnbdbackup.qemu import util as qemu from libvirtnbdbackup.objects import processInfo from libvirtnbdbackup.nbdcli.exceptions import NbdClientException from libvirtnbdbackup.exceptions import DiskBackupFailed def setup(args: Namespace, disk: DomainDisk, remoteHost: str, port: int) -> processInfo: """Start background qemu-nbd process used during backup if domain is offline, in case of remote backup, initiate ssh session and start process on remote system.""" bitMap: str = "" if args.level in ("inc", "diff"): bitMap = args.cpt.name socket = f"{args.socketfile}.{disk.target}" if remoteHost != "": logging.info( "Offline backup, starting remote NBD server, socket: [%s:%s], port: [%s]", remoteHost, socket, port, ) nbdProc = qemu.util(disk.target).startRemoteBackupNbdServer( args, disk, bitMap, port ) logging.info("Remote NBD server started, PID: [%s].", nbdProc.pid) return nbdProc logging.info("Offline backup, starting local NBD server, socket: [%s]", socket) nbdProc = qemu.util(disk.target).startBackupNbdServer( disk.format, disk.path, socket, bitMap ) logging.info("Local NBD Service started, PID: [%s]", nbdProc.pid) return nbdProc def connect( # pylint: disable=too-many-arguments args: Namespace, disk: DomainDisk, metaContext: str, remoteIP: str, port: int, virtClient: virt.client, ): """Connect to started nbd endpoint""" socket = args.socketfile if args.offline is True: socket = f"{args.socketfile}.{disk.target}" cType: Union[nbdcli.TCP, nbdcli.Unix] if virtClient.remoteHost != "": cType = nbdcli.TCP(disk.target, metaContext, remoteIP, args.tls, port) else: cType = nbdcli.Unix(disk.target, metaContext, socket) nbdClient = nbdcli.client(cType, args.no_sparse_detection) try: return nbdClient.connect() except NbdClientException as e: raise DiskBackupFailed( f"NBD endpoint: [{cType}]: connection failed: [{e}]" ) from e ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/backup/target.py0000644000000000000000000000355415033267156021503 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging from typing import BinaryIO from argparse import Namespace from libvirtnbdbackup.virt.client import DomainDisk from libvirtnbdbackup.common import getIdent def get( args: Namespace, fileStream, targetFile: str, targetFilePartial: str ) -> BinaryIO: """Open target file based on output writer""" if args.stdout is True: logging.info("Writing data to zip archive.") fileStream.open(targetFile) else: logging.info("Write data to target file: [%s].", targetFilePartial) fileStream.open(targetFilePartial) return fileStream def Set(args: Namespace, disk: DomainDisk, ext: str = "data"): """Set Target file name to write data to, used for both data files and qemu disk info""" targetFile: str = "" level = args.level_filename if level in ("full", "copy"): if disk.format == "raw": level = "copy" targetFile = f"{args.output}/{disk.target}.{level}.{ext}" elif level in ("inc", "diff"): cptName = getIdent(args) targetFile = f"{args.output}/{disk.target}.{level}.{cptName}.{ext}" targetFilePartial = f"{targetFile}.partial" return targetFile, targetFilePartial ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/block.py0000644000000000000000000000440515033267156020036 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ from typing import Generator, IO, Any, Union from nbd import Error as nbdError from libvirtnbdbackup import lz4 from libvirtnbdbackup.exceptions import BackupException def step(offset: int, length: int, maxRequestSize: int) -> Generator: """Process block and ensure to not exceed the maximum request size from NBD server. If length parameter is dict, compression was enabled during backup, thus we cannot use the offsets and sizes for the original data, but must use the compressed offsets and sizes to read the correct lz4 frames from the stream. """ blockOffset = offset if isinstance(length, dict): blockOffset = offset compressOffset = list(length.keys())[0] for part in length[compressOffset]: blockOffset += part yield part, blockOffset else: blockOffset = offset while blockOffset < offset + length: blocklen = min(offset + length - blockOffset, maxRequestSize) yield blocklen, blockOffset blockOffset += blocklen def write( writer: IO[Any], block, nbdCon, btype: str, compress: Union[bool, int] ) -> int: """Write single block that does not exceed nbd maxRequestSize setting. In case compression is enabled, single blocks are compressed using lz4. """ if btype == "raw": writer.seek(block.offset) try: data = nbdCon.nbd.pread(block.length, block.offset) except nbdError as e: raise BackupException(e) from e if compress is not False: data = lz4.compressFrame(data, compress) return writer.write(data) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/chunk.py0000644000000000000000000000646415033267156020063 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ from typing import List, Any, Tuple, IO, Union from nbd import Error as nbdError from libvirtnbdbackup import block from libvirtnbdbackup import lz4 from libvirtnbdbackup.exceptions import DiskBackupFailed # pylint: disable=too-many-arguments def write( writer: IO[Any], blk, nbdCon, btype: str, compress: Union[bool, int], pbar ) -> Tuple[int, List[int]]: """During extent processing, consecutive blocks with the same type(data or zeroed) are unified into one big chunk. This helps to reduce requests to the NBD Server. But in cases where the block to be saved exceeds the maximum recommended request size (nbdClient.maxRequestSize), we need to split one big request into multiple not exceeding the limit If compression is enabled, function returns a list of offsets for the compressed frames, which is appended to the end of the stream. """ wSize = 0 cSizes = [] for blocklen, blockOffset in block.step( blk.offset, blk.length, nbdCon.maxRequestSize ): if btype == "raw": writer.seek(blockOffset) try: data = nbdCon.nbd.pread(blocklen, blockOffset) except nbdError as e: raise DiskBackupFailed(e) from e if compress is not False: compressed = lz4.compressFrame(data, compress) wSize += writer.write(compressed) cSizes.append(len(compressed)) else: wSize += writer.write(data) pbar.update(blocklen) return wSize, cSizes def read( reader: IO[Any], offset: int, length: int, nbdCon, compression: bool, pbar, ) -> int: """Read data from reader and write to nbd connection If Compression is enabled function receives length information as dict, which contains the stream offsets for the compressed lz4 frames. Frames are read from the stream at the compressed size information (offset in the stream). After decompression, data is written back to original offset in the virtual machine disk image. If no compression is enabled, data is read from the regular data header at its position and written to nbd target directly. """ wSize = 0 for blocklen, blockOffset in block.step(offset, length, nbdCon.maxRequestSize): if compression is True: data = lz4.decompressFrame(reader.read(blocklen)) nbdCon.nbd.pwrite(data, offset) offset += len(data) wSize += len(data) else: data = reader.read(blocklen) nbdCon.nbd.pwrite(data, blockOffset) wSize += len(data) pbar.update(blocklen) return wSize ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/common.py0000644000000000000000000002071015033267156020231 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import sys import glob import json import logging import logging.handlers import signal import shutil import pprint from time import time from threading import current_thread from argparse import Namespace from typing import Optional, List, Any, Union, Dict from tqdm import tqdm import colorlog from libvirtnbdbackup import ssh from libvirtnbdbackup.ssh.exceptions import sshError from libvirtnbdbackup import output from libvirtnbdbackup.logcount import logCount log = logging.getLogger("lib") logFormat = ( "%(asctime)s %(levelname)s %(name)s %(module)s - %(funcName)s" " [%(threadName)s]: %(message)s" ) logFormatColored = ( "%(green)s%(asctime)s%(reset)s%(blue)s %(log_color)s%(levelname)s%(reset)s " "%(name)s %(module)s - %(funcName)s" " [%(threadName)s]: %(log_color)s %(message)s" ) logDateFormat = "[%Y-%m-%d %H:%M:%S]" defaultCheckpointName = "virtnbdbackup" def argparse(parser) -> Namespace: """Parse arguments""" return parser.parse_args() def printVersion(version) -> None: """Print version and passed arguments""" log.info("Version: %s Arguments: %s", version, " ".join(sys.argv)) def humanize(num, suffix="B"): """Print size in human readable output""" for unit in ["", "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi"]: if abs(num) < 1024.0: return f"{num:3.1f}{unit}{suffix}" num /= 1024.0 return f"{num:.1f}Yi{suffix}" def setThreadName(tn="main") -> None: """Set thread name reported by logging function""" current_thread().name = tn def setLogLevel(verbose: bool) -> int: """Set loglevel""" level = logging.INFO if verbose is True: level = logging.DEBUG return level def sshSession( args: Namespace, remoteHost: str, mode: ssh.Mode = ssh.Mode.DOWNLOAD ) -> Union[ssh.client, None]: """Use ssh to copy remote files""" try: return ssh.client(remoteHost, args.ssh_user, args.ssh_port, mode) except sshError as err: log.warning("Failed to setup SSH connection: [%s]", err) return None def getLogFile(fileName: str) -> Optional[logging.FileHandler]: """Try setup log handler, if this fails, something is already wrong, but we can at least provide correct error message.""" try: return logging.FileHandler(fileName) except OSError as e: logging.error("Failed to open logfile: [%s].", e) return None def configLogger( args: Namespace, fileLog: Optional[logging.FileHandler], counter: logCount ): """Setup logging""" syslog = False try: syslog = args.syslog is True except AttributeError: pass handler: List[Any] handler = [ fileLog, counter, ] stderrh = logging.StreamHandler(stream=sys.stderr) if args.nocolor is False: formatter = colorlog.ColoredFormatter( logFormatColored, datefmt=logDateFormat, log_colors={ "WARNING": "yellow", "ERROR": "red", "DEBUG": "cyan", "CRITICAL": "red", }, ) stderrh.setFormatter(formatter) if args.quiet is False: handler.append(stderrh) if syslog is True: handler.append(logging.handlers.SysLogHandler(address="/dev/log")) logging.basicConfig( level=setLogLevel(args.verbose), format=logFormat, datefmt=logDateFormat, handlers=handler, ) def hasFullBackup(args: Namespace) -> int: """Check if full backup file exists in target directory""" fullFiles = glob.glob(f"{args.output}/*.full.data") return len(fullFiles) > 0 def exists(args: Namespace, filePath: str) -> bool: """Check if file exists either remotely or locally.""" if args.sshClient: return args.sshClient.exists(filePath) return os.path.exists(filePath) def targetIsEmpty(args: Namespace) -> bool: """Check if target directory does not include an backup already (no .data or .data.partial files)""" if exists(args, args.output) and args.level in ("full", "copy", "auto"): dirList = glob.glob(f"{args.output}/*.data*") if len(dirList) > 0: return False return True def getLatest(targetDir: str, search: str, key=None) -> List[str]: """get the last backed up file matching search from the backupset, used to find latest vm config, data files or data files by disk. """ ret: List[str] = [] try: files = glob.glob(f"{targetDir}/{search}") files.sort(key=os.path.getmtime) if key is not None: ret.append(files[key]) else: ret = files log.debug("Sorted data files: \n%s", pprint.pformat(ret)) return ret except IndexError: return [] def hasQcowDisks(diskList: List[Any]) -> bool: """Check if the list of attached disks includes at least one qcow image based disk, else checkpoint handling can be skipped and backup module falls back to type copy""" for disk in diskList: if disk.format.startswith("qcow"): return True return False def copy(args: Namespace, source: str, target: str) -> None: """Copy file, handle exceptions""" try: if args.sshClient: args.sshClient.copy(source, target) else: shutil.copyfile(source, target) except OSError as e: log.warning("Failed to copy [%s] to [%s]: [%s]", source, target, e) except sshError as e: log.warning("Remote copy from [%s] to [%s] failed: [%s]", source, target, e) def remove(args: Namespace, file: str) -> None: """Remove file either locally or remote""" try: if args.sshClient: args.sshClient.run(f"rm -f {file}") else: os.remove(file) log.debug("Removed: [%s]", file) except FileNotFoundError: pass except OSError as e: log.warning("Failed to remove [%s]: [%s]", file, e) except sshError as e: log.warning("Remote remove failed: [%s]: [%s]", file, e) def progressBar(total: int, desc: str, args: Namespace, count=0) -> tqdm: """Return tqdm object""" return tqdm( total=total, desc=desc, unit="B", unit_scale=True, unit_divisor=1024, disable=args.noprogress, position=count, leave=False, ) def killProc(pid: int) -> bool: """Attempt kill PID""" log.debug("Killing PID: %s", pid) while True: try: os.kill(pid, signal.SIGTERM) return True except ProcessLookupError: return True def getIdent(args: Namespace) -> Union[str, int]: """Used to get an unique identifier for target files, usually checkpoint name is used, but if no checkpoint is created, we use timestamp""" try: ident = args.cpt.name except AttributeError: ident = int(time()) if args.level == "diff": ident = int(time()) if args.level == "copy": ident = "copy" return ident def dumpExtentJson(extents) -> str: """Dump extent object as json""" extList = [] for extent in extents: ext = {} ext["start"] = extent.offset ext["length"] = extent.length ext["data"] = extent.data extList.append(ext) return json.dumps(extList, indent=4, sort_keys=True) def dumpMetaData(dataFile: str, stream): """read metadata header""" with output.openfile(dataFile, "rb") as reader: _, _, length = stream.readFrame(reader) return stream.loadMetadata(reader.read(length)) def isCompressed(meta: Dict[str, str]) -> bool: """Return true if stream is compressed""" try: version = meta["stream-version"] == 2 except KeyError: version = meta["streamVersion"] == 2 if version: if meta["compressed"] is not False: return True return False ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/exceptions.py0000644000000000000000000000244615033267156021130 0ustar00rootroot""" Exceptions """ class CheckpointException(Exception): """Base checkpoint Exception""" class NoCheckpointsFound(CheckpointException): """Inc or differential backup attempted but no existing checkpoints are found.""" class RedefineCheckpointError(CheckpointException): """During redefining existing checkpoints after vm relocate, an error occurred""" class ReadCheckpointsError(CheckpointException): """Can't read checkpoint file""" class RemoveCheckpointError(CheckpointException): """During removal of existing checkpoints after an error occurred""" class SaveCheckpointError(CheckpointException): """Unable to append checkpoint to checkpoint file""" class ForeignCeckpointError(CheckpointException): """Checkpoint for vm found which was not created by virtnbdbackup""" class BackupException(Exception): """Base backup Exception""" class DiskBackupFailed(BackupException): """Backup of one disk failed""" class DiskBackupWriterException(BackupException): """Opening the target file writer failed""" class RestoreException(Exception): """Base restore Exception""" class UntilCheckpointReached(RestoreException): """Base restore Exception""" class RestoreError(RestoreException): """Base restore error Exception""" ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4864874 virtnbdbackup-2.30/libvirtnbdbackup/extenthandler/0000755000000000000000000000000015033267565021240 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/extenthandler/__init__.py0000644000000000000000000000140215033267156023342 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ __title__ = "extenthandler" __version__ = "0.1" from .extenthandler import ExtentHandler ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/extenthandler/extenthandler.py0000644000000000000000000002520415033267156024456 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging from typing import List, Any, Generator, Dict from nbd import CONTEXT_BASE_ALLOCATION from libvirtnbdbackup.objects import Extent, _ExtentObj from libvirtnbdbackup.common import humanize log = logging.getLogger("extenthandler") # pylint: disable=too-many-instance-attributes class ExtentHandler: """Query extent information about allocated and zeroed regions from the NBD server started by libvirt/qemu This implementation should return the same extent information as nbdinfo or qemu-img map """ def __init__(self, nbdFh, cType, no_sparse_detection: bool) -> None: self.useQemu = False self._maxRequestBlock: int = 4294967295 self._align: int = 512 self.lastExtentLen: int = 0 self.offset: int = 0 if nbdFh.__class__.__name__ == "util": self.useQemu = True self._nbdFh = nbdFh self._cType = cType self._extentEntries: Dict = {} self.no_sparse_detection = no_sparse_detection if cType.metaContext == "": self._metaContext = CONTEXT_BASE_ALLOCATION else: self._metaContext = cType.metaContext if self.useQemu is False: contexts = self._nbdFh.nbd.get_nr_meta_contexts() log.debug("NBD server exports [%d] metacontexts:", contexts) for i in range(0, contexts): ctx = self._nbdFh.nbd.get_meta_context(i) if self.no_sparse_detection is True and ctx == CONTEXT_BASE_ALLOCATION: continue self._extentEntries[ctx] = [] else: if self.no_sparse_detection is False: self._extentEntries[CONTEXT_BASE_ALLOCATION] = [] self._extentEntries[self._metaContext] = [] log.debug("Primary meta context for backup: %s", self._metaContext) def _getExtentCallback( self, metacontext: str, offset: int, entries: List, status: str ) -> None: """Callback function called by libnbd for each extent that is returned """ log.debug("Metacontext is: %s", metacontext) log.debug("Offset is: %s", offset) log.debug("Status is: %s", status) self.lastExtentLen = len(self._extentEntries[self._metaContext]) for entry in entries: self._extentEntries[metacontext].append(entry) log.debug("entries: %s", len(self._extentEntries[metacontext])) log.debug("Processed offsets: %s", self.offset) self.offset += sum( self._extentEntries[self._metaContext][self.lastExtentLen :: 2] ) self.lastExtentLen = len(self._extentEntries[self._metaContext]) def _setRequestAligment(self) -> int: """Align request size to nbd server""" align = self._nbdFh.nbd.get_block_size(0) if align == 0: align = self._align return self._maxRequestBlock - align + 1 def queryExtents(self) -> List[Any]: """Query extents either via qemu or custom extent handler""" if self.useQemu: return self.queryExtentsQemu() return self.queryExtentsNbd() def queryExtentsQemu(self) -> List[Any]: """Use qemu utils to query extents from nbd server""" extents = [] for ctx in iter(self._extentEntries): for extent in self._nbdFh.map(self._cType, ctx): extentObj = _ExtentObj(ctx, extent["length"], extent["type"]) extents.append(extentObj) log.debug("Got %s extents from qemu command", len(extents)) return extents def _extentsToObj(self) -> List[_ExtentObj]: """Go through extents and create a list of extent objects""" extentList = [] for context, values in self._extentEntries.items(): extentSizes = values[0::2] extentTypes = values[1::2] assert len(extentSizes) == len(extentTypes) ct = 0 while ct < len(extentSizes): extentObj = _ExtentObj(context, extentSizes[ct], extentTypes[ct]) extentList.append(extentObj) ct += 1 return extentList @staticmethod def _unifyExtents(extentObjects: List[_ExtentObj]) -> Generator: """Unify extents. If a sequence of extents has the same type (data or zero) it is better to unify them into a bigger block, so during backup, less requests to the nbd server have to be sent """ log.debug("Attempting to unify %s extents", len(extentObjects)) cur = None for myExtent in extentObjects: if cur is None: cur = myExtent elif cur.type == myExtent.type and cur.context == myExtent.context: cur.length += myExtent.length else: yield cur cur = myExtent yield cur def queryExtentsNbd(self) -> List[_ExtentObj]: """Request used blocks/extents from the nbd service""" maxRequestLen = self._setRequestAligment() size = self._nbdFh.nbd.get_size() while self.offset < size: if size < maxRequestLen: request_length = size else: request_length = min(size - self.offset, maxRequestLen) log.debug("Block status request length: %s", request_length) self._nbdFh.nbd.block_status( request_length, self.offset, self._getExtentCallback ) log.debug("Extents: %s", self._extentEntries) return self._extentsToObj() def setBlockType(self, context: str, blockType: int) -> bool: """Returns block type The extent types are as follows: For full backup: case 0 ("allocated") case 1: ("hole") case 2: ("zero") case 3: ("hole,zero") For checkpoint based inc/diff: case 0: ("clean") case 1: ("dirty") """ data = None if context == CONTEXT_BASE_ALLOCATION: assert blockType in (0, 1, 2, 3) if blockType == 0: data = True if blockType == 1: data = False elif blockType == 2: data = True elif blockType == 3: data = False else: assert blockType in (0, 1) data = bool(blockType) assert data is not None return data def overlap(self, extents: List[Extent]) -> List[Extent]: """Find overlaps between base allocation and incremental bitmap to detect zero regions""" base_extents = [e for e in extents if e.context == CONTEXT_BASE_ALLOCATION] backup_extents = [ e for e in extents if e.context == self._metaContext and e.data ] totalLength: int = 0 result = [] i = 0 # index for base_extents j = 0 # index for backup_extents while i < len(base_extents) and j < len(backup_extents): base = base_extents[i] backup = backup_extents[j] log.debug( "base: %d:%d(%s) bitmap: %d:%d(%s)", base.length, base.offset, str(base.data), backup.length, backup.offset, str(backup.data), ) # Skip if either extent has data=False or no real intersection if not base.data or base.offset + base.length <= backup.offset: i += 1 continue if not backup.data or backup.offset + backup.length <= base.offset: j += 1 continue offset = max(base.offset, backup.offset) end = min(backup.offset + backup.length, base.offset + base.length) log.debug("-->: %d:%d", offset, end - offset) result.append( Extent( context=base.context, data=True, offset=offset, length=end - offset, ) ) totalLength += end - offset # advance if end == base.offset + base.length: i += 1 if end == backup.offset + backup.length: j += 1 if totalLength > 0: log.info( "Detected %d bytes [%s] non-sparse blocks for current bitmap.", totalLength, humanize(totalLength), ) return result def queryBlockStatus(self) -> List[Extent]: """Check the status for each extent, whether if it is real data or zeroes, return a list of extent objects """ log.info("Start receiving backup extents.") extents = self.queryExtents() log.info("Finished receiving extents.") extentList: List[Extent] = [] start: int = 0 baseStart: int = 0 totalLength: int = 0 for extent in self._unifyExtents(extents): extObj = Extent( extent.context, self.setBlockType(extent.context, extent.type), baseStart if extent.context == CONTEXT_BASE_ALLOCATION else start, extent.length, ) extentList.append(extObj) if extent.context == CONTEXT_BASE_ALLOCATION: baseStart += extent.length else: start += extent.length if extObj.data: totalLength += extent.length log.debug( "%s %d %d %d", extObj.context, extObj.data, extObj.offset, extObj.offset + extObj.length, ) if self.no_sparse_detection is True: log.info("Skipping detection of sparse/fstrimmed blocks.") return extentList if self._metaContext != CONTEXT_BASE_ALLOCATION: log.debug("Detected [%d] bytes of changed data regions.", totalLength) extentList = self.overlap(extentList) return extentList ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/logcount.py0000644000000000000000000000241615033267156020576 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging class logCount(logging.Handler): """Custom log handler keeping track of issued log messages""" class LogType: """Log message type""" def __init__(self) -> None: self.warnings = 0 self.errors = 0 def __init__(self) -> None: super().__init__() self.count = self.LogType() def emit(self, record: logging.LogRecord) -> None: if record.levelno == logging.WARNING: self.count.warnings += 1 if record.levelno in (logging.ERROR, logging.CRITICAL): self.count.errors += 1 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/lz4.py0000644000000000000000000000234715033267156017460 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging import lz4.frame log = logging.getLogger() def decompressFrame(data: bytes) -> bytes: """Decompress lz4 frame, print frame information""" frameInfo = lz4.frame.get_frame_info(data) log.debug("Compressed Frame: %s", frameInfo) return lz4.frame.decompress(data) def compressFrame(data: bytes, level: int) -> bytes: """Compress block with to lz4 frame, checksums enabled for safety """ return lz4.frame.compress( data, content_checksum=True, block_checksum=True, compression_level=level, ) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4874876 virtnbdbackup-2.30/libvirtnbdbackup/map/0000755000000000000000000000000015033267565017150 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/map/__init__.py0000644000000000000000000000131615033267156021256 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ __title__ = "map" __version__ = "0.1" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/map/changes.py0000644000000000000000000000370715033267156021135 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import logging from argparse import Namespace from typing import List from libvirtnbdbackup import common as lib from libvirtnbdbackup import output def replay(dataRanges: List, args: Namespace) -> None: """Replay the changes from an incremental or differential backup file to the NBD device""" logging.info("Replaying changes from incremental backups") blockListInc = list( filter( lambda x: x["inc"] is True, dataRanges, ) ) dataSize = sum(extent["length"] for extent in blockListInc) progressBar = lib.progressBar(dataSize, "replaying..", args) with output.openfile(args.device, "wb") as replayDevice: for extent in blockListInc: if args.noprogress: logging.info( "Replaying offset %s from %s", extent["offset"], extent["file"] ) with output.openfile(os.path.abspath(extent["file"]), "rb") as replaySrc: replaySrc.seek(extent["offset"]) replayDevice.seek(extent["originalOffset"]) replayDevice.write(replaySrc.read(extent["length"])) replayDevice.seek(0) replayDevice.flush() progressBar.update(extent["length"]) progressBar.close() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/map/ranges.py0000644000000000000000000000723715033267156021006 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import logging import json from typing import List, Dict, Tuple, IO from libvirtnbdbackup import common as lib from libvirtnbdbackup import output from libvirtnbdbackup.output.exceptions import OutputException from libvirtnbdbackup.exceptions import RestoreError from libvirtnbdbackup.sparsestream.exceptions import StreamFormatException def _parse(stream, sTypes, reader) -> Tuple[List, Dict]: """Read block offsets from backup stream image""" try: kind, start, length = stream.readFrame(reader) meta = stream.loadMetadata(reader.read(length)) except StreamFormatException as errmsg: logging.error("Unable to read metadata header: %s", errmsg) raise RestoreError from errmsg if lib.isCompressed(meta): logging.error("Mapping compressed images currently not supported.") raise RestoreError assert reader.read(len(sTypes.TERM)) == sTypes.TERM dataRanges: List = [] count: int = 0 while True: kind, start, length = stream.readFrame(reader) if kind == sTypes.STOP: dataRanges[-1]["nextBlockOffset"] = None break blockInfo = {} blockInfo["count"] = count blockInfo["offset"] = reader.tell() blockInfo["originalOffset"] = start blockInfo["nextOriginalOffset"] = start + length blockInfo["length"] = length blockInfo["data"] = kind == sTypes.DATA blockInfo["file"] = os.path.abspath(reader.name) blockInfo["inc"] = meta["incremental"] if kind == sTypes.DATA: reader.seek(length, os.SEEK_CUR) assert reader.read(len(sTypes.TERM)) == sTypes.TERM nextBlockOffset = reader.tell() + sTypes.FRAME_LEN blockInfo["nextBlockOffset"] = nextBlockOffset dataRanges.append(blockInfo) count += 1 return dataRanges, meta def get(args, stream, sTypes, dataFiles: List) -> List: """Get data ranges for each file specified""" dataRanges = [] for dFile in dataFiles: try: reader = output.openfile(dFile, "rb") except OutputException as e: logging.error("[%s]: [%s]", dFile, e) raise RestoreError from e Range, meta = _parse(stream, sTypes, reader) if Range is False or meta is False: logging.error("Unable to read meta header from backup file.") raise RestoreError("Invalid header") dataRanges.extend(Range) if args.verbose is True: logging.info(json.dumps(dataRanges, indent=4)) else: logging.info( "Parsed [%s] block offsets from file [%s]", len(dataRanges), dFile ) reader.close() return dataRanges def dump(tfile: IO, dataRanges: List) -> bool: """Dump block map to temporary file""" try: tfile.write(json.dumps(dataRanges, indent=4).encode()) return True except OSError as e: logging.error("Unable to write blockmap file: %s", e) return False ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/map/requirements.py0000644000000000000000000000360515033267156022245 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import sys import shutil import logging from argparse import Namespace from libvirtnbdbackup import common as lib def executables() -> None: """Check if required utils are installed""" for exe in ("nbdkit", "qemu-nbd"): if not shutil.which(exe): logging.error("Please install required [%s] utility.", exe) def device(args: Namespace) -> None: """Check if /dev/nbdX exists, otherwise it is likely nbd module isn't loaded on the system""" if not args.device.startswith("/dev/nbd"): logging.error("Target device [%s] seems not to be an NBD device?", args.device) if not lib.exists(args, args.device): logging.error( "Target device [%s] does not exist, please load nbd module: [modprobe nbd]", args.device, ) def plugin(args: Namespace) -> str: """Attempt to locate the nbdkit plugin that is passed to the nbdkit process""" pluginFileName = "virtnbd-nbdkit-plugin" installDir = os.path.dirname(sys.argv[0]) nbdkitModule = f"{installDir}/{pluginFileName}" if not lib.exists(args, nbdkitModule): logging.error("Failed to locate nbdkit plugin: [%s]", pluginFileName) return nbdkitModule ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4884875 virtnbdbackup-2.30/libvirtnbdbackup/nbdcli/0000755000000000000000000000000015033267565017626 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/nbdcli/__init__.py0000644000000000000000000000146215033267156021736 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ __title__ = "nbdcli" __version__ = "0.1" from libvirtnbdbackup.objects import Unix, TCP from .client import client from . import context ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/nbdcli/client.py0000644000000000000000000001077215033267156021461 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import logging from time import sleep import nbd from libvirtnbdbackup.nbdcli import exceptions log = logging.getLogger("nbd") # pylint: disable=too-many-instance-attributes class client: """Helper functions for NBD""" def __init__(self, cType, no_sparse_detection: bool): """ Connect NBD backend """ self.cType = cType self._exportName = cType.exportName self._metaContext = "" if cType.metaContext != "": self._metaContext = cType.metaContext else: self._metaContext = nbd.CONTEXT_BASE_ALLOCATION self.maxRequestSize = 33554432 self.minRequestSize = 65536 self.no_sparse_detection = no_sparse_detection self.nbd = nbd.NBD() def debug(func, args): """Write NBD debugging messages to logfile instead of stderr""" log.debug("%s: %s", func, args) self.nbd.set_debug_callback(debug) self.connection = None def _getBlockInfo(self) -> None: """Read maximum request/block size as advertised by the nbd server. This is the value which will then be used by default """ maxSize = self.nbd.get_block_size(nbd.SIZE_MAXIMUM) if maxSize != 0: self.maxRequestSize = maxSize log.debug("Block size supported by NBD server: [%s]", maxSize) def _connect(self) -> nbd.NBD: """Setup connection to NBD server endpoint, return connection handle """ if self.cType.tls and not self.nbd.supports_tls(): raise exceptions.NbdConnectionError( "Installed python nbd binding is missing required tls features." ) try: if self.cType.tls: self.nbd.set_tls(nbd.TLS_REQUIRE) if self.no_sparse_detection is False: self.nbd.add_meta_context(nbd.CONTEXT_BASE_ALLOCATION) if self._metaContext != "": log.debug( "Adding meta context to NBD connection: [%s]", self._metaContext ) self.nbd.add_meta_context(self._metaContext) self.nbd.set_export_name(self._exportName) self.nbd.connect_uri(self.cType.uri) except nbd.Error as e: raise exceptions.NbdConnectionError(f"Unable to connect nbd server: {e}") self._getBlockInfo() return self.nbd def connect(self) -> nbd.NBD: """Wait until NBD endpoint connection can be established. It can take some time until qemu-nbd process is running and reachable. Attempt to connect and fail if no connection can be established. In case of unix domain socket, wait until socket file is created by qemu-nbd.""" log.info("Waiting until NBD server at [%s] is up.", self.cType.uri) retry = 0 maxRetry = 20 sleepTime = 1 while True: sleep(sleepTime) if retry >= maxRetry: raise exceptions.NbdConnectionTimeout( "Timeout during connection to NBD server backend." ) if self.cType.backupSocket and not os.path.exists(self.cType.backupSocket): log.info("Waiting for NBD Server unix socket, Retry: %s", retry) retry = retry + 1 continue try: connection = self._connect() except exceptions.NbdConnectionError as e: self.nbd = nbd.NBD() log.info("Waiting for NBD Server connection, Retry: %s [%s]", retry, e) retry = retry + 1 continue log.info("Connection to NBD backend succeeded.") self.connection = connection return self def disconnect(self) -> None: """Close nbd connection handle""" self.nbd.shutdown() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/nbdcli/context.py0000644000000000000000000000241715033267156021664 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging from argparse import Namespace from libvirtnbdbackup.virt.client import DomainDisk log = logging.getLogger("nbdctx") def get(args: Namespace, disk: DomainDisk) -> str: """Get required meta context string passed to nbd server based on backup type""" metaContext = "" if args.level not in ("inc", "diff"): return metaContext if args.offline is True: metaContext = f"qemu:dirty-bitmap:{args.cpt.name}" else: metaContext = f"qemu:dirty-bitmap:backup-{disk.target}" logging.debug("Using NBD meta context [%s]", metaContext) return metaContext ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/nbdcli/exceptions.py0000644000000000000000000000036015033267156022354 0ustar00rootroot""" Exceptions """ class NbdClientException(Exception): """Nbd exceptions""" class NbdConnectionError(NbdClientException): """Connection failed""" class NbdConnectionTimeout(NbdClientException): """Connection timed out""" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/objects.py0000644000000000000000000000461315033267156020376 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import ipaddress from dataclasses import dataclass @dataclass class processInfo: """Process info object returned by functions calling various qemu commands """ pid: int logFile: str err: str out: str pidFile: str @dataclass class DomainDisk: """Domain disk object holding information about the disk attached to a virtual machine""" target: str format: str filename: str path: str backingstores: list discardOption: str @dataclass class nbdConn: """NBD connection""" exportName: str metaContext: str @dataclass class Unix(nbdConn): """NBD connection type unix for connection via socket file""" backupSocket: str tls: bool = False def __post_init__(self): self.uri = f"nbd+unix:///{self.exportName}?socket={self.backupSocket}" @dataclass class TCP(nbdConn): """NBD connection type tcp for remote backup""" hostname: str tls: bool port: int = 10809 backupSocket: str = "" uri_prefix = "nbd://" def __post_init__(self): if self.tls: self.uri_prefix = "nbds://" try: ip = ipaddress.ip_address(self.hostname) if ip.version == 6: self.hostname = f"[{self.hostname}]" except ValueError: pass self.uri = f"{self.uri_prefix}{self.hostname}:{self.port}/{self.exportName}" @dataclass class Extent: """Extent description containing information if block contains data, offset and length of data to be read/written""" context: str data: bool offset: int length: int @dataclass class _ExtentObj: """Single Extent object as returned from the NBD server""" context: str length: int type: int ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4884875 virtnbdbackup-2.30/libvirtnbdbackup/output/0000755000000000000000000000000015033267565017733 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/output/__init__.py0000644000000000000000000000020415033267156022034 0ustar00rootroot"""Output helper class""" __title__ = "output" __version__ = "0.1" from .target import target openfile = target.Directory().open ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/output/exceptions.py0000644000000000000000000000037115033267156022463 0ustar00rootroot""" Exceptions """ class OutputException(Exception): """Outpuhelper exceptions""" class OutputOpenException(OutputException): """File open failed""" class OutputCreateDirectory(OutputException): """Can't create output directory""" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/output/stream.py0000644000000000000000000000227615033267156021603 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ from argparse import Namespace from typing import Union from libvirtnbdbackup import output def get( args: Namespace, repository: output.target ) -> Union[output.target.Directory, output.target.Zip]: """Get filehandle for output files based on output mode""" fileStream: Union[output.target.Directory, output.target.Zip] if args.stdout is False: fileStream = repository.Directory() else: fileStream = repository.Zip() args.output = "./" args.worker = 1 return fileStream ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/output/target.py0000644000000000000000000001431315033267156021571 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import sys import zlib import zipfile import logging import time import builtins from typing import IO, Union, Tuple, Any from libvirtnbdbackup.output import exceptions if sys.version_info >= (3, 8): from typing import Literal else: from typing_extensions import Literal log = logging.getLogger("output") class target: """Directs output stream to either regular directory or zipfile. If other formats are added class should be used as generic wrapper for open()/write()/close() functions. """ class Directory: """Backup to target directory""" def __init__(self) -> None: self.fileHandle: IO[Any] self.chksum: int = 1 def create(self, targetDir) -> None: """Create wrapper""" log.debug("Create: %s", targetDir) if os.path.exists(targetDir): if not os.path.isdir(targetDir): raise exceptions.OutputCreateDirectory( "Specified target is a file, not a directory" ) if not os.path.exists(targetDir): try: os.makedirs(targetDir) except OSError as e: raise exceptions.OutputCreateDirectory( f"Failed to create target directory: [{e}]" ) def open( self, targetFile: str, mode: Union[ Literal["w"], Literal["wb"], Literal["rb"], Literal["r"] ] = "wb", ) -> IO[Any]: """Open target file""" try: # pylint: disable=unspecified-encoding,consider-using-with self.fileHandle = builtins.open(targetFile, mode) return self.fileHandle except OSError as e: raise exceptions.OutputOpenException( f"Opening target file [{targetFile}] failed: {e}" ) from e def write(self, data: bytes) -> int: """Write wrapper""" self.chksum = zlib.adler32(data, self.chksum) written = self.fileHandle.write(data) assert written == len(data) return written def read(self, size=-1) -> int: """Read wrapper""" return self.fileHandle.read(size) def flush(self) -> None: """Flush wrapper""" return self.fileHandle.flush() def truncate(self, size: int) -> None: """Truncate target file""" try: self.fileHandle.truncate(size) self.fileHandle.seek(0) except OSError as e: raise exceptions.OutputException( f"Failed to truncate target file: [{e}]" ) from e def close(self) -> None: """Close wrapper""" log.debug("Close file") self.fileHandle.close() def seek(self, tgt: int, whence: int = 0) -> int: """Seek wrapper""" return self.fileHandle.seek(tgt, whence) def checksum(self) -> int: """Return computed checksum""" cur = self.chksum self.chksum = 1 return cur class Zip: """Backup to zip file""" def __init__(self) -> None: self.zipStream: zipfile.ZipFile self.zipFileStream: IO[bytes] log.info("Writing zip file stream to stdout") try: # pylint: disable=consider-using-with self.zipStream = zipfile.ZipFile( sys.stdout.buffer, "x", zipfile.ZIP_STORED ) except zipfile.error as e: raise exceptions.OutputOpenException( f"Failed to open zip file: {e}" ) from e def create(self, targetDir) -> None: """Create wrapper""" log.debug("Create: %s", targetDir) target.Directory().create(targetDir) def open(self, fileName: str, mode: Literal["w"] = "w") -> IO[bytes]: """Open wrapper""" zipFile = zipfile.ZipInfo( filename=os.path.basename(fileName), ) dateTime: time.struct_time = time.localtime(time.time()) timeStamp: Tuple[int, int, int, int, int, int] = ( dateTime.tm_year, dateTime.tm_mon, dateTime.tm_mday, dateTime.tm_hour, dateTime.tm_min, dateTime.tm_sec, ) zipFile.date_time = timeStamp zipFile.compress_type = zipfile.ZIP_STORED try: # pylint: disable=consider-using-with self.zipFileStream = self.zipStream.open( zipFile, mode, force_zip64=True ) return self.zipFileStream except zipfile.error as e: raise exceptions.OutputOpenException( f"Failed to open zip stream: {e}" ) from e return False def truncate(self, size: int) -> None: """Truncate target file""" raise RuntimeError("Not implemented") def write(self, data: bytes) -> int: """Write wrapper""" return self.zipFileStream.write(data) def close(self) -> None: """Close wrapper""" log.debug("Close file") self.zipFileStream.close() def checksum(self) -> None: """Checksum: not implemented for zip file""" return ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4894874 virtnbdbackup-2.30/libvirtnbdbackup/qemu/0000755000000000000000000000000015033267565017342 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/qemu/__init__.py0000644000000000000000000000131715033267156021451 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ __title__ = "qemu" __version__ = "0.2" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/qemu/command.py0000644000000000000000000000602415033267156021330 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging import tempfile import subprocess from typing import List, Tuple, Union from libvirtnbdbackup.qemu.exceptions import ( ProcessError, ) from libvirtnbdbackup.output import openfile from libvirtnbdbackup.objects import processInfo log = logging.getLogger(__name__) def _readlog(logFile: str, cmd: str) -> str: try: with openfile(logFile, "rb") as fh: return fh.read().decode().strip() except Exception as errmsg: log.exception(errmsg) raise ProcessError( f"Failed to execute [{cmd}]: Unable to get error message: {errmsg}" ) from errmsg def _readpipe(p) -> Tuple[str, str]: out = p.stdout.read().decode().strip() err = p.stderr.read().decode().strip() return out, err def run(cmdLine: List[str], pidFile: str = "", toPipe: bool = False) -> processInfo: """Execute passed command""" logFileName: str = "" logFile: Union[int, tempfile._TemporaryFileWrapper] if toPipe is True: logFile = subprocess.PIPE else: # pylint: disable=consider-using-with logFile = tempfile.NamedTemporaryFile( delete=False, prefix=cmdLine[0], suffix=".log" ) logFileName = logFile.name log.debug("CMD: %s", " ".join(cmdLine)) try: with subprocess.Popen( cmdLine, close_fds=True, stderr=logFile, stdout=logFile, ) as p: p.wait() log.debug("Return code: %s", p.returncode) err: str = "" out: str = "" if p.returncode != 0: log.error("CMD: %s", " ".join(cmdLine)) log.debug("Read error messages from logfile") if toPipe is True: out, err = _readpipe(p) else: err = _readlog(logFileName, cmdLine[0]) raise ProcessError(f"Unable to start [{cmdLine[0]}] error: [{err}]") if toPipe is True: out, err = _readpipe(p) if pidFile != "": realPid = int(_readlog(pidFile, "")) else: realPid = p.pid process = processInfo(realPid, logFileName, err, out, pidFile) log.debug("Started [%s] process: [%s]", cmdLine[0], process) except FileNotFoundError as e: raise ProcessError(e) from e return process ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/qemu/exceptions.py0000644000000000000000000000041415033267156022070 0ustar00rootroot""" Exceptions """ class QemuHelperError(Exception): """Errors during qemu helper""" class NbdServerProcessError(QemuHelperError): """Unable to start nbd server for offline backup""" class ProcessError(QemuHelperError): """Unable to start process""" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/qemu/util.py0000644000000000000000000002021515033267156020665 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import json import logging import tempfile import subprocess from typing import List from argparse import Namespace from libvirtnbdbackup.ssh.exceptions import sshError from libvirtnbdbackup.objects import processInfo from libvirtnbdbackup.qemu import command from libvirtnbdbackup.virt.client import DomainDisk log = logging.getLogger(__name__) class util: """Wrapper for qemu executables""" def __init__(self, exportName: str) -> None: self.exportName = exportName @staticmethod def map(cType, context: str) -> str: """Read extent map using nbdinfo utility""" cmd = f"nbdinfo --json --map={context} '{cType.uri}'" log.debug("Starting CMD: [%s]", cmd) extentMap = subprocess.run( cmd, shell=True, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) return json.loads(extentMap.stdout) @staticmethod def create( # pylint: disable=R0913: disable=R0913 args: Namespace, targetFile: str, fileSize: int, diskFormat: str, qcowOptions: list, sshClient=None, ) -> processInfo: """Create the target qcow image""" fileParam = f"{targetFile}" if sshClient: fileParam = f"'{targetFile}'" cmd = [ "qemu-img", "create", "-f", f"{diskFormat}", fileParam, "-o", f"size={fileSize}", ] if args.preallocate: cmd.append("-o") cmd.append("preallocation=full") if qcowOptions: cmd = cmd + qcowOptions if not sshClient: return command.run(cmd) return sshClient.run(" ".join(cmd)) def info(self, targetFile: str, sshClient=None) -> processInfo: """Return qemu image information""" fileParam = f"{targetFile}" if sshClient: fileParam = f"'{targetFile}'" cmd = [ "qemu-img", "info", fileParam, "--output", "json", "--force-share", ] if not sshClient: return command.run(cmd, toPipe=True) return sshClient.run(" ".join(cmd)) def startRestoreNbdServer(self, targetFile: str, socketFile: str) -> processInfo: """Start local nbd server process for restore operation""" pidFile = self._gt("qemu-nbd", ".pid") cmd = [ "qemu-nbd", "--discard=unmap", "--format=qcow2", "-x", f"{self.exportName}", f"{targetFile}", "-k", f"{socketFile}", "--pid-file", f"{pidFile}", "--fork", ] return command.run(cmd, pidFile=pidFile) @staticmethod def _gt(prefix: str, suffix: str, delete: bool = False) -> str: """Create named temporary file.""" with tempfile.NamedTemporaryFile( delete=delete, prefix=prefix, suffix=suffix ) as tf1: return tf1.name @staticmethod def _addTls(cmd: List[str], certpath: str) -> None: """Add required tls related options to qemu-nbd command line.""" cmd.append("--object") cmd.append( f"tls-creds-x509,id=tls0,endpoint=server,dir={certpath},verify-peer=false" ) cmd.append("--tls-creds tls0") def startRemoteRestoreNbdServer( self, args: Namespace, targetFile: str ) -> processInfo: """Start nbd server process remotely over ssh for restore operation""" pidFile = self._gt("qemu-nbd-restore", ".pid") logFile = self._gt("qemu-nbd-restore", ".log") cmd = [ "qemu-nbd", "--discard=unmap", "--format=qcow2", "-x", f"{self.exportName}", f"'{targetFile}'", "-p", f"{args.nbd_port}", "--pid-file", f"{pidFile}", "--fork", ] if args.tls is True: self._addTls(cmd, args.tls_cert) if args.nbd_ip != "": cmd.append("-b") cmd.append(args.nbd_ip) cmd.append(f"> {logFile} 2>&1") try: return args.sshClient.run(" ".join(cmd), pidFile, logFile) except sshError: log.error("Executing command failed: check [%s] for errors.", logFile) raise def startNbdkitProcess( self, args: Namespace, nbdkitModule: str, blockMap, fullImage: str ) -> processInfo: """Execute nbdkit process for virtnbdmap""" debug = "0" hexdump = "0" pidFile = self._gt("nbdkit", ".pid") if args.verbose: debug = "1" if args.hexdump: hexdump = "1" debug = "1" cmd = [ "nbdkit", "--pidfile", f"{pidFile}", "-i", f"{args.listen_address}", "-p", f"{args.listen_port}", "-e", f"{self.exportName}", "--filter=blocksize", "--filter=cow", "-v", "python", f"{nbdkitModule}", f"maxlen={args.blocksize}", f"blockmap={blockMap}", f"disk={fullImage}", f"debug={debug}", f"hexdump={hexdump}", "-t", f"{args.threads}", ] return command.run(cmd, pidFile=pidFile) def startBackupNbdServer( self, diskFormat: str, diskFile: str, socketFile: str, bitMap: str ) -> processInfo: """Start nbd server process for offline backup operation""" bitmapOpt = "--" if bitMap != "": bitmapOpt = f"--bitmap={bitMap}" pidFile = f"{socketFile}.pid" cmd = [ "qemu-nbd", "-r", f"--format={diskFormat}", "-x", f"{self.exportName}", f"{diskFile}", "-k", f"{socketFile}", "-t", "-e 2", "--fork", "--detect-zeroes=on", f"--pid-file={pidFile}", bitmapOpt, ] return command.run(cmd, pidFile=pidFile) def startRemoteBackupNbdServer( self, args: Namespace, disk: DomainDisk, bitMap: str, port: int ) -> processInfo: """Start nbd server process remotely over ssh for restore operation""" pidFile = self._gt("qemu-nbd-backup", ".pid") logFile = self._gt("qemu-nbd-backup", ".log") cmd = [ "qemu-nbd", "-r", f"--format={disk.format}", "-x", f"{self.exportName}", f"'{disk.path}'", "-p", f"{port}", "--pid-file", f"{pidFile}", "--fork", ] if args.nbd_ip != "": cmd.append("-b") cmd.append(args.nbd_ip) if bitMap != "": cmd.append(f"--bitmap={bitMap}") if args.tls is True: self._addTls(cmd, args.tls_cert) cmd.append(f"> {logFile} 2>&1") try: return args.sshClient.run(" ".join(cmd), pidFile, logFile) except sshError: log.error("Executing command failed: check [%s] for errors.", logFile) raise def disconnect(self, device: str) -> processInfo: """Disconnect device""" log.info("Disconnecting device [%s]", device) cmd = ["qemu-nbd", "-d", f"{device}"] return command.run(cmd) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4914875 virtnbdbackup-2.30/libvirtnbdbackup/restore/0000755000000000000000000000000015033267565020056 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/restore/__init__.py0000644000000000000000000000132215033267156022161 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ __title__ = "restore" __version__ = "0.1" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/restore/data.py0000644000000000000000000001354515033267156021345 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging import pprint from argparse import Namespace from libvirtnbdbackup import chunk from libvirtnbdbackup import lz4 from libvirtnbdbackup import common as lib from libvirtnbdbackup.sparsestream import types from libvirtnbdbackup.sparsestream import streamer from libvirtnbdbackup.sparsestream.exceptions import StreamFormatException from libvirtnbdbackup.exceptions import RestoreError from libvirtnbdbackup.exceptions import UntilCheckpointReached def restore( args: Namespace, stream: streamer.SparseStream, disk: str, targetFile: str, connection, ) -> bool: """Restore the data stream to the target file""" diskState = False diskState = _write(args, stream, disk, targetFile, connection) # no data has been processed if diskState is None: diskState = True return diskState def _write( # pylint: disable=too-many-branches,too-many-locals,too-many-statements args: Namespace, stream: streamer.SparseStream, dataFile: str, targetFile: str, connection, ) -> bool: """Restore data for disk""" sTypes = types.SparseStreamTypes() try: # pylint: disable=consider-using-with reader = open(dataFile, "rb") except OSError as errmsg: logging.error("Failed to open backup file for reading: [%s].", errmsg) raise RestoreError from errmsg try: kind, start, length = stream.readFrame(reader) meta = stream.loadMetadata(reader.read(length)) except StreamFormatException as errmsg: logging.fatal(errmsg) raise RestoreError from errmsg trailer = None if lib.isCompressed(meta) is True: trailer = stream.readCompressionTrailer(reader) logging.info("Found compression trailer.") logging.debug("%s", trailer) if meta["dataSize"] == 0: logging.info("File [%s] contains no dirty blocks, skipping.", dataFile) if meta["checkpointName"] == args.until: logging.info("Reached checkpoint [%s], stopping", args.until) raise UntilCheckpointReached return True logging.info( "Applying data from backup file [%s] to target file [%s].", dataFile, targetFile ) pprint.pprint(meta) assert reader.read(len(sTypes.TERM)) == sTypes.TERM progressBar = lib.progressBar( meta["dataSize"], f"restoring disk [{meta['diskName']}]", args ) dataSize: int = 0 dataBlockCnt: int = 0 while True: try: kind, start, length = stream.readFrame(reader) except StreamFormatException as err: logging.error("Can't read stream at pos: [%s]: [%s]", reader.tell(), err) raise RestoreError from err if kind == sTypes.ZERO: logging.debug("Zero segment from [%s] length: [%s]", start, length) elif kind == sTypes.DATA: logging.debug( "Processing data segment from [%s] length: [%s]", start, length ) originalSize = length if trailer: logging.debug("Block: [%s]", dataBlockCnt) logging.debug("Original block size: [%s]", length) length = trailer[dataBlockCnt] logging.debug("Compressed block size: [%s]", length) if originalSize >= connection.maxRequestSize: logging.debug( "Chunked read/write, start: [%s], len: [%s]", start, length ) try: written = chunk.read( reader, start, length, connection, lib.isCompressed(meta), progressBar, ) except Exception as e: logging.exception(e) raise RestoreError from e logging.debug("Wrote: [%s]", written) else: try: data = reader.read(length) if lib.isCompressed(meta): data = lz4.decompressFrame(data) connection.nbd.pwrite(data, start) written = len(data) except Exception as e: logging.exception(e) raise RestoreError from e progressBar.update(written) assert reader.read(len(sTypes.TERM)) == sTypes.TERM dataSize += originalSize dataBlockCnt += 1 elif kind == sTypes.STOP: progressBar.close() if dataSize != meta["dataSize"]: logging.error( "Restored data size does not match [%s] != [%s]", dataSize, meta["dataSize"], ) raise RestoreError("Data size mismatch") break logging.info("End of stream, [%s] of data processed", lib.humanize(dataSize)) if meta["checkpointName"] == args.until: logging.info("Reached checkpoint [%s], stopping", args.until) raise UntilCheckpointReached if connection.nbd.can_flush() is True: logging.debug("Flushing NBD connection handle") connection.nbd.flush() return True ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/restore/disk.py0000644000000000000000000001145115033267156021360 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging from argparse import Namespace from libvirtnbdbackup import virt from libvirtnbdbackup import common as lib from libvirtnbdbackup.objects import DomainDisk from libvirtnbdbackup.restore import server from libvirtnbdbackup.restore import files from libvirtnbdbackup.restore import image from libvirtnbdbackup.restore import header from libvirtnbdbackup.restore import data from libvirtnbdbackup.restore import vmconfig from libvirtnbdbackup.sparsestream import types from libvirtnbdbackup.sparsestream import streamer from libvirtnbdbackup.exceptions import RestoreError, UntilCheckpointReached from libvirtnbdbackup.nbdcli.exceptions import NbdConnectionTimeout def _backingstore(args: Namespace, disk: DomainDisk) -> None: """If an virtual machine was running on an snapshot image, warn user, the virtual machine configuration has to be adjusted before starting the VM is possible. User created external or internal Snapshots are not part of the backup. """ if len(disk.backingstores) > 0 and not args.adjust_config: logging.warning( "Target image [%s] seems to be a snapshot image.", disk.filename ) logging.warning("Target virtual machine configuration must be altered!") logging.warning("Configured backing store images must be changed.") def restore( # pylint: disable=too-many-branches,too-many-statements,too-many-locals args: Namespace, ConfigFile: str, virtClient: virt.client ) -> bytes: """Handle disk restore operation and adjust virtual machine configuration accordingly.""" stream = streamer.SparseStream(types) vmConfig = vmconfig.read(ConfigFile) vmDisks = virtClient.getDomainDisks(args, vmConfig) if not vmDisks: raise RestoreError("Unable to parse disks from config") restConfig: bytes = vmConfig.encode() for disk in vmDisks: if args.disk not in (None, disk.target): logging.info("Skipping disk [%s] for restore", disk.target) continue restoreDisk = lib.getLatest(args.input, f"{disk.target}*.data") logging.debug("Restoring disk: [%s]", restoreDisk) if len(restoreDisk) < 1: logging.warning( "No backup file for disk [%s] found, assuming it has been excluded.", disk.target, ) if args.adjust_config is True: restConfig = vmconfig.removeDisk(restConfig.decode(), disk.target) continue targetFile = files.target(args, disk) if args.raw and disk.format == "raw": logging.info("Restoring raw image to [%s]", targetFile) lib.copy(args, restoreDisk[0], targetFile) continue if "full" not in restoreDisk[0] and "copy" not in restoreDisk[0]: logging.error( "[%s]: Unable to locate base full or copy backup.", restoreDisk[0] ) raise RestoreError("Failed to locate backup.") cptnum = -1 if args.until is not None: cptnum = int(args.until.split(".")[-1]) meta = header.get(restoreDisk[cptnum], stream) try: image.create(args, meta, targetFile, args.sshClient) except RestoreError as errmsg: raise RestoreError("Creating target image failed.") from errmsg try: connection = server.start(args, meta["diskName"], targetFile, virtClient) except NbdConnectionTimeout as e: raise RestoreError(e) from e for dataFile in restoreDisk: try: data.restore(args, stream, dataFile, targetFile, connection) except UntilCheckpointReached: break except RestoreError: break _backingstore(args, disk) if args.adjust_config is True: restConfig = vmconfig.adjust(disk, restConfig.decode(), targetFile) logging.debug("Closing NBD connection") connection.disconnect() if args.adjust_config is True: restConfig = vmconfig.removeUuid(restConfig.decode()) restConfig = vmconfig.setVMName(args, restConfig.decode()) return restConfig ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/restore/files.py0000644000000000000000000001054715033267156021535 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import zlib import json import logging from typing import List from argparse import Namespace from libvirtnbdbackup import virt from libvirtnbdbackup import output from libvirtnbdbackup.restore import vmconfig from libvirtnbdbackup.restore import header from libvirtnbdbackup import common as lib from libvirtnbdbackup.objects import DomainDisk from libvirtnbdbackup.sparsestream import streamer from libvirtnbdbackup.exceptions import RestoreError def restore(args: Namespace, vmConfig: str, virtClient: virt.client) -> None: """Notice user if backed up vm had loader / nvram""" config = vmconfig.read(vmConfig) info = virtClient.getDomainInfo(config) for setting, val in info.items(): f = lib.getLatest(args.input, f"*{os.path.basename(val)}*", -1) if lib.exists(args, val): logging.info( "File [%s]: for boot option [%s] already exists, skipping.", val, setting, ) continue logging.info( "Restoring configured file [%s] for boot option [%s]", val, setting ) lib.copy(args, f[0], val) def verify(args: Namespace, dataFiles: List[str]) -> bool: """Compute adler32 checksum for exiting data files and compare with checksums computed during backup.""" for dataFile in dataFiles: if args.disk is not None and not os.path.basename(dataFile).startswith( args.disk ): continue logging.debug("Using buffer size: %s", args.buffsize) logging.info("Computing checksum for: %s", dataFile) sourceFile = dataFile if args.sequence: sourceFile = os.path.join(args.input, dataFile) with output.openfile(sourceFile, "rb") as vfh: adler = 1 data = vfh.read(args.buffsize) while data: adler = zlib.adler32(data, adler) data = vfh.read(args.buffsize) chksumFile = f"{sourceFile}.chksum" logging.info("Checksum result: %s", adler) if not os.path.exists(chksumFile): logging.info("No checksum found, skipping: [%s]", sourceFile) continue logging.info("Comparing checksum with stored information") with output.openfile(chksumFile, "r") as s: storedSum = int(s.read()) if storedSum != adler: logging.error("Stored sums do not match: [%s]!=[%s]", storedSum, adler) return False logging.info("OK") return True def dump(args: Namespace, stream: streamer.SparseStream, dataFiles: List[str]) -> bool: """Dump stream contents to json output""" logging.info("Dumping saveset meta information") entries = [] for dataFile in dataFiles: if args.disk is not None and not os.path.basename(dataFile).startswith( args.disk ): continue logging.info(dataFile) sourceFile = dataFile if args.sequence: sourceFile = os.path.join(args.input, dataFile) try: meta = header.get(sourceFile, stream) except RestoreError as e: logging.error(e) continue entries.append(meta) if lib.isCompressed(meta): logging.info("Compressed stream found: [%s].", meta["compressionMethod"]) print(json.dumps(entries, indent=4)) return True def target(args: Namespace, disk: DomainDisk) -> str: """Based on disk information, return target file to create during restore.""" if disk.filename is not None: targetFile = os.path.join(args.output, disk.filename) else: targetFile = os.path.join(args.output, disk.target) return targetFile ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/restore/header.py0000644000000000000000000000272515033267156021662 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ from typing import Dict from libvirtnbdbackup import common as lib from libvirtnbdbackup.sparsestream import streamer from libvirtnbdbackup.sparsestream.exceptions import StreamFormatException from libvirtnbdbackup.exceptions import RestoreError from libvirtnbdbackup.output.exceptions import OutputException def get(diskFile: str, stream: streamer.SparseStream) -> Dict[str, str]: """Read header from data file""" try: return lib.dumpMetaData(diskFile, stream) except StreamFormatException as errmsg: raise RestoreError( f"Reading metadata from [{diskFile}] failed: [{errmsg}]" ) from errmsg except OutputException as errmsg: raise RestoreError( f"Reading data file [{diskFile}] failed: [{errmsg}]" ) from errmsg ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/restore/image.py0000644000000000000000000000737515033267156021522 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import logging import json from argparse import Namespace from typing import List, Dict from libvirtnbdbackup.qemu import util as qemu from libvirtnbdbackup import output from libvirtnbdbackup import common as lib from libvirtnbdbackup.exceptions import RestoreError from libvirtnbdbackup.qemu.exceptions import ProcessError from libvirtnbdbackup.output.exceptions import OutputException from libvirtnbdbackup.ssh.exceptions import sshError def getConfig(args: Namespace, meta: Dict[str, str]) -> List[str]: """Check if backup includes exported qcow config and return a list of options passed to qemu-img create command""" opt: List[str] = [] qcowConfig = None qcowConfigFile = lib.getLatest(args.input, f"{meta['diskName']}*.qcow.json*", -1) if not qcowConfigFile: logging.warning("No qcow image config found, will use default options.") return opt lastConfigFile = qcowConfigFile[0] try: with output.openfile(lastConfigFile, "rb") as qFh: qcowConfig = json.loads(qFh.read().decode()) logging.info("Using QCOW options from backup file: [%s]", lastConfigFile) except ( OutputException, json.decoder.JSONDecodeError, ) as errmsg: logging.warning( "Unable to load original QCOW image config, using defaults: [%s].", errmsg, ) return opt try: opt.append("-o") opt.append(f"compat={qcowConfig['format-specific']['data']['compat']}") except KeyError as errmsg: logging.warning("Unable apply QCOW specific compat option: [%s]", errmsg) try: opt.append("-o") opt.append(f"cluster_size={qcowConfig['cluster-size']}") except KeyError as errmsg: logging.warning("Unable apply QCOW specific cluster_size option: [%s]", errmsg) try: if qcowConfig["format-specific"]["data"]["lazy-refcounts"]: opt.append("-o") opt.append("lazy_refcounts=on") except KeyError as errmsg: logging.warning( "Unable apply QCOW specific lazy_refcounts option: [%s]", errmsg ) return opt def create(args: Namespace, meta: Dict[str, str], targetFile: str, sshClient): """Create target image file""" logging.info( "Create virtual disk [%s] format: [%s] size: [%s] based on: [%s] preallocated: [%s]", targetFile, meta["diskFormat"], meta["virtualSize"], meta["checkpointName"], args.preallocate, ) options = getConfig(args, meta) if lib.exists(args, targetFile): logging.error( "Target file already exists: [%s], won't overwrite.", os.path.abspath(targetFile), ) raise RestoreError qFh = qemu.util(meta["diskName"]) try: qFh.create( args, targetFile, int(meta["virtualSize"]), meta["diskFormat"], options, sshClient, ) except (ProcessError, sshError) as e: logging.error("Failed to create restore target: [%s]", e) raise RestoreError from e ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/restore/sequence.py0000644000000000000000000000410515033267156022234 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os from typing import List from argparse import Namespace from libvirtnbdbackup import virt from libvirtnbdbackup import common as lib from libvirtnbdbackup.restore import header from libvirtnbdbackup.restore import server from libvirtnbdbackup.restore import image from libvirtnbdbackup.restore import data from libvirtnbdbackup.sparsestream import types from libvirtnbdbackup.sparsestream import streamer from libvirtnbdbackup.exceptions import RestoreError def restore(args: Namespace, dataFiles: List[str], virtClient: virt.client) -> bool: """Reconstruct image from a given set of data files""" stream = streamer.SparseStream(types) result: bool = False sourceFile = os.path.join(args.input, dataFiles[-1]) meta = header.get(sourceFile, stream) if not meta: return result diskName = meta["diskName"] targetFile = os.path.join(args.output, diskName) if lib.exists(args, targetFile): raise RestoreError(f"Targetfile {targetFile} already exists.") try: image.create(args, meta, targetFile, args.sshClient) except RestoreError as errmsg: raise errmsg connection = server.start(args, diskName, targetFile, virtClient) for disk in dataFiles: sourceFile = os.path.join(args.input, disk) result = data.restore(args, stream, sourceFile, targetFile, connection) connection.disconnect() return result ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/restore/server.py0000644000000000000000000000501215033267156021730 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging from argparse import Namespace from typing import Union from libvirtnbdbackup import nbdcli from libvirtnbdbackup import virt from libvirtnbdbackup.qemu import util as qemu from libvirtnbdbackup.ssh.exceptions import sshError from libvirtnbdbackup.qemu.exceptions import ProcessError from libvirtnbdbackup.exceptions import RestoreError log = logging.getLogger("restore") def setup(args: Namespace, exportName: str, targetFile: str, virtClient: virt.client): """Setup NBD process required for restore, either remote or local""" qFh = qemu.util(exportName) cType: Union[nbdcli.TCP, nbdcli.Unix] if not virtClient.remoteHost: logging.info("Starting local NBD server on socket: [%s]", args.socketfile) proc = qFh.startRestoreNbdServer(targetFile, args.socketfile) cType = nbdcli.Unix(exportName, "", args.socketfile) else: remoteIP = virtClient.remoteHost if args.nbd_ip != "": remoteIP = args.nbd_ip logging.info( "Starting remote NBD server on socket: [%s:%s]", remoteIP, args.nbd_port, ) proc = qFh.startRemoteRestoreNbdServer(args, targetFile) cType = nbdcli.TCP(exportName, "", remoteIP, args.tls, args.nbd_port) nbdClient = nbdcli.client(cType, False) logging.info("Started NBD server, PID: [%s]", proc.pid) return nbdClient.connect() def start(args: Namespace, diskName: str, targetFile: str, virtClient: virt.client): """Start NDB Service""" try: return setup(args, diskName, targetFile, virtClient) except ProcessError as errmsg: logging.error(errmsg) raise RestoreError("Failed to start local NBD server.") from errmsg except sshError as errmsg: logging.error(errmsg) raise RestoreError("Failed to start remote NBD server.") from errmsg ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/restore/vmconfig.py0000644000000000000000000001274115033267156022241 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import tempfile import logging from argparse import Namespace from libvirtnbdbackup import output from libvirtnbdbackup import common as lib from libvirtnbdbackup.objects import DomainDisk from libvirtnbdbackup.virt import xml from libvirtnbdbackup.virt import disktype def read(ConfigFile: str) -> str: """Read saved virtual machine config'""" try: return output.openfile(ConfigFile, "rb").read().decode() except: logging.error("Can't read config file: [%s]", ConfigFile) raise def removeDisk(vmConfig: str, excluded) -> bytes: """Remove disk from config, in case it has been excluded from the backup.""" tree = xml.asTree(vmConfig) logging.info("Removing excluded disk [%s] from vm config.", excluded) try: target = tree.xpath(f"devices/disk/target[@dev='{excluded}']")[0] disk = target.getparent() disk.getparent().remove(disk) except IndexError: logging.warning("Removing excluded disk from config failed: no object found.") return xml.ElementTree.tostring(tree, encoding="utf8", method="xml") def removeUuid(vmConfig: str) -> bytes: """Remove the auto generated UUID from the config file to allow for restore into new name""" tree = xml.asTree(vmConfig) try: logging.info("Removing uuid setting from vm config.") uuid = tree.xpath("uuid")[0] tree.remove(uuid) except IndexError: pass return xml.ElementTree.tostring(tree, encoding="utf8", method="xml") def setVMName(args: Namespace, vmConfig: str) -> bytes: """Change / set the VM name to be restored""" tree = xml.asTree(vmConfig) name = tree.xpath("name")[0] if args.name is None and not name.text.startswith("restore"): domainName = f"restore_{name.text}" logging.info("Change VM name from [%s] to [%s]", name.text, domainName) name.text = domainName else: logging.info("Set name from [%s] to [%s]", name.text, args.name) name.text = args.name return xml.ElementTree.tostring(tree, encoding="utf8", method="xml") def adjust(restoreDisk: DomainDisk, vmConfig: str, targetFile: str) -> bytes: """Adjust virtual machine configuration after restoring. Changes the paths to the virtual machine disks and attempts to remove components excluded during restore.""" tree = xml.asTree(vmConfig) for disk in tree.xpath("devices/disk"): if disk.get("type") == "volume": logging.info("Disk has type volume, resetting to type file.") disk.set("type", "file") dev = disk.xpath("target")[0].get("dev") logging.debug("Handling target device: [%s]", dev) device = disk.get("device") driver = disk.xpath("driver")[0].get("type") if disktype.Optical(device, dev): logging.info("Removing device [%s], type [%s] from vm config", dev, device) disk.getparent().remove(disk) continue if disktype.Raw(driver, device): logging.warning( "Removing raw disk [%s] from vm config, use --raw to copy as is.", dev, ) disk.getparent().remove(disk) continue backingStore = disk.xpath("backingStore") if backingStore: logging.info("Removing existent backing store settings") disk.remove(backingStore[0]) originalFile = disk.xpath("source")[0].get("file") if dev == restoreDisk.target: abspath = os.path.abspath(targetFile) logging.info( "Change target file for disk [%s] from [%s] to [%s]", restoreDisk.target, originalFile, abspath, ) disk.xpath("source")[0].set("file", abspath) return xml.ElementTree.tostring(tree, encoding="utf8", method="xml") def restore( args: Namespace, vmConfig: str, adjustedConfig: bytes, targetFileName: str, ) -> None: """Restore either original or adjusted vm configuration to new directory""" targetFile = os.path.join(args.output, os.path.basename(targetFileName)) if args.adjust_config is True: if args.sshClient: with tempfile.NamedTemporaryFile(delete=True) as fh: fh.write(adjustedConfig) lib.copy(args, fh.name, targetFile) else: with output.openfile(targetFile, "wb") as cnf: cnf.write(adjustedConfig) logging.info("Adjusted config placed in: [%s]", targetFile) if args.define is False: logging.info("Use 'virsh define %s' to define VM", targetFile) else: lib.copy(args, vmConfig, targetFile) logging.info("Copied original vm config to [%s]", targetFile) logging.info("Note: virtual machine config must be adjusted manually.") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/sighandle.py0000644000000000000000000000525215033267156020703 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import sys from argparse import Namespace from typing import Any from libvirt import virDomain from libvirtnbdbackup import virt from libvirtnbdbackup import common as lib from libvirtnbdbackup.objects import processInfo from libvirtnbdbackup.qemu import util as qemu class Backup: """Handle signal during backup operation""" @staticmethod def catch( args: Namespace, domObj: virDomain, virtClient: virt.client, log: Any, signum: int, _, ) -> None: """Catch signal, attempt to stop running backup job.""" log.error("Signal caught: %s", signum) if args.offline is True: log.error("Exiting.") sys.exit(1) if virtClient.remoteHost != "": log.info("Reconnecting remote system to stop backup job.") try: virtClient = virt.client(args) domObj = virtClient.getDomain(args.domain) except virt.exceptions.connectionFailed as e: log.error("Reconnecting remote host failed: [%s]", e) log.error("Unable to stop backup job on remote system.", e) sys.exit(1) log.info("Cleanup: Stopping backup job.") virtClient.stopBackup(domObj) virtClient.close() sys.exit(1) class Map: """Handle signal during map operation""" @staticmethod def catch( args: Namespace, nbdkitProcess: processInfo, blockMap, log: Any, signum, _, ): """Catch signal, attempt to stop processes.""" log.info("Received signal: [%s]", signum) qemu.util("").disconnect(args.device) if not args.verbose: log.info("Removing temporary blockmap file: [%s]", blockMap.name) os.remove(blockMap.name) log.info("Removing nbdkit logfile: [%s]", nbdkitProcess.logFile) os.remove(nbdkitProcess.logFile) lib.killProc(nbdkitProcess.pid) sys.exit(0) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4924874 virtnbdbackup-2.30/libvirtnbdbackup/sparsestream/0000755000000000000000000000000015033267565021104 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/sparsestream/__init__.py0000644000000000000000000000132715033267156023214 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ __title__ = "sparsestream" __version__ = "0.1" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/sparsestream/exceptions.py0000644000000000000000000000054115033267156023633 0ustar00rootroot""" Exceptions """ class StreamFormatException(Exception): """Wrong metadata header""" class MetaHeaderFormatException(StreamFormatException): """Wrong metadata header""" class BlockFormatException(StreamFormatException): """Wrong metadata header""" class FrameformatException(StreamFormatException): """Frame Format is wrong""" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/sparsestream/streamer.py0000644000000000000000000001200515033267156023272 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier Copyright (C) 2020 Red Hat, Inc. This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import json import os import datetime from typing import List, Any, Tuple, Dict from argparse import Namespace from libvirtnbdbackup.objects import DomainDisk from libvirtnbdbackup.sparsestream import exceptions class SparseStream: """Sparse Stream writer/reader class""" def __init__(self, types, version: int = 2) -> None: """Stream version: 1: base version 2: stream version with compression support """ self.version = version self.compressionMethod: str = "lz4" self.types = types.SparseStreamTypes() def dumpMetadata( self, args: Namespace, virtualSize: int, dataSize: int, disk: DomainDisk, ) -> bytes: """First block in backup stream is Meta data information about virtual size of the disk being backed up, as well as various information regarding backup. Dumps Metadata frame to be written at start of stream in json format. """ meta = { "virtualSize": virtualSize, "dataSize": dataSize, "date": datetime.datetime.now().isoformat(), "diskName": disk.target, "diskFormat": disk.format, "checkpointName": args.cpt.name, "compressed": args.compress, "compressionMethod": self.compressionMethod, "parentCheckpoint": args.cpt.parent, "incremental": (args.level in ("inc", "diff")), "streamVersion": self.version, } return json.dumps(meta, indent=4).encode("utf-8") def writeCompressionTrailer(self, writer, trailer: List[Any]) -> None: """Dump compression trailer to end of stream""" size = writer.write(json.dumps(trailer).encode()) writer.write(self.types.TERM) self.writeFrame(writer, self.types.COMP, 0, size) def _readHeader(self, reader) -> Tuple[str, str, str]: """Attempt to read header""" header = reader.read(self.types.FRAME_LEN) try: kind, start, length = header.split(b" ", 2) except ValueError as err: raise exceptions.BlockFormatException( f"Invalid block format: [{err}]" ) from err return kind, start, length @staticmethod def _parseHeader(kind, start: str, length: str) -> Tuple[str, int, int]: """Return parsed header information""" try: return kind, int(start, 16), int(length, 16) except ValueError as err: raise exceptions.FrameformatException( f"Invalid frame format: [{err}]" ) from err def readCompressionTrailer(self, reader) -> Dict[int, Any]: """If compressed stream is found, information about compressed block sizes is appended as last json payload. Function seeks to end of file and reads trailer information. """ pos = reader.tell() reader.seek(0, os.SEEK_END) reader.seek(-(self.types.FRAME_LEN + len(self.types.TERM)), os.SEEK_CUR) _, _, length = self._readHeader(reader) reader.seek(-(self.types.FRAME_LEN + int(length, 16)), os.SEEK_CUR) trailer = self.loadMetadata(reader.read(int(length, 16))) reader.seek(pos) return trailer @staticmethod def loadMetadata(s: bytes) -> Any: """Load and parse metadata information Parameters: s: (str) Json string as received during data file read Returns: json.loads: (dict) Decoded json string as python object """ try: return json.loads(s.decode("utf-8")) except json.decoder.JSONDecodeError as err: raise exceptions.MetaHeaderFormatException( f"Invalid meta header format: [{err}]" ) from err def writeFrame(self, writer, kind, start: int, length: int) -> None: """Write backup frame Parameters: writer: (fh) Writer object that implements .write() """ writer.write(self.types.FRAME % (kind, start, length)) def readFrame(self, reader) -> Tuple[str, int, int]: """Read backup frame Parameters: reader: (fh) Reader object which implements .read() """ kind, start, length = self._readHeader(reader) return self._parseHeader(kind, start, length) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/sparsestream/types.py0000644000000000000000000000542115033267156022620 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier Copyright (C) 2020 Red Hat, Inc. This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ from dataclasses import dataclass @dataclass(frozen=True) class SparseStreamTypes: # pylint: disable=too-many-instance-attributes """Sparse stream format Extended format based on the examples provided by the ovirt-imageio project: https://github.com/oVirt/ovirt-imageio/tree/master/examples META: start of meta information header DATA: data block marker ZERO: zero block marker STOP: stop block marker TERM: termination identifier FRAME: assembled frame FRAME_LEN: length of frame Stream format ============= Stream is composed of one of more frames. Meta frame ---------- Stream metadata, must be the first frame. "meta" space start length "\r\n" \r\n Metadata keys in the json payload: - virtual-size: image virtual size in bytes - data-size: number of bytes in data frames - date: ISO 8601 date string Data frame ---------- The header is followed by length bytes and terminator. "data" space start length "\r\n" "\r\n" Zero frame ---------- A zero extent, no payload. "zero" space start length "\r\n" Stop frame ---------- Marks the end of the stream, no payload. "stop" space start length "\r\n" Regular stream Example ------- meta 0000000000000000 0000000000000083\r\n { [.]] }\r\n data 0000000000000000 00000000000100000\r\n <1 MiB bytes>\r\n zero 0000000000100000 00000000040000000\r\n data 0000000040100000 00000000000001000\r\n <4096 bytes>\r\n stop 0000000000000000 00000000000000000\r\n Compressed stream: ------- Ends with compression marker: stop 0000000000000000 00000000000000000\r\n \r\n comp 0000000000000000 00000000000000010\r\n """ META: bytes = b"meta" DATA: bytes = b"data" COMP: bytes = b"comp" ZERO: bytes = b"zero" STOP: bytes = b"stop" TERM: bytes = b"\r\n" FRAME: bytes = b"%s %016x %016x" + TERM FRAME_LEN: int = len(FRAME % (STOP, 0, 0)) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4934876 virtnbdbackup-2.30/libvirtnbdbackup/ssh/0000755000000000000000000000000015033267565017170 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/ssh/__init__.py0000644000000000000000000000136015033267156021275 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ __title__ = "ssh" __version__ = "0.1" from .client import client, Mode ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/ssh/client.py0000644000000000000000000001214515033267156021017 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging import socket from typing import Tuple, Callable from enum import Enum from paramiko import ( AutoAddPolicy, SSHClient, SFTPClient, SSHException, AuthenticationException, ) from libvirtnbdbackup.ssh import exceptions from libvirtnbdbackup.objects import processInfo log = logging.getLogger("ssh") class Mode(Enum): """Up or download mode""" UPLOAD = 1 DOWNLOAD = 2 class client: """Wrapper around paramiko/sftp put and get functions, to be able to remote copy files from hypervisor host""" def __init__( self, host: str, user: str, port: int = 22, mode: Mode = Mode.DOWNLOAD ): self.client = None self.host = host self.user = user self.port = port self.copy: Callable[[str, str], None] = self.copyFrom if mode == Mode.UPLOAD: self.copy = self.copyTo self.connection = self.connect() def connect(self) -> SSHClient: """Connect to remote system""" log.info( "Connecting remote system [%s] via ssh, username: [%s]", self.host, self.user, ) try: cli = SSHClient() cli.load_system_host_keys() cli.set_missing_host_key_policy(AutoAddPolicy()) cli.connect( self.host, username=self.user, port=self.port, timeout=5000, ) return cli except AuthenticationException as e: raise exceptions.sshError(f"SSH key authentication failed: {e}") except socket.gaierror as e: raise exceptions.sshError(f"Unable to connect: {e}") except SSHException as e: raise exceptions.sshError(e) except Exception as e: log.exception(e) raise exceptions.sshError(f"Unhandled exception occurred: {e}") @property def sftp(self) -> SFTPClient: """Copy file""" return self.connection.open_sftp() def exists(self, filepath: str) -> bool: """ Check if remote file exists """ try: self.sftp.stat(filepath) return True except IOError: return False def copyFrom(self, filepath: str, localpath: str) -> None: """ Get file from remote system """ log.info("Downloading file [%s] to [%s]", filepath, localpath) try: self.sftp.get(filepath, localpath) except SSHException as e: log.warning("Unable to download file: [%s]", e) def copyTo(self, localpath: str, remotepath: str) -> None: """ Put file to remote system """ log.info("Uploading file [%s] to [%s]", localpath, remotepath) try: self.sftp.put(localpath, remotepath) except SSHException as e: log.warning("Unable to upload file: [%s]", e) def _execute(self, cmd) -> Tuple[int, str, str]: _, stdout, stderr = self.connection.exec_command(cmd) ret = stdout.channel.recv_exit_status() err = stderr.read().strip().decode() out = stdout.read().strip().decode() return ret, err, out def run(self, cmd: str, pidFile: str = "", logFile: str = "") -> processInfo: """ Execute command """ pid: int = 0 pidOut: str log.debug("Executing remote command: [%s]", cmd) ret, err, out = self._execute(cmd) logerr = "" if ret == 127: raise exceptions.sshError(err) if ret != 0: log.error( "Executing remote command failed, return code: [%s] stderr: [%s], stdout: [%s]", ret, err, out, ) if logFile: log.debug("Attempting to catch errors from logfile: [%s]", logFile) _, _, logerr = self._execute(f"cat {logFile}") raise exceptions.sshError( f"Error during remote command: [{cmd}]: [{err}] [{logerr}]" ) if pidFile: log.debug("PIDfile: [%s]", pidFile) _, _, pidOut = self._execute(f"cat {pidFile}") pid = int(pidOut) log.debug("PID: [%s]", pid) return processInfo(pid, logFile, err, out, pidFile) def disconnect(self): """Disconnect""" if self.sftp: self.sftp.close() if self.connection: self.connection.close() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/ssh/exceptions.py0000644000000000000000000000013615033267156021717 0ustar00rootroot""" Exceptions """ class sshError(Exception): """Exception thrown during ssh session""" ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4944875 virtnbdbackup-2.30/libvirtnbdbackup/virt/0000755000000000000000000000000015033267565017357 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/virt/__init__.py0000644000000000000000000000135315033267156021466 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ __title__ = "virt" __version__ = "0.1" from .client import client ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/virt/checkpoint.py0000644000000000000000000003067615033267156022070 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import glob import json import logging from argparse import Namespace from typing import Optional, Union, Any, List from lxml import etree as ElementTree import libvirt from libvirtnbdbackup import output from libvirtnbdbackup.virt import xml from libvirtnbdbackup.output.exceptions import OutputException from libvirtnbdbackup.common import defaultCheckpointName from libvirtnbdbackup.exceptions import ( NoCheckpointsFound, ReadCheckpointsError, CheckpointException, SaveCheckpointError, ForeignCeckpointError, RedefineCheckpointError, RemoveCheckpointError, ) log = logging.getLogger() def exists( domObj: libvirt.virDomain, checkpointName: str ) -> libvirt.virDomainCheckpoint: """Check if an checkpoint exists""" return domObj.checkpointLookupByName(checkpointName) def getXml(cptObj: libvirt.virDomainCheckpoint) -> str: """Get Checkpoint XML including size, if possible. Flag is not supported amongst all libvirt versions.""" try: return cptObj.getXMLDesc(libvirt.VIR_DOMAIN_CHECKPOINT_XML_SIZE) except libvirt.libvirtError as e: log.warning("Failed to get checkpoint info with size information: [%s]", e) return cptObj.getXMLDesc() def getSize(domObj: libvirt.virDomain, checkpointName: str) -> int: """Return current size of checkpoint for all disks""" size = 0 cpt = exists(domObj, checkpointName) cptTree = xml.asTree(getXml(cpt)) for s in cptTree.xpath("disks/disk/@size"): size += int(s) return size def delete(cptObj: libvirt.virDomainCheckpoint, checkpointName: str) -> bool: """Delete checkpoint""" checkpointName = cptObj.getName() if defaultCheckpointName not in checkpointName: log.debug( "Skipping checkpoint removal: [%s]: not from this application", checkpointName, ) return True log.debug("Attempt to remove checkpoint: [%s]", checkpointName) try: cptObj.delete() log.debug("Removed checkpoint: [%s]", checkpointName) return True except libvirt.libvirtError as errmsg: log.error("Error during checkpoint removal: [%s]", errmsg) return False def backup(args: Namespace, domObj: libvirt.virDomain) -> bool: """save checkpoint config to persistent storage""" checkpointFile = f"{args.checkpointdir}/{args.cpt.name}.xml" log.info("Saving checkpoint config to: [%s]", checkpointFile) try: with output.openfile(checkpointFile, "wb") as f: c = exists(domObj, args.cpt.name) f.write(getXml(c).encode()) return True except OutputException as errmsg: log.error( "Failed to save checkpoint config to file: [%s]: %s", checkpointFile, errmsg, ) return False def _hasForeign(domObj: libvirt.virDomain, checkpointName: str) -> Optional[str]: """Check if the virtual machine has an checkpoint which was not created by virtnbdbackup If an user or a third party utility creates an checkpoint, it is in line with the complete checkpoint chain, but virtnbdbackup does not save it. We can ensure consistency only if the complete chain of checkpoints is created by ourself. In case we detect an checkpoint that does not match our name, return so. """ cpts = domObj.listAllCheckpoints() if cpts: for cpt in cpts: checkpointName = cpt.getName() log.debug("Found foreign checkpoint: [%s]", checkpointName) if defaultCheckpointName not in checkpointName: return checkpointName return None def checkForeign( args: Namespace, domObj: libvirt.virDomain, ) -> bool: """Check and warn user if virtual machine has checkpoints not originating from this utility""" foreign = None if args.level in ("full", "inc", "diff"): foreign = _hasForeign(domObj, defaultCheckpointName) if not foreign: return True log.fatal("Foreign checkpoint found: [%s]", foreign) log.fatal("This checkpoint has not been created by this utility.") log.fatal( "To ensure backup chain consistency, " "remove existing checkpoints " "and start a new backup chain by creating a full backup." ) raise ForeignCeckpointError def removeAll( domObj: libvirt.virDomain, checkpointList: Union[List[Any], None], args: Namespace, checkpointName: str, ) -> bool: """Remove all existing checkpoints for a virtual machine, used during FULL backup to reset checkpoint chain """ log.debug("Cleaning up persistent storage %s", args.checkpointdir) try: for checkpointFile in glob.glob(f"{args.checkpointdir}/*.xml"): log.debug("Remove checkpoint file: %s", checkpointFile) os.remove(checkpointFile) except OSError as e: log.error("Failed to clean persistent storage %s: %s", args.checkpointdir, e) return False if checkpointList is None: cpts = domObj.listAllCheckpoints() if cpts: for cpt in cpts: if delete(cpt, checkpointName) is False: return False return True for cp in checkpointList: cptObj = exists(domObj, cp) if cptObj: if delete(cptObj, checkpointName) is False: return False return True def redefine(domObj: libvirt.virDomain, args: Namespace) -> bool: """Redefine checkpoints from persistent storage""" log.info("Loading checkpoint list from: [%s]", args.checkpointdir) checkpointList = glob.glob(f"{args.checkpointdir}/*.xml") checkpointList.sort(key=os.path.getmtime) for checkpointFile in checkpointList: log.debug("Loading checkpoint config from: [%s]", checkpointFile) try: with output.openfile(checkpointFile, "rb") as f: checkpointConfig = f.read() root = ElementTree.fromstring(checkpointConfig) except OutputException as e: log.error("Opening checkpoint file failed: [%s]: %s", checkpointFile, e) return False except ElementTree.ParseError as e: log.error( "Failed to load checkpoint config from [%s]: %s", checkpointFile, e ) return False try: checkpointName = root.find("name").text except ElementTree.ParseError as e: log.error("Failed to find checkpoint name: [%s]", e) return False try: _ = exists(domObj, checkpointName) log.debug("Checkpoint [%s] found", checkpointName) continue except libvirt.libvirtError as e: # ignore VIR_ERR_NO_DOMAIN_CHECKPOINT, report other errors if e.get_error_code() != libvirt.VIR_ERR_NO_DOMAIN_CHECKPOINT: log.error("libvirt error: %s", e) return False log.info("Redefine missing checkpoint: [%s]", checkpointName) try: domObj.checkpointCreateXML( checkpointConfig.decode(), libvirt.VIR_DOMAIN_CHECKPOINT_CREATE_REDEFINE | libvirt.VIR_DOMAIN_CHECKPOINT_CREATE_REDEFINE_VALIDATE, ) except libvirt.libvirtError as e: log.error("Redefining checkpoint failed: [%s]: %s", checkpointName, e) return False return True def read(cFile: str) -> List[str]: """Open checkpoint file and read checkpoint information""" checkpoints: List[str] = [] if not os.path.exists(cFile): return checkpoints try: with output.openfile(cFile, "rb") as fh: checkpoints = json.loads(fh.read().decode()) return checkpoints except OutputException as e: raise ReadCheckpointsError(f"Failed to read checkpoint file: [{e}]") from e except json.decoder.JSONDecodeError as e: raise ReadCheckpointsError(f"Invalid checkpoint file: [{e}]") from e def save(args: Namespace) -> None: """Append created checkpoint to checkpoint file""" try: checkpoints = read(args.cpt.file) checkpoints.append(args.cpt.name) with output.openfile(args.cpt.file, "wb") as cFw: cFw.write(json.dumps(checkpoints).encode()) except CheckpointException as e: raise CheckpointException from e except OutputException as e: raise SaveCheckpointError from e def validate(domObj: libvirt.virDomain, checkpointName: str) -> bool: """Validate that checkpoint exists""" try: c = domObj.checkpointLookupByName(checkpointName) checkpointXml = c.getXMLDesc(0) flags = ( libvirt.VIR_DOMAIN_CHECKPOINT_CREATE_REDEFINE | libvirt.VIR_DOMAIN_CHECKPOINT_CREATE_REDEFINE_VALIDATE ) # Redefine the checkpoint using the provided XML description c = domObj.checkpointCreateXML(checkpointXml, flags) if not c: return False return True except libvirt.libvirtError as e: log.warning("Failed to validate checkpoint: [%s]", e) return False def create( args: Namespace, domObj: libvirt.virDomain, ) -> None: """Checkpoint handling for different backup modes to be executed. Create, check and redefine checkpoints based on backup mode. Creates a new namespace in the argparse object, for easy pass around in further functions. """ checkpointName: str = f"{defaultCheckpointName}.0" parentCheckpoint: str = "" cptFile: str = f"{args.output}/{args.domain}.cpt" log.info("Loading checkpoints from: [%s]", cptFile) checkpoints: List[str] = read(cptFile) if args.offline is False: if redefine(domObj, args) is False: raise RedefineCheckpointError("Failed to redefine checkpoints.") # save level to reuse it properly when filename is selected for the backup args.level_filename = args.level log.info("Checkpoint handling.") if args.level == "full" and checkpoints: log.info("Removing all existent checkpoints before full backup.") if not removeAll(domObj, checkpoints, args, defaultCheckpointName): raise RemoveCheckpointError("Failed to remove checkpoint.") os.remove(cptFile) checkpoints = [] elif args.level == "full" and len(checkpoints) < 1: if not removeAll(domObj, None, args, defaultCheckpointName): raise RemoveCheckpointError("Failed to remove checkpoint.") checkpoints = [] if checkpoints and args.level in ("inc", "diff"): nextCpt = len(checkpoints) checkpointName = f"{defaultCheckpointName}.{nextCpt}" parentCheckpoint = checkpoints[-1] log.info("Next checkpoint id: [%s].", nextCpt) log.info("Parent checkpoint name [%s].", parentCheckpoint) if args.offline is True: log.info("Offline backup, using latest checkpoint, saving only delta.") checkpointName = parentCheckpoint # Autostart is disabled, validation could not be performed elif not validate(domObj, parentCheckpoint): log.warning( "Checkpoint [%s] is invalid, switching backup to full.", parentCheckpoint, ) args.level = "full" # reset back to defaults parentCheckpoint = "" if args.level in ("inc", "diff") and len(checkpoints) < 1: raise NoCheckpointsFound( "No existing checkpoints found, execute full backup first." ) if args.level == "diff": log.info("Diff backup: saving delta since checkpoint: [%s].", parentCheckpoint) if args.level in ("full", "inc"): log.info("Using checkpoint name: [%s].", checkpointName) args.cpt = Namespace() args.cpt.name = checkpointName args.cpt.parent = parentCheckpoint args.cpt.file = cptFile log.debug("Checkpoint info: [%s].", vars(args.cpt)) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/virt/client.py0000644000000000000000000004430115033267156021205 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import string import random import logging from argparse import Namespace from typing import Any, Dict, List, Tuple, Union import libvirt from libvirtnbdbackup.objects import DomainDisk from libvirtnbdbackup.virt.exceptions import ( domainNotFound, connectionFailed, startBackupFailed, ) from libvirtnbdbackup.virt import fs from libvirtnbdbackup.virt import xml from libvirtnbdbackup.virt import disktype def libvirt_ignore( _ignore: None, _err: Tuple[int, int, str, int, str, str, None, int, int] ) -> None: """this is required so libvirt.py does not report errors to stderr which it does by default. Error messages are fetched accordingly using exceptions. """ libvirt.registerErrorHandler(f=libvirt_ignore, ctx=None) log = logging.getLogger("virt") class client: """Libvirt related functions""" def __init__(self, uri: Namespace) -> None: self.remoteHost: str = "" self._conn = self._connect(uri) self._domObj = None self.libvirtVersion = self._conn.getLibVersion() @staticmethod def _connectAuth(uri: str, user: str, password: str) -> libvirt.virConnect: """Use openAuth if connection for advanced SASL authentication mechanisms if username and password are set""" def _cred(credentials, user_data) -> int: for credential in credentials: if credential[0] == libvirt.VIR_CRED_AUTHNAME: credential[4] = user_data[0] elif credential[0] == libvirt.VIR_CRED_PASSPHRASE: credential[4] = user_data[1] return 0 log.debug("Username: %s", user) log.debug("Password: %s", password) try: flags: List[Any] = [libvirt.VIR_CRED_AUTHNAME, libvirt.VIR_CRED_PASSPHRASE] auth: List[Any] = [flags] if user is not None and password is not None: user_data = [user, password] auth.append(_cred) auth.append(user_data) return libvirt.openAuth(uri, auth, 0) except libvirt.libvirtError as e: raise connectionFailed(e) from e @staticmethod def _connectOpen(uri: str) -> libvirt.virConnect: """Open connection with regular libvirt URI for local authentication without further authentication mechanisms required""" try: return libvirt.open(uri) except libvirt.libvirtError as e: if e.get_error_code() == 45: errmsg = f"{e}: --user and --password options for SASL authentication are required." raise connectionFailed(errmsg) from e raise connectionFailed(e) from e def _connect(self, args: Namespace) -> libvirt.virConnect: """return libvirt connection handle and check if connection is established to a remote host.""" log.debug("Libvirt URI: [%s]", args.uri) if args.user and args.password: conn = self._connectAuth(args.uri, args.user, args.password) else: conn = self._connectOpen(args.uri) # Detect if we are connected to a remote libvirt daemon by # comparing the local and remote hostname. If qemu+ssh is # part of the libvirt URI, set the remote host as well. # This will spawn the NBD service for data transfer via # TCP socket instead of local socket file and related virtual # domain files will be copied via SFTP. if "qemu+ssh" in args.uri: remoteHostname = conn.getHostname() log.info( "Connected to remote host: [%s]", remoteHostname, ) self.remoteHost = remoteHostname return conn def close(self) -> None: """Disconnect""" log.debug("Close connection to libvirt.") self._conn.close() def getDomain(self, name: str) -> libvirt.virDomain: """Lookup domain""" try: return self._conn.lookupByName(name) except libvirt.libvirtError as e: raise domainNotFound(e) from e def refreshPool(self, path: str) -> None: """Check if specified path matches an existing storage pool and refresh its contents""" try: pool = self._conn.storagePoolLookupByTargetPath(path) except libvirt.libvirtError: log.warning( "Restore path [%s] seems not to be an libvirt managed pool, skipping refresh.", path, ) return try: pool.refresh() log.info("Refreshed contents of libvirt pool [%s]", pool.name()) except libvirt.libvirtError as e: log.warning("Failed to refresh libvirt pool [%s]: [%s]", pool.name(), e) @staticmethod def blockJobActive(domObj: libvirt.virDomain, disks: List[DomainDisk]) -> bool: """Check if there is already an active block job for this virtual machine, which might block""" for disk in disks: blockInfo = domObj.blockJobInfo(disk.target) if ( blockInfo and blockInfo["type"] == libvirt.VIR_DOMAIN_BLOCK_JOB_TYPE_BACKUP ): log.debug("Running block jobs for disk [%s]", disk.target) log.debug(blockInfo) return True return False def hasIncrementalEnabled(self, domObj: libvirt.virDomain) -> bool: """Check if virtual machine has enabled required capabilities for incremental backup Libvirt version >= 7006000 have the feature enabled by default without the domain XML including the capability statement. """ if self.libvirtVersion >= 7006000: return True tree = xml.asTree(domObj.XMLDesc(0)) for target in tree.findall( "{http://libvirt.org/schemas/domain/qemu/1.0}capabilities" ): for cap in target.findall( "{http://libvirt.org/schemas/domain/qemu/1.0}add" ): if "incremental-backup" in cap.items()[0]: return True return False @staticmethod def getDomainConfig(domObj: libvirt.virDomain) -> str: """Return Virtual Machine configuration as XML""" return domObj.XMLDesc(0) @staticmethod def startDomain(domObj: libvirt.virDomain) -> bool: """Start virtual machine in paused state to allow full / inc backup""" return domObj.createWithFlags( flags=libvirt.VIR_DOMAIN_START_PAUSED | libvirt.VIR_DOMAIN_START_AUTODESTROY ) @staticmethod def domainAutoStart(domObj: libvirt.virDomain) -> None: """Mark virtual machine for autostart""" try: domObj.setAutostart(1) log.info("Setting autostart config for domain.") except libvirt.libvirtError as errmsg: log.warning("Failed to set autostart flag for domain: [%s]", errmsg) def defineDomain(self, vmConfig: bytes, autoStart: bool) -> bool: """Define domain based on restored config""" try: log.info("Redefining domain based on adjusted config.") domObj = self._conn.defineXMLFlags(vmConfig.decode(), 0) log.info("Successfully redefined domain [%s]", domObj.name()) except libvirt.libvirtError as errmsg: log.error("Failed to define domain: [%s]", errmsg) return False if autoStart is True: self.domainAutoStart(domObj) return True def getDomainInfo(self, vmConfig: str) -> Dict[str, str]: """Return object with general vm information relevant for backup""" tree = xml.asTree(vmConfig) settings = {} for flag in ["loader", "nvram", "kernel", "initrd"]: try: settings[flag] = tree.find("os").find(flag).text except AttributeError as e: log.debug("No setting [%s] found: %s", flag, e) log.debug("Domain Info: [%s]", settings) return settings def getTPMDevice(self, vmConfig: str) -> bool: """Check if virtual machine has configured an emulated (swtpm based) TPM device""" tree = xml.asTree(vmConfig) device = tree.find("devices/tpm") if device is not None: tpm = device.xpath("backend")[0].get("type") return tpm == "emulator" return False @staticmethod def getBackingStores(disk: xml._Element) -> List[str]: """Get list of backing store files defined for disk, usually the case if virtual machine has external snapshots.""" backingStoreFiles: List[str] = [] backingStore = disk.find("backingStore") while backingStore is not None: backingStoreSource = backingStore.find("source") if backingStoreSource is not None: backingStoreFiles.append(backingStoreSource.get("file")) if backingStore.find("backingStore") is not None: backingStore = backingStore.find("backingStore") else: backingStore = None return backingStoreFiles def _getDiskPathByVolume(self, disk: xml._Element) -> Union[str, None]: """If virtual machine disk is configured via type='volume' get path to disk via appropriate libvirt functions, pool and volume setting are mandatory as by xml schema definition""" vol = disk.xpath("source")[0].get("volume") pool = disk.xpath("source")[0].get("pool") try: diskPool = self._conn.storagePoolLookupByName(pool) diskPath = diskPool.storageVolLookupByName(vol).path() except libvirt.libvirtError as errmsg: log.error("Failed to detect vm disk by volumes: [%s]", errmsg) return None return diskPath def getDomainDisks(self, args: Namespace, vmConfig: str) -> List[DomainDisk]: """Parse virtual machine configuration for disk devices, filter all non supported devices """ tree = xml.asTree(vmConfig) devices = [] excludeList = None if args.exclude is not None: excludeList = args.exclude.split(",") for disk in tree.xpath("devices/disk"): discardOption = None dev = disk.xpath("target")[0].get("dev") device = disk.get("device") diskFormat = disk.xpath("driver")[0].get("type") discardOption = disk.xpath("driver")[0].get("discard") if excludeList is not None and dev in excludeList: log.warning("Excluding disk [%s] from backup as requested", dev) continue if args.include is not None and dev != args.include: log.info( "Skipping disk: [%s] as requested: does not match disk [%s]", dev, args.include, ) continue # skip cdrom/floppy devices if disktype.Optical(device, dev): continue diskPath = None diskType = disk.get("type") if diskType == "volume": log.debug("Disk [%s]: volume notation", dev) diskPath = self._getDiskPathByVolume(disk) elif diskType == "file": log.debug("Disk [%s]: file notation", dev) diskPath = disk.xpath("source")[0].get("file") elif diskType == "block": if args.raw is False and diskFormat == "raw": log.warning( "Skipping direct attached block device [%s], use option --raw to include.", dev, ) continue diskPath = disk.xpath("source")[0].get("dev") else: log.error("Unable to detect disk volume type for disk [%s]", dev) continue if diskPath is None: log.error("Unable to detect disk source for disk [%s]", dev) continue # include other direct attached devices if --raw option is enabled if args.raw is False and ( disktype.Block(disk, dev) or disktype.Lun(device, dev) or disktype.Raw(diskFormat, dev) ): continue diskFileName = os.path.basename(diskPath) backingStoreFiles = self.getBackingStores(disk) devices.append( DomainDisk( dev, diskFormat, diskFileName, diskPath, backingStoreFiles, discardOption, ) ) log.debug("Device list: %s ", devices) return devices def _createBackupXml(self, args: Namespace, diskList) -> str: """Create XML file for starting an backup task using libvirt API.""" top = xml.ElementTree.Element("domainbackup", {"mode": "pull"}) if self.remoteHost == "": xml.ElementTree.SubElement( top, "server", {"transport": "unix", "socket": f"{args.socketfile}"} ) else: listen = self.remoteHost tls = "no" if args.tls: tls = "yes" if args.nbd_ip != "": listen = args.nbd_ip xml.ElementTree.SubElement( top, "server", {"tls": f"{tls}", "name": f"{listen}", "port": f"{args.nbd_port}"}, ) disks = xml.ElementTree.SubElement(top, "disks") if args.cpt.parent != "": inc = xml.ElementTree.SubElement(top, "incremental") inc.text = args.cpt.parent for disk in diskList: scratchId = "".join( random.choices(string.ascii_uppercase + string.digits, k=5) ) scratchFile = f"{args.scratchdir}/backup.{scratchId}.{disk.target}" log.debug("Using scratch file: %s", scratchFile) dE = xml.ElementTree.SubElement(disks, "disk", {"name": disk.target}) xml.ElementTree.SubElement(dE, "scratch", {"file": f"{scratchFile}"}) return xml.indent(top) def _createCheckpointXml( self, diskList: List[Any], parentCheckpoint: str, checkpointName: str ) -> str: """Create valid checkpoint XML file which is passed to libvirt API""" top = xml.ElementTree.Element("domaincheckpoint") desc = xml.ElementTree.SubElement(top, "description") desc.text = "Backup checkpoint" name = xml.ElementTree.SubElement(top, "name") name.text = checkpointName if parentCheckpoint != "": pct = xml.ElementTree.SubElement(top, "parent") cptName = xml.ElementTree.SubElement(pct, "name") cptName.text = parentCheckpoint disks = xml.ElementTree.SubElement(top, "disks") for disk in diskList: # No persistent checkpoint will be created for raw disks, # because it is not supported. Backup will only be crash # consistent. If we would like to create a consistent # backup, we would have to create an snapshot for these # kind of disks, example: # virsh checkpoint-create-as vm4 --diskspec sdb # error: unsupported configuration: \ # checkpoint for disk sdb unsupported for storage type raw # See also: # https://lists.gnu.org/archive/html/qemu-devel/2021-03/msg07448.html if disk.format != "raw": xml.ElementTree.SubElement(disks, "disk", {"name": disk.target}) return xml.indent(top) def startBackup( self, args: Namespace, domObj: libvirt.virDomain, diskList: List[Any], ) -> None: """Attempt to start pull based backup task using XML description""" backupXml = self._createBackupXml(args, diskList) checkpointXml = None freezed = False # do not create checkpoint during copy/diff backup. # backup saves delta until the last checkpoint if args.level not in ("copy", "diff"): checkpointXml = self._createCheckpointXml( diskList, args.cpt.parent, args.cpt.name ) freezed = fs.freeze(domObj, args.freeze_mountpoint) try: log.debug("Starting backup job via libvirt API.") domObj.backupBegin(backupXml, checkpointXml) log.debug("Started backup job via libvirt API.") except libvirt.libvirtError as errmsg: code = errmsg.get_error_code() if code == libvirt.VIR_ERR_CHECKPOINT_INCONSISTENT: raise startBackupFailed( "Bitmap inconsistency detected: please cleanup checkpoints using virsh " f"and execute new full backup: {errmsg}" ) from errmsg raise startBackupFailed(f"Failed to start backup: [{errmsg}]") from errmsg except Exception as e: log.exception(e) raise startBackupFailed( f"Unknown exception during backup start: [{e}]" ) from e finally: # check if filesystem is freezed and thaw # in case creating checkpoint fails. if freezed is True: fs.thaw(domObj) @staticmethod def stopBackup(domObj: libvirt.virDomain) -> bool: """Cancel the backup task using job abort""" try: domObj.abortJob() return True except libvirt.libvirtError as err: log.warning("Failed to stop backup job: [%s]", err) return False ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/virt/disktype.py0000644000000000000000000000354115033267156021564 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging from lxml.etree import _Element log = logging.getLogger() def Optical(device: list, dev: str) -> bool: """Check if device is cdrom or floppy""" if device in ("cdrom", "floppy"): log.info("Skipping attached [%s] device: [%s].", device, dev) return True return False def Lun(device: list, dev: str) -> bool: """Check if device is direct attached LUN""" if device == "lun": log.warning( "Skipping direct attached lun [%s], use option --raw to include", dev, ) return True return False def Block(disk: _Element, dev: str) -> bool: """Check if device is direct attached block type device""" if disk.xpath("target")[0].get("type") == "block": log.warning( "Block device [%s] excluded by default, use option --raw to include.", dev, ) return True return False def Raw(diskFormat: str, dev: str) -> bool: """Check if disk has RAW disk format""" if diskFormat == "raw": log.warning( "Raw disk [%s] excluded by default, use option --raw to include.", dev, ) return True return False ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/virt/exceptions.py0000644000000000000000000000051115033267156022103 0ustar00rootroot""" Exceptions """ class virtHelperError(Exception): """Errors during libvirt helper""" class domainNotFound(virtHelperError): """Can't find domain""" class connectionFailed(virtHelperError): """Can't connect libvirtd domain""" class startBackupFailed(virtHelperError): """Can't start backup operation""" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/virt/fs.py0000644000000000000000000000334115033267156020336 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging import libvirt log = logging.getLogger("fs") def freeze(domObj: libvirt.virDomain, mountpoints: None) -> bool: """Attempt to freeze domain filesystems using qemu guest agent""" state, _ = domObj.state() if state == libvirt.VIR_DOMAIN_PAUSED: log.info("Skip freezing filesystems: domain is in paused state") return False log.debug("Attempting to freeze filesystems.") try: if mountpoints is not None: frozen = domObj.fsFreeze(mountpoints.split(",")) else: frozen = domObj.fsFreeze() log.info("Freezed [%s] filesystems.", frozen) return True except libvirt.libvirtError as errmsg: log.warning(errmsg) return False def thaw(domObj: libvirt.virDomain) -> bool: """Thaw freezed filesystems""" log.debug("Attempting to thaw filesystems.") try: thawed = domObj.fsThaw() log.info("Thawed [%s] filesystems.", thawed) return True except libvirt.libvirtError as errmsg: log.warning(errmsg) return False ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/libvirtnbdbackup/virt/xml.py0000644000000000000000000000256715033267156020537 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import logging from lxml.etree import _Element from lxml import etree as ElementTree log = logging.getLogger() def asTree(vmConfig: str) -> _Element: """Return Etree element for vm config""" return ElementTree.fromstring(vmConfig) def indent(top: _Element) -> str: """Indent xml output for debug log""" try: ElementTree.indent(top) except ElementTree.ParseError as errmsg: log.debug("Failed to parse xml: [%s]", errmsg) except AttributeError: # older ElementTree versions dont have the # indent method, skip silently and use # non formatted string pass xml = ElementTree.tostring(top).decode() log.debug("\n%s", xml) return xml ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4954875 virtnbdbackup-2.30/man/0000755000000000000000000000000015033267565013621 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/man/virtnbdbackup.10000644000000000000000000001233015033267156016534 0ustar00rootroot.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.49.3. .TH VIRTNBDBACKUP "1" "July 2025" "virtnbdbackup 2.30" "User Commands" .SH NAME virtnbdbackup \- backup utility for libvirt .SH DESCRIPTION usage: virtnbdbackup [\-h] \fB\-d\fR DOMAIN [\-l {copy,full,inc,diff,auto}] .TP [\-t {stream,raw}] [\-r] \fB\-o\fR OUTPUT [\-C CHECKPOINTDIR] [\-\-scratchdir SCRATCHDIR] [\-S] [\-i INCLUDE] [\-x EXCLUDE] [\-f SOCKETFILE] [\-n] [\-z [COMPRESS]] [\-w WORKER] [\-F FREEZE_MOUNTPOINT] [\-e] [\-\-no\-sparse\-detection] [\-T THRESHOLD] [\-U URI] [\-\-user USER] [\-\-ssh\-user SSH_USER] [\-\-ssh\-port SSH_PORT] [\-\-password PASSWORD] [\-P NBD_PORT] [\-I NBD_IP] [\-\-tls] [\-\-tls\-cert TLS_CERT] [\-L] [\-\-quiet] [\-\-nocolor] [\-q] [\-s] [\-k] [\-p] [\-v] [\-V] .PP Backup libvirt/qemu virtual machines .SS "options:" .TP \fB\-h\fR, \fB\-\-help\fR show this help message and exit .SS "General options:" .TP \fB\-d\fR DOMAIN, \fB\-\-domain\fR DOMAIN Domain to backup .TP \fB\-l\fR {copy,full,inc,diff,auto}, \fB\-\-level\fR {copy,full,inc,diff,auto} Backup level. (default: copy) .TP \fB\-t\fR {stream,raw}, \fB\-\-type\fR {stream,raw} Output type: stream or raw. (default: stream) .TP \fB\-r\fR, \fB\-\-raw\fR Include full provisioned disk images in backup. (default: False) .TP \fB\-o\fR OUTPUT, \fB\-\-output\fR OUTPUT Output target directory .TP \fB\-C\fR CHECKPOINTDIR, \fB\-\-checkpointdir\fR CHECKPOINTDIR Persistent libvirt checkpoint storage directory .TP \fB\-\-scratchdir\fR SCRATCHDIR Target dir for temporary scratch file. (default: \fI\,/var/tmp\/\fP) .TP \fB\-S\fR, \fB\-\-start\-domain\fR Start virtual machine if it is offline. (default: False) .TP \fB\-i\fR INCLUDE, \fB\-\-include\fR INCLUDE Backup only disk with target dev name (\fB\-i\fR vda) .TP \fB\-x\fR EXCLUDE, \fB\-\-exclude\fR EXCLUDE Exclude disk(s) with target dev name (\fB\-x\fR vda,vdb) .TP \fB\-f\fR SOCKETFILE, \fB\-\-socketfile\fR SOCKETFILE Use specified file for NBD Server socket (default: \fI\,/var/tmp/virtnbdbackup.4086134\/\fP) .TP \fB\-n\fR, \fB\-\-noprogress\fR Disable progress bar .TP \fB\-z\fR [COMPRESS], \fB\-\-compress\fR [COMPRESS] Compress with lz4 compression level. (default: False) .TP \fB\-w\fR WORKER, \fB\-\-worker\fR WORKER Amount of concurrent workers used to backup multiple disks. (default: amount of disks) .TP \fB\-F\fR FREEZE_MOUNTPOINT, \fB\-\-freeze\-mountpoint\fR FREEZE_MOUNTPOINT If qemu agent available, freeze only filesystems on specified mountpoints within virtual machine (default: all) .TP \fB\-e\fR, \fB\-\-strict\fR Change exit code if warnings occur during backup operation. (default: False) .TP \fB\-\-no\-sparse\-detection\fR Skip detection of sparse ranges during incremental or differential backup. (default: False) .TP \fB\-T\fR THRESHOLD, \fB\-\-threshold\fR THRESHOLD Execute backup only if threshold is reached. .SS "Remote Backup options:" .TP \fB\-U\fR URI, \fB\-\-uri\fR URI Libvirt connection URI. (default: qemu:///session) .TP \fB\-\-user\fR USER User to authenticate against libvirtd. (default: None) .TP \fB\-\-ssh\-user\fR SSH_USER User to authenticate against remote sshd: used for remote copy of files. (default: abi) .TP \fB\-\-ssh\-port\fR SSH_PORT Port to connect to remote sshd: used for remote copy of files. (default: 22) .TP \fB\-\-password\fR PASSWORD Password to authenticate against libvirtd. (default: None) .TP \fB\-P\fR NBD_PORT, \fB\-\-nbd\-port\fR NBD_PORT Port used by remote NBD Service, should be unique for each started backup. (default: 10809) .TP \fB\-I\fR NBD_IP, \fB\-\-nbd\-ip\fR NBD_IP IP used to bind remote NBD service on (default: hostname returned by libvirtd) .TP \fB\-\-tls\fR Enable and use TLS for NBD connection. (default: False) .TP \fB\-\-tls\-cert\fR TLS_CERT Path to TLS certificates used during offline backup and restore. (default: /etc/pki/qemu/) .SS "Logging options:" .TP \fB\-L\fR, \fB\-\-syslog\fR Additionally send log messages to syslog (default: False) .TP \fB\-\-quiet\fR Disable logging to stderr (default: False) .TP \fB\-\-nocolor\fR Disable colored output (default: False) .SS "Debug options:" .TP \fB\-q\fR, \fB\-\-qemu\fR Use Qemu tools to query extents. .TP \fB\-s\fR, \fB\-\-startonly\fR Only initialize backup job via libvirt, do not backup any data .TP \fB\-k\fR, \fB\-\-killonly\fR Kill any running block job .TP \fB\-p\fR, \fB\-\-printonly\fR Quit after printing estimated checkpoint size. .TP \fB\-v\fR, \fB\-\-verbose\fR Enable debug output .TP \fB\-V\fR, \fB\-\-version\fR Show version and exit .SH EXAMPLES .IP # full backup of domain 'webvm' with all attached disks: .IP virtnbdbackup \-d webvm \-l full \-o /backup/ .IP # incremental backup: .IP virtnbdbackup \-d webvm \-l inc \-o /backup/ .IP # differential backup: .IP virtnbdbackup \-d webvm \-l diff \-o /backup/ .IP # full backup, exclude disk 'vda': .IP virtnbdbackup \-d webvm \-l full \-x vda \-o /backup/ .IP # full backup, backup only disk 'vdb': .IP virtnbdbackup \-d webvm \-l full \-i vdb \-o /backup/ .IP # full backup, compression enabled: .IP virtnbdbackup \-d webvm \-l full \-z \-o /backup/ .IP # full backup, create archive: .IP virtnbdbackup \-d webvm \-l full \-o \- > backup.zip .IP # full backup of vm operating on remote libvirtd: .IP virtnbdbackup \-U qemu+ssh://root@remotehost/system \-\-ssh\-user root \-d webvm \-l full \-o /backup/ ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/man/virtnbdmap.10000644000000000000000000000416715033267156016055 0ustar00rootroot.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.49.3. .TH VIRTNBDMAP "1" "July 2025" "virtnbdmap 2.30" "User Commands" .SH NAME virtnbdmap \- map virtnbdbackup image files to nbd devices .SH DESCRIPTION usage: virtnbdmap [\-h] \fB\-f\fR FILE [\-b BLOCKSIZE] [\-d DEVICE] [\-e EXPORT_NAME] .IP [\-t THREADS] [\-l LISTEN_ADDRESS] [\-p LISTEN_PORT] [\-n] [\-L LOGFILE] [\-\-nocolor] [\-r] [\-H] [\-v] [\-V] .PP Map backup image(s) to block device .SS "options:" .TP \fB\-h\fR, \fB\-\-help\fR show this help message and exit .SS "General options:" .TP \fB\-f\fR FILE, \fB\-\-file\fR FILE List of Backup files to map .TP \fB\-b\fR BLOCKSIZE, \fB\-\-blocksize\fR BLOCKSIZE Maximum blocksize passed to nbdkit. (default: 4096) .TP \fB\-d\fR DEVICE, \fB\-\-device\fR DEVICE Target device. (default: \fI\,/dev/nbd0\/\fP) .TP \fB\-e\fR EXPORT_NAME, \fB\-\-export\-name\fR EXPORT_NAME Export name passed to nbdkit. (default: sda) .TP \fB\-t\fR THREADS, \fB\-\-threads\fR THREADS Amount of threads passed to nbdkit process. (default: 1) .TP \fB\-l\fR LISTEN_ADDRESS, \fB\-\-listen\-address\fR LISTEN_ADDRESS IP Address for nbdkit process to listen on. (default: 127.0.0.1) .TP \fB\-p\fR LISTEN_PORT, \fB\-\-listen\-port\fR LISTEN_PORT Port for nbdkit process to listen on. (default: 10809) .TP \fB\-n\fR, \fB\-\-noprogress\fR Disable progress bar .SS "Logging options:" .TP \fB\-L\fR LOGFILE, \fB\-\-logfile\fR LOGFILE Path to Logfile (default: \fI\,/home/abi/virtnbdmap.log\/\fP) .TP \fB\-\-nocolor\fR Disable colored output (default: False) .SS "Debug options:" .TP \fB\-r\fR, \fB\-\-readonly\fR Map image readonly (default: False) .TP \fB\-H\fR, \fB\-\-hexdump\fR Hexdump data to logfile for debugging (default: False) .TP \fB\-v\fR, \fB\-\-verbose\fR Enable debug output .TP \fB\-V\fR, \fB\-\-version\fR Show version and exit .SH EXAMPLES .IP # Map full backup to device /dev/nbd0: .IP virtnbdmap \-f /backup/sda.full.data .IP # Map full backup to device /dev/nbd2: .IP virtnbdmap \-f /backup/sda.full.data \-d /dev/nbd2 .IP # Map sequence of full and incremental to device /dev/nbd2: .IP virtnbdmap \-f /backup/sda.full.data,/backup/sda.inc.1.data \-d /dev/nbd2 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/man/virtnbdrestore.10000644000000000000000000001063415033267156016757 0ustar00rootroot.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.49.3. .TH VIRTNBDRESTORE "1" "July 2025" "virtnbdrestore 2.30" "User Commands" .SH NAME virtnbdrestore \- restore utility for libvirt .SH DESCRIPTION usage: virtnbdrestore [\-h] [\-a {dump,restore,verify}] \fB\-i\fR INPUT \fB\-o\fR OUTPUT .TP [\-u UNTIL] [\-s SEQUENCE] [\-d DISK] [\-n] [\-f SOCKETFILE] [\-r] [\-c] [\-D] [\-C CONFIG_FILE] [\-N NAME] [\-B BUFFSIZE] [\-A] [\-U URI] [\-\-user USER] [\-\-ssh\-user SSH_USER] [\-\-ssh\-port SSH_PORT] [\-\-password PASSWORD] [\-P NBD_PORT] [\-I NBD_IP] [\-\-tls] [\-\-tls\-cert TLS_CERT] [\-L LOGFILE] [\-\-nocolor] [\-v] [\-V] .PP Restore virtual machine disks .SS "options:" .TP \fB\-h\fR, \fB\-\-help\fR show this help message and exit .SS "General options:" .TP \fB\-a\fR {dump,restore,verify}, \fB\-\-action\fR {dump,restore,verify} Action to perform: (default: restore) .TP \fB\-i\fR INPUT, \fB\-\-input\fR INPUT Directory including a backup set .TP \fB\-o\fR OUTPUT, \fB\-\-output\fR OUTPUT Restore target directory .TP \fB\-u\fR UNTIL, \fB\-\-until\fR UNTIL Restore only until checkpoint, point in time restore. .TP \fB\-s\fR SEQUENCE, \fB\-\-sequence\fR SEQUENCE Restore image based on specified backup files. .TP \fB\-d\fR DISK, \fB\-\-disk\fR DISK Process only disk matching target dev name. (default: None) .TP \fB\-n\fR, \fB\-\-noprogress\fR Disable progress bar .TP \fB\-f\fR SOCKETFILE, \fB\-\-socketfile\fR SOCKETFILE Use specified file for NBD Server socket (default: \fI\,/var/tmp/virtnbdbackup.4086139\/\fP) .TP \fB\-r\fR, \fB\-\-raw\fR Copy raw images as is during restore. (default: False) .TP \fB\-c\fR, \fB\-\-adjust\-config\fR Adjust vm configuration during restore. (default: False) .TP \fB\-D\fR, \fB\-\-define\fR Register/define VM after restore. (default: False) .TP \fB\-C\fR CONFIG_FILE, \fB\-\-config\-file\fR CONFIG_FILE Name of the vm config file used for restore. (default: vmconfig.xml) .TP \fB\-N\fR NAME, \fB\-\-name\fR NAME Define restored domain with specified name .TP \fB\-B\fR BUFFSIZE, \fB\-\-buffsize\fR BUFFSIZE Buffer size to use during verify (default: 8192) .TP \fB\-A\fR, \fB\-\-preallocate\fR Preallocate restored qcow images. (default: False) .SS "Remote Restore options:" .TP \fB\-U\fR URI, \fB\-\-uri\fR URI Libvirt connection URI. (default: qemu:///session) .TP \fB\-\-user\fR USER User to authenticate against libvirtd. (default: None) .TP \fB\-\-ssh\-user\fR SSH_USER User to authenticate against remote sshd: used for remote copy of files. (default: abi) .TP \fB\-\-ssh\-port\fR SSH_PORT Port to connect to remote sshd: used for remote copy of files. (default: 22) .TP \fB\-\-password\fR PASSWORD Password to authenticate against libvirtd. (default: None) .TP \fB\-P\fR NBD_PORT, \fB\-\-nbd\-port\fR NBD_PORT Port used by remote NBD Service, should be unique for each started backup. (default: 10809) .TP \fB\-I\fR NBD_IP, \fB\-\-nbd\-ip\fR NBD_IP IP used to bind remote NBD service on (default: hostname returned by libvirtd) .TP \fB\-\-tls\fR Enable and use TLS for NBD connection. (default: False) .TP \fB\-\-tls\-cert\fR TLS_CERT Path to TLS certificates used during offline backup and restore. (default: /etc/pki/qemu/) .SS "Logging options:" .TP \fB\-L\fR LOGFILE, \fB\-\-logfile\fR LOGFILE Path to Logfile (default: \fI\,/home/abi/virtnbdrestore.log\/\fP) .TP \fB\-\-nocolor\fR Disable colored output (default: False) .SS "Debug options:" .TP \fB\-v\fR, \fB\-\-verbose\fR Enable debug output .TP \fB\-V\fR, \fB\-\-version\fR Show version and exit .SH EXAMPLES .IP # Dump backup metadata: .IP virtnbdrestore \-i /backup/ \-o dump .IP # Verify checksums for existing data files in backup: .IP virtnbdrestore \-i /backup/ \-o verify .IP # Complete restore with all disks: .IP virtnbdrestore \-i /backup/ \-o /target .IP # Complete restore, adjust config and redefine vm after restore: .IP virtnbdrestore \-cD \-i /backup/ \-o /target .IP # Complete restore, adjust config and redefine vm with name 'foo': .IP virtnbdrestore \-cD \-\-name foo \-i /backup/ \-o /target .IP # Restore only disk 'vda': .IP virtnbdrestore \-i /backup/ \-o /target \-d vda .IP # Point in time restore: .IP virtnbdrestore \-i /backup/ \-o /target \-\-until virtnbdbackup.2 .IP # Restore and process specific file sequence: .IP virtnbdrestore \-i /backup/ \-o /target \-\-sequence vdb.full.data,vdb.inc.virtnbdbackup.1.data .IP # Restore to remote system: .IP virtnbdrestore \-U qemu+ssh://root@remotehost/system \-\-ssh\-user root \-i /backup/ \-o /remote_target ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/requirements.txt0000644000000000000000000000011715033267156016325 0ustar00rootrootlibvirt-python>=6.0.0 tqdm lz4>=2.1.2 lxml paramiko typing_extensions colorlog ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4964876 virtnbdbackup-2.30/setup.cfg0000644000000000000000000000100615033267565014664 0ustar00rootroot[metadata] description_file = README.md long_description = Backup utility for libvirt, using latest changed block tracking features [bdist_rpm] release = 1 packager = Michael Ablassmeier doc_files = README.md Changelog LICENSE requires = python3-libvirt python3-libnbd python3-lxml python3-tqdm python3-lz4 nbdkit-server nbdkit-python-plugin python3-dataclasses python3-paramiko python3-typing-extensions python3-colorlog qemu-img openssh-clients [egg_info] tag_build = tag_date = 0 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/setup.py0000644000000000000000000000147715033267156014565 0ustar00rootroot#!/usr/bin/env python3 """Setup virtnbdbackup""" from setuptools import setup, find_packages import libvirtnbdbackup with open("requirements.txt") as f: install_requires = f.read().splitlines() setup( name="virtnbdbackup", version=libvirtnbdbackup.__version__, description="Backup utility for libvirt", url="https://github.com/abbbi/virtnbdbackup/", author="Michael Ablassmeier", author_email="abi@grinser.de", license="GPL", keywords="libnbd backup libvirt", packages=find_packages(exclude=("docs", "tests", "env")), include_package_data=True, scripts=["virtnbdbackup", "virtnbdrestore", "virtnbdmap", "virtnbd-nbdkit-plugin"], install_requires=install_requires, extras_require={ "dev": [], "docs": [], "testing": [], }, classifiers=[], ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/virtnbd-nbdkit-plugin0000644000000000000000000001633415033267156017211 0ustar00rootroot""" Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import builtins import json import pprint import nbdkit API_VERSION = 2 blockMap = None blockMapFile = None image = None debug = "0" hexdump = "0" # pylint: disable=global-statement,global-variable-not-assigned,redefined-builtin,too-many-statements,too-many-branches def config(key, value): """Read parameter values, needs to use open function via builtins as the python plugin itself defines it again""" global blockMap global blockMapFile global image global debug global hexdump if key == "blockmap": blockMapFile = value with builtins.open(blockMapFile, "rb") as fh: blockMap = json.loads(fh.read()) return if key == "disk": image = value return if key == "debug": debug = value return if key == "hexdump": hexdump = value return raise RuntimeError(f"Unsupported parameter: {key}") def log(msg): """Debug logging""" global debug if debug == "1": nbdkit.debug(msg) def config_complete(): """Check if we have all required parameters""" global image global blockMap global blockMapFile if image is None or blockMap is None: raise RuntimeError( "Missing parameter: path to blockmap and disk files required." ) if not os.path.exists(image): raise RuntimeError(f"Specified image file: [{image}] does not exist.") if not os.path.exists(blockMapFile): raise RuntimeError(f"Specified blockmap file: [{blockMapFile}] does not exit.") pprint.pprint(blockMap) def thread_model(): """nbdkit threading model""" return nbdkit.THREAD_MODEL_PARALLEL def open(_): """Open backup files and return FD for each""" fd = os.open(image, os.O_RDONLY) log(f"File descriptors: {fd}") return fd def close(_): """Close""" return 1 def get_size(_): """Loop through the metadata and calculate the complete virtual disk size""" global blockMap size = 0 for m in blockMap: # use only size from full backup image if m["file"] == image: size += m["length"] log(f"DISK SIZE: {size}") return size def _hexdump(data: bytearray, width: int = 16): """Hexdump for debugging, dumps requested blocks in an hexdump -C compatible format""" zero_count = 0 skip_mode = False for i in range(0, len(data), width): chunk = data[i : i + width] if all(byte == 0 for byte in chunk): zero_count += len(chunk) skip_mode = True continue # Skip this block if skip_mode: log(f"... ({zero_count} zero bytes skipped)") zero_count = 0 # Reset counter skip_mode = False hex_part = " ".join(f"{byte:02X}" for byte in chunk) ascii_part = "".join(chr(byte) if 32 <= byte < 127 else "." for byte in chunk) log(f"{i:08X} {hex_part:<{width * 3}} |{ascii_part}|") if zero_count > 0: log(f"... ({zero_count} zero bytes skipped)") def pread(h, buf, offset, _): """Return the right data during read operation. Function uses the generated block map to check where in the stream format the required block offset is to be found and maps it accordingly and returns requested data. There might be situations where this goes really wrong and usually this results in an non-readable disk image. By using the blockfilter plugin, it "should" work most of the time, because the requested block size does not span amongst multiple frames within the sparse backup format.. as it should match the physical sector size .. hopefully """ global blockMap global hexdump log(f"Handle: {h}") # get block where offset sort of matches data = bytearray() blockListFull = sorted( [ b for b in blockMap if b["originalOffset"] <= offset < (b["originalOffset"] + b["length"]) and not b["inc"] ], key=lambda x: x["originalOffset"], ) log("-------------------------------------------") log(f"matching blocklist from full backup: {blockListFull}") log("-------------------------------------------") if len(blockListFull) == 1: log("using first block from blocklist") block = blockListFull[0] else: log("Using block from full backup") block = blockListFull[-1] log(f"Processing block: {block}") # where to read in the stream file fileOffset = block["offset"] - block["originalOffset"] + offset dataFile = block["file"] log(f"READ FROM: {dataFile}") log(f"READ AT: {fileOffset}") log(f"READ: {len(buf)}") log(f"BLOCK LENGTH: {block['length']}") if len(buf) <= block["length"]: log("Block found contains all required data") if block["data"] is False: data += b"\0" * len(buf) else: data += os.pread(h, len(buf), fileOffset) buf[:] = data if hexdump == "1": _hexdump(data) return remaining = len(buf) - block["length"] included = block["length"] log(f"Read spans multiple blocks, need to read: {remaining} from next block.") log(f"Read available data size {included} from current block.") data += os.pread(h, included, fileOffset) count = block["count"] + 1 while len(data) != len(buf): log(f"Locate next available block number {count}") next_block = [b for b in blockMap if b["count"] == count] if not next_block: raise RuntimeError("Unable to locate next block") next_block = next_block[0] log(f"Found next block: {next_block}") if remaining >= next_block["length"]: log("Next block does not contain all remaining data") to_read = next_block["length"] count += 1 else: to_read = remaining if next_block["data"]: log(f"Read {to_read} data size at {next_block['offset']} from this block.") data += os.pread(h, to_read, next_block["offset"]) else: log(f"Next block contains zeroes, return {remaining} zeroes") data += b"\0" * to_read remaining -= to_read if remaining == 0: log("All requested data read") else: log(f"{remaining} data left to read, next block to request: {count}") if len(data) != len(buf): raise RuntimeError( f"Unexpected short read from file. Read: {len(data)} requested: {len(buf)}" ) if hexdump == "1": _hexdump(data) buf[:] = data ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/virtnbdbackup0000755000000000000000000004166715033267156015644 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import sys import signal import logging import argparse from typing import List from datetime import datetime from functools import partial from concurrent.futures import ThreadPoolExecutor, as_completed from nbd import __version__ as __nbdversion__ from libvirtnbdbackup import sighandle from libvirtnbdbackup import argopt from libvirtnbdbackup import __version__ from libvirtnbdbackup import virt from libvirtnbdbackup.objects import DomainDisk from libvirtnbdbackup.virt import checkpoint from libvirtnbdbackup import output from libvirtnbdbackup.output import stream from libvirtnbdbackup import common as lib from libvirtnbdbackup.logcount import logCount from libvirtnbdbackup import exceptions from libvirtnbdbackup.backup import partialfile from libvirtnbdbackup.backup import job from libvirtnbdbackup.backup import disk from libvirtnbdbackup.backup import metadata from libvirtnbdbackup.backup import check from libvirtnbdbackup.ssh.exceptions import sshError from libvirtnbdbackup.virt.exceptions import ( domainNotFound, connectionFailed, ) from libvirtnbdbackup.output.exceptions import OutputException def main() -> None: """Handle backup operation and settings.""" parser = argparse.ArgumentParser( description="Backup libvirt/qemu virtual machines", epilog=( "Examples:\n" " # full backup of domain 'webvm' with all attached disks:\n" "\t%(prog)s -d webvm -l full -o /backup/\n" " # incremental backup:\n" "\t%(prog)s -d webvm -l inc -o /backup/\n" " # differential backup:\n" "\t%(prog)s -d webvm -l diff -o /backup/\n" " # full backup, exclude disk 'vda':\n" "\t%(prog)s -d webvm -l full -x vda -o /backup/\n" " # full backup, backup only disk 'vdb':\n" "\t%(prog)s -d webvm -l full -i vdb -o /backup/\n" " # full backup, compression enabled:\n" "\t%(prog)s -d webvm -l full -z -o /backup/\n" " # full backup, create archive:\n" "\t%(prog)s -d webvm -l full -o - > backup.zip\n" " # full backup of vm operating on remote libvirtd:\n" "\t%(prog)s -U qemu+ssh://root@remotehost/system " "--ssh-user root -d webvm -l full -o /backup/\n" ), formatter_class=argparse.RawTextHelpFormatter, ) opt = parser.add_argument_group("General options") opt.add_argument("-d", "--domain", required=True, type=str, help="Domain to backup") opt.add_argument( "-l", "--level", default="copy", choices=["copy", "full", "inc", "diff", "auto"], type=str, help="Backup level. (default: %(default)s)", ) opt.add_argument( "-t", "--type", default="stream", type=str, choices=["stream", "raw"], help="Output type: stream or raw. (default: %(default)s)", ) opt.add_argument( "-r", "--raw", default=False, action="store_true", help="Include full provisioned disk images in backup. (default: %(default)s)", ) opt.add_argument( "-o", "--output", required=True, type=str, help="Output target directory" ) opt.add_argument( "-C", "--checkpointdir", required=False, default=None, type=str, help="Persistent libvirt checkpoint storage directory", ) opt.add_argument( "--scratchdir", default="/var/tmp", required=False, type=str, help="Target dir for temporary scratch file. (default: %(default)s)", ) opt.add_argument( "-S", "--start-domain", default=False, required=False, action="store_true", help="Start virtual machine if it is offline. (default: %(default)s)", ) opt.add_argument( "-i", "--include", default=None, type=str, help="Backup only disk with target dev name (-i vda)", ) opt.add_argument( "-x", "--exclude", default=None, type=str, help="Exclude disk(s) with target dev name (-x vda,vdb)", ) opt.add_argument( "-f", "--socketfile", default=f"/var/tmp/virtnbdbackup.{os.getpid()}", type=str, help="Use specified file for NBD Server socket (default: %(default)s)", ) opt.add_argument( "-n", "--noprogress", default=False, help="Disable progress bar", action="store_true", ) opt.add_argument( "-z", "--compress", default=False, type=int, const=2, nargs="?", help="Compress with lz4 compression level. (default: %(default)s)", action="store", ) opt.add_argument( "-w", "--worker", type=int, default=None, help=( "Amount of concurrent workers used " "to backup multiple disks. (default: amount of disks)" ), ) opt.add_argument( "-F", "--freeze-mountpoint", type=str, default=None, help=( "If qemu agent available, freeze only filesystems on specified mountpoints within" " virtual machine (default: all)" ), ) opt.add_argument( "-e", "--strict", default=False, help=( "Change exit code if warnings occur during backup operation. " "(default: %(default)s)" ), action="store_true", ) opt.add_argument( "--no-sparse-detection", default=False, help=( "Skip detection of sparse ranges during incremental or differential backup. " "(default: %(default)s)" ), action="store_true", ) opt.add_argument( "-T", "--threshold", type=int, default=None, help=("Execute backup only if threshold is reached."), ) remopt = parser.add_argument_group("Remote Backup options") argopt.addRemoteArgs(remopt) logopt = parser.add_argument_group("Logging options") logopt.add_argument( "-L", "--syslog", default=False, action="store_true", help="Additionally send log messages to syslog (default: %(default)s)", ) logopt.add_argument( "--quiet", default=False, action="store_true", help="Disable logging to stderr (default: %(default)s)", ) argopt.addLogColorArgs(logopt) debopt = parser.add_argument_group("Debug options") debopt.add_argument( "-q", "--qemu", default=False, action="store_true", help="Use Qemu tools to query extents.", ) debopt.add_argument( "-s", "--startonly", default=False, help="Only initialize backup job via libvirt, do not backup any data", action="store_true", ) debopt.add_argument( "-k", "--killonly", default=False, help="Kill any running block job", action="store_true", ) debopt.add_argument( "-p", "--printonly", default=False, help="Quit after printing estimated checkpoint size.", action="store_true", ) argopt.addDebugArgs(debopt) repository = output.target() args = lib.argparse(parser) lib.setThreadName() args.stdout = args.output == "-" args.sshClient = None args.diskInfo = [] args.offline = False if args.quiet is True: args.noprogress = True fileStream = stream.get(args, repository) try: if not args.stdout: fileStream.create(args.output) except OutputException as e: logging.error("Can't open output file: [%s]", e) sys.exit(1) if args.worker is not None and args.worker < 1: args.worker = 1 now = datetime.now().strftime("%m%d%Y%H%M%S") logFile = f"{args.output}/backup.{args.level}.{now}.log" fileLog = lib.getLogFile(logFile) or sys.exit(1) counter = logCount() # pylint: disable=unreachable lib.configLogger(args, fileLog, counter) lib.printVersion(__version__) logging.info("Backup level: [%s]", args.level) if args.compress is not False: logging.info("Compression enabled, level [%s]", args.compress) try: check.arguments(args) except exceptions.BackupException as e: logging.error(e) sys.exit(1) if partialfile.exists(args): sys.exit(1) if not args.checkpointdir: args.checkpointdir = f"{args.output}/checkpoints" else: logging.info("Store checkpoints in: [%s]", args.checkpointdir) fileStream.create(args.checkpointdir) def connectionError(_, reason, args): """Callback if the libvirt connection drops mid data transfer, used to potentially cleanup the leftover backup job. """ virConnectCloseReason = ( "Misc I/O error", "End-of-file from server", "Keepalive timer triggered", "Client side connection close", "Unknown", ) logging.error( "Libvirt connection error [%s], trying to reconnect", virConnectCloseReason[reason], ) try: virtClient = virt.client(args) except connectionFailed as e: logging.error("Unrecoverable connection error: %s", e) sys.exit(1) domObj = virtClient.getDomain(args.domain) if not args.offline: logging.error("Attempting to stop backup task") virtClient.stopBackup(domObj) sys.exit(1) try: virtClient = virt.client(args) domObj = virtClient.getDomain(args.domain) except domainNotFound as e: logging.error("%s", e) sys.exit(1) except connectionFailed as e: logging.error("Can't connect libvirt daemon: [%s]", e) sys.exit(1) virtClient._conn.registerCloseCallback( # pylint: disable=W0212 connectionError, args ) logging.info("Libvirt library version: [%s]", virtClient.libvirtVersion) logging.info("NBD library version: [%s]", __nbdversion__) try: check.vmfeature(virtClient, domObj) checkpoint.checkForeign(args, domObj) check.vmstate(args, virtClient, domObj) check.targetDir(args) except exceptions.BackupException as e: logging.error(e) sys.exit(1) except exceptions.CheckpointException: sys.exit(1) if args.raw is True and args.level in ("inc", "diff"): logging.warning( "Raw disks can't be included during incremental or differential backup." ) logging.warning("Excluding raw disks.") args.raw = False signal.signal( signal.SIGINT, partial(sighandle.Backup.catch, args, domObj, virtClient, logging), ) if args.level not in ("inc", "diff") and args.no_sparse_detection is True: args.no_sparse_detection = False vmConfig = virtClient.getDomainConfig(domObj) disks: List[DomainDisk] = virtClient.getDomainDisks(args, vmConfig) args.info = virtClient.getDomainInfo(vmConfig) if virtClient.getTPMDevice(vmConfig): logging.warning("Emulated TPM device attached: User action required.") logging.warning( "Please manually backup contents of: [/var/lib/libvirt/swtpm/%s/]", domObj.UUIDString(), ) try: check.diskformat(args, disks) except exceptions.BackupException as e: logging.info(e) if not disks: logging.error("Unable to detect disks suitable for backup.") metadata.saveFiles(args, vmConfig, disks, fileStream, logFile) sys.exit(1) try: check.blockjobs(args, virtClient, domObj, disks) except exceptions.BackupException as e: logging.error(e) sys.exit(1) logging.info( "Backup will save [%s] attached disks.", len(disks), ) if args.worker is None or args.worker > int(len(disks)): args.worker = int(len(disks)) logging.info("Concurrent backup processes: [%s]", args.worker) if args.killonly is True: logging.info("Stopping backup job") if not virtClient.stopBackup(domObj): sys.exit(1) sys.exit(0) try: checkpoint.create(args, domObj) except exceptions.CheckpointException as errmsg: logging.error(errmsg) sys.exit(1) if args.printonly and args.cpt.parent and not args.offline: size = checkpoint.getSize(domObj, args.cpt.parent) logging.info("Estimated checkpoint backup size: [%s] Bytes", size) sys.exit(0) if args.threshold and args.cpt.parent and not args.offline: size = checkpoint.getSize(domObj, args.cpt.parent) if size < args.threshold: logging.info( "Backup size [%s] does not meet required threshold [%s], skipping backup.", size, args.threshold, ) sys.exit(0) if virtClient.remoteHost != "": args.sshClient = lib.sshSession(args, virtClient.remoteHost) if not args.sshClient: logging.error("Remote backup detected but ssh session setup failed") sys.exit(1) logging.info( "Remote NBD Endpoint host: [%s]", virtClient.remoteHost, ) if args.offline is True: logging.info( "Remote ports used for backup: [%s-%s]", args.nbd_port, args.nbd_port + args.worker, ) else: logging.info("Local NBD Endpoint sockets:") for sdisk in disks: logging.info( "%s: [nbd+unix:///%s?socket=%s]", sdisk.target, sdisk.target, args.socketfile, ) if args.offline is not True: logging.info("Temporary scratch file target directory: [%s]", args.scratchdir) fileStream.create(args.scratchdir) if not job.start(args, virtClient, domObj, disks): sys.exit(1) if args.level not in ("copy", "diff") and args.offline is False: logging.info("Started backup job with checkpoint, saving information.") try: checkpoint.save(args) except exceptions.CheckpointException as e: logging.error("Extending checkpoint file failed: [%s]", e) sys.exit(1) if not checkpoint.backup(args, domObj): virtClient.stopBackup(domObj) sys.exit(1) if args.startonly is True: logging.info("Started backup job for debugging, exiting.") sys.exit(0) backupSize: int = 0 try: with ThreadPoolExecutor(max_workers=args.worker) as executor: futures = { executor.submit( disk.backup, args, Disk, count, fileStream, virtClient ): Disk for count, Disk in enumerate(disks) } for future in as_completed(futures): size, state = future.result() backupSize += size if state is not True: raise exceptions.DiskBackupFailed("Backup of one disk failed") except exceptions.BackupException as e: logging.error("Disk backup failed: [%s]", e) except sshError as e: logging.error("Remote Disk backup failed: [%s]", e) except Exception as e: # pylint: disable=broad-except logging.critical("Unknown Exception during backup: %s", e) logging.exception(e) if args.offline is False: logging.info("Backup jobs finished, stopping backup task.") virtClient.stopBackup(domObj) virtClient.close() metadata.saveFiles(args, vmConfig, disks, fileStream, logFile) if domObj.autostart() == 1: metadata.backupAutoStart(args) if counter.count.errors > 0: logging.error("Error during backup") sys.exit(1) if args.sshClient: args.sshClient.disconnect() if counter.count.warnings > 0 and args.strict is True: logging.info( "[%s] Warnings detected during backup operation, forcing exit code 2", counter.count.warnings, ) sys.exit(2) logging.info("Total saved disk data: [%s]", lib.humanize(backupSize)) logging.info("Finished successfully") if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1752002421.4964876 virtnbdbackup-2.30/virtnbdbackup.egg-info/0000755000000000000000000000000015033267565017376 5ustar00rootroot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002421.0 virtnbdbackup-2.30/virtnbdbackup.egg-info/PKG-INFO0000644000000000000000000000062615033267565020477 0ustar00rootrootMetadata-Version: 2.1 Name: virtnbdbackup Version: 2.30 Summary: Backup utility for libvirt Home-page: https://github.com/abbbi/virtnbdbackup/ Author: Michael Ablassmeier Author-email: abi@grinser.de License: GPL Keywords: libnbd backup libvirt Provides-Extra: dev Provides-Extra: docs Provides-Extra: testing License-File: LICENSE Backup utility for libvirt, using latest changed block tracking features ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002421.0 virtnbdbackup-2.30/virtnbdbackup.egg-info/SOURCES.txt0000644000000000000000000000456215033267565021271 0ustar00rootrootChangelog LICENSE MANIFEST.in README.md requirements.txt setup.cfg setup.py virtnbd-nbdkit-plugin virtnbdbackup virtnbdmap virtnbdrestore docker/Dockerfile docker/README.md libvirtnbdbackup/__init__.py libvirtnbdbackup/argopt.py libvirtnbdbackup/block.py libvirtnbdbackup/chunk.py libvirtnbdbackup/common.py libvirtnbdbackup/exceptions.py libvirtnbdbackup/logcount.py libvirtnbdbackup/lz4.py libvirtnbdbackup/objects.py libvirtnbdbackup/sighandle.py libvirtnbdbackup/backup/__init__.py libvirtnbdbackup/backup/check.py libvirtnbdbackup/backup/disk.py libvirtnbdbackup/backup/job.py libvirtnbdbackup/backup/metadata.py libvirtnbdbackup/backup/partialfile.py libvirtnbdbackup/backup/server.py libvirtnbdbackup/backup/target.py libvirtnbdbackup/extenthandler/__init__.py libvirtnbdbackup/extenthandler/extenthandler.py libvirtnbdbackup/map/__init__.py libvirtnbdbackup/map/changes.py libvirtnbdbackup/map/ranges.py libvirtnbdbackup/map/requirements.py libvirtnbdbackup/nbdcli/__init__.py libvirtnbdbackup/nbdcli/client.py libvirtnbdbackup/nbdcli/context.py libvirtnbdbackup/nbdcli/exceptions.py libvirtnbdbackup/output/__init__.py libvirtnbdbackup/output/exceptions.py libvirtnbdbackup/output/stream.py libvirtnbdbackup/output/target.py libvirtnbdbackup/qemu/__init__.py libvirtnbdbackup/qemu/command.py libvirtnbdbackup/qemu/exceptions.py libvirtnbdbackup/qemu/util.py libvirtnbdbackup/restore/__init__.py libvirtnbdbackup/restore/data.py libvirtnbdbackup/restore/disk.py libvirtnbdbackup/restore/files.py libvirtnbdbackup/restore/header.py libvirtnbdbackup/restore/image.py libvirtnbdbackup/restore/sequence.py libvirtnbdbackup/restore/server.py libvirtnbdbackup/restore/vmconfig.py libvirtnbdbackup/sparsestream/__init__.py libvirtnbdbackup/sparsestream/exceptions.py libvirtnbdbackup/sparsestream/streamer.py libvirtnbdbackup/sparsestream/types.py libvirtnbdbackup/ssh/__init__.py libvirtnbdbackup/ssh/client.py libvirtnbdbackup/ssh/exceptions.py libvirtnbdbackup/virt/__init__.py libvirtnbdbackup/virt/checkpoint.py libvirtnbdbackup/virt/client.py libvirtnbdbackup/virt/disktype.py libvirtnbdbackup/virt/exceptions.py libvirtnbdbackup/virt/fs.py libvirtnbdbackup/virt/xml.py man/virtnbdbackup.1 man/virtnbdmap.1 man/virtnbdrestore.1 virtnbdbackup.egg-info/PKG-INFO virtnbdbackup.egg-info/SOURCES.txt virtnbdbackup.egg-info/dependency_links.txt virtnbdbackup.egg-info/requires.txt virtnbdbackup.egg-info/top_level.txt././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002421.0 virtnbdbackup-2.30/virtnbdbackup.egg-info/dependency_links.txt0000644000000000000000000000000115033267565023444 0ustar00rootroot ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002421.0 virtnbdbackup-2.30/virtnbdbackup.egg-info/requires.txt0000644000000000000000000000015115033267565021773 0ustar00rootrootcolorlog libvirt-python>=6.0.0 lxml lz4>=2.1.2 paramiko tqdm typing_extensions [dev] [docs] [testing] ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002421.0 virtnbdbackup-2.30/virtnbdbackup.egg-info/top_level.txt0000644000000000000000000000002115033267565022121 0ustar00rootrootlibvirtnbdbackup ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/virtnbdmap0000755000000000000000000001735315033267156015147 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import sys import tempfile import signal import time import argparse import logging from functools import partial from libvirtnbdbackup import argopt from libvirtnbdbackup import __version__ from libvirtnbdbackup import sighandle from libvirtnbdbackup.map import changes from libvirtnbdbackup.map import ranges from libvirtnbdbackup.map import requirements from libvirtnbdbackup.qemu import util as qemu from libvirtnbdbackup.qemu.exceptions import ProcessError, QemuHelperError from libvirtnbdbackup.exceptions import RestoreError from libvirtnbdbackup.output.exceptions import OutputException from libvirtnbdbackup import common as lib from libvirtnbdbackup.logcount import logCount from libvirtnbdbackup.sparsestream import streamer from libvirtnbdbackup.sparsestream import types # pylint: disable=unreachable def main() -> None: """Map full backup file to nbd device for single file or instant recovery""" parser = argparse.ArgumentParser( description="Map backup image(s) to block device", epilog=( "Examples:\n" " # Map full backup to device /dev/nbd0:\n" "\t%(prog)s -f /backup/sda.full.data\n" " # Map full backup to device /dev/nbd2:\n" "\t%(prog)s -f /backup/sda.full.data -d /dev/nbd2\n" " # Map sequence of full and incremental to device /dev/nbd2:\n" "\t%(prog)s -f /backup/sda.full.data,/backup/sda.inc.1.data -d /dev/nbd2\n" ), formatter_class=argparse.RawTextHelpFormatter, ) opt = parser.add_argument_group("General options") opt.add_argument( "-f", "--file", required=True, type=str, help="List of Backup files to map" ) opt.add_argument( "-b", "--blocksize", required=False, type=str, default="4096", help="Maximum blocksize passed to nbdkit. (default: %(default)s)", ) opt.add_argument( "-d", "--device", default="/dev/nbd0", type=str, help="Target device. (default: %(default)s)", ) opt.add_argument( "-e", "--export-name", default="sda", type=str, help="Export name passed to nbdkit. (default: %(default)s)", ) opt.add_argument( "-t", "--threads", default=1, type=str, help="Amount of threads passed to nbdkit process. (default: %(default)s)", ) opt.add_argument( "-l", "--listen-address", default="127.0.0.1", type=str, help="IP Address for nbdkit process to listen on. (default: %(default)s)", ) opt.add_argument( "-p", "--listen-port", default="10809", type=str, help="Port for nbdkit process to listen on. (default: %(default)s)", ) opt.add_argument( "-n", "--noprogress", required=False, action="store_true", default=False, help="Disable progress bar", ) logopt = parser.add_argument_group("Logging options") argopt.addLogArgs(logopt, parser.prog) argopt.addLogColorArgs(logopt) debopt = parser.add_argument_group("Debug options") debopt.add_argument( "-r", "--readonly", required=False, action="store_true", help="Map image readonly (default: %(default)s)", ) debopt.add_argument( "-H", "--hexdump", required=False, action="store_true", help="Hexdump data to logfile for debugging (default: %(default)s)", ) argopt.addDebugArgs(debopt) args = lib.argparse(parser) args.sshClient = None args.quiet = False fileLog = lib.getLogFile(args.logfile) or sys.exit(1) lib.setThreadName() counter = logCount() lib.configLogger(args, fileLog, counter) lib.printVersion(__version__) nbdkitModule = requirements.plugin(args) logging.info("Logfile: [%s]", args.logfile) logging.info("Plugin location: [%s]", nbdkitModule) requirements.executables() requirements.device(args) dataFiles = args.file.split(",") if len(dataFiles) > 1 and not "full.data" in dataFiles[0]: logging.error("Sequence must start with a full backup") if len(dataFiles) > 1 and args.readonly: logging.error("Device mapping with incrementals doesn't work in readonly mode") if counter.count.errors > 0: sys.exit(1) fullImage = os.path.abspath(dataFiles[0]) stream = streamer.SparseStream(types) sTypes = types.SparseStreamTypes() # pylint: disable=consider-using-with blockMap = tempfile.NamedTemporaryFile(delete=False, prefix="block.", suffix=".map") logging.info("Write blockmap to temporary file: [%s]", blockMap.name) try: dataRanges = ranges.get(args, stream, sTypes, dataFiles) except RestoreError as e: logging.error(e) sys.exit(1) if not ranges.dump(blockMap, dataRanges): sys.exit(1) blockMap.flush() blockMap.close() logging.info("Target device: %s", args.device) qFh = qemu.util(args.export_name) try: nbdkitProcess = qFh.startNbdkitProcess( args, nbdkitModule, blockMap.name, fullImage ) except QemuHelperError as e: logging.error("Failed to start nbdkit process: [%s]", e) sys.exit(1) logging.info( "Started nbdkit process pid: [%s], Logfile: [%s]", nbdkitProcess.pid, nbdkitProcess.logFile, ) signal.signal( signal.SIGINT, partial(sighandle.Map.catch, args, nbdkitProcess, blockMap, logging), ) maxRetry = 10 retryCnt = 0 nbdCmd = [ "qemu-nbd", "-c", f"{args.device}", f"nbd://{args.listen_address}:{args.listen_port}/{args.export_name}", "-f", "raw", ] if args.readonly: logging.warning("Device will be mapped readonly without cow.") logging.warning("Mounting will only work with '-o norecovery,ro'") nbdCmd.append("-r") logging.debug(nbdCmd) while True: try: qemu.command.run(cmdLine=nbdCmd, toPipe=True) break except ProcessError as e: if retryCnt >= maxRetry: logging.info("Unable to connect device after service start: %s", e) lib.killProc(nbdkitProcess.pid) break if "Connection refused" in str(e): logging.info("NBD server refused connection, retry [%s]", retryCnt) time.sleep(1) retryCnt += 1 else: logging.error("Failed to map device:") logging.error("Stderr: [%s]", str(e)) lib.killProc(nbdkitProcess.pid) if len(dataFiles) > 1: try: changes.replay(dataRanges, args) except OutputException as e: logging.error("Failed to replay changes: %s", e) lib.killProc(nbdkitProcess.pid) sys.exit(1) logging.info("Done mapping backup image to [%s]", args.device) logging.info("Press CTRL+C to disconnect") while True: time.sleep(60) if __name__ == "__main__": main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1752002158.0 virtnbdbackup-2.30/virtnbdrestore0000755000000000000000000002323715033267156016053 0ustar00rootroot#!/usr/bin/python3 """ Copyright (C) 2023 Michael Ablassmeier This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . """ import os import io import sys import logging import argparse from typing import List from libvirtnbdbackup import argopt from libvirtnbdbackup import __version__ from libvirtnbdbackup import virt from libvirtnbdbackup.restore import vmconfig from libvirtnbdbackup.restore import files from libvirtnbdbackup.restore import sequence from libvirtnbdbackup.restore import disk from libvirtnbdbackup import output from libvirtnbdbackup import common as lib from libvirtnbdbackup.logcount import logCount from libvirtnbdbackup.sparsestream import streamer from libvirtnbdbackup.sparsestream import types from libvirtnbdbackup.ssh import Mode from libvirtnbdbackup.virt.exceptions import connectionFailed from libvirtnbdbackup.exceptions import RestoreError def main() -> None: """main function""" defaultConfig = "vmconfig.xml" parser = argparse.ArgumentParser( description="Restore virtual machine disks", epilog=( "Examples:\n" " # Dump backup metadata:\n" "\t%(prog)s -i /backup/ -o dump\n" " # Verify checksums for existing data files in backup:\n" "\t%(prog)s -i /backup/ -o verify\n" " # Complete restore with all disks:\n" "\t%(prog)s -i /backup/ -o /target\n" " # Complete restore, adjust config and redefine vm after restore:\n" "\t%(prog)s -cD -i /backup/ -o /target\n" " # Complete restore, adjust config and redefine vm with name 'foo':\n" "\t%(prog)s -cD --name foo -i /backup/ -o /target\n" " # Restore only disk 'vda':\n" "\t%(prog)s -i /backup/ -o /target -d vda\n" " # Point in time restore:\n" "\t%(prog)s -i /backup/ -o /target --until virtnbdbackup.2\n" " # Restore and process specific file sequence:\n" "\t%(prog)s -i /backup/ -o /target " "--sequence vdb.full.data,vdb.inc.virtnbdbackup.1.data\n" " # Restore to remote system:\n" "\t%(prog)s -U qemu+ssh://root@remotehost/system" " --ssh-user root -i /backup/ -o /remote_target" ), formatter_class=argparse.RawTextHelpFormatter, ) opt = parser.add_argument_group("General options") opt.add_argument( "-a", "--action", required=False, type=str, choices=["dump", "restore", "verify"], default="restore", help="Action to perform: (default: %(default)s)", ) opt.add_argument( "-i", "--input", required=True, type=str, help="Directory including a backup set", ) opt.add_argument( "-o", "--output", required=True, type=str, help="Restore target directory" ) opt.add_argument( "-u", "--until", required=False, type=str, help="Restore only until checkpoint, point in time restore.", ) opt.add_argument( "-s", "--sequence", required=False, type=str, default=None, help="Restore image based on specified backup files.", ) opt.add_argument( "-d", "--disk", required=False, type=str, default=None, help="Process only disk matching target dev name. (default: %(default)s)", ) opt.add_argument( "-n", "--noprogress", required=False, action="store_true", default=False, help="Disable progress bar", ) opt.add_argument( "-f", "--socketfile", default=f"/var/tmp/virtnbdbackup.{os.getpid()}", type=str, help="Use specified file for NBD Server socket (default: %(default)s)", ) opt.add_argument( "-r", "--raw", default=False, action="store_true", help="Copy raw images as is during restore. (default: %(default)s)", ) opt.add_argument( "-c", "--adjust-config", default=False, action="store_true", help="Adjust vm configuration during restore. (default: %(default)s)", ) opt.add_argument( "-D", "--define", default=False, action="store_true", help="Register/define VM after restore. (default: %(default)s)", ) opt.add_argument( "-C", "--config-file", default=defaultConfig, type=str, help=f"Name of the vm config file used for restore. (default: {defaultConfig})", ) opt.add_argument( "-N", "--name", default=None, type=str, help="Define restored domain with specified name", ) opt.add_argument( "-B", "--buffsize", default=io.DEFAULT_BUFFER_SIZE, type=int, help="Buffer size to use during verify (default: %(default)s)", ) opt.add_argument( "-A", "--preallocate", default=False, action="store_true", help="Preallocate restored qcow images. (default: %(default)s)", ) remopt = parser.add_argument_group("Remote Restore options") argopt.addRemoteArgs(remopt) logopt = parser.add_argument_group("Logging options") argopt.addLogArgs(logopt, parser.prog) argopt.addLogColorArgs(logopt) debopt = parser.add_argument_group("Debug options") argopt.addDebugArgs(debopt) args = lib.argparse(parser) args.quiet = False args.sshClient = None # default values for common usage of lib.getDomainDisks args.exclude = None args.include = args.disk lib.setThreadName() stream = streamer.SparseStream(types) fileLog = lib.getLogFile(args.logfile) or sys.exit(1) counter = logCount() # pylint: disable=unreachable lib.configLogger(args, fileLog, counter) lib.printVersion(__version__) if not lib.exists(args, args.input): logging.error("Backup source [%s] does not exist.", args.input) sys.exit(1) dataFiles: List[str] = [] if args.sequence is not None: logging.info("Using manual specified sequence of files.") logging.info("Disabling redefine and config adjust options.") args.define = False args.adjust_config = False dataFiles = args.sequence.split(",") if "full" not in dataFiles[0] and "copy" not in dataFiles[0]: logging.error("Sequence must start with full or copy backup.") sys.exit(1) else: dataFiles = lib.getLatest(args.input, "*.data") if not dataFiles: logging.error("No data files found in directory: [%s]", args.input) sys.exit(1) if args.action == "dump" or args.output == "dump": files.dump(args, stream, dataFiles) sys.exit(0) if args.action == "verify" or args.output == "verify": if not files.verify(args, dataFiles): sys.exit(1) sys.exit(0) if args.action == "restore": if args.define is True: args.adjust_config = True try: virtClient = virt.client(args) except connectionFailed as e: logging.error("Unable to connect libvirt: [%s]", e) sys.exit(1) if virtClient.remoteHost: if not args.output.startswith("/"): logging.error( "Absolute target path required for restore to remote system" ) sys.exit(1) args.sshClient = lib.sshSession( args, virtClient.remoteHost, mode=Mode.UPLOAD ) if not args.sshClient: logging.error("Remote restore detected but ssh session setup failed") sys.exit(1) if not args.sshClient.exists(args.output): logging.info("Create target directory: [%s]", args.output) args.sshClient.sftp.mkdir(args.output) else: output.target.Directory().create(args.output) ConfigFiles = lib.getLatest(args.input, "vmconfig*.xml") if not ConfigFiles: logging.error("No domain config file found") sys.exit(1) if args.until is not None: ConfigFile = ConfigFiles[int(args.until.split(".")[-1])] else: ConfigFile = ConfigFiles[-1] logging.info("Using config file: [%s]", ConfigFile) autoStart = False if lib.getLatest(args.input, "autostart.*", -1): autoStart = True restConfig: bytes = b"" try: if args.sequence is not None: sequence.restore(args, dataFiles, virtClient) else: restConfig = disk.restore(args, ConfigFile, virtClient) except RestoreError as errmsg: logging.error("Disk restore failed: [%s]", errmsg) sys.exit(1) files.restore(args, ConfigFile, virtClient) vmconfig.restore(args, ConfigFile, restConfig, args.config_file) virtClient.refreshPool(args.output) if args.define is True: if not virtClient.defineDomain(restConfig, autoStart): sys.exit(1) if __name__ == "__main__": main()