This guide provides technical details for deploying Proxmox VE with Blockbridge iSCSI and NVME storage using the Blockbridge storage driver for Proxmox.

Most readers will want to start with the Quickstart section. It’s an ordered list of configuration steps and is the fastest path to an installation. The rest of the document provides details on all aspects of using Proxmox with Blockbridge.

Last updated on Apr 10, 2024.

FEATURE OVERVIEW

Formats & Content Types

Blockbridge provides block-level storage optimized for performance, security, and efficiency. Block storage is used by Proxmox to store raw disk images. Disk images are attached to virtual machines and typically formatted with a filesystem for use by the guest.

Proxmox supports several built-in storage types. Environments with existing enterprise or datacenter storage systems can use the LVM or iSCSI/kernel storage types for shared storage in support of high-availability. For service providers, however, these solutions are simply not scalable. The configuration management required to implement and maintain Proxmox on traditional shared storage systems is too large a burden. We developed our Proxmox-native driver specifically to address these challenges.

The table below provides a high-level overview of the capabilities of popular block storage types. For a complete list of storage types, visit the Proxmox Storage Wiki.

Note 1: LVM and iSCSI inherit the availability characteristics of the underlying storage.
Note 2: LVM can be deployed on iSCSI-based storage to achieve shared storage.

High Availability

Blockbridge provides highly-available storage that is self-healing. Controlplane (i.e., API) and dataplane (i.e., iSCSI, NVME) services transparently failover in the event of hardware failure. Depending on your network configuration, it may be appropriate to deploy Linux multipathing for protection against network failure. The Blockbridge driver supports automated multipath management.

Multi-Tenancy & Multi-Proxmox

Blockbridge implements features critical for multi-tenant environments, including management segregation, automated performance shaping, and always-on encryption. The Blockbridge driver leverages these functions, allowing you to create storage pools dedicated for different users, applications, and performance tiers. Service providers can safely deploy multiple Proxmox clusters on Blockbridge storage without the risk of collision.

High Performance

Blockbridge is heavily optimized for performance. Expect approximately a 5x write latency and IOPS advantage when compared to native Proxmox CEPH/RBD solution. Optionally, the Blockbridge driver can tune your hosts for the best possible latency and performance.

At-Rest & In-Flight Encryption

Blockbridge implements always-on per-virtual disk encryption, automated key management, and instant secure erase for at-rest security. The Blockbridge driver also supports in-flight encryption for end-to-end protection.

Snapshots & Clones

Snapshots and Clones are thin and instantaneous. Both technologies take advantage of an allocate-on-write storage architecture for significantly improved latency compared to copy-on-write strategies.

Blockbridge 5.2.0 adds support for rolling back to the most recent snapshot. Support for snapshot rollback is available in version 2.1.0+ of our Proxmox Plugin.

Thin Provisioning & Data Reduction

Blockbridge supports thin-provisioning, pattern elimination, and latency-optimized adaptive data reduction. These features are transparent to Proxmox.

QUICKSTART

This is a quick reference for installing and configuring the Blockbridge Proxmox VE shared storage plugin.

Some of these topics have more information available by selecting the information ⓘ links next to items where they appear.

Driver Installation

Drivers for Blockbridge Version 6 Storage

1. Import the Blockbridge release signing key. ⓘ

   curl -fsSL https://get.blockbridge.com/tools/6.0/debian/blockbridge-archive-keyring.gpg > \
     /usr/share/keyrings/blockbridge-archive-keyring.gpg
   

2. Verify the key fingerprint. ⓘ

   gpg --show-keys --fingerprint /usr/share/keyrings/blockbridge-archive-keyring.gpg
   

   pub   rsa4096 2016-11-01 [SC]
         9C1D E2AE 5970 CFD4 ADC5  E0BA DDDE 845D 7ECF 5373
   uid                      Blockbridge (Official Signing Key) <security@blockbridge.com>
   sub   rsa4096 2016-11-01 [E]
   

3. Add the Blockbridge Tools repository and install the plugin. ⓘ

   curl -fsSL https://get.blockbridge.com/tools/6.0/debian/blockbridge-$(source /etc/os-release && echo "$VERSION_CODENAME").sources > \
     /etc/apt/sources.list.d/blockbridge-tools.sources
   apt update
   apt install blockbridge-proxmox
   

Drivers for Blockbridge Version 5 Storage

1. Import the Blockbridge release signing key. ⓘ

   curl -fsSL https://get.blockbridge.com/tools/5.2/debian/blockbridge-archive-keyring.gpg > \
     /usr/share/keyrings/blockbridge-archive-keyring.gpg
   

2. Verify the key fingerprint. ⓘ

   gpg --show-keys --fingerprint /usr/share/keyrings/blockbridge-archive-keyring.gpg
   

   pub   rsa4096 2016-11-01 [SC]
         9C1D E2AE 5970 CFD4 ADC5  E0BA DDDE 845D 7ECF 5373
   uid                      Blockbridge (Official Signing Key) <security@blockbridge.com>
   sub   rsa4096 2016-11-01 [E]
   

3. Add the Blockbridge Tools repository and install the plugin. ⓘ

   curl -fsSL https://get.blockbridge.com/tools/5.2/debian/blockbridge-$(source /etc/os-release && echo "$VERSION_CODENAME").sources > \
     /etc/apt/sources.list.d/blockbridge-tools.sources
   apt update
   apt install blockbridge-proxmox
   

Authentication Token

This section describes creating a dedicated Blockbridge account for your Proxmox storage, and then creating an authorization token to use it. These steps only need to happen once.

1. Log in to your Blockbridge controlplane as the system user. ⓘ

    root@proxmox-1:~# bb auth login
    Enter a default management host: blockbridge.yourcompany.com
    Authenticating to https://blockbridge.yourcompany.com/api
   
    Enter user or access token: system
    Password for system:
    Authenticated; token expires in 3599 seconds.
    == Authenticated as user system.
   

2. Create a dedicated proxmox account. ⓘ

    root@proxmox-1:~# bb account create --name proxmox
   

3. Use the ‘substitute user’ option to switch your session to the newly created proxmox account. ⓘ

   Note that you will have to re-authenticate as the system user.

    root@proxmox-1:~# bb auth login --su proxmox
    Authenticating to https://blockbridge.yourcompany.com/api
   
    Enter user or access token: system
    Password for system: ......
    Authenticated; token expires in 3599 seconds.
   
    == Authenticated as user proxmox.
   

4. Create a persistent authorization token. ⓘ

    root@proxmox-1:~# bb authorization create --notes "Proxmox Cluster token"
    == Created authorization: ATH4762194C412D97FE
    ... [output trimmed] ...
   
    == Access Token
    access token          1/LtVVws54+bGvb/l...njz8A
   

   Remember to record your access token!

Proxmox Configuration

1. Configure a blockbridge storage backend by adding a new storage object

    pvesm add blockbridge shared-block-gp -api_url https://blockbridge.yourcompany.com/api \
    -auth_token 1/nalF+/S1pO............2qitqUX79LWtpw -ssl_verify_peer 0
   

2. Restart the pvedaemon, pveproxy, pvescheduler and pvestatd services. ⓘ

   Though the configuration is automatically synchronized to all Proxmox nodes, you must restart services on all Proxmox nodes.

    systemctl try-reload-or-restart pvedaemon pveproxy pvestatd pvescheduler
   

DEPLOYMENT & MANAGEMENT

This section describes how to install and configure the Blockbridge Proxmox storage plugin. To see what’s changed in each plugin release, check out Version History.

Supported Versions

Driver Packages

Import the Blockbridge Archive Signing Key

Blockbridge Version 6 Storage

Install the Blockbridge archive signing key:

curl -fsSL https://get.blockbridge.com/tools/6.0/debian/blockbridge-archive-keyring.gpg > \
  /usr/share/keyrings/blockbridge-archive-keyring.gpg

Verify the key fingerprint:

gpg --show-keys --fingerprint /usr/share/keyrings/blockbridge-archive-keyring.gpg

pub   rsa4096 2016-11-01 [SC]
      9C1D E2AE 5970 CFD4 ADC5  E0BA DDDE 845D 7ECF 5373
uid                      Blockbridge (Official Signing Key) <security@blockbridge.com>
sub   rsa4096 2016-11-01 [E]

Blockbridge Version 5 Storage

Install the Blockbridge archive signing key:

curl -fsSL https://get.blockbridge.com/tools/5.2/debian/blockbridge-archive-keyring.gpg > \
  /usr/share/keyrings/blockbridge-archive-keyring.gpg

Verify the key fingerprint:

gpg --show-keys --fingerprint /usr/share/keyrings/blockbridge-archive-keyring.gpg

pub   rsa4096 2016-11-01 [SC]
      9C1D E2AE 5970 CFD4 ADC5  E0BA DDDE 845D 7ECF 5373
uid                      Blockbridge (Official Signing Key) <security@blockbridge.com>
sub   rsa4096 2016-11-01 [E]

Add the Blockbridge Tools Repository and Install the Plugin

Add the tools apt repository, and install the Blockbridge storage plugin:

Blockbridge Version 6 Storage

curl -fsSL https://get.blockbridge.com/tools/6.0/debian/blockbridge-$(source /etc/os-release && echo "$VERSION_CODENAME").sources > \
  /etc/apt/sources.list.d/blockbridge-tools.sources
apt update
apt install blockbridge-proxmox

Blockbridge Version 5 Storage

curl -fsSL https://get.blockbridge.com/tools/5.2/debian/blockbridge-$(source /etc/os-release && echo "$VERSION_CODENAME").sources > \
  /etc/apt/sources.list.d/blockbridge-tools.sources
apt update
apt install blockbridge-proxmox

Optional Packages

To use TLS transport encryption for iSCSI traffic, install the stunnel package.

apt install stunnel

Driver Options

Driver Authentication

Create a persistent authorization for Proxmox use

Log in to your Blockbridge controlplane as the system user.

root@proxmox-1:~# bb auth login
Enter a default management host: blockbridge.yourcompany.com
Authenticating to https://blockbridge.yourcompany.com/api

Enter user or access token: system
Password for system:
Authenticated; token expires in 3599 seconds.
== Authenticated as user system.

Create a dedicated proxmox account for storage and management isolation.

root@proxmox-1:~# bb account create --name proxmox
== Created account: proxmox (ACT0762194C407BA625)

== Account: proxmox (ACT0762194C407BA625)
name                  proxmox
label                 proxmox
serial                ACT0762194C407BA625
created               2021-01-27 16:58:53 -0500
disabled              no

With the system username and password, use the “substitute user” function to switch to the newly created proxmox account:

root@proxmox-1:~# bb auth login --su proxmox
Authenticating to https://blockbridge.yourcompany.com/api

Enter user or access token: system
Password for system: ......
Authenticated; token expires in 3599 seconds.

== Authenticated as user proxmox.

Create a persistent authorization for use by the Blockbridge storage plugin.

root@proxmox-1:~# bb authorization create --notes "Proxmox Cluster token"
== Created authorization: ATH4762194C412D97FE

== Authorization: ATH4762194C412D97FE
notes                 Proxmox Cluster token
serial                ATH4762194C412D97FE
account               proxmox (ACT0762194C407BA625)
user                  proxmox (USR1B62194C407BA0E5)
enabled               yes
created at            2021-01-27 16:59:08 -0500
access type           online
token suffix          rDznjz8A
restrict              auth
enforce 2-factor      false

== Access Token
access token          1/LtVVws54+bGvb/l...njz8A

*** Remember to record your access token!

Proxmox Storage Definition

Configure a blockbridge storage backend by adding a new storage object:

pvesm add blockbridge shared-block-gp -api_url https://blockbridge.yourcompany.com/api \
-auth_token 1/nalF+/S1pO............2qitqUX79LWtpw -ssl_verify_peer 0

Alternatively manually add a new section to /etc/pve/storage.cfg. The /etc/pve directory is an automatically synchronized filesystem (proxmox cluster filesystem, or just pmxcfs), so you only need to edit the file on a single node; the changes are synchronized to all cluster members.

For example, edit storage.cfg to add this section:

blockbridge: shared-block-gp
        api_url https://blockbridge.yourcompany.com/api
        auth_token 1/nalF+/S1pO............2qitqUX79LWtpw
        ssl_verify_peer 0

After adding new storage, editing storage.cfg, or updating the blockbridge plugin, restart the pvedaemon, pveproxy, pvescheduler and pvestatd services.

systemctl try-reload-or-restart pvedaemon pveproxy pvestatd pvescheduler

Upgrading the Blockbridge Plugin

Take extra care when upgrading core PVE packages: new Proxmox releases are frequently accompanied by Storage Plugin API changes, which need a corresponding Blockbridge Plugin update. Always deploy to a staging environment before going live to production!

These instructions are careful to only install Blockbridge package updates. Follow these instructions to upgrade the Blockbridge storage plugin using the apt package management CLI. If preferred, the web-based Proxmox management interface can be used to list and install package updates.

Upgrade via the CLI

Upgrades must be performed on all proxmox nodes. For environments with many nodes, we strongly recommend using a configuration management tool, such as Ansible to orchestrate package updates.

First, update the list of available packages:

# apt update

Check to see if any updated Blockbridge packages are available:

# apt list --upgradable -q blockbridge-\*
Listing...
blockbridge-proxmox/bullseye 2.3.2-193~bullseye1 all [upgradable from: 2.3.1-161~bullseye1]

In this example, an update from 2.3.1 to 2.3.2 is available. Take a look at Version History to see a summary of changes for a given release.

Install the updated blockbridge-proxmox package using the apt install command:

# apt install blockbridge-proxmox
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
  blockbridge-proxmox
1 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.
Need to get 21.0 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 https://get.blockbridge.com/tools/5.2/debian bullseye/main amd64 blockbridge-proxmox all 2.3.2-193~bullseye1 [21.0 kB]
Fetched 21.0 kB in 0s (130 kB/s)
Reading changelogs... Done
(Reading database ... 87522 files and directories currently installed.)
Preparing to unpack .../blockbridge-proxmox_2.3.2-193~bullseye1_all.deb ...
Unpacking blockbridge-proxmox (2.3.2-193~bullseye1) over (2.3.1-161~bullseye1) ...
Setting up blockbridge-proxmox (2.3.2-193~bullseye1) ...

Finally, reload PVE services to load the updated driver:

# systemctl try-reload-or-restart pvedaemon pveproxy pvestatd pvescheduler

The Unattended Upgrades Service

By default, PVE is configured to perform unattended daily package upgrades. To confirm your system is configured for automatic updates, use the apt-config tool:

# apt-config dump APT::Periodic::Unattended-Upgrade
APT::Periodic::Unattended-Upgrade "1";

# apt-config dump Unattended-Upgrade::Origins-Pattern
Unattended-Upgrade::Origins-Pattern "";
Unattended-Upgrade::Origins-Pattern:: "origin=Debian,codename=${distro_codename},label=Debian";
Unattended-Upgrade::Origins-Pattern:: "origin=Debian,codename=${distro_codename},label=Debian-Security";
Unattended-Upgrade::Origins-Pattern:: "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";

The APT::Perodic::Unattended-Upgrade value of "1" indicates unattended-upgrades will execute once per day; a value of 0 disables unattended upgrades. The patterns specified in Unattended-Upgrade::Origins-Pattern list what package origins are eligible for unattended upgrade. By default, security updates and general updates for the current named Debian release are considered.

Blockbridge software is published using an origin of Blockbridge, so will not be automatically updated. Depending on your security policy and appetite for surprise package updates, you may want to adjust the unattended-upgrades configuration. We recommend disabling automatic updating for all but critical security updates. To do this, edit /etc/apt/apt.conf.d/50unattended-upgrades and comment out the non-security origins. After these changes, the Unattended-Upgrade::Origins-Pattern setting will look something like this:

Unattended-Upgrade::Origins-Pattern {
        // Codename based matching:
        // This will follow the migration of a release through different
        // archives (e.g. from testing to stable and later oldstable).
        // Software will be the latest available for the named release,
        // but the Debian release itself will not be automatically upgraded.
//      "origin=Debian,codename=${distro_codename}-updates";
//      "origin=Debian,codename=${distro_codename}-proposed-updates";
//      "origin=Debian,codename=${distro_codename},label=Debian";
        "origin=Debian,codename=${distro_codename},label=Debian-Security";
        "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";

        // Archive or Suite based matching:
        // Note that this will silently match a different release after
        // migration to the specified archive (e.g. testing becomes the
        // new stable).
//      "o=Debian,a=stable";
//      "o=Debian,a=stable-updates";
//      "o=Debian,a=proposed-updates";
//      "o=Debian Backports,a=${distro_codename}-backports,l=Debian Backports";
};

Finally, confirm that only security origins are enabled using apt-config:

# apt-config dump Unattended-Upgrade::Origins-Pattern
Unattended-Upgrade::Origins-Pattern "";
Unattended-Upgrade::Origins-Pattern:: "origin=Debian,codename=${distro_codename},label=Debian-Security";
Unattended-Upgrade::Origins-Pattern:: "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";

Troubleshooting

The Blockbridge plugin logs all interactions with both Proxmox and your Blockbridge installation to syslog at LOG_INFO level. You can see the logs with journalctl -f | grep blockbridge:.

If you need to further diagnose Proxmox integration, inspect storage configuration, or perform low-level Blockbridge API operations, the bbpve command line utility will come in handy.

Inspect software components and pool configuration

Use bbpve to summarize package and kernel versions:

root@pmx02:~# bbpve --version
Component versions:

OS:     Debian GNU/Linux 11 (bullseye)
PVE:    pve-manager/7.1-11/8d529482 (running kernel: 5.15.27-1-pve)
Plugin: 2.3.2-193~bullseye1
CLI:    5.3.0-1779~bullseye1

Display a compact summary of all Blockbridge storage pools. This display omits the API access token, to avoid inadvertently leaking sensitive information:

root@pmx02:~# bbpve --pools
bb1
    api_url https://blockbridge-01.example.com/api

bb2
    api_url https://blockbridge-02.example.com/api
    transport_encryption tls

bb2-mpath
    api_url https://blockbridge-02.example.com/api
    multipath 1

Issue low-level commands directly to the Blockbridge API

bbpve provides a wrapper around the underlying Blockbridge command line suite. This allows you to execute the CLI in the context of a defined Proxmox storage pool.

As an example, say we wanted to enumerate the storage devices logically contained within the bb1 storage pool. Instead of logging in to the correct account by hand using bb auth login, we can simply execute the operation using the configured api endpoint and access token:

root@pmx1:~# bbpve bb1 disk list
label [2]  serial               vss [1]               capacity  size      size limit  status
---------  -------------------  --------------------  --------  --------  ----------  ------
base       DSK1962194C4062644E  bb1:base-100-disk-0   40.0GiB   2.333GiB  none        online
base       DSK1962194C40626417  bb1:vm-100-cloudinit  4.0MiB    128.0KiB  none        online
base       DSK1962194C406268BA  bb1:vm-101-cloudinit  4.0MiB    128.0KiB  none        online
base       DSK1962194C4062697E  bb1:vm-102-cloudinit  4.0MiB    128.0KiB  none        online
base       DSK1962194C40626907  bb1:vm-102-disk-1     1.0GiB    0b        none        online
base       DSK1962194C40626CA8  bb1:vm-105-disk-0     32.0GiB   0b        none        online
base       DSK1962194C406269DD  bb1:vm-110-cloudinit  4.0MiB    128.0KiB  none        online
base       DSK1962194C406268A2  bb1:vm-110-disk-0     40.0GiB   2.453GiB  none        online
base       DSK1962194C406269C5  bb1:vm-110-disk-1     2.0GiB    0b        none        online
base       DSK1962194C40626984  bb1:vm-110-disk-2     112.0MiB  23.0MiB   none        online
base       DSK1962194C406269BC  bb1:vm-110-disk-3     112.0MiB  23.0MiB   none        online
base       DSK1962194C406269FD  bb1:vm-111-cloudinit  4.0MiB    0b        none        online
base       DSK1962194C406269E5  bb1:vm-111-disk-0     40.0GiB   2.453GiB  none        online
base       DSK1962194C4062699C  bb1:vm-111-disk-1     2.0GiB    0b        none        online
base       DSK1962194C40626FC2  bb1:vm-200-disk-2     1.0GiB    0b        none        online

This can be useful in diagnosing configuration errors, tuning storage performance, or making adjustments to disk configuration that’s not currently possible using the PVE interface directly.

Blockbridge Plugin Version History

3.1.0 - Mon, 05 June 2023

This update brings features, bug fixes, and performance improvements.

We’ve added selectors to handle multi-tenant Proxmox deployments where the backend storage system contains dedicated networking interfaces corresponding to customer isolated networks. This feature ensures that a Proxmox deployment selects the optimal physical network for storage connectivity.

We’ve improved large-scale startup and migration performance by limiting vdisk inspection when attaching w/ multipath enabled. Performance gains correspond to reduced API calls to the Blockbridge control plane.

Additional performance gains were realized by converting the volume inspection helper function to use direct API calls instead of the CLI, eliminating CLI startup overhead.

We also patched a potential DoS vulnerability related to automatic LVM detection. The driver now automatically edits the LVM global_filter to skip plugin-managed block devices.

3.0.1 - Mon, 08 May 2023

This is a minor update containing bug fixes and performance improvements.

Minor improvements for processing udev change events.

3.0.0 - Mon, 03 Oct 2022

Major release adding NVME/TCP support.

Both iSCSI and NVMe are supported. NVMe support requires Proxmox 7.2.

2.3.3 - Mon, 02 May 2022

This update fixes a rare multipath volume deactivation issue.

A poorly timed administrative LVM probe could interfere with volume decativation, leaving a stranded multipath map device behind. The stranded device had I/O queueing enabled and no valid paths – any subsequent attempt to access the device resulted in a hung process.

This issue has been fixed by ensuring I/O queueing is disabled prior to removing the multipath device.

2.3.2 - Mon, 21 Mar 2022

This update brings bug fixes and a performance-optimized overhaul of our device management rules.

By default, most Linux distributions (including Debian and PVE) are configured for traditional server or desktop computing. Attached storage devices containing partitions, filesystems, and LVM volumes are probed to facilitate automatic mounting procedures. The end goal is to make interactive use and administration more streamlined. None of this makes sense when running a hypervisor, where the guests are the ultimate consumer of storage devices. Problems with the default model include:

• Device links for guest devices, partitions and filesystems clutter the /dev/disk/... directory tree.
• Collisions become much more likely, and are guaranteed when using a cloud-init drive for guest configuration (every cloudinit drive shares the cidata label).
• Bottlenecks in device attachment directly impact VM startup time, and waste hypervisor resources.

With this update, these problematic and unnecessary probing rules are avoided for Blockbridge managed storage. This dramatically improves the performance and reliability of Linux device management. There’s a significant reduction in CPU utilization during system reboot and virtual machine migration. If you’ve ever wondered why the udev daemon is consuming all of your CPU for minutes at a time, this update is for you!

Bugfixes

• Fix compatibility with older Perl dependecies in PVE6.
• Clean up the occasional stranded iSCSI session on volume deactivation.

2.3.1 - Thu, 24 Feb 2022

This patch release fixes an occasional failure when moving a disk between virtual machines.

Under the covers, the qm move-disk command uses the rename volume storage operation. In Proxmox, disk ownership is encoded directly in the volume label: moving a disk between virtual machines results in a single backend label update.

Unfortunately, if a disk was renamed while attached its device path would no longer be correct. This is now fixed by explicitly detaching at the start of a volume rename operation.

2.3.0 - Fri, 18 Feb 2022

This release focuses on improved integration, reliability, and performance with Linux multipathing.

• Ignore dm-multipath partition devices.
• Storage pools now support switching between single-path and multipath.
• Improve interaction with multipathd, especially for read-only media.
• Fix online volume resize when using multipath.

2.2.1 - Fri, 21 Jan 2022

This is a minor release, primarily focused on bugfixes and supportability enhancements.

• Unexpected task failures now persistently log additional diagnostic state.
• The storage plugin now includes bbpve, a management and diagnostics tool. See the Support section for more details.

Bugfixes

• Fixed a couple remaining paths which allowed tainted data to be suppled to Proxmox.
• Fixed a (harmless, but annoying) warning about undefined size values.

2.2.0 - Wed, 08 Dec 2021

PVE7.1 Support

• Support Proxmox APIVER 10.
• Added support for volume rename.
• Added support for the protected volume flag.

Bugfixes

• Closed a window where volume activation could succeed before its associated devpath was populated. This most often occurred with cloud-init drives.
• Fix multipath devpath management.
• Clean up handling of certificate verification errors.

2.1.1 - Tue, 26 Oct 2021

Bugfixes

• Update blockbridge-cli minimum version to 5.2.1: fix host attach –repair so it leaves healthy connections intact.
• Change the meaning of vss_label_prefix; it’s now a an enum which defaults to ‘pool’. To disable vss label prefixing, set to ‘none’.

2.1.0 - Tue, 24 Aug 2021

PVE7 and container filesystem support

Updated to support PVE7’s new driver API version, as well as the necessary packaging updates to support Debian 11 (Bullseye).

PVE7 adds the ability for external storage plugins to support the rootdir content type. The rootdir content type is used to store container filesystem data. Prior to PVE7, container storage was limited to a fixed set of storage types.

Snapshot Rollback

We’ve added support for snapshot rollback! Rollback support requires 5.2.0 or greater, and is currently limited to the most recent snapshot.

Bugfixes

• Resizing an attached disk was inadvertently running afoul of perl’s taint checking – this is now fixed.
• Optimized linked clones by using a single shared base snapshot per disk. Put additional bumpers in place to ensure base (a.k.a. template) disks remain static.
• Snapshot names were, in some cases, limited to characters permitted in iSCSI target IQNs. This restriction has been relaxed by escaping/encoding user-supplied names.
• Prefix backend vss labels with the Proxmox pool ID – this avoids collisions and confusion when two pools are configured to use the same backend account.

PROXMOX STORAGE PRIMITIVES

Proxmox offers multiple interfaces for storage management.

• The GUI offers storage management scoped to the context of virtual machine.
• The pvesm command provides granular storage management for specific node.
• The qm command allows for VM specific volume management.
• The pvesh API tool provides fine-grained storage and VM management, and can operate on any node in your Proxmox cluster. To see the available resources, check out the browsable api viewer

For additional detail and for topics not covered in this guide, head over to the Proxmox VE Documentation Index.

Device Naming Specification

Proxmox does not maintain internal state about storage devices or connectivity. In practice, this means that Proxmox relies on device naming to know which devices are associated with virtual machines and how those device are connected to the virtual storage controller. The general device name format is as follows:

Device Filename Specification:
vm-<vmid>-disk-<unique-id>

<vmid>: <integer> (100 - N)
Specify owner VM

<disk-id>: <integer> (1 - N)
Unique naming of disk files

Show Storage Pools

Proxmox supports multiple pools of storage. This flexibility allows for optimization of storage resources based on requirements. With Blockbridge, you can offer different classes of storage. For example, one pool can be IOPS-limited, while another can impose quality-of-service with strict performance guarantees.

Not all Proxmox storage pools allow for shared access. As such, the interfaces that you use to view storage pools are scoped to a node. When working with shared storage types, such as Blockbridge, each node will return its own view of the storage, consistent with the other nodes’ views.

PVESM

Show available storage types on the local node:

$ pvesm status
Name                      Type  Status      Total      Used  Available       %
backup                     pbs  active   65792536   7402332   55018432  11.25%
local                      dir  active    7933384   6342208    1168472  79.94%
shared-block-gp    blockbridge  active  268435456  83886080  184549376  31.25%
shared-block-iops  blockbridge  active  268435456  33669120  234766336  12.54%
shared-file             cephfs  active   59158528    995328   58163200   1.68%

PVESH

Show available storage types on proxmox-1

$ pvesh get /nodes/proxmox-1/storage/
┌──────────────────────┬───────────────────┬─────────────┬────────┬────────────┬─────────┬────────┬────────────┬────────────┬─────────┐
│ content              │ storage           │ type        │ active │      avail │ enabled │ shared │      total │       used │  used % │
╞══════════════════════╪═══════════════════╪═════════════╪════════╪════════════╪═════════╪════════╪════════════╪════════════╪═════════╡
│ backup               │ backup            │ pbs         │ 1      │  52.47 GiB │ 1       │ 0      │  62.74 GiB │   7.06 GiB │  11.25% │
├──────────────────────┼───────────────────┼─────────────┼────────┼────────────┼─────────┼────────┼────────────┼────────────┼─────────┤
│ images               │ shared-block-gp   │ blockbridge │ 1      │ 240.00 GiB │ 1       │ 1      │ 256.00 GiB │  16.00 GiB │   6.25% │
├──────────────────────┼───────────────────┼─────────────┼────────┼────────────┼─────────┼────────┼────────────┼────────────┼─────────┤
│ images               │ shared-block-iops │ blockbridge │ 1      │ 191.89 GiB │ 1       │ 1      │ 256.00 GiB │  64.11 GiB │  25.04% │
├──────────────────────┼───────────────────┼─────────────┼────────┼────────────┼─────────┼────────┼────────────┼────────────┼─────────┤
│ iso,images,vztmpl,.. │ local             │ dir         │ 1      │   1.11 GiB │ 1       │ 0      │   7.57 GiB │   6.05 GiB │  79.99% │
├─────────────────────-┼───────────────────┼─────────────┼────────┼────────────┼─────────┼────────┼────────────┼────────────┼─────────┤
│ vztmpl,backup,iso    │ shared-file       │ cephfs      │ 1      │  55.47 GiB │ 1       │ 1      │  56.42 GiB │ 972.00 MiB │   1.68% │
└──────────────────────┴───────────────────┴─────────────┴────────┴────────────┴─────────┴────────┴────────────┴────────────┴─────────┘

List Volumes

You can enumerate volumes stored in a storage pool using the GUI, pvesm, and pvesh tools.

GUI

To generate a list of all volumes in a storage pool, we recommend Folder View. To see devices connected to a specific virtual machine, select the VM from the primary navigation plane. Then select Hardware.

To see a list of all devices in the storage pool, select a storage pool from the Storage folder in the primary navigation plane (all nodes have a consistent view of storage.) Then select VM Disks.

PVESM

pvesm list <storage> [--vmid <integer>]

Example

List all volumes from the shared-block-iops pool.

$ pvesm list shared-block-iops
Volid                              Format  Type             Size VMID
shared-block-iops:vm-101-disk-0    raw     images    34359738368 101
shared-block-iops:vm-101-disk-1    raw     images    42949672960 101
shared-block-iops:vm-101-disk-2    raw     images    34359738368 101
shared-block-iops:vm-101-state-foo raw     images     4819255296 101
shared-block-iops:vm-10444-disk-1  raw     images    34359738368 10444
shared-block-iops:vm-2000-disk-0   raw     images      117440512 2000

List volumes of VM 101 stored in the shared-block-iops pool.

$ pvesm list shared-block-iops --vmid 101
Volid                              Format  Type             Size VMID
shared-block-iops:vm-101-disk-0    raw     images    34359738368 101
shared-block-iops:vm-101-disk-1    raw     images    42949672960 101
shared-block-iops:vm-101-disk-2    raw     images    34359738368 101
shared-block-iops:vm-101-state-foo raw     images     4819255296 101

PVESH

pvesh get <api_path> [-vmid <integer>]

Show volumes from the shared-block-iops pool:

$ pvesh get /nodes/proxmox-1/storage/shared-block-iops/content --vmid 101
┌────────┬────────────┬────────────────────────────────────┬────────────┬───────────┬───────┬────────┬──────┬──────────────┬───────┐
│ format │       size │ volid                              │      ctime │ encrypted │ notes │ parent │ used │ verification │  vmid │
╞════════╪════════════╪════════════════════════════════════╪════════════╪═══════════╪═══════╪════════╪══════╪══════════════╪═══════╡
│ raw    │  32.00 GiB │ shared-block-iops:vm-101-disk-0    │ 1612628760 │           │       │        │      │              │   101 │
├────────┼────────────┼────────────────────────────────────┼────────────┼───────────┼───────┼────────┼──────┼──────────────┼───────┤
│ raw    │  40.00 GiB │ shared-block-iops:vm-101-disk-1    │ 1612627879 │           │       │        │      │              │   101 │
├────────┼────────────┼────────────────────────────────────┼────────────┼───────────┼───────┼────────┼──────┼──────────────┼───────┤
│ raw    │  32.00 GiB │ shared-block-iops:vm-101-disk-2    │ 1612564950 │           │       │        │      │              │   101 │
├────────┼────────────┼────────────────────────────────────┼────────────┼───────────┼───────┼────────┼──────┼──────────────┼───────┤
│ raw    │   4.49 GiB │ shared-block-iops:vm-101-state-foo │ 1612725210 │           │       │        │      │              │   101 │
├────────┼────────────┼────────────────────────────────────┼────────────┼───────────┼───────┼────────┼──────┼──────────────┼───────┤
│ raw    │  32.00 GiB │ shared-block-iops:vm-10444-disk-1  │ 1612566379 │           │       │        │      │              │ 10444 │
├────────┼────────────┼────────────────────────────────────┼────────────┼───────────┼───────┼────────┼──────┼──────────────┼───────┤
│ raw    │ 112.00 MiB │ shared-block-iops:vm-2000-disk-0   │ 1612478241 │           │       │        │      │              │  2000 │
└────────┴────────────┴────────────────────────────────────┴────────────┴───────────┴───────┴────────┴──────┴──────────────┴───────┘

List volumes of VM 101 that are stored in the shared-block-iops pool:

$ pvesh get /nodes/proxmox-1/storage/shared-block-iops/content
┌────────┬───────────┬────────────────────────────────────┬────────────┬───────────┬───────┬────────┬──────┬──────────────┬──────┐
│ format │      size │ volid                              │      ctime │ encrypted │ notes │ parent │ used │ verification │ vmid │
╞════════╪═══════════╪════════════════════════════════════╪════════════╪═══════════╪═══════╪════════╪══════╪══════════════╪══════╡
│ raw    │ 32.00 GiB │ shared-block-iops:vm-101-disk-0    │ 1612628760 │           │       │        │      │              │  101 │
├────────┼───────────┼────────────────────────────────────┼────────────┼───────────┼───────┼────────┼──────┼──────────────┼──────┤
│ raw    │ 40.00 GiB │ shared-block-iops:vm-101-disk-1    │ 1612627879 │           │       │        │      │              │  101 │
├────────┼───────────┼────────────────────────────────────┼────────────┼───────────┼───────┼────────┼──────┼──────────────┼──────┤
│ raw    │ 32.00 GiB │ shared-block-iops:vm-101-disk-2    │ 1612564950 │           │       │        │      │              │  101 │
├────────┼───────────┼────────────────────────────────────┼────────────┼───────────┼───────┼────────┼──────┼──────────────┼──────┤
│ raw    │  4.49 GiB │ shared-block-iops:vm-101-state-foo │ 1612725210 │           │       │        │      │              │  101 │
└────────┴───────────┴────────────────────────────────────┴────────────┴───────────┴───────┴────────┴──────┴──────────────┴──────┘

Allocate A Volume

Proxmox volumes are provisioned in the context of a VM. In fact, the naming scheme for volumes includes the VMID. When using the GUI, volume allocation automatically attaches the volume to the VM. When pvesm or pvesh are used, you are required to attach volumes as a separate step (see: Attach A Volume). This section covers explicit allocation of volumes as a distinct action.

PVESM

pvesm alloc <storage> <vmid> <filename> <size>

Arguments

Example

Allocate a 10G volume for VMID 100 from the general purpose performance pool.

$ pvesm alloc shared-block-gp 100 vm-100-disk-1 10G
successfully created 'shared-block-gp:vm-100-disk-1'

PVESH

pvesh create <api_path> -vmid <vmid> -filename <filename> -size <size>

Arguments

Volume management with pvesh is node-relative. However, Blockbridge’s shared storage permits uniform access to storage from all Proxmox nodes. You are free to execute allocation requests against any cluster member. The volume will be available globally.

Example

Allocate a 10G volume for VMID 100 from the general purpose performance pool.

$ pvesh create /nodes/proxmox-1/storage/shared-block-gp/content -vmid 100 -filename vm-100-disk-1 -size 10G
shared-block-gp:vm-100-disk-1

Delete A Volume

You can use either pvesm or pvesh commands to delete a volume. It may appear as though the tools use inconsistent terminology. However, keep in mind that pvesh is submitting a DELETE HTTP request to the resource URL.

PVESM

pvesm free <volume> --storage <storage>

Example

Destroy a volume allocated from the general purpose performance pool.

$ pvesm free vm-100-disk-10 --storage shared-block-gp
Removed volume 'shared-block-gp:vm-100-disk-10'

PVESH

pvesh delete <api_path>

Example

Destroy a volume allocated from the general purpose performance pool.

$ pvesh delete /nodes/proxmox-1/storage/shared-block-gp/content/vm-100-disk-1
Removed volume 'shared-block-gp:vm-100-disk-1'

Attach A Volume

An attachment is effectively a VM configuration reference to a storage device. An attachment describes how a storage device is connected to a VM and how the guest OS sees it. The attach operation is principally a VM operation.

GUI

The GUI allows you to attach devices from the Hardware list that are identified as Unused. Select an Unused disk from the Hardware table and click the Edit button. Assign a Bus and Device number. Then Add the device to the VM.

QM

qm set <vmid> --scsihw <scsi-adapter> --scsi<N> <storage>:<volume>

Example

Attach device vm-100-disk-1 to VM 100.

$ qm set 100 --scsihw virtio-scsi-pci --scsi1 shared-block-gp:vm-100-disk-1
update VM 100: -scsi1 shared-block-gp:vm-100-disk-1 -scsihw virtio-scsi-pci

PVESH

pvesh create <api_path> -scsihw <scsi-adapter> -scsi<n> <storage>:<volume>

Example

Attach device vm-100-disk-1 to VM 100.

$ pvesh create /nodes/proxmox-1/qemu/100/config -scsihw virtio-scsi-pci -scsi1 shared-block-gp:vm-100-disk-1
update VM 100: -scsi1 shared-block-gp:vm-100-disk-1 -scsihw virtio-scsi-pci

Detach A Volume

The detach operation updates the configuration of a VM to remove references to a storage device. If the VM is running, the device will disappear from the guest. Detach is a non-destructive operation. It does not overwrite or release storage.

GUI

The GUI allows you to detach devices in Hardware list. Select a disk from the Hardware table and click the Detach button.

QM

qm unlink <vmid> --idlist scsi<N>

Example

Unlink the scsi1 device from VM 100.

$ qm unlink 100 --idlist scsi1
update VM 100: -delete scsi1

PVESH

pvesh set <api_path> -idlist scsi<N>

Example

Unlink the scsi1 device from VM 100.

$ pvesh set /nodes/proxmox-1/qemu/100/unlink -idlist scsi1
update VM 100: -delete scsi1

Resize A Volume

The resize operation extends the logical address space of a storage device. Reducing the size of a device is not permitted by Proxmox. The resize operation can only execute against devices that are attached to a VM.

GUI

The GUI allows you to resize devices available from Hardware list. Select a disk from the Hardware table and click the Resize button.

QM

qm resize <vmid> scsi<N> <size>

Example

Extend the device attached to scsi1 of VM 100 by 1GiB.

$ qm resize 100 scsi1 +1G

PVESH

pvesh set <api_path> -disk scsi<N> -size <size>

Example

Extend the device attached to scsi1 of VM 100 by 1GiB.

$ pvesh set /nodes/proxmox-1/qemu/100/resize -disk scsi1 -size +1G

Create A Snapshot

Snapshots provide a recovery point for a virtual machine’s state, configuration, and data. Proxmox orchestrates snapshots via QEMU and backend storage providers. When you snapshot a Proxmox VM that uses virtual disks backed by Blockbridge, your disk snapshots are thin, they complete instantly, and they avoid copy-on-write (COW) performance penalties.

GUI

In the Snapshots panel for the VM, click Take Snapshot. The duration of the operation depends on whether VM state is preserved.

QM

qm snapshot <vmid> <snapname> --description <desc> --vmstate <save>

Example

Take a snapshot of VM 100, including RAM.

qm snapshot 100 snap_1 --description "hello world" --vmstate 1

PVESH

pvesh create <api_path> -snapname -description <desc> -vmstate <save>

Example

Take a snapshot of VM 100, including RAM.

pvesh create /nodes/proxmox-1/qemu/100/snapshot -snapname snap_1 -description "hello world" -vmstate 1

Remove A Snapshot

Delete a VM snapshot and release associated storage resources.

GUI

In the Snapshots panel for the VM, select the snapshot to remove, and then click Remove. A dialog will appear to confirm your intent.

QM

qm delsnapshot <vmid> <snapname> --force <force>

Example

Gracefully delete the snapshot snap1 of VM 100.

qm delsnapshot 100 snap1

PVESH

pvesh delete <api_path> -force <force>

Example

Gracefully Delete the snapshot snap1 of VM 100.

pvesh delete /nodes/proxmox-1/qemu/100/snapshot/snap1