Understanding LVM Shared Storage In Proxmox

 ┌──────────────────────────────────────────────────────────────────────────────────────┐         ┌─────────────┐
 │ PVE HOST [A]                                                                         │         │ ┌───┐       │
 │             ┌──────────────────LVM───────────────────┬──MULTIPATH SHARED STORAGE───┐ │         │ │ P │       │
 │ ┌─────────┐ ▼ ┌──────────┐ ┌──────────┐ ┌──────────┐ ▼ ┌──────────┐ ┌────────────┐ ▼ │         │ │ O │       │
 │ │ VIRTUAL │   │ LOGICAL  │ │ VOLUME   │ │ PHYSICAL │   │ MULTIPATH├─► SCSI [SDA] ├───┼─────────┼─► R ├───┐   │
 │ │ MACHINE ┼───► VOLUME   ┼─► GROUP    ┼─► VOLUME   ┼───► DEVICE   │ ├────────────┤   │         │ │ T │ ┌─▼─┐ │
 │ │ [A]     │   │ [LV0]    │ │ [VG0]    │ │ [PV0]    │   │ [MPATH0] ├─► SCSI [SDB] ├───┼───┐ ┌───┼─► A │ │ S │ │
 │ └─────────┘   └──────────┘ └──────────┘ └──────────┘   └──────────┘ └────────────┘   │   │ │   │ │ L │ │ H │ │
 └──────────────────────────────────────────────────────────────────────────────────────┘   │ │   │ │   │ │ A │ │
                                                                                            │ │   │ │ 0 │ │ R │ │
                                                                                            │ │   │ └───┘ │ E │ │
                                                                                          ┌─┼─┘   │ ┌───┐ │ D │ │
                                                                                          │ │     │ │ P │ │   │ │
 ┌──────────────────────────────────────────────────────────────────────────────────────┐ │ │     │ │ O │ │ L │ │
 │ PVE HOST [B]                                                                         │ │ │     │ │ R │ │ U │ │
 │             ┌──────────────────LVM───────────────────┬──MULTIPATH SHARED STORAGE───┐ │ │ │     │ │ T │ │ N │ │
 │ ┌─────────┐ ▼ ┌──────────┐ ┌──────────┐ ┌──────────┐ ▼ ┌──────────┐ ┌────────────┐ ▼ │ │ │     │ │ A │ └─▲─┘ │
 │ │ VIRTUAL │   │ LOGICAL  │ │ VOLUME   │ │ PHYSICAL │   │ MULTIPATH├─► SCSI [SDA] ├───┼─┘ └─────┼─► L ├───┘   │
 │ │ MACHINE ┼───► VOLUME   ┼─► GROUP    ┼─► VOLUME   ┼───► DEVICE   │ ├────────────┤   │         │ │   │       │
 │ │ [B]     │   │ [LV1]    │ │ [VG0]    │ │ [PV0]    │   │ [MPATH0] ├─► SCSI [SDB] ├───┼─────────┼─► 1 │       │
 │ └─────────┘   └──────────┘ └──────────┘ └──────────┘   └──────────┘ └────────────┘   │         │ └───┘       │
 └──────────────────────────────────────────────────────────────────────────────────────┘         └─────────────┘

If you are migrating to a Proxmox Virtual Environment (PVE) from an existing VMware or Hyper-V environment, chances are you have an existing SAN. If you want to leverage your existing SAN for shared storage, the best practices are to use PVE’s native support for the Linux Logical Volume Manager (LVM).

LVM provides flexibility in managing disk space, enabling the creation of shared storage pools for virtual machines (VMs) and containers. This article explores how LVM Shared Storage works with PVE, its benefits, limitations, and the concepts required to set it up correctly for high availability.

What is LVM?

LVM (Logical Volume Manager) is a Linux device mapper framework that provides a flexible way to manage disk storage. Unlike traditional partitioning schemes that allocate fixed partitions, LVM provides dynamic logical volume management.

LVM introduces the following key concepts:

Physical Volume (PV): A disk or partition used as part of a storage pool.
Volume Group (VG): A collection of physical volumes that combine to form a storage pool.
Logical Volume (LV): An allocation within the volume group, which can be used as a block device for storing data (like a virtual disk for VMs).

In the context of PVE, LVM is often used to manage the storage of virtual disks, providing flexibility and efficiency.

What makes LVM shared storage possible?

LVM provisions logical volumes (LV) from a volume group (VG). When using Thick Volumes, LVM tracks the physical extents (continuous regions of physical storage) that make up the logical volume’s address space using lightweight persistent metadata. The logical volume’s address space is “directly mapped,” meaning that logical addresses are translated to physical addresses in a linear fashion, typically using an in-memory data structure such as an array or linked list.

LVM is not natively a clustered volume manager. It wasn’t designed to support coordinated, multi-writer access to its metadata without additional synchronization mechanisms. In a clustered environment, PVE enables shared access to LVM storage by ensuring that only one node is responsible for updating the LVM metadata at any given time. The other nodes use coordinated polling to detect metadata changes and refresh their local view of the storage. This creates a single-writer, multi-reader model, where only one node can write the LVM metadata, while the others can read it.

This synchronization model works well for thick LVM volumes but is not compatible with thin LVM, which allocates storage dynamically during write operations. Thin LVM requires more advanced multi-writer, multi-reader metadata synchronization, similar to what you might find in systems like VMFS (VMware File System), to support coordinated access across multiple nodes.

Why Use LVM Shared Storage?

LVM enables centralized storage to be shared across multiple PVE nodes in a cluster, enabling high availability functions, virtual machine migration, and simplified management. LVM is a simple storage management solution that can leverage your existing storage solution.

Key Benefits of LVM Shared Storage in PVE

High Availability: PVE can support live migration of virtual machines by sharing storage between multiple nodes, ensuring minimal downtime. If one node fails, VMs on that node can be restarted on another node with access to the shared storage.
Storage Efficiency: LVM enables the creation of logical volumes that can be resized or extended. PVE can dynamically allocate storage for VMs as needed, improving utilization.
Capacity Expansion: LVM allows you to grow a storage pool by adding additional physical volumes (i.e., devices) to a volume group, without disrupting running VMs or the PVE cluster.

Key Limitations of LVM Shared Storage in PVE

Thin Provisioning: LVM’s thin provisioning features are not compatible with cluster-shared storage. LVM is not a cluster-aware volume manager and does not support multi-writer metadata access.
Snapshots: LVM’s snapshot features are not compatible with cluster-shared storage. LVM is not a cluster-aware volume manager and does not support multi-writer metadata access.
Performance: LVM shared storage configurations built on iSCSI and FC leverage a shared SCSI task set corresponding to a controller-limited queue depth. There are currently no client-side mechanisms in PVE to ensure fairness or prevent starvation like those in ESX.
Security: When a single LUN is shared and accessed by all hosts for all VMs in a deployment, operator errors or software defects on any one host could jeopardize the entire installation.

Tip: The impact of LVM limitations may be surprising to those migrating from ESX. It’s important to note that LVM is a volume manager, not a cluster filesystem like VMFS.

How to Set Up LVM Shared Storage in PVE

Setting up LVM Shared Storage in PVE requires configuring a shared storage device that multiple PVE nodes can access concurrently. Typically, the device is an enterprise multipath-capable SAN supporting iSCSI and/or Fibre Channel.

Here’s an overview of the procedure for configuring LVM shared storage in PVE, with more detailed steps provided in the following sections:

Prepare the Shared Storage Device
Understand Multipath Reporting for Your Device
Connect All Hosts to the Shared Storage
Enable Device Mapper Multipath
Initialize the Multipath Device as an LVM Physical Volume (PV)
Create an LVM Volume Group (VG) Using the Physical Volume (PV)
Register the LVM Volume Group with PVE

Tip: It is important to follow these steps in the specified order, as each step builds upon the previous one.

Prepare the Shared Storage Device

To start, you need a shared storage device that all nodes in your PVE cluster can access. Your storage device should have multiple independent network paths.

There is a wide range of makes, models, manufacturers, and configurations available. While the recommended best practice is to use a solution with high availability and disk-level data protection, it is not mandatory. However, ensure you fully understand the risks before opting for less reliable alternatives.

Understand Multipath Reporting for Your Device

To properly configure shared storage with PVE, it’s absolutely crucial to understand how your storage system reports its available paths. Storage vendors utilize various architectures to implement high availability (i.e., dual controller, virtual controller, l3-failover, etc.), and the underlying implementation directly affects how paths are presented and discovered.

To identify how your storage device reports its available paths, use the iscsiadm command. You should execute iSCSI discovery on all IP addresses configured on the device. The example below demonstrates discovery executed against two portals, each returning identical portal results.

iscsiadm -m discovery -t sendtargets -p 172.16.0.2
172.16.0.2:3260,1 iqn.2009-12.com.blockbridge:t-pewlkzsafdnfl-target-1
172.17.0.2:3260,1 iqn.2009-12.com.blockbridge:t-pewlkzsafdnfl-target-1

iscsiadm -m discovery -t sendtargets -p 172.17.0.2
172.16.0.2:3260,1 iqn.2009-12.com.blockbridge:t-pewlkzsafdnfl-target-1
172.17.0.2:3260,1 iqn.2009-12.com.blockbridge:t-pewlkzsafdnfl-target-1

Tip: See if your storage system’s user manual has a section on configuring Linux multipath. There may be specific parameters such as ALUA or path selection policies that your vendor recommends.

Multipath Reporting Behaviors

The four most common behaviors are illustrated below. For our examples, let’s assume that the hosts have access only to the 172.16.0.0/16 and 172.17.0.0/16 networks and that the device has two known portals, controllers, or paths.

Behavior A: The device reports multiple paths, all of which are accessible to your hosts.

┌────────────────────────────────────┐
│           SHARED STORAGE           │
└────────────────────────────────────┘
│ ┌──────────────┐  ┌──────────────┐ │
│ │   PORTAL 1   │  │   PORTAL 2   │ │
│ └──────────────┘  └──────────────┘ │
└────────────────────────────────────┘

   ✓  172.16.0.2    ✓  172.16.0.2
   ✓  172.17.0.2    ✓  172.17.0.2

Behavior B: The device reports multiple paths, but some are inaccessible to your hosts.

┌────────────────────────────────────┐
│           SHARED STORAGE           │
└────────────────────────────────────┘
│ ┌──────────────┐  ┌──────────────┐ │
│ │   PORTAL 1   │  │   PORTAL 2   │ │
│ └──────────────┘  └──────────────┘ │
└────────────────────────────────────┘

   ✓  172.16.0.2     ✓  172.16.0.2
   ✓  172.17.0.2     ✓  172.17.0.2
   ⓧ  10.16.0.2     ⓧ  10.16.0.2
   ⓧ  10.17.0.2     ⓧ  10.17.0.2

Behavior C: The device uses discrete controllers, each presenting a mutually exclusive set of paths, all of which are accessible to your hosts.

┌────────────────────────────────────┐
│           SHARED STORAGE           │
└────────────────────────────────────┘
│ ┌──────────────┐  ┌──────────────┐ │
│ │   PORTAL 1   │  │   PORTAL 2   │ │
│ └──────────────┘  └──────────────┘ │
└────────────────────────────────────┘

  ✓  172.16.0.2     ✓  172.17.0.2

Behavior D: The device uses discrete controllers, each presenting a mutually exclusive set of paths, some of which are inaccessible.

┌────────────────────────────────────┐
│           SHARED STORAGE           │
└────────────────────────────────────┘
│ ┌──────────────┐  ┌──────────────┐ │
│ │   PORTAL 1   │  │   PORTAL 2   │ │
│ └──────────────┘  └──────────────┘ │
└────────────────────────────────────┘

   ✓  172.16.0.2     ✓  172.17.0.2
   ⓧ  10.16.0.2     ⓧ  10.17.0.2

Connect All Hosts to the Shared Storage

The correct way to configure PVE access to multipath storage depends on the path behavior identified in the previous section.

Here are the appropriate configuration strategies for each behavior:

Behavior A: Add your iSCSI device to /etc/pve/storage.cfg. PVE will automatically connect to the storage from all nodes in the cluster.

  pvesm add iscsi <storage_id> --portal <portal_ip> --target <iqn> --content none

Behavior B: Manually configure multiple iSCSI paths on all of your PVE hosts. PVE will not automatically connect the iSCSI paths, so you must ensure they are established after each reboot.

iscsiadm -m node -T iqn.2025-01.com.example:storage -p <portal_ip>:<portal_port> --login
iscsiadm -m node -T iqn.2025-01.com.example:storage -p <portal_ip>:<portal_port> --op update -n node.startup -v automatic

Behavior C: Add multiple iSCSI devices to /etc/pve/storage.cfg, one for each controller path. PVE will automatically connect to each controller from all nodes in the cluster.

pvesm add iscsi <storage_id> --portal <portal_ip> --target <iqn> --content none
pvesm add iscsi <storage_id> --portal <portal_ip> --target <iqn> --content none

Behavior D: Manually configure multiple iSCSI paths on all of your PVE hosts. PVE will not automatically connect the iSCSI paths, so you must ensure the paths are established after each reboot.

iscsiadm -m node -T iqn.2025-01.com.example:storage -p <portalA_ip>:<portal_port> --login
iscsiadm -m node -T iqn.2025-01.com.example:storage -p <portalA_ip>:<portal_port> --op update -n node.startup -v automatic
iscsiadm -m node -T iqn.2025-01.com.example:storage -p <portalB_ip>:<portal_port> --login
iscsiadm -m node -T iqn.2025-01.com.example:storage -p <portalB_ip>:<portal_port> --op update -n node.startup -v automatic

In summary, PVE expects that storage devices correctly report all accessible paths during the iSCSI discovery process. If your storage does not meet this expectation, you must manually configure the storage on all hosts using iscsiadm.

Enable Device Mapper Multipath

 ┌────────────────────────────────────┐
 │            PROXMOX HOST            │
 ├────────────────────────────────────┤
 │          ┌──────────────┐          │
 │       ┌──┼  /dev/dm-0   ┼──┐       │
 │       │  └──────────────┘  │       │
 │ ┌─────▼────────┐  ┌────────▼─────┐ │
 │ │   /dev/sda   │  │   /dev/sdb   │ │
 │ └─────┬────────┘  └────────┬─────┘ │
 │       │                    │       │
 │ ┌─────▼────────┐  ┌────────▼─────┐ │
 │ │    iscsi 0   │  │   iscsi 1    │ │
 │ └─────┬────────┘  └────────┬─────┘ │
 └───────┼────────────────────┼───────┘
         │                    │
         │                    │
         │                    │
 ┌───────▼────────────────────▼───────┐
 │           SHARED STORAGE           │
 └────────────────────────────────────┘
 │ ┌──────────────┐  ┌──────────────┐ │
 │ │   PORTAL 1   │  │   PORTAL 2   │ │
 │ └──────────────┘  └──────────────┘ │
 └────────────────────────────────────┘

In Linux, each SCSI device path is treated as a separate, independent SCSI block device. This means that if you have a device with multiple physical paths (for example, in a multipath configuration for high availability or redundancy), each path is represented individually, and the operating system sees them as separate devices.

To present a single, unified SCSI device that spans multiple paths, Linux uses the device-mapper framework. Device-mapper allows you to combine these multiple block devices into a single logical block device, creating a virtual device that abstracts the underlying physical paths. This logical device can then be used like a regular block device, even though it is backed by multiple physical paths, enabling features like load balancing and fault tolerance.

For specific instructions on device-mapper multipath configuration, please refer to the Official Proxmox Multipath documentation.

Tip: A common mistake is initializing LVM on one of the device paths instead of the dm device.

Warning: Linux performs an automatic LVM scan. It will fail if duplicate devices are detected and Multipath configuration is not in place.

Initialize the Multipath Device as an LVM Physical Volume (PV)

 ┌────────────────────────────────────┐
 │            PROXMOX HOST            │
 ├────────────────────────────────────┤
 │          ┌──────────────┐          │
 │          │   LVM PV     │          │
 │          └──────┬───────┘          │
 │          ┌──────▼───────┐          │
 │       ┌──┼  /dev/dm-0   ┼──┐       │
 │       │  └──────────────┘  │       │
 │ ┌─────▼────────┐  ┌────────▼─────┐ │
 │ │   /dev/sda   │  │   /dev/sdb   │ │
 │ └─────┬────────┘  └────────┬─────┘ │
 │       │                    │       │
 │ ┌─────▼────────┐  ┌────────▼─────┐ │
 │ │    iscsi 0   │  │   iscsi 1    │ │
 │ └─────┬────────┘  └────────┬─────┘ │
 └───────┼────────────────────┼───────┘
         │                    │
         │                    │
         │                    │
 ┌───────▼────────────────────▼───────┐
 │           SHARED STORAGE           │
 └────────────────────────────────────┘
 │ ┌──────────────┐  ┌──────────────┐ │
 │ │   PORTAL 1   │  │   PORTAL 2   │ │
 │ └──────────────┘  └──────────────┘ │
 └────────────────────────────────────┘

Creating a Physical Volume (PV) is the first step in setting up an LVM storage system. A PV is simply a storage device (or a portion of a device) that is prepared for use by LVM.

Creating a PV involves writing a special metadata header to the device that LVM can later use to track it. The base LVM metadata includes information such as:

The size of the device
Configuration details are required for LVM to manage the device.
Which Volume Group (VG) the device belongs to

pvcreate /dev/mpath0

Create an LVM Volume Group (VG) Using the Physical Volume (PV)

 ┌────────────────────────────────────┐
 │            PROXMOX HOST            │
 ├────────────────────────────────────┤
 │       ┌───────────────────┐        │
 │       │      LVM VG       │        │
 │       │  ┌──────────────┐ │        │
 │       │  │   LVM PV     │ │        │
 │       │  └──────┬───────┘ │        │
 │       └─────────┼─────────┘        │
 │          ┌──────▼───────┐          │
 │       ┌──┼  /dev/dm-0   ┼──┐       │
 │       │  └──────────────┘  │       │
 │ ┌─────▼────────┐  ┌────────▼─────┐ │
 │ │   /dev/sda   │  │   /dev/sdb   │ │
 │ └─────┬────────┘  └────────┬─────┘ │
 │       │                    │       │
 │ ┌─────▼────────┐  ┌────────▼─────┐ │
 │ │    iscsi 0   │  │   iscsi 1    │ │
 │ └─────┬────────┘  └────────┬─────┘ │
 └───────┼────────────────────┼───────┘
         │                    │
         │                    │
         │                    │
 ┌───────▼────────────────────▼───────┐
 │           SHARED STORAGE           │
 └────────────────────────────────────┘
 │ ┌──────────────┐  ┌──────────────┐ │
 │ │   PORTAL 1   │  │   PORTAL 2   │ │
 │ └──────────────┘  └──────────────┘ │
 └────────────────────────────────────┘

Once you’ve created a Physical Volume (PV), it can be used to form a Volume Group (VG). A Volume Group is essentially a logical collection of one or more PVs and serves as a pool of storage resources. By grouping multiple PVs together, a VG aggregates their total capacity, making it available for use in creating and managing Logical Volumes (LVs). This pooling mechanism abstracts the physical layout of storage, allowing you to allocate and manage space more flexibly.

vgcreate <volume_group_name> /dev/mpath0

Register the LVM Volume Group with PVE

PVE integrates Volume Groups (VGs) as a native storage backend, offering a flexible and dynamic method for managing storage resources in virtualized environments. This support allows users to dynamically create, delete, add, remove, and resize logical volumes (block devices) for VMs without manual partitioning.

PVE’s LVM integration allows for the dynamic creation of Logical Volumes (LVs) from a Volume Group (VG), which are then used as virtual disks for virtual machines (VMs). Although LVM itself is not cluster-aware, PVE provides coordination mechanisms to ensure that LVM management operations are carried out from a single node within the cluster. Additionally, PVE incorporates discovery and synchronization logic to ensure that all nodes in the cluster maintain a consistent and up-to-date view of shared LVM devices. In essence, the PVE LVM management model follows a single-writer, multiple-reader approach.

To register an LVM Volume Group with PVE, add it to the /etc/pve/storage.cfg.

pvesm add lvm <storage_id> --vgname <volume_group_name> --content images

You can find syntax for the LVM type in the Official Proxmox Storage Documentation

COMMON PITFALLS

When working with LVM shared storage in PVE, there are several common pitfalls that can lead to confusion, errors, and even data loss. Here are some key considerations to help you avoid these issues.

Reusing a LUN Previously Used with LVM/PVE

If you add a disk that was previously used with PVE or LVM to an existing environment, you may encounter LVM naming collisions. To avoid this, ensure you rename or remove any existing LVM metadata on the disk before reusing it in a new environment.

Lack of Multipath Configuration

Attempting to manage LVM on a system with multiple paths to storage, but without a proper Multipath configuration, can lead to inconsistent behavior. Always operate on the multipath device, not the underlying paths!

Modifying LVM Outside of PVE

Direct LVM operations executed outside the PVE interface can conflict with in-flight PVE operations. PVE ensures single-writer access to the LVM metadata, and any uncoordinated external modifications can potentially cause data corruption. Always use the PVE interfaces to manage LVM, and avoid modifying LVM directly.

CONCLUSION

Using LVM Shared Storage in a Proxmox Virtual Environment can significantly improve flexibility, storage efficiency, and high availability in a clustered environment. However, you must carefully configure multipath devices, understand the limitations of LVM on shared storage (i.e., no thin provisions or snapshots), and ensure your PVE nodes are synchronized to manage storage effectively. This approach is simple and particularly valuable for users with existing SAN infrastructure looking to integrate with PVE.