Why LVM-Thin Cannot Be Shared Storage in Proxmox

Proxmox Virtual Environment (PVE) offers two ways to put virtual disks on LVM. Thick LVM works across a cluster on a shared SAN LUN. Before PVE 9 it had no snapshots. PVE 9 adds them as a technology preview that layers QCOW2 images on thick logical volumes (LVs). LVM-thin has snapshots built into the volume manager, but a thin pool cannot be shared.

The obvious question is why not. The idea seems simple enough: put a thin pool on the shared LUN, and since PVE only ever runs a VM on one node, only that node will have the pool active. Give each VM its own pool and you’ve also limited the damage if something does go wrong.

Note: Stock PVE won’t let you set this up. The lvmthin storage type can’t be marked shared.

This technote explains why that design corrupts data, and why no release of PVE or the kernel is exempt. The kernel behavior described here was checked against Linux 5.4 through 7.2, and the migration sequence against qemu-server 6.0 through 9.2.9. Between them, those cover every kernel PVE has shipped since PVE 6.2 and every migration path since PVE 6.0. For background on thick LVM as shared storage, see Understanding LVM Shared Storage in Proxmox. For the QCOW2 snapshot support in PVE 9, see Inside Proxmox VE 9 SAN Snapshot Support.

Summary

Thick LVM works as shared storage because its block map is text on disk that every node reads and that PVE serializes writes to.
A thin pool’s block map lives in the kernel of whichever node activated it, and that kernel refreshes its view only when it commits a transaction of its own.
One pool per VM limits how much a second activation can destroy. It does nothing to prevent one, and PVE live migration performs a second activation every time it runs.

The Block Map

Thick LVM and LVM-thin differ in where the block map is kept and in who may change it. The block map is the table that translates a volume’s logical blocks to physical locations on the LUN.

A thick LV is a list of physical extents. That list is part of the VG metadata, the volume group’s text metadata. The VG metadata is stored in a reserved area at the front of each physical volume. Every node reads the same text. When a node runs lvcreate or lvextend, it rewrites the text with a new sequence number. The other nodes pick up the change the next time they run an LVM command. PVE wraps each change in a cluster-wide lock.

Activation is the LVM step that makes an LV appear as a block device on a node. Activating a linear thick LV loads its extent list into the Linux device mapper as a fixed map. Two nodes can hold the same fixed map without harm, because both point at the same sectors and neither holds anything the other lacks. Which node’s guest is allowed to write is a separate question, and it is PVE’s job to make sure a VM only ever runs on one node.

A thin pool keeps a persistent B-tree that maps each block of each thin LV to its location on disk. This B-tree is the pool metadata, and it is separate from the VG metadata, which records only that the pool and its thin LVs exist and says nothing about where their blocks are. The kernel owns the pool metadata and updates it as thin LVs are written and snapshotted.

  THICK LOGICAL VOLUME                          THIN POOL
  ┌─────────────────────────────────────┐       ┌────────────────────────────────────────────┐
  │ PVE NODE                            │       │ PVE NODE                                   │
  │ ┌─────────┐   ┌───────────────────┐ │       │ ┌─────────┐   ┌───────────────────┐        │
  │ │ VIRTUAL │   │ FIXED MAP         │ │       │ │ VIRTUAL │   │ THIN LV           │        │
  │ │ MACHINE ┼───► built from the    │ │       │ │ MACHINE ┼───►                   │        │
  │ │ [A]     │   │ VG text metadata  │ │       │ │ [A]     │   └────────┬──────────┘        │
  │ └─────────┘   └────────┬──────────┘ │       │ └─────────┘            │                   │
  │                        │            │       │               ┌────────▼──────────┐        │
  │                        │            │       │               │ THIN POOL         │        │
  │                        │            │       │               │ B-tree roots in   │        │
  │                        │            │       │               │ KERNEL MEMORY on  │        │
  │                        │            │       │               │ this node only    │        │
  │                        │            │       │               └────────┬────────┬─┘        │
  └────────────────────────┼────────────┘       └────────────────────────┼────────┼──────────┘
                           │                                             │        │           
  ┌────────────────────────▼────────────┐       ┌────────────────────────▼────────▼──────────┐
  │ SHARED LUN                          │       │ SHARED LUN                                 │
  │ ┌──────────────┐ ┌────────────────┐ │       │ ┌──────────────┐ ┌──────────┐ ┌──────────┐ │
  │ │ VG METADATA  │ │ LV DATA        │ │       │ │ VG METADATA  │ │ POOL     │ │ POOL     │ │
  │ │ (text, read  │ │                │ │       │ │ (names only, │ │ METADATA │ │ DATA     │ │
  │ │  by all)     │ │                │ │       │ │  no map)     │ │ (B-tree) │ │ (blocks) │ │
  │ └──────────────┘ └────────────────┘ │       │ └──────────────┘ └──────────┘ └──────────┘ │
  └─────────────────────────────────────┘       └────────────────────────────────────────────┘

Activating a Thin Pool

When a node activates a thin pool, the kernel opens the pool metadata and reads its superblock. The superblock is a small header that records where the B-tree starts, where the space map starts, and the current transaction number. The space map is the record of which chunks of the data area are free. Those starting points are called roots, and the kernel keeps them in memory, walking the on-disk B-tree from the roots it captured for as long as the pool stays open.

The kernel refreshes the B-tree root only when it commits a transaction of its own. A commit writes the kernel’s current state to the superblock and then reads the superblock back. The space map is opened once, at activation, and is not read again in normal operation.

When a VM writes to a thin LV and the map changes, the kernel commits from its own in-memory state. It does not check whether another kernel has written a newer superblock since it read its own. The newer superblock is overwritten. dm-thin, the device-mapper target that implements thin pools, was designed for a single owner.

The LVM documentation states this explicitly. Even under lvmlockd, LVM’s own lock manager for shared volume groups, a thin pool can be active on only one node at a time. The lvmlockd(8) manual’s section on shared activation lists the LV types that “cannot be used concurrently from multiple hosts” as thin, cache, raid, mirror, and snapshot.

One Pool Per VM

The case for one pool per VM is containment. If a pool gets corrupted, you lose that VM’s disks and snapshots. Every one of those pools is still a thin pool, though, and every one still has to be open on exactly one kernel at a time.

PVE guarantees that a VM runs on one node at a time. It does not guarantee that an LV is active on one node at a time. For thick LVs that never mattered. For thin pools it does, and live migration is the routine operation that activates an LV on a second node while the first still has it open.

Live Migration

Live migration with shared storage moves a VM’s memory and device state. The PVE migration code runs this sequence:

The source node starts a migration. The VM keeps running.
The target node activates the VM’s LVs.
The target node starts a QEMU process, opens the disks, and waits for the migration stream.
Memory is copied to the target. The guest is still running, so pages it writes to during the copy have to be sent again. This repeats until the leftover set is small enough to send during a short pause.
The source pauses. The last changed memory pages are sent.
The target resumes the VM.
Lastly, the source stops its QEMU process and then deactivates its LVs.

From step 2 through step 7 the LV is active on both nodes. That is just how live migration works, and a thick LV does not care. A thin LV cannot be activated without its pool, so for the same stretch of time two kernels have the same pool open.

The target read the pool metadata at step 2 and caches that view. The source kept writing until step 5. Three kinds of write operation change the map: a write to a region that has never been written, a write to a chunk shared with a snapshot, and a discard. The source committed every such change to disk as it went, the last time at step 7. The target’s copy of the space map is stale. When the VM resumes and writes, the target allocates from that stale map and commits, and whatever the guest wrote to those regions during the migration is lost or overwritten.

Warning: LVM does not reliably detect and report this. thin_check can pass afterwards. When it does, the surviving metadata is consistent with itself and nothing records what was lost.

Alternatives

If you need snapshots on a shared LUN in PVE, there are two options today. Both keep the block map somewhere that has exactly one writer, with no node caching a copy of it.

Approach	Where the block map lives	Migration handoff	Trade-off
PVE 9 snapshots as volume chains	In a chain of QCOW2 images on thick LVs, written by QEMU	QEMU is the only writer and flushes before the handoff	QCOW2 metadata on the data path; technology preview as of PVE 9.2
Array-side snapshots	In the storage array, through a storage plugin	No map on the node; the array holds it	Requires an array with a PVE plugin

CONCLUSION

LVM-thin cannot be shared storage in PVE. A thin pool belongs to the kernel that opened it. There is no check, in LVM or in the kernel, for a second kernel opening the same pool. The owning kernel reads the pool metadata’s roots at activation and writes its own view back whenever it commits. PVE live migration opens the pool on the target before the source is finished with it, so every live migration of a thin LV puts the pool in two kernels at once. The map changes the source made in between are lost when the target commits. Giving each VM its own pool does not change any of this. It only decides whether one VM is corrupted or all of them are.

Proxmox’s own answer, shipped in PVE 9 as a technology preview, is snapshots as volume chains: QCOW2 images on thick LVs. The LVs stay thick, so two nodes can hold them without harm, and the block map lives in the QCOW2 chain, where QEMU is the only writer. The details are in Inside Proxmox VE 9 SAN Snapshot Support.

If you are new to PVE and must use an existing SAN that has no native PVE plugin, QCOW2 on thick LVs is the way to get snapshots on it.

ADDITIONAL RESOURCES

lvmthin(7), the LVM thin provisioning manual
lvmlockd(8), including the LV types that cannot be used from multiple hosts
Kernel documentation: dm-thin provisioning
drivers/md/dm-thin-metadata.c, the kernel’s thin pool metadata code
PVE::QemuMigrate, the PVE live migration code
Proxmox VE Storage Overview, which lists LVM-thin as non-shared
Blockbridge // Understanding LVM Shared Storage in Proxmox
Blockbridge // Inside Proxmox VE 9 SAN Snapshot Support