The two cluster nodes and qdevice host running on Rocky Linux 10.1
Pacemaker version 3.0.1
Corosync version 3.1.9
DRBD version 9.3.1

There are two stress tests:

Test1 - Configuration with Corosync quorum running on qdevice host and configured DRBD handlers and “fencing resource-and-stonith”.
active node memverge, run on memverge “reboot -f -n” and resources switch to memverge2.

Test2 - Configuration with DRBD diskless quorum and Corosync quorum running together on qdevice host.
active node memverge, run on memverge “reboot -f -n” and resources didn’t switch to memverge2.

The real problem with the Test2 is DRBD marks volumes as “Outdated” before STONITH/fencing succeed a few seconds later, on the remaining healthy node memverge2. Is there a way to recover from “Outdated” on the remaining healthy memverge2 node after successful fencing memverge ?

The even more problem, with enabled DRBD handlers and set “fencing resource-only”, there is still no switch to memverge2. And according to logs below, it looks like the handlers didn’t even call…

Here is below one of two DRBD resource config (with latest Test2 “The even more problem, with enabled DRBD handlers and set “fencing resource-only”, there is still no switch to memverge2.”), secondary resource config has exactly the same in critical sections.

[root@memverge ~]# cat /etc/drbd.d/ha-nfs.res
resource ha-nfs {

options {
       auto-promote yes;
       quorum majority;
       on-no-quorum suspend-io;
       on-no-data-accessible suspend-io;
       on-suspended-primary-outdated force-secondary;
        }

handlers {
      fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh";
      unfence-peer "/usr/lib/drbd/crm-unfence-peer.9.sh";
    }

disk {
        c-plan-ahead 0;
        resync-rate 3G;
        c-max-rate 4G;
        c-min-rate 2G;
#        max-buffers 8000;
        al-extents 65536;
        c-fill-target 16M;
#       no-disk-flushes;
#       no-md-flushes;
#       no-disk-barrier;
#       no-disk-drain;
     }

  volume 1 {
    device      /dev/drbd1;
    disk        /dev/mapper/object_block_nfs_vg-ha_nfs_exports_lv_with_vdo_1x8;
    meta-disk   internal;
  }
  volume 2 {
    device      /dev/drbd2;
    disk        /dev/mapper/object_block_nfs_vg-ha_nfs_internal_lv_without_vdo;
    meta-disk   internal;
  }
  volume 5 {
    device      /dev/drbd5;
    disk        /dev/mapper/object_block_nfs_vg-ha_samba_exports_lv_with_vdo_1x8;
    meta-disk   internal;
  }

  on memverge {
    address   10.72.14.152:7900;
    node-id   27;
  }
  on memverge2 {
    address   10.72.14.154:7900;
    node-id   28;
  }
  on qdevice {
    address    10.72.14.186:7900;
    node-id   29;
    volume 1 {
        disk none;
             }
    volume 2 {
        disk none;
             }
    volume 5  {
        disk none;
             }
             }

#  connection-mesh {
#    hosts memverge memverge2 qdevice;
#                  }

net
    {
        transport tcp;
        protocol  C;
        sndbuf-size 64M;
        rcvbuf-size 64M;
        max-buffers 128K;
        max-epoch-size 16K;
        timeout 15;             # 1.5 seconds (must be < Token), active replication (non-idle), time for waiting an expected response packet from the partner
        ping-timeout 5;         # 0.5 second, no active replication (idle), check if its partner is still alive
        ping-int 3;             # 3 seconds, no active replication (idle), interval between two keep-alive packet to check if its partner is still alive
        connect-int 3;          # 3 seconds, link failed, interval between DRBD keeps on trying to connect
       fencing resource-only;
    }

connection
    {
        host memverge address 192.168.0.6:7900;
        host memverge2 address 192.168.0.8:7900;
        path
        {
            host memverge address 192.168.0.6:7900;
            host memverge2 address 192.168.0.8:7900;
        }
        path
        {
            host memverge address 1.1.1.6:7900;
            host memverge2 address 1.1.1.8:7900;
        }
    }

 connection
     {
   host memverge address 10.72.14.152:7900;
   host qdevice address 10.72.14.186:7900;
   path {
     host memverge address 10.72.14.152:7900;
     host qdevice address 10.72.14.186:7900;
   }
 }

 connection
     {
   host memverge2 address 10.72.14.154:7900;
   host qdevice address 10.72.14.186:7900;
   path {
     host memverge2 address 10.72.14.154:7900;
     host qdevice address 10.72.14.186:7900;
   }
 }

}
[root@memverge ~]#

And here is log file from remaining healthy node memverge2:

[root@memverge2 ~]# cat /var/log/messages
Mar 15 10:22:01 memverge2 corosync[760961]:  [KNET  ] link: host: 27 link: 1 is down
Mar 15 10:22:01 memverge2 corosync[760961]:  [KNET  ] link: host: 27 link: 2 is down
Mar 15 10:22:01 memverge2 corosync[760961]:  [KNET  ] host: host: 27 (passive) best link: 0 (pri: 2)
Mar 15 10:22:01 memverge2 corosync[760961]:  [KNET  ] host: host: 27 (passive) best link: 0 (pri: 2)
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs memverge: PingAck did not arrive in time.
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/1 drbd1: Would lose quorum, but using tiebreaker logic to keep
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/2 drbd2: Would lose quorum, but using tiebreaker logic to keep
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/5 drbd5: Would lose quorum, but using tiebreaker logic to keep
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs memverge: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/1 drbd1: disk( UpToDate -> Consistent )
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/1 drbd1 memverge: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/2 drbd2: disk( UpToDate -> Consistent )
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/2 drbd2 memverge: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/5 drbd5: disk( UpToDate -> Consistent )
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/5 drbd5 memverge: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/1 drbd1: Enabling local AL-updates
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/2 drbd2: Enabling local AL-updates
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/5 drbd5: Enabling local AL-updates
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs memverge: Terminating sender thread
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs memverge: Starting sender thread (peer-node-id 27)
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs: Preparing remote state change 3384123242: 27->all empty
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs memverge: Connection closed
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs memverge: helper command: /sbin/drbdadm disconnected
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs memverge: helper command: /sbin/drbdadm disconnected exit code 0
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs memverge: conn( NetworkFailure -> Unconnected ) [disconnected]
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs memverge: Restarting receiver thread
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs memverge: conn( Unconnected -> Connecting ) [connecting]
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs qdevice: Committing remote state change 3384123242 (primary_nodes=8000000)
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/1 drbd1: disk( Consistent -> Outdated ) [far-away]
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/2 drbd2: disk( Consistent -> Outdated ) [far-away]
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs/5 drbd5: disk( Consistent -> Outdated ) [far-away]
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs: Preparing cluster-wide state change 1847087918: 28->all empty
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs: State change 1847087918: primary_nodes=8000000, weak_nodes=FFFFFFFFD7FFFFFF
Mar 15 10:22:01 memverge2 kernel: drbd ha-nfs: Committing cluster-wide state change 1847087918 (26ms)
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi memverge: PingAck did not arrive in time.
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi/3 drbd3: Would lose quorum, but using tiebreaker logic to keep
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi/4 drbd4: Would lose quorum, but using tiebreaker logic to keep
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi memverge: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi/3 drbd3: disk( UpToDate -> Consistent )
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi/3 drbd3 memverge: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi/4 drbd4: disk( UpToDate -> Consistent )
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi/4 drbd4 memverge: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi/3 drbd3: Enabling local AL-updates
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi/4 drbd4: Enabling local AL-updates
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi memverge: Terminating sender thread
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi memverge: Starting sender thread (peer-node-id 27)
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi memverge: Connection closed
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi memverge: helper command: /sbin/drbdadm disconnected
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi memverge: helper command: /sbin/drbdadm disconnected exit code 0
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi memverge: conn( NetworkFailure -> Unconnected ) [disconnected]
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi memverge: Restarting receiver thread
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi memverge: conn( Unconnected -> Connecting ) [connecting]
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi: Preparing remote state change 1224727545: 27->all empty
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi qdevice: Committing remote state change 1224727545 (primary_nodes=8000000)
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi/3 drbd3: disk( Consistent -> Outdated ) [far-away]
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi/4 drbd4: disk( Consistent -> Outdated ) [far-away]
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi: Preparing cluster-wide state change 4009752090: 28->all empty
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi: State change 4009752090: primary_nodes=8000000, weak_nodes=FFFFFFFFD7FFFFFF
Mar 15 10:22:02 memverge2 kernel: drbd ha-iscsi: Committing cluster-wide state change 4009752090 (35ms)
Mar 15 10:22:04 memverge2 pacemaker-attrd[760992]: notice: Setting master-ha-iscsi[memverge2] in instance_attributes: 10000 -> (unset)
Mar 15 10:22:04 memverge2 pacemaker-attrd[760992]: notice: Setting master-ha-nfs[memverge2] in instance_attributes: 10000 -> (unset)
Mar 15 10:22:18 memverge2 kernel: mlx5_core 0000:d8:00.0 ens7f0np0: Link down
Mar 15 10:22:18 memverge2 kernel: mlx5_core 0000:d8:00.0 mlx5_0: Port: 1 Link DOWN
Mar 15 10:22:18 memverge2 kernel: mlx5_core 0000:d8:00.1 ens7f1np1: Link down
Mar 15 10:22:19 memverge2 kernel: mlx5_core 0000:d8:00.1 mlx5_1: Port: 1 Link DOWN
Mar 15 10:22:20 memverge2 kernel: drbd ha-nfs: Preparing remote state change 1819376490: 29->all empty
Mar 15 10:22:20 memverge2 kernel: drbd ha-nfs qdevice: Committing remote state change 1819376490 (primary_nodes=0)
Mar 15 10:22:20 memverge2 kernel: drbd ha-iscsi: Preparing remote state change 1736200617: 29->all empty
Mar 15 10:22:20 memverge2 kernel: drbd ha-iscsi qdevice: Aborting remote state change 1736200617
Mar 15 10:22:20 memverge2 kernel: drbd ha-iscsi: Preparing remote state change 2304927656: 29->all empty
Mar 15 10:22:20 memverge2 kernel: drbd ha-iscsi qdevice: Committing remote state change 2304927656 (primary_nodes=0)
Mar 15 10:22:21 memverge2 corosync[760961]:  [KNET  ] link: host: 27 link: 0 is down
Mar 15 10:22:21 memverge2 corosync[760961]:  [KNET  ] host: host: 27 (passive) best link: 0 (pri: 2)
Mar 15 10:22:21 memverge2 corosync[760961]:  [KNET  ] host: host: 27 has no active links
Mar 15 10:22:22 memverge2 corosync[760961]:  [TOTEM ] Token has not been received in 5250 ms
Mar 15 10:22:24 memverge2 corosync[760961]:  [TOTEM ] A processor failed, forming new configuration: token timed out (7000ms), waiting 8400ms for consensus.
Mar 15 10:22:26 memverge2 kernel: mlx5_core 0000:d8:00.0 ens7f0np0: Link up
Mar 15 10:22:26 memverge2 NetworkManager[1828]: <info>  [1773562946.3024] device (ens7f0np0): carrier: link connected
Mar 15 10:22:26 memverge2 kernel: mlx5_core 0000:d8:00.0 mlx5_0: Port: 1 Link ACTIVE
Mar 15 10:22:26 memverge2 kernel: mlx5_core 0000:d8:00.1 ens7f1np1: Link up
Mar 15 10:22:26 memverge2 NetworkManager[1828]: <info>  [1773562946.5014] device (ens7f1np1): carrier: link connected
Mar 15 10:22:26 memverge2 kernel: mlx5_core 0000:d8:00.1 mlx5_1: Port: 1 Link ACTIVE
Mar 15 10:22:33 memverge2 corosync[760961]:  [QUORUM] Sync members[1]: 28
Mar 15 10:22:33 memverge2 corosync[760961]:  [QUORUM] Sync left[1]: 27
Mar 15 10:22:33 memverge2 corosync[760961]:  [VOTEQ ] waiting for quorum device Qdevice poll (but maximum for 10000 ms)
Mar 15 10:22:33 memverge2 corosync[760961]:  [TOTEM ] A new membership (1c.4f93) was formed. Members left: 27
Mar 15 10:22:33 memverge2 corosync[760961]:  [TOTEM ] Failed to receive the leave message. failed: 27
Mar 15 10:22:33 memverge2 pacemaker-controld[760994]: notice: Our peer on the DC (memverge) is dead
Mar 15 10:22:33 memverge2 pacemaker-attrd[760992]: notice: Lost attribute writer memverge
Mar 15 10:22:33 memverge2 pacemaker-controld[760994]: notice: State transition S_NOT_DC -> S_ELECTION
Mar 15 10:22:33 memverge2 pacemaker-based[760989]: notice: Node memverge state is now lost
Mar 15 10:22:33 memverge2 pacemaker-based[760989]: notice: Removed 1 inactive node with cluster layer ID 27 from the membership cache
Mar 15 10:22:33 memverge2 pacemaker-attrd[760992]: notice: Node memverge state is now lost
Mar 15 10:22:33 memverge2 pacemaker-attrd[760992]: notice: Removing all memverge attributes for node loss
Mar 15 10:22:33 memverge2 pacemaker-attrd[760992]: notice: Removed 1 inactive node with cluster layer ID 27 from the membership cache
Mar 15 10:22:33 memverge2 pacemaker-fenced[760990]: notice: Node memverge state is now lost
Mar 15 10:22:33 memverge2 pacemaker-fenced[760990]: notice: Removed 1 inactive node with cluster layer ID 27 from the membership cache
Mar 15 10:22:34 memverge2 corosync[760961]:  [QUORUM] Members[1]: 28
Mar 15 10:22:34 memverge2 corosync[760961]:  [MAIN  ] Completed service synchronization, ready to provide service.
Mar 15 10:22:34 memverge2 pacemaker-controld[760994]: notice: Node memverge state is now lost
Mar 15 10:22:34 memverge2 pacemaker-controld[760994]: notice: State transition S_ELECTION -> S_INTEGRATION
Mar 15 10:22:34 memverge2 pacemaker-controld[760994]: notice: Finalizing join-1 for 1 node (sync'ing CIB 0.6307.198 with schema pacemaker-4.0 and feature set 3.20.1 from memverge2)
Mar 15 10:22:34 memverge2 pacemaker-attrd[760992]: notice: Recorded local node as attribute writer (was unset)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: Cluster node memverge will be fenced: peer is no longer part of the cluster
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: memverge is unclean
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: pb_nfs_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: pb_nfs_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ip0_nfs_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ip0_nfs_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: fs_nfs_internal_info_HA_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: fs_nfs_internal_info_HA_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: fs_nfsshare_exports_HA_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: fs_nfsshare_exports_HA_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: nfsserver_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: nfsserver_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: expfs_nfsshare_exports_HA_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: expfs_nfsshare_exports_HA_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: samba_service_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: samba_service_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: fs_sambashare_exports_HA_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: fs_sambashare_exports_HA_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: punb_nfs_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: punb_nfs_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: pb_iscsi_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: pb_iscsi_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ip0_iscsi_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ip0_iscsi_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ip1_iscsi_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ip1_iscsi_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: iscsi_target_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: iscsi_target_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: iscsi_lun_drbd3_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: iscsi_lun_drbd3_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: iscsi_lun_drbd4_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: iscsi_lun_drbd4_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: punb_iscsi_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: punb_iscsi_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-nfs:0_demote_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-nfs:0_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-nfs:0_demote_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-nfs:0_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-nfs:0_demote_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-nfs:0_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-nfs:0_demote_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-nfs:0_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-iscsi:0_demote_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-iscsi:0_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-iscsi:0_demote_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-iscsi:0_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-iscsi:0_demote_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-iscsi:0_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-iscsi:0_demote_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ha-iscsi:0_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ipmi-fence-memverge2_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: ipmi-fence-memverge2_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: watchdog_stop_0 on memverge is unrunnable (node is offline)
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: Scheduling node memverge for fencing
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Fence (reboot) memverge 'peer is no longer part of the cluster'
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       pb_nfs     ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       ip0_nfs    ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       fs_nfs_internal_info_HA     ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       fs_nfsshare_exports_HA      ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       nfsserver                   ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       expfs_nfsshare_exports_HA   ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       samba_service               ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       fs_sambashare_exports_HA    ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       punb_nfs                    ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       pb_iscsi                    ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       ip0_iscsi                   ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       ip1_iscsi                   ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       iscsi_target                ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       iscsi_lun_drbd3             ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       iscsi_lun_drbd4             ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       punb_iscsi                  ( memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       ha-nfs:0                    ( Promoted memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       ha-iscsi:0                  ( Promoted memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Stop       ipmi-fence-memverge2        (          memverge )  due to node availability
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: notice: Actions: Move       watchdog                    ( memverge -> memverge2 )
Mar 15 10:22:34 memverge2 pacemaker-schedulerd[760993]: warning: Calculated transition 1 (with warnings), saving inputs in /var/lib/pacemaker/pengine/pe-warn-3838.bz2
Mar 15 10:22:34 memverge2 pacemaker-controld[760994]: notice: Requesting fencing (reboot) targeting node memverge
Mar 15 10:22:34 memverge2 pacemaker-controld[760994]: notice: Requesting local execution of notify operation for ha-nfs on memverge2
Mar 15 10:22:34 memverge2 pacemaker-fenced[760990]: notice: Client pacemaker-controld.760994 wants to fence (reboot) memverge using any device
Mar 15 10:22:34 memverge2 pacemaker-fenced[760990]: notice: Requesting peer fencing (reboot) targeting memverge
Mar 15 10:22:34 memverge2 pacemaker-controld[760994]: notice: Requesting local execution of notify operation for ha-iscsi on memverge2
Mar 15 10:22:34 memverge2 pacemaker-fenced[760990]: notice: Requesting that memverge2 perform 'reboot' action targeting memverge using ipmi-fence-memverge
Mar 15 10:22:34 memverge2 pacemaker-fenced[760990]: notice: Delaying 'reboot' action targeting memverge using ipmi-fence-memverge for 3s
Mar 15 10:22:34 memverge2 pacemaker-controld[760994]: notice: Requesting local execution of start operation for watchdog on memverge2
Mar 15 10:22:34 memverge2 pacemaker-controld[760994]: notice: Result of notify operation for ha-nfs on memverge2: OK
Mar 15 10:22:34 memverge2 pacemaker-controld[760994]: notice: Result of notify operation for ha-iscsi on memverge2: OK
Mar 15 10:22:34 memverge2 pacemaker-controld[760994]: notice: Result of start operation for watchdog on memverge2: OK
Mar 15 10:22:34 memverge2 pacemaker-controld[760994]: notice: Requesting local execution of monitor operation for watchdog on memverge2
Mar 15 10:22:34 memverge2 pacemaker-controld[760994]: notice: Result of monitor operation for watchdog on memverge2: OK
Mar 15 10:22:43 memverge2 kernel: mlx5_core 0000:d8:00.0 ens7f0np0: Link down
Mar 15 10:22:43 memverge2 kernel: mlx5_core 0000:d8:00.0 mlx5_0: Port: 1 Link DOWN
Mar 15 10:22:43 memverge2 kernel: mlx5_core 0000:d8:00.1 ens7f1np1: Link down
Mar 15 10:22:44 memverge2 kernel: mlx5_core 0000:d8:00.1 mlx5_1: Port: 1 Link DOWN
Mar 15 10:22:55 memverge2 pacemaker-fenced[760990]: notice: Operation 'reboot' [763039] targeting memverge using ipmi-fence-memverge returned 0
Mar 15 10:22:55 memverge2 pacemaker-fenced[760990]: notice: Action 'reboot' targeting memverge using ipmi-fence-memverge on behalf of pacemaker-controld.760994@memverge2: Done
Mar 15 10:22:55 memverge2 pacemaker-fenced[760990]: notice: Operation 'reboot' targeting memverge by memverge2 for pacemaker-controld.760994@memverge2: OK (Done)
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Peer memverge was terminated (reboot) by memverge2 on behalf of pacemaker-controld.760994@memverge2: OK
Mar 15 10:22:55 memverge2 pacemaker-attrd[760992]: notice: Removing all memverge attributes for node memverge2
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Requesting local execution of notify operation for ha-nfs on memverge2
Mar 15 10:22:55 memverge2 pacemaker-attrd[760992]: notice: Removing all memverge attributes for node memverge2
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Requesting local execution of notify operation for ha-iscsi on memverge2
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Result of notify operation for ha-nfs on memverge2: OK
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Requesting local execution of notify operation for ha-nfs on memverge2
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Result of notify operation for ha-iscsi on memverge2: OK
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Requesting local execution of notify operation for ha-iscsi on memverge2
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Result of notify operation for ha-nfs on memverge2: OK
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Requesting local execution of notify operation for ha-nfs on memverge2
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Result of notify operation for ha-iscsi on memverge2: OK
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Requesting local execution of notify operation for ha-iscsi on memverge2
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Result of notify operation for ha-nfs on memverge2: OK
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Result of notify operation for ha-iscsi on memverge2: OK
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: Transition 1 (Complete=65, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-3838.bz2): Complete
Mar 15 10:22:55 memverge2 pacemaker-controld[760994]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE
Mar 15 10:23:01 memverge2 kernel: mlx5_core 0000:d8:00.1 ens7f1np1: Link up
Mar 15 10:23:01 memverge2 NetworkManager[1828]: <info>  [1773562981.0393] device (ens7f1np1): carrier: link connected
Mar 15 10:23:01 memverge2 kernel: mlx5_core 0000:d8:00.1 mlx5_1: Port: 1 Link ACTIVE
Mar 15 10:23:01 memverge2 NetworkManager[1828]: <info>  [1773562981.2379] device (ens7f0np0): carrier: link connected
Mar 15 10:23:01 memverge2 kernel: mlx5_core 0000:d8:00.0 ens7f0np0: Link up
Mar 15 10:23:01 memverge2 kernel: mlx5_core 0000:d8:00.0 mlx5_0: Port: 1 Link ACTIVE
[root@memverge2 ~]#
[root@memverge2 ~]# drbdadm status
ha-iscsi role:Secondary
  volume:3 disk:Outdated open:no
  volume:4 disk:Outdated open:no
  memverge connection:Connecting
  qdevice role:Secondary
    volume:3 peer-disk:Diskless
    volume:4 peer-disk:Diskless

ha-nfs role:Secondary
  volume:1 disk:Outdated open:no
  volume:2 disk:Outdated open:no
  volume:5 disk:Outdated open:no
  memverge connection:Connecting
  qdevice role:Secondary
    volume:1 peer-disk:Diskless
    volume:2 peer-disk:Diskless
    volume:5 peer-disk:Diskless

[root@memverge2 ~]#

latest drdb patch

two nodes are fully synched - one node has some inconsistent vdisks

drbdadm status all

pm-479d5da8 role:Primary
  disk:UpToDate open:yes
  pveAMD-AI role:Secondary
    peer-disk:UpToDate
  pveAMD02 role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:4.00

pm-8b1faeab role:Primary
  disk:UpToDate open:yes
  pveAMD-AI role:Secondary
    peer-disk:UpToDate
  pveAMD02 role:Secondary
    peer-disk:UpToDate

pm-8f4a8d6b role:Primary
  disk:UpToDate open:yes
  pveAMD-AI role:Secondary
    peer-disk:UpToDate
  pveAMD02 role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:1.93


drbdadm adjust all

drbdadm -V
DRBDADM_BUILDTAG=GIT-hash:\ d10b5f53cdf6a445d6fc02cfc2477a129f4e7e83\ build\ by\ @buildsystem\,\ 2026-03-11\ 12:53:05
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090300
DRBD_KERNEL_VERSION=9.3.0
DRBDADM_VERSION_CODE=0x092101
DRBDADM_VERSION=9.33.1

linstor -v
linstor-client 1.27.1; GIT-hash: 9c57f040eb3834500db508e4f04d361d006cb6b5

Simply no progress for two vdisks

Any suggestions?

Thanks and happy we.

I’ve found the hack…. you have to manually down / up the resource on the corresponding node by:

drbdadm down pm-479d5da8

drbdadm up pm-479d5da8

drbdadm status pm-479d5da8

pm-479d5da8 role:Secondary
disk:UpToDate open:no
pveAMD01 role:Primary
peer-disk:UpToDate
pveAMD02 role:Secondary
peer-disk:UpToDate

ERROR:
Description:
    The property 'DrbdOptions/ExactSize' cannot be set to True while there are currently deployed resources(3).
Details:
    Resource definition: pm-3f00aa01

If the VM’s are stopped it works fine, but this is a 20TB disk and it will take a long time to move, so I would prefer to do a live move.

linstor resource-definition list
linstor resource-definition set-property <resource_name> DrbdOptions/ExactSize yes

also in /etc/pve/storage.cfg set exactsize on your drbd volume.
I can do a live move of the storage, i encountered the same issue as you.

I did also test without exactsize in the storage.cfg file while the VM was off, and then started it again after setting the above settings, so if you have a VM with a lots of data you can certainly live move after a reboot of your VM.

I hope this helps!

Thank you for your reply.

Please note that I do not want to migrate from local storage to drbd/linstor, but the other way around. My VM disk in already in Linstor.

exactsize yes

example:

drbd: drbdpool
content images
controller 172.16.20.50
resourcegroup pve-rg
exactsize yes

But only do this with VM you want to migrate from local storage to drbd storage, when done migrating do no forget to remove this settings or as I did add a hashtag.

Now I see you already tried moving it but you are stuck at this moment, so I am guessing your VM is still using it’s local storage duo to the error you’ve seen?

If that answer is yes then you should - in my opinion - delete pm-3f00aa01 and try again when you have enabled the above setting.

Please continue with caution and first create a back-up of your VM!

For example, copying 11400 MB starting out at 1.3 GB/s rapidly deteriorates afterward. The final 100 MB segments complete at just 2.5 MB/s and 1.1 MB/s:

# dd if=/dev/zero of=test bs=1M count=11400 status=progress oflag=direct
10686038016 bytes (11 GB, 10 GiB) copied, 8 s, 1.3 GB/s
# dd if=/dev/zero of=test bs=1M count=100 status=progress oflag=direct
104857600 bytes (105 MB, 100 MiB) copied, 43 s, 2.5 MB/s
# dd if=/dev/zero of=test bs=1M count=100 status=progress oflag=direct
104857600 bytes (105 MB, 100 MiB) copied, 94.2242 s, 1.1 MB/s

It fails with the error:

TASK ERROR: storage migration failed: block job (mirror) error: drive-scsi2: Source and target image have different sizes (io-status: ok)

I found this forum post and this github issue where I learned that I should set the DrbdOptions/ExactSize property to true. So I tried that, but I get:

# linstor resource-definition set-property pm-3f00aa01 DrbdOptions/ExactSize true
ERROR:
Description:
    The property 'DrbdOptions/ExactSize' cannot be set to True while there are currently deployed resources(3).
Details:
    Resource definition: pm-3f00aa01

What is the recommended procedure to set this property?

For anyone struggling:
CREATE WORK DIRECTORIES

mkdir -p linstor-ctrl/ssl pve01 pve02 pbs01

GENERATE CONTROLLER KEYSTORE

keytool -keyalg rsa
-keysize 2048
-genkey -keystore linstor-ctrl/ssl/keystore.jks
-storepass linstor
-keypass linstor
-alias linstor-ctrl
-dname “CN=linstor-ctrl”
-ext SAN=dns:linstor-ctrl,ip:172.16.20.50
-validity 3650

GENERATE LINSTOR-CONTROLLER CA

keytool -exportcert
-alias linstor-ctrl
-keystore linstor-ctrl/ssl/keystore.jks
-storepass linstor
-rfc
-file linstor-ctrl/controller-ca.crt

GENERATE SATELLITE KEYSTORES
PVE01 KEYSTORE

keytool -keyalg rsa
-keysize 2048
-genkey -keystore pve01/keystore.jks
-storepass linstor
-keypass linstor
-alias pve01
-dname “CN=pve01”
-ext SAN=dns:pve01,ip:172.16.20.10
-validity 3650

PVE02 KEYSTORE

keytool -keyalg rsa
-keysize 2048
-genkey -keystore pve02/keystore.jks
-storepass linstor
-keypass linstor
-alias pve02
-dname “CN=pve02”
-ext SAN=dns:pve02,ip:172.16.20.20
-validity 3650

PBS01 KEYSTORE

keytool -keyalg rsa
-keysize 2048
-genkey -keystore pbs01/keystore.jks
-storepass linstor
-keypass linstor
-alias pbs01
-dname “CN=pbs01”
-ext SAN=dns:pbs01,ip:172.16.20.30
-validity 3650

BUILD THE LINSTOR-CONTROLLER TRUSTSTORE

keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor -srckeystore pve01/keystore.jks -destkeystore linstor-ctrl/ssl/truststore.jks

keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor -srckeystore pve02/keystore.jks -destkeystore linstor-ctrl/ssl/truststore.jks

keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor -srckeystore pbs01/keystore.jks -destkeystore linstor-ctrl/ssl/truststore.jks

CREATE SATELLITE TRUSTSTORES
PVE01 TRUSTSTORE

keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor -srckeystore linstor-ctrl/ssl/keystore.jks -destkeystore pve01/certificates.jks

PVE02 TRUSTSTORE

keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor -srckeystore linstor-ctrl/ssl/keystore.jks -destkeystore pve02/certificates.jks

PBS01 TRUSTSTORE

keytool -importkeystore -srcstorepass linstor -deststorepass linstor -keypass linstor -srckeystore linstor-ctrl/ssl/keystore.jks -destkeystore pbs01/certificates.jks

COPY KEYSTORES AND TRUSTSTORES
CREATE THE SSL DIRECTORY FOR THE LINSTOR CONTROLLER

mkdir -p /etc/linstor/ssl
ssh -t root@pve02 “mkdir -p /etc/linstor/ssl”

COPY LINSTOR-CONTROLLER KEYSTORE AND TRUSTSTORE

cp linstor-ctrl/ssl/keystore.jks /etc/linstor/ssl/keystore.jks
cp linstor-ctrl/ssl/truststore.jks /etc/linstor/ssl/certificates.jks

COPY LINSTOR-CONTROLLER KEYSTORE AND TRUSTSTORE TO PVE02

scp linstor-ctrl/ssl/keystore.jks root@pve02:/etc/linstor/ssl/keystore.jks
scp linstor-ctrl/ssl/truststore.jks root@pve02:/etc/linstor/ssl/certificates.jks

COPY PVE01 SATELLITE KEYSTORE AND TRUSTSTORE

cp pve01/keystore.jks /etc/linstor/keystore.jks
cp pve01/certificates.jks /etc/linstor/certificates.jks

COPY PVE02 SATELLITE KEYSTORE AND TRUSTSTORE

scp pve02/keystore.jks root@pve02:/etc/linstor/keystore.jks
scp pve02/certificates.jks root@pve02:/etc/linstor/certificates.jks

COPY PBS01 SATELLITE KEYSTORE AND TRUSTSTORE

scp pbs01/keystore.jks root@pbs01:/etc/linstor/keystore.jks
scp pbs01/certificates.jks root@pbs01:/etc/linstor/certificates.jks

ENABLE HTTPS COMMUNICATION

sed -i ‘0,/[https]/s/[https]/#[https]/’ /etc/linstor/linstor.toml

cat << EOF | sudo tee -a /etc/linstor/linstor.toml

[https]
enabled = true
port = 3371
listen_addr = “0.0.0.0”
keystore = “/etc/linstor/ssl/keystore.jks”
keystore_password = “linstor”

ONLY USE TRUSTSTORE WHEN YOU WANT RESTRICTED CLIENT ACCESS!

truststore = “/etc/linstor/ssl/certificates.jks”

truststore_password = “linstor”

EOF

scp /etc/linstor/linstor.toml root@pve02:/etc/linstor/linstor.toml
systemctl restart linstor-controller.service // LINSTOR-GUI IS NOW AVAILABLE AT HTTPS://<YOUR-VIRTUAL-IP:3371

CONVERT KEYSTORES TO PEM CLIENT CERTIFICATES
EXPORT THE PVE01 SATELLITE KEYSTORE TO A PKCS#12 FILE

keytool -importkeystore -srckeystore pve01/keystore.jks -destkeystore pve01/client.p12 -storepass linstor -keypass linstor -srcalias pve01 -srcstoretype jks -deststoretype pkcs12

CONVERT TO PEM WITH A PASSWORD

openssl pkcs12 -in pve01/client.p12 -out pve01/client_with_pw.pem

REMOVE THE PASSWORD, EXTRACT THE KEY AND APPEND THE CERTIFICATE

openssl rsa -in pve01/client_with_pw.pem -out pve01/client.pem
openssl x509 -in pve01/client_with_pw.pem >> pve01/client.pem

EXPORT THE PVE02 SATELLITE KEYSTORE TO A PKCS#12 FILE

keytool -importkeystore -srckeystore pve02/keystore.jks -destkeystore pve02/client.p12 -storepass linstor -keypass linstor -srcalias pve02 -srcstoretype jks -deststoretype pkcs12

CONVERT TO PEM WITH A PASSWORD

openssl pkcs12 -in pve02/client.p12 -out pve02/client_with_pw.pem

REMOVE THE PASSWORD, EXTRACT THE KEY AND APPEND THE CERTIFICATE

openssl rsa -in pve02/client_with_pw.pem -out pve02/client.pem
openssl x509 -in pve02/client_with_pw.pem >> pve02/client.pem

EXPORT THE PBS01 SATELLITE KEYSTORE TO A PKCS#12 FILE

keytool -importkeystore -srckeystore pbs01/keystore.jks -destkeystore pbs01/client.p12 -storepass linstor -keypass linstor -srcalias pbs01 -srcstoretype jks -deststoretype pkcs12

CONVERT TO PEM WITH A PASSWORD

openssl pkcs12 -in pbs01/client.p12 -out pbs01/client_with_pw.pem

REMOVE THE PASSWORD, EXTRACT THE KEY AND APPEND THE CERTIFICATE

openssl rsa -in pbs01/client_with_pw.pem -out pbs01/client.pem
openssl x509 -in pbs01/client_with_pw.pem >> pbs01/client.pem

DEPLOY SATELLITE CERTIFICATES
PVE01

cp pve01/client.pem /etc/linstor/client.pem

PVE02

scp pve02/client.pem root@pve02:/etc/linstor/client.pem

PBS01

scp pbs01/client.pem root@pbs01:/etc/linstor/client.pem

UPDATE LINSTOR-CLIENT CONFIGURATION
PVE01

rm -rf /etc/linstor/linstor-client.conf

cat << EOF > /etc/linstor/linstor-client.conf
[global]
controllers = linstor+ssl://172.16.20.50:3371
certfile = /etc/linstor/client.pem
EOF

COPY LINSTOR-CLIENT CONFIGURATION TO PVE02

ssh -t root@pve02 “rm -rf /etc/linstor/linstor-client.conf”
scp /etc/linstor/linstor-client.conf root@pve02:/etc/linstor/linstor-client.conf

COPY LINSTOR-CLIENT CONFIGURATION TO PBS01

ssh -t root@pbs01 “rm -rf /etc/linstor/linstor-client.conf”
scp /etc/linstor/linstor-client.conf root@pbs01:/etc/linstor/linstor-client.conf

CONFIGURE SATELLITE NETWORK COMMUNICATION
PVE01

cat << EOF > /etc/linstor/linstor_satellite.toml
[netcom]
type=“ssl”
port=3367
server_certificate=“/etc/linstor/keystore.jks”
trusted_certificates=“/etc/linstor/certificates.jks”
key_password=“linstor”
keystore_password=“linstor”
truststore_password=“linstor”
ssl_protocol=“TLSv1.2”
EOF

COPY TO PVE02

scp /etc/linstor/linstor_satellite.toml root@pve02:/etc/linstor/linstor_satellite.toml

COPY TO PBS01

scp /etc/linstor/linstor_satellite.toml root@pbs01:/etc/linstor/linstor_satellite.toml

RESTART SATELLITE SERVICE ON ALL NODES

systemctl restart linstor-satellite.service

ssh -t root@pve02 “systemctl restart linstor-satellite.service”
ssh -t root@pbs01 “systemctl restart linstor-satellite.service”

SWITCH SATELLITE COMMUNICATION TO SSL
PVE01

linstor node interface modify --communication-type ssl -p 3367 pve01 default

PVE02

linstor node interface modify --communication-type ssl -p 3367 pve02 default

PBS01

linstor node interface modify --communication-type ssl -p 3367 pbs01 default

DISABLE HTTP COMMUNICATION

sed -i ‘0,/[http]/s/[http]/#[http]/’ /etc/linstor/linstor.toml

systemctl restart linstor-controller.service
systemctl restart linstor-satellite.service

ssh -t root@pve02 “systemctl restart linstor-satellite.service”
ssh -t root@pbs01 “systemctl restart linstor-satellite.service”

GENERATE CLIENT CERTIFICATES FOR PROXMOX REST API
PVE01

openssl pkcs12 -in pve01/client.p12 -clcerts -nokeys -out pve01/client.crt
openssl pkcs12 -in pve01/client.p12 -nocerts -nodes -out pve01/client.key

cp pve01/client.crt /etc/linstor/client.crt
cp pve01/client.key /etc/linstor/client.key

PVE02

openssl pkcs12 -in pve02/client.p12 -clcerts -nokeys -out pve02/client.crt
openssl pkcs12 -in pve02/client.p12 -nocerts -nodes -out pve02/client.key

scp pve02/client.crt root@pve02:/etc/linstor/client.crt
scp pve02/client.key root@pve02:/etc/linstor/client.key

COPY LINSTOR-CONTROLLER CA

cp linstor-ctrl/controller-ca.crt /etc/linstor/controller-ca.crt
scp linstor-ctrl/controller-ca.crt root@pve02:/etc/linstor/controller-ca.crt

EDIT DRBD STORAGE CONFIGURATION

nano /etc/pve/storage.cfg

apicrt /etc/linstor/client.crt
apikey /etc/linstor/client.key
apica /etc/linstor/controller-ca.crt

EXAMPLE

drbd: drbdpool
content images
controller 172.16.20.50
resourcegroup pve-rg

# exactsize yes # Use exactsize parameter only temporarly to migrate running VMs

apicrt /etc/linstor/client.crt
apikey /etc/linstor/client.key
apica /etc/linstor/controller-ca.crt

I hope this makes some sense.
My config is a 2-node PVE Cluster setup with 1 PBS node acting a quorom (qdevice) for PVE and a diskless/tie-breaker (quorum) for LINSTOR/DRBD
The linstor-controller (172.16.20.50) is HA and can move between PVE01 and PVE02.

I’m also continously improving so if someone knows a better way, please share

A quick note: This does not encrypt the DRBD replication data, that needs another method.

I had a very similar case occur where this all worked as designed in my test environment, but was not working with another user. After further analysis, we discovered that my environment did not specify transport tcp; but instead omitted this line to let the utils just automatically work this.

It is not currently documented (just filed a documentation bug to fix that), but there is three possible transports you can configure which align with our three different transport kernel modules, tcp, rdma, and lb-tcp. In order for load-balanced TCP to work, you need to specify the lb-tcp transport specifically, or, as I mentioned, just remove that line from the configuration entirely. For example:

        load-balance-paths yes;
        transport "lb-tcp";
    }

It has been a few months, but I hope this helps. At least it may help others who stumble across this post.

There is two-node shared-nothing DRBD-based sync replication active/standby Pacemaker storage metro-cluster.
DRBD replication links are directly (no switch) connected.
Configuration with Corosync quorum and DRBD diskless quorum are running on additional qdevice host.
Heuristics (parallel fping) and STONITH/fencing - fence_ipmilan and diskless sbd (hpwdt, /dev/watchdog).

Data consistency and integrity is absolute priority during any node/network breaks.

In such setup, should I add to DRBD resource files “handlers” and “fencing”,

handlers {
      fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh";
      unfence-peer "/usr/lib/drbd/crm-unfence-peer.9.sh";
    }

and

fencing resource-and-stonith; or fencing resource-only

Or is it unnecessary and may even be counterproductive ?

We are happy and proud to announce the availability of LINBIT SDS For Windows.
LINBIT SDS For Windows is a fully featured Software Defined Storage (SDS) solution
for Windows clusters (or also for mixed Linux/Windows clusters).

It comes with following components:

* WinDRBD: Replication engine (driver) that makes your data highly available
* LINSTOR: Storage orchestrator for large (100+ physical nodes) clusters
* LINSTOR GUI: Easy to use, intuitive web based GUI for LINSTOR
* LINSTOR Client: Command line client for LINSTOR
* DRBD Reactor / Promoter: Cluster manager: starts and stops
resources in response to events (node lost, ...)

For Windows, LINBIT SDS comes with a single installer EXE that allows users
to get started quickly. There is a community edition for evaluation purposes
which requires nodes to be set to test mode but has no other restrictions
in storage size or number of nodes. LINBIT SDS For Windows can be downloaded
from the LINBIT website (www.linbit.com, it is currently filed under Windows
DRBD driver).

I will give an introduction and 3 demos tomorrow at the community meeting.
I would like to invite you to join and ask any questions that may arise:

Best regards,

- Johannes

This time, we have taken the plunge to update LINSTOR to 1.33.1, along
with updating to the latest DRBD 9.2.17. These fix a number of bugs,
so updating is recommended. In the next feature release, we will
update DRBD to 9.3.x.

On the Kubernetes front, we have updated the CSI driver to 1.10.6,
which brings some fixes for IPv6 only clusters, and better capacity
calculation for storage pools with disabled overprovisioning.

We recommend all users of release 2.10.x to update to this latest release.

To upgrade, either point your kustomization.yaml at the new manifest:

  https://charts.linstor.io/static/v2.10.5.yaml

Or, if using helm, upgrade the linstor-operator chart:

  helm repo update
  helm upgrade linstor-operator linstor/linstor-operator --wait

To get specific instructions to apply the update, check our users guide[1].

Source code is, as always, available upstream[2].

Best regards,
Moritz

[1]: LINSTOR 1.0 en - LINBIT
[2]: GitHub - piraeusdatastore/piraeus-operator: The Piraeus Operator manages LINSTOR clusters in Kubernetes. · GitHub
---
### Fixed

- Updated NFS server template to work with NFS Ganesha instances with
monitoring enabled.
- Ensure Satellites are always rescheduled on NoScheduled nodes as
long as the LinstorSatellite resource exists.

### Changed

- Updated images:
    * LINSTOR 1.33.1
    * LINSTOR CSI 1.10.6
    * DRBD 9.2.17
    * DRBD Reactor 1.11.0
    * Latest CSI sidecars

this is the first RC of the upcoming version 9.34.0. It contains
improvements and fixes all over the place. The main reason for this
release is the release of DRBD kernel module 9.3.1 which introduced two
new parameters, namely "discard-granularity" and "tie-breaker".

Please test, if we don't find any issues the final release will be about
a week from now.

Regards, rck

GIT: Prepare 9.34.0-rc.1 · LINBIT/drbd-utils@696ee0f · GitHub
TGZ: https://pkg.linbit.com//downloads/drbd/utils/drbd-utils-9.34.0-rc.1.tar.gz
PPA: LINBIT DRBD9 Stack : “LINBIT” team

Changelog:
9.34.0-rc.1
-----------
* drbdadm: fix adjust for a v8 config with a v9 drbd kernel driver
* drbdadm: fix segfault if we don't know about peer devices
* drbdsetup,v9: suspend-io: add --bdev-freeze option
* drbdadm,adjust: split disk-options into early/late stage
   based on bitmap direction
* drbdadm,adjust: fix segfault on _unknown options
* drbdmeta: create-md: add --initialize-bitmap=set-all
* drbd: add per-peer_device tiebreaker config flag for quorum
* drbd-utils: add discard-granularity device option

Here is the final release of drbd-9.2.17 and drbd-9.3.1.

This development cycle brought a slightly higher number of bug fixes
than the previous cycles. In nature, they are following the trend we
have seen for quite some time. It is the unusual corner cases.

The set of tests in the CI loop, nightly test stability tests, and I/O
endurance tests is a good net that catches major bugs before I tag a
release. Of course, we constantly add new tests to further improve the
quality. Also in this cycle, we introduced a static analyzer to
eliminate a whole class lifetime/locking bugs of the bitmap data
structure. And since it checks every future commit, it also guarantees
that we never ever introduce bugs of that class.

The 9.3.1 brings fundamental modernization to how DRBD handles I/O. From
now on, DRBD can allocate compound pages from the kernel as I/O
buffers. When you make large I/O requests or streaming accesses, that
can speed things up.

Do not be shy about switching from the 9.2.x series to the 9.3.x
series. We use 9.3.1 on our internal production system exclusively by
now. As of today, I plan that the 9.2.x series sees its last release in
September 2026.

9.3.1 (api:genl2/proto:86-101,118-124/transport:21)
--------
* Use compound pages to optimize large I/O performance
* New option for setting the discard granularity exposed on
   diskless nodes
* Add per-peer_device tiebreaker config flag for quorum
* Fix compat-8.4 /proc/drbd output for connected resources
* Fix a race that could lead that a resyncs stalls when it
   should complete.
* All fixes from 9.2.17
  - Fix a kernel crash triggered by a crafted/invalid netlink message
  - Fix a crash when resyncing with a peer supporting proto 121,
    e.g. drbd-9.1.23
  - Fixed two corner case regressions regarding bitmap lifetime
    (introduced with 9.2.16)
  - Add a static analyzer that checks for correctly handling bitmap
    object lifetime; it identified 6 bugs
  - Correctly request BLK_FEAT_STABLE_WRITES from the kernel; Kernels
    side changes lead to this queue flag not being active
  - Fix lb-tcp to not corrupt data when it reassembles a data chunk, it
    split for sending over multiple paths in parallel
  - Add a special case to ensure LINSTOR re-provisioned nodes with thin
    volumes get the necessary incremental resync
  - Always do a two-phase-commit for attach, closes a race that could
    lead to an inconsistent repl states
  - Skip unnecessary resync when a missed end-of-resync has 0 bits;
    before that fix, DRBD could end up in inconsistent repl states
  - Fix a case where a send error was ignored on congested connections
  - Make detection of missed end-of-resync symmetric; the bug led to
    inconsistent repl states
  - Make the RDMA transport compatible with RXE (software RoCE),
    and some mainly performance-related fixes

https://pkg.linbit.com//downloads/drbd/9/drbd-9.3.1.tar.gz

https://pkg.linbit.com//downloads/drbd/9/drbd-9.2.17.tar.gz

-Philipp

this is drbdadm status results

• DRBD version: 9.3.0

• Cluster size: 3 diskful nodes

• Resource: r1

• Nodes: drbd1, drbd2, drbd3

• Protocol: C

• Single primary (no dual-primary)

Problem Description

In a 3-node DRBD cluster we occasionally encounter a situation where resynchronization stalls and all write I/O to the mounted filesystem blocks indefinitely.

When this happens:

• resync progress stops

• writes to the mounted filesystem hang

• the filesystem cannot be unmounted (umount blocks)

• DRBD status shows peer / dependency suspended states between nodes

Restarting DRBD on one node (drbdadm down/up) causes the resync topology to change and the system recovers.

Reproduction Scenario

The issue can be reproduced with the following sequence.

Nodes involved:

drbd1
drbd2
drbd3

Initial state: all nodes are UpToDate.

Step 1

Promote drbd3 to Primary and mount the filesystem.

Step 2

Disconnect the network between drbd3 and drbd2.

Step 3

Write data on drbd3.

Result:

drbd2 becomes Outdated

Step 4

Restore the network between drbd3 and drbd2.

Result:

drbd3 → drbd2 resync starts

Step 5

Promote drbd2 to Primary and mount the filesystem.

Then disconnect the network between drbd2 and drbd3.

Because drbd2 is now Primary and receives writes:

drbd3 becomes Outdated

Step 6

Restore the network between drbd2 and drbd3.

Current state becomes:

drbd1 : UpToDate
drbd2 : Inconsistent (Primary)
drbd3 : Outdated

DRBD chooses the following resync direction:

drbd3 → drbd2

Step 7

Write data on drbd2.

At this point the problem occurs.

Observed Behavior

1. The resync from drbd3 → drbd2 stalls (no further progress).

2. The filesystem cannot be unmounted.

DRBD status shows the following relationship:

drbd1 → drbd2
    drbd1: resync-suspended: peer
    drbd2: resync-suspended: dependency

This suggests that:

• drbd1 → drbd2 resync is waiting for another resync to complete

• drbd2 ← drbd3 is expected to complete first

However:

drbd3 → drbd2 resync shows "suspended: no"
but makes no progress.

As a result, the whole system appears to be stuck.

Recovery

If we restart DRBD on drbd3:

drbdadm down r1
drbdadm up r1

The resync topology changes to:

drbd1 → drbd2
drbd1 → drbd3

Both nodes resync from drbd1, and the cluster returns to normal operation.

Question

Is this behavior expected in DRBD 9.3.0 when a node that is still Inconsistent is promoted to Primary and receives writes?

Or could this indicate a resync scheduling issue where DRBD chooses a suboptimal sync source (Outdated node) and the resync pipeline becomes stalled?

Any guidance on how to avoid or diagnose this situation would be appreciated.

We just released WinDRBD 1.2.8. The major change in this version is
that the Linux crypto framework was imported into WinDRBD. This means
that (currently) crc32c, sha1 and hmac(sha1) are available for crypto
algorithms. Also, more crypto algorithms can be added easily, since
the Linux kernel's crypto code is now also part of the WinDRBD source
code.

This eliminates the need for ugly patches in LINSTOR, which is why we
did the release now. In addition an endless loop when updating WinDRBD
was fixed.

Once again please join us at the LINBIT community meeting on March 12
where we have exciting news about the future of WinDRBD to share.

You can register at:

All the best,

- Johannes

% egrep -R '\bauto-verify-alg\b' .
./debian/changelog:  * auto-verify-alg: Improve disabled check
./server/src/main/java/com/linbit/linstor/InternalApiConsts.java:    public static final String DRBD_AUTO_VERIFY_ALGO = "auto-verify-alg";
% egrep -R '\bDRBD_AUTO_VERIFY_ALGO\b' .
./server/src/main/java/com/linbit/linstor/InternalApiConsts.java:    public static final String DRBD_AUTO_VERIFY_ALGO = "auto-verify-alg";
./controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscDfnAutoVerifyAlgoHelper.java:                        InternalApiConsts.DRBD_AUTO_VERIFY_ALGO, ApiConsts.NAMESPC_DRBD_OPTIONS
./controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscDfnAutoVerifyAlgoHelper.java:                            InternalApiConsts.DRBD_AUTO_VERIFY_ALGO,
./controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscDfnAutoVerifyAlgoHelper.java:                        rscDfnProps.removeProp(InternalApiConsts.DRBD_AUTO_VERIFY_ALGO, ApiConsts.NAMESPC_DRBD_OPTIONS);
./controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscDfnAutoVerifyAlgoHelper.java:                        InternalApiConsts.DRBD_AUTO_VERIFY_ALGO, ApiConsts.NAMESPC_DRBD_OPTIONS) != null)
./controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscDfnAutoVerifyAlgoHelper.java:                    rscDfnProps.removeProp(InternalApiConsts.DRBD_AUTO_VERIFY_ALGO, ApiConsts.NAMESPC_DRBD_OPTIONS);
./satellite/src/main/java/com/linbit/linstor/layer/drbd/resfiles/ConfFileBuilder.java:                InternalApiConsts.DRBD_AUTO_VERIFY_ALGO, ApiConsts.NAMESPC_DRBD_OPTIONS);

And I noticed this:

./controller/src/main/java/com/linbit/linstor/dbcp/migration/k8s/crd/Migration_01_v1_15_0_init.java:            "DrbdOptions/auto-verify-algo-allowed-list", "crct10dif-pclmul;crct10dif-generic;sha384-generic;sha512-generic;sha256-generic;md5-generic"

candlerb:

Aside: have you found this option documented anywhere?

No, I haven’t found this option in the documentation either.

Is the value of DrbdOptions/auto-verify-alg hardcoded directly in the source code?

with 9.3.1-rc.1 discard on diskless node works again. Tested with latest 6.17.13-1-pve
kernel.

kurbanaliev:

DrbdOptions/auto-verify-alg

Aside: have you found this option documented anywhere? I see it in the linstor source code, but I don’t see it in the linstor manual. (I’m sure it’s valid - linstor does validate the property names - but I’d like to understand the semantics)

Also, I’m not sure if resource-definition list-properties shows properties which are inherited from the resource group - and if it does, whether it’s possible to distinguish between inherited and explicitly-set properties.

brian@virtual1:~$ linstor rd l -r linstor_db
╭──────────────────────────────────────────────────────╮
┊ ResourceName ┊ ResourceGroup  ┊ Layers       ┊ State ┊
╞══════════════════════════════════════════════════════╡
┊ linstor_db   ┊ linstor-db-grp ┊ DRBD,STORAGE ┊ ok    ┊
╰──────────────────────────────────────────────────────╯
brian@virtual1:~$ linstor rg lp linstor-db-grp
╭──────────────────────────────────────────────────────────────────────╮
┊ Key                                                ┊ Value           ┊
╞══════════════════════════════════════════════════════════════════════╡
┊ DrbdOptions/Resource/auto-promote                  ┊ no              ┊
┊ DrbdOptions/Resource/on-no-data-accessible         ┊ io-error        ┊
┊ DrbdOptions/Resource/on-no-quorum                  ┊ io-error        ┊
┊ DrbdOptions/Resource/on-suspended-primary-outdated ┊ force-secondary ┊
┊ DrbdOptions/Resource/quorum                        ┊ majority        ┊
┊ Internal/Drbd/QuorumSetBy                          ┊ user            ┊
╰──────────────────────────────────────────────────────────────────────╯
brian@virtual1:~$ linstor rd lp linstor_db
╭─────────────────────────────────────────╮
┊ Key                         ┊ Value     ┊
╞═════════════════════════════════════════╡
┊ DrbdOptions/Resource/quorum ┊ majority  ┊
┊ DrbdOptions/auto-verify-alg ┊ crct10dif ┊
┊ DrbdPrimarySetOn            ┊ VIRTUAL1  ┊
╰─────────────────────────────────────────╯
brian@virtual1:~$ /usr/sbin/drbdadm dump
...
        verify-alg       crct10dif;

• OS: Ubuntu 24.04

• LINSTOR Controller/Satellite: 1.33.1

• DRBD version: 9.3

I attempted to change the verification algorithm at the Resource Group level using the following command:
linstor rg opt --verify-alg crc32c test

As expected, the new value is visible at the Resource Group level:

root@test-node2:~# linstor rg lp test
+-----------------------------------------------------------+

| Key                         | Value                       |
|-----------------------------------------------------------|
| DrbdOptions/Net/verify-alg  | crc32c                      |
+-----------------------------------------------------------+

However, when checking the Resource Definition properties, I still see a different key with the old value:

root@test-node2:~# linstor rd lp test
+---------------------------------------------------+

| Key                           | Value             |
|---------------------------------------------------|
| DrbdOptions/auto-verify-alg   | crct10dif         |
+---------------------------------------------------+

Interestingly, the actual DRBD configuration correctly reflects the new setting. In the DRBD config, I see the new algorithm that I assigned to the Resource Group:

root@test-node3:~# drbdsetup show test
...
net {
    verify-alg "crc32c";
}
...

I can see that DRBD receives the correct configuration, but after changing the verification algorithm on the Resource Group, I expected this change to be reflected in the Resource Definition as well. The simultaneous coexistence of auto-verify-alg and verify-alg creates confusion from an administrative perspective.
I would appreciate a clarification on the proper way to change the verification algorithm in LINSTOR.
Is it possible to change the default algorithm for auto-verify-alg?
Why is this property not updated or hidden when verify-alg is explicitly defined at the Resource Group level?

I am preparing another double release. 9.2.17-rc.1 brings a bit more
bug-fixes as in the last development cycles, but in nature they are
following the trend we see since quite some time. It is the unusual
corner cases.

The 9.3.1-rc.1 brings fundamental modernization to how DRBD handles
I/O. From now on, DRBD can allocate compound pages from the kernel as
I/O buffers. When you make large I/O requests or streaming accesses,
that can speed things up.

This is a release candidate. Please help test it. If everything goes by
plan, I will do the final release in one week.

9.3.1-rc.1 (api:genl2/proto:86-101,118-124/transport:21)
--------
* Use compound pages to optimize large I/O performance
* New option for setting the discard granularity exposed on
   diskless nodes
* Add per-peer_device tiebreaker config flag for quorum
* Fix compat-8.4 /proc/drbd output for connected resources
* All fixes from 9.2.17
  - Fix a kernel crash triggered by a crafted/invalid netlink message
  - Fix a crash when resyncing with a peer supporting proto 121,
    e.g. drbd-9.1.23
  - Fixed two corner case regressions regarding bitmap lifetime
    (introduced with 9.2.16)
  - Add a static analyzer that checks for correctly handling bitmap
    object lifetime; it identified 6 bugs
  - Correctly request BLK_FEAT_STABLE_WRITES from the kernel; Kernels
    side changes lead to this queue flag not being active
  - Fix lb-tcp to not corrupt data when it reassembles a data chunk, it
    split for sending over multiple paths in parallel
  - Add a special case to ensure LINSTOR re-provisioned nodes with thin
    volumes get the necessary incremental resync
  - Always do a two-phase-commit for attach, closes a race that could
    lead to an inconsistent repl states
  - Skip unnecessary resync when a missed end-of-resync has 0 bits;
    before that fix, DRBD could end up in inconsistent repl states
  - Fix a case where a send error was ignored on congested connections
  - Make detection of missed end-of-resync symmetric; the bug led to
    inconsistent repl states
  - Make the RDMA transport compatible with RXE (software RoCE),
    and some mainly performance-related fixes

https://pkg.linbit.com//downloads/drbd/9/drbd-9.3.1-rc.1.tar.gz

https://pkg.linbit.com//downloads/drbd/9/drbd-9.2.17-rc.1.tar.gz

cheers,
Philipp

There is a two-node active/standby shared-nothing DRBD-based pacemaker cluster. The cluster is configured with qdevice on additional host and two-level STONITH/fencing - fence_ipmilan and diskless sbd (hpwdt, /dev/watchdog). All cluster resources are configured to always run together on the same node.

As additional block-level protection, I would like to add a diskless DRBD quorum on the qdevice host.

How to configure diskless DRBD quorum, if DRBD replication links are directly connected without switch ?

this is drbd-reactor version 1.11.0. There has been a minor new feature
that allows disabling adjusting of DRBD resources on startup[4].

The promoter now supports an on-disk-detach policy[1]. When the local
DRBD disk on a Primary gets detached, the new "fail-over" policy stops
services if an UpToDate peer is available, triggering a failover. The
default ("ignore") preserves existing behavior.

We fixed initial startup ordering on intentionally diskless nodes[2].
Diskless nodes tend to be ready faster and could win the startup race
even when they shouldn't. We now also consider disk state during initial
startup.

drbd-reactorctl status learned a --json flag[3] for machine-readable
output.

Please test, if we don't find any bugs we will release the final version
in about a week from now.

Regards, rck

GIT: prepare v1.11.0 · LINBIT/drbd-reactor@8fb5c64 · GitHub
TGZ: https://pkg.linbit.com//downloads/drbd/utils/drbd-reactor-1.11.0.tar.gz
PPA: https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack/

Changelog:
[ Christoph Böhmwalder ]
* grafana: add $resource variable to dashboard
* doc: add configuration options for prometheus exporter

[ Moritz Wanzenböck ]
* ci: replace gitlab internal registry image
* ci: use custom trivy image
* promoter: add adjust-resource-on-start setting

[ Roland Kammerer ]
* doc: add info about systemd symbols
* core: add toml/target monitoring
* fix minor linter warnings
* ctl: add json output for status command
* promoter: fix initial startup on diskless
* promoter: add on-disk-detach policy for local disk failure
* ctl: rm runner from template
* all: switch to captured identifiers

[1] promoter: add on-disk-detach policy for local disk failure · LINBIT/drbd-reactor@2d3ebad · GitHub
[2] promoter: fix initial startup on diskless · LINBIT/drbd-reactor@2539fc9 · GitHub
[3] ctl: add json output for status command · LINBIT/drbd-reactor@a19a507 · GitHub
[4] promoter: add adjust-resource-on-start setting · LINBIT/drbd-reactor@cbd7950 · GitHub

1. mgmt for server management
2. enp8s0 for iSCSI traffic
3. enp2s0 for DRBD traffic

All the information is:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet 10.30.10.10/24 brd 10.30.10.255 scope global enp2s0
valid_lft forever preferred_lft forever
5: mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
inet
75/26 brd 172.16.35.127 scope global mgmt
valid_lft forever preferred_lft forever
7: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet 172.16.35.6/26 brd 172.16.35.63 scope global enp8s0
valid_lft forever preferred_lft forever

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet 10.30.10.20/24 brd 10.30.10.255 scope global enp2s0
valid_lft forever preferred_lft forever
5: mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
inet 172.16.35.76/26 brd 172.16.35.127 scope global mgmt
valid_lft forever preferred_lft forever
6: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet 172.16.35.7/26 brd 172.16.35.63 scope global enp8s0
valid_lft forever preferred_lft forever

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet 10.30.10.30/24 brd 10.30.10.255 scope global enp2s0
valid_lft forever preferred_lft forever
5: mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
inet 172.16.35.77/26 brd 172.16.35.127 scope global mgmt
valid_lft forever preferred_lft forever
6: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet 172.16.35.8/26 brd 172.16.35.63 scope global enp8s0
valid_lft forever preferred_lft forever

╭───────────────────────────────────────────────────────────╮
┊ Node ┊ NodeType ┊ Addresses ┊ State ┊
╞═══════════════════════════════════════════════════════════╡
┊ domvme01 ┊ SATELLITE ┊ 172.16.35.75:3366 (PLAIN) ┊ Online ┊
┊ domvme02 ┊ SATELLITE ┊ 172.16.35.76:3366 (PLAIN) ┊ Online ┊
┊ domvme03 ┊ SATELLITE ┊ 172.16.35.77:3366 (PLAIN) ┊ Online ┊
╰───────────────────────────────────────────────────────────╯

╭─────────────────────────────────────────────────────────────────╮
┊ domvme01 ┊ NetInterface ┊ IP ┊ Port ┊ EncryptionType ┊
╞═════════════════════════════════════════════════════════════════╡
┊ + StltCon ┊ default ┊ 172.16.35.75 ┊ 3366 ┊ PLAIN ┊
┊ + ┊ stg ┊ 10.30.10.10 ┊ ┊ ┊
╰─────────────────────────────────────────────────────────────────╯

╭─────────────────────────────────────────────────────────────────╮
┊ domvme02 ┊ NetInterface ┊ IP ┊ Port ┊ EncryptionType ┊
╞═════════════════════════════════════════════════════════════════╡
┊ + StltCon ┊ default ┊ 172.16.35.76 ┊ 3366 ┊ PLAIN ┊
┊ + ┊ stg ┊ 10.30.10.20 ┊ ┊ ┊
╰─────────────────────────────────────────────────────────────────╯

╭─────────────────────────────────────────────────────────────────╮
┊ domvme03 ┊ NetInterface ┊ IP ┊ Port ┊ EncryptionType ┊
╞═════════════════════════════════════════════════════════════════╡
┊ + StltCon ┊ default ┊ 172.16.35.77 ┊ 3366 ┊ PLAIN ┊
┊ + ┊ stg ┊ 10.30.10.30 ┊ ┊ ┊
╰─────────────────────────────────────────────────────────────────╯

linstor storage-pool list
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ domvme01 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ domvme01;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ domvme02 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ domvme02;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ domvme03 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ domvme03;DfltDisklessStorPool ┊
┊ gateway-storage ┊ domvme01 ┊ LVM ┊ storage ┊ 800.00 GiB ┊ 800.00 GiB ┊ False ┊ Ok ┊ domvme01;gateway-storage ┊
┊ gateway-storage ┊ domvme02 ┊ LVM ┊ storage ┊ 800.00 GiB ┊ 800.00 GiB ┊ False ┊ Ok ┊ domvme02;gateway-storage ┊
┊ gateway-storage ┊ domvme03 ┊ LVM ┊ storage ┊ 800.00 GiB ┊ 800.00 GiB ┊ False ┊ Ok ┊ domvme03;gateway-storage ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

linstor-gateway check-health --iscsi-backends lio

[✓] System Utilities
[✓] LINSTOR
[✓] drbd-reactor
[✓] Resource Agents
[✓] iSCSI
[!] NVMe-oF
✗ The nvmetcli tool is not available
exec: “nvmetcli”: executable file not found in $PATH
Please install the nvmetcli package
Hint: nvmetcli is not (yet) packaged on all distributions. See 
 for instructions on how to manually install it.
[!] NFS
✗ Service nfs-server.service is in the wrong state (not loaded).
This systemd service conflicts with LINSTOR Gateway.
It needs to be loaded, but not started.
Make sure that:
• the nfs-server package is installed
• the nfs-server.service systemd unit is stopped and disabled
Execute systemctl disable --now nfs-server.service to disable and stop the service.

I configured everything, but at the final step, the creation of the iSCSI target I get an error:

linstor-gateway iscsi create \
  iqn.2001-09.com.linbit:linstorhci \
  172.16.35.9/26 \
  800G

ERRO[0000] failed to create iscsi resource: failed to create linstor resource: failed to autoplace resources: Message: ‘Not enough available nodes’; Details: 'Not enough nodes fulfilling the following auto-place criteria:

the current access context has enough privileges to use the node and the storage pool

the node is online

Auto-place configuration details:
Replica count: 2
Additional replica count: 2
Do not place with resource:
linstorhci
Diskless on remaining: false

Auto-placing resource: linstorhci’

Why do I have an additional replica count: 2?

Other useful information:

linstor-gateway version 2.1.0

cat /proc/drbd

version: 9.3.0 (api:2/proto:118-123)
GIT-hash: f5e0e7ebfa6112db11a7fa0bc755af85c89d968d build by root@domvme01, 2026-02-25 01:44:42
Transports (api:21):

linstor-client 1.27.1; GIT-hash: 9c57f040eb3834500db508e4f04d361d006cb6b5

root@domvme01:~# uname -r
6.8.0-84-generic

root@domvme01:~# uname -a
Linux domvme01 6.8.0-84-generic #84-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 5 22:36:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
root@domvme01:~# cat /etc/os-release
PRETTY_NAME=“Ubuntu 24.04.3 LTS”
NAME=“Ubuntu”
VERSION_ID=“24.04”
VERSION=“24.04.3 LTS (Noble Numbat)”
VERSION_CODENAME=noble

After adding the LINSTOR driver to OpenStack Cinder, three separate cinder-volume services were created (one per controller node).
When I create a volume using a LINSTOR volume type, the volume is assigned to one specific cinder-volume host.
The problem is that when the related cinder-volume service fails, I am no longer able to manage the volume lifecycle operations such as: attach ,detach, resize , . . . .
After extensive testing and research, I tried the following:
1. Setting the backend_host option in cinder.conf for the LINSTOR backend
2. Testing the cluster configuration option in cinder.conf
However, neither approach worked, and the volumes remain tied to a specific cinder-volume host.
My question is:
Does the LINSTOR Cinder driver support solution such as Active-Active operation at the Cinder service layer?
If not, what is the recommended High Availability design for cinder-volume when using LINSTOR as a backend?

Thank you in advance for your guidance

this is version 8.2.0 of the plugin. We did not find any issues in the
RC.

We added NVMe-oF support, allowing LINSTOR-managed NVMe over Fabrics
resources alongside existing DRBD setups. See the commit for
requirements and limitations. We will update the official documentation
soon. We also added a configurable allowtwoprimaries option that allows
using DRBD replication protocols other than C (at the cost of disabling
live migration), and fixed list_images to correctly show all volumes in
the storage pool regardless of which node you query from.

Regards, rck

GIT: linstor-proxmox version 8.2.0 · LINBIT/linstor-proxmox@ca3230d · GitHub
TGZ: https://pkg.linbit.com/downloads/connectors/linstor-proxmox-8.2.0.tar.gz

Changelog:
[ Roland Kammerer ]
* Add configurable allow-two-primaries option

[ Tydus ]
* Fix list_images to show all volumes in storage pool
* Add NVMe-oF support

I encountered a strange problem after an upgrade to proxmox 9.1.5 using kernel 6.17.4-2-pve. When vm is live migrated to a diskless node discard doesn’t work anymore. fstrim command has no effect. Syslog shows the following error at the time of migration, which might be related to the problem:

Feb 24 11:52:42 tn3 kernel: drbd vm-3005-disk-1/0 drbd1009: setting new queue limits failed

If I live migrate the vm back to a diskfull node and do the fstrim then the space is properly reclaimed. I use LVMThin storage pool. Discard works properly on diskless nodes with an older kernel.

The list of my tested combinations:

kernel drbd result

6.2.16-20-pve 9.2.14 works
6.2.16-20-pve 9.2.15 works
6.2.16-20-pve 9.2.16 works
6.2.16-20-pve 9.3.0 works
6.17.4-2-pve 9.2.16 doesn't work
6.17.4-2-pve 9.3.0 doesn't work

Am I missing something here? Thanks.

I believe the core issue comes down to buffer limitations. Sending 25 MB over a 400 Gb/s link takes approximately half a millisecond, which means a single RDMA connection cannot keep up with the throughput.

The real bottleneck appears to be the maximum number of Work Requests per RDMA connection, which constrains the total buffer size. With a Scatter-Gather Element of 8, the limit is 13,106 WRs. Assuming 4 KB buffers, the effective buffer capacity (send + receive) reaches only about 51 MB.

I believe the problem could potentially be addressed in two ways:

1. Use multiple RDMA connections - NVMe over Fabrics (RDMA) uses one connection per CPU, which might distribute the load more effectively
2. Increase buffer sizes per Work Request - using buffers larger than 4 KB could maximize the WR capacity.

Thank you for reporting. I will make sure we open an issue to track this internally.

I have ConnectX-5 cards available for my own testing and typically see around ~15% better performance when using RDMA rather than TCP (IPoIB), but I usually test random writes with small block sizes. When testing with sequential writes and large block sizes, I am also seeing that RDMA is ~3% slower than TCP. So not as drastic as your results, but still counter intuitive.

I will ask the LINBIT kernel module developers for input and report back with any insight.

Steven:

If I understand correctly at this moment the controller needs to be stopped to copy the database files?

Yes, to ensure the database is stable at the time of backup.

Yes, that is correct.

If db files are just “copied” i guess recovery of a controller is just as easy as copying them back in place?

this is the first RC of the upcoming 1.11.0 release. The highlights:

The promoter now supports an on-disk-detach policy[1]. When the local
DRBD disk on a Primary gets detached, the new "fail-over" policy stops
services if an UpToDate peer is available, triggering a failover. The
default ("ignore") preserves existing behavior.

We fixed initial startup ordering on intentionally diskless nodes[2].
Diskless nodes tend to be ready faster and could win the startup race
even when they shouldn't. We now also consider disk state during initial
startup.

drbd-reactorctl status learned a --json flag[3] for machine-readable
output.

Please test, if we don't find any bugs we will release the final version
in about a week from now.

Regards, rck

GIT: prepare v1.11.0-rc.1 · LINBIT/drbd-reactor@12e4b4f · GitHub
TGZ: https://pkg.linbit.com//downloads/drbd/utils/drbd-reactor-1.11.0-rc.1.tar.gz
PPA: LINBIT DRBD9 Stack : “LINBIT” team

Changelog:
[ Christoph Böhmwalder ]
* grafana: add $resource variable to dashboard
* doc: add configuration options for prometheus exporter

[ Moritz Wanzenböck ]
* ci: replace gitlab internal registry image
* ci: use custom trivy image

[ Roland Kammerer ]
* doc: add info about systemd symbols
* core: add toml/target monitoring
* fix minor linter warnings
* ctl: add json output for status command
* promoter: fix initial startup on diskless
* promoter: add on-disk-detach policy for local disk failure
* ctl: rm runner from template
* all: switch to captured identifiers

[1] promoter: add on-disk-detach policy for local disk failure · LINBIT/drbd-reactor@2d3ebad · GitHub
[2] promoter: fix initial startup on diskless · LINBIT/drbd-reactor@2539fc9 · GitHub
[3] ctl: add json output for status command · LINBIT/drbd-reactor@a19a507 · GitHub

this is the first RC of the upcoming 8.2.0 release. We added NVMe-oF
support, allowing LINSTOR-managed NVMe over Fabrics resources alongside
existing DRBD setups. See the commit for requirements and limitations.
We will update the official documentation soon. We also added a
configurable allowtwoprimaries option that allows using DRBD replication
protocols other than C (at the cost of disabling live migration), and
fixed list_images to correctly show all volumes in the storage pool
regardless of which node you query from.

Please test, if we don't find any issues I will release the final
version in about a week from now. Packages are available in the public
testing repos.

Regards, rck

GIT: linstor-proxmox version 8.2.0-rc.1 · LINBIT/linstor-proxmox@a61b58a · GitHub
TGZ: https://pkg.linbit.com/downloads/connectors/linstor-proxmox-8.2.0-rc.1.tar.gz

Changelog:
[ Roland Kammerer ]
* Add configurable allow-two-primaries option

[ Tydus ]
* Fix list_images to show all volumes in storage pool
* Add NVMe-oF support

The RDMA transport driver is very slow on modern hardware like NVIDIA ConnectX-7. It struggles to utilize the adapter’s (400 Gb/s) bandwidth. It appears that sndbuf-size and rcvbuf-size maximum values are too small. It is impossible to specify buffer sizes larger than 25M (sndbuf-size + rcvbuf-size + rdma-ctrl-sndbuf-size + rdma-ctrl-rcvbuf-size = 52424K max), and even with 25M buffers, I see thousands of Not sending flow_control msg, no receive window messages during heavy load.

Additionally, there are many messages about switching to WQ_UNBOUND in dmesg (for both RDMA and TCP transports):

workqueue: iomap_dio_complete_work hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND

Even when switching the transport from RDMA to TCP (using IP over Infiniband), I observe two times the performance improvement.

I have configured 24 DRBD resources. For each DRBD instance, I allocated a dedicated CPU core (reserved only for DRBD):

resource "disk0" {
    device minor 0;
    disk "/dev/disk/by-partlabel/drbd0";
    meta-disk internal;
    options {
        cpu-mask "00000000,00000000,00000001";
    }
    on "hosta" {
        node-id 0;
    }
    on "hostb" {
        node-id 1;
    }
    connection {
        host "hosta" address 192.168.1.1:7789;
        host "hostb" address 192.168.1.2:7789;
    }
}
...
resource "disk23" {
    device minor 23;
    disk "/dev/disk/by-partlabel/drbd23";
    meta-disk internal;
    options {
        cpu-mask "00000000,00000008,00000000";
    }
    on "hosta" {
        node-id 0;
    }
    on "hostb" {
        node-id 1;
    }
    connection {
        host "hosta" address 192.168.1.1:7812;
        host "hostb" address 192.168.1.2:7812;
    }
}

For my write performance test, I used this FIO script:

[seq-write-24]
filename=/disks/0/fio:/disks/1/fio:/disks/2/fio:/disks/3/fio:/disks/4/fio:/disks/5/fio:/disks/6/fio:/disks/7/fio:/disks/8/fio:/disks/9/fio:/disks/10/fio:/disks/11/fio:/disks/12/fio:/disks/13/fio:/disks/14/fio:/disks/15/fio:/disks/16/fio:/disks/17/fio:/disks/18/fio:/disks/19/fio:/disks/20/fio:/disks/21/fio:/disks/22/fio:/disks/23/fio
file_service_type=roundrobin:1
size=2400G
rw=write
bs=1M
direct=1
buffered=0
numjobs=24
time_based=1
runtime=60
ioengine=libaio
iodepth=24
group_reporting

With RDMA and this configuration, I got 12.2 GB/s overall:

net {
    protocol C;
    transport "rdma";
    sndbuf-size 25M;
    rcvbuf-size 25M;
    rdma-ctrl-sndbuf-size 600K;
    rdma-ctrl-rcvbuf-size 600K;
    max-buffers 128K;
    max-epoch-size 20000;
}

With TCP and this configuration, I got 24.3 GB/s overall:

net {
    protocol C;
    transport "tcp";
    sndbuf-size 128M;
    rcvbuf-size 128M;
    max-buffers 128K;
    max-epoch-size 20000;
}

Without replication (DRBD down on hostb), I achieved 129 GB/s

We just released WinDRBD 1.2.7. The main reason for this release is that
an issue with connecting to a Linux DRBD was fixed. If you are running
cross-platform (Linux/Windows) clusters, then you should upgrade to
WinDRBD 1.2.7.

Other than that I would like to invite you to the next LINBIT community
meeting on March 12 2026 at 5 PM (CET) / 8 AM (PT), where we have
some exciting news regarding the WinDRBD ecosystem to share.

Details about the community meetings can be found at:

Best regards,

- Johannes

Downgraded to 9.2 to see if we can reproduce.

New to this forum I would like to share something with you. We have been using DRBD with no issue with the following configuration on Debian 12.

resource "r0" {
    disk {
        al-extents 10000;
    }

    net {
        protocol A;
        verify-alg md5;
        connect-int 60;
        max-buffers 32k;
    }

    volume 0 {
        device minor 0;
        disk "/dev/vg0/drbd-r0-0";
        meta-disk "/dev/vg0/drbd-r0-0-md";

        disk {
            disk-flushes no;
            md-flushes no;
        }
    }

    volume 1 {
        device minor 1;
        disk "/dev/vg0/drbd-r0-1";
        meta-disk "/dev/vg0/drbd-r0-1-md";

        disk {
            resync-after r0/0;
            disk-flushes no;
            md-flushes no;
        }
    }

    on "storage-1" {
        node-id 1;
    }

    on "storage-2" {
        node-id 2;
    }

    on "storage-4" {
        node-id 3;
    }

    connection {
        host storage-1 address *.*.*.*:30012;
        host storage-2 address *.*.*.*:30012;
    }

    connection {
        host storage-1 address *.*.*.*:30013 via proxy on drbd-proxy-main {
            inside *.*.*.*:31013;
            outside *.*.*.*:32013;
            options {
                memlimit 2G;
            }
        }

        host storage-4 address *.*.*.*:30013 via proxy on storage-4 {
            inside *.*.*.*:31013;
            outside *.*.*.*:32013;
            options {
                memlimit 2G;
            }
        }

        net {
            cram-hmac-alg sha1;
            shared-secret "****";
            allow-remote-read no;
        }

        volume 0 {
            disk {
                c-plan-ahead 20;
                c-fill-target 6M;
            }
        }

        volume 1 {
            disk {
                c-plan-ahead 20;
                c-fill-target 6M;
            }
        }
    }

    connection {
        host storage-2 address *.*.*.*:30023 via proxy on drbd-proxy-main {
            inside *.*.*.*:31023;
            outside *.*.*.*:32023;
            options {
                memlimit 2G;
            }
        }

        host storage-4 address *.*.*.*:30023 via proxy on storage-4 {
            inside *.*.*.*:31023;
            outside *.*.*.*:32023;
            options {
                memlimit 2G;
            }
        }

        net {
            cram-hmac-alg sha1;
            shared-secret "****";
            allow-remote-read no;
        }

        volume 0 {
            disk {
                c-plan-ahead 20;
                c-fill-target 6M;
            }
        }

        volume 1 {
            disk {
                c-plan-ahead 20;
                c-fill-target 6M;
            }
        }
    }

    options {
        cpu-mask FFFFFF;
    }
}

Filesystems are mostly btrfs above LVM above dm-crypt above drbd above lvm above BBWC RAID. We have a fast SSD raid and a slow one which is why we have two volumes in the resource.

We updated the system in Debian 13 a few weeks ago and since then we experience some of the following errors :

drbd r0 storage-2: BAD! BarrierAck #47682 received with n_writes=4, expected n_writes=7!

We wonder if there is compatibility issues with DRBD on this kernel version (6.12) or btrfs (6.14), or is there is something we should adjust in our configuration, particularly enabling/disabling drain/flushes/barriers.

Thank you for time by advance

a new version of LINSTOR Gateway has just been released: v2.1.0

Here is the relevant excerpt from the changelog:

-----------------------------------
## [2.1.0] - 2026-02-05

### Changes

* Add configurable resource timeout for all operations. (31276c2)
* Add CORS configuration for the gateway server. (722d7e6)
-----------------------------------

Regards,
Christoph

--
Christoph Böhmwalder
LINBIT | Keeping the Digital World Running
DRBD HA — Disaster Recovery — Software defined Storage

According to the user guide,

“DRBD allows you to set an explicit CPU mask for its kernel threads. By default, DRBD picks a single CPU for each resource. All the threads for this resource run on this CPU. This policy is generally optimal when the goal is maximum aggregate performance with more DRBD resources than CPU cores. If instead you want to maximize the performance of individual resources at the cost of total CPU usage, you can use the CPU mask parameter to allow the DRBD threads to use multiple CPUs.”

Test DRBD resource has,

resource test {
options {
cpu-mask 0000ffff;
}

But looks it still uses single core on Primary and Secondary servers, limiting replication bandwidth up to ~4GByte/s, no matter TCP or RDMA. Test environment completely idle.
Hardware - 400 Gb/s dedicated link for DRBD, PCIe 5.0. NVMe SSDs, AMD EPYC 9455P 48-Core.

So, how let DRBD utilize modern hardware keeping all replication volumes with single DRBD resource ?