LVM PV to fill the primary disk

**Caveat** I'm new to OpenEBS, and I apologize if anything here is misleadingly wrong.

## What I'd like

BROS currently fills the primary with the data partition, which is an ext4. This is useful for container images, logs and similar ephemeral data.

In my attempts to build a production-grade Kubernetes cluster that can scale from 1 to many nodes, I'm looking for a way to run OpenEBS in a way that can use the spare capacity for replicated block storage.

E.g. Hetzner's cheap servers (Hetzner Robot, not Hetzner Cloud) can have TBs of storage, sometimes with HW RAID. They often have two disks of the same size. Either disk configuration (left separate, mirroring or striping) will make one device wastefully huge for the BROS data partition. It would be very useful to install BROS, tag the node with "is block storage" and have the unused quantity show up as available OpenEBS block devices. (Any non-primary disk would be no problem, since BROS doesn't care about them.)

## Any alternatives you've considered

OpenEBS supports two modes: local and replicated, each with multiple drivers available.

### Local PVs

For local PVs, useful only for `StatefulSets`, OpenEBS can use

* a local directory
* a local block device
* a [ZFS pool](https://github.com/openebs/zfs-localpv#features)
* an [LVM VG](https://github.com/openebs/lvm-localpv#features).

The local directory is the only currently viable option, pointing it to the data file system. A local block device is not viable right now, since the disk is filled with the data partition. These drivers provide no OpenEBS benefits in terms of snapshotting, or backups. Using local directory would directly interfere with quota isolation of the Kubelet vs OpenEBS. Using local block device is simplest to implement, but requires statically deciding the size of the data partition, which may not scale well. (Note that we're talking about stateful servers, so "just tearing it down and make a new" might be more painful/costly than the normal BROS usecase.)

The ZFS and LVM drivers are the most versatile, in that OpenEBS can orchestrate snapshotting and backups. (And of the two, ZFS is the most capable, see links above.) They require appropriate kernel support, and user-space utilities. Supporting this would require

* installing ZFS or LVM kernel modules
* installing ZFS or LVM userspace utilities (some of which would have to live in the rootfs, to resize the data file system)
* the disk builder creates a ZFS pool, or an LVM VG instead of the data file system
* the data file system is provisioned on top of that
* this pool or VG is resized to fill the disk on first boot
* the data file system needs to have an upper size limit so it's not grown too large.

**Conclusion** Combining versatility and simplicity, I propose supporting the LVM driver for `ReadWriteOnce` volumes, used by `StatefulSets`. This allows easily adjusting the data file system after the fact, and still make use of advanced OpenEBS's snapshotting capabilities. Using ZFS would be a larger change to BROS.

### Replicated PVs

For replicated PVs, the [cStor driver](https://github.com/openebs/cstor-operators/blob/develop/docs/quick.md) is essentially a forked ZFS, but is managed outside the kernel. In this case, one gives OpenEBS unused block devices, and it will create zpools and sprinkle stardust. (I don't believe ZFS kernel modules are required for this.)

Using this requires having `open-iscsi` installed on nodes, and of course leaving an unused block device. There is no specific benefit to using LVM here, since snapshotting and backup functionality is implemented in the cStor layer. However, using LVM would allow the same benefits as for the local case: ability to easily adjust the data file system size while remaining online.

**Conclusion** Making the data partition have a max size, and adding a dedicated partition to fill the disk would work, but using LVM will make it far easier to adjust the relationship between data file system and cStor device size at installation time.

### Together

For both simple local cases, and replicated volumes, it seems that using LVM for the data partition has benefits. The default could still be that the data file system spans the entire LVM PV, and a user-data option would give it a max-size. Any remaining LVM space would be given to an unused PV.

No source file or test is named. Start by tracing the disk builder and first-boot resizing path, then review how the data filesystem and user-data options are defined. Done means the proposed LVM-backed layout, size limit, and remaining unused capacity are specified and integrated for both local and replicated storage use cases.

Written by the indexing model from the issue text.