Trying out Proxmox
this is or will be a long one, part documenting what I am doing and part figuring out what I am aiming for
I am becoming restless again. Time to stir things up. I installed Proxmox a few times on temp hardware and inside virtual machines but for me the only way to get familiar with such a product is to actually use it. That means installing things on it and running it as ‘production’ (whatever that means in a homelab).
Why? I started to dislike the direction that TrueNAS is taking with their apps (talking about TrueNAS Scale here). All fair, TrueNAS is a company that makes a great product and can do whatever they want with their product. The move to kubernetes for the application platform is one I can live with, but various breaking changes that lead to not being able to update apps got to me (ingress changes, Traefik, PSSQL …). Kubernetes is fine but it is too much and too complex for what I want to do.
My use case for TrueNAS is to have a rock solide storage platform and the ability to run apps on top of it so they can access the data locally; think exposing my music collection via Navidrome, playing with Immich to expose my locally stored pictures and adding pics from my phone. Where the apps were a single entity in TrueNAS Core running inside jails and iocage on FreeBSD allowing me to mount local ZFS datasets into those jails
Long rant but in short it’s time to see if something else fits my needs better. And change is good, I like to tinker and work my way through such little challenges to move an application or service from one platform to another. I did that a few years ago moving from vSphere to XCP-ng. Is it better? No idea but I learned a lot and enjoyed the trip.
The long(er) term goal for now is to move away from TrueNAS and explore Proxmox. Proxmox is not a NAS solution but my needs for a storage server are very basic. TrueNAS is overkill; it was the combo of reliable ZFS storage and the apps on top that made it interesting. So, Proxmox, reliable storage, NAS functionality and a platform to install apps. I like ZFS and understand it well enough to work with it outside of the guards of TrueNAS; that covers data storage. Proxmox works fine with ZFS and a NAS function is easily installed in a container or vm.
So, where to start? I created a vm in XCP-ng and installed Proxmox 8.1. Gave it a little bit of storage and installed a few containers (ha, new technology, nice to explore). XCP-ng is running on a small machine with connections for 2 x SATA and 2 x PCIe nvme. I boot from a small SATA SSD and was running my VMs from a 1 Tb nvme. My first experiment was to see if I could pass through a disk to let a Proxmox vm fully control that disk and see if ZFS would work. So I added another SATA SSD and moved all VM disks over to that. With the freed up nvme SSD I started experimenting. I found instructions on how to pass through disks to a VM and tested it. That worked. Now what? Part of the fun is that I make things up as I go along. With this idea working I starting dreaming about building a mini NAS with Proxmox, running as a VM but with passed through disks. Because I could then reuse (import) those disks when reinstalling Proxmox directly on the system. This way I can already build the new storage and app system but without disrupting everything else. Yes it is more complex than that (make back ups of the Proxmox configs etc but the basic idea should work).
I found a nice deal for a WD Blue SN580 2 Tb and bought two. I like it when my storage is mirrored. I put both nvme SSDs in the XCP-ng host, and passed them through to the Proxmox VM. Setting up a ZFS mirror is easy and all of a sudden I had a tiny Proxmox server in a VM with 2 Tb mirrored storage. I could immediately migrate the few containers over to the ZFS pool, +1 for Proxmox. Setting up CIFS/Samba is also rleatively straight forward looking at this.
Another side project from a couple of months ago was buying and setting up a small system dedicated for gaming. I wanted to get back into a bit of gaming. Of course the buy and build part was more interesting than the gaming part. I temporary repurposed this machine (a Minisforum UM 560 XT with 32 Gb memory) as a second XCP-ng host. The idea is to move all my running VMs over and then see if I can go from Proxmox in-a-VM to Proxmox bare-metal.
As I write this I am migrating the VMs over, this will take several hours, and I am not in a hurry.
To be continued.
Update Dec 17
All but the Proxmox vm have moved to the Minisforum host, everything running absolutely fine. I have been experimenting with LXCs which I like very much. I am still looking to run Proxmox bare metal but instead of running it bare metal on the Xen host I am now considering using the TrueNAS hardware. The storage in the TrueNAS server is already set up in ZFS pools. As long as I have a way of sharing the datasets I want via Samba I should be good to go.
I am now replacing the apps I am running on top of TrueNAS Scale (Navidrome, Jellyfin, Calibre/Calibre-web) with docker versions inside a LXC Alpine container and so far it is fun and straightforward. I need to experiment with bind mounts to get local ZFS datasets into the docker containers and find a manageable way of setting up and maintaining shares/users/ACLs to access the data also via network. I have managed Samba for years by editing the samba config files directly. It works and I don’t want to do that anymore. An LXC with with Samba and Cockpit seems a light weight way to do this; my needs are simple.