New year - new virtualization host
I have been running my homelab on several machines:
-
Dell PowerEdge T20, Xeon 3-1225 v3, 32 Gb ECC memory, Dell Perc H200 controller and several disks as fileserver with TrueNAS
-
Dell PowerEdge T20, Xeon 3-1225 v3, 24 Gb memory (16 Gb ECC and 8 Gb non-ECC), boot SSD and storage SSD as virtualization host running XCP-ng
-
HP Pro 6300 (desktop), i5 3470 32 Gb memory, boot SSD and storage SSD as virtualization host running XCP-ng
-
HP T620, AMD GX-415GA , 2 Gb memory, 32 Gb storage, running OPNSense for internal DHCP and DNS (see New machine for more details)
-
HP T620, AMD GX-415GA , 16 Gb memory, 32 Gb storage, running Ubuntu 20.04 LTS with Docker (see New machine for more details)
- It really is convenient to have management services like Xen Orchestra separate from the managed XCP-ng hosts
After reading a lot, pondering a lot I decided that server hardware is great but expensive and not adding much value for my use case. I’ll probably miss out-of-band management every now and then but for a virtualization host, in my lab, consumer grade hardware is fine.
I am looking for a minimum 64 Gb of memory, (relatively) low power usage, local storage to boot from and to store vm disks, small(ish) footprint even though I have plenty of space and reasonably priced. I will most likely use it for years so a little bit of investment is fine.
I settled for this setup:
- Asrock Deskmini X300
- this is a barebone AM4 socket system accepting 2 SODIMM memory modules for up to 64 Gb of memory
- AMD Ryzen 5 5600G
- 6 cores/12 threads and pretty decent power usage when idling
- Kingston ValueRAM KVR32S22D8/32
- Listed at the Kingston website as compatible with the X300
- 32 Gb to start with, with the option to add 32 Gb more
- Noctua NH-L9a-AM4
- Research tells us that the stock cooler does not fit into the Deskmini
With this in place I am looking at moving the vm’s running at the T20/XCP-ng host to this one, freeing it up to run a few test setups like ProxMox and possibly TrueNAS temporarily to help with some storage migrations.
The Deskmini has been ordered and is on it’s way. It looks like this model is out of stock or is not manufactured anymore because it is becoming hard to find. Putting something similar together myself is not that difficult but there is no guarantee that the result is low power. Processor, memory, Noctua cooler and a power meter are ordered as well.
While I plan to decommission at least one of the older systems there is no hurry. Today (Feb 22) TrueNAS Scale has been released and it is something I have been looking forward to test. While I love TrueNAS the fact it is running on FreeBSD has caused some excellent learning opportunities but also some frustation. I am not a purist in the sense that I want my NAS to be NAS only; I do see the value of running workloads that benefit from low-latency access to (shared) storage. Hosting music streaming apps, photo publishing software where the photos are stored on the same machine (no duplication or NFS/CIFS shares with multi-machine dependencies) is a plus in my view. It is my lab after all.
So it looks like the 24 Gb T20 will be repurposed to properly test TrueNAS Scale before I bite the bullet and migrate my full-time NAS.
Update Feb 25
All the pieces arrived. It has been a (long) while since I inserted a CPU and added a cooler but with a bit of RTFM I got there. CPU installed, the Ryzen 5600G, with the Noctua cooler and 32 Gb of memory and then the moment when you press the power switch. I was very much aware that early batches of the X300 shipped with an older BIOS that would not recognize the Ryzen5. But no such problem here, system booted, CPU recognized, memory all there.
A little bit of testing, booting various distros, including XCP-ng, to see if there were any showstoppers, but all started fine. So I shut down my XCP-ng host with 24 Gb of memory (a Dell T20), pulled out the SATA SSDs and connected them. Uh, no disks to boot from? Ah, UEFI. Disable that and he presto, there is XCP-ng booting nicely from the SATA disk, vm’s starting etc. At this point I have to mention I was testing all of this without a network connection because, why would I run a long cable to the room where I was doing all this? Network card was properly recognized and all. Well, if you ’transplant’ disks to new systems they remember what network cards were configured before. No problem, XCP-ng has this option to reset the network. Fill in the details, reboot and … nothing. Huh? Down the rabbit hole of kernel modules, RealTek network cards/versions. Until I just went to the machine and looked at the output of the boot process. No eth0 indeed, but what was eth3 doing there? Apparently the network reset did only so much. As soon as I pointed XCP-ng at eth3 as the network card to use all was well.
So now I have the same XCP-ng instance running in less then 1/8 of the volume, using less power, and with triple the amount of threads and more memory. What’s not to like?
For now I am really happy. I’ll take some time to see if this remains stable and then find a nice 1 Tb NVME disk to replace the current 500 Gb SATA disk so I can reuse it somewhere else. I might replace the boot disk as well so I can use it in the ‘old’ T20 and play with that a bit more; TrueNas Scale is on the list, but also Proxmox VE.
Update March 6
The new machine is running perfectly fine. 12 threads is overkill but since I want to consolidate two machines into this one (eight older cores into six core/twelve threads) it is nice to have the overhead. I ordered another 32 Gb memory DIMM. I can run, with a bit of squeezing, all current VM’s on 32 Gb but since the whole point of this environment is the ability to add and remove systems as I see fit again overhead is good.
◎ Deskmini running XCP-ng
I did some power measurements. This new system runs at around 20W idle which is fine. I also took a look at the older HP Deskpro and this runs at 30W idle. I am currently looking at the NAS and with 5 HDD and 3 SSD this idles at 60W. That might be a nice little project for the near future. There are some optimizations possible for that system, such as removing the additional dual-port network card that is now unused and removing two older 3 Tb HDDs.
Update March 11 - final update
The additional 32 Gb memory arrived earlier this week. I now have a 12 thread - 64 Gb playground all for myself. I did some quick power measurements on the other Dell T20, with 2 HDD, and found it varying between 35 and 40 Watt. TrueNAS Scale installed just fine on this machine but there is no need to run this on a physical machine now that I have enough resources on the new host. So I switched it off and built a vm with 8 Gb of memory to play with TrueNAS Scale and also a vm with 8 Gb to test Proxmox because, why not?
The system is running fine, it is silent and cool. Enough for the next couple of years. I might swap the current 500 Gb SDD for a 1 Tb NVME drive but for now this is not necessary.