Should you buy dedicated hardware or a OP workstation for you next testing environment. If you are not sharing it with others, this might be useful for you.
History
For many years now VMware Workstation has been my secret weapon an daily tool for just about everything in regards to customer remote connections, test environments as so on.
Recently I needed to do some advanced testing with NSX-V and NSX-T. This required a lot more power than what I normally use so I needed to upgrade my testing platform.
The consideration is always with these things. How much are you going to invest, and what are the benefits. For a long time I have been considering buying 4 Intel NUC PC’s for doing these tests, but the problem is that to get a real setup that is flexible you need to invest a lot. Also it is not very flexible as you need to maintain them, and reinstall them everytime you need to play with a newer or older version.
After doing some testing with TrueNAS I decided to run everything on VMware Workstation. It is more flexible to run the ESXi hosts nested if it would perform to my satisfaction.
So I decided to upgrade my Physical Workstation to be up for the task. I already has a 32 Core Threadripper in the rig along with NVME disk and some very large Spinning Disks in a raid setup. It was equipped with 64GB of memory, but I quickly decided that was not enough. I do not want to spend a lot of my precious time resizing every appliance, also that does not give great performance. So I bought 256GB of replacement memory. RGB of course. YOU NEED THE BLING!
Hardware Setup
Here are the important bits. There is also some spinning disks in a RAID1 setup.
| CPU | AMD Ryzen Threadripper 3970X Processor 32-Core, 64-Thread, 3.7/4.50GHz |
| Memory | Gskill D4 3600 MHz 256GB C18 TridentZ RGB K8 F4-3600C18Q2-256GTZR |
| SSD | Corsair Force MP600 NVMe Gen4 M.2 – 2TB |
Storage
As I mentioned I decided to use TrueNAS as my storage. I did this because I did not want all my VM storage to based on NVME even though that would be nice. I have great faith in the caching algorithms of ZFS, which turned out to be justified, as it really performs great.
I just installed the TrueNAS appliance using the ISO in a VM. To start out with I gave it 8GB of memory. 100 GB of NVME storage for caching, 100 GB of NVME storage for Log and 1 TB of spinning disk storage. Which performs great.
I my experience most caching algorithms perform really poorly, and that shows in an environment like this. The ones I found to be doing well in this regard is VMware vSAN Hybrid, NetApp Storage and ZFS. To keep it simple I went with ZFS.
VM Environment
I have created the following VMs in Workstation
| VM | CPU | Memory |
| Virtual Router | 1 | 64MB |
| TrueNAS | 4 | 8GB |
| ESXi Host 1 | 8 | 32GB |
| ESXi Host 2 | 8 | 32GB |
| ESXi Host 3 | 8 | 32GB |
I could easily have set the hosts to 64GB of memory, but that was not necessary for this environment.
The TrueNAS provides shared storage using NFS to my ESXi hosts.
I have created the following nested VMs on my virtual ESXi hosts.
| VM | CPU | Memory |
| Windows 2012 AD DC | 2 | 4GB |
| Windows 2019 AD DC | 2 | 6GB |
| vCenter | 2 | 12GB |
| NSX-V Manager | 4 | 16GB |
| NSX-T Manager | 6 | 24GB |
| Log Insight | 4 | 8GB |
Apart from the listes VMs there are also some NSX controllers and other stoff for testing.
Conclusion
The environment does everything it is supposed to, and is very flexible. I think it is a better solutions than the Intel NUC’s, as that would be very cumbersome to maintain, and could also only run one version at the time. I do not feel I could get enough CPU into thoses to run nested ESXi hosts on them that would perform good enough.
The performance it really great in VMware Workstation when you have the right hardware. I will say that you can’t really feel that you are running nested servers. They feel native. Also the boot time or quite normal, so you can start it up quite quickly thanks to TrueNAS and ZFS caching.
Tips
It is important to configure VMware Workstation to fit all VM memory into physical memory, otherwise you will not get great performance.
If you TrueNas has some disks on NVME/SSD, but the VM itself is not located on SSD, then you snapshot redo logs will be kept in you VMs home folder. This means that all writes will go to spinning disks instead of NVME/SSD’s while running on snapshots. You need make a change to you vmx file as described here: https://kb.vmware.com/s/article/2007069 Please note that you need to set it to false.
snapshot.redoNotWithParent = "FALSE"