I'll do my best to describe the problem and hopefully someone can explain to me what I'm doing wrong. The host only has 1 VM running on it.
Host consists of 3 datastores.
Datastore1 465.66 GB - local - vmfs 3.33 - Max File size 256 GB - Block size 1 MB
Datastore2 931.25 GB - local - vmfs 3.33 - Max File size 256 GB - Block size 1 MB
Datastore3 1.86 TB - iscsi - vmfs 3.54 - Max File size 2048 GB - Block Size 8 MB
Guest Configuration
Hard disk 1 32GB Thick on datastore1
Hard disk 2 100GB thick on datastore2
Hard disk 3 256GB thin on datastore2
Hard disk 4 250GB thick on datastore 3
So the original problem we had was creating a backup with Veeam. I don't recall the original error, but I looked it up and discovered that it was due to trying to create a snapshot of hard disk 3 on a datastore with a 1MB block size. The file plus the snapshot overhead was too large for the datastore max file size.
I shut the VM down and added workingDir = "/vmfs/volumes/4f5eedb2-3b4780ce-93b6-001a4bbe3fe0/vm-snapshots" which moved the snapshot directory to datastore 3.
I am still not able to get a Veeam backup. I contacted Veeam support and after looking at the job logs they told me it was an ESXi problem because datastore2 Hard disk 2 is not available when trying to create snapshots.
Getting downtime on this system is pretty rare, and when I do get it, it's not for very long. Can anyone point me in the right direction? I'll be happy to provide vmware and veeam logs if someone thinks that will help.