Recently we had a weird issue in our office. We had one virtual machine with a snapshot. That by itself isn’t an issue, a snapshot is nothing special. But someone created that snapshot before a software upgrade and forget to delete it. So this snapshot was growing and growing. We found out that there is a snapshot when the VM or service owner requested some additional disk space. We weren’t able to add disk space because of that snapshot. So we scheduled a maintenance window to delete the snapshot. Faster said than done.
The VM went offline because of disk consolidation. That could happen, depending on snapshot size and storage system. But the VM not only went offline for some time, but unexpectedly for hours. Together with VMware support we were able to stop the snapshot deletion. The VM came back online but with the known “Disk Consolidation Needed status”. We found out that this snapshot was about 400 GB in size. What a bummer! So we scheduled another maintenance window to consolidate that snapshot. Unfortunately that didn’t work well. Consolidation timed at around 96%, not sure why. “Error communicating with the host” isn’t very helpful in that moment.
Some research and again having a chat with VMware support led us towards cloning the disk files. During the cloning of a disk file the snapshot will be consolidated. And as you’re doing a disk clone locally on the ESXi host with “vmkfstools” and not withing vCenter, there shouldn’t be a timeout either. So we had out action plan. And we scheduled another maintenance window with the service owner.
The process in short
- Logon to your ESXi host with PuTTY / SSH where the VM is registered (or at least to any ESXi host which has access to the datastore where the VM resides on)
- Shutdown the VM you want to clone because of the file locks
- On the datastore create a new folder where you put the cloned VMDK files
- Then navigate to the current folder where the VM files are located
- Clone the VMDK files from source to target
- Create a new empty shell VM and add the cloned VMDK as existing disk
A little more into detail
When you’re in the source folder, make sure you get the correct disk / descriptor file. If you do a “ls- lah” within the folder (that command shows you a list of files and the filesize in a human-readable format) you will see all vmdk disk files, flat vmdk’s, delta vmdk’s and the vmdk descriptor files. What we should use here as a source file is the descriptor file that points to our snapshot. In our example this is “vmname-000001.vmdk”.
Execute the vmkfstools command and let the magic happen:
vmkfstools --i /vmfs/volumes/vsanDatastore/vmfolder_source/vmname-000001.vmdk /vmfs/volumes/vsanDatastore/newvm/newvm.vmdk -d thin
The actual syntax of the vmkfstools is:
vmkfstools -i source_path destination_path -d disk_format
With the one-liner above you’re going now to clone the VMDK file with a snapshot from the source VM to another folder and name the target file. At the end you can specify the target disk format with “-d” (like for example thin-provisioned). During the cloning process the snapshot will be consolidated / committed with the target VMDK.
Now as you have cloned the VMDK file (and committed the snapshot) into a new folder, you can either create an empty shell VM with the exact configuration as the source VM, add the cloned VMDK as an existing disk, or you can remove the source disk from the source VM configuration and add it there as existing disk.
To clean up that little mess we’ve just created, you can move all files to the same folder (you probably have to remove the VM from inventory and register it again later), and you have to edit the VMX file for sure because the path to the VMDK file changed. So all files (VMX, VMDK etc.) are on the same place / in the same folder. Oh, did I mention that, if you rename the VM, the Storage vMotion will rename all the files too, if the VM will be moved around? Handy feature, isn’t it?
And a little vmkfstools reference
As for your support, please see the help output of vmkfstools below.
[root@eagle:~] vmkfstools
No valid command specified
OPTIONS FOR FILE SYSTEMS:
vmkfstools -C --createfs [vmfs5|vmfs6|vfat]
-S --setfsname fsName
-Y --unmapGranularity #[bBsSkKmMgGtT]
-O --unmapPriority <none|low|medium|high>
-Z --spanfs span-partition
-G --growfs grown-partition
deviceName
-P --queryfs -h --humanreadable
-T --upgradevmfs
vmfsPath
-y --reclaimBlocks vmfsPath [--reclaimBlocksUnit #blocks]
OPTIONS FOR VIRTUAL DISKS:
vmkfstools -c --createvirtualdisk #[bBsSkKmMgGtT]
-d --diskformat [zeroedthick
|thin
|eagerzeroedthick
]
-a --adaptertype [deprecated]
-W --objecttype [file|vsan|vvol|pmem|upit]
--policyFile <fileName>
-w --writezeros
-j --inflatedisk
-k --eagerzero
-K --punchzero
-U --deletevirtualdisk
-E --renamevirtualdisk srcDisk
-i --clonevirtualdisk srcDisk
-d --diskformat [zeroedthick
|thin
|eagerzeroedthick
|rdm:<device>|rdmp:<device>
|2gbsparse]
-W --object [file|vsan|vvol]
--policyFile <fileName>
-N --avoidnativeclone
-X --extendvirtualdisk #[bBsSkKmMgGtT]
[-d --diskformat eagerzeroedthick]
-M --migratevirtualdisk
-r --createrdm /vmfs/devices/disks/...
-q --queryrdm
-z --createrdmpassthru /vmfs/devices/disks/...
-v --verbose #
-g --geometry
-x --fix [check|repair]
-e --chainConsistent
-Q --objecttype name/value pair
--uniqueblocks childDisk
--dry-run [-K]
vmfsPath
OPTIONS FOR DEVICES:
-L --lock [reserve|release|lunreset|targetreset|busreset|readkeys|readresv
] /vmfs/devices/disks/...
-B --breaklock /vmfs/devices/disks/...
OPTIONS FOR VMFS MODULE:
--traceConfig [0|1]
--dataTracing [0|1]
--traceSize <x> (MB)
vmkfstools -H --help