Ever had the issue that after trying to delete snapshot which has filled your harddrives, that the snapshots/drives/vmdks needed consolidation, but you get an error message that the files are locked/in use? (Taken from this discussion, which refers to this VMWare KB article ). Steps to take:
- ssh into the vm host
- find out which files are being locked
- release the locks by killing the processes which use them
- retry consolidation
The issue I was facing was to identify the files being locked. Since the drives were full I assume the logs were not being written to anymore, so parsing the logs to find out which files were locked was not working. However, if you use unique filenames for your vms, their datastores and folders you can easily find the right processes. Remember that in our case we had the affected vm shut down!
/ # cd OS_160GB/
/vmfs/volumes/51e469de-32cf9bcd-9ea8-9ab11345d700 # cd "Server-06"
/vmfs/volumes/51e469de-32cf9bcd-9ea8-9ab11345d700/Server-06 # lsof | grep "Server-06"
9405 vpxa-worker FILE 21 /vmfs/volumes/ 51e3f2b6-acba7fca-5980-90b11c45a700/Server-06/Server-06-000002-delta.vmdk
9405 vpxa-worker FILE 25 /vmfs/volumes/ 51e3f2b6-acba7fca-5980-90b11c45a700/Server-06/Server-06-flat.vmdk
9405 vpxa-worker FILE 26 /vmfs/devices/ deltadisks/f658926-Server-06-000002-delta.vmdk
9405 vpxa-worker FILE 28 /vmfs/volumes/ 51e3f2b6-acba7fca-5980-90b11c45a700/Server-06/Server-06-000002-ctk.vmdk
/vmfs/volumes/51e469de-32cf9bcd-9ea8-9ab11345d700/Server-06 # kill 9405
/vmfs/volumes/51e469de-32cf9bcd-9ea8-9ab11345d700/Server-06 # lsof | grep "Server-06"
If the consolidation process seems to get stuck at 95%/99%, you can check whether the vm host is still working on it or not:
watch -d 'ls -luth | grep -E "delta|flat|sesparse"'
It should update the timestamps if it still working on the consolidation process.
