Making cloned boot LUNs boot with ESX4

With advanced storage arrays like NetApp FAS and V-Series LUNs can be easily and efficiently cloned. This is appealing for boot from SAN solutions where many nearly-identical images are needed for farms of SAN-connected servers such as ESX. While there are solutions such as ESXi or PXE-based installations, boot from SAN is appealing to many VMware users who have a SAN infrastructure. This has worked well for ESX 3.x, but in vSphere / ESX 4, the solution breaks due to some changes in how the service console (COS) virtual disk (VMDK) is laid out.

The Problem

It seems quite a few people have hit a wall in getting ESX 4 to boot from cloned LUNs. In spite of a few clever tricks, people keep coming to this:

I’ve figured it out and have made it work in a reproducible and fairly simple fashion. In fact, there are a couple slightly different methods that can be used.

Researching the root cause

Feel free to skip this if you're not interested. It's only here because a few people asked.

The problem comes from a couple things. First, the ESX service console no longer lives fully in partitions it owns on the boot device, as it did for ESX 3.x. Only /boot is a partition accessible/mountable by the service console on the boot device. The rest of the service console partitions are in a VMDK within a VMFS that consumes the rest of the boot disk not used by /boot and VMKCORE.

The second issue is standard ESX behaviour when encountering a clone of a LUN that contains a VMFS. The metadata signature of the VMFS will usually not match, primarily the LUN S/N (or in ESX4, the NAA ID, which also contains the LUN S/N). You can resignature, but you need access to the service console to do that! (Yes, in vSphere you can resignature from the vSphere client, but that also requires a working service console.)

The obvious answer to those familiar with VMFS and LUNs and signatures is to set LVM.EnableResignature to 1 before you shut down the gold/master image in preparation for the snapshot àclone steps. Those who have tried this have found that while the volume does get resignatured, the service console still doesn’t mount. It just chokes at vsd-mount and drops to a recovery shell running from a RAM file system, with nothing mounted and many esxcfg commands missing, etc. (Lots of geeks nodding heads saying, “Yeah, exactly what I saw!”)

On a few threads like this one there was discussion on editing esx.conf and changing /boot/cosvmdk to match the new UUID of your resignatured VMFS. There’s a catch-22 with that which is that you can’t change it ahead of time because you don’t know what the resig’d UUID will be. You can’t change it after, because you can’t get to the service console copy of esx.conf, at least not in any way I could find that didn’t totally default the purpose of simple boot LUN cloning.

The third piece to this is how the console VMDK gets its name and what the startup script that mounts it looks for. The COS VMDK, by default, is located in

/vmfs/volumes/<VMFS_UUID>/esxconsole-<system_UUID>/esxconsole.vmdk

(and esxconsole-flat.vmdk which is the actual disk image).

This is the default if you do an interactive installation without a kickstart file. If you install using a kickstart file, even the one that the installer leaves in /root/ks.cfg after an interactive install, the installer script prepends "default-" to the COS VMDK file names, unless you also specify a path option, like this:

virtualdisk 'esxconsole' --size=8004 --onvmfs='Storage1' --path=esxconsole-esx5/esxconsole.vmdk

This is somewhat documented on pp.52-53 of the ESX and vCenter Server Installation Guide for ESX 4.0 and vCenter Server 4.0, in the Installation Script Commands section under "virtualdisk". One thing to be careful of is that the path is meant to be unique, although the boot LUN should also only be used by a single ESX server.

The python script that names and creates the COS VMDK is usr/lib/vmware/weasel/scriptedinstall/preparser.py located in the initrd.img in the isolinux directory of the ESX DVD. The script that actually mounts the COS VMDK during normal boot is etc/vmware/init/init.d/66.vsd-mount located in /boot/initrd.img of an installed ESX server. Note that I don't show leading slashes for files in initrds because they are relative to wherever you extract them.

Once people get past the resignaturing bit, the thing they run into is that 66.vsd-mount does not look for all the likely names that preparser.py might have given the COS VMDK. Here's what 66.vsd-mount does:

1. Check to see if the GRUB boot option "cosvmdk" was specified and if so attempt to mount that.

2. Check esx.conf for /boot/cosvmdk= and if so attempt to mount that. The esx.conf it checks at this point is in /boot/initrd.img, but under normal circumstances that matches the copy in /etc/vmware.

3. Look for a VMDK in /vmfs/volumes/*/esxconsole-*/esxconsole.vmdk and attempt to mount the first one it finds. (Not likely there would be more than one!)

4. Look for a VMDK in /vmfs/volumes/*/cos/default-cos.vmdk and attempt to mount the first one it finds.

It never looks for a default-esxconsole.vmdk, which is most likely how scripted installs name the COS VMDK.

But there's another problem that we run into first. When 66.vsd-mount finds cosvmdk specified in GRUB or in esx.conf, if the file specified doesn’t actually exist, it bombs out the whole script. I think if the file doesn’t exist, it should move on to the next search method.

OK, so how do we make it work?

There are 3 things to "fix" on your golden image to make cloned boot LUNs work, and for 2 of them there is more than one way to do it.

1. Enable resignaturing.

2. Remove /boot/cosvmdk from esx.conf or make 66.vsd-mount ignore the missing file and try something else.

3. Make the name of the COS VMDK match what 66.vsd-mount actually looks for, which you can achieve in a few ways:

a. Use the --path option in ks.cfg to force a VMDK path the script actually looks for. OR

b. Add another search to 66.vsd-mount. OR

c. Rename the COS VMDK to remove the default- prefix. Note that you technically only have to rename the descriptor. If you also rename the -flat, you will have to edit the descriptor file to match. This is probably the quickest and easiest method.

I've tried all of these and they all work, but anything that involved editing a startup script is unlikely to be supported by VMware. One interesting detail is that unlike a running VM, the COS VMDK (both descriptor and -flat) can be renamed while ESX is running and using the VMDK - it's not locked and moving it doesn’t PSOD as I kinda expected.

Another detail is that there are two instances of the files used for boot, including 66.vsd-mount and esx.conf. We're used to /etc/vmware/esx.conf in the running service console, but there is also a copy in /boot/initrd.img. There are discussions on the communities that tell you to extract, edit and repack /boot/initrd.img. Reality is if you do a proper shutdown or reboot, manually rebuilding initrd is completely unnecessary. /etc/rc<6 or 0>.d/K01vmware runs esxcfg-boot --rebuild which rebuilds /boot/initrd.img. How nice for us.

Here's a screen shot of a cloned boot LUN working using a modified version of 66.vsd-mount. The comment after vsd-mount is because I made two versions of the script - one in /etc and one in /boot/initrd.img. Yes, I built the initrd by hand, and when the script in the COS VMDK ran rather than the one in the initrd, that's how I noticed ESX was building initrd for us. I was surprised that the script that actually ran was the one in the COS VMDK, not in /boot/initrd.

Later on, I found the logfile /var/log/boot-logs/sysboot.log. You can see it working (or not) on the lines starting with “sysboot: vsd-mount ...”.

Putting it all together

The first part of this is standard SAN / boot LUN setup, so I’m intentionally vague on some details. Feel free to skim ahead. The workaround stuff is mainly steps 12) through 16).

1) Enable HBA BIOS

2) Set up zoning such that all the HBAs of the servers can access all target ports of the storage. Details on this and best practices such as single-initiator zones are well-documented elsewhere.

3) Boot LUN setup. Create an aggregate (or use an existing one with space). Create a volume. Set the appropriate volume and snapshot options. Create the LUN.

f33> vol create esxgold aggr0 50g

f33> vol options esxgold no_atime_update on

f33> snap sched esxgold 0 0 0

f33> snap reserve esxgold 0

f33> lun create -s 40g -t vmware /vol/esxgold/lun

4) Create igroup for just this ESX (both HBA WWPN)

f33> igroup create -f -t vmware esx9 21:00:00:e0:8b:94:84:0e 21:01:00:e0:8b:b4:84:0e

5) Map boot LUN (ID 0)

f33> lun map /vol/esxgold/lun esx9 0

6) Set the boot device in your HBA BIOS. For a QLogic HBA, you set this in Selectable Boot

7) Enable selectable boot

8) Next to (Primary) hit enter, then select the correct LUN (should be LUN ID 0, and the WWPN of the port of the filer that has this LUN).

9) Install ESX and any goodies you want in your standard boot LUN image.

10) Log in to the service console

11) Edit /etc/vmware/esx.conf

12) Find the line with /system/uuid and delete it.

/system/uuid = "4adf6585-f1ed-6630-3740-001a64249120"

13) Find the line with /boot/cosvmdk and delete it.

/boot/cosvmdk = "/vmfs/volumes/4a30feb6-7d3064c0-534c-0019b9e03288/esxconsole-4a30fe53-5ef0-68b7-226d-0019b9e03286/default-esxconsole.vmdk"

(Note: You don't have to remove this if you modify 66-vsd-mount to check for the file before attempting to mount and exiting with FATAL_ERROR.)

14) Set resignature

[root@esx9 ~]# esxcfg-advcfg -g /LVM/EnableResignature

Value of EnableResignature is 0

[root@esx9 ~]# esxcfg-advcfg -s 1 /LVM/EnableResignature

Value of EnableResignature is 1

[root@esx9 ~]# esxcfg-advcfg -g /LVM/EnableResignature

Value of EnableResignature is 1

15) Make 66.vsd-mount and the name of the COS VMDK match in one of the following ways:

a)

Fix /etc/vmware/init/init.d/66.vsd-mount or copy in my version OR

b) Rename the COS VMDK (easiest)

[root@esx9 ~]# cd /vmfs/volumes/<LOCAL_VMFS_UUID>/esxconsole-<UUID>

[root@esx9 ~]# mv default-esxconsole.vmdk esxconsole.vmdk

OR

c)

Install the golden image of ESX using kickstart specifying the COS VMDK --path option in your kickstart file:

virtualdisk 'esxconsole' --size=8004 --onvmfs='Storage1' --path=esxconsole-esx5/esxconsole.vmdk

17)

Copy in /etc/rc3.d/S99zzclonecleanup script to reset LVM resig and add the correct entry for cosvmdk into esx.conf. Set permissions to 777 so it will execute.

[root@esx9 ~]# chmod 777 /etc/rc3.d/S99zzclonecleanup

18)

You will need some method of resetting hostname, IP, etc. DHCP is an option, but you may need a script to set the virtual MAC address of the service console to some known value, possibly based on a MAC address of a physical NIC. How to do all this is out of the scope of this article, however, the virtual MAC and IP addresses must be changed before booting a clone in order to avoid conflicts.

19) Set NMP SATP PS ALUA RR, and other fun stuff you want the same for every ESX server.

20) Halt the ESX server

21) Create a snapshot of the halted ESX LUN

f33> snap create esxgold master

22) Clone from the snapshot

f33> vol clone create esx1boot -b esxgold master

Thu Oct 22 02:35:33 GMT [f33: wafl.volume.clone.created:info]: Volume clone esx1boot of volume esxgold was created successfully.

Creation of clone volume 'esx1boot' has completed.

Thu Oct 22 02:35:34 GMT [f33: lun.newLocation.offline:warning]: LUN /vol/esx1boot/lun has been taken offline to prevent map conflicts after a copy or move operation.

23) If you are testing on the same server, unmap the original boot LUN.

f33> lun unmap /vol/esxgold/lun esx9

24) If setting up a new server, set it up like the first one – HBA, zoning, igroup.

25) Map the cloned boot LUN to the new server.

f33> lun map /vol/esx1boot/lun esx9 0

26) If you saw the “LUN … taken offline” message when you created the clone, bring the cloned LUN on line.

f33> lun online /vol/esx1boot/lun

27) Boot the server

28) Watch it magically work.

Other notes

If you boot a server from your master/gold image to update or change it, you will need to remove uuid and cosvmdk parameters from esx.conf, reset /LVM/EnableResignature and copy the S99zzclonecleanup script into /etc/rc3.d again prior to shutting down and cloning.

What to do about the CosCorefile entry in esx.conf during this process? It fixes this itself. See the logfile /var/log/boot-logs/sysboot.log after “sysboot: vsd-mount”. The last thing it does (just before “sysboot: mount-root”) is create a core-dumps dir and add the entry to esx.conf using esxcfg-advcfg.

For zoning and igroup creation, it helps to use the HBA BIOS/Fast!Util and scan for devices from each HBA port. This forces each port to login to the fabric, and the WWPN will be visible to the switch. You may need to do this twice (once to set up zoning and once for igroups). This saves having to type (and maybe typo) the WWPNs.

Fixing 66.vsd-mount

This script is located in /etc/vmware/init/init.d and in initrd.img in the same path. It finds and mounts the COS VMDK. Edit the copy in /etc and avoid messing with the initrd. initrd gets rebuilt during a normal shutdown.

I also didn’t put a fix to check for the existence of the VMDK specified in esx.conf before trying to mount and failing because it’s not there. This means you must remove the entry from esx.conf. I didn’t go through the fix of only looking in UUID paths. Maybe later.

[root@esx9 ~]# vi etc/vmware/init/init.d/66.vsd-mount

There are 4 code blocks that look for the COS VMDK. Insert the following between the 3^rd and 4^th block (or copy the 3^rd block and insert “default-” in front of “esxconsole.vmdk”.

Log "Scanning VMFS for COS VMDK..."

for disk in /vmfs/volumes/*/esxconsole-*/default-esxconsole.vmdk ; do

Log "Attempting to use VMDK found by inspection: ${disk}"

echo "Found $disk"

MountDisk "${disk}" continue

return ${SUCCESS}

done

The whole 66-vsd-mount file as it worked for me

[root@esx9 init.d]# cat 66.vsd-mount

#!/bin/sh

# Copyright 2008 VMware Inc.,

# Modified by peterl@netapp.com 27 Oct 09

# Version 0.04prl

echo "COS VMDK instance of 66.vsd-mount ..."

MountDisk()

{

local COSCoreFile=

if [ ! -f "$@" ] ; then

return ${FALSE}

fi

Exec vsd -scu -f "$@" {

if [ "${VERBOSE}" != "no" ] ; then

Error "Failed to mount VMDK Disk: $@"

else

Log "Failed to mount VMDK Disk: $@"

fi

return ${FALSE}

}

COSCoreFile="$(esxcfg-advcfg -qg /Misc/CosCorefile)"

if [ "${COSCoreFile}" = "" -o ! -d "$(dirname "${COSCoreFile}")" ] ; then

Exec mkdir -p "$(dirname "$@")/core-dumps/"

Exec esxcfg-advcfg -qs "$(dirname "$@")/core-dumps/cos-core" /Misc/CosCorefile

fi

return ${TRUE}

}

echo "Method 1 - GRUB command line..."

CommandLineVMDK="$(GetBootOption 'cosvmdk')"

if [ -n "${CommandLineVMDK}" ] ; then

Log "COS VMDK Specified on Kerenel Boot Line: ${CommandLineVMDK}"

MountDisk "${CommandLineVMDK}" return ${FATAL_ERROR}

return ${SUCCESS}

fi

echo "Method 2 - cosvmdk in esx.conf..."

ConfigVMDK="$(GetRawConfigOption '/boot/cosvmdk')"

if [ -n "${ConfigVMDK}" ] ; then

Log "COS VMDK Specified in esx.conf: ${ConfigVMDK}"

if [ -f "${ConfigVMDK}" ] ; then

MountDisk "${ConfigVMDK}" return ${FATAL_ERROR}

return ${SUCCESS}

else

echo "COS VMDK specified in esx.conf does not exist."

echo "${ConfigVMDK}"

sleep 5

fi

fi

echo "VMFS..."

echo

ls -l /vmfs/volumes

sleep 5

echo "Method 3 - Scan VMFS for esxconsole..."

Log "Scanning VMFS for COS VMDK..."

for disk in /vmfs/volumes/*/esxconsole-*/esxconsole.vmdk ; do

Log "Attempting to use VMDK found by inspection: ${disk}"

MountDisk "${disk}" continue

return ${SUCCESS}

done

Log "Scanning VMFS for COS VMDK..."

echo "Method 3P - Scan VMFS for esxconsole (Peter's way) ..."

for disk in /vmfs/volumes/*/esxconsole-*/default-esxconsole.vmdk ; do

Log "Attempting to use VMDK found by inspection: ${disk}"

echo "Found $disk"

MountDisk "${disk}" continue

return ${SUCCESS}

done

echo 'Did not find COS VMDK using esxconsole-*/default-esxconsole.vmdk!!!'

sleep 5

echo "Method 4 - Scan VMFS for cos/default-cos.vmdk..."

Log "Scanning VMFS for legacy COS VMDK..."

for disk in /vmfs/volumes/*/cos/default-cos.vmdk ; do

Log "Attempting to use VMDK found by inspection: ${disk}"

MountDisk "${disk}" continue

return ${SUCCESS}

done

Error "Unable to find COS VMDK"

return ${FATAL_ERROR}

S99zzclonecleanup

This is a post-boot run-once script to reset /LVM/EnableResignature to 0, check and fix the cosvmdk entry in esx.conf. Put this script in /etc/rc3.d. Set permissions to 777!

[root@esx9 ~]# cat S99zzclonecleanup

#!/bin/sh

#S99zzclonecleanup

# Script to reset certain parameters back to sane values after a cloned boot

# LUN boots successfully for the first time.

# 27 Oct 09 Peter Learmonth, NetApp, peterl@netapp.com

# Place this script in /etc/rc3.d in the master boot LUN image and make it

# executable (chmod 777). The script deletes itself upon execution.

# /LVM/EnableResignature should never be left as 1 in order to prevent

# resignaturing VMFS that should be fixed in other ways. In fact, in vSphere

# / ESX 4.0 and greater, there are more granular and safer ways to fix

# signatures.

# It may not be essential to fix the cosvmdk parameter, but in the interest of

# consistency and adherence to normal ESX configuration we check and set the

# esx.conf parameter to match what we're actually running.

# Can't set cosvmdk using esxcfg-advcfg -s

# Exception occured: Unable to find branch boot

resig=`esxcfg-advcfg -g /LVM/EnableResignature`

echo $resig

resig=`echo $resigsed 's/.*is//'`

if [ $resig -gt 0 ]

then

echo "Setting /LVM/EnableResignature to 0"

esxcfg-advcfg -s 0 /LVM/EnableResignature

fi

cosvmdk_esxconf=`esxcfg-advcfg -c /boot/cosvmdk`

cosvmdk_live=`vsd -g`

fixesxconf() {

grep -v '^/boot/cosvmdk' /etc/vmware/esx.conf >/tmp/esxconf$$

mv -f /etc/vmware/esx.conf /etc/vmware/esx.conf.save

mv -f /tmp/esxconf$$ /etc/vmware/esx.conf

echo "/boot/cosvmdk = \"$cosvmdk_live\"" >>/etc/vmware/esx.conf

}

if [ -z "$cosvmdk_esxconf" ]

then

echo "/boot/cosvmdk is not set in /etc/vmware/esx.conf"

echo "Fixing"

fixesxconf

else

if [ "$cosvmdk_esxconf" != "$cosvmdk_live" ]

then

echo "Current service console VMDK does not match "

echo "/boot/cosvmdk in /etc/vmware/esx.conf"

echo "Fixing"

fixesxconf

fi

fi

rm -f /etc/rc3.d/S99zzclonecleanup