I have been working with migrating our internal Exchange 2007 infrastructure to Exchange 2010. Wow Exchange 2010 is just a great product. I will be documenting several of the things I discovered about the process over the next several days/weeks.
Update: A major “All Paths Dead (ADP)” storage related issue with vSphere 4 and how to workaround it
December 29, 2009Great writeup on the ADP bug and the workarounds from the Virtual Geek blog…
Common behavior:
- They want to remove a LUN from a vSphere 4 cluster
- They move or Storage vMotion the VMs off the datastore who is being removed (otherwise, the VMs would hard crash if you just yank out the datastore)
- After removing the LUN, VMs on OTHER datastores would become unavailable (not crashing, but becoming periodically unavailable on the network)
- the ESX logs would show a series of errors starting with “NMP”
Examples of the error messages include:
- “NMP: nmp_DeviceAttemptFailover: Retry world failover device “naa._______________” – failed to issue command due to Not found (APD)”“NMP: nmp_DeviceUpdatePathStates: Activated path “NULL” for NMP device “naa.__________________”.
This is affecting multiple storage vendors (suggesting an ESX-side issue). You can see the VMTN thread on this here.
Here’s what’s happening, and the workaround options:
When a LUN supporting a datastore becomes unavailable, the NMP stack in vSphere 4 attempts failover paths, and if no paths are available, an APD (All Paths Dead) state is assumed for that device (starts a different path state detection routine). If after that you do a rescan, periodically VMs on that ESX host will lose network connectivity and become non-responsive.
This is a bug, and a known bug.
What was commonly happening in these cases was that the customer was changing LUN masking or zoning in the array or in the fabric, removing it from all the ESX hosts before removing the datastore and the LUN in the VI client. It is notable that this could also be triggered by anything making the LUN inaccessible to the ESX host – intentional, outage, or accidental.
Workaround 1
This workaround falls under “operational excellence”. The sequence of operations here is important – the issue only occurs if the LUN is removed while the datastore and disk device are expected by the ESX host. The correct sequence for removing a LUN backing a datastore.
- In the vSphere client, vacate the VMs from the datastore being removed (migrate or Storage vMotion)
- In the vSphere client, remove the Datastore
- In the vSphere client, remove the storage device
- Only then, in your array management tool remove the LUN from the host.
- In the vSphere client, rescan the bus.
Workaround 2 (only available in ESX/ESXi 4 u1)
This workaround is available only in update 1, and changes what the vmkernel does when it detects this APD state for a storage device, basically just immediately failing to open a datastore volume if the device’s state is APD. Since it’s an advanced parameter change – I wouldn’t make this change unless instructed by VMware support.
esxcfg-advcfg -s 1 /VMFS3/FailVolumeOpenIfAPD
Again all credit for documenting this go to the Virtual Geek blog…. http://virtualgeek.typepad.com/virtual_geek/2009/12/an-important-vsphere-4-storage-bug-and-workaround.html
—————————————————————————————————–
Update 1:
Has VMware fixed the ADP issue? Maybe, only time will tell.
How to setup Exchange 2010 to use a single certificate for internal and external use
December 29, 2009A special shout out to The Exchange Ninjas for this script for Exchange 2007. http://www.exchangeninjas.com/set-allvdirs
I have modified that script to work with Exchange 2010. Enjoy. Copy and paste into a .ps1 file and run from the Exhange Comand Shell.
# Script to allow you to set all virtual directories to a common name like mail.company.com
Start-Transcript
# Variables
[string]$UMExtend = “/UnifiedMessaging/Service.asmx”
[string]$OWAExtend = “/OWA”
[string]$OABExtend = “/OAB”
[string]$SCPExtend = “/Autodiscover/Autodiscover.xml”
[string]$EWSExtend = “/EWS/Exchange.asmx”
[string]$ECPExtend = “/ECP”
[string]$ConfirmPrompt = “Set this Value? (Y/N)”
[string]$NoChangeForeground = “white”
[string]$NoChangeBackground = “red”
Write-host “This will allow you to set the virtual directories associated with setting up a single SSL certificate to work with Exchange 2010.”
Write-host “”
[string]$base = Read-host “Base name of virtual directory (e.g. mail.company.com)”
write-host “”
# =======================================================
# Validate if a third party trusted certificate is being used
# because BITS won’t use untrusted certificates
[string]$set = Read-host “Is the certificate being used an internally generated certificate? (Y/N)”
Write-host “”
if ($set -eq “Y”) {
[string]$OABprefix = “http://”
} else {
[string]$OABprefix = “https://”
}
# =======================================================
# Build the Autodiscover URL and set the SCP Value
Write-host “Setting Autodiscover Service Connection Point” -foregroundcolor Yellow
write-host “”
$SCPURL = “https://” + $base + $SCPExtend
[array]$SCPCurrent = Get-ClientAccessServer
Foreach ($value in $SCPCurrent) {
Write-host “Looking at Server: ” $value.name
Write-host “Current SCP value: ” $value.AutoDiscoverServiceInternalUri.absoluteuri
Write-host “New SCP Value: ” $SCPURL
[string]$set = Read-host $ConfirmPrompt
write-host “”
if ($set -eq “Y”) {
Set-ClientAccessServer -id $value.identity -AutoDiscoverServiceInternalUri $SCPURL
} else {
write-host “Autodiscover Service Connection Point internal value NOT changed” -foregroundcolor $NoChangeForeground -backgroundcolor $NoChangeBackground
}
}
# =======================================================
# Build the EWS URL and set the internal Value
Write-host “Setting Exchange Web Services Virtual Directories” -foregroundcolor Yellow
write-host “”
$EWSURL = “https://” + $base + $EWSExtend
[array]$EWSCurrent = Get-WebServicesVirtualDirectory
Foreach ($value in $EWSCurrent) {
Write-host “Looking at Server: ” $value.server
Write-host “Current Internal Value: ” $value.internalURL
Write-host “New Internal Value: ” $EWSUrl
[string]$set = Read-host $ConfirmPrompt
write-host “”
if ($set -eq “Y”) {
Set-WebServicesVirtualDirectory -id $value.identity -InternalURL $EWSURL
} else {
write-host “Exchange Web Services Virtual Directory internal value NOT changed” -foregroundcolor $NoChangeForeground -backgroundcolor $NoChangeBackground
}
Write-host “Looking at Server: ” $value.server
Write-host “Current External Value: ” $value.externalURL
Write-host “New External Value: ” $EWSUrl
[string]$set = Read-host $ConfirmPrompt
write-host “”
if ($set -eq “Y”) {
Set-WebServicesVirtualDirectory -id $value.identity -ExternalURL $EWSURL
} else {
write-host “Exchange Web Services Virtual Directory external value NOT changed” -foregroundcolor $NoChangeForeground -backgroundcolor $NoChangeBackground
}
}
# ======================================================
# Build the OAB URL and set the internal Value
Write-host “Setting OAB Virtual Directories” -foregroundcolor Yellow
write-host “”
$OABURL = $OABprefix + $base + $OABExtend
[array]$OABCurrent = Get-OABVirtualDirectory
Foreach ($value in $OABcurrent) {
Write-host “Looking at Server: ” $value.server
Write-host “Current Internal Value: ” $value.internalURL
Write-host “New Internal Value: ” $OABUrl
[string]$set = Read-host $ConfirmPrompt
write-host “”
if ($set -eq “Y”) {
Set-OABVirtualDirectory -id $value.identity -InternalURL $OABURL
} else {
write-host “OAB Virtual Directory internal value NOT changed” -foregroundcolor $NoChangeForeground -backgroundcolor $NoChangeBackground
}
Write-host “Looking at Server: ” $value.server
Write-host “Current External Value: ” $value.externalURL
Write-host “New External Value: ” $OABUrl
[string]$set = Read-host $ConfirmPrompt
write-host “”
if ($set -eq “Y”) {
Set-OABVirtualDirectory -id $value.identity -ExternalURL $OABURL
} else {
write-host “OAB Virtual Directory external value NOT changed” -foregroundcolor $NoChangeForeground -backgroundcolor $NoChangeBackground
}
}
# =======================================================
# Build the UM URL and set the internal Value
Write-host “Setting UM Virtual Directories” -foregroundcolor Yellow
write-host “”
$UMURL = “https://” + $base + $UMExtend
[array]$UMCurrent = Get-UMVirtualDirectory
foreach ($value in $UMCurrent) {
Write-host “Looking at Server: ” $value.server
Write-host “Current Internal Value: ” $value.internalURL
Write-host “New Internal Value: ” $UMUrl
[string]$set = Read-host $ConfirmPrompt
write-host “”
if ($set -eq “Y”) {
Set-UMVirtualDirectory -id $value.identity -InternalURL $UMURL
} else {
write-host “UM Virtual Directory internal value NOT changed” -foregroundcolor $NoChangeForeground -backgroundcolor $NoChangeBackground
}
Write-host “Looking at Server: ” $value.server
Write-host “Current External Value: ” $value.externalURL
Write-host “New External Value: ” $UMUrl
[string]$set = Read-host $ConfirmPrompt
write-host “”
if ($set -eq “Y”) {
Set-UMVirtualDirectory -id $value.identity -ExternalURL $UMURL
} else {
write-host “UM Virtual Directory external value NOT changed” -foregroundcolor $NoChangeForeground -backgroundcolor $NoChangeBackground
}
}
# =======================================================
# Build the ECP URL and set the internal Value
Write-host “Setting ECP Virtual Directories” -foregroundcolor Yellow
write-host “”
$ECPURL = “https://” + $base + $ECPExtend
[array]$ECPCurrent = Get-ECPVirtualDirectory
foreach ($value in $ECPCurrent) {
Write-host “Looking at Server: ” $value.server
Write-host “Current Internal Value: ” $value.internalURL
Write-host “New Internal Value: ” $ECPUrl
[string]$set = Read-host $ConfirmPrompt
write-host “”
if ($set -eq “Y”) {
Set-ECPVirtualDirectory -id $value.identity -InternalURL $ECPURL
} else {
write-host “ECP Virtual Directory internal value NOT changed” -foregroundcolor $NoChangeForeground -backgroundcolor $NoChangeBackground
}
Write-host “Looking at Server: ” $value.server
Write-host “Current External Value: ” $value.externalURL
Write-host “New External Value: ” $ECPUrl
[string]$set = Read-host $ConfirmPrompt
write-host “”
if ($set -eq “Y”) {
Set-ECPVirtualDirectory -id $value.identity -ExternalURL $ECPURL
} else {
write-host “ECP Virtual Directory external value NOT changed” -foregroundcolor $NoChangeForeground -backgroundcolor $NoChangeBackground
}
}
# =======================================================
# Build the OWA URL and set the internal Value
Write-host “Setting OWA Virtual Directories” -foregroundcolor Yellow
write-host “”
$OWAURL = “https://” + $base + $OWAExtend
[array]$OWACurrent = Get-OWAVirtualDirectory
foreach ($value in $OWACurrent) {
Write-host “Looking at Server: ” $value.server
Write-host “Current Internal Value: ” $value.internalURL
Write-host “New Internal Value: ” $OWAUrl
[string]$set = Read-host $ConfirmPrompt
write-host “”
if ($set -eq “Y”) {
Set-OWAVirtualDirectory -id $value.identity -InternalURL $OWAURL
} else {
write-host “OWA Virtual Directory internal value NOT changed” -foregroundcolor $NoChangeForeground -backgroundcolor $NoChangeBackground
}
Write-host “Looking at Server: ” $value.server
Write-host “Current External Value: ” $value.externalURL
Write-host “New External Value: ” $OWAUrl
[string]$set = Read-host $ConfirmPrompt
write-host “”
if ($set -eq “Y”) {
Set-OWAVirtualDirectory -id $value.identity -ExternalURL $OWAURL
} else {
write-host “OWA Virtual Directory external value NOT changed” -foregroundcolor $NoChangeForeground -backgroundcolor $NoChangeBackground
}
}
End-Transcript
Posted by barrymmartin 
