vSphere 6.5 – Top KB Articles for Upgrade Prep

Here are the top KB articles for upgrading to vSphere 6.5:

KB 2146420 – Estimating vCenter Server 5.5 to vCenter Server Appliance 6.x migration time
KB 2147711 – Estimating vCenter Server 5.5 or 6.0 to vCenter Server Appliance 6.5 migration time
KB 2112283 – Regenerate vCenter Certificates
KB 2147548 – Important information before upgrading to vSphere 6.5
KB 2147686 – vCenter Server 6.5 upgrade best practices
KB 2147824 – Migrating VMFS 5 datastore to VMFS 6 datastore
KB 2147289 – Update Sequence for vSphere 6.5 and its compatible VMware products
KB 2147929 – vSphere Client (HTML5) and vSphere Web Client 6.5 FAQ
KB 2113917 – Repointing VC 6.x to a PSC
KB 2113115 – VMware PSC 6.x FAQs
KB 2147454 – Linux VMware Tools update fails to complete
KB 2147672 – Supported and deprecated topologies for VMware vSphere 6.5

Mass Update Media in vCloud Dicrector

In vCloud 5.1 it is very difficult to move media from LUN to LUN once it is uploaded. In vCloud 5.5 this gets better but it is still a pain.

One method I have found that works is to disable the LUN you want the media moved off of, and change something on the media record. Something such as adding a ‘ ‘ (space) to the description will cause vCloud Director to copy the media to a new LUN.

Below is a script to mass update media entries to allow them to move easier.

$MediaToUpdate = Get-Catalog -Org NAME | Get-Media 
Foreach ( $Media in $MediaToUpdate ) { 
$RefMedia = Get-Media -Id $Media.Id
Write-Host “Updating “ $RefMedia.Name 
$OldDescription = $RefMedia.Description 
$OldDescription += "." 
$Media.ExtensionData.Description = $OldDescription 

Just make sure the LUN you want cleaned off is ‘disabled’ and you have enough space on the other LUNs for the newly copied/moved media.

Relocating a VM in vCloud Director

Occasionally I need to move a VM or vApp from datastore to datastore, various reasons why. Sometimes I need to break the linked clone tree and force vCenter to consolidate it. Sure I can power off the VM and execute a ‘consolidate’ via the UI, but what if I need to keep the VM powered on? A storage vMotion does just that.

Sure you can fire up your vCenter Client or WebClient and move the VM that way. If you like your cloud / VM and like them running I would not do that.

Instead run this tid-bit of code in a PowerShell window.

To move a vApp:

$destDatastoreName = “LUN NAME”

$vm_names = Get-CIVApp “VAPP NAME” | get-civm

foreach ($vm in $vm_names) {

$dsQuery = Search-Cloud -QueryType Datastore -Name $destDatastoreName

$dsRef = New-Object vmware.vimautomation.cloud.views.reference

$dsRef.Href = “https://$($global:DefaultCIServers%5B0%5D.name)/api/admin/extension/datastore/$($dsquery.id.split(‘:’)[-1])”


To move a single VM:

$destDatastoreName = “LUN NAME”

$vm_name = get-civm “VM NAME”

$dsQuery = Search-Cloud -QueryType Datastore -Name $destDatastoreName

$dsRef = New-Object vmware.vimautomation.cloud.views.reference

$dsRef.Href = “https://$($global:DefaultCIServers%5B0%5D.name)/api/admin/extension/datastore/$($dsquery.id.split(‘:’)[-1])”

An important rule to remember… a VM can only live on 1 datastore, this means all VMDK’s and the VMX of that VM must exist on the same datasore. Executing the above will make sure that happens. If you fell like breaking things and use the vSphere Client, keep this in mind.

Remove vApp Template With No Catalog Item Link

First… Did you know that vApp Templates and Catalog Items are two different things in vCloud Director? If so, did you know that you can have a vApp Tempalte that does not link to a catalog item? Finally, you can’t have a Catalog Item without a vApp Tempalte backing though.

Well I guess you could. If you manage to get one of those, you will want to follow my guide on fixing that here.

Okay back to detecting if you have a vApp Template that is not mapped to a Catalog Item. To detect these ‘hung’ vApp Templates run the following script as a ‘system administrator’

$VappTemplates = Search-Cloud -QueryType AdminVappTemplate

foreach ( $VappT in $VappTemplates ) {
$vAppTemplate = Get-CIVAppTemplate -ID $VappT.Id

 $vapptid = $vapptemplate.id
 $CatItem = Search-Cloud -QueryType AdminCatalogItem -Filter Entity==$VappTId
 if ( !$CatItem ) { Write-Host "Get-CIVappTemplate -ID $($vAppTemplate.Id) " }


This will procduce an out put of ‘Get-CIVappTemplate’ and the ID of the hung vApp Template. It is up to you to pick what to do with that hung vApp Template. I recommend removing them and re-creating whatever they were. To do that append this to the output and run it in the same PowerShell window:

| Remove-CIVappTemplate -RemoveCatalogItem $False

Detecting and Removing Invalid vApps

After you run vCloud for a while, heck run any management product for a while, and you will have left overs, hung items, or even corrupt vApps.

With vCloud Director you can run the below to find those items and remove them.

$badvapp = Search-Cloud -QueryType AdminVapp | Where-Object { $_.Status -eq “UNRESOLVED” }

foreach ( $id in $badvapp ) {

Get-CIVApp -Id $id.id | Remove-CIVApp


This probably wont remove everything, those items that are left, generally can only be removed via editing the database (I would not recommend that).

Detecting and Removing Invalid vApp Templates

In the event that a vApp Template or Catalog Item fails during the upload process parts of that vApp Template can be left behind. To find and remove those items run the following:

$badvappt = Search-Cloud -QueryType AdminVappTemplate | Where-Object { $.Status -eq “FAILEDCreation” }

foreach ( $id in $badvappt ) {

Get-CIVAppTemplate -Id $id.id | Remove-CIVAppTemplate -RemoveCatalogItem $false


Run the first part to see what there is, run it all to remove broken vApp Templates.

You will need to download the custom powershell pack from here.

Modify Bulk vApp Network Attachments

With vCloud Director if you need to change the Organization vDC Network, for instance you run out of IP addresses. You can use the script below to change the attached network of all powered off vApps.

$vapps = search-cloud -QueryType VAppOrgNetworkRelation | Where-Object { $.OrgNetworkName -eq “Old Network Name” -and $.Status -eq “Stopped” }

foreach ($vapp in $vapps) {

get-civapp -id $vapp.id | Get-CIVAppNetwork -ErrorAction SilentlyContinue | Where-Object { $_.ConnectionType -eq “Routed” } | Set-CIVAppNetwork -ParentOrgNetwork “New Network Name”

Write-Host “Updated ” $vapp.id ” Owned by: ” $Vapp.OwnerName


Simply change the New Network Name and Old Network Name in the commands above.

Query vCloud Director for Storage Profile Usage

Often times an organization will have multiple Storage Profiles, different class fo storage, etc. The simple command of:


Returns a summary or cummulative view of the storage profile usage. To get the actual usage for each profile, execute this command:

$OrgVdc = get-orgvdc (Org VDC Name)

search-cloud -querytype AdminOrgVdcStorageProfile | where { $_.VdcName -eq $orgvdc } | get-ciview

Find Broken Catalog Items in vCloud 5.x

Sometimes when uploading content into vCloud there is a chance that the content can be broken. This can cause programs to have errors. To find these items, run the following in PowerShell as the a ‘system administrator’.

search-cloud -QueryType AdminCatalogItem | Where-Object { $_.Status -eq “Unknown” }

You can now remove them utilizing the API or vCO and the ‘Remove Catalog Item’ workflow.

Monitoring Compute Performance of a VM

Even when using vC Ops from VMware, vCenter metrics, vCloud Metrics, and Hyperic Montoring there are things that can be missed.

What if the datastore shows 1ms latency on the vSphere host, the CPU usage is low, the Memory is not being swapped, but a user insists it is just ‘slow’. What would you check?

In our cloud deployment (25,000 VMs in a single vCloud instance, and we have 4); I have deployed 4 – 6 VMs per vCenter across differnet datastores, on different hosts. These VMs are 512MB of RAM, 16GB of Hard Drive, and 1 vCPU. On these VMs I installed CentOS, MySQL and sysbench.

sysbench for those that do not know runs artificial tests against the HDD, RAM, CPU and MySQL. Since we have the CPU, RAM and HDD monitored by other products, I configured it to only point at the MySQL server. The MySQL test will test every portion of the compute stack. While the RAM and CPU may report within acceptable ranges, they maybe high in those ranges. Combinded that may cause a performance impact.

My script runs every 5 minutes via a cron job, and logs the data to a SQL database, for later reporting and analysis.

I won’t cover how to install sysbench or MySQL here, since that can be found else where on the internet. sysbench & MySQL

You will also need to install iSQL for CentOS, to write your data to a SQL server (if that is your target), those instructions can be found here.

After you have the above installed and ready, you can run the following to get a report on the status of your VMs performance:

sysbench –num-threads=16 –max-requests=10000 –test=oltp –oltp-table-size=500000 –mysql-socket=/var/lib/mysql/mysql.sock –oltp-test-mode=complex –mysql-user=root –mysql-password=VMware1! run > mysql.sysbench

$testvaule = cat mysql.sysbench | egrep ” cat|transactions:” | awk {‘print substr($3,2) ‘}

$date = date -u “+%F %R”

$machine = ifconfig eth0 | grep inet | awk ‘{ print substr($2,6) }’

$hostname = hostname

inssql=”insert into TABLENAME VALUES (‘$date’, ‘$machine’, ‘$testvalue’,’$hostname’)”


The above will insert the data in to a SQL table, for later reporting.

Keep in mind you do not want to make a ‘monster’ VM, the smaller the better, you want to tax all of the components. If you give it 4GB of RAM, then MySQL will cache all of the disk I/O in RAM and not report an accurate number. If you give it 4 vCPU’s you may actually artificially lower your score by causing the hypervisor to schedule 4 vCPUs worth of tasks. In this case smaller is most definitely better.

This will not replace the other monitoring solutions you may have, but it will help to augment those solutions.

Browse Categories