Get the enic and fnic and check for any drivers that are NOT the VMware ones.
$vmhs = Get-Cluster dc1-clv07 | Get-VMHost | sort Name
foreach ($vmh in $vmhs){
Write-Host $vmh.Name " - " $vmh.build -ForegroundColor Blue
#$vmh = Get-VMHost "ral-dev-esx27.rd.allscripts.com"
$esxcli = Get-EsxCli -VMHost $vmh
$face = $esxcli.software.vib.list()
$face | where {$_.Vendor -ne "VMware"} | ft Name,Vendor,Version
$esxcli.system.module.get("enic").version
$esxcli.system.module.get("fnic").version
Write-Host ""
}
Auto Draft
vcac 6.0.1.1 update changes configuration files.
So there is a little issue when upgrading to 6.0.1.1 from 6.0.1 (which is the only supported way.. have to upgrade to 6.0.1 from 6.0 first).
This has only been noticed on the ISO attached cd rom for the update, so I don’t know about the others (seeing how it’s not released on the repository yet)
That removes A LOT of info from the server information. Like it’s db connection, it’s cluster settings… etc.. etc…
Here are the files I have noticed it changing (note only the files that are talked about in the distributed architecture have been checked against, so more configuration file might have actually been changed)
Located in /etc/vcac/
encryption.key
security.properties
server.xml
setenv-core
solution-users.properties
vcac.keystore
vcac.properties
Located in /etc/apache2
server.pem
The following files are changed,
Server.xml
setenv-core
The lines are missing in setenv-core for clustering and the café.node
VCAC_OPTS=”$VCAC_OPTS -Dspring.profiles.active=default,cluster”
VCAC_OPTS=”$VCAC_OPTS -Dcluster.cache.invalidation.poll.enabled=true”
Server.xml appears to be a different format all together, but contains the data, here you can see where it over wrote the database connection in Global Naming Resources, not sure what else has changed here.
the local host setting for clustering appears changed
However we are manually reconfiguring those two files and modifying the the café.node. instance id manually.
Just throwing this out there for anyone that is about to uprade to 6.0.1.1, you have been warned!!
vcpus, more the merrier? pt2
a while ago I published a blog post that is located here that talked about the concept that more vcpus don’t necessary mean more performance, due to ready time, cpu timing and various other things. Read more on the blog post above if you are looking for more information!
This client went live with there system and was experiencing some database time outs, some sluggishness for consumer reports and meter usage. nothing something you want to have happen…
I was able to convince the vendor to let me actually try to REDUCE the number of cpus from 8 to 4, if you recall last time, the server was at 16.3% rdy for barely any load, it moved up to 28.9% during some data migration testing. so, let’s see the results with the cpus reduced….
as you can see it’s not 2.64% ! amazing!!!
the VM was no longer having any performance issues, was generating reports over 4x faster, and had no hiccups or sluggishness about it!
less cpus = more work!
If you look at the picture you will notice there are 43 vms on this host… with 59 vcpus. running the math 59/16 = 3.6875, so the ratio is 3.6875 vcpus to 1 pcpus. The VM runs around 13% util with burst of 66%, we could drop it down to 2 cpus or 3 cpus for even better ratios, and even more performance to all vms, however, the vendor still wont let me 🙂
so there we have it, proof, more cpus, doesn’t always mean better performance, and can in some cases, hurt performance.
always remember to start with low vcpus and move up, it’ll make the VMadmins and the VMs happy!
Oracle on VMware Lic and Support
This will hopefully be a good article for the SMB’s and all my COOP friends over something that is a pain point… Oracle Licensing, and support, first off let me state that there will be multiple links on this post. Some will explain and go to direct links and some will be to other blogs. This will cover Oracle Database Standard edition. That appears to be what is typically deployed in the COOP land
First, let’s talk about support:
http://www.vmware.com/support/policies/oracle-support.html
VMware says they got your back if oracle tells you to reproduce on physical hardware (which I have NEVER, NEVER, EVER NOT ONCE seen)
https://blogs.oracle.com/UPGRADE/entry/is_oracle_certified_to_run_on
Note you will need a support account to view above.
Now that we are all happy because Oracle is supported, let’s move on!
http://www.oracle.com/us/products/database/standard-edition/comparisons/index.html
Oracle states max of 4 sockets; this is fine since mostly all servers in the SMB land are 2 socket.
Standard edition is licensed as follows:
“Standard Edition Per-socket licensing
If you use Standard Edition or Standard Edition One on a 2 processor system you simply need 2 licenses. However, if you use Enterprise Edition you need to take the number of cores into account as well.”
http://www.orafaq.com/wiki/Oracle_licensing#Processor_licensing
Oracle defines VMware as “Soft partitioning”
http://www.oracle.com/us/corporate/pricing/partitioning-070609.pdf
This states the following:
“Soft partitioning is not permitted as a means to determine or limit the number of software licenses required for any given server.”
So you have to License the Physical host based on the number of CPU sockets installed.…
Nowhere does oracle state that you have to license EVERY HOST in a cluster, just the ones where the application is installed,
http://longwhiteclouds.com/2013/06/04/the-fud-strikes-back-oracle-licensing-on-vmware/
“The OLSA does not say you must license every host where Oracle might possibly at some point in the future be installed and/or run, else you would have to license every host in your datacenter. Do you need to license all hosts connected into every SAN where Oracle is installed and/or run? NO! You must license the hosts were Oracle IS installed and/or run. As soon as a VM comes onto an unlicensed host, or Oracle is installed and/or run on an unlicensed host, then you must license that host, end of story.”
Summary:
Now then, once you license a CPU, any VM can run that version of Oracle without the need for addition Licenses.. So let’s Simply create a DRS rule that states the oracle VMs HAVE to run on a host, this is easy. So now that is done, run as many Oracle Standard Edition, in as many VMs are you want too!
Knowing this will hopefully prevent my client from purchasing another piece of physical hardware, and it will cover them for DR.
They will get 4 or 6 licenses, this will cover 2 or 4 cpus at HQ, and 2 at dr, they will be fully covered on Oracle as I will have audit logs saved for vMotion and drs events. If customer decides to do 4 cpu at hq then we have maintenance without shutting down the guest VMs
This will allow them to be uber flexible with everything, and save some cash due to reduced hardware and all the other fun things virtualization brings, as well as adhere to their virtualization first policy!
Alan
Links:
VMware:
https://www.vmware.com/files/pdf/techpaper/vmw-understanding-oracle-certification-supportlicensing-environments.pdf
“When a customer does not have enough Oracle application instances to justify creating a dedicated cluster
for those applications, only a subset of the hosts in the cluster are licensed for the Oracle application….
. At present, Oracle does not have any stated policy regarding clustering mechanisms or DRS Host Affinity. Customers can easily maintain records for compliance..”
http://blogs.vmware.com/apps/2012/11/update-on-virtualizing-oracle.html
http://www.vmware.com/support/policies/oracle-support.html
Oracle:
http://www.oracle.com/us/corporate/contracts/license-service-agreement/index.html
http://www.oracle.com/us/corporate/pricing/partitioning-070609.pdf
https://blogs.oracle.com/UPGRADE/entry/is_oracle_certified_to_run_on
Community Blogs:
http://longwhiteclouds.com/2012/07/21/fight-the-fud-oracle-licensing-and-support-on-vmware-vsphere/
http://longwhiteclouds.com/2013/06/04/the-fud-strikes-back-oracle-licensing-on-vmware/
powershell esxcli set psp
[powershell]
$face = Get-Cluster HQ | Get-VMHost | Get-ScsiLun -LunType disk | Where {$_.MultipathPolicy -ne “RoundRobin”}
Get-Cluster [ClusterName] | Get-VMHost | Get-ScsiLun -LunType disk | Where {$_.MultipathPolicy -ne “DELL_PSP_EQL_ROUTED ”} | Set-ScsiLun -MultipathPolicy “RoundRobin”
$face = Get-VMHost “esxi-hq-03*” | Get-ScsiLun -LunType disk | Set-ScsiLun -MultipathPolicy “Unknown”
$esxcli = Get-EsxCli -VMHost “esxi-hq-03*”
$esxcli.storage.nmp.psp.list()
esxcli storage nmp device set –device naa.64ed2ac55694dea85e16f5d08d01e0dd –psp DELL_PSP_EQL_ROUTED
$meh = $esxcli.storage.nmp.device.list()
: VMW_PSP_MRU
$meh2 = $meh | where {$_.Device -like “naa.64ed*”}
foreach ($ds in $meh2){
$esxcli.storage.nmp.device.set($null,$ds.Device, “DELL_PSP_EQL_ROUTED”)
}
[/powershell]
Auto Draft
how to make an iso with the latest image!
ever wanted to make an ISO with the latest stuff so no need to reinstall or reupdate the host?
Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml Get-EsxImageProfile ESXi-5.1.0-20130304001-standard | Export-EsxImageProfile -ExportToIso -FilePath C:\AlanTemp\esx51latest.iso
SSH, suppress shell, syslog, firewall
making ssh and syslog and enabling via powercli
Get-VMHost atl* | sort name | Foreach {
Start-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} )
$_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} | Set-VMHostService -Policy on
$_ | Set-VMHostAdvancedConfiguration UserVars.SuppressShellWarning 1
$_ | Set-VMHostAdvancedConfiguration -Name Syslog.global.logHost -Value "10.101.114.19"
$_ | set-vmhostAdvancedconfiguration -name Vpx.Vpxa.config.log.level -Value "info"
$_ | set-vmhostAdvancedconfiguration -name Config.HostAgent.log.level -value "info"
$_ | Get-VMHostFirewallException |?{$_.Name -eq 'syslog'} | Set-VMHostFirewallException -Enabled:$true
}
vcpus, more the merrier?
Here we are, back to the tried and true thought that more vcpus equals better performance.
If you are running a fairly low vm:host ratio, this might not be as big of a deal because your physical cpu to virtual cpu ratio is lower. I recently ran into a vendor that “had” to have 8 cpus. Now this client has the latest and greatest most awesomeness host, some dell r620 2x 8 core (ht enabled) with 256gb of ram at 1600mhz – I mean this host is quick. The problem is there are two of them for the whole cluster. I’ve built it this way on purpose: it’s a basic smb that runs everything (and I do mean everything, virtualized, 99% of it is for dr purposes) and it’s backed by some nice eql arrays. So now that you know the background of the client, it’s time I introduce everyone to some important concepts in virtualization.
Welcome to the world of the vmk scheduler. Its job is to tell the vmk when to run vcpus; it likes to run Symmetric multiprocessing (SMP) vcpus at the same time. It will wait until it can, if smp vcpus aren’t run at the same time and the application is multithreaded, some very bad things will happen (cpu returns instruction sets out of order, blah blah blah). So now that we have a basic understanding of that, let’s look and see what some metrics are to see how long it’s taking the scheduler to do its thing…
First let’s discuss the metrics we will be using…
Taking directly from VMware
• Run – Amount of time the virtual machine is consuming CPU resources.
• Wait – Amount of time the virtual machine is waiting for a VMkernel resource.
• Ready – Amount of time the virtual machine was ready to run, waiting in a queue to be scheduled.
• Co-Stop – Amount of time a SMP virtual machine was ready to run, but incurred delay due to co-vCPU scheduling contention.
Esxtop and advanced perf stats to the rescue!!
If you don’t know how to use esxtop, go read elsewhere.
These are the values of this server: (click the image below)
16% of it’s time trying to do work and it can’t…
Ouch… this server just spent 16.3% of its time waiting to be run… poor thing, sad part is it is using less than 600mhz at this time.
Let’s try some things, like only running this one vm on a host, and clearing off all other vms. We would expect ready time to decrease because the scheduler isn’t doing anything because it’s only one vm! It doesn’t even have to cross numa nodes!! (click image below)
As expected, it’s low… super low. That’s great – it’ll be able to rock out any time now..
Okay, so what if I change it so everything is running on one host again…
As expected, it’s back to high again (16.5%). Typically over 5% and you’ll notice performance issues, unless you’re reading this article, or you’ve been around the VMware block before, you won’t really know how to describe it other than “sluggish.”
Stay tuned for part two where we decrease the number of vpcus and watch the efficiency of the vm increase, even though the total amount of work it could do is decreased.