Installing Syslog Collector and ESXi Dump Colector for vSphere 5.1

This is not quite as simple as it seems. But, once you know how to do it, it is easy. It took me a couple hours to figure it out.

Of course,  you first need to install these applications on a Windows server. You can use your vCenter Server since they do not take up much space or resources. This part is pretty straight forward.

Once the applications are installed, you must configure your esxi servers. This took me some time to figure out.

For syslog, start by enabling the security port for "syslog" in your firewall settings on each esxi server from the vsphere client. This step is not documented and can easily be overlooked.

Your next step is to go to syslog under "Advanced Settings" on each esxi server. Go to syslog.global.loghost and enter: tcp://<vcenter51 fully qualified server name>:514   This assumes you installed Syslog Collector on your vCenter 5.1 server. If not, use the name of the server you installed it on. And to use port 514.

Once this is done, you will immediately see syslog files being generated.

ESXi Dump Collector is just as easy, but you need to do it from your vSphere Management Server or from the command line on your esxi servers. There is no firewall port to enable for this.

From either vma or the command line on your host, execute the following commands:

> esxcli system coredump network set -v vmk0 -i <ip address where you are running Dump Collector> -o 6500
>esxcli system coredump network set --enable true
>esxcli system coredump network get

The first command tells your host to use the vmkernel port for your Management Port, lets it know the IP address where ESXi Dump Collector is running (probably your vCenter 5.1 Server), and  use port 6500

The second command enables the esxi host to use ESXi Dump Collector

The third command verifies this is working correctly.

Now you have a central location configured for your syslog files and coredumps.

Using vSphere ESXi Image Builder

You need to use vSPhere ESXi Image Builder if you want to upgrade you ESX 4 hosts to ESXi5.1. Minimally you will need to build a custom ISO that contains your updated server drivers. If you use 10GB, you will need to add that as well.

One important task is you must add all your offline bundles into a depot and then list the software packages to verify they are there. After that, it is a simple matter to add the software packages into your image. You will need to do this to upgrade your ESXi 4 hosts to ESXi5.1.

The first step is to download your ESXi 5.x depot zip file. If you are creating an image for esxi5.1, it is called: VMware-ESXi-5.1.0-799733-depot.zip. This can be download from VMware's download page. Make sure you download the zip file and not the ISO.

Once your ESXi zip is downloaed, download all the drivers you will need for your ESXi server. If you are using HP servers, you will need the latest drivers (zip) from HP's site. I also needed the Intel 10GB driver.

Download all your drivers and copy them and the ESXi zip file to a folder on your C drive where you are running vSphere PowerCLI.

Once your files are in a folder, look at the properties of each file and make sure it is unblocked.

Assuming you have Power Shell and VMware vSphere PowerCLI5.1 installed , you are ready to go.

Step 1: Open vSphere PowerCLI and run: Add-ESXSoftwareDepot -DepotUrl c:\folder\zipfile
       This will add your offline bundle or ESXi zip file to a depot

Step 2: Repeat Step 1 for all the drivers you downloaded. These must be in the form of a bundle or offline_bundle and have a zip extension. (vib files do not work here). If you do not do this for your downloaded drivers, you will not be able to add them to your image later.

Step 3: Run: Get-ESXImageProfile to list your current profiles. You will have to clone one of these.

Step 4: Now you need to create a new profile by cloning an existing one you just listed. Run the following:
  New-ESXImageProfile -CloneProfile "current profile" -Name "give it a name" -Vendor "Tom"
   -AcceptanceLevel  "CommunitySupported"  ;  Name yourself as the vendor. Use CommunitySupported to avoid not being able to add a software package to your profile. If you run Get-ESXImageProfile again, you should see your newly created cloned profile listed.

Step5: List the software packages and look for the ones from your vendors (ie...HP , Intel, etc....)
  Run: Get-ESXSoftwarePackage  ; Look for the vendor packages and document them

Step 6: Now that you have done all the above, here comes the easy part. Run the following to add your software packages you documented in step 5 to the profile you created in step 4
  Add-SoftwarePackage -ImageProfile "your profile"  -SoftwarePackage "partner-package"

Step 7: You will need to repeat step 6 for every softwre package

Step 8: Export your Image Profile to an ISO. You can use this to install ESXi or upload the ISO into Update Manager to upgrade your existing hosts. I recommend using the name of the ISO -custom.iso For example VMware-VMvisor-Installer-5.1.0-799733.x86_64-custom.iso
Run: Export-ESXImageProfile -ImageProfile "clone created in step 4" -ExportToISO -FilePath "c:\folder\ISO name".

snapvault

Struggled with snapvault this week. First, we were missing some datastores in one of the snapvault jobs, then the snapvault copies on another datastore disappeared.

When snapvault works, it is great. But when it does not, it is a pain. If a datastore should be getting snapvaulted, but is not, the only way to know this is to be aware which datastores are suppossed to get snapvaulted and verify the snapshots on either the promary or secondary. It is very easy to create a new datastore, and then forget to add it to the snapvault job.

I have a script that checks for snapshots, but I need to remember to add new datastores to the script. Time to review my scripts!

Passed!

Passed the VCP5 exam on 2/28/12... This was relatively easy compared to the VCAP, which I am scheduled to re-take in a couple weeks.

4.1 Upgrade postmoderm

Upgraded our vDS switches to 4.1 and enabled Network I/O.  Thi was very easy to do and had no surprises. The only thing left to do is to finish upgrading vmware tools, and  one cluster still needs to be upgraded. Two clusters have older servers and will not get upgraded to 4.1.

Once again, after switching the vCenter Server database to SQL, everything went very smooth. There were just a couple gotchas along the way. One was with 4.1U1, we were unable to edit the settings in a template without removing it from inventory first. Upgrading vCenter Server to 4.1 U2 foxed this. The other issue was the historical perfromance data quit working a couple times. We fixed that by re-creating the view tables in the database.

I almost do not want to upgrade to vSphere 5, now that 4.1 is working so well. But we will probably do that prior to migrating our ERP physical servers to Vms which should happen in the fall.

iscsi on Intel 10 GB NICs

Software iscsi using the INtel 10 GB nics was relatively easy to setup. The only issue I ran into is 4.1 recognizes my broadcom nics as being iscsi compatible, but that is not want tI want to use, So I had to make sure when I was in the CLI that I selected the software iscsi and not the hardware dependent iscsi hba.

after this, the fun part starts The main reason I was interested in configuring iSCSI was to test the ability of Snap Drive to create a lun from within a VM using the VMware iSCSI initiators and VMware RDM.
I can create a lun using Snap Drive when the VM is in an NFS datastore. The RDM pointer gets created in a separate dedicated  iscsi datastore. Both vmotion and storage vmotion work great.

Next on the agenda is testing snap manager for sql and the ability to connect to existing FC luns create with snap drive.

software iscsi with Intel 10 GB NICs and esxi 4.1 Update1

I'm looking fiorward to testing this in the lab. I have two 10 GB NICs that handle my NFS traffic, guest iscsi traffic, VM port traffic, and vmotion.

I've configured software iSCSI using the 10 GB nics and will see if I can add this to the mix. That will provide a combination of NFS and iSCSI storage over redundant 10 GB paths, and iSCSI muti-pathing. Also give me the option of creating vmware RDM drives.

ESXi 4.1 Upgrade

I wanted to go with the latest and greatest vMA, so installed vma5, which is compatible with 4.0 U2 and 4.1. Since my system is running at 4.1, I figured we were good.

This has worked well. I was using the vCLI and CLI to execute commands from the command line. But the VMA definately makes it simpler and easier to do. I especially like it for adding additional NFS volumes, which in a 9 host cluster can be time consuming.

The only disapppointment so far is vmware has removed the vi-logger from vma5. I was looking forward to consolidating my logs with this, but now it will be a little more complex.

One last step I almost forgot is to upgrade the vDS networking to 4.1. I'll have to schedule that and then I can enable Network I/O control.

OnCommand 5 Update

We made more progress on getting smvi and snapvault to work with OnCommand 5. OnCommand 5 replaces SMVI and the sv-smvi script.

We needed the follwoing KB articla to configure this and to understand the differences between how OnCommand 5 does snapshots compared to VSC and SMVI.

http://communities.netapp.com/thread/19792

vSphere 4.1 Upgrade Progress

I wrapped up the ESX upgrades on Sunday. Upgraded the final five ESXi servers to 4.1 Update1. This worked flawlessly. A huge help in doing this was incresing  the memory on two esxi hosts to 192 GB. This allowed me to vmotion many VMs ahead of time to get five hosts ready.  Upgrading five hosts and verifying each took about 2.5 hours.

I am impressed with 4.1 so far. It seems to be a very solid version with significant improvements over 4.0.

Goals for 2012

Here are my goals for 2012:

• Get certified in vSphere 5 - done , 2/28/12... JIT to save me from taking the What's New course.
• Get a VCAP certification
• Learn  Power CLI (change to just Power CLI)
• Get experience with cloud self-servicing portals and cloud automation
• implement a vmware monitoring tool
• Get VSP

vSphere 4.1 upgrade is almost finished

With five ESXi servers left to upgrade, I'll be completing the vSphere 4.0 to 4.1 this coming Sunday. This will make 29 ESXi upgrades to ESXi 4.1 Update1. vCenter was upgraded to 4.1 Update1 and a month after Update 2 was released, we upgraded vCenter to Update 2. Update 2 resolved an issue with editing the settings on a template.

I did experience one issue in this upgrade, and that was that  vCenter stopped updating the performance data. This happenned about two weeks after upgrading vCenter Server from 4.1 Update 1 to 4.1 Update 2. And, and it took approximately one month to get it completely resolved. The KB's for this are on a previous post. For us, we had to re-create the views within the database before it was completely resolved. This was the first time I had experienced this issue.

We've already upgraded VMware Tools on three clusters, and will finish upgrading this within the next month or two after all the ESXi servers are upgraded.

NetApp Management Console 5.0

NetApp has release OnCommand 5.0, which replaces OnCommand System Manager 2 and VSC. This provides NetApp storage management, provisioning and cloning, and SMVI.

Provisioning and cloning and smvi was previously located in vCenter with VSC,  and has been moved to the OnCommand 5 suite.

If you currently use VSC, you will need to uninstall it and remove the Pluig-In. Uninstalling VSC is straight forward and is done by running the installation program for VSC.

Removing the Plug-in is not so obvious. You will need to go to the MOB (Managed Object Reference) page for your  vCenter server. To do this, point your browser to: http:// vCenter Server name/MOB
  1) Select "content"
  2) Select "extension manager"
  3) Select your NetApp plug-in from the list
  4) Click on the extension list, look at the properties and copy the key string
  5) Go back one page on the browser, and look for unregister Extension on the Methods table
  6) On the next screen, paste the copied string and click invoke Method, you will get the status as: "Method Invocation Result: void"
   7) close the window and log back into the vSphere Client.

Now that VSC is uninstalled and the Plug-in removed, you can finish installing the OnCommand 5.0 suite.

Quickest Way to Add NFS Storage

If you have three or more esx serves in a cluster with a couple dozen NFS datastores, it can be time consuming adding 24 datastores to one server or adding one or two new datastores to each server in the cluster.

I have found the quickest and easiest way to do this is using VMware’s Management Assistant - either the Linux or Windows flavor. To do this, run the following command from your Management Assistant CLI:
esxcfg-nas.pl - server 'esx server name' -username 'user' -a 'name of your new datastore' -o 'ip address or name of your NFS server' -s 'name and path of the nfs export'
for example: esxcfg-nas.pl -  myesxserver - username me -a mynewdatastore -o 1.2.3.4 -s /vol/newvol/newqtree

That's it.