Backup Exec And VMware ESX 4.1

A couple of days ago I had a sudden panic when all of a sudden my Virtual Machines began to shut down. You can only imagine how bad this can be when a fully redundant system, simply fails. As I began looking into the issue, I start to receive errors that some of my LUN’s on my brand new EVA had reached capacity.

Baffled as you can imagine I began to look at my usage in VMware and found that I still had approx 180 Gig available. When I opened up the datastore I noticed that my VM’s had a massive amount of snapshots that have piled up in the directories.

We use a utility called Backup Exec from Symantec to provide a backup utility for our entire production environment. One of the plugins for Backup Exec allows you to make backups of your VMDK’s. When BE makes backups of these files it makes a call to VMware and creates a snapshot of the VM at that moment. If for whatever reason the backup job fails, the snapshot is not deleted and it becomes the current working version of the VM. After several failed jobs this began to pile up for the VM’s and the LUN ran out of room.

Well as you remember I told you that when I looked at the LUN I had 180 Gig available. Well unknown to me that coupled with the fact that my LUN had run out of space but now VMware was not reporting my usage correctly. What this caused was a failure of the alarms triggering to tell me that my LUN utilization was getting high.

After calling VMware to assist me with getting my environment back online and clear out all my snapshots, I found the issue with the alarms not triggering. The recommendation was to edit your datastore alerts and make a change to it in some way so that when you click ok, VCenter server will reset the trigger and start to poll the actual datastore size at that moment.

After all is said and done I have learned and am now passing on to all of you. Always check your backups and make sure that there is no snapshots left behind that were not cleaned up. I have also learned to double check my usage stats and look more closely at VSphere client for anomalies.

Advertisement

Removing Ghost NIC

One issue I seem to run into more and more now that I have a VMWare virtual environment, is Ghost Nic’s. yes it can be a little scary, but easily resolved. So what is a Ghost Nic?. It is when a Nic card is removed from a VM or physical box prior to removing it’s settings. So when another Nic is installed it shows as Nic #2 , and windows will complain that the IP is already in use by another Nic if you are trying to reuse the IP from the old one. Here is the odd part, there is no other Nic at least not anymore so how could this be?. Well by removing a Nic before uninstalling it, the windows registry will continue to map to a Nic that is no longer present. It holds it’s settings in the registry so that if it is ever put back in, then all your settings will show back up. Each device in windows has a unique ID that identifies it within windows and this is what windows ties the settings to.

This seems to be more of an issue now because in VMWare all your devices are now virtual to the VM, and If anything happens to your VMWare tools or you need to P2V a machine it is very easy to forget to uninstall your Nic cards before re installing or virtualizing your physical box. Of course I always forget to do this. Now if it is a physical box, the easy fix is just pop in the old Nic and uninstall the drivers and settings by going to device manager >> uninstall device then you are all set. In VMWare, well you can’t pop in a virtual Nic and uninstall the drivers because every time you install a new Nic in a VM, it is given a new UUID. So I will show you how to take care of it another way.

First goto your command prompt and run this command

Then right click on My Computer >> click on Properties >> then goto Device Manager.

Once you are in Device Manager click on the View menu option, then select “Show Hidden Devices”. You should see your ghost Nic in the list of devices in your Device Manager window. At this point right click the device, and select uninstall. Now your Nic settings are freed up and you can assign your static IP along with your other settings without having any issues.

If you have any problems, questions, comments. Feel free to leave me a comment or email me.