26-03-15

Upgrading The Home Lab Part III : Upgrading VMware Tools and Virtual Hardware

We’ve arrived at the final part of our odyssey (a small odyssey in my case, but an odyssey none the less!) in our upgrade to vSphere 6.0. We’ve upgraded vCenter (relatively trouble free), ESXi (not so much, but that was down to my Jurassic era hardware) and now we have the small matter of the VMs left, to upgrade VMware Tools and virtual hardware to the latest versions.

This might seem like the easiest task of the lot, but actually in my experience this is the hardest part. Not so much from a technical level, but from the perspective of there being large numbers of VMs to touch, and of course in times of Change Management, getting agreement to down VMs to upgrade their virtual hardware can sometimes feel like rutting stags in a field. Although from vSphere 5.1 onwards, a Windows reboot for an upgrade of VMware Tools was eliminated, we still need to power off VMs in order to upgrade their virtual hardware.

29fc1f37d00bf7a74decf694dce0559225f242e6 “Barry, let me upgrade the virtual hardware on your Exchange Server!..” “…No! Bugger off, Maurice! I can’t have 5 minutes downtime!..”

Thankfully, VUM can come to our rescue again. When it’s installed, it creates some default patch baselines. Two of which include baselines for upgrading virtual hardware and VMware Tools. You can see these by clicking on the VMware Update Manager button in the Home view in the vSphere Client. You need to click on the “Baselines and Groups” tab and then on the “VMs/VAs” button. You should see the following in your VUM screen:-

vum-vm-baselines

There is also an upgrade path for virtual appliances you can see at the bottom, we’re not going to cover that here as usually appliances are in the minority. VMs are what we’re looking at here. In order get VUM to bring our VMs up to date, we need to create a couple of Baseline Groups, or we can just use a single group if we want to consolidate both upgrades into a single operation, which is what I’ll be doing. We can do this from the same screen as above, in the right hand pane. Click on the “Create” button to start the Baseline Group as shown below:-

create-baseline-group

This starts the Baseline Group creation wizard, which only really has a couple of steps to set up, nothing too tricky. Give the Baseline Group a name, as below. And no, don’t use one of the Bee Gees like I did with the stag picture:-

baseline-1

Click Next and as we’re only upgrading VMware Tools and virtual hardware, we’re going to leave virtual appliances alone. We are going to create VM upgrades, so tick the radio buttons next to the following groups:-

  • VM Hardware Upgrade to match host (predefined)
  • VMware Tools Upgrade to match host (predefined)

This is shown below:-

baseline-types

Click Next..review the settings and click Finish and you’ll see the following screen:-

baseline-group-complete

So now we have our baseline groups created, we now need some VMs to attach them to. As I’ve said countless times before, this is a test environment, so I don’t suffer from the same constraints as a production system. That’s another way of saying “if something explodes, I don’t care”, but that being said, I do want to stage these updates to make sure everything works as I expect before I push the baseline group to a wider audience.

I am not going to update any virtual appliances as I mentioned previously, and I have no VMs right now that are Linux based. Rather than pushing out the baseline to all Windows VMs, I’m going to stage them by folder. First up is my seldom used Windows Cluster folder. This has two Windows Server 2012 R2 nodes and an iSCSI target also running Windows 2012 R2. As I hardly use this cluster, it spends most of it’s life powered off, meaning it’s a good place to test my rolling VM updates.

So to start with, if you haven’t already, create a folder and move the VMs you want to update into this folder (hint: you’ll need to be in the “VMs and Templates” view in vSphere Client to do this). Once you’ve done this, you can add the baseline group to the folder by clicking the Update Manager tab and clicking the Attach.. button. You’ll then see the dialog below:-

attach-baseline-to-folder

As you can see, I already ticked the box to add the Baseline Group to the folder. The sharper eyed readers amongst others will notice I could have done this without creating a baseline group first, but I think my way is neater 😉

Click Attach and then you will need to perform a Scan.. just as we did with the hosts. In fact, it’s exactly the same process. Remember at this stage, we don’t care about virtual appliance updates, so make sure you untick that box and tick the other boxes for VMware Tools and virtual hardware, as below:-

confirm-tools-scan

The scan results are in, and lo and behold I’m not compliant:-

vm-scan

In which case, I need to hit the Remediate.. button to apply both sets of upgrades, just like we did with the hosts. This starts an upgrade wizard, as shown below:-

remediate-vm-1

On clicking Next.. the next step is to schedule when we want the upgrades to occur. Like I said, these boxes are my guinea pigs as they are hardly ever powered on, so I can go ahead and do it immediately. In the production world, you’d probably have to do this out of hours or whenever your maintenance windows are:-

remediate-vm-2

Give the task a name and description as shown above, and decide when you want the process to run. The scheduled intervals are applied via powered on machines, powered off machines and suspended virtual machines. By default, Immediately is set for all cases. Take care here!

One really useful feature of using VUM to upgrade VMs is the ability to create snapshots ahead of the actual upgrade processes. This is very handy on the off chance that something goes badly pear shaped. There’s no reason it should, but it’s always nice to have a safety net, isn’t it? And you are creating full offline backups, aren’t you?

remediate-vm-3

So as you can see above, I’m keeping the snapshot for 24 hours (default is 18, for some reason). You can keep them forever if you like, but if there are a lot of VMs to be upgraded, this could swallow a lot of expensive storage in a busy environment very quickly. I just want to make sure the VM boots and reports back in as up to date once the process is complete. 24 hours is plenty of time for me to validate the update hasn’t eaten my VM. As these particular VMs are already powered off, no need for me to select Take a snapshot of the memory for the virtual machine. This requires a running instance of VMware Tools and can add a lot of time to the process, so use sparingly.

Time for one last sanity check and then hit Finish if you’re happy:-

remediate-vm-4

You can then monitor the upgrade task in the tasks pane at the bottom of the screen, as below (click to expand):-

remediate-vm-5

Once the upgrade task completes (and this could take a while, so go and make a coffee or something), you should see a fully compliant bunch of VMs. If you don’t, you can use the Tasks/Events window (Events mainly) to help troubleshoot what went wrong. The law of averages says that a couple of VMs out of dozens will need some minor hand holding. To get through them all without issues is pretty much unheard of, so don’t worry. As you can see below from the Events window, the upgrade process is ongoing:-

vum-progress

And then after a little while of VUM whirring away in the background, skidoosh! We have 100% compliance!

remediate-vms-100-percent

Don’t believe me? Here’s what one of the VMs says..

vm-status

We’re on version 11 virtual hardware (ESXi 6.0 compatible) and VMware Tools are current. All done by VUM in the background. Multiply that by a few dozen VMs and you’ve got a nice time saver there! I also wanted to show that the pre-upgrade snapshot is available for us, on the off chance something went septic:-

vm-snapshot

As you can see, VUM even puts in a useful description so we know what the snapshot is, when it was created and when it will be deleted (if applicable).

Conclusion

Upgrading VMs can often be the trickiest part of the upgrade process as there can be hundreds or thousands of objects to be updated. However, VUM can make this process pretty painless by automating the upgrades and scheduling them for a time that suits you. Don’t be like Barry and Maurice at the top of the article – get a maintenance window with the VM owner and get VUM to do all of the heavy lifting for you.

 

25-03-15

Upgrading the home lab Part II : ESXi hosts

In Part I of the “Upgrading the home lab” series, we migrated/upgraded the vCenter appliance from version 5.5 to 6.0. That all seemed to go pretty well, so the next major step on the road to vSphere 6.0 is to upgrade the ESXi hosts in the environment to ESXi 6.0. Just before we get to that, we’ve actually missed a step out. Once vCenter has been upgraded to version 6.0, you should take a few minutes to upgrade VMware Update Manager (VUM) to version 6.0 too. In my case I hadn’t got around to building a 5.5 VUM server, so I just built one out from scratch with the vSphere 6.0 installer DVD. I just did a simple install and used the SQL Server 2012 Express version for the database, as I’m just managing a single host. For 5 or more hosts, you should go and get the “full fat” SQL Server.

What’s new with VUM?

Not really a lot as far as I can see. It still requires a Windows Server (minimum 2008, but 2012 R2 should be your aim these days), still requires a SQL database (see above) and still requires the vSphere Client (not the Web Client) to perform any kind of meaningful management. In that respect, it doesn’t look much different than it did in 5.5 days. You can read what’s new in the VUM 6.0 documentation, but it seems to be more database support than anything to get excited about (like baking it into the appliance, for example).

Upgrading the ESXi host(s)

To upgrade your hosts, there are a couple of different ways you can do it. You can boot from the DVD (or remotely attach an ISO image if you have an iLO/DRAC card etc.) and perform an in-place upgrade, you can use VUM to upgrade your hosts, or you can boot from DVD/ISO and perform a fresh installation. It depends what you want to achieve in the process, obviously you want a quick and supported way of getting your hosts up to date, and VUM is VMware’s recommended method.

However, in most enterprise environments, ESXi hosts are commodity items – by this I mean all VM data (and even ISOs) are stored on shared datastores on SAN/NAS etc. In this case, you can achieve a “clean slate” installation by using the installation DVD to perform a fresh installation with the original addressing information. Consider the use of host profiles to “backup” the host configuration before you start (requires Enterprise Plus licencing).

You can also use scripted upgrades, using Auto Deploy or the esxcli command, see here for further information on supported methods. I’m lazy, so I’m using VUM.

Using VUM to upgrade your hosts

As noted above, VUM is the recommended method of upgrading hosts to the latest version of ESXi. In terms of supported prior versions of ESXi, if you’re version 5.x or above, you’re pretty much in clover. Anything older than that is basically a fresh new installation. That’s not all bad, depending of course on how many hosts you have to get through. Remember to check the VMware HCL to ensure your host hardware is supported with ESXi 6.0 and if you can, obtain the custom vendor ISO for ESXi for the best level of driver support and functionality. At the time of writing however, I was only able to find the HP version of the custom ISO (as you can see below), so I will have to use the GA ISO to upgrade my PowerEdge. Hurry up, Dell!

esxi-downloads

Once you have obtained the ESXi 6.0 ISO, ensure your VUM plugin in the vSphere Client is installed and enabled (and one step I haven’t specifically called out is to ensure you upgrade your vSphere Client to version 6.0 before you start this part. Reports of it’s demise have been somewhat premature!).  You can check this by going to the Plugins menu and selecting Manage Plugins.., you should see something similar to the following:-

plugin-manager

If you have a prior version of the VUM plugin installed or you don’t have the plugin installed, you will have to select the “Download and install” option. This runs a brief installer and does not require a reboot nor a restart of the vSphere Client. If the installation has been successful, you’ll see the plugin enabled in the Plugin Manager and you’ll also have a button on the home screen and an extra tab on the host view.

The next step is to upload our ESXi ISO into the VUM repository and create a patch baseline. To do this, you need to go to the Home view in the vSphere Client and then click on the VUM button in the Solutions and Applications section, as shown below:-

vum-home

This button takes you into the VUM management view and from here we need the ESXi Images tab, as shown below:-

esxi-images

And then from there, click on “Import ESXi Image” as shown above. Browse to and select the ESXi 6.0 ISO you downloaded, click next to start the import process and you should see the following progress dialog. This only usually takes a couple of minutes or so.

iso-upload

If the import has been successful, you’ll see the following dialog:-

import-successful

We now need to create an upgrade patch baseline from this ISO so we can add it to our hosts to be upgraded. Leave the “create a baseline” option ticked and give it a meaningful name, as shown below:-

baseline-name

And click “Finish”. All being well, you should now have both the ISO imported and the baseline created, as shown below:-

baseline-iso

So now we have imported our ISO and we have created a baseline. Now we need to associate this baseline with an object to be upgraded. We basically have three choices here – we can apply the baseline at datacenter level, cluster level or we can apply the baseline at individual host level. I’m going to go for the first option, just so I can call out some differences between the options. To apply the baseline to the datacenter object, select the datacenter object in the vSphere Client, select the Update Manager tab and click the Attach.. button on the far right, as shown below:-

attach-baseline

As you can see, my datacenter has no baselines already attached. In the “Attach baseline or Group” dialog, you should see the upgrade baseline we created earlier. Tick the box and tick “Attach” as shown below:-

attach-baseline-group

Once you have attached the patch baseline to the datacenter object, the view in Update Manager should change. You will see the hosts added and a 0% compliance report. This is because we haven’t yet run a scan against the host to check what version of ESXi already exists and if the host is compatible with the ESXi 6.0 upgrade. Next, select your hosts and click the “Scan..” button in the top right.

vum-scan

In our case we just want to scan against upgrade baselines, so be sure to tick this box in the “Confirm Scan” dialog:-

confirm-scan

Click the “Scan” button and VUM will go off and query each host in turn for their compliance against the ESXi 6.0 upgrade baseline we created. This should only take a couple of minutes per host. Once the scan is complete, you should see new information in the VUM tab. In my case, my host as come back as “Incompatible”, which doesn’t surprise me in the least as this host hardware is prehistoric by any measure. However, I can still force the upgrade to run if I know the installer will complete successfully. This isn’t strictly supported by VMware, but all this basically means is that only current generations of servers are tested by VMware and their partners for HCL purposes. To recertify every piece of server hardware for each new release of ESXi does not make sense. This does not however mean that your server can’t run ESXi 6.0, I suggest you test it on some development kit first before moving forward. In my lab, I don’t care!

vum-incompatible

As you can see in the above graphic, my host is older than Bruce Forsyth and as such comes back as non compliant in VUM. No surprises there. In order to force this upgrade through, I can hit the “Remediate” button to force the upgrade to start. I have seen in the field some HCL certified kit come back as incomplete, so sometimes you do need to know how to do this to get the upgrade done. This in turn starts a 6 step wizard to push the upgrade down to the host via VUM. First up, we need to select which hosts and which baseline to use, as below:-

remediate-1

Then we thoroughly read and agree to the software EULA:-

remediate-2

The next step is “signing the death warrant”. If this goes toes up, that’s down to you! Check the box to ignore warnings and in my case, hope my offline backups are good!

remediate-3

Then we give the task a name and description (you can call it anything you like, really) and schedule when this upgrade should be done. I’m going to do it immediately, because I just can’t wait for ESXi 6.0 goodness!

remediate-4

In the final configuration step, I need to tell VUM what to do if there are running VMs on the host to be upgraded. Normally you wouldn’t change anything here as really you should already have your host in maintenance mode before you target it with the upgrade. As my vCenter appliance is on the host to be upgraded, I need to be slightly more creative and get vCenter to power VMs off.

remediate-5

Then one last sanity check before we hit the chicken switch…

remediate-6

And off we went. However, big problems lay ahead. VUM spat out my upgrade saying the CPU in my host was not compatible. OK, fair enough. I did buy it from Fred Flintstone! What I did instead was to burn the ESXi 6.0 ISO to CD and boot it off the physical DVD drive in the host. This way I can basically tell the installer I don’t care about compatibility and support issues, I’m going to bear the risk of it all turning to toast.

First off, I booted from the CD and as the files were copying (black screen with yellow thin progress bar at the top), I got the error “Error loading /s.v00 Fatal error: 6 (Buffer too small)” and the whole thing just stopped. I didn’t get it – the MD5 matched the VMware download site! I downloaded the ISO again, but this time I performed a “direct” download rather than using the Download Manager. The MD5 matched again, I burned a new CD and this time it all worked just fine.

Even though the installer complained bitterly about the host CPU, CPU virtualisation modes and a PCI device not being supported, it all seems to work just fine. The host booted and my vCenter appliance auto started as usual. For completeness, time to go back into VUM and validate my upgrade. To do this, select the host, go to the Update Manager tab and select “Scan” again, as previously. This should take just a minute or so, and then we get what we were hoping for. Green!

vum-green

So now we have vCenter at 6.0 and ESXi at 6.0. Not without a few niggles, but that’s just a consequence of using such old hardware. The moral of the story for me is that it’s high time I gave my boxen an overhaul. Finally, as a last piece of housekeeping, I’m going to validate the status of my vSphere Client plug-ins :-

plugin-error

It seems everyone is happy except Mr Auto Deploy. I don’t use Auto Deploy in my lab, but red errors kind of piss me off anyway. I performed a quick Google and found a blog post by Kyle Gleed that tells you how to fix this. You simply start the Auto Deploy service on the appliance as it’s disabled by default. God bless the internet. However, Kyle’s instructions reference a management interface to the appliance which is no longer used in 6.0. In order to configure appliance based services, you must login to the Web Client as an administrator and enable it from there.

In the main Web Client home screen, click on the “Administration” button on the left and navigate down to Deployment/System Configuration as shown below:-

deploy-sysconfig

Then click “Services”..

services

And then right click on “Auto Deploy”.. and select “Edit Startup Type”..

rightclick-autodeploy

 

Select the Startup Type, depending on how you want the service to start on appliance boot – I’m choosing “Automatic“and click OK..

startup-type

And then manually start the service by right clicking again on “Auto Deploy” and selecting “Start”..

autodeploy-start

I then go back into Plugin Manager, enable the Auto Deploy plugin (accepting any certificate warnings) and we should be free of errors!

plugin-manager-fixed

As a side effect, we also have a nice button added to the home screen:-

autodeploy-button

Conclusion

So there we go, we now have an upgraded vCenter Server and ESXi host. I wouldn’t advise forcing the upgrade through the way I did unless you have nothing to lose – i.e. it’s a test lab or something non production. If the host hardware you are installing to is HCL listed, you should whizz through a VUM focused upgrade just fine.

If you get any odd errors in the Web Client, try deleting your browser cache and also the Flash Player cache from Control Panel. There could be something caught up in there from 5.x days.

Next stop is upgrades to the virtual machines – virtual hardware and VMware Tools. That is to come in Part III !

 

23-03-15

Upgrading the home lab Part I : vCenter Server

vSphere 6.0 has finally shipped, so I decided to take the plunge and upgrade the home lab to vSphere 6.0. In the next couple of posts, I’ll outline the steps required to perform the upgrade, plus any issues I encountered along the way. I think most people know that most articles I write are focused around VDI, so let me express this straight out of the gate. vSphere 6.0 does not support Horizon View until release 6.1. So basically, don’t upgrade any environments to vSphere 6.0 where View is in use, stuff will probably break. When is View 6.1 out? I don’t have a clue right now, but seeing as the release notes have been posted up, I can’t imagine it’s too far away.

If you’re project planning and you need to have certified upgrade paths (I know some project documentation requires this in some companies), the current (at the time of writing) interoperability matrix result is shown below:-

Compatibility Matrix

 

So my home lab is a very simple affair indeed. It comprises a single host which is a Dell PowerEdge 1435SC with 32GB RAM and two Opteron processors. Old hat I know, but it gets the job done. For those wondering how I deal with power management on such old kit, it’s simple. I turn the bugger off when I’m not using it! As I’m often on the road, I don’t see a lot of value having the beast humming away in the attic when I’m not around to use it.

Anyway, that aside, it’s currently on ESXi 5.5 U2 and runs the vCenter Server Appliance. I chose the appliance because it’s quick and simple, and I don’t have to faff around with Windows licenses. I know Linux quite well, so I don’t have any fear knocking around on the command line. In vSphere 6.0, the back end architecture of vCenter changes somewhat. If you recall, in vCenter 4.x and 5.0, everything was pretty much just baked into one installer. In vSphere 5.1 and 5.5, Single Sign On was broken out (and made good in 5.5!), as was the Inventory Service to provide a more modular environment should you wish to break things out a little for scalability and to mitigate the “all eggs in one basket” factor.

Further to that, vCenter 6.0 now has the concept of the “Platform Controller”.  Put simply, the Platform Controller are infrastructure or access services such as SSO, License Service, Lookup Service and the Certificate Authority. vCenter Server is basically everything else, so Inventory Service, PostgreSQL, Web Client, Dump Collector, et al. For my simple purposes, I’m just going to upgrade from my current vCenter 5.5 appliance to the 6.0 appliance, I don’t really need to start making the design overly complex. In fact, because it is just a lab environment, I’m not massively bothered if the upgrade process blows up, I can just rebuild from scratch. Obviously in a live environment, I’d be slightly more circumspect!

One important caveat to note is in the VMware documentation:-

You cannot switch the models after deployment, which means that after you deploy vCenter Server with an embedded Platform Services Controller, you cannot switch to vCenter Server with an external Platform Services Controller, and the reverse.

For full information of the pros and cons of either method, please refer to the product documentation. I’m not going to go into that level of detail here. What is reassuring for me with my one host and a dozen VMs is the following comment in the documentation:-

vCenter Server with an embedded Platform Services Controller is suitable for most environments.

Cool, I feel kind of validated now. I couldn’t see at first blush how the sizing of your environment effects your design decision, I suspect it’s more to do with geographical constraints, multiple vCenters and other VMware products that integrate with it, such as vRealise Automation. More on that in the future once I understand it better!

The Appliance Upgrade Process

The process of upgrading your vCenter appliance from 5.x to 6.0 is actually more of a migration than an upgrade. In essence what you’re doing is spinning up a new appliance as 6.0 with temporary IP address information, using SSH to copy over the database from the source 5.x appliance (and historical data if you so choose), changing the target appliance’s IP address to the source’s address and then dropping the source 5.x appliance.

Meeting Prerequisites

As you might expect, there are several prerequisites to be met before actually copying over any files or making any changes. First and foremost – have a backup and no, a snapshot is not a backup! By all means take a snapshot of your vCenter Server prior to starting the process, but have a block based backup too, whether that’s via Veeam or any other backup method. Don’t rely on snapshots. If you do, you’re upgrade deserves to fail!

Again the product documentation is the best place to refer to, as I’m sure over time these will change as experiences come back from the field when the product is being deployed. Once pre-requisites have been met, we’re hot to trot and ready to install our vCenter Server 6.0 appliance.

Download the appliance ISO file from MyVMware (at the time of writing this is VMware-VCSA-all-6.0.0-2562643.iso) and you may notice from the off we’re not downloading an OVA/OVF as we did previously. This is because the upgrade method is slightly different. Instead we’re going to take the ISO and mount it locally on our Windows machine (if you’re on Windows 8.1, you can right click and select Mount to mount the ISO to your DVD drive) as shown below:-

mount-vcsa

 

Alternatively, you can right click and extract with 7-Zip (for example) and create a source directory of files. However, for my purposes, I’m going to keep it simple and mount the ISO in Windows. Once mounted, we need to navigate to the DVD drive and go to the \vcsa folder. In here is the installer for the Client Integration Plugin, which we will need for this process. As a good habit, don’t forget to right click the installer and select “Run as Administrator”, as shown below:-

client-runasadmin

You’ll need to close any running browsers as plugins are installed, and then step through the simple installer, which should take just a minute or so. Once this install is complete, no reboot should be required and we can jump back into the root of the installation DVD and run the main installer vcsa-setup.html. I ran this with IE, I don’t know how well it works with other browsers. You will need the browser plugin access to start the installer, so click “Allow” (if you’re using IE):-

allow_installer

You should then be greeted with the option to install or upgrade. If you don’t see this screen, go back and check your client integration plugin installation (a reboot may help):-

install

Time for another quick sanity check at this point that the pre-requisites have truly been met. In my case I’m running a 5.5 version of the virtual appliance, as shown below:-

old-vc

 

so I’m good to go:-

sanity-check

Now to the meat and potatoes of the installer itself, and a nine part wizard now has to be negotiated in order to kick the upgrade process off. First up is the usual EULA. I read this thoroughly (twice!) and consider myself happy with it and click “I accept..” and click Next:-

part1-eula

Next I need to input details of the ESXi host to which I’d like to push out the new virtual appliance. Note the caveats listed at the bottom of this step – disable lockdown or maintenance mode and if you’re using a VDS, initial deployment of the appliance must be to an ephemeral port group. Click Next:-

part2-esxi

Accept the certificate warning by clicking Yes in order to continue:-

part2-cert

Your host credentials will then be validated as shown below:-

part2-creds

Step 3 is setting up the appliance name and SSH options. I’m calling mine the highly imaginative and original name of “vCenter” and I’m turning SSH on because it’s a lab and I’m not massively fussed about security. In the real world, leave this disabled unless you specifically need it. Click Next to proceed.

part3-vm

Step 4 is configuring the source information for the migration, so your existing vCenter Server. In this screen you need to enter IP/DNS details of the current appliance, version number, your administrator@vsphere.local SSO password (and if you’re a buffoon like me and forgot what the password was, you can reset it using this KB!), the root password for the appliance and host information along similar lines. You can optionally choose to migrate performance and other historical data. I’m going to enable this option, and I can’t think of any great reason in the real world why you wouldn’t do this (unless your database is mah-oosive). Before you proceed, check the caveats at the bottom of the page – that lockdown mode and maintenance mode is disabled and the source appliance’s DRS setting is disabled so it doesn’t move off that source host part way through. Click Next:-

part4-source

Once the credentials have been accepted, take care to check the information presented to you on what happens post upgrade. The new appliance will inherit the old appliance’s IP address. Also make sure no firewalls prevent SSH access between both appliances as this is the transfer method used during the migration/upgrade. The installer will start the SSH process will be started on the source appliance if it is not already running. Click Yes:-

part4-warning

In step 5, we have the choice of choosing what size appliance we’d like. This is pretty similar to the Windows based installation and helps determine JVM heap sizes are set appropriately, amongst other things. I feel quite inadequate choosing the “Tiny” option, but it is what it is and I have no requirement to go any bigger than that. Size clearly is everything in this case. Make your selection as appropriate and click Next:-

step5-size

Step 6 is choosing which datastore you’d like the appliance to go on. I’m going to choose the one with the most available space and I’m also going to thin provision it. This appliance won’t be doing a great deal in my lab and the datastore isn’t likely to fill up quickly, so I have no real need to thick provision here. Click Next:-

step6-datastore

Step 7 is the creation of some temporary network details for the new appliance as this is a migration really as opposed to an in place upgrade. In this step, we should pick the same port group as the source appliance and use the same subnet as well if possible, especially if the data migration is going to be large. My database is small and it’s a single host, so speed shouldn’t be an issue. Fill out the details appropriate to your environment and click Next:-

step7-network

Step 8 is the “Ready to complete” or final sanity check. Review the information presented and check for typos, wrong IP addresses, what size appliance you need (sometimes “Tiny” just ain’t enough!) and when you’re happy, click Finish:-

step8-ready

All being well, the process should start by initiating the appliance deployment to your ESXi host as below:-

Deploy

You can also monitor what is going on by connecting to the ESXi host with your vSphere Client and clicking on the “Events” tab of your target appliance, you should see something similar to below:-

transfer

And some while later, I was greeted with the following screen. The process took around an hour, but bear in mind this is a “tiny” and simple deployment. Larger enterprises should budget a lot more time for this process when migrating between appliances:-

finished

I double checked my “old” appliance has been powered off, and the new appliance is up with the original appliance’s identity:-

appliance-screen

Once the install is completed, you may like to perform some housekeeping before you put the appliance into production. By default, my appliance came up with the DNS name localhostI don’t really want that, so I quickly logged into the appliance console to change it. Something you may notice that’s new is that the vCenter appliance has now been set to behave just like an ESXi host, so once you press F2 to customise the system and entered the root password, the menu structure should be something that’s pretty familiar:-

appliance-menu

I like that VMware use a different colour scheme on the appliance to save any confusion with connecting with an ESXi host. Even though you can see it’s the appliance at the bottom of the screen, with many screens open it may help prevent costly mistakes! To go back to the original housekeeping, go into Configure Management Network and then DNS Configuration. Input the appropriate values for your appliance, as shown below:-

new-dns

I also like to disable IPv6, though there is a mixed bag of opinion on this. I say if you don’t use it, don’t enable it. However, this is a subjective thing and purely optional. To disable IPv6, go into Configure Management Network and then into IPv6 Configuration. To disable it, hit the space bar to uncheck the box as shown below and restart your appliance.

disable-ipv6

Once rebooted, you can see we’re up to vCenter 6.0!

about-vsphere6

Post upgrade issues

I have only really come across two issues so far – firstly I got an “Error #1009” which I cleared by deleting cookies etc from my web browser (and also upgrading the Flash Player in Firefox to the latest version).

As you can also see from the above screen shot, I was having issues with the Client Integration Plug-In. It was definitely installed from when we started the migration process, and all three browsers I had reported the issue (IE11, Firefox, Chrome) so I uninstalled the plug-in from Add/Remove Programs, rebooted, downloaded the plug-in again from the Web Client login page, installed it and as you can see below, all was good:-

integation-enabled

Conclusion

In conclusion, I’d say well done to VMware for streamlining the upgrade process for the vCenter appliance. Yes, it has a couple of quirks and yes you should ensure all pre-reqs are met, but by and large I was pretty impressed with the whole process. Next up, my ESXi host….!

 

16-02-15

Elite Implementer Status : A Few Thoughts

Cert_Roadmap_2015Q1_v5_final_WEB

 

(Image taken from vmware.com)

There is a lively thread going on over at LinkedIn regarding the new VCx 6.x tracks that I felt compelled to jot down a few thoughts on. Firstly, once the new track becomes live, the VCAP level certs will be renamed to VCIX (VMware Certified Implementation Expert) and will require two exams as before. One for administration and one for design. So far, so good. Two exams as before, presumably of similar lengths as the VCAPs now with the same core set of skills being measured. However, instead of having two certifications to your name (VCAP-DCx and VCAP-DTx), you’ll have one. Fine, I suppose it makes sense and I don’t have a problem with that.

Now comes the interesting bit – “Elite Implementer status will be granted for candidates who complete multiple VCIX certifications”. I’m glad VMware have recognised the amount of effort and skill required to complete multiple Advanced tracks, however these exams aren’t yet live (I’m guessing it will be  around VMworld time before we see them in the wild) and there are a lot of people out there whose VCAP certifications are current and have completed multiple tracks.

In my opinion, there is no reason why VMware cannot enact this change right now. It costs them nothing and provides recognition to those who have spent a minimum of around 12/14 hours sitting these tough VCAP exams and getting through them. Think about it. Yes, we’d all like to be VCDXs, but the crushing reality is often that this certification requires a level of commitment way over and above anything I’ve seen from any other certification. I simply don’t have the time and energy to commit to around 100-150 hours on putting together a design and submitting it to VMware and then defending it in front of a panel, much as I’d love to.

The VCAP exams are tough, make no mistake. Not only do you need to have “operational” experience with all the respective products, but you also need to have a good understanding of the overlapping ecosystem – such things as third party solutions, Active Directory, Group Policy, storage, networking and more. Anyone with a VCAP cert has been through the mill to get it and deserves a pat on the back. To have both design and administration certs for multiple different VMware technologies elevates you to another level still.

So in short, come on VMware, recognise your multi-track vRockstars now and give them Elite Implementer status. It’s a small gesture that will go a long way and keep existing holders motivated for when the 6.x track comes on line. For more information on the 2105 track announcements, please visit MyLearn.

Comments and opinions are welcome, maybe with enough weight we can make it happen!

 

18-11-14

download

UK VMUG Event Review

Yesterday I had the pleasure of attending the fourth UK-VMUG annual conference at the National Motorcycle Museum in Solihull. For those that didn’t make it, I’ve put together an event review for your viewing pleasure. Apologies for the crapness of the pictures, taken with my phone unfortunately!

 

Joe Baguley Keynote

IMG_20141118_091319

After a brief introduction from VMUG leader Alaric Davies, the day started with the now usual keynote from Joe Baguley, CTO for EMEA. This year the keynote was entitled “Rant as a Service” and after setting the scene for around 30 minutes, the key message is still around software defined enterprise. It was my interpretation that there was a small pop at a hyper converged company whose name may or may not contain nuts on the basis that EVO:Rail and EVO:Rack can give you the same level of support and performance without having to buy into a single vendor. I’ve been feeling for a while that there isn’t a lot of love lost between the two parties, and I don’t know if that’s true, but I don’t find it particularly helpful when constant implied barbs are being traded. Just my opinion!

The point of EVO:Rail is to have the infrastructure up and running within 15 minutes. The value here is that you can go to 8 partners and pick which stack and value add you want. It’s not a single vendor lock in as such, as most customers already have an existing relationship with the likes of Dell, etc. EVO:Rail is 2U in size and has four blades installed. Not dissimilar to Nutanix and UCS in that respect, though of course the UCS chassis has a larger form factor. For larger installations or special use cases such as VDI with NVIDIA graphics, the bigger EVO:Rack will be required.

One interesting line was the ongoing idea now that abstraction and obfuscation takes place in as much as key components such as disks and raid controllers are being replaced by software and public and hybrid cloud solutions. This of course is becoming transparent to the “end user” as we move towards a hybrid cloud type of world. If a disk controller fails, it’s OK, software can take care of that. Lose a data centre? That’s OK too, we’ll just move to another one in the background. I’m not sure we’re totally there with that one yet, but it’s an interesting concept none the less.

Then we had the discussion about what is “Enterprise Scale” these days? As consumer electronics demands increase exponentially (photo uploads, data requirements, data production, etc) then most things these days really are “Enterprise” grade as they have millions of people using them daily, not just the tens, hundreds or even thousands in an “enterprise” environment.

White boxes are also now are taking the place of large monolithic proprietary solutions. EVO:Rail again was mentioned as an example of this, where you get a pre-built, predictable and validated vSphere environment from whichever hardware vendor you prefer. The irony that you’re still locked into VMware technologies was missed at this point, but I think I see where the point is here.

IMG_20141118_091425

VMworld Update – Julian Wood

I then went to the VMworld Update session with Julian Wood. One thing I’d have to say is that there was too much to fit in for 40 minutes. That’s not Julian’s fault – as he noted, once you look into it, there are so many product releases, updates and acquisitions to keep track of that you could spend all day talking about it! There was some discussion around the vRealise suite (I’m not spelling it with a “z”!),  what that means and how there is on and off premises solutions for that now. vRealise is essentially the management and automation tools bundled into a suite, so products such as vCloud Automation Center, Log Insight, vCenter Operations Manager etc.

CloudVolumes was also discussed, where applications are installed to a VMDK and then this VMDK is connected and presented to desktops in a fraction of the time it takes to do ThinApp etc. As I was listening to this, I started to think “what are the storage requirements though?  Read intensive, or are blocks cached.  How does this work?”. “Do we require any back end infrastructure such as MS-SQL etc.?”

On the EUC side, big strides continue to be made and VMware are really competing with Citrix in the application presentation stakes, as well as adding further improvements to the core View product, including Cloud Pods (or Linked Mode for View, as I like to call it!), where you can break the current scalability limits but also provide an additional site failover for virtual desktops if required, plugging one gap in the previous product set.

vSphere Futures – Duncan Epping

The next session was with Duncan Epping. His sessions are always well attended as he’s usually on the bleeding edge of what the company is doing internally, plus I’ve found him to be pretty honest in his responses to some issues that have cropped up, especially around Virtual SAN. I made quite a few notes around what was discussed, and it’s probably easier to break them down into bullet points:-

  • All flash Virtual SAN coming, to increase the configuration options for two slot blades, where currently you need flash for cache and spinning disk for content
  • Virtual volumes (VVols) policies coming that will based per VM
  • This functionality will be based on an array that supports virtual volumes
  • IO filters directly in the hypervisor for those arrays not VVol aware
  • Storage DRS VM IOPS reservations, so we can migrate workloads to other storage if reservations are not met
  • Storage DRS has better awareness of thin provisioning, dedupe and replication
  • Resource and Availability Service is a new web based tool that uses exported DRS settings to simulate failure of resources and ensure design is correct, validating such things as Admission Control settings
  • FT support for up to 4 vCPU
  • No more vLockstep or shared VMDK for Fault Tolerance,  10Gbps networking will be required
  • The ability to vMotion “anywhere”, requirement is that both vCenters must be in same SSO domain
  • vMotion has a 10ms latency tolerance now, working on 100ms tolerance for long distances
  • The vCenter Appliance will scale as well as Windows version now, and will be the future of vCenter releases
  • SQL server supported externally for the vCenter Appliance
  • Task pane will be coming into bottom of Web Client
  • Less nested right click options to make the Web Client interface cleaner
  • Task  concurrency, performance  charts and other features will be introduced into the Web Client
  • Linked Mode will be available for the vCenter Appliance
  • Content library for ISOs etc, replicated across sites. Also includes templates, OVFs etc. Same as Hyper-V Libraries, by the sounds of it

IMG_20141118_105540

One very interesting thread was around Project Fargo. This in essence is a “re-imagining” of the snapshot process and will allow for the creation of Windows virtual machines in around 2 seconds. In the lab, Linux VMs were spun up in less than that, the overhead on the Windows side was mainly down to customisation and joining AD etc. Another way of thinking about it is “Linked Clones on steroids” in the sense that you have a parent virtual machine and lots of child virtual machines. Duncan’s blog entry as linked above goes into some good detail on what you can expect from this initiative.

Horizon View Architecture and Design – Barry Coombs & Peter Von Oven

I then went to the session by Barry Coombs and Peter Von Oven about Horizon View Design and Architecture. This session wasn’t really a “death by PowerPoint” session, but more a key points and brief discussion at a high level as to what you should be looking for in a good Horizon View design. There are always little nuggets or anecdotes that can be useful that maybe you haven’t come across before that only really come out of experience. One good point from this session was that you should never let the IT team speak on behalf of the end users, so in other words, don’t assume IT know necessarily what the user experience is like, because they can’t know every individual use case.

The key point of performing a desktop assessment phase and also a proof of concept was also re-iterated, and I can’t agree with this enough. To chat to IT and some end users is not enough. It’s useful as part of the whole engagement, but you also need key performance metrics and also a proof of concept to see what works and what doesn’t work. Think of a PoC as the first draft of a document that requires lots of iterations to get it “just right”. To perform a desktop assessment and some stakeholder interviews and then think you can roll an effective VDI environment first time out of the gate is total fantasy.

Any VDI deployment (whether it’s View or AN Other solution) should be an improvement on the current “physical” end user experience. Again this is a given. If you’re spending time and money replacing a solution people are familiar with and comfortable with, it needs to be visually an improvement on what they already have, or the solution will simply acquire a “bad name”. One interesting idea was the notion of having a “Departmental Champion” – an end user who wants to positively influence the outcome of the project. They can interface with other users and help cascade information and feedback backwards and forwards. This can give you a view inside the PoC that you would not normally have.

Some other brief points included not forgetting to factor in VM and graphics overhead when right sizing a solution, these are commonly forgotten about (guilty!) and user concurrency should be measured in advance. Generally I use the rule of thumb of 80% concurrency, but in an organisation that has shift patterns, this may not be appropriate. Make sure the solution scale!

IMG_20141118_131653

EUC Update – Peter Von Oven

My next session was another EUC session, this time with Peter Von Oven from VMware. Again, a lot of key messages came out pretty thick and fast, so a bullet point summary is included below:-

  • VMware’s strategy is still the three pillar strategy of SDDC,  EUC and Hybrid Cloud
  • AppVolumes (formerly known as CloudVolumes) will be available in December
  • Horizon Workspace can disable icons based on physical location. It’s context aware in that sense. So for example, R&D portal is not accessible from Starbucks, but is from a corporate LAN
  • Horizon Workspace provides a central point of management
  • AppVolumes will be in the Enterprise Edition of Horizon View
  • View 6 makes it possible to co-exist and transition from XenApp environments
  • Windows 2008 or 2012 server required for RDSH Application Publishing, and can mix and match if required
  • Easier than upgrade to XenApp 7.5 in the sense that a new infrastructure does not need to be stood up
  • Seamless application remoting,  even on Mac
  • Use vCOps for View and do a 60 day assessment of your environment – though I’m not sure you get the same level of information as you do with say Stratusphere FIT
  • Use thin clients not zero for unified communications in VDI
  • Fully supported by Microsoft for Lync over PCoIP
  • Webcam and mic done using USB redirection
  • Use case for Thinapp is portability and isolation, AppVolumes for performance
  • Application catalogue allows user self service of applications, can remove after 30 days etc
  • Workspace Suite is Horizon + AirWatch, includes Horizon Advanced for Workspace
  • vGPU like Citrix,  coming Q1 next year – vGPU is covered here and is essentially dedicated hardware VGA acceleration but with the consolidation ratio of sVGA. Still uses NVIDIA driver for application validation and support
  • Horizon Flex out in December, delivers containerised desktops in much the same way as the old VMware ACE product
  • No dependency for Horizon Flex on Mirage at the back end
  • Requires Flex policy management server and provides time limits, grace period, remote lock and wipe, USB lock down, etc

IMG_20141118_140229

Cisco and VMware – Chris Bashforth

For my final breakout of the day, I went to the Cisco partner presentation on UCS and VMware View. I have to say I didn’t find this session all that useful. I don’t know if it was due to the graveyard slot at the end of a long day or if it was just the general dryness of the topic, but I never really felt like the audience engaged with the speaker and the atmosphere fell a little flat. We were given a brief overview of UCS for those who have never seen it before and then a quick run through of the blade and chassis models available and which are recommended for VDI deployments.

I’m still quite new to UCS having been a HP guy all of my career, so there were some interesting items in there but I didn’t feel I got a lot out of this session and left a little disappointed. For those folks wanting to use NVIDIA GRID cards in their UCS deployments, you will need to use C class rackmount servers for this purpose, with two slots available per server for this purpose. B class blades are densely packed and simply do not have the space to accommodate this card.

One thing to correct is the speaker’s comment that NVIDIA vDGA will support 8 users per server – this isn’t true. Direct passthrough means that you connect the physical VGA card to the virtual desktop on a 1:1 basis. I can only assume he got mixed up with the upcoming vGPU which will be a similar passthrough arrangement, but with the ability to get a higher consolidation ratio of up to 8. If I misinterpreted these comments, please feel free to let me know.

IMG_20141118_154030

Closing Keynote – Chris Wahl

The closing keynote was from Chris Wahl, industry legend and double VCDX. The force is strong with this one! The session was entitled “Don’t Be a Minesweeper”. I went into the session wondering what the correlation was between stealing bits of beer from tables (my definition of a Minesweeper) and the IT industry, but it turns out he was referring to the cheesy clicky clicky game of previous Windows’ vintages. The general gist was that automation is the way forward, we’re seeing that now, and it pays dividends to be ahead of the curve by learning some scripting now. Whether that be PowerShell, PowerCLI, Python or anything else.

I did particularly enjoy Chris’s attempt at using British slang. Top marks to him for differentiating between bollocks (bad) and dog’s bollocks (very good). It’s not always easy for an American to grasp such as concept depending on whether or not said objects are canine connected, but I think he did pretty well!

IMG_20141118_164222

Summary

Overall it was a very good day and a hearty well done to the VMUG committee who put it all together. This was my third VMUG-UK and each time it just keeps getting bigger. I don’t know how many showed yesterday, but I heard on the Twittervine that nearly 600 had pre-registered, which is absolutely fantastic. I did wonder if the event is now starting to outgrow the venue – the solutions hall was packed and difficult to navigate and lunch and brew breaks got quite cramped for space, but that’s a relatively minor thing.

I didn’t get much chance to look at the booths in the solutions hall, but it’s difficult when you’re a partner with long standing relationships with vendors to have something new to talk about sometimes. I did however get to see some old ex-colleagues as well as chatting to some folks I hadn’t seen in years, which was great.

 

08-11-14

Adventures in NVIDIA GRID K1 and Horizon View

Just had a really interesting week helping a customer with a proof of concept environment using Horizon View 6 and an NVIDIA GRID K1 card, so I thought I’d blog about it. You know, like you do. Anyway, this customer already had a View 5.2 environment running on vSphere 5.1 on a Cisco UCS back end. They use 10 Zig V-1200 zero clients on the desktop to provide virtual desktops to end users. As they are an education customer providing tuition on 3D rendering and computer games production, they wanted to see how far they could push a K1 card and get acceptable performance for end users. Ultimately, the customer’s goal is to move away from fat clients as much as possible and move over to thin or zero clients.

I have to say that in my opinion,  there is not a great deal of content out there about K1 cards and Horizon View. One of the best articles I’ve seen is by my good friend Steve Dunne, where he conducted a PoC with a customer using dedicated VGA. In our case, we were testing out sVGA or Shared VGA as the TCO of dedicated would be far too high and impossible to justify to the cheque signers. This PoC involved a Cisco UCS C240-M3 with an NVIDIA K1 card pre-installed. The K1 card has four GPU cores and 16GB RAM on board and takes up two PCI slots.

I’m not going to produce a highly structured and technical test plan with performance metrics as this really isn’t the way we ran the initial testing. It was really a bit more ad hoc than that. We ran the following basic tests:-

– Official 720p “Furious 7” trailer from YouTube (replete with hopelessly unrealistic stunts)

– Official 1080p “Fury” trailer from YouTube  (replete with hopelessly unrealistic acting)

– Manipulating a pre-assembled dune buggy type unit in LEGO Digital Designer

– Manipulating objects in 3DS Max

When we started off at View 5.2, we found that video playback was very choppy, lip sync was out on the trailers and 3D objects would just hang on the screen. Not the most auspicious of starts, so I ensured the NVIDIA VIB for ESXi 5.1 was the latest version (it was) and ESXi was at 5.1 U1. We made a single change at a time and went back over the test list above to see what difference it made. We initially set the test pool to be hardware 3D renderer, with 256MB of video RAM. The command gpuvm was checked to ensure the test VMs were indeed assigned to the GRID K1 card and we used nvidia-smi -l to monitor the GPU usage during the testing process.

Remember that when you configure video RAM in a desktop pool, half of the RAM is assigned from host memory and the other half is from the GRID K1 card, so keep this in mind when sizing your environment. The goal of the customer is to provide enough capacity for 50 concurrent heavy 3D users per host, so two GRID K1 cards per host is what they’re looking at, pending testing and based on the K120Q profile.

So after performing some Google-Fu, the next change we decided to make was to add the Teradici audio driver to the base image, the main reason for this was that there is apparently a known issue with 10Zig devices and audio sync, so we thought we’d give it a try. Although audio quality and sync did improve, it still didn’t really give us the results we were looking for.

Having gone back to the drawing board (and forums), the next change we made was to upgrade the View agent on the virtual desktop from 5.2 to 5.3. Some customers in the VMware Communities forums had observed some fairly major performance improvements doing this, without the need to upgrade the rest of the View infrastructure to 5.3. We did this and boy did things improve! It was like night and day and the video was much improved, barely flickered at all and the audio was perfect. At this point we decided that View 5.2 was obviously not going to give us the performance we needed to put this environment in front of power users for UAT.

The decision was taken to upgrade the identical but unused parallel environment at another site from vSphere 5.1 and View 5.2 to vSphere 5.5 U2 and View 6.0.1. The reasoning behind this was that I knew that PCoIP had improved markedly in 6.0.1 from the 5.x days, plus is meant we were testing on the latest platform available. Once we upgraded the environment and re-ran the tests, we saw further improvement without major pool or image changes. We updated VMware Tools and also the View Agent to the latest versions as part of the upgrade and the customer was really impressed with the results.

In fact, as we were watching the 720p and 1080p videos, we remarked that you’d never know you were watching it on a zero client, basically streamed from a data centre. That remark is quite telling, as if a bunch of grizzled techies say that, end users are likely to be even more chirpy! We also performed more rudimentary testing with LEGO Digital Designer and also 3DS Max, with improved results. The PoC kit has now been left with the customer, as really we need subject matter experts to test whether or not this solution provides acceptable end user performance to totally replace fat clients.

What is the takeway from this?

The takeaway is that  in my opinion, VDI is following a similar path to what datacentre virtualisation followed a few years back. First you take the quick and easy wins such as web servers and file servers and then you get more ambitious and virtualise database servers and messaging servers once confidence in the platform has been established.

VDI has started with the lighter use of the “knowledge user” who uses a web browser and some Office applications to a basic level. Now this target has been proven and conquered, we’re moving up the stack to concentrate on users who require more grunt out of their virtual desktop. The improvements in Horizon View, ESXi and now with the support of hardware vendors such as NVIDIA and Teradici, native levels of performance for some heavy use cases can be achieved at a sensible cost.

That being said, running a PoC will also help finding where the performance tipping point is and what realistic expectations are. In our case, early testing has shown that video, audio and smaller scale 3D object manipulation is more than feasible and a realistic goal for production. However, much larger vehicles such as the Unreal Development Kit may still be best suited to dedicated gaming hardware rather than a virtual desktop environment. The one thing of course I haven’t mentioned is the fact that the customer is GigE to the desktop and have a 10GigE backbone between their two sites. This makes a huge difference. I doubt we’d have seen the same results on a 100Mbps/1Gbps equivalent environment.

The customer will be testing the PoC until the end of December, hopefully I can share the results nearer the time as to whether or not they proceed and if so, how they do it. Hopefully for anyone researching or testing NVIDIA GRID cards in a VDI environment, this has been some help.

NVIDIA GRID Card

 

16-04-13

VMware related offers of the week – be quick! Just another quick post to bring to your attention a couple of offers that might be of use to fellow virtualisation professionals. Firstly, the new book from Chris Wahl and Steve Pantol “Networking for VMware Administrators” is currently 50% off cover price at Pearson IT Certification. I haven’t read the book as yet, but you can read reviews from my friends Ather Beg and Seb Hakiel to see what it covers. Quite apart from anything else, it fills a notable gap in the market and should be a useful addition to anyone’s library. This offer expires this Sunday, 20th April. ShowCover The other deal is for VMUG Advantage membership. If you are already a “free” VMUG member, you can upgrade to VMUG Advantage status with a 20% discount when you use the code ADVSALE at the checkout. This offer expires a little sooner, at 12pm Central Time tomorrow. Don’t ask me what that is in “real money”, aka GMT 😉 nvvyrqmf As Maury Finkle would say – “Do it!”