VMware – Clone a VM with snapshots (and consolidate it)

Recently we had a weird issue in our office. We had one virtual machine with a snapshot. That by itself isn’t an issue, a snapshot is nothing special. But someone created that snapshot before a software upgrade and forget to delete it. So this snapshot was growing and growing. We found out that there is a snapshot when the VM or service owner requested some additional disk space. We weren’t able to add disk space because of that snapshot. So we scheduled a maintenance window to delete the snapshot. Faster said than done.

The VM went offline because of disk consolidation. That could happen, depending on snapshot size and storage system. But the VM not only went offline for some time, but unexpectedly for hours. Together with VMware support we were able to stop the snapshot deletion. The VM came back online but with the known “Disk Consolidation Needed status”. We found out that this snapshot was about 400 GB in size. What a bummer! So we scheduled another maintenance window to consolidate that snapshot. Unfortunately that didn’t work well. Consolidation timed at around 96%, not sure why. “Error communicating with the host” isn’t very helpful in that moment.

Some research and again having a chat with VMware support led us towards cloning the disk files. During the cloning of a disk file the snapshot will be consolidated. And as you’re doing a disk clone locally on the ESXi host with “vmkfstools” and not withing vCenter, there shouldn’t be a timeout either. So we had out action plan. And we scheduled another maintenance window with the service owner.

Read more

Recap of the latest VMware vSphere 6.7 releases

vSphere 6.7

Oh boy, what a week! Some say that winter is now finally gone, nice and warm weather, not wearing winter jackets anymore. But hey, i’m not a weatherman. When you’re sitting in the office i think it doesn’t matter if it’s raining or snowing outside. Just kidding… Let’s get back to business.

There was some rumor about the next upcoming version. Will it be version 7? Or something just above 6.5? VMware did release several new products versions! And it’s all with version number 6.7. What a list! It’s one of those email notifications that I usually like to scroll down, a little more, and more and more, to get all the news soaked up like a sponge. I’d like to dive in right now and provide you a recap of this weeks VMware releases. And as i said, it’s quite a list. I’ll pick out just some new key features. You can find the full release news on the VMware Blogs (links provided here).

New product versions

vSphere 6.7

  • several new APIs that improve the efficiency and experience to deploy vCenter, to deploy multiple vCenters based on a template, to make management of vCenter Server Appliance significantly easier, as well as for backup and restore
  • significantly simplifies the vCenter Server topology through vCenter with embedded platform services controller in enhanced linked mode
  • 2X faster performance in vCenter operations per second
  • 3X reduction in memory usage
  • 3X faster DRS-related operations (e.g. power-on virtual machine)
  • vSphere 6.7 improves efficiency when updating ESXi hosts, significantly reducing maintenance time by eliminating one of two reboots normally required for major version upgrades (Single Reboot). In addition to that, vSphere Quick Boot is a new innovation that restarts the ESXi hypervisor without rebooting the physical host, skipping time-consuming hardware initialization
  • The HTML5-based vSphere Client provides a modern user interface experience that is both responsive and easy to use, and it’s now including other key functionality like managing NSX, vSAN, VUM as well as third-party components.
  • enabling encrypted vMotion across different vCenter instances
  • enhancements to Nvidia GRID vGPU
  • vSphere 6.7 introduces vCenter Server Hybrid Linked Mode, which makes it easy and simple for customers to have unified visibility and manageability across an on-premises vSphere environment running on one version and a vSphere-based public cloud environment, such as VMware Cloud on AWS, running on a different version of vSphere.
  • vSphere 6.7 also introduces Cross-Cloud Cold and Hot Migration
  • Delivers a new capability that is key for the hybrid cloud, called Per-VM EVC

More information here: Introducing VMware vSphere 6.7 / VMware Blogs

vSAN 6.7

  • vSAN 6.7 provides intuitive operations that align with other VMware products from a UI and workflow perspective to provide a “one team, one tool” experience
  • Iintroduces a new HTML5 UI based on the “Clarity” framework as seen in other VMware products (All products in the VMware portfolio are moving toward this UI framework)
  • A new feature known as “vRealize Operations within vCenter” provides an easy way for customers to see vRealize intelligence directly in the vSphere Client
  • vSAN 6.7 now expands the flexibility of the vSAN iSCSI service to support Windows Server Failover Clusters (WSFC)
  • vSAN 6.7 introduces an all-new Adaptive Resync feature to ensure a fair-share of resources are available for VM I/Os and Resync I/Os during dynamic changes in load on the system
  • Optimizes the de-staging mechanism, resulting in data that “drains” more quickly from the write buffer to the capacity tier.  The ability to de-stage this data quickly allows the cache tier to accept new I/O, which reduces or eliminates periods of congestion
  • New health checks include:
    • Maintenance mode verification ensures proper decommission state
    • Consistent configuration verification for advanced settings
    • vSAN and vMotion network connectivity checks improved
    • Improved vSAN Health service installation check
    • Improved physical disk health check combines multiple checks (software, physical, metadata) into a single notification
    • Firmware check is independent from driver check

More information here: What’s New with VMware vSAN 6.7 / VMware Blogs and also here: Extending Hybrid Cloud Leadership with vSAN 6.7

vCenter Server 6.7

  • The vSphere Client (HTML5) is full of new workflows and closer to feature parity
  • built-in file-based vCenter Server backup now includes a scheduler

Installation

  • No load balancer required for high availability and fully supports native vCenter Server High Availability.
  • SSO Site boundary removal provides flexibility of placement.
  • Supports vSphere scale maximums.
  • Allows for 15 deployments in a vSphere Single Sign-On Domain.
  • Reduces the number of nodes to manage and maintain.

Migration

  • vSphere 6.7 is also the last release to include vCenter Server for Windows, which has been deprecated.
  • migrate to the vCenter Server Appliance with the built-in Migration Tool
  • Deploy & import all data
  • Deploy & import data in the background
  • Customers will also get an estimated time of how long each option will take when migrating

Upgrading

  • vSphere 6.7. will support upgrades and migrations only from vSphere 6.0 or 6.5
  • vSphere 5.5 does not have a direct upgrade path to vSphere 6.7
  • Upgrade path: vSphere 5.5 to vSphere 6.0 or 6.5, and then to vSphere 6.7
  • vCenter Server 6.0 or 6.5 managing ESXi 5.5 hosts cannot be upgraded or migrated until the hosts have been upgraded to at least ESXi 6.0
  • Reminder: end of general support for vSphere 5.5 is September 19, 2018.

Monitoring and Management

  • vSphere Appliance Management Interface (VAMI) on port 5480 has received an update to the Clarity UI
  • There is now a tab dedicated to monitoring. Here you can see CPU, memory, network, database and disk utilization.
  • Another new tab called Services is also within the VAMI, giving the option to start, stop, and restart vCenter Server services if needed
  • vSphere 6.7 also marks the final release of the vSphere Web Client (Flash). Some of the newer workflows in the updated vSphere HTML5 Client release include:
    • vSphere Update Manager
    • Content Library
    • vSAN
    • Storage Policies
    • Host Profiles
    • vDS Topology Diagram
    • Licensing

More information here: Introducing vCenter Server 6.7 / VMware Blogs

vSphere with Operations Management 6.7

  • new plugin for the vSphere Client. This plugin is available out-of-the-box and provides some great new functionality
  • When interacting with this plugin, you will be greeted with 6 vRealize Operations Manager (vROps) dashboards directly in the vSphere client
  • overview, cluster view, and alerts for both vCenter and vSAN views
  • The new Quick Start page is making it easier to get directly to the data you need to
  • four use cases: Optimize Performance, Optimize Capacity, Troubleshoot, and Manage Configuration
  • The Workload Optimization dashboard was updated. Workload Optimization takes predictive analytics and uses them in conjunction with vSphere Distributed Resource Scheduler (DRS) to move workloads between clusters. New with vROps 6.7, you can now fine tune the configuration for workload optimization
  • vROps 6.7 introduced a completely new capacity engine that is smarter and much faster

More information here: vSphere with Operations Management 6.7 / VMware Blogs

vSphere 6.7 Security

  • TPM 2.0 support for ESXi
  • Virtual TPM 2.0 for VMs
  • Support for Microsoft Virtualization Based Security
  • UI updates (combined all encryption functions (VM Encryption, vMotion Encryption) into one panel in VM Options)
  • Multiple SYSLOG targets
  • FIPS 140-2 validated cryptographic modules – by default!

More information here: vSphere 6.7 Security / VMware Blogs

Developer and Automation Interfaces for vSphere 6.7

  • Added functionality to existing APIs in vSphere 6.7
  • Coverage of new areas
  • Appliance API updates: from prechecks to staging to installation and validation, it’s all available by API now
  • vCenter API updates: new APIs have been added to interact with the VM’s guest operating system (OS), viewing Storage Policy Based Management (SPBM) policies, and managing vCenter server services
  • also a handful of new APIs to handle the deployment and lifecycle of the vCenter server
  • a handful of updates to the vSphere Web Services (SOAP) APIs as well

More information here: Developer and Automation Interfaces for vSphere 6.7 / VMware Blogs

Faster Lifecycle Management Operations in VMware vSphere 6.7

  • brand-new Update Manager interface which is now part of the HTML5 Client
  • Update Manager in vSphere 6.7 keeps VMware ESXi 6.0 to 6.7 hosts reliable and secure
  • the new UI provides a much more streamlined remediation process, requiring just a few clicks to begin the procedure. It’s not just a port from the old Flash client
  • Hosts that are currently on ESXi 6.5 will be upgraded to 6.7 significantly faster than ever before
  • Several optimizations have been made for that upgrade path, including eliminating one of two reboots traditionally required for a host upgrade
  • Quick Boot eliminates the time-consuming hardware initialization phase by shutting down ESXi in an orderly manner and then immediately re-starting it

More information here: Faster Lifecycle Management Operations in VMware vSphere 6.7 / VMware Blogs

vSphere 6.7 for Enterprise Applications

  • include support for Persistent Memory (PMEM) and enhanced support for Remote Directory Memory Access (RDMA)
  • PMEM is a new layer called Non-Volatile Memory (NVM) and sits between NAND flash and DRAM, providing faster performance relative to NAND flash but also providing the non-volatility not typically found in traditional memory offerings
  • new protocol support for Remote Direct memory Access (RDMA) over Converged Ethernet, or RoCE (pronounced “rocky”) v2, a new software Fiber Channel over Ethernet (FCoE) adapter, and iSCSI Extension for RDMA (iSER)

More information here: vSphere 6.7 for Enterprise Applications / VMware Blogs

How to download vSphere software with VMware Software Manager

The VMware Software Manager is a very helpful tool if you want to download your VMware vSphere, vCloud or vRealize software packages. It is also very helpful to keep your VMware software repository up to date. All you need is your VMware login and the installer package. Let me introduce this software to you.

Download

Visit this website to download the VMware Software Manager (login / registration required). It’s an 18 megabyte MSI package. So big deal.

You have to accept the End User License Agreement to download the software:

Installation

  1. Double click the MSI package to start the setup process.

    Click Next.

    Software Manager

  2. Accept the EULA and click Next.

    EULA

  3. Choose the installation folder (just let the standard is also fine).

    Choose the depot location; either a local drive, an external disk or even a UNC path is possible.

    Click Next to continue.

    Basic Settings

  4. When all things are set, click Install to continue.

    Ready

  5. You can check the box to open Download Service web application.

    Click Finish to close the setup assistant.

    Completed

Now after installation we can move on with the further configuration. When you checked the box to open the web application, then you’re ready. Otherwise look on your desktop for the icon to start the web application manually.

Accessing VMware Software Manager

  1. When you start the web application you’ll see the login screen.

    Use your My VMware login (or one of the other two possibilities) to login.

    Click Connect to login.

    Login

  2. Now Software Manager reads the VMware software depot for the first time. This can take some time…

    Reading VMware depot

  3. After reading the VMware software Depot you’ll see the main software dashboard.

    When you click the arrow (>) on the right side of a product you can see more details.

    This is also the way how to find the download button.

    Dashobard

Conclusion

As mentioned above the VMWare Software Manager is a great tool to download various VMware software all around data center virtualization and automation. If you have a VMware software repository in your company it’s also a good way to keep that repository up to date. The software itself does not run as a service, you have to start it manually.

VMware – vSphere Update Manager in a DRS cluster (white paper)

white paper

Yesterday i published my first white paper. It’s nothing special. Just a small guide on how to use the vSphere Update Manager (VUM) in your DRS enabled cluster.

In today’s world of IT, datacenter and cloud automation, maintenance windows and downtime are a special topic. A few years ago the IT department did updates mostly on weekends because nobody was working then. On Monday everyone came back to the office, the mail server was patched and driver updates were installed. Anybody uses IT like running water. And nobody except the IT knows what effort it is to keep the IT thus the business running.

Today at least maintenance windows with service interruption are somewhat of the past, but not to be forgotten, because everyone want’s access to their data whenever it’s needed, wherever it’s located. You can’t shut down a mail server to install updates, you can’t restart virtualization hosts just to install a driver or a patch. IT has to continue to run like water from the tap.

I’m working as a system engineer for an IT company in Switzerland. We provide different services to our customers, ranging from small to medium sized businesses. I saw so many transformations in business needs, but most of the customers had the same whish. The employees of the customers should have access to their emails, wanted to work from home or when they are on the go. So the IT systems had to run twenty four hours and seven days a week.

VMware vSphere Update Manager is a powerful tool to update your ESXi hosts. You can automatically set your hosts into maintenance mode, and if DRS is enabled, your virtual machines are moved to other hosts automatically. At least from infrastructure perspective you can avoid any maintenance window or even downtime. Because the DRS cluster manages the VMs and you can patch your ESXi hosts in the middle of the day.

Read and download this white paper here.

VMware – vSAN Deploy and Manage course – Day 3

Today it was the last day in our VMware vSAN Deploy and Manage course. Nevertheless today we have given everything again. We had a deep dive in designing vSAN solutions, we discussed the key topics in design decisions and also played around with some what-if scenarios. But as every day we kicked off with some review what we discovered yesterday, and again to make for everyone clear what vSAN really is.

Day 3

Daily review

What is vSAN:

  • Software Defined Storage
  • Hyper Converged Infrastructure
  • Network Storage Topology
  • Hypervisor integrated
    • That means less latency
    • no dependencies on VMs
    • support
    • distributed
  • local disks presenting one datastore per cluster

Use cases:

  • VDI (licensing, offload of IOPS, scalable)
  • Test / DEv environments (projects, easy, growth)
  • Branch Office / Remote Office (same solution, backup)

Install vSAN:

  • Simple, with GUI, all from the web client (with just few clicks)
  • install vSphere
  • create a Cluster
  • set a VMkernel for vSAN
  • disable HA
  • claim disks
    • create disk groups
    • claim them as cache / capacity tier
  • enable vSAN

What’s in the default vSAN policy:

  • FTT = 1
  • Stripes = 1
  • No reservation (neither cache nor capacity)
  • Thin provisioning

What is a Fault Domain:

  • an area which may can fail
  • plan to recover impact of Ops
    • Rack awareness
    • Site awerness

Availability:

In vSAN there are two states of compliance…

  • Compliant
  • non compliant
    • Absent => wait for 60 minutes, then rebuild
    • Degraded => rebuild immediately

What-if failures:

  • Cache disk fails => lose disk group => latency increases
  • Capacity disk fails => degraded => rebuild => VM back online
  • Controller => host issue => HA response
  • Host outtage (complete loss of host) => HA response => VM response

Module 7 Lesson 2 – Troubleshooting

Some topics we covered already yesterday. Today it was also some repetition and a quick overview about troubleshooting and some of the tools we discovered yesterday. There are so many tools for troubleshooting available, either already built-in or community driven, i think the list could be longer. But at least some of the most known tools i will provide you with this list.

  • vCenter (you don’t say…)
  • vROPS
  • esxtop
  • Wireshark (yes indeed; capture packets on ESXi and analyze them with Wireshark => pcap)
  • vSAN Observer (based on Ruby)
  • RVC (Ruby vSphere Console)
  • Health Service Plugin
  • vCheck
  • PowerCLI scripts (combined with Onyx)

A cool tool indeed is vCheck. It’s based on Powershell scripts and runs against your vSphere infrastructure (there are scripts for other stuff too). You schedule the scripts and you can reveive notifications about changes, issues (before they become a real deal). So when you arrive in your office you already know what’s going on (or what’s not). Also worth to mention is vSAN Observer. It’s already there, just start it and access the built-in webserver to get an overview what’s going on in your vSAN environment.

Module 8 -Stretched Cluster

After doing some work in the labs we talked about design. And having a stretched cluster is also a question of design, how to create a solution which covers rack outtages or even a complete site outtage. You can do that with a stretched cluster. And the failover happens automatic (what may probably not the best solution in every fail over situation…).

When planning a stretched cluster you have to concern about resources. You need 50% spare capacity on both sites (talking about two racks or two sites) in HA admission control. Imagine that one site / rack should keep the other one online, and the stuff which is already running on the secondary site too.

You don’t have to use SRM (Site Recovery Manager) for a failover. vSAN does that for you automatically. If you use SRM then you have to have a recovery plan for each and every VM. Thats a lot of planning and even checks if there are new or changed VMs. Not to think about the costs. You need SRM licenses and a second vCenter license.

Talking about the vSAN witness. A witness is a separate ESXi box. This can be a physical server with ESXi which needs to be licensed. This physical server can’t be a member of a cluster, but it can run some VMs on it. Or you can get a witness appliance, which represents a special ESXi as an appliance, which runs on a ESXi server. This appliance cannot run VMs on it.

You can have a ROBO vSAN cluster in your remote office / branch office which consists only of two ESXi hosts in this cluster. If you’re doing so you have to have a witness host / appliance in your main office site. You always need somewhere a witness to have the quorum in case of an HA event. And remember the 5 heartbeats. In the case of an outtage, after 5 missed hearbeats your host is gone and a failover happens.

Module 10 – Designing a vSAN deployment

That’s not a random list of IT buzzwords, folks. You have to consider these key points when you’re designing a vSAN solution (probably any other scalable solution too).

Availability Management
Managability Virtual machines
Performance Compute
Recoverability Network
Security Storage

Let me give you some more things to consider. In the way of designing a vSAN solution you will have to find answers to these questions. Some answers you will get from your customer when talking with him about a solution for his specific needs. Some other answers you will find when you design the solution. And you will find some more questions too…

Requirements (must have / be / do)

  • “RPO of 15 minutes”
  • “RTO of 5 minutes”
  • Location of data / data center

Constraints (design decisions)

  • “Must work with existing network hardware”
  • “Must work at this site”

Assumptions

  • “We have enough bandwith”

Risks

  • “If the bandwith is not enough => risk of not meeting the SLA”

If you covered the topics above (and the bullet points are just ideas, there are lot more to cover) then you will proceed with the design.

Conception

Logical

  • “Keep data at this location”
  • “We want two sites, one for failover”
  • Should also be vendor independent

Physical

  • Here you can come in with the vendor and create the solution
  • vSAN here, stretched cluster there
  • Network links from here to here
  • Backup then with this
  • and so on…

And if you are searching some benchmark tools for your newly created solution, there you go:

  • HCIBench (https://labs.vmware.com/flings/hcibench)
    • simplify and accelerate customer POC performance testing
    • not only a benchmark tool designed for Virtual SAN
    • evaluate the performance of all kinds of Hyper-Converged Infrastructure Storage in vSphere
  • HammerDB (http://www.hammerdb.com/)
    • open source database load testing and benchmarking tool
    • Oracle Database, Microsoft SQL Server, IBM DB2, TimesTen, MySQL, MariaDB,  PostgreSQL
    • Postgres Plus Advanced Server, Greenplum, Redis
    • Amazon Aurora and Redshift and Trafodion SQL on Hadoop

Here you can find the other blog posts about the vSAN deploy and manage course: