How to integrate the Synology VAAI Plugin into the vLCM Image?

I had a customer support case recently where the customer was struggling with updating a vSphere cluster. The customer was already using the vSphere Lifecycle Manager (vLCM), but the composed image always returned the error that the hosts were not compatible. So he asked for help.

We knew the installed hardware, we knew the installed (and targeted) ESXi version, and we also knew the currently installed and targeted version of the vendor addon, too. But we were both a bit perplexed. Until we took a closer look at one of the “incompatible” hosts and the error message that revealed the cause of this incompatibility.

The error message told us that vLCM should obviously downgrade something, but couldn’t. Thanks to the listing of what’s currently on the host and what’s in the image of the desired state, we could find out what the issue is. And there were some “components” that differed in the versions. And we saw that the ESXi version was fine, and the image is more up-to-date than the host, and the same for the vendor addon. But we saw that two or three drivers were a little above on the host than in the image. We couldn’t explain this. But with the possibility of manually adding “components” to an image, we were able to fix that.

vSphere Lifecycle Manager (vLCM) shows an error
Source: knowledge.broadcom.com

Long story short, in the vLCM image, you cannot only choose the desired ESXi version and vendor addon, and if you have the hardware support manager (HSM) installed, the latest server firmware. But you can also add specific drivers and updates manually to the image. And you can even upload some updates to the vLCM.

And exactly the last step, uploading a plugin and manually adding such “components” is the topic of this blog post. Let’s dive into it.

Disclaimer

The following steps are related to hardware and software, which may or may not be on the hardware compatibility list (HCL). I’m showing screenshots from my homelab, where I don’t care much for compatibility. It is more kind of “the main thing is that it (somehow) runs and works and does what I want.”

Please use it at your own risk, and probably not in production, except the vendor and / or Broadcom have it on the compatibility list and everything has been checked for being properly supported.

How can I upload a plugin to vLCM?

Well, I’m glad you’ve asked. That’s pretty simple. But before we move on, you may know what plugin I’m using for my homelab and as an example for this blog post. It is the Synology VAAI plugin. You get more information on how to install the plugin (the manual approach), and also the download link, from the Synology website here.

But what is VAAI?

VAAI, short for “vStorage APIs for Array Integration,” is a feature introduced in ESXi that provides hardware acceleration functionality. It enables your host to offload specific virtual machine and storage management operations to compliant storage hardware. With the storage hardware assistance, your host performs these operations faster and consumes less CPU, memory, and storage fabric bandwidth. (Source: knowledge.broadcom.com – FAQ for vStorage APIs for Array Integration).

And now, let’s continue. I assume that you’re already logged in to the vCenter web client.

  1. In the vSphere web client, click the burger menu in the top-left corner. Then, click the “Lifecycle Manager” entry.
  2. On the top menu, click “ACTIONS“, then “Import Updates

    Notice: it doesn’t matter on which tab you’re on in the Lifecycle Manager; you can import updates from any tab.
  3. Next, click “BROWSE” to open the file dialog, and search for the plugin you’d like to upload
  4. Next, click “IMPORT” to start the upload of the plugin to the vLCM library

And we’re done for the first part, we successfully uploaded a plugin to the vLCM library. We continue now with using this plugin in a vLCM image.

How to add a plugin to a vLCM image?

  1. As we’d like to create an image with this plugin included, navigate to the “Image Library” tab, and click “CREATE IMAGE
  2. We start by providing an image name and choosing the desired ESXi version.
  3. To add the recently uploaded plugin, you may have to click “SHOW DETAILS” first, and then click “ADD COMPONENTS
  4. In the following dialog box, you can search for the plugin. Just click the filter icon in the “Component Name” column and enter the plugin name
  5. Now select the plugin (check the box in front of it), then click “SELECT” in the lower-right corner
  6. We can now see the added plugin in the list of components. Next, click “SAVE” and you’re done with creating the image

You can now move on and assign this image to your cluster and start updating the hosts. Have fun!

Storage performance upgrade – and how to make use of Vmknic binding

It’s been a while now (exactly two years and one month), since I bought my last Synology NAS, both for homelab and personal/private usage. Why did I buy a new NAS? There are some requirements that my new NAS had to fulfill:

  • Capable of 10 Gigabit Ethernet (integrated or with an addon card)
  • Enough slots for disks (at least five)
  • Two NVMe slots for using NVMe SSDs as read/write cache (integrated or with an addon card)
  • Can be used for homelab storage, as well as backup storage for home computers, laptops, etc.

When looking for that one NAS that could handle all of the above, I stumbled across the Synology DS1621xs+. Synology launched it in September 2020, and I ordered it about two months later.

Some of the technical specifications of the NAS:

  • It has 2 x M.2 2280 NVMe slots
  • It comes with 8 GB DDR4 ECC SODIMM (expandable up to 32 GB, and yes I upgraded to 32 GB)
  • It is equipped with an Intel Xeon D-1527 4-core 2.2 GHz CPU that has a Turbo Boost up to 2.7 GHz
  • In terms of connectivity, there are two 1 GbE and one 10 GbE RJ-45 ports
  • If you fancy, you could install some PICe card (the NAS has one 8-lane x8 slot), but I don’t know why or what
  • And you can install up to six 3.5″ or 2.5″ drives

So that’s the basics covered. Let’s get back to the topic.

Read more

How to shut down a vSAN cluster – and start it again

Just a few weeks ago, to my shame, I stumbled across an interesting feature in VMware vSphere when trying some things with vSAN. To be honest, and to make it clear before we dive into this topic in this blog post: I screwed up when I tested this feature the first time. Because I didn’t know about this feature and because I didn’t proceed as I should have, as per this feature. At the end I had to reinstall all my vSAN nodes and create a new clean environment after that I screwed up. It was somehow needed anyway because of the most recent homelab rebuild. So, somehow a win-win for me and the lab.

So don’t screw up! No, just kidding. you may know the feature better than me. And I can tell you, vSAN is stronger and more resilient than you may think.

In this blog post, I’d like to show you how to shut down a vSAN cluster, and how to start it again. The feature is hidden in plain view, right-click the vSAN cluster and you’re good to go. Or not?

In this blog post, I’m assuming that the vCenter is NOT running on the vSAN cluster. I may update this blog post, or create another one, with vCenter running on the cluster. Without searching the internet and checking the VMware docs, I don’t know by heart if this is even possible. Anyway. So how do you shut down the vSAN cluster?

Read more

Bulletproof tip on how to find the right Windows VMDK

I did this so many times already, and I never thought that I could document it, or put it somehow into a blog post. And maybe you already know how that works, how you can find the right Windows VMDK to resize it.

It’s not that complicated if you know what to search or look for. And it’s easy as pie when the VM has only one disk, or maybe two. But imagine a SQL server, which has like eight disks or more, depending on its setup or software recommendation? Then it might get tricky to catch the right VMDK at the first shot.

But this bulletproof guide should help you out! Maybe that gets a new category, bulletproof. We’ll see. But let’s get back to the main topic here.

How do I find the right Windows VMDK?

I deliberately write Windows because I only work on Windows machines for that blog post here and not Linux VMs. It might get another blog post for that sometime in the future. But that depends on the Linux distribution or flavor because Linux is not Linux (don’t hate me please). But there are Debians, RPMs, Unixes, and so many other derivates. Not always the same tools available, not the same commands or syntax. You’ll get the point. Let’s focus on Windows VMs for now.

So, you’ve got that huge VM with several disks, and you have to resize one of them. Lucky you if it’s the C: drive because usually, that’s the first VMDK. Usually. But this guide shows you how to find the right VMDK. And always, making the disk bigger is easier than shrinking it. So it’s better to find the right candidate in the first shot, isn’t it?

Read more

Recap of the latest VMware vSphere 6.7 releases

vSphere 6.7

Oh boy, what a week! Some say that winter is now finally gone, nice and warm weather, not wearing winter jackets anymore. But hey, i’m not a weatherman. When you’re sitting in the office i think it doesn’t matter if it’s raining or snowing outside. Just kidding… Let’s get back to business.

There was some rumor about the next upcoming version. Will it be version 7? Or something just above 6.5? VMware did release several new products versions! And it’s all with version number 6.7. What a list! It’s one of those email notifications that I usually like to scroll down, a little more, and more and more, to get all the news soaked up like a sponge. I’d like to dive in right now and provide you a recap of this weeks VMware releases. And as i said, it’s quite a list. I’ll pick out just some new key features. You can find the full release news on the VMware Blogs (links provided here).

New product versions

vSphere 6.7

  • several new APIs that improve the efficiency and experience to deploy vCenter, to deploy multiple vCenters based on a template, to make management of vCenter Server Appliance significantly easier, as well as for backup and restore
  • significantly simplifies the vCenter Server topology through vCenter with embedded platform services controller in enhanced linked mode
  • 2X faster performance in vCenter operations per second
  • 3X reduction in memory usage
  • 3X faster DRS-related operations (e.g. power-on virtual machine)
  • vSphere 6.7 improves efficiency when updating ESXi hosts, significantly reducing maintenance time by eliminating one of two reboots normally required for major version upgrades (Single Reboot). In addition to that, vSphere Quick Boot is a new innovation that restarts the ESXi hypervisor without rebooting the physical host, skipping time-consuming hardware initialization
  • The HTML5-based vSphere Client provides a modern user interface experience that is both responsive and easy to use, and it’s now including other key functionality like managing NSX, vSAN, VUM as well as third-party components.
  • enabling encrypted vMotion across different vCenter instances
  • enhancements to Nvidia GRID vGPU
  • vSphere 6.7 introduces vCenter Server Hybrid Linked Mode, which makes it easy and simple for customers to have unified visibility and manageability across an on-premises vSphere environment running on one version and a vSphere-based public cloud environment, such as VMware Cloud on AWS, running on a different version of vSphere.
  • vSphere 6.7 also introduces Cross-Cloud Cold and Hot Migration
  • Delivers a new capability that is key for the hybrid cloud, called Per-VM EVC

More information here: Introducing VMware vSphere 6.7 / VMware Blogs

vSAN 6.7

  • vSAN 6.7 provides intuitive operations that align with other VMware products from a UI and workflow perspective to provide a “one team, one tool” experience
  • Iintroduces a new HTML5 UI based on the “Clarity” framework as seen in other VMware products (All products in the VMware portfolio are moving toward this UI framework)
  • A new feature known as “vRealize Operations within vCenter” provides an easy way for customers to see vRealize intelligence directly in the vSphere Client
  • vSAN 6.7 now expands the flexibility of the vSAN iSCSI service to support Windows Server Failover Clusters (WSFC)
  • vSAN 6.7 introduces an all-new Adaptive Resync feature to ensure a fair-share of resources are available for VM I/Os and Resync I/Os during dynamic changes in load on the system
  • Optimizes the de-staging mechanism, resulting in data that “drains” more quickly from the write buffer to the capacity tier.  The ability to de-stage this data quickly allows the cache tier to accept new I/O, which reduces or eliminates periods of congestion
  • New health checks include:
    • Maintenance mode verification ensures proper decommission state
    • Consistent configuration verification for advanced settings
    • vSAN and vMotion network connectivity checks improved
    • Improved vSAN Health service installation check
    • Improved physical disk health check combines multiple checks (software, physical, metadata) into a single notification
    • Firmware check is independent from driver check

More information here: What’s New with VMware vSAN 6.7 / VMware Blogs and also here: Extending Hybrid Cloud Leadership with vSAN 6.7

vCenter Server 6.7

  • The vSphere Client (HTML5) is full of new workflows and closer to feature parity
  • built-in file-based vCenter Server backup now includes a scheduler

Installation

  • No load balancer required for high availability and fully supports native vCenter Server High Availability.
  • SSO Site boundary removal provides flexibility of placement.
  • Supports vSphere scale maximums.
  • Allows for 15 deployments in a vSphere Single Sign-On Domain.
  • Reduces the number of nodes to manage and maintain.

Migration

  • vSphere 6.7 is also the last release to include vCenter Server for Windows, which has been deprecated.
  • migrate to the vCenter Server Appliance with the built-in Migration Tool
  • Deploy & import all data
  • Deploy & import data in the background
  • Customers will also get an estimated time of how long each option will take when migrating

Upgrading

  • vSphere 6.7. will support upgrades and migrations only from vSphere 6.0 or 6.5
  • vSphere 5.5 does not have a direct upgrade path to vSphere 6.7
  • Upgrade path: vSphere 5.5 to vSphere 6.0 or 6.5, and then to vSphere 6.7
  • vCenter Server 6.0 or 6.5 managing ESXi 5.5 hosts cannot be upgraded or migrated until the hosts have been upgraded to at least ESXi 6.0
  • Reminder: end of general support for vSphere 5.5 is September 19, 2018.

Monitoring and Management

  • vSphere Appliance Management Interface (VAMI) on port 5480 has received an update to the Clarity UI
  • There is now a tab dedicated to monitoring. Here you can see CPU, memory, network, database and disk utilization.
  • Another new tab called Services is also within the VAMI, giving the option to start, stop, and restart vCenter Server services if needed
  • vSphere 6.7 also marks the final release of the vSphere Web Client (Flash). Some of the newer workflows in the updated vSphere HTML5 Client release include:
    • vSphere Update Manager
    • Content Library
    • vSAN
    • Storage Policies
    • Host Profiles
    • vDS Topology Diagram
    • Licensing

More information here: Introducing vCenter Server 6.7 / VMware Blogs

vSphere with Operations Management 6.7

  • new plugin for the vSphere Client. This plugin is available out-of-the-box and provides some great new functionality
  • When interacting with this plugin, you will be greeted with 6 vRealize Operations Manager (vROps) dashboards directly in the vSphere client
  • overview, cluster view, and alerts for both vCenter and vSAN views
  • The new Quick Start page is making it easier to get directly to the data you need to
  • four use cases: Optimize Performance, Optimize Capacity, Troubleshoot, and Manage Configuration
  • The Workload Optimization dashboard was updated. Workload Optimization takes predictive analytics and uses them in conjunction with vSphere Distributed Resource Scheduler (DRS) to move workloads between clusters. New with vROps 6.7, you can now fine tune the configuration for workload optimization
  • vROps 6.7 introduced a completely new capacity engine that is smarter and much faster

More information here: vSphere with Operations Management 6.7 / VMware Blogs

vSphere 6.7 Security

  • TPM 2.0 support for ESXi
  • Virtual TPM 2.0 for VMs
  • Support for Microsoft Virtualization Based Security
  • UI updates (combined all encryption functions (VM Encryption, vMotion Encryption) into one panel in VM Options)
  • Multiple SYSLOG targets
  • FIPS 140-2 validated cryptographic modules – by default!

More information here: vSphere 6.7 Security / VMware Blogs

Developer and Automation Interfaces for vSphere 6.7

  • Added functionality to existing APIs in vSphere 6.7
  • Coverage of new areas
  • Appliance API updates: from prechecks to staging to installation and validation, it’s all available by API now
  • vCenter API updates: new APIs have been added to interact with the VM’s guest operating system (OS), viewing Storage Policy Based Management (SPBM) policies, and managing vCenter server services
  • also a handful of new APIs to handle the deployment and lifecycle of the vCenter server
  • a handful of updates to the vSphere Web Services (SOAP) APIs as well

More information here: Developer and Automation Interfaces for vSphere 6.7 / VMware Blogs

Faster Lifecycle Management Operations in VMware vSphere 6.7

  • brand-new Update Manager interface which is now part of the HTML5 Client
  • Update Manager in vSphere 6.7 keeps VMware ESXi 6.0 to 6.7 hosts reliable and secure
  • the new UI provides a much more streamlined remediation process, requiring just a few clicks to begin the procedure. It’s not just a port from the old Flash client
  • Hosts that are currently on ESXi 6.5 will be upgraded to 6.7 significantly faster than ever before
  • Several optimizations have been made for that upgrade path, including eliminating one of two reboots traditionally required for a host upgrade
  • Quick Boot eliminates the time-consuming hardware initialization phase by shutting down ESXi in an orderly manner and then immediately re-starting it

More information here: Faster Lifecycle Management Operations in VMware vSphere 6.7 / VMware Blogs

vSphere 6.7 for Enterprise Applications

  • include support for Persistent Memory (PMEM) and enhanced support for Remote Directory Memory Access (RDMA)
  • PMEM is a new layer called Non-Volatile Memory (NVM) and sits between NAND flash and DRAM, providing faster performance relative to NAND flash but also providing the non-volatility not typically found in traditional memory offerings
  • new protocol support for Remote Direct memory Access (RDMA) over Converged Ethernet, or RoCE (pronounced “rocky”) v2, a new software Fiber Channel over Ethernet (FCoE) adapter, and iSCSI Extension for RDMA (iSER)

More information here: vSphere 6.7 for Enterprise Applications / VMware Blogs