How to integrate the Synology VAAI Plugin into the vLCM Image?

by

I had a customer support case recently where the customer was struggling with updating a vSphere cluster. The customer was already using the vSphere Lifecycle Manager (vLCM), but the composed image always returned the error that the hosts were not compatible. So he asked for help.

We knew the installed hardware, we knew the installed (and targeted) ESXi version, and we also knew the currently installed and targeted version of the vendor addon, too. But we were both a bit perplexed. Until we took a closer look at one of the “incompatible” hosts and the error message that revealed the cause of this incompatibility.

The error message told us that vLCM should obviously downgrade something, but couldn’t. Thanks to the listing of what’s currently on the host and what’s in the image of the desired state, we could find out what the issue is. And there were some “components” that differed in the versions. And we saw that the ESXi version was fine, and the image is more up-to-date than the host, and the same for the vendor addon. But we saw that two or three drivers were a little above on the host than in the image. We couldn’t explain this. But with the possibility of manually adding “components” to an image, we were able to fix that.

vSphere Lifecycle Manager (vLCM) shows an error
Source: knowledge.broadcom.com

Long story short, in the vLCM image, you cannot only choose the desired ESXi version and vendor addon, and if you have the hardware support manager (HSM) installed, the latest server firmware. But you can also add specific drivers and updates manually to the image. And you can even upload some updates to the vLCM.

And exactly the last step, uploading a plugin and manually adding such “components” is the topic of this blog post. Let’s dive into it.

Disclaimer

The following steps are related to hardware and software, which may or may not be on the hardware compatibility list (HCL). I’m showing screenshots from my homelab, where I don’t care much for compatibility. It is more kind of “the main thing is that it (somehow) runs and works and does what I want.”

Please use it at your own risk, and probably not in production, except the vendor and / or Broadcom have it on the compatibility list and everything has been checked for being properly supported.

How can I upload a plugin to vLCM?

Well, I’m glad you’ve asked. That’s pretty simple. But before we move on, you may know what plugin I’m using for my homelab and as an example for this blog post. It is the Synology VAAI plugin. You get more information on how to install the plugin (the manual approach), and also the download link, from the Synology website here.

But what is VAAI?

VAAI, short for “vStorage APIs for Array Integration,” is a feature introduced in ESXi that provides hardware acceleration functionality. It enables your host to offload specific virtual machine and storage management operations to compliant storage hardware. With the storage hardware assistance, your host performs these operations faster and consumes less CPU, memory, and storage fabric bandwidth. (Source: knowledge.broadcom.com – FAQ for vStorage APIs for Array Integration).

And now, let’s continue. I assume that you’re already logged in to the vCenter web client.

  1. In the vSphere web client, click the burger menu in the top-left corner. Then, click the “Lifecycle Manager” entry.
  2. On the top menu, click “ACTIONS“, then “Import Updates”

    Notice: it doesn’t matter on which tab you’re on in the Lifecycle Manager; you can import updates from any tab.
  3. Next, click “BROWSE” to open the file dialog, and search for the plugin you’d like to upload
  4. Next, click “IMPORT” to start the upload of the plugin to the vLCM library

And we’re done for the first part, we successfully uploaded a plugin to the vLCM library. We continue now with using this plugin in a vLCM image.

How to add a plugin to a vLCM image?

  1. As we’d like to create an image with this plugin included, navigate to the “Image Library” tab, and click “CREATE IMAGE”
  2. We start by providing an image name and choosing the desired ESXi version.
  3. To add the recently uploaded plugin, you may have to click “SHOW DETAILS” first, and then click “ADD COMPONENTS”
  4. In the following dialog box, you can search for the plugin. Just click the filter icon in the “Component Name” column and enter the plugin name
  5. Now select the plugin (check the box in front of it), then click “SELECT” in the lower-right corner
  6. We can now see the added plugin in the list of components. Next, click “SAVE” and you’re done with creating the image

You can now move on and assign this image to your cluster and start updating the hosts. Have fun!

VCF 9 – How to migrate your Content Library to a different Data Store

by

I wasn’t aware that you couldn’t migrate your vSphere content library to a different data store. At least not just like you would migrate a virtual machine. It was a feature that I wasn’t missing much in my homelab, and in production (at my employer or my customers), content libraries weren’t a thing anyway. Back in the days, the only possible way to “migrate” the content library from one data store to another was to export everything from the “old” content library, create a new one, and import everything into the new content library.

When I was driving to a customer beginning this week for a project meeting, I thought, “let’s listen to some tech podcasts”. Long story short, I looked up the Virtually Speaking Podcast on Spotify. During the drive, I heard that you can now migrate a content library to a different data store! I didn’t memorize the title of this podcast, so I had to go through many podcasts again back at home to find the one. And here it is, not on Spotify but on YouTube: What’s New with vSphere in VCF 9.0 – (YouTube.com) at around 37:28 in the video.

What is a content library?

Just in case you’re not aware of this great thing called a content library, I’ll just add some explanations here. A vSphere content library is a centralized repository within VMware vSphere used to store and manage various types of files, including VM templates, vApp templates, ISO images, and other files, enabling efficient sharing and deployment across multiple vCenter Server instances. It simplifies the management of virtual machine resources by providing a single source for templates and other files, eliminating the need to manually update and copy them across different environments. So, a pretty nice thing, isn’t it?

Why would you migrate a content library?

There are multiple reasons, I’ll just name a few. Every content library has a specific data store configured. You may want to migrate the content library because you’re running out of disk space.  Space contention may happen. Or your storage system is undergoing maintenance, or is going to be replaced, or you want to store all these VM templates, ISO files, etc., on a “cheaper” data store. The use case is up to you.

How do you migrate a content library?

The following steps should guide you through the process. It is pretty easy, but it comes with one downside. The migration workflow doesn’t support the migration of VM templates. But I think you don’t have to worry much. At least I didn’t have any issues during setting up a content library and also the migration from one data store to another worked just fine.

What did I store in the content library for the test?

  • An ISO file
  • A VM template (traditional, right-click a VM, convert to template, right-click the template, clone to library)
  • A VM template (imported from an OVA file)

So nothing special, for sure, but I’m not sure what exactly is meant by “The workflow doesn’t support the migration of VM templates”. I didn’t test the migration with versioned VM templates, but I may update this blog post at a later time.

So let’s get started now.

Two ways to initiate the migration

You can initiate the migration of your content library in two ways:

  1. You can right-click the content library and select “Migrate”
  2. Or you go into the content library, and select from the actions menu “ACTIONS => Migrate”.
  3. Next, the migration assistant will start.

    NOTICE: You can see the yellow warning, stating that “The workflow doesn’t support the migration of VM templates”
  4. Next, select the storage where you want to migrate the content library to
  5. Check once if everything is correct, then click “FINISH”
  6. In the Tasks view below, you can see the migration progress

And that’s it. Not that complicated. The duration of the content library migration depends on the amount of data as well as the performance of the network and storage.

 

Using the VCF Download Tool on Windows Subsystem for Linux to download VCF 9 binaries

by

You may have read it here first, but I recently started a new job. And I’m going deep down the rabbit hole, the VMware Cloud Foundation rabbit hole. I recently sat my first VMware Cloud Foundation exam (VCF-VCP administrator) and passed. The needle for this had been in the arm for a long time if one can put it that way.

VCF version 9 has just been released recently too, and many people are trying to get it up and running in their homelab. And I’m no exception. Getting the stuff done is not as easy as it was back in the days with setting up the ESXi hosts, installing vCenter, and then configuring the vSphere environment as it should be (storage, vSAN, distributed virtual switches, etc.).

I’m also not yet quite sure if my whole homelab infrastructure (talking about the Supermicro servers here, and the HP workstation here) has enough resources. It should be enough when doing the math in terms of CPU cores, amount of memory, and storage. But when going deep technical, with CPU specs, etc., that might be a killer. We will see.

Installing the new VMware Cloud Foundation (VCF) version 9 has been simplified somewhat compared to version 5. There is now an installer available to be deployed as an OVA file, which then pulls all the required files (like ESXi installers, vCenter ISO, and all that stuff) automatically, either from an online depot or an offline depot. No matter which way you go, a valid download token is required to pull the installer files.

A quick word on the VCF download tool

The following guide should help you download the binaries from Broadcom through the VCF download tool, running in Windows Subsystem for Linux (WSL). I’m assuming that you have already a WSL environment with a supported Linux version set up and running on your computer. The VCF download tool is designed to run on Linux. The only way to get it working on Windows is to use the WSL environment. Some information (not much, to be honest) is available here on the Broadcom support website: VCF Download Tool under Windows OS (broadcom.com).

A sidenote on the offline depot

One cool thing I stumbled across during my first steps is a blog post from William Lam, where he is setting up a Synology NAS as an offline depot. You may check this out if you have a Synology NAS, that would be a great fit for having the VCF 9 installer configured with this: VCF 9.0 Offline Depot using Synology (williamlam.com).

How to download the binaries with the VCF download tool

Let’s download the binaries with the VCF download tool within WSL.

  1. First, download the VCF download tool from the Broadcom support website. It’s packed as a tar.gz file.
  2. Upload the package to your WSL environment (if your WSL is running, you should see the mounted disk in Windows Explorer. Nice, isn’t it?).
  3. Next, switch to your WSL command line and extract the package to a folder somewhere on the disk.
    Notice: You may have to create a folder first

    tar -xvf vcf-download-tool-9.0.0.0.24703747.tar.gz -C /path/to/folder/
  4. Now, navigate to the /bin subfolder within the extracted folder 
    cd /path/to/folder/bin
  5. As mentioned earlier, a valid download token is required to download the binaries. Create a text document (downloadtoken.txt) somewhere on the disk and put your download token in one line. Only the token itself, nothing else. We will reference that file when issuing the download command.
  6. You may create a specific folder where the binaries will be stored, at least I’m referencing to such a folder in this guide.
    Notice: Make sure you have enough free disk space on your Windows computer.
  7. Next, still in the /bin subfolder of the extracted VCF download tool, execute the following command to start the download of the binaries: 
    ./vcf-download-tool binaries download --vcf-version 9.0.0 --automated-install -d /path/to/downloadfolder/offline-bundles --depot-download-token-file /path/to/downloadtoken.txt
  8. If everything is configured correctly, you should get a notification to either enable or not the customer experience program. After that, the download should start automatically.
    It should look somewhat like the screenshot below:

When I was downloading the binaries, something went wrong, and four downloads have failed. No worries if something like this happens. Just executed the same command again. The VCF download tool will start again, check what’s already downloaded, and also notice what failed to download. Only these binaries will be downloaded again:

After executing the download command again, all was fine:

One word to disk space. Shortly before the download starts, the VCF download tool will tell you how much disk space is needed, and will then immediately start downloading the binaries. So, really make sure you have enough “physical” disk space on your Windows computer, so that the WSL disk can automatically grow as needed.

Next, I’m going to prepare my Synology NAS as an offline depot and upload all the binaries to it. The ESXi hosts have been reinstalled from scratch already with ESXi 9, but have no configuration yet (except the DHCP IP address).

Bye for now 🙂

Back to the Mothership!

by

My last blog post was from February 2024. That was over a year ago (at the time of this blog post, which you are currently reading). Far too long! And now I’m writing a new blog post titled “Back to the Mothership.” What does that mean?

I’m glad you asked.

What happened?

Towards the end of March, I lost my job. I was laid off. It was quite a surprise and also a bitter disappointment. Especially when there had been no prior warnings or any discussions with the boss or anything like that. And especially when the Outlook invitation was “Personnel / Goal Setting” (that sounds promising, right?). And when someone from HR is also in the meeting room, you can put two and two together.

But that’s in the past now. I’m over it. Done and forgotten (but not forgiven). I will miss the team I was privileged to lead there. Including me, half the team has resigned or was resigned within three weeks.

Wow, bummer, what now?

I have spent a lot of time with my family in the last two months since I was laid off, spending a lot of time with my son and my wife. And I’ve also been able to get a lot of things in and around the house in order. So I cleaned all the windows (finally, once again after a few years), repaired the robotic lawnmower and gave it a new garage, and did a lot of those little things that take time and that I’ve been putting off until now.

But I also took time to figure out what I really wanted professionally. I enjoyed some good discussions with my wife (she is an emergency doctor, so she knows urgent and special situations), and also with good friends about my situation. What are my strengths? What skills do I have? What is my expertise? Is computer science (well, I mean the big IT), especially the field of virtualization and technology, still my passion? And the answer to this last question is clearly “YES”.

And what’s that with the Mothership now?

Now I can let the cat out of the bag, since the contracts are signed. I’ll be starting a new position as a Senior System Engineer in the VCF stream at the company “soulTec” on June 2nd. I will work a lot with everything related to vSphere, datacenter, a lot of customer projects, both onsite and remote, and I will learn a lot! I’m so excited and anxious, I can’t wait to show up at the office for the first time on Monday.

Technology with passion!

An ode to joy – or why you don’t have to mourn the free ESXi

by

An ode to joy, or why you don’t have to mourn the free ESXi. An obituary.

But before I go into depth here, I would like to declare the following:

This blog post is reflecting my personal opinion. Based on my knowledge and experience, as well as the things I have read on the internet and heard from people directly and discussed with them, I have written this blog post.

The title of this blog post says a lot. At least I think so. And I’m glad you stumbled across this blog post or clicked the link wherever it has been posted. In the last few days, there has been a lot of reading about the fact that VMware by Broadcom has canceled the free ESXi hypervisor. And yes, that may not be the fine English way to mash such a product. I don’t even know if there is a “good” way to do something like that.

Yes. It’s bad that the “free ESXi” has been canceled. At least according to what I read about on X (Twitter) and some IT news portals. People were (and still are) angry about this move of thinning out the product portfolio. I never used this free version, and I wasn’t even aware that this was such a big thing. But that doesn’t count here.

Yes. There are alternative hypervisors, like Hyper-V, Proxmox, and others, and they are free. You can run virtual machines even on storage systems like TrueNAS or Unraid (the last one does cost some bucks). And a free hypervisor, I don’t care about the flavor, might be just enough for some people. They may run a homelab with some VMs, maybe some production services for their home network like Pi-Hole, to name one.

But this is it.

Read more