vExpert

How to integrate the Synology VAAI Plugin into the vLCM Image?

by

I had a customer support case recently where the customer was struggling with updating a vSphere cluster. The customer was already using the vSphere Lifecycle Manager (vLCM), but the composed image always returned the error that the hosts were not compatible. So he asked for help.

We knew the installed hardware, we knew the installed (and targeted) ESXi version, and we also knew the currently installed and targeted version of the vendor addon, too. But we were both a bit perplexed. Until we took a closer look at one of the “incompatible” hosts and the error message that revealed the cause of this incompatibility.

The error message told us that vLCM should obviously downgrade something, but couldn’t. Thanks to the listing of what’s currently on the host and what’s in the image of the desired state, we could find out what the issue is. And there were some “components” that differed in the versions. And we saw that the ESXi version was fine, and the image is more up-to-date than the host, and the same for the vendor addon. But we saw that two or three drivers were a little above on the host than in the image. We couldn’t explain this. But with the possibility of manually adding “components” to an image, we were able to fix that.

vSphere Lifecycle Manager (vLCM) shows an error
Source: knowledge.broadcom.com

Long story short, in the vLCM image, you cannot only choose the desired ESXi version and vendor addon, and if you have the hardware support manager (HSM) installed, the latest server firmware. But you can also add specific drivers and updates manually to the image. And you can even upload some updates to the vLCM.

And exactly the last step, uploading a plugin and manually adding such “components” is the topic of this blog post. Let’s dive into it.

Disclaimer

The following steps are related to hardware and software, which may or may not be on the hardware compatibility list (HCL). I’m showing screenshots from my homelab, where I don’t care much for compatibility. It is more kind of “the main thing is that it (somehow) runs and works and does what I want.”

Please use it at your own risk, and probably not in production, except the vendor and / or Broadcom have it on the compatibility list and everything has been checked for being properly supported.

How can I upload a plugin to vLCM?

Well, I’m glad you’ve asked. That’s pretty simple. But before we move on, you may know what plugin I’m using for my homelab and as an example for this blog post. It is the Synology VAAI plugin. You get more information on how to install the plugin (the manual approach), and also the download link, from the Synology website here.

But what is VAAI?

VAAI, short for “vStorage APIs for Array Integration,” is a feature introduced in ESXi that provides hardware acceleration functionality. It enables your host to offload specific virtual machine and storage management operations to compliant storage hardware. With the storage hardware assistance, your host performs these operations faster and consumes less CPU, memory, and storage fabric bandwidth. (Source: knowledge.broadcom.com – FAQ for vStorage APIs for Array Integration).

And now, let’s continue. I assume that you’re already logged in to the vCenter web client.

  1. In the vSphere web client, click the burger menu in the top-left corner. Then, click the “Lifecycle Manager” entry.
  2. On the top menu, click “ACTIONS“, then “Import Updates”

    Notice: it doesn’t matter on which tab you’re on in the Lifecycle Manager; you can import updates from any tab.
  3. Next, click “BROWSE” to open the file dialog, and search for the plugin you’d like to upload
  4. Next, click “IMPORT” to start the upload of the plugin to the vLCM library

And we’re done for the first part, we successfully uploaded a plugin to the vLCM library. We continue now with using this plugin in a vLCM image.

How to add a plugin to a vLCM image?

  1. As we’d like to create an image with this plugin included, navigate to the “Image Library” tab, and click “CREATE IMAGE”
  2. We start by providing an image name and choosing the desired ESXi version.
  3. To add the recently uploaded plugin, you may have to click “SHOW DETAILS” first, and then click “ADD COMPONENTS”
  4. In the following dialog box, you can search for the plugin. Just click the filter icon in the “Component Name” column and enter the plugin name
  5. Now select the plugin (check the box in front of it), then click “SELECT” in the lower-right corner
  6. We can now see the added plugin in the list of components. Next, click “SAVE” and you’re done with creating the image

You can now move on and assign this image to your cluster and start updating the hosts. Have fun!

Using the VCF Download Tool on Windows Subsystem for Linux to download VCF 9 binaries

by

You may have read it here first, but I recently started a new job. And I’m going deep down the rabbit hole, the VMware Cloud Foundation rabbit hole. I recently sat my first VMware Cloud Foundation exam (VCF-VCP administrator) and passed. The needle for this had been in the arm for a long time if one can put it that way.

VCF version 9 has just been released recently too, and many people are trying to get it up and running in their homelab. And I’m no exception. Getting the stuff done is not as easy as it was back in the days with setting up the ESXi hosts, installing vCenter, and then configuring the vSphere environment as it should be (storage, vSAN, distributed virtual switches, etc.).

I’m also not yet quite sure if my whole homelab infrastructure (talking about the Supermicro servers here, and the HP workstation here) has enough resources. It should be enough when doing the math in terms of CPU cores, amount of memory, and storage. But when going deep technical, with CPU specs, etc., that might be a killer. We will see.

Installing the new VMware Cloud Foundation (VCF) version 9 has been simplified somewhat compared to version 5. There is now an installer available to be deployed as an OVA file, which then pulls all the required files (like ESXi installers, vCenter ISO, and all that stuff) automatically, either from an online depot or an offline depot. No matter which way you go, a valid download token is required to pull the installer files.

A quick word on the VCF download tool

The following guide should help you download the binaries from Broadcom through the VCF download tool, running in Windows Subsystem for Linux (WSL). I’m assuming that you have already a WSL environment with a supported Linux version set up and running on your computer. The VCF download tool is designed to run on Linux. The only way to get it working on Windows is to use the WSL environment. Some information (not much, to be honest) is available here on the Broadcom support website: VCF Download Tool under Windows OS (broadcom.com).

A sidenote on the offline depot

One cool thing I stumbled across during my first steps is a blog post from William Lam, where he is setting up a Synology NAS as an offline depot. You may check this out if you have a Synology NAS, that would be a great fit for having the VCF 9 installer configured with this: VCF 9.0 Offline Depot using Synology (williamlam.com).

How to download the binaries with the VCF download tool

Let’s download the binaries with the VCF download tool within WSL.

  1. First, download the VCF download tool from the Broadcom support website. It’s packed as a tar.gz file.
  2. Upload the package to your WSL environment (if your WSL is running, you should see the mounted disk in Windows Explorer. Nice, isn’t it?).
  3. Next, switch to your WSL command line and extract the package to a folder somewhere on the disk.
    Notice: You may have to create a folder first

    tar -xvf vcf-download-tool-9.0.0.0.24703747.tar.gz -C /path/to/folder/
  4. Now, navigate to the /bin subfolder within the extracted folder 
    cd /path/to/folder/bin
  5. As mentioned earlier, a valid download token is required to download the binaries. Create a text document (downloadtoken.txt) somewhere on the disk and put your download token in one line. Only the token itself, nothing else. We will reference that file when issuing the download command.
  6. You may create a specific folder where the binaries will be stored, at least I’m referencing to such a folder in this guide.
    Notice: Make sure you have enough free disk space on your Windows computer.
  7. Next, still in the /bin subfolder of the extracted VCF download tool, execute the following command to start the download of the binaries: 
    ./vcf-download-tool binaries download --vcf-version 9.0.0 --automated-install -d /path/to/downloadfolder/offline-bundles --depot-download-token-file /path/to/downloadtoken.txt
  8. If everything is configured correctly, you should get a notification to either enable or not the customer experience program. After that, the download should start automatically.
    It should look somewhat like the screenshot below:

When I was downloading the binaries, something went wrong, and four downloads have failed. No worries if something like this happens. Just executed the same command again. The VCF download tool will start again, check what’s already downloaded, and also notice what failed to download. Only these binaries will be downloaded again:

After executing the download command again, all was fine:

One word to disk space. Shortly before the download starts, the VCF download tool will tell you how much disk space is needed, and will then immediately start downloading the binaries. So, really make sure you have enough “physical” disk space on your Windows computer, so that the WSL disk can automatically grow as needed.

Next, I’m going to prepare my Synology NAS as an offline depot and upload all the binaries to it. The ESXi hosts have been reinstalled from scratch already with ESXi 9, but have no configuration yet (except the DHCP IP address).

Bye for now 🙂

An ode to joy – or why you don’t have to mourn the free ESXi

by

An ode to joy, or why you don’t have to mourn the free ESXi. An obituary.

But before I go into depth here, I would like to declare the following:

This blog post is reflecting my personal opinion. Based on my knowledge and experience, as well as the things I have read on the internet and heard from people directly and discussed with them, I have written this blog post.

The title of this blog post says a lot. At least I think so. And I’m glad you stumbled across this blog post or clicked the link wherever it has been posted. In the last few days, there has been a lot of reading about the fact that VMware by Broadcom has canceled the free ESXi hypervisor. And yes, that may not be the fine English way to mash such a product. I don’t even know if there is a “good” way to do something like that.

Yes. It’s bad that the “free ESXi” has been canceled. At least according to what I read about on X (Twitter) and some IT news portals. People were (and still are) angry about this move of thinning out the product portfolio. I never used this free version, and I wasn’t even aware that this was such a big thing. But that doesn’t count here.

Yes. There are alternative hypervisors, like Hyper-V, Proxmox, and others, and they are free. You can run virtual machines even on storage systems like TrueNAS or Unraid (the last one does cost some bucks). And a free hypervisor, I don’t care about the flavor, might be just enough for some people. They may run a homelab with some VMs, maybe some production services for their home network like Pi-Hole, to name one.

But this is it.

Read more

Storage performance upgrade – and how to make use of Vmknic binding

by

It’s been a while now (exactly two years and one month), since I bought my last Synology NAS, both for homelab and personal/private usage. Why did I buy a new NAS? There are some requirements that my new NAS had to fulfill:

  • Capable of 10 Gigabit Ethernet (integrated or with an addon card)
  • Enough slots for disks (at least five)
  • Two NVMe slots for using NVMe SSDs as read/write cache (integrated or with an addon card)
  • Can be used for homelab storage, as well as backup storage for home computers, laptops, etc.

When looking for that one NAS that could handle all of the above, I stumbled across the Synology DS1621xs+. Synology launched it in September 2020, and I ordered it about two months later.

Some of the technical specifications of the NAS:

  • It has 2 x M.2 2280 NVMe slots
  • It comes with 8 GB DDR4 ECC SODIMM (expandable up to 32 GB, and yes I upgraded to 32 GB)
  • It is equipped with an Intel Xeon D-1527 4-core 2.2 GHz CPU that has a Turbo Boost up to 2.7 GHz
  • In terms of connectivity, there are two 1 GbE and one 10 GbE RJ-45 ports
  • If you fancy, you could install some PICe card (the NAS has one 8-lane x8 slot), but I don’t know why or what
  • And you can install up to six 3.5″ or 2.5″ drives

So that’s the basics covered. Let’s get back to the topic.

Read more

vExpert Applications are open

by

The vExpert applications for 2024 are open now. Nice! But what is this?

Some vendors have special programs for the IT folks around the world. These programs are designed for ambassadors and evangelists. Microsoft with its MVP, Veeam with the Vanguards, and many more.

As benefits, you may get early access to beta versions, marketing information, they help you promote your blog posts and many other great things. The VMware vExpert program is VMware’s global evangelism and advocacy program.

There is no certification exam for becoming a vExpert, no course requirement. It is kind of an award or an accreditation. You have to apply for the vExpert program, and a committee will review your application, and then accept your application or reject it.

Do I have what it takes?

Yes! The vExpert program is about “giving back”. You work in your job, you do great things, and help your users and customers. You’ve got experience in IT technology and working for some years already in that area. You may work at VMware, at a partner, or you may be a VMware customer.

Giving back means sharing your knowledge. Did you fix a problem? Blog about it! Did you set up a new solution for a customer? You may speak about it at VMworld! If you’re not a writer, then there are various groups on social media, or you can find forums like the VMTN (and many other non VMware related forums). There are a lot of users, both partners, and customers, asking for help. Help them with your experience and knowledge! You may have written a book about VMware solutions, or you’re focusing on a specific VMware product. Maybe you are a public speaker or a VMUG leader. You have the knowledge, you have experience, share it!

How can I apply?

It is very easy to apply. There are two applications per year. One is starting during the summertime, and one is starting in early December. Both application windows are open for some days. After this period, the committee will close the applications and start their voting. This may take some time too. The results for the December applications are usually announced in February, the summer application will be announced somewhen in August.

If you’re already a vExpert or even a VCDX, you have to reapply. There are also vExpert sub-programs, like Application Modernization, HCX, Cloud Management, and many others, that you can apply for if you’ve been awarded a vExpert.

The December vExpert applications are open now, and the results will be announced in February 2024 (my guess). And it might be one of the most eagerly awaited emails, at least if you read about it on some blogs, or see it popping up on Twitter.

Click here to apply for vExpert 2024!

Can I get help?

Yes! VMware announced the vExpert PRO program. There are more than 100 vExpert PROs around the world that are helping you. They can help you with mentoring, tell you what you have to do, and outline the process of the applications. Just reach out to the vExpert PRO close to your area!

What benefits can I get from the vExpert program?

If you check the list on the vExpert website, you can see that there are a lot of benefits. I’m not listing every point here, but I’d like to mention at least some of them.

You will be part of a global network roughly 1600 other vExperts. They are on social media, Twitter, active on blogs and forums. There is also a Slack channel for vExperts. You’ll get a certificate signed by the VMware CEO. One of my personal favorites is the access to 365 day eval licenses for most of the VMware products, like vSphere, vSAN, etc. Throughout the year, you got the opportunity to join private webinars with VMware partners. You will be featured on the vExpert Directory. Also before VMware Explorer (former VMworld), there are pre-launch briefings for bloggers, which means that you may get a sneak peek at some products or announcements. Also, at the VMware Explore event, there will be vExpert parties both in the US and Europe locations. And I tell you, these parties are awesome!

My personal view

I’m working for about 23 years in IT now, including my apprenticeship. I learned many things, and I’m still learning. I had the chance to gain insights into different industries during my work. When I was awarded a vExpert for the first time, I felt honored. It meant a lot to me to be one of only a few hundred to receive such an award. It showed also that my efforts are worth it, and that they have been noticed. I was working on a partner level for many years, helped my customers to build their infrastructure, helped them with troubleshooting. Now, I’m working at a customer, fighting with daily business problems, architecting data centers, migrating workloads, and designing cloud concepts. I just recently applied to a new job and switched my focus from managing datacenters to managing VMware Horizon VDI workloads. But I still have strong bounds to the datacenter, as I am an infrastrucutre guy, and will ever be an infrastructure guy.

Some people kept smiling at my vExpert status because they didn’t believe in certifications, accreditations, etc. But that didn’t bother me. I did my thing. I wrote blog posts, and I still write (even if not as often or regularly as I want). I’m active in forums and help people with troubleshooting. And I’ve always stood up for one thing. I’m grateful for becoming a vExpert, it meant a lot to me, and it still does.

Personally, I have the feeling that, especially in contact with technical support, it enables a different level of communication if the other person recognizes my status as vExpert. A problem may escalate upwards faster, and you may have faster access to 3rd level support. Things like this. But also during projects, when I’m in contact with internal customers and external consultants, they recognize my status and see that I’m not just some random IT technician or IT dude. I know some stuff, I’m experienced, and someone voted for me so I could become a vExpert. As mentioned, it’s not a certification exam you can learn for. It is the personal commitment that shines through here.