No vMotion possible after ESXi host BIOS update

by

I was working on some ESXi upgrades recently. We’re currently preparing everything to make the upgrade to vSphere 7 somewhen smooth as silk. That means that we’re rolling out vSphere 6.7 on all of our systems. Recently, I was tasked to upgrade some hosts in a facility some hundred miles away. The task itself was super easy, managing that with vSphere Update Manager was working like a charm. But before the vSphere upgrade, I had to upgrade the BIOS and server firmware to make sure that we’re fine with the VMware HCL.

The second host was done within one hour and received the complete care package. But the first host took a bit longer due to unforeseen troubleshooting. I’d like to share some helpful tips (hopefully they’re helpful).

What happened?

As mentioned, upgrading the ESXi host through the vSphere Update Manager worked like a charm. But before that, I booted the server remotely with the Service Pack for ProLiant ISO image to upgrade the BIOS and firmware of that server. Also, that went very well and. As there are two ESXi hosts at this location, we had shared storage available and we were able to move the VMs from one host to the other without further issues. One host placed into maintenance mode, upgrade, remove from maintenance mode, and the same for the second server. That was the idea.

But unfortunately, the gods of IT had something different in mind. After upgrading the first host, we tried to move the VMs back to this host to prepare the upgrade for the second host. Well, some VMs were able to be moved there, some were not. But why?

When we move some particular VMs back from the second to the upgraded host, we received the below error:

That made us curious. Why did this happen? When checking the tasks and events of that host and also the affected VM in vCenter, we didn’t find much information. After some internet research, we’ve found some possible causes. But not all of them didn’t fit our issue. So we had to dig a bit deeper. Thanks to the vmware.log file, located in the VM folder, we were able to find out the following:

Ok, that sounds funny that VMX has left the building, but not sure why. Seems to be a boring party…

Some more digging brought some more helpful information:

Obviously, the vMotion failed because some CPU features are different between the first (the already updated) and the second host. But wait, the two servers are the same model, with the same hardware configuration? How can that be?

The solution

That led us to the conclusion that it must be something with the VM compatibility level. But wait again, some movable VMs were on VM HW version 8, and the VMs with failed vMotion were also on VM HW version 8? Well, to say it here, we weren’t able to find the exact differences here. But that led us to two solutions. Either upgrade the VM HW version or install an ESXi patch. We decided to install the patch as we didn’t want to reboot some VMs (but we did it later).

And before you complain now, yes, I’m aware of the fact that the patch in the linked KB article is not the most recent ESXi build. It’s somewhat historical. When we started with the global vSphere 6.7 rollout, vSphere 6.7 Update 2 was the latest version available. And yes, we’re currently again planning new rollouts, as it was sadly neglected in the past. But you know, time and human resources, and change requests…

New Year – New Hosting?

by

To make the long story short: this week I moved my blog to a new web host. And I was surprised and pleased with how well and smooth everything went. But I’ll don’t just let you alone here. I’d like to explain the how and why I moved.

When I started with WordPress as my blogging platform, it was all just fun and games. Nothing technical, no helpful blog posts, just tinkering around, having fun writing. But with my engagement in the IT community, with my career in IT, I have rethought. I stopped playing around, and I started writing actual helpful blog posts. I started to write in German because this is my native language. At some time I switched to English, not without a reason (or more actually more than one). I have often dealt with English-speaking customers, with hotlines from international companies. And today at my current employer, English is the de-facto standard when opening tickets internally, or talking to other people in different time zones. I switched to English because most of the IT people I know, personally and on various social media, are native English speaking or understand English. Maybe I also switched because of reaching more people with helpful blog posts. And when I check the blog statistics, most visitors are from the United States. So, not a wrong decision at all, switching to English.

But enough of the forewords.

The why

When I moved my blog from one host to the other back in the days, I was looking for more speed. If you know WordPress, then you know it’s all PHP and MySQL, which can be highly dynamic content. And from a webserver perspective, dynamic content can’t be delivered as fast as static content. But, in my humble opinion, that was back in the days when there wasn’t much SSD storage in the webservers, or it was expensive, or with old PHP versions, etc.

My previous web host had also WordPress, but not the traditional way. He offered static WordPress hosting, which made me curious, and I wanted to give it a try. You’ll get a WordPress instance which you can start whenever you want, write your blog posts, do the other stuff, and shut it down. After that, you’ll create a so-called artifact, which renders all the dynamic content from your blog into static files. All your text, CSS, and JS files, images, etc. will then be put onto Amazon CloudFront automatically. And that’s the static content you’ll get presented when visiting the website. The performance was good, good enough for me.

But it has also some downsides. Some native WordPress features, like comments, search, or some plugins, just don’t work like this. They can’t be static because they relate to the dynamic WordPress in the backend. I had to find solutions for many problems. And I’m still not sure, even if I was able to test it successfully if it really worked.

I decided to move back to my original web host, where I have my domain running since 2008. Not only because of that but also because of the aforementioned circumstances. I’m also born in Switzerland, and my blog has a Swiss TLD (.ch), but with that TLD it’s still (highly) visible also international. My new web host caught me with a good hosting package, which has 250GB of website storage and is backed by 100% NVMe SSD, with Nginx and Apache running in parallel, 50MB of Nginx cache, and many other things. And that for a reasonable price, at least by Swiss standards. Yes, it may be true that many things are more expensive in Switzerland than abroad. But not everything.

That’s the why.

Read more

Setting up Visual Studio Code for WSL 2

by

Recently, I’ve published a blog post on how to set up the Windows Subsystem for Linux version 2 (WSL 2). I’m currently learning Ansible and I was searching for a solution that fits my needs in terms of usability, knowledge, etc. I’ve tested some Linux distributions, tried to connect remotely with my coding tool of choice, Visual Studio Code, but all were complex or didn’t work as expected. That’s the reason I gave WSL 2 a try.

I really like Visual Studio Code. It’s fast, supports a wide range of languages, and it’s free. Yes, free. And you don’t even need a registration nor a login to download it! VSCode also supports a variety of extensions. If it detects that you’re writing something in YAML, it might help you with a pop-up that there is an extension for it, for example, to properly highlight the syntax of that language. And that’s just one great example. With the combination of WSL 2 and VSCode, I’m able to write scripts (or playbooks in Ansible terms) and run them directly in the same tool. How cool is this?

Today, I’m going to show you how you can set up Visual Studio Code to use it with your already installed WSL 2 Linux distribution (at least when you read my previous blog post and followed the guide there).

Read more

Setting up Windows Subsystem for Linux Version 2 (WSL 2)

by

A few weeks ago I started getting familiar with Ansible. I’m far away from being an expert, and I’m probably not going so far anyway. But I want to learn some new things and train my skills. One quote which reminds me every day when I try and fail at something:

Even a lesson learned the hard way is a lesson learned.

Before getting deeper into Ansible, I had to find out how I can use Ansible, how I have to set up everything I need to get started. And it wasn’t easy. But I might have found a very convenient way. I’m not a Linux pro, but I know some things, and I’m flexible in learning new things. I have created the following guide for my own documentation, but hopefully, you find it helpful If you’re new to Ansible and you want to find out more like I wanted to do.

And before we dive deep here, I just assume that you already know that Ansible is an automation engine, driven by so-called playbooks. The playbooks contain your code (like for example, the instruction to search updates and install them in a specific Linux VM), which you then run against your infrastructure.

So let’s dive into the topics now. And yes, there are many guides available on the internet, showing you how to set up WSL 2. I’ve checked many of these guides during my initial setup tests etc. Unfortunately, most were not complete, others missed some steps (which means more research and tests needed). This guide has been developed and tested by myself, step by step, to make the setup of WSL 2 as easy as possible for you.

Read more

My homelab hardware gets its own rack

by

This project started a long time ago. When I planned the hardware needs for my homelab, I also thought of getting a rack. I had a real IT rack in mind, as you know it from your daily business, maybe back in the days when at least some stuff was on-premises and not everything in the cloud. I wanted to get a small rack with enough space to mount my whole homelab hardware into it, to have a proper cabling solution, and to have flexibility in case my homelab gets an extension.

But that wasn’t easy. There are various flavors of racks. The normal 42 unit IT rack, half-hight racks, and also various wall-mountable racks for patch panels, switches, and smaller devices. I was thinking and tinkering, looking for specs. But in the end, nothing satisfied me. Well, at least not from a price perspective, of being not able to transport it. And then, there was something going on on Twitter:

https://twitter.com/widmerkarl/status/1175392396974145542

Thanks to my colleague Michael Schroeder I’ve found something. He mentioned his IKEA rack, and that made me curious. Earlier in June, my colleague Fred Hofer announced that he moved his hardware into a bigger rack and that it was easier as when he moved from an IKEA Lack rack to the small rack:

https://twitter.com/Fred_vBrain/status/1267580223727550470

And that was the trigger! Why not building my own rack and tailor it to my needs? I don’t have to spend much money on a real IT rack, and I can do something handcrafted. The rack didn’t have to be anything special, there was not much in my personal specification book.

That’s the specifications planned:

  • Small (not full 42 rack units)
  • It should be lightweight
  • Enough space for at least three servers, some switches, and a NAS (or two)
  • Enough space for future homelab upgrades
  • Extensible, if needed
  • Should withstand some weight
  • Wheels!

The idea of building my own IKEA Lack Rack was born.

This whole homelab IKEA Lack Rack story will be covered in a small blog series. This blog post will start the series with some planning stuff, the first pictures, and the BOM, as far as I can provide it already. At least the BOM will be updated if there is a reason for it.

Read more