VMware – Homelab storage extension installed

28. December 2016 by Karl Widmer

Recently i ordered the last piece of hardware for 2016 for use in my VMware vSphere homelab. I failled in the fourth VCP exam in December 2016 and that gave me the kick to extend my homelab a little, and look into storage stuff in detail.

Thoughts and requirements

I had some ideas in mind and received good inputs from my fellow homelab colleagues, but there are so much possibilities for extending storage. There are various NAS manufacturers and storage vendors. You can “extend” your storage even virtually with some virtual storage appliances. But i have to keep my budget small, well as small as possible for my needs. I don’t have a sponsor (would be nice indeed). So for the extension of my homelab any storage device other than a NAS costs way too much money. And i want to use real physical existent storage, so also a no-go for virtual storage appliances (which also requires some physical storage in the back end). This made the field of choice at least a little smaller, not much, and i’m still kicking out some devices to find the one which suits my needs the best.

Another point is network connectivity. My decision was to have four network ports on this specific NAS device. It should support link aggregation, load balancing and failover. The NAS device should also support NFS and iSCSI protocols so i can reach it from my ESXi hosts and use it. It would be the best for the integration into my homelab when i’m already familiar with a specific kind of device / operating system / manufacturer. Yes, i know, that’s not a real decision maker, at least not the best. But why struggle if there exist easy to setup systems? And last but not least it should be supported within VMware, for example with VAAI.

With all this points from above i decided to go for a Synology NAS device.

The hardware

The base system is a Synology DS1515+ NAS device. The technical specifications:

CPU Model	Intel Atom C2538
CPU Architecture	64-bit
CPU Frequency	Quad Core 2.4 GHz
System Memory	2 GB DDR3
Memory Expandable up to	6 GB (2 GB + 4 GB)
Drive Bay(s)	5
Hot Swappable Drive	YES
RJ-45 1GbE LAN Port	4
VMware vSphere 5 with VAAI	YES
VMware vSphere 6 with VAAI	YES

Details specifications are available here: Synology DS1515+

Disks (capazity / cache)

I ordered also three WD Red SATA disk with 4TB each and two Sandisk X400 SSDs with 512GB each. In this configuration i’ll get enough raw storage space (roughly 8TB usable capazity). With two SSD in a Synology multi-bay NAS i can also configure read-write cache (you’ll get read cache only with one SSD).

So let’s get our hands on the hardware…

Veeam – Backup your Office 365 mailboxes (on a UNC path)

15. December 2016 by Karl Widmer

Today i’d like to provide you another one of my straight-forward walk through guides. Everybody is talking about cloud. Is it either a private cloud or a public cloud, or probably something between (a hybrid cloud, you don’t say…). So many people and companies want to have their stuff migrated to the cloud or are already using cloud technologies.

Microsoft Office 365 is also one of these cloud technologies. It’s your mail hosted in the cloud. It is available in different sizes and versions, combined with powerful office and collaboration applications (Word, Excel, Outlook, Skype for Business and Sharepoint Online and so on) or just the mail part of it. You can choose whatever you need, and at very attractive pricing conditions.

Compare the Microsoft Office 365 business plans here:

https://products.office.com/en-us/business/compare-more-office-365-for-business-plans

But today we don’t talk about how cool and great this Office 365 stuff is. We wanna talk about how you backup your mails. Your mails are stored somewhere in the cloud (actually a physical datacenter located somewhere). I don’t know if Microsoft does backups for YOUR mails. It’s always better to do that by yourself. Then you know it’s working and you can recover in case of a user deleted something, or whatever can happen with these mails.

So let’s talk about Veeam Backup for Office 365.

This nice piece of software from Veeam is now available since a few days. It’s working very fine, and it is simple to install as you know that from other Veeam software. As mentioned above, this post should guide you through the simple setup and configuration of Veeam Backup for Office 365. When i don’t count the time it took to take screenshots, then the installation is done in a few minutes. Promise. Let’s dig into that.

Windows 7 – No access to UNC path or network drives

14. December 2016 by Karl Widmer

Today i had again some of the problems worth writing about it afterwards. A customer called because of a strange network issue on a computer in his network. He can’t access the network drives. They are connected but he can’t access them. Also no access to UNC paths (like \\server\sharedfolder). Both ways he receives an “Access denied” error.

So i started with some tests just to make sure we are talking the same and if at least the basics on the computer, server and network are fine. It was one of the weirdest problems i have had the last few weeks. And the solution was so simple and cheap that it is embarrassing. But i don’t want to anticipate it. Let’s have a review.

The following list describe the troubleshooting steps which we did today. I’ll hope that this “checklist” (well, actually more a list of steps i tried) will help you if you are in the same situation as i was (or probably in a situation close to this).

Client steps

checked network configuration with “ipconfig /all”
- everything fine (DHCP, all expected values were correct)
checked DNS lookup
- resolving server hostname, domain, different websites, all good
nslookup (reverse DNS lookup)
- IP addresses are resolved to hostnames

So basically on the client it looked all good. DNS was fine, internet connection was good. But we still don’t have access to UNC paths or network drives.

Server steps

the same as above (DNS / reverse DNS, generall network configuration, all was fine
checked permission for the specific user on there shares, permissions were granted correctly

DNS or permission issues are common in most IT environments. But at this time it wasn’t the root cause of this problem. Because we checked DNS and permissions and it was all fine. Let’s dig a little deeper.

Further client steps

on another computer we tried to login with the user which the customer called for
- that worked all fine, no issues
on the affected computer we tried to login with another user
- that worked, but the issues were the same (neither access to UNC paths nor network drives)

So now we knew that there has to be a computer related issues. Nothing with the user profile, no permission conflicts or DNS errors. But what the hell can cause this problem?

More client steps

Just to make sure there is no software causing some issues, we tried several things.

uninstall and re-installation of the antivirus program
- did not help
checked other software that might could cause suche problems
- we uninstalled some old pre-installed software
- checked which Windows update were installed recently
- we did not find any suspicious updates nor software which could cause something like that

The customer was in a hurry because he had to leave after lunch time (which we both didn’t had today). But he will call me in a few hours he said. So i ended up like a donkey at a five-barred gate. I can’t imagine were this issue has it’s root cause. All checked were good, all settings were correct. But just with this computer there is still no access to UNC paths or network drives.

A few hours later…

Later this afternoon the customer called me when he was back in the office. He said he don’t have that much time. So, let’s do this, computer. Let’s finally solve your problem.

The second last attempt was to remove this computer from the existing domain, delete the computer object in Active Directory and to join the domain again.

leaving the domain worked fine
no problems deleting the computer account
after the necessary reboots the computer wasn’t able to join the domain
- “Could not join the domain. The network path was not found”

Damn, what have i done? What should i do now? Come on, it can’t be that hard! One last attempt i had in mind. What about to uninstall the network card of this computer? It can’t get any worse. Let’s try that!

uninstalled the network card via Windows device manager
- also checked the box to delete driver software for this device
let Windows find a driver
- surprisingly Windows found a driver (it was four years older than the previous)
- network connection comes up again
restarted the computer

The customer tried to login after the last restart. Then we checked the network drives, and here they are! Access granted! A quick check with the UNC paths was also working now. Uninstalling the network card, deleting its software and letting Windows search for a driver did the trick.

The customer was happy today (the computer is for the secretariat, the user was out of office today) and the user will be happy tomorrow 🙂

Veeam – Configure Azure Direct Restore with Veeam 9.5

8. December 2016 by Karl Widmer

Since the direct restore to Azure was announced with Veeam version 9.5 i wanted to test that and to get some hands on. But there is always either lacking of time or Murphy, which both are against me. This week I’ve been digging into this topic. I wanted to create a working configuration for a direct restore to Azure. Let me tell you this fairy tale…

I’m not completely new to cloud computing at all. I did some first steps with Amazon AWS and my first website, i tried out Microsoft Azure when it was relatively new, and i had also some tryouts with VMware vCloud Air. But all was just for testing, looking how that stuff works. No big deal at all. Now with Veeam version 9.5 this new feature called “Direct Restore to Microsoft Azure” comes in handy.

Before you try to just configure the direct restore in Veeam Backup & Replication 9.5 there are some requirements you have to fulfill, otherwise you will do many of the setup assistants twice or more (like i had to do because i didn’t know what exactly are the requirements).

Requirements

Veeam Backup & Replication 9.5
- Supported in Standard, Enterprise and Enterprise Plus
Microsoft Azure Account
- Pay-as-you-go or any other subscription based account
Azure Storage (blob or general storage)
Azure Virtual Network

Hint: if you organize all of this stuff above in the same ressource group on Azure you will have a better overview (at least then if you are already a heavy Azure user and if you’ve got lots of things on your Azure dashboard).

With this basics above configured we are ready to set the things up in Veeam Backup & Replication.

VMware – vSAN Deploy and Manage course – Day 3

7. December 2016 by Karl Widmer

Today it was the last day in our VMware vSAN Deploy and Manage course. Nevertheless today we have given everything again. We had a deep dive in designing vSAN solutions, we discussed the key topics in design decisions and also played around with some what-if scenarios. But as every day we kicked off with some review what we discovered yesterday, and again to make for everyone clear what vSAN really is.

Day 3

Daily review

What is vSAN:

Software Defined Storage
Hyper Converged Infrastructure
Network Storage Topology
Hypervisor integrated
- That means less latency
- no dependencies on VMs
- support
- distributed
local disks presenting one datastore per cluster

Use cases:

VDI (licensing, offload of IOPS, scalable)
Test / DEv environments (projects, easy, growth)
Branch Office / Remote Office (same solution, backup)

Install vSAN:

Simple, with GUI, all from the web client (with just few clicks)
install vSphere
create a Cluster
set a VMkernel for vSAN
disable HA
claim disks
- create disk groups
- claim them as cache / capacity tier
enable vSAN

What’s in the default vSAN policy:

FTT = 1
Stripes = 1
No reservation (neither cache nor capacity)
Thin provisioning

What is a Fault Domain:

an area which may can fail
plan to recover impact of Ops
- Rack awareness
- Site awerness

Availability:

In vSAN there are two states of compliance…

Compliant
non compliant
- Absent => wait for 60 minutes, then rebuild
- Degraded => rebuild immediately

What-if failures:

Cache disk fails => lose disk group => latency increases
Capacity disk fails => degraded => rebuild => VM back online
Controller => host issue => HA response
Host outtage (complete loss of host) => HA response => VM response

Module 7 Lesson 2 – Troubleshooting

Some topics we covered already yesterday. Today it was also some repetition and a quick overview about troubleshooting and some of the tools we discovered yesterday. There are so many tools for troubleshooting available, either already built-in or community driven, i think the list could be longer. But at least some of the most known tools i will provide you with this list.

vCenter (you don’t say…)
vROPS
esxtop
Wireshark (yes indeed; capture packets on ESXi and analyze them with Wireshark => pcap)
vSAN Observer (based on Ruby)
RVC (Ruby vSphere Console)
Health Service Plugin
vCheck
PowerCLI scripts (combined with Onyx)

A cool tool indeed is vCheck. It’s based on Powershell scripts and runs against your vSphere infrastructure (there are scripts for other stuff too). You schedule the scripts and you can reveive notifications about changes, issues (before they become a real deal). So when you arrive in your office you already know what’s going on (or what’s not). Also worth to mention is vSAN Observer. It’s already there, just start it and access the built-in webserver to get an overview what’s going on in your vSAN environment.

Module 8 -Stretched Cluster

After doing some work in the labs we talked about design. And having a stretched cluster is also a question of design, how to create a solution which covers rack outtages or even a complete site outtage. You can do that with a stretched cluster. And the failover happens automatic (what may probably not the best solution in every fail over situation…).

When planning a stretched cluster you have to concern about resources. You need 50% spare capacity on both sites (talking about two racks or two sites) in HA admission control. Imagine that one site / rack should keep the other one online, and the stuff which is already running on the secondary site too.

You don’t have to use SRM (Site Recovery Manager) for a failover. vSAN does that for you automatically. If you use SRM then you have to have a recovery plan for each and every VM. Thats a lot of planning and even checks if there are new or changed VMs. Not to think about the costs. You need SRM licenses and a second vCenter license.

Talking about the vSAN witness. A witness is a separate ESXi box. This can be a physical server with ESXi which needs to be licensed. This physical server can’t be a member of a cluster, but it can run some VMs on it. Or you can get a witness appliance, which represents a special ESXi as an appliance, which runs on a ESXi server. This appliance cannot run VMs on it.

You can have a ROBO vSAN cluster in your remote office / branch office which consists only of two ESXi hosts in this cluster. If you’re doing so you have to have a witness host / appliance in your main office site. You always need somewhere a witness to have the quorum in case of an HA event. And remember the 5 heartbeats. In the case of an outtage, after 5 missed hearbeats your host is gone and a failover happens.

Module 10 – Designing a vSAN deployment

That’s not a random list of IT buzzwords, folks. You have to consider these key points when you’re designing a vSAN solution (probably any other scalable solution too).

Availability	Management
Managability	Virtual machines
Performance	Compute
Recoverability	Network
Security	Storage

Let me give you some more things to consider. In the way of designing a vSAN solution you will have to find answers to these questions. Some answers you will get from your customer when talking with him about a solution for his specific needs. Some other answers you will find when you design the solution. And you will find some more questions too…

Requirements (must have / be / do)

“RPO of 15 minutes”
“RTO of 5 minutes”
Location of data / data center

Constraints (design decisions)

“Must work with existing network hardware”
“Must work at this site”

Assumptions

“We have enough bandwith”

Risks

“If the bandwith is not enough => risk of not meeting the SLA”

If you covered the topics above (and the bullet points are just ideas, there are lot more to cover) then you will proceed with the design.

Conception

What the business wants
High level design (HLD)
Should be vendor independent

Logical

“Keep data at this location”
“We want two sites, one for failover”
Should also be vendor independent

Physical

Here you can come in with the vendor and create the solution
vSAN here, stretched cluster there
Network links from here to here
Backup then with this
and so on…

And if you are searching some benchmark tools for your newly created solution, there you go:

HCIBench (https://labs.vmware.com/flings/hcibench)
- simplify and accelerate customer POC performance testing
- not only a benchmark tool designed for Virtual SAN
- evaluate the performance of all kinds of Hyper-Converged Infrastructure Storage in vSphere
HammerDB (http://www.hammerdb.com/)
- open source database load testing and benchmarking tool
- Oracle Database, Microsoft SQL Server, IBM DB2, TimesTen, MySQL, MariaDB, PostgreSQL
- Postgres Plus Advanced Server, Greenplum, Redis
- Amazon Aurora and Redshift and Trafodion SQL on Hadoop