Benchmarking Virtual Desktops (VDI)

Benchmarking is a way to compare the performance of one device versus another using a standardized test. Benchmarking applications eliminate the human factor to measure only the performance of a device or application running on that device.

Benchmarking is strictly a device or application measurement. It measures performance of the hardware or an application on that hardware. It allows us to compare the relative performance of one hardware platform versus another. We can measure whether one platform performs better or worse than another. We can also use a benchmark to compare the performance of old versus new technology.

Benchmarking does not allow you to predict how well a person will perform using a particular piece of hardware. A talented employee might still work faster and accomplish more on a slower piece of hardware than another less skilled employee. One thing for certain is that giving the most talented most productive users better equipment will boost their performance even more. When the performance difference between old and new technology becomes too great then it is time to replace older equipment to regain the competitive edge. Benchmarks can be used to measure application performance and help us judge how well an application will perform on a specific platform. It will also allow us to compare the performance of the application between competing platforms. Benchmarks do a good job of eliminating the human factor which is hard to control.

While testing new technologies, and developing new ways to work, we constantly benchmark new products and new technologies. We measure overall performance as well as price/performance. We use benchmarks to give us an objective way to compare technologies and to give us an indication of how the technology will perform in the real world compared to what we are currently using.

WHAT DOES A BENCHMARK MEASURE?

There are many things we can benchmark. We can benchmark

Application performance
Storage speed
RAM speed
CPU performance
GPU performance

There are many commercial applications available that we can use to benchmark desktops and compare relative performance. Here are some we use:

PCMark 8
3DMark 11
7-Zip
wPrime
Cinebench
Geekbench
Revit RFO Benchmark – application benchmark
Cadalyst CAD Benchmark – application benchmark

Keep in mind that a benchmark does not precisely reflect real life usage. But it does give an indication of how well the same person will perform on one workstation versus another. In real life, you can never realize 100% of the speed difference between competing platforms. But you can be reasonably sure the same person working on a faster workstation will be faster. Even if the difference is minutes over the course of the day, time savings adds up over the weeks and months to yield real dollar benefits.

An application benchmark is used to compare application performance on one workstation versus another and eliminates the human factor. However, a person cannot work like a benchmark, so things that affect a benchmark might never affect the live performance of a person. There is always going to be a human factor, but an automated application benchmark can demonstrate and measure the true device performance difference.

USING BENCHMARKS TO SHOW ROI

For example, I evaluated a new workstation for my company that benchmarked 20% faster than my existing workstations. The new workstation was expensive at about $4500 each. Is it worth it? How long until I see a payback on my investment?

I calculated productivity first. I thought my staff might see 10% improvement in speed using the new workstation, about half the benchmarked speed difference of 20%. I knew a human could never capture 100% of the difference in speed.

Productivity Measurement

480 min / day (8 hours/day x 60 min/hour)
x 95% billable time target (allow for 24 minutes of wasted time/day, bathroom, water cooler etc.)
= 456 billable minutes / day
45.6 minutes / day saved (10% savings)
x $1.67 / minute ($100 /hour average staff billable rate)
===

$76.15 estimated savings per day per person in gained productivity by using a 20% faster workstation

Next I looked at the cost of the workstation and payback / break even calculation.

New Workstation Cost

$4,300 new workstation cost
$195 labor to install / swap out (3 hours @ $65/hour)
=$4,495 total new workstation cost

$76.15 savings / user / day
about 60 days to break even ($4,495 / $76.15/day)

So in about 3 months (20 work days / month) I could recover the cost of the new workstation paid by the gains in staff productivity. Even if I estimated incorrectly and my productivity gains were only half what I thought (5% versus 10%), I would still break even in about 6 months. If each machine had a usable life of 3 years (conservatively) then buying the faster workstation was a no brainer. I presented this calculation to management, got the ok, ordered the new machines and our productivity shot off the charts, not to mention everyone loved me for giving them a newer faster workstation.

VDI VERSUS DESKTOP PC

Advance2000 specializes in providing Cloud Based Virtual Desktops or VDIs (Desktops as a Service) to the AEC industry. Can a VDI show comparable performance to a physical high powered workstation? Can it really compete with hardware sitting under your desk?

Using benchmarks, we can demonstrate that a VDI workstation can go head to head in performance with a physical workstation. And, along with matching performance you also gain all the other benefits of a VDI, such as lower maintenance costs, faster to deploy, BYOD, and complete mobility. Is it time to rethink your hardware strategy and consider going 100% cloud based virtual workstations for your workstation needs?

Let’s take a closer look…

How does a virtual workstation (VDI) compare to a physical workstation in terms of performance?

In the last 3 years, VDI performance has improved for several reasons.

The first and most significant change was the development of vGPU or virtual graphics card. This allowed a VDI to use a physical graphics card shared among several VDIs. Using a real graphics card instead of a virtualized GPU in our VDI, we can greatly enhance graphics performance and eliminate the cursor lag that was characteristic of remote and virtual technologies for so long.
The second development was the constant improvement in the quality and availability of bandwidth. Bandwidth costs continue to drop and bandwidth speeds continue to improve with offerings like Google Fiber and Verizon FIOS. This allows companies to buy more and better quality bandwidth for the same or lower cost. Good bandwidth is essential to a good VDI experience.
Finally, hyper converged infrastructure, that is storage, computing, networking, and virtualization all on a single physical host greatly improves VDI performance by eliminating the network hop and also helps to control costs.

These factors have improved VDI performance making it competitive with physical workstations.

Next, let’s look at some benchmark numbers and see how they compare.

I will share the benchmarks of some of the newest VDIs we provide for our clients. You can run the same benchmarks on the hardware at your own company and see how you compare.

For this comparison, I ran the RFO Revit Benchmark v2.1. This is a benchmark provided by RevitForum.org. It is available here. https://www.revitforum.org/hardware-infrastructure/26519-rfo-benchmark-v2.html

This benchmark uses a script to run Revit 2016 (or 2017), a popular design authoring tool used by architects and engineers. Lower numbers are FASTER and BETTER.

Workstation Type	RFO Revit Benchmark 2016 Standard
	Standard Test Set	Rendering Test
Lenovo P50 Workstation* (32GB Physical Laptop Workstation 2016)	200.83	105.42
Entry Designer VDI (16 GB)	193.55	163.05
Designer VDI (32 GB)	191.34	182.21
Cloud 3.0 VDI (32 GB)	160.63	103.75
Cloud 4.0 VDI (our latest 32 GB)	138.37	66.88

*Lenovo P50 is a mobile workstation class computer.

CONCLUSION

Recent improvements in VDI design have really pushed the performance of virtual machines to the point they can completely replace or even outperform a traditional physical desktop. At Advance2000, we think the physical PC is obsolete and the future of computing is virtual desktops (VDI). What do you think?

Check our VDI Glossary page to learn more about what is a VDI (Virtual Desktop Infrastructure) and its key concepts.