There really isn’t any recommendation for disabling or enabling Hyper-Threading for HyperV hosts. The general consensus seems to be, it doesn’t hurt performance, so it’s better to leave it on. We did leave it on, and saw something weird. We had a 20 real core host with Hyper-Threading enabled, so it shows up having 40 cores in windows. The screenshot below is from a 24 real core system:

20150702_no_cores_hostIt shows 24 real cores with 48 logical processors. In HyperV, the number of vCPU’s doesn’t mean the number of real cores the virtual machine gets to use. But it represents the total allowed CPU time of the host, the guest is allowed to use. First, a bit more information on this topic. This percentage can be seen in HyperV manager:

20150207_1vcpu

with 1 vCPU, this guest is allowed to use 5% of the hosts cpu resources, with 2vCPU’s:

20150207_2vcpu

It’s allowed to use 10% of the hosts CPU resources. This makes sense, since HyperV doesn’t bind the vCPU’s to physical CPU’s, but schedules the threads of the guest to a real CPU, and doesn’t allow those threads to use more than the percentage of total CPU time. To my knowledge, HyperV calculates the percentage per vCPU: 1 vCPU percentage = 100/Number of cores in the host. In the example above, this is 5%, since is has 20 cores.

On a host with 20 real cores and HT enabled, the percentage/vCPU is 2.5%, and this is where it gets interesting, because those 40 cores, aren’t real 40 cores. I was under the impression, that HyperV was HT aware, and therefore should know the difference between the number of real and logical cores. In our case. We had a host with 15 virtual machines, of which, only 5 are in production at all times, the other 10 remain idle. We gave each machine 6 vCPU’s which means that each guest is allowed up to 15% total CPU time of the host, with a maximum of 75% total host CPU usage.

When in production, we never saw the hosts CPU usage go above 35%. None of the Virtual Machines with load saw a CPU usage over 6%. But we were experiencing stalls, and CPU performance issues on the virtual machines on that host.

This was strange, and it looked as if the hypervisor was having issues with scheduling the threads to the correct cores (with the lowest load). So we disabled Hyper-Threading on the host, and gave each virtual machine, 3 vCPU’s. This allowed each virtual machine to use the same 15% of total CPU resources. Now, the hosts sees a highest CPU utilization of 65%. And the guests have a max CPU usage of about 14%. That’s more like it, but apart from the higher utilization of the host and guests. The Effect was that the weird stalls in the guest machines were gone. It was a much smoother experience.

So, where does this leave us with Hyper-Threading on HyperV hosts? There really isn’t a lot of information about how the HyperV scheduler really works. Since the general consensus was that Hyper-Threading shouldn’t hurt HyperV performance. Combined with the fact that Windows Server 2012R2 is Hyper-Threading aware. The assumption was that the scheduler would be smart enough to take Hyper-threaded cores in account. As this case has shown, this isn’t the case. In fact, Hyper-Threading can actually really hurt you’re overall performance on a HyperV host. In the discussed situation, the virtual machines were all remote desktop hosts. With lot’s of users and specific applications, and the load tends to be quite high compered to other scenario’s.

As it makes sense that it’s really hard for a hypervisor scheduler to correctly schedule threads to an already virtualized CPU core. Especially if the host has a high workload. I would recommend to turn off Hyper-Threading on HyperV hosts. It worked out quite well for this scenario, and I can imagine, that in other situations where the host is under heavy load, the scheduling would also be an issue with Hyper-threading enabled.

Categories: Hyper-V, Remote Desktop

6 Comments

  1. Michael

    Great post, i suggest to have a look at your NUMA configuration and boundries . Also check hyperV performance counters

  2. Eric Verdurmen

    Thnaks! Those were the first things I checked, but there wasn’t anything wrong with those. No counter showed any problems, and Numa was running correctly. With all vm’s running on a single socket. The only problem we had here, was HT. After changing this setting, the performance was a lot better. And the performance numbers were more accurately showing the real load.

  3. Pat

    Is your CPU running at 1ghz?

    Dynamic (BIOS) or Balanced (windows) power settings are incredibly brutal.

    Task manager will not tell you the true CPU speed – you need to use CPU-z, or you can try the latest 7zip – “7z b” on the command line will ‘sense’ the CPU speed.

    1. Eric Verdurmen

      Indeed they are, but we don’t use either, we always choose the power option. (no dynamic speed in BIOS or balanced power settings in Windows).

  4. With Wings

    They really should enable core affinity for Hyper-V.

    1. Eric Verdurmen

      That won’t work well together with the way HyperV uses threading. It differs from ESX, both have their ups and downs.

Leave a Reply

Your email address will not be published. Required fields are marked *