The implications of Hyper-threading on machines with Simulator Workloads

by Vince Weaver ( vince _at_ csl.cornell.edu )
11 November 2005

The Problem

I disabled hyper-threading on the sampaka and cluizel clusters, a decision multiple people have questioned.

I have conducted a set of experiments to determine the performance implications of hyper-threading for the most common workload of the clusters, namely long running processor simulations.

The Experiment

The test was run on "cluizel39", one node of the cluizel cluster.

The node was running Linux 2.6.11 with perfctr and bmcsensor support patched in. This kernel does have a hyper-threading aware scheduler.

The node has the following hardware: The test program was the alpha version of simple-scalar 4.0.

The test benchmark was equake from the spec2k benchmarks, compiled statically on boulder. The Minnispec lgred.in input was used.

Multiple identical copies of the simulation were launched simultaneously via a script, and the resulting time it took each thread to complete was recorded.

The test was conducted both with and without Hyperthreading enabled via the BIOS.

Results

Hyperthreading Performance Plot

The errorbars indicate maximum and minimum time of the simultaneous runs, while the plotted line connects the average times.

Note the errorbars are much wider for odd-numbers of processes. This is due to the way the Linux scheduler works. In the 3-thread case, one process gets a CPU to itself, thus finishing much sooner than the two processes that must share the remaining CPU. Bouncing processes from cpu to cpu is avoided because it can cause poor cache performance.

There is no significant performance gain from hypertheading; in fact in the 3 thread case the hyperthreading performance is worse. These results are far from the 25-30% maximum performance gains Intel claim are possible.

Also note that the OS does take hyperthreading into account when scheduling. If it didn't, then the 2 and 4 cpu cases would be worse for the HT case (as the scheduler would have assigned the jobs to the first two cpus it saw (parent and sibling of cpu#0), making one cpu overload and leaving the other idle).

Conclusion

Hyper-threading would not be a benefit on our clusters.

In the two process case, the machine performs identically whether hyperthreading is enabled on disabled.

When running more than 2 threads, hyperthreading has no significant impact on the time taken to finish a job. In fact, hyperthreading can make performance slightly worse.

There are numerous other reasons to disable hyperthreading: