Bauerwulf: The Unit Test Accelerator
The EliteForm team recently faced a tough problem, and perhaps you’ve experienced something similar. We’re currently amassing loads of athlete workout video for further refinement of our PowerTracker product. PowerTracker, in short, is a hardware/software solution that uses video to analyze an athlete’s form in real time at the rack. This provides never-before-seen athletic analysis. It’s magic. It’s math. It’s processor-intensive. And it requires A LOT of time to test.
When it was time to test our latest code to see how our changes affected the accuracy of the software, we ran a unit test suite against all of our video. During the test, video is processed nearly in real time at roughly 30 frames per second. You can imagine the developer unhappiness that comes with a unit test that runs 24 hours due to the 24 hours of workout video the code is being tested against. Luckily, we were able to alleviate this problem via parallelization and cluster computing.
We commandeered a bookshelf from Doug’s desk, a massive Ethernet switch, and several off-the-shelf (now discontinued) Dell Zinos to create what we believe is the world’s first Bauerwulf cluster. Perhaps you’ve heard of a Beowulf cluster, a computing cluster created from commodity hardware. A Bauerwulf cluster is very much like a Beowulf cluster, but is named after somebody with the last name “Bauer” (hint: me).
Bauerwulf runs the Microsoft Windows Server 2008 High Performance Computing Edition (HPC Server) operating system, which provides us super-easy cluster setup, node management, and job scheduling. We investigated a few open-source solutions for job scheduling before thinking, “surely Microsoft has something for this,” and stumbling upon this golden egg.
HPC Server is a joy to use and develop for, and it beats the pants off open-source alternatives this team has used in the past. One missing feature from HPC Server today is Hadoop support, though Microsoft has recently adopted Hadoop as its “big data” platform for the future; we’re looking forward to seeing what happens with that.
HPC Server also provides a straightforward API, which we’ve leveraged for our needs. We wrote a simple program to schedule a job, which HPC Server executes across several nodes. Each node gets one video per core, and when it’s finished, another video is provided. Results are written back to the database. Before we built Bauerwulf, we’d run a test against roughly four hours of video. Since the test executes in real time, it would take about four hours to run. The same test performed by Bauerwulf completes in about 20 minutes.
We also had some fun and built our own cluster monitor. This displays all the cores in the cluster and their status (available and busy). It even comes complete with sound effects that fire when jobs are completed.
In a testament to how easy it is to use HPC Server, Bauerwulf became fully-functional in about two weeks as an after-hours project for a few of us. Since then, it has become the pride of EliteForm. We’re beginning to put it to work in other, more advanced ways. Nothing we can mention yet, but look for more magic in the future.
