Now that I have decided to go all the way and try to stuff 160 machines into a single cabinet, I needed to think about the rest of the stuff that makes the machines work, like power, ethernet, cooling, etc… Until now I had only given this a slight thought, I know I would need network switches in the cabinet, but now I know I need 4 — 48 Port Gigabit ehternet switches, 9 PDU’s, a NetBoot Server (Xserve) and some fans to prevent it from becoming less of a Mac Mini rack and more of an aluminum smelter.
The racks have 45 Rack Units “U” available, that mean I had room for 40 shelves of Mac Mini’s, 4 1U Network Switches and one XServe for NetBooting, but where in the world was I going to fit 9 PDU’s!!!! They were not going to fit, no matter how hard I tried, there was just not enough room (and these are the 0U PDU’s that hang vertically at the back of the cabinet… and I only had room for 4 at the most. I had to engineer something different. I had seen in the past some vendors included “Y” power cable splitters with their multi-power supply servers, I thought why not do something like that for the Mac Mini’s? The Mac Mini’s only draw 40 watts, that’s nothing. I could run at least 8 from a single plug, maybe more, but I still wanted to be safe. I had to figure out how to physically plug 160 machines. Working with vendors and because the power draw of the mini’s was so low, we were able to design a 4 to 1 cable allowing me to plug 4 mini’s in using a power outlet on the PDU.
And a finished sample::
Also, I had another issue, these machines were going to be in a datacenter 45 minutes away from my office. Traditionally, I would rely on two tools to help manage remote servers, one was IPMI, and the other was remote power management through a managed PDU. Both of these methods allow me to cut power to a machine so it can reboot if in a stuck state, but I was about to lose both of these tools, the IPMI didn’t exist, and the Mac Mini was engineered in such a way that it was “Supposed” to power itself back up after losing power, but it didn’t work if the machine was hung when the power was cut, so that was useless… I had to come up with a solution. On went the engineering hat and I came up with the idea of attaching a micro hobby servo to each Mac Mini when controlled by a “Phidget” servo controller and some custom code I wrote allowed me to have total control over the Mac Mini’s power switch No matter what state the Mac mini was in, I could control the power switch. I proceeded to deploy my solution onto the test bed of 16 Mac mini’s and things worked great, but the solution was left out of the final design to keep things simple. If the machines proved too susceptible to power problems I would implement the servo solution at that time.
I decided that I needed to do some testing to determine how these machines worked when put in close proximity with each other. The first step was to find a rack solution that would allow me to get as many as possible per rack unit. These were a few shelving makers in the market space, but they all had either a 2 machine per 1U solution (80 machines per cabinet), or an 8 machine per 5U solution (64 machines per cabinet). I thought this number was way too low, so I contacted one of the vendors (H-Squared) and spoke to the owner, he told me they had been mulling over a design that would allow 4 mini’s per 1U, without restricting airflow to any of the machines, I immediately asked if I could beta test 5 prototypes, he said “Sure, right after I finish hand making them for you!”
In the mean time I had to order some test Mac Mini’s. I used the online configuration tool from Apple, but soon learned that I couldn’t get a Mac Mini with 16GB of RAM, so I decided to order the Mac Mini Server Core i7 chip with a 256GB SSD and 4GB of RAM (I would swap out those RAM chips for 16GB of Kingston memory once the machines arrived)
The final configuration of the Mac Mini’s is:
Mac Mini Server Core i7 Quad Core Processor
256GB SSD (Solid State Drive)
In a week, the shelving arrived and I was ready to do some scaled testing.
I commandeered some space in a rack and placed the 16 machines in their new cozy shelving and proceeded to run non-stop testing for days. The results of which were surprising. It turns out the Mac Mini’s run pretty hot, with the CPU core temps at 210 degrees and the rest of the machine at 150, but these temps didn’t set off any alarms, nor were the fans running at top speed. I also ran the same tests with 8 Mac Mini’s in the 8 across racking configuration with similar resuls. At this point I was convinced that I was going to try to stuff 160 Mac Mini’s into a single cabinet…. I might be crazy, but this is going to be fun.
Here are some pictures of the shelving::
“How Many can I fit in a rack?” This became the main question. I had been used to putting 40 XServe’s in a single rack netting me 320 cores per rack, but I wanted to do better. I began thinking that this may not be such a bad thing after all. The XServe’s were limited to a single machine per rack unit “U”, but I may be able to get multiple Mac Mini’s per “U”. The Xserve’s were very power hungry, requiring powerful Power distribution units (PDU’s), while the Mac Mini’s sipped the juice. The XServe’s ran hot, the Mini’s were cool. So, the question became, “Just how many can I fit in a rack?”
Now that Apple has eliminated the XServe, and I can no longer find them in the marketplace I need to come up with a way to get the same or better CPU core density into a standard 2′ by 2′ datacenter rack. (A Datacenter Rack is a cabinet that has 44 spaces, or horizontal units inside, each unit is referred to as a “U”. Computer manufacturers measure the height of their servers in “U”, so some servers are 1 “U” like the XServe). In the past this would have been easy, I would call Apple and order 40 top of the line multi-core XServe’s and simply build the rack with 40 servers, each with 2 quad core processors, but now Apple is calling the Mac Mini it’s new server. This poses some issues for me.
1. The Mac Mini is not a server grade hardware package
2. It only has a single power supply
3. No IPMI (Out of Band Remote Management)
But, It’s the only option, and we need Macintosh processing BAD!!!
The problem, how do we build a high density Macintosh rack, with as many cores as possible, that occupies a single rack footprint, now that Apple has eliminated the Xserve.
The solution, stuff as many of the highest powered Mac Mini servers as you possibly can into a standard data center rack.
In this blog, I will outline the process I developed to get as many Mac Minis as I can into a single 2′ x 2′ datacenter footprint.