96 MacBook Pro’s in one rack.

96 MacBook Pro’s in a single Rack space

We had a need to introduce 96 MacBook Pro’s for our product testing.  Our requirements included Retina displays, small form factor, low power, cool running, Apple branded hardware, high density design, i7 CPU’s, 16GB RAM, etc…  This is what I came up with.  Some of the parts are custom 3D printed and others are standard off the shelf parts. We use 3D printed wedges to keep each lid open to 7mm, and other 3D printed parts to help hold the power bricks to the cable management arm. I know some of you will reply with the standard “Why didn’t you just use Linux?” or “Just use OS X in a VM on <insert hardware name here>”, but the testing we do requires Apple branded hardware, so this is what we came up with.  This rack has been in service for a few months now and is running great.

Here are some of the pictures:

New MacBook Pro’s

Rack with the front door closed.. Those fans really move some air.

Shelf pulled out. Notice 3 MacBook Pro’s nested together. They are actually being held open 7mm by a custom 3D printed wedge. This opening allows for the screen to be used for testing as well as ample air circulation. You can’t see the temperature sensors tucked into each notebook’s keyboard area.

Another angle of the front with a shelf extended.

A shot of the back. Notice the cable management arm. Each 1U shelf has it’s own cable management arm which also carries the 3 power bricks. We did this to make sure there was enough air flow over the power bricks to keep them cool while allowing the shelf to fully extend for maintenance.

Hi All,  Thanks for all of the questions and posts on all of the websites..

I would like to answer some of the questions posted.

Q. How fast are the hard drives

A. These machines all have SSD’s, so they are pretty fast.

 

Q. Why on earth would you build this?  There are easier and cheaper ways to get this density.

A. The company I work for requires large numbers of machines to build and test the software products we make, these products support Windows, Linux and Mac so we have data centers with thousands of machines configured with all 3 OS’s running constant build and test operations 24 hours a day 365 days a year. This is just a small look at the Mac side of things.

Q. There are virtualization solutions that you could configure to run OS X “Hackintosh” why not just do that?

A. There were plenty of discussions of Virtual environments and other “Bare Motherboard”/Google Datacenter-type solutions, but the simple fact is, the Apple EULA requires that Mac OS X run on Apple Hardware, since we are a software company we adhere to these rules without exception.

Q. Why Mac OS X? Linux is free..

A. We require Mac OS X because the products we make run on OS X and we believe in testing on the same hardware our customers use, this helps produce a better product.

Q. How do you get all of those machines to run as a single server, what is one fails?

A. These machines aren’t servers, they serve nothing, they are treated as individual machines.  They are all managed and allocated by specialized “Cluster” software that allows us to run our Build and Test functions. As for device failure, we treat these machines like pixels in a very large display, if a few fail, it’s ok, the management software disables them until we can switch them out. This approach allows us to continue our operations regardless of machine failures.

Q. Have you tried to mount them vertically?

A. I tried the vertical approach, but manufacturing the required plenum to keep the air clean to the rear machines cost too much for this project, but it’s not off the table for the next rack

 

 

Please keep the questions coming 🙂

Build It Bigger (I must be dense)

I love all of the interest in the current rack of 160 Mac  Minis, but yesterday I spoke to the shelving vendor and we are going to begin working on a solution to fit 6 mini’s per 1U of space, resulting in a rack with 240 Mac Minis in it.  I can’t wait to start working on that in January.  People thought I was crazy building this???  Wait till they see the next one 🙂

In Production…..

I have introduced these machines to the production environment, they run great…  I had to add a 10Gb SFP+ Ethernet card to the XServe, and now I can NetBoot all 160 machines at the same time.. It works so well that all of the machines are back up and running within 45 seconds….  That’s fast for a NetBoot..  Here are some final pics of the cabinet.

 

 

 

 

 

Time Goes By………..

 

Well, here we are, 2 years have passed and these 2 racks (yup there are 2 of them now) have been running non-stop at ~97% utilization.  I have had 10 hosts fail, but these failures have all been related to SSD or RAM, and were quickly fixed.  Last year, these racks endured an 40 mile move to our new datacenter and continue to run fine.  Over the past year I did have to replace the front fans on the first rack, but the second rack is running without issue.

Stay Tuned for the next build.  It should be very interesting!!

Steven —

Time to Light the Fires and Kick the Tires

Well, The rack is finished. It’s time to power all of these machines up. I have installed the fans, custom power cables and wired everything. Once powered up I expect the entire rack to use less than 30 amps. I have done some testing with 80 machines running full power and with the fans running half speed the machines stay considerably cooler then expected. If you remember when I initially tested the machines in this configuration I had core temps hovering around 210F. That was hot… Now that I have them installed and the fans running I am seeing core temps around 85F, a drastic improvement.

It’s Getting Hot In Here!!

Another consideration I had to deal with was the sheer amount of heat generated by these machines as well as their inability to move air very well. The Mac Mini was designed to be a small quiet desktop machine. It was designed to have access to open air and not be stacked or crowded. The Mac Mini’s design causes us 2 problems, 1. The fans are small and underpowered and 2. the design relies on the case as a large aluminum heat sink. If the fan can’t move air and the case is stuffed in a rack with 159 of it’s closest friends, that means we have to help the air through the cabinet and across all of those cases. I have done some experimenting with some fans designed for these racks, but none of the models met our criteria (220v and a ton of air movement) so I decided to do things my way, and build a solution. My design started with a high volume DC electric fan, it looks just like one you would find on a car radiator, because it is from a car radiator!, I sourced a 220VAC to 12VDC @ 40A power supply and a 40A DC motor controller. Once connected I had the ability (at high speed) to move so much air I could blow my office door shut from 6 feet away, while at lower speeds I was able to move plenty of air and have a nice quiet environment. I decided that 4 fans attached to the inside of the front door pulling air into the rack was sufficient to move air across all of the shelves, so I built the door in the pictures. You will notice the door is all self contained with the power supply, motor controller and fan all mounted together. The only thing the door needs is power from within the cabinet and it uses a standard 220v power cord.

 

Racking it all up

Now I’m ready to begin packing all of these Mac Mini’s into their assigned locations within this single rack. I have taken the time to label everything and make sure I know where all of the parts are going. Once if the biggest problems is going to be cable management, even with only 2 cables per mini.

 

Filling The Void

Now that I have decided to go all the way and try to stuff 160 machines into a single cabinet, I needed to think about the rest of the stuff that makes the machines work, like power, ethernet, cooling, etc… Until now I had only given this a slight thought, I know I would need network switches in the cabinet, but now I know I need 4 — 48 Port Gigabit ehternet switches, 9 PDU’s, a NetBoot Server (Xserve) and some fans to prevent it from becoming less of a Mac Mini rack and more of an aluminum smelter.

The racks have 45 Rack Units “U” available, that mean I had room for 40 shelves of Mac Mini’s, 4 1U Network Switches and one XServe for NetBooting, but where in the world was I going to fit 9 PDU’s!!!! They were not going to fit, no matter how hard I tried, there was just not enough room (and these are the 0U PDU’s that hang vertically at the back of the cabinet… and I only had room for 4 at the most. I had to engineer something different. I had seen in the past some vendors included “Y” power cable splitters with their multi-power supply servers, I thought why not do something like that for the Mac Mini’s? The Mac Mini’s only draw 40 watts, that’s nothing. I could run at least 8 from a single plug, maybe more, but I still wanted to be safe.  I had to figure out how to physically plug 160 machines. Working with vendors and because the power draw of the mini’s was so low, we were able to design a 4 to 1 cable allowing me to plug 4 mini’s in using a power outlet on the PDU.

And a finished sample::

Also, I had another issue, these machines were going to be in a datacenter 45 minutes away from my office. Traditionally, I would rely on two tools to help manage remote servers, one was IPMI, and the other was remote power management through a managed PDU. Both of these methods allow me to cut power to a machine so it can reboot if in a stuck state, but I was about to lose both of these tools, the IPMI didn’t exist, and the Mac Mini was engineered in such a way that it was “Supposed” to power itself back up after losing power, but it didn’t work if the machine was hung when the power was cut, so that was useless… I had to come up with a solution. On went the engineering hat and I came up with the idea of attaching a micro hobby servo to each Mac Mini when controlled by a “Phidget” servo controller and some custom code I wrote allowed me to have total control over the Mac Mini’s power switch No matter what state the Mac mini was in, I could control the power switch. I proceeded to deploy my solution onto the test bed of 16 Mac mini’s and things worked great, but the solution was left out of the final design to keep things simple.  If the machines proved too susceptible to power problems I would implement the servo solution at that time.

Let The Testing Begin

I decided that I needed to do some testing to determine how these machines worked when put in close proximity with each other. The first step was to find a rack solution that would allow me to get as many as possible per rack unit. These were a few shelving makers in the market space, but they all had either a 2 machine per 1U solution (80 machines per cabinet), or an 8 machine per 5U solution (64 machines per cabinet). I thought this number was way too low, so I contacted one of the vendors (H-Squared) and spoke to the owner, he told me they had been mulling over a design that would allow 4 mini’s per 1U, without restricting airflow to any of the machines, I immediately asked if I could beta test 5 prototypes, he said “Sure, right after I finish hand making them for you!”

In the mean time I had to order some test Mac Mini’s. I used the online configuration tool from Apple, but soon learned that I couldn’t get a Mac Mini with 16GB of RAM, so I decided to order the Mac Mini Server Core i7 chip with a 256GB SSD and 4GB of RAM (I would swap out those RAM chips for 16GB of Kingston memory once the machines arrived)

The final configuration of the Mac Mini’s is:

Mac Mini Server Core i7 Quad Core Processor
16GB RAM
256GB SSD (Solid State Drive)
In a week, the shelving arrived and I was ready to do some scaled testing.

I commandeered some space in a rack and placed the 16 machines in their new cozy shelving and proceeded to run non-stop testing for days. The results of which were surprising. It turns out the Mac Mini’s run pretty hot, with the CPU core temps at 210 degrees and the rest of the machine at 150, but these temps didn’t set off any alarms, nor were the fans running at top speed. I also ran the same tests with 8 Mac Mini’s in the 8 across racking configuration with similar resuls. At this point I was convinced that I was going to try to stuff 160 Mac Mini’s into a single cabinet…. I might be crazy, but this is going to be fun.

Here are some pictures of the shelving::

 

 

How many can I fit in a rack

“How Many can I fit in a rack?”  This became the main question. I had been used to putting 40 XServe’s in a single rack netting me 320 cores per rack, but I wanted to do better.  I began thinking that this may not be such a bad thing after all.  The XServe’s were limited to a single machine per rack unit “U”, but I may be able to get multiple Mac Mini’s per “U”.  The Xserve’s were very power hungry, requiring powerful Power distribution units (PDU’s), while the Mac Mini’s sipped the juice. The XServe’s ran hot, the Mini’s were cool. So, the question became, “Just how many can I fit in a rack?”