Author Archives: obmibmis

About obmibmis

I am the "Mac Guy", but I can do much more:: Talented and accomplished “Hands On” Information Technology Expert with more than 20 years of systems, program and department management experience. Expert in Unix/Macintosh system design, administration and security. Additional background in facilities management. Expertise in global network design and implementation and able to bridge the gap between technical and non-technical persons. I also build robots in my spare time.

Hi All,  Thanks for all of the questions and posts on all of the websites..

I would like to answer some of the questions posted.

Q. How fast are the hard drives

A. These machines all have SSD’s, so they are pretty fast.

 

Q. Why on earth would you build this?  There are easier and cheaper ways to get this density.

A. The company I work for requires large numbers of machines to build and test the software products we make, these products support Windows, Linux and Mac so we have data centers with thousands of machines configured with all 3 OS’s running constant build and test operations 24 hours a day 365 days a year. This is just a small look at the Mac side of things.

Q. There are virtualization solutions that you could configure to run OS X “Hackintosh” why not just do that?

A. There were plenty of discussions of Virtual environments and other “Bare Motherboard”/Google Datacenter-type solutions, but the simple fact is, the Apple EULA requires that Mac OS X run on Apple Hardware, since we are a software company we adhere to these rules without exception.

Q. Why Mac OS X? Linux is free..

A. We require Mac OS X because the products we make run on OS X and we believe in testing on the same hardware our customers use, this helps produce a better product.

Q. How do you get all of those machines to run as a single server, what is one fails?

A. These machines aren’t servers, they serve nothing, they are treated as individual machines.  They are all managed and allocated by specialized “Cluster” software that allows us to run our Build and Test functions. As for device failure, we treat these machines like pixels in a very large display, if a few fail, it’s ok, the management software disables them until we can switch them out. This approach allows us to continue our operations regardless of machine failures.

Q. Have you tried to mount them vertically?

A. I tried the vertical approach, but manufacturing the required plenum to keep the air clean to the rear machines cost too much for this project, but it’s not off the table for the next rack

 

 

Please keep the questions coming :-)

Build It Bigger (I must be dense)

I love all of the interest in the current rack of 160 Mac  Minis, but yesterday I spoke to the shelving vendor and we are going to begin working on a solution to fit 6 mini’s per 1U of space, resulting in a rack with 240 Mac Minis in it.  I can’t wait to start working on that in January.  People thought I was crazy building this???  Wait till they see the next one :-)

In Production…..

I have introduced these machines to the production environment, they run great…  I had to add a 10Gb SFP+ Ethernet card to the XServe, and now I can NetBoot all 160 machines at the same time.. It works so well that all of the machines are back up and running within 45 seconds….  That’s fast for a NetBoot..  Here are some final pics of the cabinet.

 

photo-2 copy

Time to Light the Fires and Kick the Tires

Well, The rack is finished. It’s time to power all of these machines up. I have installed the fans, custom power cables and wired everything. Once powered up I expect the entire rack to use less than 30 amps. I have done some testing with 80 machines running full power and with the fans running half speed the machines stay considerably cooler then expected. If you remember when I initially tested the machines in this configuration I had core temps hovering around 210F. That was hot… Now that I have them installed and the fans running I am seeing core temps around 85F, a drastic improvement.

20121203-102107 copy

20121203-102117 copy

It’s Getting Hot In Here!!

Another consideration I had to deal with was the sheer amount of heat generated by these machines as well as their inability to move air very well. The Mac Mini was designed to be a small quiet desktop machine. It was designed to have access to open air and not be stacked or crowded. The Mac Mini’s design causes us 2 problems, 1. The fans are small and underpowered and 2. the design relies on the case as a large aluminum heat sink. If the fan can’t move air and the case is stuffed in a rack with 159 of it’s closest friends, that means we have to help the air through the cabinet and across all of those cases. I have done some experimenting with some fans designed for these racks, but none of the models met our criteria (220v and a ton of air movement) so I decided to do things my way, and build a solution. My design started with a high volume DC electric fan, it looks just like one you would find on a car radiator, because it is from a car radiator!, I sourced a 220VAC to 12VDC @ 40A power supply and a 40A DC motor controller. Once connected I had the ability (at high speed) to move so much air I could blow my office door shut from 6 feet away, while at lower speeds I was able to move plenty of air and have a nice quiet environment. I decided that 4 fans attached to the inside of the front door pulling air into the rack was sufficient to move air across all of the shelves, so I built the door in the pictures. You will notice the door is all self contained with the power supply, motor controller and fan all mounted together. The only thing the door needs is power from within the cabinet and it uses a standard 220v power cord.

20121107-102640 copy

20121203-102107 copy

 

20121203-102117 copy

Racking it all up

Now I’m ready to begin packing all of these Mac Mini’s into their assigned locations within this single rack. I have taken the time to label everything and make sure I know where all of the parts are going. Once if the biggest problems is going to be cable management, even with only 2 cables per mini.

20121107-101018 copy 20121107-101032 copy

Filling The Void

Now that I have decided to go all the way and try to stuff 160 machines into a single cabinet, I needed to think about the rest of the stuff that makes the machines work, like power, ethernet, cooling, etc… Until now I had only given this a slight thought, I know I would need network switches in the cabinet, but now I know I need 4 — 48 Port Gigabit ehternet switches, 9 PDU’s, a NetBoot Server (Xserve) and some fans to prevent it from becoming less of a Mac Mini rack and more of an aluminum smelter.

The racks have 45 Rack Units “U” available, that mean I had room for 40 shelves of Mac Mini’s, 4 1U Network Switches and one XServe for NetBooting, but where in the world was I going to fit 9 PDU’s!!!! They were not going to fit, no matter how hard I tried, there was just not enough room (and these are the 0U PDU’s that hang vertically at the back of the cabinet… and I only had room for 4 at the most. I had to engineer something different. I had seen in the past some vendors included “Y” power cable splitters with their multi-power supply servers, I thought why not do something like that for the Mac Mini’s? The Mac Mini’s only draw 40 watts, that’s nothing. I could run at least 8 from a single plug, maybe more, but I still wanted to be safe.  I had to figure out how to physically plug 160 machines. Working with vendors and because the power draw of the mini’s was so low, we were able to design a 4 to 1 cable allowing me to plug 4 mini’s in using a power outlet on the PDU.

And a finished sample::

20120918-0905162 copy

Also, I had another issue, these machines were going to be in a datacenter 45 minutes away from my office. Traditionally, I would rely on two tools to help manage remote servers, one was IPMI, and the other was remote power management through a managed PDU. Both of these methods allow me to cut power to a machine so it can reboot if in a stuck state, but I was about to lose both of these tools, the IPMI didn’t exist, and the Mac Mini was engineered in such a way that it was “Supposed” to power itself back up after losing power, but it didn’t work if the machine was hung when the power was cut, so that was useless… I had to come up with a solution. On went the engineering hat and I came up with the idea of attaching a micro hobby servo to each Mac Mini when controlled by a “Phidget” servo controller and some custom code I wrote allowed me to have total control over the Mac Mini’s power switch No matter what state the Mac mini was in, I could control the power switch. I proceeded to deploy my solution onto the test bed of 16 Mac mini’s and things worked great, but the solution was left out of the final design to keep things simple.  If the machines proved too susceptible to power problems I would implement the servo solution at that time.