I have introduced these machines to the production environment, they run great… I had to add a 10Gb SFP+ Ethernet card to the XServe, and now I can NetBoot all 160 machines at the same time.. It works so well that all of the machines are back up and running within 45 seconds…. That’s fast for a NetBoot.. Here are some final pics of the cabinet.
Time Goes By………..
Well, here we are, 2 years have passed and these 2 racks (yup there are 2 of them now) have been running non-stop at ~97% utilization. I have had 10 hosts fail, but these failures have all been related to SSD or RAM, and were quickly fixed. Last year, these racks endured an 40 mile move to our new datacenter and continue to run fine. Over the past year I did have to replace the front fans on the first rack, but the second rack is running without issue.
Stay Tuned for the next build. It should be very interesting!!
Steven —
Steve's Blog 
Great job. It must make the other racks look like kludges by comparison.
Are you managing without being able to remotely cut power?
The PDU’s I used allow me control at the plug level, so if I really had to, I could power cycle a shelf at a time, but as of yet, I haven’t needed to power cycle them
Great.
Thank you 😉
Sorry this might be a silly question, but what are these machines doing? Are they nodes in a virtualised environment? In reality what are you doing with this processing power.
Not a silly question at all.. These machines are one small part of a farm of ~2000 machines consisting of Windows, Mac and linux hosts running on some very powerful systems. These systems run 24/7 in our Build and Test environment constantly building and testing the software we produce.
Cool, thanks for the reply. Can you say who you work for?
Sorry, unfortunately no, but if you want more information on the rack I’m happy to chat.
Hi! Can you you provide more information about using this macs? How can you use them together? Is it possible to use all of them for one task? If yes, how?
A difficult problem solved with unexpected elegance. Well done. This should be a product.
This is very impressive. I’m just curious as to why there are so few pictures included.
And the cooling difference… that’s really impressive.
I’m far from apple, all of this is basically because apple doesn’t allow to run their OS on other HW?
Pingback: 160 Mac minis crammed into custom 2′ x 2′ datacenter rack | 9to5Mac
Pingback: 160 Mac minis crammed into custom 2′ x 2′ datacenter rack | MobileAdmin
So you need an Xserve along with the minis. minis along won’t work correct?
Pingback: 160 Mac minis crammed into custom 2′ x 2′ datacenter rack | Apple News Gator
for as much as apple appears care about their consumer products, they sure seem to not care about enterprise datacenter scalablity…. Don’t get me wrong, you daisy chaining a bunch of mac mini’s is an innovative on-off solution, and you did a great job. Unforuntatly not mainstream solution, which is what is needed if they want to be as dominant in the datacenter as they are with consumer electronics.
Pingback: This Custom Datacenter Rack Has 160 Mac Minis Crammed Inside of It « VidenOmkring
I totally agree.
The mini’s will work fine alone, but in our environment we require that each time a machine is rebooted it returns with a fresh copy of our internal OS X build. This requirement reduces the occurrences of sporadic errors in out Build and Test environment. The XServe is there to provide NetBoot services to the 160 Clients.
Pingback: 160 Mac minis crammed into custom 2′ x 2′ datacenter rack | Apple Store Valencia
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | allcom.se
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Tips for the Unready
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower ← techtings
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower « Whats-Hots.com | What Hots News and Trends in The World
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | GabsGroup
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | The Worlds Tech Blog
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Hi-Tech Centre
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower - FourTech Plus
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | i-Newz
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower - Andys Roms
Pingback: Pctwix.com – Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Pctwix – Technical News never delivered
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Google Android news and more!
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Best Mobile Gadgets
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower « WEBLICIT
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower - IT Lounge
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | BeachTechnology
Pingback: Tech Gadget Reviews » Custom enclosure designs shove 160 Mac minis into a single rackmount tower
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Information News
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower « Datapple
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Electronics & computer accessories
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Blog Feeds
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | 1v8 NET
Which DC fan did you use? Source? Thx! Nice work!
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower » Breaking News | Top Stories
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | eMagility :: defining mobile agility
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower » Developages - Development and Technology Blog
Pingback: Free Warez at putraidomandiri.com » Blog Archive » Custom enclosure designs shove 160 Mac minis into a single rackmount tower
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Price Gadget Reviews
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower - The Review Blog
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower – Nieuws over Gadgets en elektronica
How does the cost compare to the say a Unix rack you have deployed with equivalent processing power?
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower » Alpha Newsroom
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | et
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower « Gadgets « Alpha Newsroom
Pingback: This Custom Data Centre Rack Crams In 160 Mac Minis | Gizmodo Australia
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | SP1RACY - All things tech & more
That is beautiful! What is the size of your image that you can deploy it in 45 seconds?
It’s a 24GB NetBoot Image.
Hmmm… With those requirements, why not use Apple Hardware running Hyper-V with RemoteFX enabled in order to virtualize GPU resources running virtualized OSX instances on top? That way, OSX gets virtualized while still retaining GPU access for OpenGL as you require.
The only thing though is that I do not know if HyperV would support running OSX/Server but maybe you guys could pull it off.
I really appreciate you taking the time to offer help, I looked into virtualization, but our testing environment has the power to max out CPU and RAM on each machine so it is in our best interest to have actual physical hardware under each copy of the OS.
Pingback: 160 Mac minis crammed into custom 2′ x 2′ datacenter rack - Just a new life! - What's Up!
Pingback: This Custom Datacenter Rack Has 160 Mac Minis Crammed Inside of It | Gadget News, Reviews and Guides
Pingback: 160 Mac mini’s crammed into a custom enclosure :: TweakTown USA Edition | Up to the hour news
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Summer Vacation Video
Mmhm, then having dedicated hardware for each copy of the OS does sound good ^^
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | tekifeed.com – Gadget Feeds, Gadget News and more!
Pingback: – Custom enclosure designs shove 160 Mac minis into a single rackmount tower
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Technology Blog
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Mobile Gadgets Reviews
I have experimented with automotive fans for cooling in racks as well, but was never able to get more than 6months out of one before they seized up. Being brush fans im concerned you may have a similar problem; unless you found brushless models? I would also make sure the fans have some sort of inline fuse between power supply. I burnt through two power supplies before I wised up to that. They sure move some mean air. Neat setup!
hi steve, this sounds like a very interesting project. it caught my eye on the endgadet website.. We are a Dubai based manufacturing company that specializes in design & manufacturing of custom made server racks. id like to get in touch and send you some more details. im sure we can work something out that meets your requirements. Please get back to me if your interested.
Pingback: » Custom enclosure designs shove 160 Mac minis into a single rackmount tower Gamez Menu
Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower « BinBon.com
We should definitely talk. There is already a discussion from my management asking for a higher density rack in the spring.
I definitely would like to talk. Please let me know the best way to contact you. Maybe a private message
UniMelb?
I think this is really cool and inventive but the price must be between $80k – $100k, is there no mainstream solution that’s comparable? We have an IBM BladeCenter with HX5 blades that have Dual 8-core processors, hyperthreaded that’s 32 Logical processors, with 256GB of ram and VMware vSphere they run approx 30 vms each for Dev and Test environments. New technology is backwards compatible as it emerges and we’ve rotated the blades out to newer powerful units every year.
What would hold me back from your solution is the availability of replacement units and maintenance contracts. Apple seems to change their Mac minis every 6 months can you simple replace with the newest model or do you have to keep replacement stock onhand?
I applaud your company for having faith in you and taking the risk to develop this project.
Interested in which 10gb card you threw in your Xserve–I have a pair of 2009’s running 10.8 sever aiming for a similar end game…
I used the Small Tree 10Gb Ethernet card with SFP+ The server is also equipped with 3 Intel 320 Series 300GB SSD’s in a triple mirror and 36GB of RAM. I can reboot all 160 Mac Minis at the exact same time and they all come back up in well under a minute. The XServe isn’t even breathing hard, CPU is under 20% and the SSD’s aren’t even accessed once the NetBoot image is in RAM, and the machine only uses 4 of the 10Gb Ethernet link. I’m going to add the Small tree card to the 3 other NetBooted racks I have.
If I can be of more help, please let me know.
Steven
It would be awesome to get everything mechanical related thats left, (eg. electric fans), out of the picture and you would have probably made the first timeless machines of all times. Keep up the awesome work. Hope to have mine soon!
Pingback: 160 Mac Mini dans une baie de serveurs
Hey steve, longtime listener, first time commenter 🙂 What a nice super computer you have here. Are you using this for any clustering purposes, or is the horsepower of each mini just self contained?
Pingback: Datacenter-Rack mit 160 Mac Minis | GIZMODO DEGIZMODO DE
Really cool design and idea, my only question/comment is how will you deal with building either more or replacing your current xserve server? Since even in such a impressive and complex system you still need a xserve, which I sadly regret Apple retiring. Besides that though, super cool. I honestly love this idea, and think it’s cool your company was willing to drop a dime on 160 mac minis, my issue is less with how you made it possible more so how Apple gave you only one way to do it. But kudos on coming up with a solution, I only wish Apple would work more with companies to deliver business hardware on a workable scale.
Pingback: Centro de datos con 160 Mac mini: la plantación de manzanas | Applesencia
Pingback: This is what 160 Mac Minis in a rack look like | SiliconChaos
Pingback: 160 Mac Minis im Rack.. « schillerfrau
What’s your process for getting new systems from their out-of-the-box config (where they want to boot from the factory installed OS on the internal disk) to NetBooting? Presumably, you’re not hooking up a keyboard to each and holding down N. 🙂
Thanks,
Jim
Jim,
Unfortunately, we are doing just that, only instead of the “n” key we hold the “option” key and choose our custom installer, once our custom image is on the machine, we choose our “Build and Test” NetBoot image, and off they go! Once they are in the Build and test Pool, we have custom scripting that allows us to change the image they NetBoot between test cycles, That scripting is built around the “bless” command. I hope to have some time to write up the entire process on this blog, and I may include some of my notes from the last time I spoke at the Splunk conference.
Hi Steve, you have inspired me to build a few of these racks for internal purposes once the 2014 mini refresh is out in a few months. Q? Source of the racks and plastic heat redirectors to order them and any special instructions I should tell them? Also did you put a power strip and gigabit switch in each rack so 2 connections out the back or insufficient room so 8 connections out the back?
Hi Steve.
This is a great job done here.
We’re in need of a similar solution. Would love to chat.
-Rybo
Pingback: chkuendig comments on “96 MacBook Pro’s in one server rack” | Exploding Ads