In Production…..

I have introduced these machines to the production environment, they run great…  I had to add a 10Gb SFP+ Ethernet card to the XServe, and now I can NetBoot all 160 machines at the same time.. It works so well that all of the machines are back up and running within 45 seconds….  That’s fast for a NetBoot..  Here are some final pics of the cabinet.

 

photo-2 copy

About these ads

92 thoughts on “In Production…..

  1. simbimbo Post author

    The PDU’s I used allow me control at the plug level, so if I really had to, I could power cycle a shelf at a time, but as of yet, I haven’t needed to power cycle them

  2. Ben Magro

    Sorry this might be a silly question, but what are these machines doing? Are they nodes in a virtualised environment? In reality what are you doing with this processing power.

  3. simbimbo Post author

    Not a silly question at all.. These machines are one small part of a farm of ~2000 machines consisting of Windows, Mac and linux hosts running on some very powerful systems. These systems run 24/7 in our Build and Test environment constantly building and testing the software we produce.

  4. simbimbo Post author

    Sorry, unfortunately no, but if you want more information on the rack I’m happy to chat.

  5. Alex

    Hi! Can you you provide more information about using this macs? How can you use them together? Is it possible to use all of them for one task? If yes, how?

  6. DiNet

    This is very impressive. I’m just curious as to why there are so few pictures included.
    And the cooling difference… that’s really impressive.
    I’m far from apple, all of this is basically because apple doesn’t allow to run their OS on other HW?

  7. Pingback: 160 Mac minis crammed into custom 2′ x 2′ datacenter rack | 9to5Mac

  8. Pingback: 160 Mac minis crammed into custom 2′ x 2′ datacenter rack | MobileAdmin

  9. Pingback: 160 Mac minis crammed into custom 2′ x 2′ datacenter rack | Apple News Gator

  10. Johnny

    for as much as apple appears care about their consumer products, they sure seem to not care about enterprise datacenter scalablity…. Don’t get me wrong, you daisy chaining a bunch of mac mini’s is an innovative on-off solution, and you did a great job. Unforuntatly not mainstream solution, which is what is needed if they want to be as dominant in the datacenter as they are with consumer electronics.

  11. Pingback: This Custom Datacenter Rack Has 160 Mac Minis Crammed Inside of It « VidenOmkring

  12. simbimbo Post author

    The mini’s will work fine alone, but in our environment we require that each time a machine is rebooted it returns with a fresh copy of our internal OS X build. This requirement reduces the occurrences of sporadic errors in out Build and Test environment. The XServe is there to provide NetBoot services to the 160 Clients.

  13. Pingback: 160 Mac minis crammed into custom 2′ x 2′ datacenter rack | Apple Store Valencia

  14. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | allcom.se

  15. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Tips for the Unready

  16. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower ← techtings

  17. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower « Whats-Hots.com | What Hots News and Trends in The World

  18. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | GabsGroup

  19. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | The Worlds Tech Blog

  20. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Hi-Tech Centre

  21. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower - FourTech Plus

  22. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | i-Newz

  23. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower - Andys Roms

  24. Pingback: Pctwix.com – Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Pctwix – Technical News never delivered

  25. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Google Android news and more!

  26. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Best Mobile Gadgets

  27. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower « WEBLICIT

  28. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower - IT Lounge

  29. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | BeachTechnology

  30. Pingback: Tech Gadget Reviews » Custom enclosure designs shove 160 Mac minis into a single rackmount tower

  31. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Information News

  32. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower « Datapple

  33. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Electronics & computer accessories

  34. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Blog Feeds

  35. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | 1v8 NET

  36. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower » Breaking News | Top Stories

  37. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | eMagility :: defining mobile agility

  38. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower » Developages - Development and Technology Blog

  39. Pingback: Free Warez at putraidomandiri.com » Blog Archive » Custom enclosure designs shove 160 Mac minis into a single rackmount tower

  40. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Price Gadget Reviews

  41. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower - The Review Blog

  42. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower – Nieuws over Gadgets en elektronica

  43. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower » Alpha Newsroom

  44. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | et

  45. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower « Gadgets « Alpha Newsroom

  46. Pingback: This Custom Data Centre Rack Crams In 160 Mac Minis | Gizmodo Australia

  47. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | SP1RACY - All things tech & more

  48. Ayu

    Hmmm… With those requirements, why not use Apple Hardware running Hyper-V with RemoteFX enabled in order to virtualize GPU resources running virtualized OSX instances on top? That way, OSX gets virtualized while still retaining GPU access for OpenGL as you require.

    The only thing though is that I do not know if HyperV would support running OSX/Server but maybe you guys could pull it off.

  49. simbimbo Post author

    I really appreciate you taking the time to offer help, I looked into virtualization, but our testing environment has the power to max out CPU and RAM on each machine so it is in our best interest to have actual physical hardware under each copy of the OS.

  50. Pingback: 160 Mac minis crammed into custom 2′ x 2′ datacenter rack - Just a new life! - What's Up!

  51. Pingback: This Custom Datacenter Rack Has 160 Mac Minis Crammed Inside of It | Gadget News, Reviews and Guides

  52. Pingback: 160 Mac mini’s crammed into a custom enclosure :: TweakTown USA Edition | Up to the hour news

  53. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Summer Vacation Video

  54. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | tekifeed.com – Gadget Feeds, Gadget News and more!

  55. Pingback: – Custom enclosure designs shove 160 Mac minis into a single rackmount tower

  56. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Technology Blog

  57. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower | Mobile Gadgets Reviews

  58. Torbs

    I have experimented with automotive fans for cooling in racks as well, but was never able to get more than 6months out of one before they seized up. Being brush fans im concerned you may have a similar problem; unless you found brushless models? I would also make sure the fans have some sort of inline fuse between power supply. I burnt through two power supplies before I wised up to that. They sure move some mean air. Neat setup!

  59. Chacko Joseph

    hi steve, this sounds like a very interesting project. it caught my eye on the endgadet website.. We are a Dubai based manufacturing company that specializes in design & manufacturing of custom made server racks. id like to get in touch and send you some more details. im sure we can work something out that meets your requirements. Please get back to me if your interested.

  60. Pingback: » Custom enclosure designs shove 160 Mac minis into a single rackmount tower Gamez Menu

  61. Pingback: Custom enclosure designs shove 160 Mac minis into a single rackmount tower « BinBon.com

  62. simbimbo Post author

    We should definitely talk. There is already a discussion from my management asking for a higher density rack in the spring.

  63. simbimbo Post author

    I definitely would like to talk. Please let me know the best way to contact you. Maybe a private message

  64. Greg

    I think this is really cool and inventive but the price must be between $80k – $100k, is there no mainstream solution that’s comparable? We have an IBM BladeCenter with HX5 blades that have Dual 8-core processors, hyperthreaded that’s 32 Logical processors, with 256GB of ram and VMware vSphere they run approx 30 vms each for Dev and Test environments. New technology is backwards compatible as it emerges and we’ve rotated the blades out to newer powerful units every year.

    What would hold me back from your solution is the availability of replacement units and maintenance contracts. Apple seems to change their Mac minis every 6 months can you simple replace with the newest model or do you have to keep replacement stock onhand?

    I applaud your company for having faith in you and taking the risk to develop this project.

  65. Chris

    Interested in which 10gb card you threw in your Xserve–I have a pair of 2009′s running 10.8 sever aiming for a similar end game…

  66. simbimbo Post author

    I used the Small Tree 10Gb Ethernet card with SFP+ The server is also equipped with 3 Intel 320 Series 300GB SSD’s in a triple mirror and 36GB of RAM. I can reboot all 160 Mac Minis at the exact same time and they all come back up in well under a minute. The XServe isn’t even breathing hard, CPU is under 20% and the SSD’s aren’t even accessed once the NetBoot image is in RAM, and the machine only uses 4 of the 10Gb Ethernet link. I’m going to add the Small tree card to the 3 other NetBooted racks I have.

    If I can be of more help, please let me know.

    Steven

  67. Repollo

    It would be awesome to get everything mechanical related thats left, (eg. electric fans), out of the picture and you would have probably made the first timeless machines of all times. Keep up the awesome work. Hope to have mine soon!

  68. Pingback: 160 Mac Mini dans une baie de serveurs

  69. Pingback: Datacenter-Rack mit 160 Mac Minis | GIZMODO DEGIZMODO DE

  70. twicked

    Really cool design and idea, my only question/comment is how will you deal with building either more or replacing your current xserve server? Since even in such a impressive and complex system you still need a xserve, which I sadly regret Apple retiring. Besides that though, super cool. I honestly love this idea, and think it’s cool your company was willing to drop a dime on 160 mac minis, my issue is less with how you made it possible more so how Apple gave you only one way to do it. But kudos on coming up with a solution, I only wish Apple would work more with companies to deliver business hardware on a workable scale.

  71. Pingback: Centro de datos con 160 Mac mini: la plantación de manzanas | Applesencia

  72. Pingback: This is what 160 Mac Minis in a rack look like | SiliconChaos

  73. Pingback: 160 Mac Minis im Rack.. « schillerfrau

  74. Jim

    What’s your process for getting new systems from their out-of-the-box config (where they want to boot from the factory installed OS on the internal disk) to NetBooting? Presumably, you’re not hooking up a keyboard to each and holding down N. :)

    Thanks,
    Jim

  75. obmibmis Post author

    Jim,

    Unfortunately, we are doing just that, only instead of the “n” key we hold the “option” key and choose our custom installer, once our custom image is on the machine, we choose our “Build and Test” NetBoot image, and off they go! Once they are in the Build and test Pool, we have custom scripting that allows us to change the image they NetBoot between test cycles, That scripting is built around the “bless” command. I hope to have some time to write up the entire process on this blog, and I may include some of my notes from the last time I spoke at the Splunk conference.

  76. Michael Price

    Hi Steve, you have inspired me to build a few of these racks for internal purposes once the 2014 mini refresh is out in a few months. Q? Source of the racks and plastic heat redirectors to order them and any special instructions I should tell them? Also did you put a power strip and gigabit switch in each rack so 2 connections out the back or insufficient room so 8 connections out the back?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s