Contexto: Dileep Bhandarkar : Na época, "Distinguished Engineer at Microsoft, responsible for Cloud Server Hardware and Datacenter Infrastructure". Ex- Texas Instruments, Digital (17 anos), Intel (12), Microsoft (5). Atualmente Vice President, Technology at Qualcomm.
James Hamilton : Vice President and Distinguished Engineer on the Amazon Web Services team where he is focused on infrastructure efficiency, reliability, and scaling (2008-) Ex-IBM (10 anos) e ex-Microsoft (12) - Hamilton's research on the use of data center containers has been adopted in Microsoft’s design of its cloud computing data centers.
Dileep’s talk: Watt Matters in Energy Efficiency
James Hamilton's notes:
-Microsoft Datacenter Capacities:
o Quincy WA: 550k sq ft, 27MW o San Antonio Tx: 477k sq ft, 27MW o Chicago, Il: 707k sq ft, 60MW § Containers on bottom floor with “medium reliability” (no generators) and standard rooms on top floor with full power redundancy o Dublin, Ireland: 570k sq ft, 27MW
· Speeds and feeds from Microsoft Consumer Cloud Services: o Windows Live: 500M IDs o Live Hotmail: 355M Active Accounts o Live Messenger: 303M users o Bing: 4B Queries/month o Xbox Live: 25M users o adCenter: 14B Ads served/month o Exchange Hosted Services: 2 to 4B emails/day
· Datacenter Construction Costs o Land: <2% o Shell: 5 to 9% o Architectural: 4 to 7% o Mechanical & Electrical: 70 to 85%
· Summarizing the above list, we get 80% of the costs scaling with power consumption and 10 to 20% scaling with floor space.
· Reports that datacenter build costs are $10M to $20M per MW and server TCO is the biggest single category. I would use the low end of this spectrum for a cost estimator with inexpensive facilities in the $9M to $10M/MW range.
· PUE is a good metric for evaluating datacenter infrastructure efficiency but Microsoft uses best server performance per watt per TCO$ o Optimize the server design and datacenter together · Cost-Reduction Strategies: o Server cost reduction: § Right size server: Low Power Processors often offer best performance/watt § Eliminate unnecessary components (very small gain) § Use higher efficiency parts § Optimize for server performance/watt/$ (cheap and low power tend to win at scale) o Infrastructure cost reduction: § Operate at higher temperatures § Use free air cooling & Eliminate chillers § Use advanced power management with power capping to support power over-subscription with peak protection
· Custom Server Design: o 2 socket, half-width server design (6.3”W x 16.7”L) o 4x SATA HDD connectors o 4x DIMM slots per CPU socket o 2x 1GigE NIC o 1x 16x PCIe slot
· Custom Rack Design: o 480 VAC 3p power directly to the rack (higher voltage over a given conductor size reduces losses in distribution) o Very tall 56 RU rack (over 98” overall height) o 12VDC distribution within the rack from two combined power supplies with distributed UPS o Power Supplies (PSU) § Input is 480VAC 3p § Output: 12V DC § Servers are 12VDC only boards § Each PSU is 4.5KW § 2 PSUs/rack so rack is 9.0KW max o Distributed UPS § Each PSU includes an UPS made up of 4 groups of 40 13.2V batteries § Overall 160 discrete batteries per UPS o By putting 2 PSUs per rack they avoid transporting low voltage (12VDC) further than 1/3 of a rack distance (under 1 yard) and only 4.5 KW is transported so moderately sized bus bars can be used. o Rack Networking: § Rack networking is interesting with 2 to 4 tor switches per rack. We know the servers are 1 GigE connected and there are up to 96 per rack which yields two possibilities: 1) they are using 24 port GigE switches or 2) they are using 48 port GigE and fat tree network topology. 24 port TORs are not particularly cost effective, so I would assume 48x1GigE TORs in a fat tree design which is nice work. o Report that the CPUs can be lower power parts including Intel Atom, AMD Bobcat, and ARM
· Power proportionality: Shows that server at 0% load consumes 32% of peak power (this is an amazingly good server – 45% to 60% is much closer to the norm)