Today as part of ACM Chennai Chapter Lectures Dr.David Chaiken, Chief Architect at Yahoo! Inc. delivered a talk on Architecture at internet-scale at the ICSR Auditorium of IIT Madras.
He started by sharing data points about Yahoo!: 640+ Million users, 4.5 Billion Pageviews per day, 368M User visits/Month. Yahoo! Mail has over 450M Mailboxes and does over 5B+ Deliverables/Day
The talk was structured into six parts:
Science, Art & Scale
Competing needs: Agility & Stability
Data Center Innovation (like the new New York Datacenter of Yahoo!)
He defined availability as below: (Definitions: MTTF, MTTD, MTTR)
He talked about an incident in late 2008 that affected 1% more of users in their advertisement platforms even after extensive testing & deployment. Their failsafe mechanisms failed too. The moral is that “if you don’t test your failsafe regularly, they don’t work”. They had a Byzantine failure where your infrastructure itself becomes your enemy. This reinforces what you learn in Computer science classes on Exponential Algorithms for common time
Science can help even in simple search results to take them beyond ten blue links. For example, see the integrated Cricket Information web part that shows up when you search for MS Dhoni in Yahoo! search
Yahoo! researchers and scientists contribute more to consumer products and teams than any other new media company
In about 10 Milliseconds Yahoo’s backend systems gather any stored information about an incoming user based on Cookie, ID or Mobile Number
Summary – in Yahoo! Hadoopis the standard for doing async & batch processing tasks. Over the years he expects Hadoop to gain near real-time update capabilities as well. Yahoo! contributes nearly 70%+ to Hadoop project and Yahoo’s Cloud infrastructure including Hadoop is completely open source.