• 7 Years of YouTube Scalability Lessons in 30 Minutes

  • YouTube 서비스 규모 확장에 대한 7년간의 경험을 30분만에 발표하기

  • If you started out building a dating site and instead ended up building a video sharing site (YouTube) that handles 4 billion views a day, then it’s just possible you learned something along the way. And indeed, Mike Solomon, one of the original engineers at YouTube, did learn a lot and he has given a talk about it at PyCon: Scalability at YouTube.
  • 처음에는 데이팅 사이트로 시작했다 나중엔 하루에 사람들이 4 억 번 시청 하는 비디오 공유 사이트(YouTube)를 만들게 됐다면, 그 과정에서 뭔가 배운 것이 있을 것이다. YouTube 창업 멤버 한 명 이었던 마이크 솔로몬(Mike Solomon)은 실제로 많은 것을 배웠고, 이를 PyCon 에서 "YouTube 의 규모 확장성" 이라는 주제로 발표했다
  • This isn’t an architecture driven talk where we are led through a description of how a lot of boxes connect to each other. Mike could give that sort of talk. He has worked on building YouTube’s servlet infrastructure, video indexing feature, video transcoding system, their full text search, a CDN, and much more. But instead, he’s taken a step back, took a long look around at what time has wrought, and shared some deep lessons, obviously hard won from experience.
  • 이것은 어떻게 수많은 박스들이 서로 연결되는지 우리에게 설명해주는 아키텍처 중심 발표가 아니다. 마이크는 이런 (아키텍처 중심의) 발표를 할 수도 있었다. 그는 유튜브의 서버 인프라, 비디오 인덱싱, 비디오 트랜스코딩 시스템, 텍스트 검색, CDN 그 외의 많은 것들의 개발에 참여 했다. 하지만 그는 대신 한걸음 물러나 시간이 어떤 변화를 가져왔는지 자세히 들여다보고, 경험을 통해서 어렵게 얻은 깊은 교훈들을 사람들과 나누었다.
  • The key takeaway away of the talk for me was doing a lot with really simple tools. While many teams are moving on to more complex ecosystems, YouTube really does keep it simple. They program primarily in Python, use MySQL as their database, they’ve stuck with Apache, and even new features for such a massive site start as a very simple Python program.
  • 이 발표에서 내가 얻은 가장 중요한 교훈은 정말 간단한 툴을 가지고 수많은 일을 하는 것이다. 많은 팀들이 더욱 복잡한 에코시스템으로 옮겨가지만, 유튜브는 정말 심플하게 유지하고 있다. 프로그램은 주로 Python, 디비는 MySQL, Apache에 매여있고, 심지어 이 거대한 사이트의 새로운 기능들도 매우 간단한 Python프로그램으로 시작한다.
  • That doesn’t mean YouTube doesn’t do cool stuff, they do, but what makes everything work together is more a philosophy or a way of doing things than technological hocus pocus. What made YouTube into one of the world’s largest websites? Read on and see...
  • 그렇다고 유튜브가 멋진 일들을 하지 않는다는 것은 아니다. 하지만 이 모든것을 가능하게 하는 것은 이상한 기술적 장난이 아니라 철학 이며, 일하는 방식이다. 무엇이 유튜브를 전 세계에서 가장 성공한 웹사이트중 하나로 만들었을까? 한번 읽어 보시라...
  • Stats

  • 현재 상태

  • - 4 billion Views a day
  • - 하루에 4 억 번 조회
  • - 60 hours of video is uploaded every minute
  • - 매 분당 60시간에 해당하는 영상이 업로드
  • - 350+ million devices are YouTube enabled
  • - 3억 5천만개 이상의 기기가 YouTube 동영상 재생을 지원
  • - Revenue double in 2010
  • - 2010년에만 수익이 두 배 증가
  • - The number of videos has gone up 9 orders of magnitude and the number of developers has only gone up two orders of magnitude.
  • - 총 비디오 9 자릿수 증가. 개발자 수는 겨우 두자릿수 증가.
  • - 1 million lines of Python code
  • - 백만줄의 Python 코드로 작성 됨
  • Stack

  • - Python - most of the lines of code for YouTube are still in Python. Everytime you watch a YouTube video you are executing a bunch of Python code.
  • - Python - YouTube의 대부분의 코드는 여전히 Python으로 되어 있다. YouTube 동영상을 볼 때마다 사실 여러분은 Python 코드를 실행 하고 있는 것이다.
  • - Apache - when you think you need to get rid of it, you don’t. Apache is a real rockstar technology at YouTube because they keep it simple. Every request goes through Apache.
  • - Apache(아파치 웹 서버) - 이걸 없애고 싶다는 생각이 들더라도 그러지 마라. (역자 주: 요즘 아파치 말고 Nginx 등으로 바꾸는 경우가 많아서 이렇게 말한 것 같습니다.) Apache는 유튜브에서 정말 중요한 기술인데, 단순하기 때문이다. 모든 요청은 Apache를 경유한다.
  • - Linux - the benefit of Linux is there’s always a way to get in and see how your system is behaving. No matter how bad your app is behaving, you can take a look at it with Linux tools like strace and tcpdump.
  • - Linux - 리눅스 시스템을 사용 할 때의 장점은 시스템이 어떤 상황에서든 시스템이 돌아가는 상황을 들여다 볼 수 있다는 점이다. strace 나 tcpdump 같은 리눅스 툴을 이용하면 여러분이 작성한 응용프로그램이 얼마나 심각하게 오동작 하든 이를 들여다 볼 수 있다.
  • - MySQL - is used a lot. When you watch a video you are getting data from MySQL. Sometime it’s used a relational database or a blob store. It’s about tuning and making choices about how you organize your data.
  • - MySQL - MySQL은 정말 많이 사용된다. 동영상을 감상할 때마다 여러분은 MySQL 로부터 데이터를 받고 있다고 할 수 있다. 관계형 데이터베이스로 사용되기도 하고 BLOB 데이터 저장소로 사용되기도 하는데, 이는 성능 최적화나 데이터를 어떻게 조직화 할 지 선택 하는데 따라 결정 된다.
  • - Vitess - a new project released by YouTube, written in Go, it’s a frontend to MySQL. It does a lot of optimization on the fly, it rewrites queries and acts as a proxy. Currently it serves every YouTube database request. It’s RPC based.
  • - Vitess - 유튜브가 내놓은 MySQL의 프론트엔드인데 Go 언어로 작성됐다. 프록시처럼 동작하면서 쿼리(SQL 질의문)를 재작성고 즉석에서 많은 최적화를 수행한다. 현재 모든 YouTube의 데이터베이스 요청이 이것을 통해 이뤄진다. RPC 기반.
  • - Zookeeper - a distributed lock server. It’s used for configuration. Really interesting piece of technology. Hard to use correctly so read the manual
  • - Zookeeper - 분산 락 서버. 설정을 위해 사용한다. 정말 흥미로운 기술이다. 제대로 사용하기 어려우니 메뉴얼을 읽어라.
  • - Wiseguy - a CGI servlet container.
  • - Wiseguy - CGI 서블릿 컨테이너.
  • - Spitfire - a templating system. It has an abstract syntax tree that let’s them do transformations to make things go faster.
  • - Serialization formats - no matter which one you use, they are all expensive. Measure. Don’t use pickle. Not a good choice. Found protocol buffers slow. They wrote their own BSON implementation which is 10-15 time faster than the one you can download.
  • General Lessons

  • 일반적인 교훈들

  • - Tao of YouTube: choose the simplest solution possible with the loosest guarantees that are practical. The reason you want all these things is you need flexibility to solve problems. The minute you over specify something you paint yourself into a corner. You aren’t going to make those guarantees. Your problem becomes automatically more complex when you try and make all those guarantees. You leave yourself no way out.
  • - YouTube 의 도: 현실적으로 최대한 느슨하게 보장하는, 가능한 한 가장 단순한 해결책을 선택하라. 이런것들이 필요한 이유는 문제를 풀기 위해선 유연해져야 하기 때문이다. 무엇인가를 너무 상세하게 명시하는 순간 자신을 코너에 몰아 넣게 된다. 무엇도 보장 해서는 안된다. 그런 보장을 하려고 하면 당신의 문제는 자동으로 더욱 복잡해진다. 스스로 막다른 골목에 자신을 몰아넣게 되는 것이다.
  • note icon
    guarantees 가 내가 아는 의미 말고도 다른게 있는건가(...)
  • - That whole process is what scalability is about. A scalable system is one that’s not in your way. That you are unaware of. It’s not buzz words. It’s a general problem solving ethos.
  • - 전체 개발 과정은 규모확장성에 대한 것이다. 규모 확장 가능한 시스템은. 유행어도 아니고 일반적인 문제 해결 방식이다.
  • - Hallmark of big system design: Every system is tailored to its specific requirements. Everything depends on the specifics of what you are building.
  • - YouTube is not asynchronous, everything is blocking.
  • - Believes more in philosophy than doctrine. Make it simple. What does that mean? You’ll know when you see it. If you do code review that changes thousands of lines of code and many files then there was probably a simpler way. Your first demo should be simple, then iterate.
  • - To solve a problem: One word - simple. Look for the most simple thing that will address the problem space. There are lots of complex problems, but the first solution doesn’t need to be complicated. The complexity will come naturally over time.
  • - A lot of YouTube systems start as one Python file and become large ecosystems after many many years. All their prototype were written in Python and survived for a surprising amount of time.
  • - In a design review:
  • - What’s the first solution?
  • - How are you going to iterate?
  • - What do we know about how this data is going to be used?
  • - Things change over time. How YouTube started out has no bearing on what happens later. YouTube started out as a dating site. If they had designed for that they would have different conversation. Stay flexible.
  • - YouTube CDN. Originally contracted it out. Was very expensive so they did it themselves. You can build a pretty good video CDN if you have a good hardware dude. You build a very large rack, stick machines in, then take lighttpd, and then override the 404 handler to find the video that you didn’t find. That took two weeks and it’s first day served 60 gigabits. You can do a lot with really simple tools.
  • - You have to measure. Vitess swapped out one its protocols for an HTTP implementation. Even though it was in C it was slow. So they ripped out HTTP and did a direct socket call using python and that was 8% cheaper on global CPU. The enveloping for HTTP is really expensive.
  • Scalability Techniques

  • 규모 확장 기법

  • - These are not new ideas, but it’s amazing how a few core ideas can apply in a lot different dimensions.
  • - 뭔가 새로운 개념이 있는게 아니라, 몇 가지 핵심 개념만 잡아두면 놀랍게도 수많은 각기 다른 분야에 적용 할 수 있다.
  • - Divide and Conquer - The Scalability Technique
  • - This is the scalability technique. Everything is about partitioning out work. Deciding how to execute it. Applies to many things, from web tier, you have a lot of web servers that are more or less identically and independently and you grow them horizontally. That’s divide and conquer.
  • - This is the crux of database sharding. How do you partitions things out and communicate between the parts that you’ve subdivided. These are things you want to figure out early on because they influence how you grow.
  • - Simple and loose connections are really valuable.
  • - 단순하고 느슨한 결합은 정말로 중요하다.
  • - The dynamic nature of Python is a win here. No matter how bad your API is you can stub or modify or decorate your way out of a lot of problems.
  • - Approximate Correctness - Cheat a Little
  • - Another favorite technique. The state of the system is that which it is reported to be. If a user can’t tell a part of the system is skewing and inconsistent, then it’s not.
  • - A real world example. If you write a comment and someone loads the page at the same time, they might not get it for 300-400ms, the user who is reading won’t care. The writer of the comment will care, so you make sure the user who wrote the comment will see it. So you cheat a little bit. Your system doesn’t have to have globally consistent transactions. That would be super expensive and overkill. Not every comment is a financial transaction. So know when you can cheat.
  • - Expert Knob Twiddling
  • - Ask, what do you know about your consistency model? For comments is eventually consistent good enough? Renting a movie is different. When renting there’s money so we’ll do the best we can to never lose that. Different consistency models are needed depending on the data.
  • - Jitter - Add Entropy Back into Your System
  • - Hot word in their group all of the time. If your system doesn’t jitter then you get thundering herds. Distributed applications are really weather systems. Debugging them is as deterministic as predicting the weather. Jitter introduces more randomness because surprisingly, things tend to stack up.
  • - For example, cache expirations. For a popular video they cache things as best they can. The most popular video they might cache for 24 hours. If everything expires at one time then every machine will calculate the expiration at the same time. This creates a thundering herd.
  • - By jittering you are saying randomly expire between 18-30 hours. That prevents things from stacking up. They use this all over the place. Systems have a tendency to self synchronize as operations line up and try to destroy themselves. Fascinating to watch. You get slow disk system on one machine and everybody is waiting on a request so all of a sudden all these other requests on all these other machines are completely synchronized. This happens when you have many machines and you have many events. Each one actually removes entropy from the system so you have to add some back in.
  • - Cheating - Know How to Fake Data
  • - Awesome technique. The fastest function call is the one that doesn’t happen. When you have a monotonically increasing counter, like movie view counts or profile view counts, you could do a transaction every update. Or you could do a transaction every once in awhile and update by a random amount and as long as it changes from odd to even people would probably believe it’s real. Know how to fake data.
  • - Scalable Components - Make Your own Luck
  • - You can look at an API and get a good feel. Are the inputs well defined? Do you know what you are getting out? A lot of this ends up being about data. Have a tight specification of what data comes out every function and how it flows actually helps you understand the application without documentation. You can tell what’s happening before and after a function is called.
  • - In Python things tend to move towards RPCs. The structure of your code is based on the discipline of your programmers. So establish good conventions, when all else fails there’s an RPC wall so you know what goes in and what comes out.
  • - Your components will not be perfect. A component might last a month or six months, who knows. By drawing these lines you are making some of your own luck. When things go south you can swap it out and do something different. Sometimes that rewriting someing in python and C and sometimes that means getting rid of it entirely. You don’t know until you are able to observe.
  • - With so many people on a team nobody can know the whole system, so you need to define components. This is video transcode it’s distinct from video search. You want well defined subcomponents. It’s good software design. These things end up talking to each other so having a good data specification is helpful. The greatest sin he made was communication between the servlet layer and the template layer to be a dictionary. Very bad idea. Should have added a WatchPage and said a watch page had a video and some comments and some related videos. This caused a lot of problems because the dictionary can have a few hundred attributes. They don’t always make the right choice.
  • - Efficiency - Traded Off for Scalability
  • - Efficiency is traded off for scalability. The most efficient thing is to write it in C and cram it into one process, but that’s not scalable.
  • - Focus on the macro level, your components, and how they break out. Does it makes sense to do this an RPC or do it inline? Break it into a subpackage and just someday this may be different.
  • - Focus on algorithms. In Python the effort to implement a good algorithm is low. There’s the bisect module, for example, where you can take a list, do something meaningful, and serialize it to disk and read it back again. There’s a penalty versus C, but it’s very easy.
  • - Measurement. In Python measurement is like reading tea leaves. There’s a lot of things in Python that are counter intuitive, like the cost of grabage colleciton. Most of chunks of their apps spend their time serializing. Profiling serialization is very depending on what you are putting in. Serializing ints is very different than serializing big blobs.
  • - Efficiency in Python - Knowing What Not to Do
  • - Python 효율적으로 사용하기 - 무엇을 하지 말아야 하는가
  • - More about knowing what not to do. How dynamic you make things correlates to how expensive it is to run your Python app.
  • -- 무엇을 하지 말아야 하는지에 더 알아야 한다. 동적으로 연관 짓는 것이 Python 응용프로그램을 동작시키는데 있어 얼마나 비용이 드는가
  • - Dummer code is easier to grep for and easier to maintain. The more magical the code is the harder is to figure out how it works.
  • -- 덜 똑똑하게 작성 된 코드가 grep 하기도 편하고 유지보수 하기도 편하다. (너무 잘짜서) 마법같은 코드는 실제로 어떻게 동작하는지 이해하기 어렵더라.
  • - They don’t do a lot of OO. They use a lot of namespaces. Use classes to organize data, but rarely for OO.
  • -- 객체지향 잘 사용 안함. 네임스페이스도 많이 사용하고 데이터를 조직화 하기 위해 클래스를 사용하지만 객체 지향적인 경우는 드물더라
  • - What is your code tree going to look like? He wants these words to describe it: simple, pragmatic, elegant, orthogonal, composable. This is an ideal, reality is a bit different.
  • -- 여러분이 작성한 코드 트리가 어떠했으면 좋겠는가? 단순하고 실용적이고 우아하고 직교적이며 구성적인 코드라고들 표현한다. 이상적이긴 한데 현실은 다르더라
2 Comments
umteeh

umteeh • Apr 3rd, 2012

전 개발자가 아니라, 전문 용어등은 번역이 어렵네요. 제가 번역한 부분에 잘못된 부분있으면 고쳐 주셔요~ 감사합니다!
  • ACodeFarmer • Apr 3rd, 2012

    우오옷 감사합니다. ^^ 확인해볼께요~