2014CDNSummit-Comcast

Scaling theComcast CDNJan van Doornjan_vandoorn@cable.comcast.com

DisclaimerComcast uses CDNs in a number of ways but the CDN I will be discussing in this presenta:on relates to how Comcast specifically uses a CDN to deliver its IP cable services over its own network to its cable television customers. Scaling the Comcast CDN 2

What is a CDN?• Lots of Caches – The HTTP/1.1 compa:ble work horses in mul:ple :ers and edge loca:ons • Content Router – Get customer to best cache for his requested content in his loca:on • Health Protocol – A way to tell CR which caches are able to take work • Management and Monitoring System – A way to manage a geographically disperse set of servers • Repor;ng System – Log file analysis of edge, mid and CR contacts for (internal) billing and sizing Scaling the Comcast CDN 3

Why does Comcast need one?• X1 / X2 Cloud based UI – Images / UI elements – Code Downloads • Next genera;on video distribu;on – Video on Demand (VoD) – Live Television – Cloud DVR – Second Screen – First Screen Scaling the Comcast CDN 4

About Video and HTTP/1.1• Adap;ve BitRate (ABR) video delivery – Video gets “chopped up” in short chunks (2 – 6s) – Chunks are retrieved from server using HTTP – Adapts quality to resource availability and needs – S:ll video (high bandwidth) – Highly cacheable – HTTP KeepAlive – Live Television – VoD / cDVR Scaling the Comcast CDN 5

Buy vs. BuildThe decision Scaling the Comcast CDN 6

The Comcast CDN• The Caches – Apache Traffic Server -­‐ More on that later • Content Router – Built in-­‐house • Tomcat/Java applica:on • Simple / Stateless • DNS or 302 redirec:on Scaling the Comcast CDN 9

About Content Routinghhp://ccr.cdn.comcast.net/nbc/nightlynews/21040512.m3u8 • Goal is to get the client to the best cache for the content requested and the loca0on it is requested from based on – Distance/network hops – Network cost – Network link quality/speed – Health of the cache – Content group limits – Availability of content in cache Scaling the Comcast CDN 10

DNS Content Routinghhp://edge.cdn.comcast.net/x1_v33_guide/images/up_arrow.png • Content Router is DNS Auth • CR makes decision based on resolver, not based on client • CR only knows the FQDN – Unaware of path in URL, HTTP headers, query string, etc • Fast – Usually used for gejng web-­‐page objects, images, etc Scaling the Comcast CDN 11

HTTP Content Routinghhp://ccr.cdn.comcast.net/nbc/nightlynews/21040512.m3u8 • CR is DNS auth, but always responds with own IP address to DNS query • Client then does HTTP connec;on to CR – CR knows all the HTTP details – CR also knows client IP address • CR sends HTTP 302 redirec;on • Slower, but much more “precise”; usually used for longer sessions, like ABR video sessions – Client will “s:ck” to cache amer ini:al redirect Scaling the Comcast CDN 12

The Comcast CDN• Health Protocol – Tomcat / Java applica:on built in-­‐house Aggregates status from enhanced stats plugin in ATS to [J | L] status per server and content group • Management and Monitoring System – Web applica:on built in-­‐house • Perl / Mojolicious framework against rela:onal db • jQuery UI • Puppet & Nagios for OS • Repor;ng System – The only thing we bought Scaling the Comcast CDN 13

Why Traffic Server●●Any HTTP 1.1 Compliant cache will work We chose Apache Traffic Server (ATS) – Top Level Apache project (NOT hhpd!) – Extremely scalable and proven – Very good with our VoD load – Efficient storage subsystem uses raw disks – Extensible through plugin API – Vibrant and friendly development community – Added / modified handful of plugins for specific use cases – Ex: Signed URL, Invalidate content, sta:s:cs Scaling the Comcast CDN 14

Our Gen1 Cache• Off-­‐the-­‐shelf hardware -­‐ ride Moore's Law! • Spinning disks (!) – 24 900Gb SAS disks for caching – 2 mirrored OS drives • 192 Gbyte of memory, for live TV • 1x10GE ini;ally, 2x10GE upgrades being planned • Linux CentOS 6.2+ Scaling the Comcast CDN 15

Traffic Server Performance• Tested very well without major tweaks – Pushing 10 Gbps on Gen1 box with very wide VoD like dispersion – Disk u:l is even and at almost 100% • Using traffic separa;on feature (volume patch) – up to 40 Gbps per server with a realis:c traffic profile • Not sure if we want more… – Failure domain almost too large • Available test suites seem to not apply to our work load – Ended up wri:ng many test tools ourselves • Teaches you a lot about the protocols and the CDN Scaling the Comcast CDN 16

Open Source and Support• Not having a support number is scary! – Most Open Source projects now have third par:es selling support… but we’re flying solo on that as well – Omen more FUD than actual ra:onal reasons • The ac;ve community is really important here • DevOps model • DIY surgical patches for your problem are usually much faster than a release (from either a vendor, or an Open Source project) • Get someone on staff to become part of the Open Source community Scaling the Comcast CDN 17

Current Status• Serving 1 – 1.5 Peta Byte / day • Over 250 caches deployed and serving traffic – Over 5 PByte total storage capacity – Each has single 10GE (star:ng to upgrade to 20GE) • 25 edge cache groups (“clusters”) – 1.7 Tbps total edge capacity – 400 Gbps served at highest peak – 300 Gbps daily peak • 3 mid-­‐;er cache groups (“clusters”) • Origin off load > 85% (varies by content and demand) • All IPv6 / IPv4 – Client decides based on connec:vity, CDN is IPv6 all the way Scaling the Comcast CDN 18

Future Plans• Double size of CDN in 2014 • New Mid-­‐Tier cache – 4RU, 512G RAM, 288TB disk • Next Gen Edge Tier – S:ll defining, probably SSD based • Dedicated “Live TV” caches • Deeper deployment • Bener tools for Opera;ons • Go(lang) • Dynamic ;ers / Popularity based rou;ng • Open Source logging / analy;cs Scaling the Comcast CDN 19

Ques:ons? Scaling the Comcast CDN 20