Video - ABR and Background Jobs

Adaptive bit-rate streaming with HLS, running slow jobs in the background, and playing video on the front end

Video Uploads

Uploading Video

• We now support file uploads
  • We allow users to upload images
• How do we support video uploads?
  • The same exact way as images!
  • Files are just bytes
• HW3 upload form: A multipart request with parts named "title", "description", and "video"
  • Parse the parts, save the video to disk, store its path in your database
  • Host it with Content-Type: video/mp4 and support Range requests

The Problem

• You host a video on your web app
• You want high quality so you host a large 1080p mp4
  • The entire file is 100's of MB
• Range requests let users start watching without downloading the entire file
• But every user gets the same quality
  • Someone visits your site using eduroam on a bad day..
  • The video buffers, stutters, or doesn't play at all
• We need a solution that:
  • Allows users with slow connections to enjoy your content
  • Delivers high quality to users with high-speed Internet

Adaptive Bit-Rate Streaming

Segments

• Split the video into many short files called segments
  • Typically ~2-10 seconds of playback each
• The browser downloads one segment at a time
  • Only a few seconds need to be downloaded before playback starts
  • If the user skips around in the video, only request the segment they skipped to
  • If the user leaves the page without finishing the video, the rest is never downloaded
• Each segment is a small, regular file
  • No Range requests needed

Adaptive Bit-Rate

• Instead of hosting a single video at a single resolution
  • Host multiple versions of the same video at different resolutions
  • Each resolution requires a different bit-rate to stream
• Create segments for every version
• With the video split into ~2-10 second segments
  • The browser can switch to a different version at any segment
  • It chooses the highest bit-rate that fits its current bandwidth

Adaptive Bit-Rate

• The browser adapts the requested bit-rate based on current conditions
• Limited interruption for the user, though quality can change over time

Segment	1	2	3	4	5	6	7
Bandwidth	Fast	Fast	Drops	Slow	Slow	Recovers	Fast
Quality	1080p	1080p	480p	potato	potato	480p	1080p

HLS vs MPEG-DASH

• Two major protocols for segmented, adaptive bit-rate video
  • Both use an index file listing the segments. The browser requests segments as it needs them
• HTTP Live Streaming (HLS) - Developed by Apple
  • Index files are .m3u8 playlists. Segments are commonly .ts files
  • Wide-spread support. Spec freely available in RFC8216
• Dynamic Adaptive Streaming over HTTP (MPEG-DASH) - Developed by MPEG
  • Index file is an XML .mpd file
  • Not supported on Apple devices
  • Spec published as ISO/IEC 23009-1 - Available for $245 (!)
• We'll use HLS

HLS Files

• A master playlist lists every version (rendition) of the video
• Each rendition has its own media playlist listing its segments
• Each segment is a few seconds of video and audio
• Your server hosts all of these files
• The browser only needs the URL of the master playlist

public/videos/abc123/
├── master.m3u8
├── 0/
│   ├── index.m3u8
│   ├── index0.ts
│   ├── index1.ts
│   └── index2.ts
└── 1/
    ├── index.m3u8
    ├── index0.ts
    ├── index1.ts
    └── index2.ts

Master Playlist

• master.m3u8 lists each rendition
  • Its bandwidth, resolution, and codecs
  • The path to its media playlist, relative to this file
• The browser chooses a rendition based on its bandwidth

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=2825900,RESOLUTION=1280x720,CODECS="avc1.f4001f,mp4a.40.2"
0/index.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=955900,RESOLUTION=640x360,CODECS="avc1.f4001e,mp4a.40.2"
1/index.m3u8

Media Playlist

• 1/index.m3u8 lists the segments of the 360p rendition
  • #EXTINF is the duration of the next segment in seconds
  • #EXT-X-ENDLIST means there are no more segments
• Segment paths are relative to the playlist

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:8
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:8.333333,
index0.ts
#EXTINF:8.333333,
index1.ts
#EXTINF:3.333333,
index2.ts
#EXT-X-ENDLIST

Watching an HLS Video

• The browser requests each file as it needs it

GET /public/videos/abc123/master.m3u8          Pick a rendition
GET /public/videos/abc123/1/index.m3u8         Start with 360p
GET /public/videos/abc123/1/index0.ts
GET /public/videos/abc123/1/index1.ts          Bandwidth looks good
GET /public/videos/abc123/0/index.m3u8         Switch to 720p
GET /public/videos/abc123/0/index2.ts

• These are all normal GET requests for files
  • Your server doesn't need to know anything about HLS to host them
  • Just the right MIME types

Converting Video

Media Processing

• You do not always want to store user uploaded media as-is
  • Users can upload anything. Never trust your users!
  • Even your most trustworthy users will upload large media files
• Process the media when it's uploaded
  • Convert/Compress the video
  • Store and serve the converted files
  • All media stored in formats you choose
• For ABR: Convert each upload into HLS at several bit-rates

ffmpeg

• ffmpeg is the answer for video manipulation
• A command line program
  • Your Go code runs it with a system call
  • Same as typing the command into the command line

exec.Command("ffmpeg", "-i", "input.avi", "-f", "mp4", "output.mp4").Run()

• ffmpeg must be installed in your container
  • Add this near the top of your Dockerfile so it's cached and you only wait for the install once

RUN apt-get update && apt-get install -y ffmpeg

The Provided Conversion

• You are not expected to research ffmpeg for the HW
• This function converts an mp4 into HLS with 2 renditions: 720p and 360p
• Let's see what each argument does

func MP4ToHLS(inputPath string, outputDir string) error {
    return exec.Command("ffmpeg", "-i", inputPath,
        "-map", "0:v:0", "-map", "0:a:0", "-map", "0:v:0", "-map", "0:a:0",
        "-filter:v:0", "scale=-2:720", "-b:v:0", "2500k",
        "-filter:v:1", "scale=-2:360", "-b:v:1", "800k",
        "-f", "hls", "-hls_playlist_type", "vod",
        "-master_pl_name", "master.m3u8",
        "-var_stream_map", "v:0,a:0 v:1,a:1",
        outputDir+"/%v/index.m3u8",
    ).Run()
}

The Provided Conversion

• -i is the input file
• -map selects streams from the input for the output
  • 0:v:0 is the first video stream of the first (only) input. 0:a:0 is its first audio stream
  • Mapped twice: Video and audio for each rendition
• If the input has no audio, 0:a:0 matches nothing and ffmpeg fails

func MP4ToHLS(inputPath string, outputDir string) error {
    return exec.Command("ffmpeg", "-i", inputPath,
        "-map", "0:v:0", "-map", "0:a:0", "-map", "0:v:0", "-map", "0:a:0",
        "-filter:v:0", "scale=-2:720", "-b:v:0", "2500k",
        "-filter:v:1", "scale=-2:360", "-b:v:1", "800k",
        "-f", "hls", "-hls_playlist_type", "vod",
        "-master_pl_name", "master.m3u8",
        "-var_stream_map", "v:0,a:0 v:1,a:1",
        outputDir+"/%v/index.m3u8",
    ).Run()
}

The Provided Conversion

• Output video stream 0: Scaled to 720 pixels tall at 2500 kb/s
• Output video stream 1: Scaled to 360 pixels tall at 800 kb/s
• -2 for the width: Compute the width that preserves the aspect ratio
  • Rounded to an even number (Required by the encoder)

func MP4ToHLS(inputPath string, outputDir string) error {
    return exec.Command("ffmpeg", "-i", inputPath,
        "-map", "0:v:0", "-map", "0:a:0", "-map", "0:v:0", "-map", "0:a:0",
        "-filter:v:0", "scale=-2:720", "-b:v:0", "2500k",
        "-filter:v:1", "scale=-2:360", "-b:v:1", "800k",
        "-f", "hls", "-hls_playlist_type", "vod",
        "-master_pl_name", "master.m3u8",
        "-var_stream_map", "v:0,a:0 v:1,a:1",
        outputDir+"/%v/index.m3u8",
    ).Run()
}

The Provided Conversion

• -f hls: Output HLS
• -hls_playlist_type vod: Video on Demand
  • Keep every segment in the playlist and end it with #EXT-X-ENDLIST
  • (As opposed to a live stream)
• -master_pl_name: Also write a master playlist named master.m3u8

func MP4ToHLS(inputPath string, outputDir string) error {
    return exec.Command("ffmpeg", "-i", inputPath,
        "-map", "0:v:0", "-map", "0:a:0", "-map", "0:v:0", "-map", "0:a:0",
        "-filter:v:0", "scale=-2:720", "-b:v:0", "2500k",
        "-filter:v:1", "scale=-2:360", "-b:v:1", "800k",
        "-f", "hls", "-hls_playlist_type", "vod",
        "-master_pl_name", "master.m3u8",
        "-var_stream_map", "v:0,a:0 v:1,a:1",
        outputDir+"/%v/index.m3u8",
    ).Run()
}

The Provided Conversion

• -var_stream_map: Group the streams into renditions
  • Rendition 0: video 0 with audio 0. Rendition 1: video 1 with audio 1
• %v in the output path is replaced by the rendition number
  • outputDir/0/index.m3u8 and outputDir/1/index.m3u8
• Run waits until ffmpeg finishes and returns an error if it failed

func MP4ToHLS(inputPath string, outputDir string) error {
    return exec.Command("ffmpeg", "-i", inputPath,
        "-map", "0:v:0", "-map", "0:a:0", "-map", "0:v:0", "-map", "0:a:0",
        "-filter:v:0", "scale=-2:720", "-b:v:0", "2500k",
        "-filter:v:1", "scale=-2:360", "-b:v:1", "800k",
        "-f", "hls", "-hls_playlist_type", "vod",
        "-master_pl_name", "master.m3u8",
        "-var_stream_map", "v:0,a:0 v:1,a:1",
        outputDir+"/%v/index.m3u8",
    ).Run()
}

Hosting HLS Files

• Two new file types to host

Extension	MIME type
.m3u8	application/vnd.apple.mpegurl
.ts	video/mp2t

• The files are in nested directories
  • eg. /public/videos/abc123/0/index0.ts
• Store the path to master.m3u8 in your database as the video's hls_path

Background Jobs

Conversions Are Slow

• Converting a video to multiple bit-rates can take a long time
  • Seconds for a short clip. Up to minutes for a long video
  • Uses a lot of CPU
• If your server converts the video while handling the upload request:
  • The user waits on the upload page with no response
  • The browser may give up and time out
  • Nothing else can happen on that keep-alive connection until the conversion finishes
  • Users will think your app is broken
• Respond before the conversion finishes!

Respond First, Process Later

1. Receive and buffer the upload
2. Save the mp4 to disk and add the video to your database
3. Respond to the user
4. Start the conversion in the background concurrently
5. Handle other requests from this user on the same connection
6. When the conversion finishes, update the video in your database

• The video can be watched as an mp4 while it's being converted
• Users see the HLS version once it's ready

Video on the Front End

The Video Element

• Can't use <img> for video
• Use the <video> element!
• Attributes:
  • controls: Displays the control buttons for the user
  • autoplay: Plays on page load [if allowed by the browser]
  • muted: Mute the audio (Required in Chromium for autoplay)

<video width="400" controls autoplay muted>
    <source src="space.mp4" type="video/mp4">
    Your browser does not support video playback
</video>

The Video Element

• Specify the file in a <source> element
  • Requested just like the src of an <img>
• Can list multiple sources
  • The browser uses the first one it supports
• The text is displayed by very old browsers that don't support video
• Can also set src on the video element
  • The provided front end does this in JavaScript

<video width="400" controls autoplay muted>
    <source src="space.mp4" type="video/mp4">
    Your browser does not support video playback
</video>

videoPlayer.src = video.video_path;

Playing HLS

• Most browsers cannot play HLS using only the <video> element
  • Safari can. Many others can't
  • You cannot rely on the browser to support it
• Use a JavaScript video player
  • It downloads the playlists and segments, and feeds them to the <video> element
• The provided front end uses hls.js
  • Included in layout.html

<script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>

The Provided Front End

• The video page requests the video from your API
• If the video has an hls_path, play it with hls.js
• Otherwise, play the mp4 directly
  • With Range requests if your server and browser supports them

const response = await fetch(`/api/videos/${videoId}`);
const video = (await response.json()).video;

if (video.hls_path) {
    const hls = new Hls();
    hls.loadSource(video.hls_path);
    hls.attachMedia(videoPlayer);
} else {
    videoPlayer.src = video.video_path;
}

Further Reading

• RFC8216 - HLS
• ffmpeg