Homework 3 - Multimedia

Introduction

You will continue to add features to your app from homework 2. In this assignment you will add multimedia to your app. By the end, users will be able to upload a profile picture, share videos with each other, watch videos at a quality that adapts to their connection, and live stream to other users of your app.

As with HW2, some objectives will require that you edit the provided front end, or build your own if you'd prefer.

All objectives must function properly using docker compose. When running "docker compose up" your app must be fully functional on localhost port 8080.

Security Concerns

Every security concern is clearly labeled. If you violate a security concern, all 5 objectives will be scored 0 unless/until you submit a proper security essay for each security risk in your submission.

Deviations

In addition to modifying the front end, or building your own, you are free to deviate from the specific details of each objective as long as you still implement the features for each objective. This offers you flexibility to implement your app the way you'd like, while still learning the concepts of each objective. There is no limit to the amount you are allowed to deviate as long as the concepts of the objective are still covered.

If you choose to deviate from a stated objective, you must justify how this deviation still covers the concepts required for the objective. Deviations never make a security concern acceptable. Any deviation that avoids handling a security concern does not cover the concepts of the objective and is not allowed (e.g. Allowing anyone to live stream without a stream key). Likewise, any deviation that uses a library to do the work an objective is teaching does not cover the concepts of the objective (e.g. Using a library to parse multipart requests or Range headers).

Objective 1: Profile Pictures

Goals:

  • Allow users to upload an image to use as their profile picture (avatar)
  • Handle requests that are too large to read without buffering
  • Parse multipart requests

Concepts Covered:

  • Buffering
  • Multipart form requests
  • Multimedia uploads
  • Hosting user-uploaded files

Pages

Add the following path, rendered using layout.html:

  • "/change-avatar" - Render change-avatar.html

Buffering

Until now, every request your server received was likely small enough to arrive in a single read from the TCP connection. Images and videos are not. A large upload will arrive over many reads, and your server must keep reading until it has received the entire body before it handles the request.

Read the Content-Length header of each request and continue reading from the connection until you have received exactly that many bytes of the body. Your buffering must work for arbitrarily large requests (e.g. a 1 GB video). You must write this buffering yourself. Functions that do the buffering for you (e.g. io.ReadAll or io.ReadFull on the connection) are not allowed for reading request bodies, and neither is attempting to read the entire body with a single, very large read. Any approach that does not require that you read the Content-Length header is not allowed (Since you are not doing the buffering).

You may assume that all the headers of a request, up to and including the first "\r\n\r\n", are received in the first read from the connection with a buffer of at least 2048 bytes (e.g. You do not have to buffer while reading the headers, and you can safely read the Content-Length before any buffering).


Parsing Multipart Requests

The change avatar form sends a multipart/form-data request containing the image chosen by the user. You must parse these requests yourself. Libraries that parse multipart requests (e.g. Go's mime/multipart package) are not allowed.

Since you'll parse multipart requests for both avatar and video uploads, it is recommended that you write a reusable function that takes an instance of your Request struct and returns a struct containing the boundary (from the Content-Type header) and all the parts of the request. Each part can be a struct containing its headers, its name (from its Content-Disposition header), and its content as bytes.

Tips:

  • Thoroughly test your parsing code and ensure that it is correct down to exact bytes. Having an extra CRLF at the end of you data for a part might corrupt your images and videos later in this assignment
  • The content of a part may be binary data (e.g. an image or video). Never convert it to a string or you will corrupt the data
  • The content of a part may contain the sequences "\r\n\r\n" and "\r\n". Be sure you are not splitting or corrupting the content when these appear in it

Avatar Uploads (`POST /api/users/avatar`)

The change avatar page sends the selected image in a part named "avatar". When your server receives a request at this endpoint:

  • Requests from users who are not logged in will be rejected with a 401 Unauthorized
  • Users can upload JPEG (Both .jpg and .jpeg), PNG, or GIF images. If the file is not one of these types, respond with a 400 and do not save the file. You may use the file extension, or MIME type of the part, to determine the file type. When serving uploaded files, you must set the correct MIME type
  • Save the image to disk using a naming convention of your choosing, with a file extension that matches the image type you identified, and store its path in your database. Do not use the user-supplied filenames. Do not store entire files in your database
  • Respond with a 200 OK and a message of your choice
  • Avatars must persist through a server restart

Displaying Avatars

  • Add an "imageURL" field to the response of GET "/api/users/@me" containing the path to the user's avatar (e.g. "imageURL": "/public/imgs/avatars/eda92e0a-eb7a-430b-a938-916d2102b480.png")
  • Add an "imageURL" field to every message returned by GET "/api/chats" containing the avatar of the message's author. As with display names in HW2, this must be their current avatar on all of their messages, including messages sent before they uploaded it
  • Users who have not uploaded an avatar, and guests, must have an "imageURL" for a default image of your choice

Tips:

  • If you reuse the same filename when a user replaces their avatar, your caching from HW1 may cause browsers to keep displaying the old image. Using a new filename for each upload avoids this
  • Files written inside a container are lost when the container is recreated. If you want uploads to survive "docker compose up --force-recreate", you may want to research and use volumes at this point. Volumes will be required for Objective 5, so setting one up now will be useful
Security Concern - XSS: There is a new XSS attack, besides HTML injection, that users can perform. Instead of uploading a valid image, they upload attack.js. To prevent this, ensure that MIME type sniffing is disabled and that you do not allow any MIME type other than they 3 listed above (e.g. Accepting an upload of attack.js and serving it with a MIME type of text/javascript will not be prevented by nosniff since the MIME type matches the type)

Optional (Optional features are not required, but you may want to add these features for real world apps):

  • This is a state-changing request made by an authenticated user, making it a target for XSRF attacks. You may want to add a XSRF token to this endpoint
  • File extensions are trivial for an attacker to spoof and they can upload javascript code with a .jpeg extension. A better method is to check the file header to verify the file type which is more difficult to fake

Objective 2: VideoTube

Goals:

  • Allow users to upload videos and share them with other users

Concepts Covered:

  • Multipart uploads with.. well, multiple parts
  • Combining multipart uploads with an API

Pages

Add the following paths, rendered using layout.html:

  • "/videotube" - Render videotube.html (Displays all videos)
  • "/videotube/upload" - Render upload.html (Upload video form)
  • "/videotube/videos/{videoID}" - Render view-video.html (Displays a single video. Host the same HTML for any videoID)

In this objective you will build a [simplified] YouTube clone where users can share videos with each other. The provided front end sends requests to the three endpoints below. The formats listed are what the provided front end expects.


Upload Videos (`POST /api/videos`)

The upload page sends a multipart request with parts named "title", "description", and "video". When your server receives a request at this endpoint:

  • Requests from users who are not logged in must be rejected with a 401 Unauthorized
  • You only need to support mp4 uploads for this endpoint. It is fine if you assume the upload is an mp4 video
  • Save the video to disk similar to how you did for images. Do not store the contents of the videos in your database
  • Respond with a 200 OK and a JSON object containing the id of the new video. The front end will redirect the user to the page for this video
  • Host uploaded videos with the MIME type "video/mp4"
  • Videos must persist through a server restart

Response (JSON): {"id": string}


Retrieve All Videos (`GET /api/videos`)

Return information about every video that has been uploaded in the following format:

Response (JSON): {"videos": [{"id": string, "author_id": string, "title": string, "description": string, "video_path": string, "created_at": string, "thumbnailURL": string}, ...]}

  • "created_at" is a timestamp that will be displayed on the front end. You may choose its format
  • The provided front end displays a thumbnail for each video using "thumbnailURL". Generating thumbnails is not required for this assignment. You may set this to an image of your choice, or edit the front end to display something else

Retrieve a Single Video (`GET /api/videos/{videoID}`)

Return all information about the video with an id matching {videoID}, using the same format as a single video from the previous endpoint. If no video has this id, respond with a 404.

Response (JSON): {"video": {"id": string, "author_id": string, "title": string, "description": string, "video_path": string, "created_at": string, "thumbnailURL": string}}

Security Concern - Escape HTML: Video titles and descriptions are user-provided content, and the provided front end inserts them into the page as HTML. You must escape any HTML in these fields before they are displayed to other users.

Tips:

  • Use short videos for most of your testing, then test at least one large video (several hundred MB) to be sure your buffering works properly

Optional:

  • Add a XSRF token to any state-changing endpoints
  • Check the file headers to verify the uploaded files are mp4
  • Extract a frame from the uploaded video to use as its thumbnail
  • Add support for video types other than mp4

Objective 3: Range Requests

Goals:

  • Allow browsers to request part of a video so playback can start, and the user can skip ahead, without downloading the entire file
  • Read and send only the bytes that were requested

Concepts Covered:

  • The Range, Content-Range, and Accept-Ranges headers
  • 206 Partial Content
  • 416 Range Not Satisfiable

When a browser plays an mp4, it doesn't need the entire file at once. If your server tells the browser that it supports range requests, the browser will ask for only the bytes it needs: the beginning of the file to start playback, then whichever part of the file the user skips to.


Requirements

  • Add support for range requests for every mp4 file your server hosts.
  • Every response for an mp4 file, including responses with the full file, must include the header "Accept-Ranges: bytes" so the browser knows it can request ranges
  • If a request for an mp4 file does not have a Range header, respond with the entire file and a 200 OK as you did before
  • If a request has a Range header requesting a range of bytes, respond with a 206 Partial Content containing only those bytes, and a Content-Range header with the appropiate value.
  • Support all three forms of a byte range:
    • "bytes={first}-{last}" - The bytes from position {first} through position {last}, inclusive. If {last} is past the end of the file, send through the end of the file
    • "bytes={first}-" - The bytes from position {first} through the end of the file
    • "bytes=-{n}" - The last {n} bytes of the file. If {n} is larger than the file, send the entire file
  • If {first} is past the end of the file, respond with a 416 Range Not Satisfiable with the header "Content-Range: bytes */{total}"
  • Functions that handle Range headers for you (e.g. http.ServeContent or http.ServeFile) are not allowed

Tips:

  • Ranges are inclusive on both ends. "bytes=0-99" is 100 bytes, not 99. Off-by-one errors are the most common bug in this objective
  • Browsers often request "bytes=0-" (the entire file) when a video first loads, then close the connection once they have enough to start playing. Your server must handle the connection being closed in the middle of a response without crashing
  • Since browser behavior can be unpredictable, you may want to test using curl and manually set the Range header of your test requests. Alternatively, you can write go code that will send custom requests to automate your testing

Optional:

  • For an open-ended range ("bytes={first}-"), you may send fewer bytes than requested by limiting each response to a maximum size (e.g. a few MB). As long as your Content-Range accurately describes the bytes you sent, the browser will request the rest when it needs them
  • Do not read the entire file. Only read the requested bytes to save on memory for range requests

Objective 4: Adaptive Bit-Rate Streaming

Goals:

  • Convert every uploaded video to HLS with multiple bit-rates
  • Perform these conversions with a background worker

Concepts Covered:

  • HLS
  • Adaptive Bit-Rate (ABR) streaming
  • Background jobs
  • Asynchronous processing

Converting a video to multiple bit-rates is slow. If your server converted each video while handling the upload request, the user would wait too long for a response leading to poor user experience. Instead, you will process this conversion in the background respond to your user immediately.


Provided Conversion Code

You are not expected to research ffmpeg for this objective. The function below converts an mp4 into HLS with two renditions (720p and 360p) with audio. You may use it as-is, or modify it if you prefer. Note that this function only works on videos with audio. If you want to support videos without audio, you will have to expand this code. 

// Converts the mp4 at inputPath into an HLS video with two
// qualities (720p and 360p) and saves the index file at outputDir/master.m3u8
func MP4ToHLS(inputPath string, outputDir string) error {
	return exec.Command("ffmpeg", "-i", inputPath,
		"-map", "0:v:0", "-map", "0:a:0", "-map", "0:v:0", "-map", "0:a:0",
		"-filter:v:0", "scale=-2:720", "-b:v:0", "2500k",
		"-filter:v:1", "scale=-2:360", "-b:v:1", "800k",
		"-f", "hls", "-hls_playlist_type", "vod",
		"-master_pl_name", "master.m3u8",
		"-var_stream_map", "v:0,a:0 v:1,a:1",
		outputDir+"/%v/index.m3u8",
	).Run()
}

For an outputDir of "public/videos/abc123", this function creates "public/videos/abc123/master.m3u8", which refers to the index files "0/index.m3u8" (720p) and "1/index.m3u8" (360p), each in its own directory with its .ts segment files. If ffmpeg fails, including for a video with no audio track, the function returns an error.

ffmpeg must be installed in your app's container. This function makes a system call that assumes ffmpeg is already installed. You can add RUN apt-get update && apt-get install -y ffmpeg to your Dockerfile to install it. You should add this line near the top of your Dockerfile, before doing anything with your app, so it will be cached and you only have to wait for the install once. You should also install ffmpeg locally if you intend to do any testing without Docker.


Background Worker

You will run the MP4ToHLS function in the background while your server responds to uploader with the following features:

  • POST "/api/videos" must respond as soon as the uploaded video is saved. It must not wait for a conversion to start or finish
  • Conversions must be performed concurrently (via goroutines). You must retain your keep-alive functionality and the video conversion must not block the next request on the connection
  • Clearly display to any user viewing a video that it is being processed during its conversion. You have flexibility in how you display this and you may need to edit the front end
  • When a conversion succeeds, add an "hls_path" field to the video containing the path to its master playlist (e.g. "hls_path": "/public/videos/video3/master.m3u8"). The provided front end plays the HLS video when this field is present and will show a quality selector for the different bit rates
  • Host the new file types with the appropriate MIME types

Tips:

  • Upload a video that is a few minutes long to give yourself time to view the page while it is processing
  • If you edit the videos title during the conversion to something like "[video title] -- processing" you can satisfy this requirement without changing the front end to display the processing status. Remember to remove this after processing is complete

Optional:

  • Build a queue of jobs and only allow 2 videos to be converted simultaneously. Running many conversions can be resource intensive and open an easy DoS opportunity for attackers. Instead, run at most 2 jobs simultaneously and queue any additional uploads
  • Alter the conversion to work with videos without audio in addition to videos with audio
  • Perform conversions in a separate Docker container using an image optimized for ffmpeg jobs, similar to the setup in the next objective. You may use an image that handles all the conversion logic out-of-the-box

Objective 5: Live Streaming

Goals:

  • Allow every user to live stream to your app using streaming software (e.g. OBS)
  • Issue each user their own secret stream key
  • Allow anyone to find and watch the streams that are currently live

Concepts Covered:

  • Live streaming (RTMP ingest to HLS)
  • Stream keys
  • Multi-container architecture and communication
  • Docker volumes
  • Live HLS playlists
  • Front end design

Let's add live-streaming to the app. You will issue every logged in user a stream key that they can use to live-stream to your other users. You do not have handle any of the RTMP ingest or HLS conversion logic. Instead, you will use an existing Docker image for you ingest service, similar to how we use an image to run Postgres. You task is to handle the rest of the logistics, including building a way to display stream keys and live streams on the front end.


The Ingest Server

Use the tiangolo/nginx-rtmp image as your ingest server with the configuration below. Save this configuration as nginx.conf in your project and copy it into the container at "/etc/nginx/nginx.conf". Replace "app" with the name of your server's service in your compose.yml, and change the two paths to the endpoints you choose. You may to use a second Dockerfile in a subdirectory to set this up, then use build: ./subdirectory in your compose.yml. 

events {}
rtmp {
    server {
        listen 1935;
        application live {
            live on;

            # Called when a stream starts and ends
            on_publish http://app:8080/api/live/start;
            on_publish_done http://app:8080/api/live/end;

            # Write each stream as HLS to /hls/{stream_key}/index.m3u8
            hls on;
            hls_path /hls;
            hls_nested on;
        }
    }
}

With this configuration:

  • A user streams by entering "rtmp://localhost:1935/live" as the server, and their stream key as the stream key in their streaming software
  • When a user starts streaming, the ingest server sends a POST request to the on_publish URL. The body of this request is URL-encoded and contains several fields, including "name", which is the stream key the user entered. Respond with a 200 OK to allow the stream, or with a 403 to reject it. Any stream with an invalid stream key should be rejected. If the stream is rejected, the ingest server disconnects the streamer
  • When a stream ends, the ingest server sends a POST request in the same format to the on_publish_done URL. Use this request to remove the stream from your active streams list
  • While a stream is live, the ingest server writes the stream as HLS to the "/hls/{stream_key}" directory inside its container. The playlist is "index.m3u8" and the segments are named "0.ts", "1.ts", etc. The playlist is constantly updated and old segments are deleted as the stream continues

Requirements

  • Running "docker compose up" must start the ingest server along with your app and database, accepting streams on localhost port 1935
  • Every registered user must have their own stream key. Stream keys must be randomly generated with at least 80 bits of entropy and treated as secret that are only shared with their owner
  • A logged in user must be able to see their own stream key. You have flexibility in how you show them, as long as it's obvious to your users where to find it (i.e. we should not have to look very hard for it during grading). The obvious choice is somewhere on their settings page. This will require some front end editing
  • Only valid stream keys may stream. Your on_publish endpoint must reject any stream key that does not belong to a user
  • You must be able to handle multiple concurrent live streams
  • Add a way to see all users who are currently live, and be able to watch each stream. You may have to manipulate the front end to accomplish this. It is acceptable to mix this in with existing VideoTube page by adding live streams to the video list with an appropriate title and description
  • When user stops their live stream, the stream must be removed from the front end
  • Viewers must only ever connect to your app on port 8080. The stream must get from the ingest server to viewers through your app (e.g. using a volume shared by both containers), not by exposing the ingest server to viewers
Security Concern - Stream keys: Anyone who has a user's stream key can stream as that user. A stream key must never be exposed to anyone other than its owner, including in any URL, playlist, or segment path requested by viewers. Note that the ingest server names each stream's HLS directory after its stream key, so hosting that directory as-is would leak the key.

Tips:

  • It's recommended that you test using OBS. To setup a stream, go to Settings, Stream, choose the "Custom" service, and enter the server and stream key
  • You can also test without streaming software by streaming a video file with ffmpeg: ffmpeg -re -i video.mp4 -c:v libx264 -preset veryfast -c:a aac -f flv rtmp://localhost:1935/live/{stream_key} although this is significantly less fun..
  • The video has to be captured, sent to the ingest server, transcoded into HLS, saved to disk, and communicated to your server (With the longest step by far being the transcoding) which will take some time. Don't panic if there's a decent amount of delay between starting your stream and seeing it in your app
  • The ingest server looks up the hostname in the on_publish URL when it starts. Use depends_on in your compose.yaml so your app's container is started first. Your app will start your TCP server quickly, but you may want to add a health check for additional assurance
  • The on_publish and on_publish_done endpoints are called by the ingest server, not by a browser, so they do not have SOP/CORS concerns. Attackers may hit these endpoints, but they don't have a way to steal a user's stream key [unless you are careless with them]
  • A live playlist changes every few seconds. Make sure your caching headers from HW1 don't cause browsers to reuse an old playlist (e.g. send "Cache-Control: no-store" for live playlists if this becomes an issue)
  • You will want to use Docker volumes to share the streams across containers. This works by mapping a local directory to a directory in the container. You can map a local directory to the /hls directory of the ingest server, then map that same local directory to your app's container. This will give you access to the files from your app, with the convenient side-effect of seeing all the files on your local machine which can be helpful while debugging. Since the directory for each stream is named by it's stream key, you will have to use a different path to host them in your app instead of simply using the file path

Submission

Submit all files for your server to Autolab in a .zip file

Complete the homework submission form after submitting your zip file to Autolab: Homework Submission Form

If you do not both submit your zip file to Autolab and complete the homework submission form before the due date, your submission will not be graded. Do not be one of the students who will get a 0 for not submitting the form after completing the assignment.

Your submission must include a ".env" file with placeholder values for all your sensitive values. This is required so we know your variable names. You can optionally include only a ".env.example" file with all your variables as is standard practice. In any case, we need a simple way to find the names of all your environment variables while grading.

To reduce hard drive usage, Autolab will limit the size of submissions. Do not include any [large] uploaded images, uploaded videos, or HLS files in your submission. You still must include everything needed to run your app, including the entire front end.

Be sure your submission includes all your required directories. You can add placeholder files in your directories to ensure they are included in the zip file, or create the directories in your code if they don't exists. Many students have lost credit by 1. Ignoring the strongly recommended testing procedure below and 2. manually creating their video directories for testing and deleting them before submission. This can cause a crash when your server is trying to save a video to a directory that does not exist.

It is strongly recommended that you download and test your submission after submitting. To do this, download your zip file into a new directory, unzip your zip file, enter the directory where the files were unzipped, run "docker compose up --build --force-recreate --renew-anon-volumes", then navigate to localhost:8080 in your browser and test all your features. This simulates exactly what the TAs will do during grading.

If you ignore this recommendation, don't be surprised when you do not earn credit even if your code functions properly on your laptop.

Grading

Each objective will be scored on a 0-3 scale as follows:

3 (Complete) Clearly correct
2 (Complete) Mostly correct, but with some minor issues
1 (Incomplete) Not all features outlined in this document are functional, but an attempt was made to complete the objective
0 (Incomplete) No attempt to complete the objective or violation of the assignment (Ex. Using an HTTP library, or a library that parses multipart requests) -or- The HW submission form was never submitted
0.X (Security Risk) If any security risk is found while testing, all objectives will be scored 0 unless a proper security essay is submitted. X is the score that will be earned if a proper security essay is submitted on time

Note that for your final grade there is no difference between a 2 and 3, or a 0 and a 1.

3 Objective Complete
2 Objective Complete
1 Objective Not Complete
0 Objective Not Complete

A Security Risk is any violation of a "Security Concern" that is explicitly labeled in this document, including those from HW2.


Security Essay


If a security risk is found in your submission, you will be assigned a 0 for all 5 objectives. However, you will still have an opportunity to earn credit for the objectives by submitting an essay about the security issue you exposed. These essays must:

  • Be at least 1000 words in length
  • Explain the security issue from your submission with specific details about your code
  • Describe how you fixed the issue in your submission with specific details about the code you changed
  • Explain why this security issue is a concern and the damage that could be done if you exposed this issue in production code with live users

If your submission contains multiple security risks, you may have to write multiple essays to recover your homework grade

Any submission that does not meet all these criteria will be rejected and your objective will remain incomplete.

Due Date: Security essays are due 1-week after grades are released. Submission instructions will be included in your feedback on Autolab.

Any essay may be subject to an interview with the course staff to verify that you understand the importance of the security issue that you exposed. If an interview is required, you will be contacted by the course staff for scheduling. Decisions of whether or not an interview is required will be made at the discretion of the course staff.