HTTP - Static Files, MIME Types, and Encodings

Encodings

MIME types and bytes

1's and 0's

  • The Internet can only transfer bits
    • Copper: High/Low voltage
    • Fiber: Light/Dark
  • All data sent over the Internet must be binary
  • How do we know what these 1's and 0's represent in HTTP?
    • Encodings and MIME Types

Encoding Text

Text

  • Only 1's and 0's can travel through the Internet
    • How do we send text?

ASCII

  • Character encoding
    • Maps characters to numbers
    • Numbers are represented in bits
    • Bits are sent through the Internet
    • Numbers are mapped back to characters by the receiver
  • ASCII uses 7 bit encodings

ASCII Chart

ASCII characters 32-127 with their decimal and hex values. Characters 0-31 are control characters and are not shown.
Dec Hex Char Dec Hex Char Dec Hex Char Dec Hex Char Dec Hex Char Dec Hex Char
32 20 space 48 30 0 64 40 @ 80 50 P 96 60 ` 112 70 p
33 21 ! 49 31 1 65 41 A 81 51 Q 97 61 a 113 71 q
34 22 " 50 32 2 66 42 B 82 52 R 98 62 b 114 72 r
35 23 # 51 33 3 67 43 C 83 53 S 99 63 c 115 73 s
36 24 $ 52 34 4 68 44 D 84 54 T 100 64 d 116 74 t
37 25 % 53 35 5 69 45 E 85 55 U 101 65 e 117 75 u
38 26 & 54 36 6 70 46 F 86 56 V 102 66 f 118 76 v
39 27 ' 55 37 7 71 47 G 87 57 W 103 67 g 119 77 w
40 28 ( 56 38 8 72 48 H 88 58 X 104 68 h 120 78 x
41 29 ) 57 39 9 73 49 I 89 59 Y 105 69 i 121 79 y
42 2A * 58 3A : 74 4A J 90 5A Z 106 6A j 122 7A z
43 2B + 59 3B ; 75 4B K 91 5B [ 107 6B k 123 7B {
44 2C , 60 3C < 76 4C L 92 5C \ 108 6C l 124 7C |
45 2D - 61 3D = 77 4D M 93 5D ] 109 6D m 125 7D }
46 2E . 62 3E > 78 4E N 94 5E ^ 110 6E n 126 7E ~
47 2F / 63 3F ? 79 4F O 95 5F _ 111 6F o 127 7F DEL

ASCII

  • As a String:
    • "hello"
    • Programming language specific representation
  • In Hex:
    • 68 65 6c 6c 6f
    • Need to encode the String into a byte representation
  • In Binary:
    • 01101000 01100101 01101100 01101100 01101111
    • Send this over the Internet

HTTP Headers

HTTP Headers can only contain

ASCII characters

HTTP Headers

  • When reading HTTP headers [And request/status lines]
    • Assume it is text encoded using ASCII
  • The body of the request/response may be encoded differently
    • Read the headers to find the encoding for the body

Character Encodings

  • ASCII can only encode 128 different characters
    • Decent enough for english text
    • Unusable for languages with different alphabets
  • With the Internet, the world became much more connected
    • Too restrictive for each alphabet to have its own encoding
  • How do we encode more characters with a single standard?
    • We need more bits!
    • Enter UTF-8

UTF-8

  • The modern standard for encoding text
  • Uses up to 4 bytes to represent a character
  • If the first bit is a 0
    • One byte used. Remaining 7 bits are ASCII
    • All ASCII encoded Strings are valid UTF-8
UTF-8 byte layouts. Each x is one bit of the code point.
Bytes Code points Bits Byte 1 Byte 2 Byte 3 Byte 4
1 U+0000 to U+007F 7 0xxxxxxx
2 U+0080 to U+07FF 11 110xxxxx 10xxxxxx
3 U+0800 to U+FFFF 16 1110xxxx 10xxxxxx 10xxxxxx
4 U+10000 to U+10FFFF 21 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

UTF-8

  • If more bytes are needed:
    • Lead with 1's to indicate the number of bytes
    • Each continuation byte begins with 10
    • Prevents decoding errors
    • No character is a subsequence of another character
UTF-8 byte layouts. Each x is one bit of the code point.
Bytes Code points Bits Byte 1 Byte 2 Byte 3 Byte 4
1 U+0000 to U+007F 7 0xxxxxxx
2 U+0080 to U+07FF 11 110xxxxx 10xxxxxx
3 U+0800 to U+FFFF 16 1110xxxx 10xxxxxx 10xxxxxx
4 U+10000 to U+10FFFF 21 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

Sending Data

  • When sending Strings over the Internet
    • The Internet does not understand language-specific Strings
    • Always convert to bytes/bits before sending
    • Encode the String using UTF-8
  • When receiving text over the Internet
    • It must have been sent as bytes/bits
    • Must convert to a language-specific String
    • Decode the bytes using the proper encoding

Sending Data

  • Use the Python methods
    • .encode()
    • .decode()
  • You can specify the encoding as an argument of these methods
    • Default is UTF-8 so providing no arguments will do what you want for this course

Content Length

  • Content-Length header must be set when there is a body to a response/request
  • Value is the number of bytes contained in the body
    • Bytes referred to as octets in some documentation
  • If all your characters are ASCII
    • This is equal to the length of the String
  • Any non-ASCII UTF-8 character uses >1 byte
    • Cannot use the length of the String as your Content-Length!

Content Length

  • To compute the content length of a UTF-8 String
    • Convert to bytes first
    • Get the length of the byte array

MIME Types

MIME Types

  • When an HTTP response [or request] contains a body, the body is an array of bytes
    • There's no restriction on the encoding used for these bytes (Cannot always assume ASCII/UTF-8)
  • Set a Content-Type header to tell the browser what those bytes represent
    • Tells the browser how to read the body of your response
  • This is the MIME type of the data

MIME Types

  • MIME type
    • Multipurpose Internet Mail Extensions
    • Developed for email and adopted for HTTP
  • Two parts separate by a /
    • <type>/<subtype>
  • Common types
    • text - Data using a text encoding (eg. UTF-8)
    • image - Raw binary of an image file
    • video - Raw binary of a video

MIME Types

  • Common Type/Subtypes
    • text/plain
    • text/html
    • text/css
    • text/javascript
    • image/png
    • image/jpeg
    • video/mp4
    • application/json

MIME Types

  • Optional settings can be added to the Content-Type header
    • Separate options by a ;
    • Options are formatted as <name>=<value>
  • Content-Type: text/html; charset=utf-8
    • The content is HTML encoded using UTF-8
    • *You must use this to tell the browser that you are using utf-8
    • Don't forget to add this exactly as it appears. One character difference will break your page

MIME Type Sniffing

  • Modern browsers will "sniff" the proper MIME type of a response
    • If the MIME type is not correct, the browser will "figure it out"™ and guess what type makes the most sense
  • Browsers can sometimes be wrong
    • Surprises when your site doesn't work with certain versions of certain browsers
  • Best practice is to disable sniffing
  • Set this HTTP header to tell the browser you set the correct MIME type
    • X-Content-Type-Options: nosniff

MIME Type Sniffing

  • X-Content-Type-Options: nosniff
  • Be sure to set this header properly!
    • Open the browser console
    • Check the headers of your response
    • Make sure this header was parsed by the browser

MIME Type Sniffing

  • Security concern:
    • You have a site where users can upload images
    • All users can view these images
    • Instead of an image, a user uploads JavaScript that steals personal data
    • You set the MIME type to image/png
    • The browser notices something is wrong and sniffs out the MIME type of text/javascript and runs the script
    • You just got hacked!
  • Solution:
    • X-Content-Type-Options: nosniff

MIME Types

  • With the proper MIME types set through a Content-Type header
    • The browser will know how to parse and render the body of your HTTP response
  • When receiving an HTTP request that contains a body
    • The Content-Type will be set to let our server know the MIME type

Sending Images

Sending Images

  • Sometimes we want to send data that is not text
  • Use different formats depending on the data
  • To send an image
    • Read the bytes from the file
    • Send the bytes as-is
    • Content-Length is the size of the file
    • Set the Content-Type to image/<image_type>

Sending Images

  • When sending images
    • Since the data is already in bytes when the file is read, no need to encode/decode
  • Never try to read an image file as a string
  • Never try to decode the bytes of an image into a string
  • An image is not encoded using UTF-8
    • The bytes will not decode properly

Sending Images

  • Don't overthink sending images
  • Read the bytes of the file. That's the body of your response
    • You may have to specify that the file should be read as a byte array since text is often assumed in File IO examples
  • Set the Content-Length to the length of the byte array
  • Set the appropriate MIME type in Content-Type
    • Ex: to send a .png the MIME type is "image/png"

Further Reading