This page intentionally left blank. ⬇️, ➡️, or spacebar 🛰 to start slidedeck. --- class: middle ### 📼 ### 📼 📼 📼 ### 📼 📼 📼 📼 📼 ### 📼 📼 📼 📼 Video ### 📼 📼 📼 📼 📼 ### 📼 📼 📼 ### 📼 --- # What is Video? - Streams - Container - Codecs - Colorspaces - Color bars - Compression - Chroma subsampling - Television display - Pixels - Bit depth / Color depth - Pixel width - Resolutions - Frame rates - Aspect ratios - Interlacing - Timecodes - 3D and beyond --- # Streams 🌊🌊🌊 Video is made up of streams, wrapped in a container, and some of those streams are encoded with a codec --- # Streams What is inside the video file can be thought of in terms of streams: - General/metadata - Video - Audio - Subtitles/text - Chapters The extent and availability of the above is contingent on the parameters of the container in use. --- # Containers See [Video - Containers](/presentations/video-containers.html) --- # Codecs See [Video - Codecs](/presentations/video-codecs.html) --- # Colorspaces How is color organized, translated, conveyed, and presented? - RGB - YUV - YCbCr - YCoCg - XYZ (CIE 1931 color space) - and more --- # RGB ❤️ 💚 💙 Red, Green, Blue, like most modern computer monitors. --- # YUV - Y = luma (brightness) - U 💙 and V ❤️ = chrominance (color) U and V are **red-difference** and **blue-difference** Sometimes people say YUV when they actually mean something else: --- # YCbCr / YPbPr - Y = luma (brightness) - Cb and Cr = chrominance (color) Cb and Cr are **blue-difference** and **red-difference** .center[] --- # YCoCg - Y = luma (brightness) - Co and Cg = chrominance (color) Co and Cg re **orange-difference** and **green-difference** .center[] --- # XYZ The CIE RGB color space is one of many RGB color spaces, distinguished by a particular set of monochromatic (single-wavelength) primary colors. [source](https://en.wikipedia.org/wiki/CIE_1931_color_space#Definition_of_the_CIE_XYZ_color_space) Visualized example: [here](https://www.youtube.com/watch?v=x0-qoXOCOow) <iframe width="560" height="315" src="https://www.youtube.com/embed/x0-qoXOCOow" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> --- # Color bars Let's look at the SMPTE color bars: .center[] What does this do? Yeah, it checks your colors and makes sure they look good! But how does it do that? --- # Color bars Top part: Different colors (white, primary, and secondary colors) set at 75% saturation (Why 75%? Because 100% would be too much and may cause clipping). Middle row: "Reverse bars" Bottom left: Subcarrier frequency at -/ phase, white, subcarrier frequency at +Q phase Bottom right: "PLUGE" element. .center[] ??? Most of this isn't really useful anymore, not useful digitally, although the bottom-left is especially not-useful. --- # Color bars Reverse bars? - If display is set to "blue-only", reverse bars should be the identical color as above bars. Saturation can be checked this way. Subcarrier frequency? - the wavelength frequency at which the video is set to stream PLUGE? - "Picture Line-Up Generating Equipment" - Reference black, one slightly more negative, and one slightly more positive --- # Compression - Uncompressed: Data stream with no attempts to decrease size - Compressed: Data is smushed in some way - Losslessly compressed: Data is smushed in some way, but can be un-smushed - Lossily Compressed: Data is smushed and this cannot be undone --- # Chroma subsampling A thing we do because human eyes can't really see color that well. Chroma subsampling is "the practice of encoding images by implementing less resolution for chroma information than for luma information." Some common ones for preservation (because we want the best): - 4:4:4 (not actually subsampling) - 4:2:2 (this is broadcast standard) - 4:2:0 (also common in production) Remember that in video, chroma is in both the Cb and Cr channels so each get subsampled. --- # Chroma subsampling, explained What do these numbers mean? x:y:z x = width (horizontal) of sample size y = number of chroma samples in the first row z = number of chroma samples in the second row --- # Chroma subsampling, explained ⬛️ = checked pixel ⬜️ = ignored pixel - 4:4:4 is full chroma and not subsampling at all because everything is being sampled. ⬛️ ⬛️ ⬛️ ⬛️ ⬛️ ⬛️ ⬛️ ⬛️ - 4:2:2 is checking every other pixel horizontally and vertically ⬛️ ⬜️ ⬛️ ⬜️ ⬛️ ⬜️ ⬛️ ⬜️ - 4:2:0 is checking every other pixel horizontally but not vertically ⬛️ ⬜️ ⬛️ ⬜️ ⬜️ ⬜️ ⬜️ ⬜️ ??? This is, of course, a simplification. Some algorithms (JPG?) don't just sample the first pixel, but sample both together and take the average results. So it's smarter than as basically outlined above. --- # Television display systems Good news, there are only three systems: - NTSC - PAL - SECAM Bad news, they are completely incompatible with each other. --- # NTSC *National Television Standards Committee* Developed in the USA in 1954. - Lines: 525 - Frame rate: 29.97 Hz - Picture resolutions: 720x480; 704x480; 352x480; 352x240 --- # PAL *Phase Alternating Line* Developed in 1967 by the United Kingdom & Germany. - Lines: 625 - Frame rate: 25 Hz - Picture resolution: 720x576; 704x576; 352x576; 352x288 --- # SECAM *Séquentiel couleur à mémoire* ("Sequential colour with memory") Developed in France in 1967. SECAM uses the same bandwidth and resolution (720x576) as PAL but transmits color differently. --- # Display resolutions  <small>By Jedi787plus - https://en.wikipedia.org/wiki/File:Vector_Video_Standards4.svg, GFDL, https://commons.wikimedia.org/w/index.php?curid=37694717</small> --- # Display resolutions But of course it's even more complicated than that. --- class: middle, center # Pixels --- # Bit depth / Color depth Pixel bit depth refers to color depth for each pixel, the amount of color information stored (as bits) in each and every pixel on screen. For perspective, 1-bit will have just two colors. Each pixel will have to answer a yes/no question for "Am I like this color?" and there is only one color to ask about. 2-bit allows for four colors. 8 bit = 256 colors 10 bit = 1024 colors 16 bit = 4,096 colors 32 bit = 4,294,967,296 colors --- # Bit depth / Color depth 256 colors isn't very much. But video comes in as three components -- YCbCr or RGB, so it is running the bit depth over one pixel three times. So when it's per channel, it means 8bit is 256x256x256 10bit is 1024x1024x1024. Both of these numbers equate to way more colors than humans are capable of perceiving. --- # Pixel Aspect Ratio Wait -- pixels can be different SIZES?  --- # Ahem... Pixel Aspect Ratio That's right, just when you think you were started to understand how video works... - PAR The horizontal to vertical ratio of each, rectangular, physical pixel - SAR The horizontal to vertical ratio of solely the number of pixels in each direction - DAR The combination (which occurs by multiplication) of both the pixel aspect ratio and storage aspect ratio giving the aspect ratio as experienced by the viewer. --- # Frame rates How fast do things go? - Pre-video rates - 24 - 25 - 29.97 - 48 - and more --- # Aspect ratios - 4:3 (1.33:1) SD - 16:9 (1.77:1) HD - 21:9 (2.35:1) Movies - 19:10 (1.9:1) IMAX - and more Practice: [Beyonce game](https://ablwr.github.io/beyonce_quiz/) --- # Interlacing Interlacing is for optimizing perceived motion in lossy video. .center[] --- # Interlacing "Interlacing issues can be seen during movement, where squiggly lines appear in places of motion. The concept of interlacing involves each frame containing 50% of the line information required for a full picture, and having even and odd frames play back half of the information quickly enough would result in a full-looking image. This was done because it was faster to send video signals, like in the case of television, when this kind of technique is used. Now it just lingers around making everything look like garbage." -- [Format-ion: Video playback errors in Beyoncé’s latest music video](https://ablwr.github.io/blog/2016/02/08/format-ion-video-playback-errors-in-beyonces-latest-music-video/) --- # TFF, BFF When interlacing, every other line is skipped. Which gets skipped depends on if the Scan Order is set to TFF or BFF. TFF = Top Field First (TFF) BFF = Bottom Field First (BFF) --- # De-interlacing MBAFF Macroblock-Level Adaptive Frame/Field PAFF Picture Adaptive Frame Field * PAFF allows to decide, on a frame basis, whether to encode each frame as a frame or as two fields. When encoded as two fields, it's as if you would have done a separatefields() on the frame. * MBAFF allows to decide, on a macroblock basis, whether to encode as field or frame. Actually, it works on vertical pairs of macroblocks, so on 16x32 areas. You can easily put two frame macroblocks ( on under the other ), or two field ones ( one with the top lines, the other with the bottom lines ). [source](http://forum.doom9.org/showthread.php?p=927647) --- # Timecodes Timecodes assign a number to each frame. BCD: Binary Coded Decimal (HH:MM:SS:FF) BITC: Burnt-In Time Code LTC: Linear Timecode VITC: Vertical Interval Time Code --- # 3D and beyond? - 3D - 360 video - Virtual reality - all of the above??? --- # Additional Resources - [A/V Artifact Atlas](https://bavc.github.io/avaa/) - [Chroma Sampling: An Investigation](http://www.nattress.com/Chroma_Investigation/chromasampling.htm) - [Charles Poynton / Chroma subsampling notation](http://poynton.ca/PDFs/Chroma_subsampling_notation.pdf) - [Computerphile - Compression](https://www.youtube.com/watch?v=Lto-ajuqW3w) - [Dave Rodriguez: Introduction to AV Codecs & Containers](https://figshare.com/articles/Introduction_to_AV_Codecs_and_Containers/6013958) - [Digital Video Introduction](https://github.com/leandromoreira/digital_video_introduction/blob/master/README.md) - [Ethan Gates: Color is Math](https://patchbay.tech/2018/07/16/color-is-math/) - [Lurker's Guide to Video Systems](https://lurkertech.com/lg/video-systems/) - [PAR, SAR, and DAR: Making Sense of Standard Definition (SD) video pixels](https://www.bavc.org/blog/par-sar-and-dar-making-sense-standard-definition-sd-video-pixels) - [Sustaining Consistent Video Presentation](http://www.tate.org.uk/about-us/projects/pericles/sustaining-consistent-video-presentation) - [411 on 422](https://www.youtube.com/watch?feature=player_embedded&v=7JYZDnenaGc) - [What is bit rate?](https://www.youtube.com/watch?v=QOn-9anLFxA) - [Video transcoding](https://github.com/donmelton/video_transcoding) --- # Additional Resources, analog - [Dead Media Research Lab](http://www.conceptlab.com/deadmedia/) - [EIA Video Art Resource Glossary](http://www.eai.org/resourceguide/glossary.html) - [LabGuy's World: The History of Video Tape Recorders before Betamax and VHS](http://www.labguysworld.com/) - [Vector Hack Festival](https://vectorhackfestival.com/) --- # Learning more - [FFmpeg](/presentations/ffmpeg.html) - [MediaInfo](/presentations/mediainfo.html) - [QCTools](/presentations/qctools.html) [Home](/)