While working on a project to allow video uploads from family members to my home server, I realized that my home videos were obnoxiously large. I took a three-minute video of my son acting crazy and it weighted in at 250MB. 15 years ago, I remember watching reasonable-looking, 2-hour long, movies that only took up 700MB. BluRay full-resolution (1080p) backups often look pretty good encoded to sizes between 1 and 2 gigabytes -- something is very wrong here!
I decided to perform a little research and conduct some tests to try and get my file sizes under control. I'd used Handbrake and ffmpeg in the past successfully, so decided to stick with those for this experiment.
The Short Version
Using the High Profile setting on Handbrake, setting Constant Quality (this is NOT the same as Constant Bit-Rate [CBR] and is actually a type of Variable Bit-Rate) to 22 and other default settings, I was able to encode a sample set of videos from multiple different recording devices ranging from low-light, flash-lit smartphone videos to sunlit family moments at a zoo taken on consumer recording devices to a much smaller file size! After deciding on h.264/AVC and trying a few constant quality settings, I settled on cq22 which reduced the size of the collection of files from 496MB to 132MB with no resolution change (resizing) and minimal visual quality loss (obviously, this is subjective). Success!
The Long Version
I can't help myself -- I have to research things to make sure I'm doing them the best possible way...especially if lossy encoding/compression is in the picture. As fast as technology changes, this must be done every few years and the best practice often heavily depends on the intended use.
Before you start...
Storage is cheap. If you have the room, you may want to just keep the original files (to negate the risk of losing data). If you have streaming needs such as myself, a second version for streaming may be a good idea to save your server and users some bandwidth at minimal quality cost. Also, for full disclosure, employing lossy compression (such as the method described here) does cost you "data." The thing to consider is whether the data lost is consequential, can be perceived, and could be valuable in the future. Additionally, is potential storage space gained is worth your time.
This can take a while! Especially if you want to perform some tests yourself to ensure my recommendations work for your video or that your eyes aren't more or less discerning than mine. Hopefully, you can figure out how to batch this process and put your computer to work at night or during long periods of, otherwise, non use. Using the Normal Handbrake profile will probably speed up the process notably, though I'm not sure what the resulting file size or quality trade-off will be.
My findings
I've compared the various containers and codecs in the past and, though I like free and open (vp8/ogg in a Matroska container) codecs and containers, I decided to go with tried and true for this test as I was interested in streaming home videos. My understanding is that using h.264/AVC and aac codecs in a mp4 (m4v) container is the most trouble-free way to prepare files for streaming using modern methods (html5) without the need for flash or multiple file encodes.
At first, I thought 2-pass encoding was supposed to be the best option based on some research I'd done in the past. That turned out to be a misunderstanding on my part. 2-pass is helpful when a specified file size is targeted (when your encoded movie has to fit on a CD/DVD, etc.), but not especially useful otherwise. For this project, Constant Quality (CQ - a type of Variable Bit-Rate[VBR]) encoding was employed as I was only interested in maintaining original subjective quality at reduced file sizes. Handbrake's High Profile and a Constant Quality of 22 produced great results for my batch of videos.
While I can discern (very) minor differences in some of the screen shot comparisons, they aren't noticeable in the video for me (mostly slight artifacting and pixelation, color variation, edge smoothing, etc.). If I planned on trying to use stills for pictures or something, I may use higher quality or, again, start with uncompressed video. Using 720 or 1080 stills as anything higher than a 3"x5" printed photo would likely look bad anyway.
How does it work?
Well, here is some speculation on my part...but since all my source files contained various types of AVC encoding performed by lower powered devices*, albeit with dedicated encoding hardware, they just don't have the power to encode optimally "on the fly." Either that or they use more of a CBR type of encoding...either way, there is room for improvement, even if it means re-encoding an already lossy encoded file to reduce storage space. The only alternative would be to try and configure any recording device to record uncompressed video and then encode it properly later, on a computer. This would avoid double lossy compression, but uncompressed video is remarkably large and would require tons of available space and very fast storage media. Likely, many consumer grade recording devices won't offer those options even if they were viable.
I can't stop myself...
...from encouraging people to back up their data. Presumably, you already back up your data. If not, you should start. $50/year will let you back up all your files with Carbonite...and it is really easy to use. There are likely other paid solutions that are easy to use and cost a similar amount per year.
I use a mix of various cloud storage solutions, network storage, and a usb-hdd. Google backs up all pictures I take (at lower resolution) and all videos (up to 1080) from all my devices and even makes fun little animations out of them that I enjoy. They also back up my music, I think. Most of my documents, code projects, etc, end up on my web server or cloud storage. I've written on backups before, so won't rehash...just make sure your data exists in no less than two locations to reduce the chance of data loss.
Recording Devices
- Canon(?) DV Camcorder (Sent to computer using DV Capture)
- Kodak Zi-8
- Motorola Droid (OG)
- Motorola Droid 3
- Motorola Droid X2
- Moto G (1st Gen)
- Moto G (4G LTE)
- Moto G (2nd Gen)
- Nikon D3100
Screenshots for Comparison
I pulled screenshots with ffmpeg and the frames are off by 1 in half the comparisons...I know that makes it unfair. Sorry. I assure you this was accidental and not maliciously as to mislead.
This is the original still. Hover/tap to show encoded still. (sorry, mobile users, this doesn't toggle for you on second tap)
This is the original still. Hover/tap to show encoded still. (sorry, mobile users, this doesn't toggle for you on second tap)
This is the original still. Hover/tap to show encoded still. (sorry, mobile users, this doesn't toggle for you on second tap)
This is the original still. Hover/tap to show encoded still. (sorry, mobile users, this doesn't toggle for you on second tap)
This is the original still. Hover/tap to show encoded still. (sorry, mobile users, this doesn't toggle for you on second tap)
This is the original still. Hover/tap to show encoded still. (sorry, mobile users, this doesn't toggle for you on second tap)
This is the original still. Hover/tap to show encoded still. (sorry, mobile users, this doesn't toggle for you on second tap)
My take is that there are (mostly, minor color) changes such as shirt color, bright spots, etc., but they are more akin to contrast adjustments, white balance settings, or saturation levels and have no meaningful impact on the viewing experience (especially video) as far as I'm concerned.
Land the Plane!
My wife'd not be happy if I posted the source videos here, but with the stills for comparison, and the very little variance there, you can imagine that the videos look even less changed after the encoding. I'll be re-visiting all my old home videos one folder at a time (starting each batch before bed), and spot checking them the next day for quality control. Hopefully this post will help if you are interested in compressing those home videos to more reasonable sizes!
