Decoding 4K in Rust
AquaTube uses no system player: video is decoded by hand, up to 4K HEVC in hardware, to pass through the cathode-ray screen. The pipeline, its traps, and the clock that sets the tempo.
Why decode it yourself
AquaTube does not open video in a system player or a separate surface. The image has to live inside the pod's cathode-ray screen, pass through the same CRT shader as everything else: curvature, scanlines, phosphor. A video dropped into its own window could not receive that treatment. So each frame had to be brought all the way to a texture I control.
And a single requirement decided the rest: play 4K HEVC in hardware, smoothly. That constraint, set early, ruled out engines that only decode in software and fixed the stack on Rust plus FFmpeg. I did not start from "I want to write Rust": I started from a video requirement, and it picked the tool.
The binding and its 7 DLLs
On the Rust side, I use `ffmpeg-the-third` (the fork maintained for FFmpeg 8, where `ffmpeg-next` fell behind), linked against a shared build of FFmpeg 8.1. Compiling is not free: it generates the bindings via bindgen, so it needs LLVM/libclang and two environment variables to find the headers and libs.
The trap is not at compile time, it is at distribution. The exe needs the FFmpeg DLLs next to it. I had shipped five (avcodec, avformat, avutil, swscale, swresample), the only ones I knowingly use. The app refused to start: `0xC0000135`, DLL not found. Two were missing, avfilter and avdevice, which the binding references at load time even though I never call them. Perfect code will not start if a load dependency is missing, and the error does not say which.
Software plateaus, hardware takes over
In software decoding, not all is equal. 1080p runs wide, around 175 to 189 frames per second on my machine. But 4K HEVC drops to 47 frames per second, under the 60 line. In software, smooth 4K is not sustainable.
To hold it, you need the GPU's hardware decoder, exposed by FFmpeg under the `cuvid` decoders. You pick the decoder by codec, with a software fallback if hardware is absent:
Then comes a subtlety that cost me a while: a hardware decoder only reveals its output format (NV12) after decoding the first frame. So you cannot prepare the NV12 to RGBA converter in advance. You have to create it lazily, once the first frame has arrived:
// cuvid only reveals its format (NV12) after the first frame.
// Create the NV12 -> RGBA converter at the 1st frame, not before.
if self.scaler.is_none() {
self.scaler = Some(Scaler::get(
frame.format(), frame.width(), frame.height(), // format known NOW
Pixel::RGBA, out_w, out_h, Flags::BILINEAR,
)?);
}Who sets the tempo
Decoding is not enough: you have to synchronize sound and image, or they drift. And synchronization is not a clever algorithm. It is the choice of a master clock, to which everything else bends.
Here, audio leads. The audio thread counts the frames actually played by the sound card; that is the reference time. Video only shows a frame once its presentation time is reached according to that clock:
There remains the case where the master disappears: a video with no audio track, or no output device. No audio clock, so video freezes. The workaround is a wall-clock fallback that takes over when audio is absent, so the image keeps advancing.
// Audio is the master clock: it counts the frames actually played.
// Video only emits a frame when its time has come by that clock.
fn next_due(&mut self, audio_played_secs: f64) -> Option<VFrame> {
if self.head_pts <= audio_played_secs { self.pop() } else { None }
}What I take away
Three things. First: a non-negotiable requirement, set early, is a useful blade. Hardware 4K settled a stack debate in one sentence that could otherwise have dragged for weeks.
Second: hardware decoding imposes its order. It only reveals its format in use, so you build the pipeline after it, not before. Trying to prepare everything ahead is fighting a component that only speaks once it is running.
Third: to synchronize is to decide who is master, not to write a formula. Once the master is named, everything aligns on it. And the day it can vanish, you give it a stand-in, or time stops with it.