Vurbl

The problem

In 2019 a podcast episode reached a listener as one long file over HTTP. The player requested the file and started playing once enough of the beginning had arrived. Every player worked this way, so nobody treated it as a problem.

It carries two costs. Playback waits on the network before the first word, which is the moment a listener decides whether to stay. And the platform pays to ship audio nobody hears, because most listeners leave an episode long before the end.

Vurbl was ad-supported and existed to pay creators, so both costs landed in the same place. Slow starts lose the audience that advertising revenue depends on, and bandwidth spent on unheard audio comes out of the money meant for the people who made it.

My role

I joined in January 2020, first as a contractor and then as an employee, and stayed until July 2021. I architected the platform, led the team that built it, and wrote the ingestion backend myself.

The work divided along those lines. The content library and the API were a team codebase: I designed them, scaffolded them in the first days of the repository, and stayed close enough to drop back into the code when a piece needed me. The ingestion and acquisition pipelines were mine to write. The edge delivery layer was mine to specify and drive, and I worked on it with a Fastly professional-services engineer rather than writing the core of it alone.

How the delivery worked

Instead of caching whole episodes, we processed audio at the CDN edge. A Fastly layer cut each episode into fragments and served them in-stream, so the player began within a moment of the request and never held more than the next couple of seconds.

Two audio delivery paths compared. In progressive download, an origin file passes through a CDN cache to a player that fills its buffer from the start of the file and waits. In Vurbl's design, the origin is fetched once and a Fastly edge layer cuts the audio into fragments served in-stream, so the player starts at once holding only the next few seconds. — The same origin file. The difference is that the edge understands it is audio.

Here is that behaviour in a browser, captured in February 2021.

Chrome network panel listing repeated requests to media1.vurbl.com, each returning HTTP 200 at a little over 42 kB, with timings between 58 and 216 milliseconds and waterfall bars spaced across the session. The summary line reads 41 of 205 requests and 1.3 MB of 2.0 MB transferred. — Chrome’s network panel on an episode page, filtered to audio. Every fragment is the same size, a little over 42 kB, and they arrive spaced across the session rather than in one burst. The player was 2:08 into a 62:57 episode and had fetched 41 of them, 1.3 MB of the 2.0 MB the whole page had transferred. Request names and episode artwork are cropped out.

That change moved three things at once. Playback started at once rather than after a wait. Delivery cost fell, because we stopped shipping audio past the point where listeners stopped. And the ceiling on growth moved off our own infrastructure, leaving CDN publishing as the only thing that gated us.

The layer was written in VCL, Fastly’s dialect of the Varnish Configuration Language. That choice was forced by the calendar: Compute@Edge, the obvious modern answer, was in beta in late 2019 and did not reach general availability until 2021. VCL was what could ship.

I want to be accurate about how it was built, because it is the piece of this platform I would most like to be judged on. I was the product manager for it. I specified the behaviour, drove it, and owned the repository. A Fastly professional-services engineer adapted custom VCL that Fastly had already built for another enterprise customer of theirs, which is the kind of head start a good vendor relationship buys you and is worth saying out loud.

The platform behind it

Delivery was the interesting end. Most of the engineering sat behind it.

Ingestion. A Python service, a Flask application factory served by gunicorn, with a Celery queue over Postgres and Redis, containerised and deployed through Cloud Build onto App Engine. It handled acquisition, processing and audio fingerprinting through ACRCloud. This was the largest codebase in the company and I wrote nearly all of it.

ETL. A separate service that managed ingestion database lifecycles and loaded downloaded metafiles in batch. I wrote all of it. Its more interesting decision is recorded in its own history: it began with a schema per content source, and I moved it to a single JSON column once the number of sources made per-source schemas the wrong trade.

The API. A Django monolith covering the content library, the player API, crawling, reporting and stats. I designed it, wrote the first commits on the second day the repository existed, and handed it to the team that built it out. My own commit rate falls away through 2020 as they ramped and returns in bursts when something needed me, which is what leading rather than owning a codebase actually looks like.

Search. Postgres replicated into Elasticsearch, using a fork of pgsync, running on a cluster I provisioned with Terraform across development, staging and production: network, VPC connector, NAT, load balancer, instance groups, IAM, service accounts and firewall rules. Primarily Google Cloud, with some AWS.

Acquisition. An RSS scraper I wrote to pull podcast feeds at scale, alongside a web scraper the team built.

Evaluated, not shipped. I stood up Apache Beam on Dataflow through Spotify’s Klio framework to test whether a streaming media pipeline belonged in our stack: Pub/Sub for event input and output, containerised workers, explicit worker sizing and autoscaling. It stayed an evaluation. I am naming it because knowing when a promising framework does not earn its place is part of the job, not because we ran it in production.

There was also a mobile application on both platforms and an embeddable web player, both built by the team.

What worked, and why it ended

The architecture worked. Audio started instantly, delivery cost stayed low against a growing library, and the scaling limit sat with our CDN rather than with anything we ran.

The business thesis did not survive the market. About four months after I joined, Spotify moved into podcasts and took the premise the company had been built on. We kept building for another year. The platform was discontinued in 2022.

That sequence is worth separating out. The engineering answered the question it was given, and the question stopped being worth answering. Those are different outcomes and only one of them was ours to control.

What the project shows

A CDN can do more than cache. Teaching the edge what the payload is turned a delivery cost problem into a delivery feature.
The constraint that shaped the best decision here was a release calendar, not a whiteboard. VCL was not the elegant choice. It was the one that existed in 2020.
Scaffolding a codebase and then handing it over is a real contribution, and the commit history shows it more honestly than a summary would.
Moving from a schema per source to a single JSON column is the sort of decision worth recording at the time, because the reasoning is what a later reader needs.
A good vendor relationship is leverage. Reusing VCL that Fastly had built for another customer saved months we did not have.

A note on ownership

Vurbl, its platform and its brand belong to Vurbl Media, Inc. My 2020 consulting agreement assigned the work product to them. This page describes engineering I did and reproduces none of their code, copy or design. The diagram above is my own.