The thing nobody warns you about with Spark is that the compute is the cheap part.
You wanted to process some Parquet. What you got was a scheduler, a shuffle service, executors that die for reasons you’ll be reading about in a stack trace at half past eleven, serialisation costs you can’t see, a memory model with three different kinds of memory, and a web UI you will come to know intimately. The job runs. It’s just that “the job runs” now involves a distributed system you’re on call for.
Sometimes that’s the deal you have to take. But the threshold where you have to take it has been moving fast, and this year it moved a lot.
My favourite detail of the year
Polars 1.41 replaced their Parquet metadata parsing with a hand-written Thrift decoder.
I love this. Sit with what it implies for a second: the bottleneck wasn’t reading the data. It was reading the description of the data. The footer. The schema and the column statistics — the bit you read to work out which parts of the file you can skip.
Anyone who’s watched a Spark job burn minutes listing files and cracking open Parquet footers before doing a single byte of real work will recognise that shape instantly. It’s the “why has nothing happened yet” phase. Polars went and looked at where the time actually went, found it was in a generated Thrift parser, and wrote one by hand.
That’s not a feature and it’ll never show up on a comparison table. It’s just someone measuring where the time actually went and then doing the boring thing about it.
Things pandas never did for you
The other half of the Polars work this year is optimiser stuff, and this is where the pandas comparison gets interesting.
pandas has no query optimiser. It never did. Every operation happens right now, in the order you wrote it, whether or not that order made any sense. If you filtered after the join instead of before it, congratulations, you materialised the whole join. The optimisation pass was you, squinting at your own code, moving lines around.
Polars 1.42 taught its optimiser to throw out contradictory filter predicates — ask for rows where x < 10 and x > 20 and it can work out that it doesn’t need to read anything at all. 1.41 added nested common subplan elimination, so a subquery you referenced four times gets computed once. 1.43 sped up joins on hive-partitioned data.
None of that is exciting on its own. Together it means the thing everyone coming from pandas finds weird — why is it lazy, why doesn’t it just do it — starts paying rent. The first time the optimiser deletes half your pipeline because it worked out you never used those columns, lazy stops feeling like ceremony.
DuckDB deleted a stall
DuckDB 1.5.0 made checkpointing non-blocking and picked up about 17% on TPC-H throughput.
The throughput number is fine. The bit I care about is that checkpointing used to be a thing that stopped. A periodic window where the database is doing its own bookkeeping instead of answering you. Now reads and writes carry on through it.
Half of what gets marketed as performance work is really this: someone found a stop-the-world pause and got rid of it. Those pauses are what make a system feel twitchy under load, and twitchy systems are how you end up buying a cluster you didn’t need — not because you ran out of capacity, but because nobody could explain the p99.
The obvious objection
Polars is building the distributed thing too. Polars Cloud 0.9.0 landed in July with distributed expressions, an Iceberg sink, Kubernetes deployment.
So I’m not claiming clusters are a mistake. Credit to them, they published a straight single-node versus distributed comparison in mid-July rather than leaving it as marketing.
But the reflex is the problem, not the technology. Somebody says “this won’t fit on one box” early, usually before anyone has measured, and from that moment it’s your problem forever. Nobody ever revisits it. A cluster sized against a 2019 assumption doesn’t tell you it’s now processing data that would fit in RAM on a laptop. It just keeps billing you and paging you.
Go and find out what your actual single-machine ceiling is on this year’s engines. If it turns out you’re under it, that’s a whole lot of spark complexity you get to not have.