Weston Pace

Columnar File Readers in Depth: Backpressure

Streaming data applications can be tricky. When you can read data faster than you can process the data then bad things tend to happen. The

Engineering 6 min read

The case for random access I/O

One of the reasons we started the Lance file format and have been investigating new encodings is because we wanted a format with better support

Engineering 10 min read

Columnar File Readers in Depth: APIs and Fusion

The API used to read files has evolved over time, from simple "full table" reads to batch reads and eventually to iterative "

Engineering 11 min read

Lance v2 is now in Beta

We've been talking for a while about a new iteration of our file format. We're pleased to announce that the new

Engineering 5 min read

Columnar File Readers in Depth: Scheduling vs Decoding

We've been working on readers / writers for our recently announced Lance v2 file format and are posting in-depth articles about writing a high

Engineering 9 min read

Columnar File Readers in Depth: Parallelism without Row Groups

Recently, I shared our plans for a new file format, Lance v2. As I'm creating a file reader for this new format I

Engineering 7 min read

Lance v2: A columnar container format for modern data

Why a new format? Lance was invented because readers and writers for existing column formats did not handle AI/ML workloads efficiently. Lance v1 solved

Engineering 12 min read