LanceDB Enterprise Architecture, Lance Community Office Hour, Petabyte Scale Multimodal AI

🔥LanceDB Enterprise Documentation

LanceDB Enterprise transforms your data lake into a high performance vector database that can operate at extreme scale. Serve millions of tables and tens of billions of rows in a single index, improve retrieval quality using hybrid search with blazing fast metadata filters, and up to 200x cheaper with object storage.

👩‍💻Lance Community Office Hour

We kicked off our first monthly community office hour in Oct, in which the LanceDb team talked about what we're currently working on, discussed rough plans and future roadmap, and answered questions from anyone in the Lance and LanceDB community. The next one is coming up soon, come say hi! 

Community contributions

Good reads

Another deep dive into the Lance file format and explain how Lance uses backpressure to balance parallelism, speedy I/O, streaming computation, and limited RAM usage.

Columnar File Readers in Depth: Backpressure

Streaming data applications can be tricky. When you can read data faster than you can process the data then bad things tend to happen. The most common scenario is you run out of memory and your process crashes. When the process doesn’t crash, it often breaks performance (e.g. swapping

LanceDB BlogWeston Pace

Event recap

Bring Vector Search And Storage To The Data Lake With Lance

Summary The rapid growth of generative AI applications has prompted a surge of investment in vector databases. While there are numerous engines available…

Data Engineering Podcast

Upcoming events

Composable AI Infra for Agents · Luma

Join us for an evening of talks about the composable AI infrastructure for agents, hosted by dltHub & LanceDB at the Continue office. Learn about the different…

dltHub

Compound AI Systems: November/SF · Luma

Welcome to the third event in the Compound AI Systems meetup series for Fall/Winter 24/25! Join experts in data and AI for an in-person deep dive into AI infra…

Ester Shmulyian

Latest releases

Lance releases:

• v0.18.1 (2024-10-03)
  • SQL now supports list element access.
  • Filter pushdown enabled for scans of Lance V2 files.
  • Zstd block compression supported, if opted in for particular columns.
• v0.18.2 (2024-10-03)
• v0.19.1 (2024-10-21)
  • Full-text search tokenizer is now customizable. It allows setting case-sensitivity, stemming, stop word removal, and ascii-folding.
  • Adds a new BlobFile API that can read and seek over large binary values. Useful, for example, to extract individual frames from video columns.
  • Can now add new columns based on existing data frame, without requiring an expensive join.

LanceDB releases:

• v0.11.0 (2024-10-09)
  • Python v0.14.0
  • From Lance:
    • SQL now supports list element access.
    • Filter pushdown enabled for scans of Lance V2 files.
    • Zstd block compression supported, if opted in for particular columns.
• v0.12.0 (2024-10-29)
  • Python v0.15.0
  • LanceDB’s native tokenizer is now configurable in Python sync API, allowing you to choose whether to enable stemming and stop word removal, as well as other settings. The default tokenizer no longer performs stemming or stop word removal. This will come soon to Nodejs and async Python.