Hybrid search, New OpenAI Embedding Models, Multimodal RAG for Video Processing

Highlights

Hybrid search with custom reranking (included in LanceDB Python version 0.6.0 release)

Explore the potential of reranking to enhance retrieval quality and downstream generation in your LanceDB workflows with minimal code adjustments and latency additions, inviting community feedback on your implementations.

Benchmarking New OpenAI Embedding Models with LanceDB

A short intro to new OpenAI embedding models and how LanceDB’s Embedding API simplifies working with embedding functions.

User Insights

Ultralytics uses LanceDB as the query engine for Dataset Exploration API & Dashboard

With LanceDB, Ultralytics is able to run semantic search across 100s of thousands of images seamlessly without setup.

Multimodal RAG for processing videos using OpenAI GPT4V and LanceDB vectorstore

A Multimodal RAG architecture designed for video processing. It utilizes OpenAI GPT4V MultiModal LLM class that employs CLIP to generate multimodal embeddings using LanceDB VectorStore for efficient vector storage.

Good Reads

Create LLM apps using RAG
- Exploring the Limits of Large Language Models: Understanding Hallucinations and Leveraging Retrieval Augmentation Generation (RAG) for Targeted Responses
Curating Custom Datasets for efficient LLM training using Lance
- 1 Million tokenized Python files from the Codeparrot dataset in Lance format - A subset of codeparrot/github-code dataset consisting of 1 Million tokenized Python files in Lance file format for blazing fast and memory efficient I/O.

Latest releases

LanceDB

Lance Format v0.10.2

Richer schema evolution functionality
Initial support for Rust-JVM interop

Contributor Spotlight

Thank you, Beinan Wang (@beinan), for contributing the initial jni-based JVM integration for Lance format. This allows Lance to serve the Java and Scala communities better and provides the foundation for connectors to large-scale distributed query engines like Spark, Presto, and Trino. Thank you, Beinan!

Give us a star on GitHub

Join the LanceDB Discord

Follow and subscribe to our LanceDB YouTube Channel

Read our Blog