Hybrid search, New OpenAI Embedding Models, Multimodal RAG for Video Processing Highlights Hybrid search with custom reranking (included in LanceDB Python version 0.6.0 release) * Explore the potential of reranking to enhance retrieval quality and
Implementing Corrective RAG in the Easiest Way Even though text-generation models are good at generating content, they sometimes need to improve in returning facts. This happens because of the way they are
Hybrid search and custom reranking with LanceDB Search has historically been a complex problem in computer science. This is partly due to limitations in natural language or contextual understanding, and partly due
Hybrid search: RAG for real-life production-grade applications by Mahesh Deshwal What is Hybrid Search, and what’s the need for it? With the increasing usage of LLMs in RAG setting, there’s
Benchmarking New OpenAI Embedding Models with LanceDB A couple of weeks ago, OpenAI launched their new and most performant embedding models with higher multilingual performance and new parameters to control the overall
LanceDB Community News — January 2024 We’re kicking off 2024 with a new LanceDB community newsletter to showcase all the updates in the LanceDB ecosystem, news, blogs, and important links.
Substrait Powered Filter Pushdown in Lance by Weston Pace Filter pushdown is one of the more fundamental optimizations in any data engineering pipeline. The premise is simple: the earlier you filter
LanceDB + Polars A (near) perfect match A spiritual successor to pandas, Polars is a new blazing fast DataFrame library for Python written in Rust. At LanceDB, we
Efficient RAG with Compression and Filtering by Kaushal Choudhary Why Contextual Compressors and Filters? RAG (Retrieval Augmented Generation) is a technique that helps add additional data sources to our existing LLM
Langroid: Multi-Agent Programming framework for LLMs In this era of Large Language Models (LLMs), there is unprecedented demand to create intelligent applications powered by this transformative technology. What is the best
Using Prediction Guard and LanceDB to Prevent Medical Hallucinations This is a collective work of the following authors: Sharan Shirodkar (Prediction Guard) Daniel Whitenack (Prediction Guard) Bingyang (Icy) Wang (Emory University) Guangming (Dola) Qiu
Using column statistics to make Lance scans 30x faster by Will Jones In Lance v0.8.21, we introduced column statistics and statistics-based page pruning. This enhancement reduces the number of IO calls needed
