Substrait Powered Filter Pushdown in Lance by Weston Pace Filter pushdown is one of the more fundamental optimizations in any data engineering pipeline. The premise is simple: the earlier you filter
LanceDB + Polars A (near) perfect match A spiritual successor to pandas, Polars is a new blazing fast DataFrame library for Python written in Rust. At LanceDB, we
Efficient RAG with Compression and Filtering by Kaushal Choudhary Why Contextual Compressors and Filters? RAG (Retrieval Augmented Generation) is a technique that helps add additional data sources to our existing LLM
Langroid: Multi-Agent Programming framework for LLMs In this era of Large Language Models (LLMs), there is unprecedented demand to create intelligent applications powered by this transformative technology. What is the best
Using Prediction Guard and LanceDB to Prevent Medical Hallucinations This is a collective work of the following authors: Sharan Shirodkar (Prediction Guard) Daniel Whitenack (Prediction Guard) Bingyang (Icy) Wang (Emory University) Guangming (Dola) Qiu
Using column statistics to make Lance scans 30x faster by Will Jones In Lance v0.8.21, we introduced column statistics and statistics-based page pruning. This enhancement reduces the number of IO calls needed
Benchmarking LanceDB I came upon a blog post yesterday benchmarking LanceDB. The numbers looked very surprising to me, so I decided to do a quick investigation on
Inverted File Product Quantization (IVF_PQ): Accelerate vector search by creating indices Vector similarity search is finding similar vectors from a list of given vectors in a particular embedding space. It plays a vital role in various
Modified RAG: Parent Document & Bigger chunk Retriever by Mahesh Deshwal In case you’re interested in modifying and improving retrieval accuracy of RAG pipelines, you should check Re-ranking post. What’s it
Search within an Image with Segment Anything 🔎 by Kaushal Choudhary Introduction SAM (Segment Anything) model by FAIR, has set a benchmark in field of Computer Vision. It seamlessly segments objects image with
MemGPT: OS inspired LLMs that manage their own memory by Ayush Chaurasia In the landscape of artificial intelligence, large language models (LLMs) have undeniably reshaped the game. However, a notable challenge persists — their restricted
Hybrid Search: Combining BM25 and Semantic Search for Better Results with Langchain Have you ever thought about how search engines find exactly what you're looking for? They usually use a mix of looking for specific
