🔥New release of Lance v0.16.1🔥
Lance v0.16.1 brings new features to help manage versions and Lance file format editions, a new optimized ETL pattern for updating columns, and an experimental API for distributed ANN index training.
💡 Dive into the future of data management with LanceDB’s random access I/O" 💡
Random access I/O has traditionally been ignored by columnar formats. One of the reasons we started the Lance file format and have been investigating new encodings is because we wanted a format with better support for random access.
Community contributions
💡
A heartfelt thank you to our community contributors of lance and lancedb this month: @gagan-bhullar-tech @mattbasta @rithikJha @rahuljo @h0rv @tonyf @dsgibbons @dentiny @maxburke @jiachengdb
Good reads
- A new report on Improving retrievers with LanceDB hybrid search and Reranking. This report explores techniques to optimize the retriever's performance without requiring a complete dataset re-ingestion. The experiments on the SQuAD and Llama2-review datasets demonstrated significant improvement. An 11% increase in accuracy on SQuAD (from 81.22% to 92.35%) and a 16% increase in hit-rate on Llama2-review (from 58.63% to 75%)
- Wonder what a summer internship looks like at LanceDB? Check out our intern Raunak Shah’s blog My summer internship experience at LanceDB, where he shares his transformative experience interning at LanceDB this summer!
Event recap
An innovative database changing the whole AI field: the idea behind LanceDB
How LanceDB tackles CAP Theorem
Latest releases
- In Lance, you can tag versions of your datasets (as of Lance v0.16.0). Blog post.
- New data_storage_version parameter allows precise control of which version of Lance files to use in tables (as of Lance v0.16.0, LanceDB Python 0.12.0, LanceDB Node/Rust 0.9.0). Blog post.
- merge_insert can be used to update a subset of columns (as of Lance v0.16.0, LanceDB Python 0.12.0, LanceDB Node/Rust 0.9.0). Blog post.
