The Curious Engineer

All blogs / Databases

Lantern

lantern.dev/blog

The most powerful database for building AI applications.

DatabasesAI / ML

’23’24’25’26
Posts
26 from 13 Sep 2023 to 20 Nov 2024
Posting
now and then
Kind
company blog

2024 20

  1. LLM Completions in PostgresWith v0.5.0, we're releasing llm_completion and add_completion_job, which enable LLM calls inside Postgres.Di QiCofounderNovember 20, 2024 · 7 min read
  2. A code chatbot in 15 minutes with PostgresLearn how to build a powerful, scalable code chatbot using Lantern on Ubicloud in under 15 minutes. All open-source tooling, from embedding models to cloud providers.Di QiCofounderOctober 24, 2024 · 7 min read
  3. Bank Compliance Automation with Lantern and EcliptorIn this article, we show how to use Lantern, Ecliptor, and OpenAI models to automate compliance checks in the banking industry.Di QiCofounderOctober 21, 2024 · 9 min read
  4. CREATE INDEX EXTERNALLY: Offloading pgvector Indexing from PostgresWe introduce external indexing for pgvector in Postgres to reduce performance issues from indexing large datasets by offloading the indexing process to external machines.Varik MatevosyanSoftware EngineerOctober 1, 2024 · 6 min read
  5. Understanding pgvector's HNSW Index Storage in PostgresIn this article, we'll explore how pgvector works under the hood, focusing on how the HNSW index is stored in Postgres.Varik MatevosyanSoftware EngineerAugust 19, 2024 · 8 min read
  6. Analysis of Key PostgreSQL Configuration Differences Among Cloud ProvidersWe dicuss some differences in PostgreSQL configuration parameters among cloud providers, why they matter, and their implications for users.Narek GalstyanCofounderAugust 13, 2024 · 5 min read
  7. Hybrid Vector Search in PostgresHybrid vector search combines the strengths of sparse and dense vector searches to improve search quality. We evaluate its performance on several datasets from the BEIR framework using Postgres.Di QiCofounderAugust 5, 2024 · 8 min read
  8. Postgres vs. PineconeWe respond to Pinecone's recent blog post comparing Postgres and Pinecone. We show that Postgres can outperform Pinecone in the same benchmarks Pinecone covered in their article.Narek GalstyanCofounderJuly 18, 2024 · 11 min read
  9. When using Pinecone, do not rely on returned scores - recalculate distances locallyWhen benchmarking Pinecone against Postgres, we noticed some inconsistencies with returned results. We recommend recalculating and re-sorting returned results locally when using Pinecone for nearest neighbor vector qu
  10. Dynamically loaded extensions in Postgres in the browserWe build on top of pglite to support loading Postgres extensions directly in the browser. We share a demo using geospatial search and vector search from Lantern, and discuss technical challenges we encountered.Varik MatevosyanSoftware EngineerJ
  11. Understanding Postgres Table PermissionsThis blog post helps interpret Postgres's terse permission strings with examples and visuals.Narek GalstyanCofounderJuly 10, 2024 · 7 min read
  12. Use separate tables for asynchronous embedding generationIn this article, we'll explore a problem faced when extending traditional Postgres tables with an additional column for embeddings. We will describe a typical setting for asynchronous embedding generation and will show how table schema design
  13. Understanding and Estimating Write-Ahead Log (WAL) Size in PostgresThis blog post looks into the underlying plumbing of WAL records - when they are created, what information they contain, and how different database operations like indexing affect the kind of WAL records being created.Narek GalstyanC
  14. April 2024 - Engineering UpdatesEngineering updates for April 2024, including asynchronous tasks, weighted vector search, and more.Di QiCofounderMay 1, 2024 · 2 min read
  15. Vector databases explainedWe give an overview of vector databases, and major concepts around them, including vector embeddings, vector indexing, and vector search.Di QiCofounderMarch 29, 2024 · 12 min read
  16. Product Quantization in PostgresWe implemented product quantization in Lantern and benchmarked it using the LAION 100M 768-dimensional vector dataset.Di QiCofounderVarik MatevosyanSoftware EngineerNarek GalstyanCofounderMarch 5, 2024 · 8 min read
  17. Evaluating OpenAI's new embedding models with Lantern and Parea AIOpenAI's newest embedding models promise huge performance increases. Using Lantern's Postgres vector database, and Parea AI's testing platform, we'll measure the new models in a real-world test.Di QiCofounderFebruary 29, 2024 · 6 min
  18. Improving vector search over documents using HyDEThe HyDE technique uses LLMs to improve the quality of vector search over text. In this article, we explain how it works and walk through an example with Lantern as a vector database.Di QiCofounderDanyil BlyschakSoftware EngineerJanuary 22, 2024 · 7 m
  19. Picking the right embedding model for your vector databaseWhy choosing the right embedding model for vector search makes all the difference, and how to experiment with embedding models more effectively with LanternDi QiCofounderDanyil BlyschakSoftware EngineerJanuary 15, 2024 · 12 min read
  20. Migrating from Pinecone to LanternPinecone is a popular closed-source vector database. We built a Python library to support migrating data from Pinecone to Lantern. This article covers how we built it and how to use it.Di QiCofounderJanuary 6, 2024 · 6 min read

2023 6

  1. Estimating memory footprint of your HNSW indexThis interactive visualization will help you quickly reason about the resources necssary to host your embeddings and serve nearest neighbor queries over themNarek GalstyanCofounderDecember 22, 2023 · 10 min read
  2. Full Text Search + Vector Search with PostgresSearch is a common need for many applications. Postgres supports search out of the box with regex matching and full text search. This can be augmented with vector search.Di QiCofounderDecember 16, 2023 · 11 min read
  3. The Hierarchial Navigable Small Worlds (HNSW) AlgorithmAn explanation of the Hierarchial Navigable Small Worlds (HNSW) AlgorithmDi QiCofounderNovember 19, 2023 · 4 min read
  4. Embeddings and choosing the right modelAn overview of embeddings and what to consider when choosing an embedding modelDi QiCofounderNovember 13, 2023 · 6 min read
  5. 90x faster than pgvector — Lantern's HNSW Index Creation TimeIndex creation time is a critical database metric. Learn more about how Lantern enables 90x faster index creation times than pgvector, and how Lantern compares to Pinecone.Varik MatevosyanSoftware EngineerOctober 20, 2023 · 8 min read
  6. Launching Lantern — a PostgreSQL vector database for building AI applicationsToday we're launching Lantern, an open-source PostgreSQL vector database.Narek GalstyanCofounderSeptember 13, 2023 · 3 min read