Zilliz представила Milvus 3.0 с поддержкой нативного доступа к данным в хранилищах

Zilliz launches Milvus vector lakebase Zilliz has announced its v3.0 update to Milvus, its open-source vector database adding lake-native data access and a more expressive retrieval engine to developers building production AI applications. Lake-native means the data's primary home is open formats on cloud object storage (a data lake) – not loaded into a separate database. Zilliz has been talking about its Vector Lakebase technology since June, public preview start time, when CEO and founder Charles Xie said it was “one data foundation where the same vectors can serve a production query, anchor a discovery session, and power a multi-petabyte training-data pipeline — without copies, migration, or a parallel stack.” Unified lake-native storage, Zilliz says, means one storage layer, on object storage, that serves both low-latency online search and large-scale analytics over the same files. James Luan, Zilliz Co-founder and CTO, said: “Milvus 3.0 brings production retrieval closer to where data already lives and gives developers a more expressive engine for modern AI applications. That same foundation powers Zilliz Vector Lakebase, where we extend it with the elasticity, operational simplicity, and enterprise capabilities required to manage AI data at scale.” Vector databases store vector embeddings, the mathematically calculated numeric representations of various dimensions of a word or image. Vector databases provide vector search facilities to find similar vectors. Items with similar values on the same dimensions have vectors that are closer to the stores ones than dissimilar items. Zilliz says Milvus is the most popular open-source vector database world-wide, with more than 10,000 organizations using it for RAG, search, recommendations, and AI agents in production. It has also accumulated more than 100 million Docker pulls. Zilliz says production AI systems commonly maintain one copy of data for real-time retrieval and another for offline processing. Moving data between the two introduces additional storage overhead, lengthy exports, synchronization pipelines, and operational complexity. Milvus 3.0 enables organizations to build and serve production-grade indexes over vector data that remains in object storage and open data formats. It also moves more query and retrieval operations into the engine, expanding Milvus beyond traditional nearest-neighbor search to support richer ranking, aggregation, sparse search, and multi-vector retrieval. V3.0 features; - External Collections define Milvus collections over data stored in Lance, Iceberg, Parquet, or Vortex. Milvus builds vector, full-text, JSON, and scalar indexes over the data and exposes them through the same APIs as native collections—without copying the source data into Milvus. Incremental synchronization keeps indexes current as the underlying dataset changes. - Loon, the new manifest-based storage engine in Milvus 3.0, is designed to substantially reduce the read amplification associated with low-latency point access on object storage. It uses Vortex, an open and Arrow-compatible columnar format, as its default storage format. - Snapshots create point-in-time, read-only views of live collections at minimal incremental storage cost. Offline jobs such as evaluation, deduplication, and backfill validation can run against a stable dataset while production writes continue. - A new Spark connector exposes Milvus as a Spark DataSource V2, allowing Spark, Databricks, and EMR pipelines to read from and write to Milvus as part of standard batch workflows. Milvus 3.0 also introduces vector-native batch operators for tasks including deduplication and clustering. Top-K vector search means finding the K most conceptually similar items to a query item, using mathematical distance rather than exact keyword matching. “K” is a number such as 5, 10 or 20. Top-K searches don’t scale well as they measure distances between vectors across an entire dataset. The more vectors and dimensions in the dataset the longer the search takes. Luan said: “Production applications rarely stop at returning the nearest vectors. They need to sort, group, build facets, and retrieve across multiple representations of the same entity. Milvus 3.0 moves more of that work into the engine while redesigning storage for efficient, low-latency access on object storage.” Milvus 3.0 expands search retrieval beyond traditional top-K vector search to include; - Server-side sorting and aggregation, together with faceted search, allow applications to combine semantic relevance with attributes such as freshness, rating, price, category, tenant, and content source. Operations that previously required over-fetching data and processing it in application code can now execute directly within the Milvus engine. - StructList introduces native multi-vector retrieval for documents, images, and other entities represented by more than one vector. A document can retain vectors for multiple chunks, an image can retain vectors for multiple visual patches, and a product can retain several multimodal representations under a single entity and metadata record. StructList also supports late-interaction retrieval models such as ColBERT and ColPali. - Milvus 3.0 strengthens sparse and hybrid retrieval with an optimized sparse index that is approximately 3 times smaller while achieving comparable recall in internal testing. - Enhancements also include SINDI for learned-sparse vectors, server-side MinHash generation, nullable vector fields, custom dictionaries for full-text search, and broader support for Faiss-compatible indexes. This release develops the shared technical foundation behind both open-source Milvus and Zilliz Cloud. Developers can deploy Milvus independently, while Zilliz Cloud builds on the Milvus core to deliver a fully-managed Vector Lakebase spanning real-time serving, interactive discovery, batch analytics, and AI data operations. Comment Zilliz has given competitors Pinecone, Qdrant and Weaviate something to think about. All four are looking to lower search times. Qdrant introduced faster indexing, 3-way clusters for instant failover, and audit logging for compliance. In April. Pinecone announced precompiled vector sets and a composable retriever in May to speed AI agent searches. They appear to be assuming that, by providing better vector-based search facilities, they can prosper while competing with multi-model databases such as Regatta, SingleStore, SurrealDB and others. The vector-only suppliers do partner upstream with analytic lakehouse suppliers such as Snowflake and Databricks, but partnering with other database suppliers might be difficult because of database core architecture and storage engine difficulties. It will be interesting to see if one or more of the vector-only database suppliers starts adding/developing support for other database model types, such as structured, graph, document and time-series data. Availability Milvus 3.0 is available under the Apache 2.0 license and remains a graduated LF AI & Data project. It can be deployed on Kubernetes or Docker, including in air-gapped environments, and supports S3-compatible object storage, Google Cloud Storage, and Azure Blob Storage. The release is available initially through the Python, Go, and Node.js SDKs, with Java support planned shortly after release. Developers can read the release notes, follow the quickstart, and access the source code on GitHub. They can also join the Milvus community or speak with project maintainers through Milvus Office Hours. For organizations that prefer a managed platform, Zilliz Cloud incorporates Milvus 3.0 as its core indexing and retrieval engine and extends it into a fully managed Vector Lakebase. Zilliz Cloud adds elastic compute, interactive discovery, batch analytics, shared indexing, enterprise security and governance, and automated operations across a single copy of AI data. Bootnote External Collection builds real vector, full-text, JSON, and scalar indexes over lake-resident data and Loon removes the read amplification that made object storage impractical for low-latency point lookups. External Collection does not copy data. It is read-only and zero-copy: Milvus maps fields from your existing Lance, Iceberg, Parquet, or Vortex data and builds indexes over it in place. The files stay in your lake under your governance; incremental sync picks up changes.