Database Manual / Sharding

Resource Allocation Considerations for mongot Deploymentsmongot部署的资源分配考虑因素

MongoDB Search and Vector Search with MongoDB Community is in Preview. The feature and the corresponding documentation might change at any time during the Preview period. MongoDB社区的MongoDB搜索和矢量搜索正在预览中。在预览期间,该功能和相应的文档可能会随时更改。To learn more, see Preview Features.要了解更多信息,请参阅预览功能

This section answers critical planning questions about your mongot deployment and provides actionable steps to design your search architecture. For a comprehensive overview, read this section linearly. 本节回答了有关mongot部署的关键规划问题,并提供了设计搜索架构的可操作步骤。如需全面概述,请线性阅读本节。Alternatively, you can use this section as a reference, skipping to relevant sections as needed. The goal of this section is to help accurately estimate resource requirements and build a robust deployment.或者,您可以将此部分用作参考,根据需要跳过相关部分。本节的目标是帮助准确估算资源需求并构建稳健的部署。

Data Characteristics and Indexes数据特征和指标

Your data's structure and index definition are the primary drivers of disk usage and RAM requirements. Use this section to estimate the final index size and the memory needed to manage indexes efficiently.数据的结构和索引定义是磁盘使用和RAM要求的主要驱动因素。使用此部分估算有效管理索引所需的最终索引大小和内存。

1

Calculate a baseline index size计算基线索引大小

Use the following formula as a starting point for your disk space estimation:使用以下公式作为磁盘空间估算的起点:

Estimated Index Size = (Number of Documents) x (Avg. Size of Indexed Text per Doc) x (Expansion Factor)

The Expansion Factor is critical and depends heavily on your tokenization strategy. Use these multipliers as a guide:扩展因子至关重要,在很大程度上取决于代币化策略。使用这些乘数作为指导:

Use Case用例Multiplier乘数
Standard/Language Tokenization标准/语言标记化1.5x - 3x
Keyword/Simple Tokenization键/简单标记化1.1x - 1.5x
edgeGram (Autocomplete)edgeGram(自动补全)5x - 8x
nGram (Partial Match)(部分比赛)8x - 15x+ (This can be extremely large, especially with a low minGram value.)

Note

The preceding expansion factor multipliers represent a generalized example and may not apply to all text based content. Make sure to test with your own data to determine the index calculation for your deployment.前面的扩展因子乘数代表了一个通用的示例,可能不适用于所有基于文本的内容。确保使用自己的数据进行测试,以确定部署的索引计算。

2

Add vector index size添加向量索引大小

For vector search, calculate the index size separately and add it to the total. The raw vector data size is a reasonable baseline for this value.对于向量搜索,分别计算索引大小并将其添加到总数中。原始矢量数据大小是该值的合理基线。

Vector Index Size = (Total Number of Vectors) x (Vector Dimensions) x 4 bytes / dimension

Note

This formula is for float32 vectors. The final HNSW index on disk will have additional overhead.此公式适用于float32向量。磁盘上的最终HNSW索引将产生额外的开销。

3

Manage memory consumption to prevent instability管理内存消耗以防止不稳定

Use Static Mappings使用静态映射
If your document structure is unpredictable, disable dynamic mappings. Dynamic mapping can lead to a "mapping explosion" where thousands of unintended fields are created, consuming excessive RAM and causing instability. 如果文档结构不可预测,请禁用动态映射。动态映射可能导致“映射爆炸”,其中创建了数千个意外字段,消耗了过多的RAM并导致不稳定。Explicitly define your index with dynamic: false. 使用dynamic:false显式定义索引。To learn more, see Dynamic and Static Mappings.要了解更多信息,请参阅动态和静态映射
Limit Stored Source Fields限制存储的源字段
Only store the minimum set of fields required for your search results within the mongot index itself. Fetching fields from the primary MongoDB collection is often more efficient and reduces the disk space used by the index. 仅将搜索结果所需的最小字段集存储在mongot索引本身中。从主MongoDB集合中获取字段通常更有效,并减少了索引使用的磁盘空间。To learn more, see Define Stored Source Fields in Your MongoDB Search Index.要了解更多信息,请参阅在MongoDB搜索索引中定义存储的源字段
Account for High-Memory Features考虑高内存特性
Be aware that synonym collections and deeply nested embeddedDocuments create significant memory artifacts. 请注意,同义词集合和深度嵌套的嵌入式文档会产生大量的内存工件。If you use these features heavily, you must allocate more Java heap space to the mongot process.如果你大量使用这些特性,你必须为mongot进程分配更多的Java堆空间。

Indexing Workload索引工作量

The rate and type of your writes (inserts, updates, deletes) determine the CPU and Disk IOPS needed to keep your search index synchronized with your database.写入(插入、更新、删除)的速率和类型决定了保持搜索索引与数据库同步所需的CPU和磁盘IOPS。

To provision for write throughput and maintenance:要提供写入吞吐量和维护:

1

Match resources to your write rate将资源与写作速度相匹配

High Write Rate (>1,000 operation per second)高写入速率(>每秒1000次操作)
This workload is CPU-bound and I/O-bound. Provision servers with a high CPU count and fast storage. Monitor the mongot process's CPU utilization and disk I/O queue length. If these metrics are consistently high and replication lag is growing, you need to scale up your hardware.此工作负载受CPU和I/O限制。为服务器配置高CPU数和快速存储。监控mongot进程的CPU利用率和磁盘I/O队列长度。如果这些指标一直很高,并且复制延迟正在增长,则需要扩展硬件。
Low Write Rate (<100 operations per second)低写入速率(每秒<100次操作)
Standard hardware configurations are usually sufficient.标准硬件配置通常就足够了。
2

Minimize replication lag最大限度地减少复制延迟

Consolidate Indexes合并索引
Avoid defining multiple, separate search indexes on a single collection. Each index adds overhead. Instead, create a single, comprehensive index definition.避免在单个集合上定义多个单独的搜索索引。每个指数都会增加开销。相反,创建一个单一的、全面的索引定义。
Materialize Complex Views实现复杂视图
If you are indexing a complex view, the change stream performance can be a constraint. Consider creating a materialized view to provide a simple, pre-aggregated data source for mongot to index.如果要为复杂视图建立索引,则更改流性能可能是一个约束。考虑创建一个物化视图,为mongot提供一个简单的预聚合数据源进行索引。
3

Plan for index rebuilds指标重建计划

Changing an index definition triggers a full, resource-intensive rebuild.更改索引定义会触发完整的、资源密集型的重建。

A rebuild creates a new index in addition to the old one before switching query requests over to the new index. Always ensure you have at least 125% of your current index size available as free disk space to accommodate this process without running out of storage.在将查询请求切换到新索引之前,重建会在旧索引之外创建一个新索引。始终确保您有至少125%的当前索引大小作为可用磁盘空间,以容纳此过程,而不会耗尽存储空间。

4

Scale horizontally with sharding使用分片进行水平缩放

For extremely demanding write workloads, scaling a single mongot node vertically might not be sufficient.对于要求极高的写入工作负载,垂直扩展单个mongot节点可能还不够。

If you anticipate sustained write loads exceeding 10,000 operations per second, sharding is the most effective scaling strategy. 如果您预计持续写入负载超过每秒10000次操作,分片是最有效的扩展策略。You need a minimum of 1 mongot per shard. mongot only indexes collections within the shard it is connected to, which helps to distribute the indexing load horizontally.每个分片至少需要1 mongotmongot仅对其连接的分片内的集合进行索引,这有助于水平分配索引负载。

Query Workload工作负载

Query performance is primarily a function of CPU for processing and RAM for caching. The complexity and volume of your queries determine the resources needed to meet your latency targets.查询性能主要取决于CPU的处理能力和RAM的缓存能力。查询的复杂性和数量决定了满足延迟目标所需的资源。

To estimate required deployment sizes for query performance:要估计查询性能所需的部署大小,请执行以下操作:

1

Estimate required CPU cores估计所需的CPU内核

To establish a CPU baseline, use your Queries per Second (QPS) target. A general starting point is 1 CPU core for every 10 QPS.要建立CPU基线,请使用每秒查询数(QPS)目标。一般的起点是每10个QPS有1个CPU核。

This baseline assumes simple queries. For workloads heavy with complex aggregations, fuzzy matching, or regex queries, you may only achieve 2-5 QPS per core. Conversely, simple term matching might yield 20+ QPS per core. Always test performance with a realistic query mix.此基线假定了简单的查询。对于具有复杂聚合、模糊匹配或正则表达式查询的繁重工作负载,每个核心只能实现2-5QPS。相反,简单的术语匹配可能会产生每个核心20+QPS。始终使用真实的查询组合测试性能。

2

Allocate RAM for low latency分配RAM以实现低延迟

RAM is the most important factor for fast queries.RAM是快速查询的最重要因素。

Full-Text Search全文检索
The goal is to fit as much of the index as possible into the operating system's file system cache. For optimal performance, provision enough RAM on the mongot node to match the total size of the index on disk. This minimizes slow disk reads.目标是将尽可能多的索引放入操作系统的文件系统缓存中。为了获得最佳性能,请在mongot节点上提供足够的RAM,以匹配磁盘上索引的总大小。这最大限度地减少了慢速磁盘读取。
Vector Search向量搜索
For low-latency vector search, the HNSW graph index must exist in memory. To calculate the minimum required RAM, use the steps in the index size estimation procedure and add a buffer of 20-25% for overhead. 对于低延迟矢量搜索,HNSW图索引必须存在于内存中。要计算所需的最小RAM,请使用索引大小估计过程中的步骤,并为开销添加20-25%的缓冲区。If the vector index doesn't fit in RAM, query latency increases.对于低延迟矢量搜索,HNSW图索引必须存在于内存中。要计算所需的最小RAM,请使用索引大小估计过程中的步骤,并为开销添加20-25%的缓冲区。如果向量索引不适合RAM,查询延迟就会增加。

Note

To reduce RAM requirements, consider using vector quantization. Vector quantization can compress vector embeddings, which lowers their memory footprint (and thus the RAM required to hold them) often with minimal impact on recall or precision.为了降低RAM要求,可以考虑使用矢量量化。矢量量化可以压缩矢量嵌入,这降低了它们的内存占用(以及保存它们所需的RAM),通常对召回率或精度的影响最小。