Boosting vector search performance with concurrent segment search

In OpenSearch, data is stored in shards, which are further divided into segments. When you execute a search query, it runs sequentially across all segments of each shard involved in the query. As the number of segments increases, this sequential execution can increase query latency (the time it takes to retrieve the results) because the query has to wait for each segment run to complete before moving on to the next one. This delay becomes especially noticeable if some segments take longer to process queries than others.

Introduced in OpenSearch version 2.12, concurrent segment search addresses this issue by enabling parallel execution of queries across multiple segments within a shard. By using available computing resources, this feature reduces overall query latency, particularly for larger datasets with many segments. Concurrent segment search is designed to provide more consistent and predictable latencies. It achieves this consistency by reducing the impact of variations in segment performance or the number of segments on query execution time.

In this blog post, we’ll explore the impact of concurrent segment search on vector search workloads.

Enabling concurrent segment search

By default, concurrent segment search is disabled in OpenSearch. For our experiments, we enabled it for all indexes in the cluster by using the following dynamic cluster setting:

PUT _cluster/settings
{
   "persistent": {
      "search.concurrent_segment_search.enabled": true
   }
}

To achieve concurrent segment searches, OpenSearch divides the segments within each shard into multiple slices, with each slice processed in parallel on a separate thread. The number of slices determines the degree of parallelism that OpenSearch can provide. You can either use Lucene’s default slicing mechanism or set the maximum slice count manually. For detailed instructions on updating the slice count, see Slicing mechanisms.

Performance results

We performed our tests on an OpenSearch 2.15 cluster using the OpenSearch Benchmark vector search workload. We used the Cohere dataset with two different configurations to evaluate the performance improvements of vector search queries when running the workload with concurrent segment search disabled, enabled with default settings, and enabled with different max slice counts.

Cluster setup

• 3 data nodes (r5.4xlarge: 128 GB RAM, 16 vCPUs, 250 GB disk space)
• 3 cluster manager nodes (r5.xlarge: 32 GB RAM, 4 vCPUs, 50 GB disk space)
• 1 OpenSearch workload client (c5.4xlarge: 32 GB RAM, 16 vCPUs)
• 1 and 4 search clients
• index_searcher thread pool size: 32

Index settings

Configuration

Service time comparison

We conducted the following experiments:

1. Concurrent search disabled
2. Concurrent search enabled:
   • Max slice count = 0 (default)
   • Max slice count = 2
   • Max slice count = 4
   • Max slice count = 8

The following sections present the results of these experiments.

Experiment 1: Concurrent search disabled

Experiment 2: Concurrent search enabled, max slice count = 0 (default)

Experiment 3: Concurrent search enabled, max slice count = 2

Experiment 4: Concurrent search enabled, max slice count = 4

Experiment 5: Concurrent search enabled, max slice count = 8

Comparing results

For simplicity, we’ll focus on the p90 metric with a single search client because this metric captures the performance of long-running vector search queries.

Service time comparison (p90)

CPU utilization comparison

As demonstrated by our performance benchmarks, enabling concurrent segment search with the default slice count delivers at least a 60% improvement in vector search service time while requiring only 25–35% more CPU. This increase in CPU utilization is expected because concurrent segment search runs on more CPU threads—the number of threads is equal to twice the number of CPU cores.

We observed a similar improvement in service time when using multiple concurrent search clients. However, maximum CPU utilization also doubled, as expected, because of the increased number of active search threads running concurrently.

Conclusion

Our experiments clearly show that enabling concurrent segment search with the default slice count improves vector search query performance, albeit at the cost of higher CPU utilization. We recommend testing your workload to determine whether the additional parallelization achieved by increasing the slice count outweighs the additional processing overhead.

Before running concurrent segment search, we recommend force-merging segments into a single segment to achieve better performance. The major disadvantage of this approach is that the time required for force-merging increases as segments grow larger. Thus, we recommend reducing the number of segments in accordance with your use case.

By combining vector search with concurrent segment search, you can improve query performance and optimize search operations. To get started with concurrent segment search, explore the documentation.