Introducing OpenSearch 2.17

Tue, Sep 17, 2024 · Saurabh Singh, Fanit Kolchina

OpenSearch 2.17 is now available! You can download the latest version of OpenSearch to explore exciting new features and enhancements designed to improve machine learning (ML) integration, scalability, cost efficiency, and search performance. This release includes significant updates such as enhanced ML inference search processors, expanded batch processing capabilities, advanced search optimization, and more. Read on to dive into these powerful new features. For a complete breakdown, check out the release notes. Be sure to explore the new OpenSearch Dashboards on OpenSearch Playground, and head to our documentation for in-depth guidance.

Vector database and generative AI

OpenSearch 2.17 adds a number of features to OpenSearch’s vector database and generative AI functionality to help accelerate application development and enable generative AI workloads.

Reduce costs and boost efficiency with seamless disk-optimized vector search

OpenSearch 2.17 introduces the new disk-optimized vector search, which significantly reduces the operational costs for vector workloads. This feature uses binary quantization (BQ), achieving an impressive 32x compression in memory usage. This feature can deliver up to a 70% cost savings while maintaining a recall rate of 0.9 or higher and p90 latencies lower than 200 milliseconds. One of the key advantages of disk-optimized vector search is its seamless out-of-the-box (OOB) integration, eliminating the need for complex preprocessing or training steps. By optimizing memory usage and reducing computational overhead, disk-optimized vector search helps you efficiently manage large-scale vector workloads.

Reduce memory usage and accelerate performance with byte vector encoding

OpenSearch’s Faiss engine now supports byte vector encoding. This memory-efficient encoding technique reduces memory usage by up to 75% with a minimal loss in recall, making it suitable for large-scale workloads. We recommend using byte vectors especially when input vectors contain values ranging from -128 to 127. This compact representation lowers search latencies and improves indexing throughput, accelerating your processes.

Enhance security and streamline updates in the Flow Framework plugin

The Flow Framework plugin now supports advanced user-level security. You can now use backend roles to configure fine-grained access to individual workflows based on roles. New for version 2.17, the reprovision parameter helps you update and provision previously provisioned templates.

Use asynchronous batch ingestion for efficient high-volume ML task processing

OpenSearch 2.17 also introduces asynchronous batch ingestion, which allows you to trigger batch inference jobs, monitor job status, and ingest results once batch processing is complete. This streamlines high-volume ML tasks, such as generating embeddings for large datasets and ingesting them into k-NN indexes.

Customize ML inference search response processors for your use case

OpenSearch 2.16 introduced ML inference search processors, enabling you to run model predictions while executing search queries. Starting with OpenSearch 2.17, you can choose to run model predictions for all documents in a single request or run model predictions for each document individually, further enhancing search response processors. This improvement provides you with the flexibility to customize ML inference search response processors to suit your specific use case.

This release also delivers new functionality focused on helping you improve search and query performance.

Optimize search performance with enhanced concurrent segment search settings and expanded support

Concurrent segment search has been generally available since OpenSearch 2.12. It supports two tuning parameters: an on/off switch and a cluster-level setting that specifies the number of slices (maximum thread count) per request. In general, a higher number of slices reduces latency but requires more processing power per request. More specific analysis has shown that some request types (including aggregations and remote store searches) benefit more from concurrent segment search than others. To help you take advantage of concurrent segment search for the right request types, OpenSearch 2.17 introduces a new setting both at the index and cluster levels. These settings, along with pluggable decider logic, provide more granular control over which requests use concurrent search. This release also supports a per-index (as opposed to per-cluster) maximum slice count setting. Additionally, we have expanded concurrent segment search to support more script-based search requests, including composite aggregations with scripting and derived fields.

Enhance query performance with efficient numeric term encoding and roaring bitmaps

OpenSearch has provided the terms query type since version 1.0, serving as a method to filter searches by including documents that contain a field value within a specified set of values. It has always supported a terms lookup, which retrieves those filter values from another index based on a document ID, essentially allowing a limited version of cross-index join. A previous limitation has been that the number of values retrieved from the lookup index was limited to the hundreds (or potentially low thousands). In OpenSearch 2.17, we have added support for encoding numeric term values as a roaring bitmap. By encoding the values more efficiently, a search request can use a stored filter that matches over a million documents, with lower retrieval latency and reduced memory usage. For more information, see Bitmap filtering.

Optimize query performance with improved multi-term query planning

In OpenSearch 2.12, we introduced a query planning feature for multi-term queries on keyword fields. These include range , terms , wildcard , prefix, and fuzzy queries. These queries tend be more expensive to evaluate on an index because you first need to collect all matching keyword terms for the field and then collect all documents that match all of the terms. With the change introduced in version 2.12, if one of these queries is part of a conjunction involving another clause that matches a few documents, you can avoid the heavy upfront work by applying the multi-term query as a secondary filter to documents that match the other clause. Unfortunately, we have seen some cases in which this query planning has chosen the wrong path due to faulty cost estimation. While we work toward improving the cost estimates, we have reverted to the pre-2.12 behavior by default in version 2.17. You can enable the pre-2.12 behavior in versions 2.12–2.16 by setting search.keyword_index_or_doc_values_enabled to true.

Accelerate search performance with new techniques for computationally intensive queries

As part of our ongoing effort to improve search performance, we are introducing a new set of experimental features intended to improve the performance of computationally expensive queries. Our approximation framework brings new techniques to short-circuit long-running queries by only scoring relevant documents in the query. The first queries to take advantage of this framework in version 2.17 are top-level range queries with no other clauses. These queries show benchmark results of up to 50x faster query running times with this optimization. To enable these new experimental features, set opensearch.experimental.feature.approximate_point_range_query.enabled to true in opensearch.yml.

Observability and log analytics

This release provides an expansion of OpenSearch’s observability and log analytics capabilities.

Enhance trace analysis with a new custom trace source for advanced filtering and cross-cluster support

OpenSearch 2.17 introduces a new custom trace source as an experimental feature. This new trace source is based on the OpenTelemetry schema and includes a redesigned overview page for traces and services that offers advanced filtering capabilities for trace metadata and optimized performance. The new experimental custom trace source will also provide flexibility for wildcards in index names and support for cross-cluster trace indexes.

Unlock enhanced anomaly detection with flexible data sources, advanced imputation methods, and domain-specific rules

With the OpenSearch 2.17 release, we have added the following enhancements to the Anomaly Detection plugin:

  • Support for remote and multiple data sources: The Anomaly Detection dashboards now support remote index selection, allowing you to select indexes from remote clusters in addition to local ones. You can choose any number of remote and local indexes for flexibility in data analysis.
  • Advanced imputation methods for missing data: To address gaps in your data streams, we’ve introduced new imputation methods, including filling missing values with zeros, previous values, or custom values. These options help maintain data continuity and improve anomaly detection accuracy in the presence of missing data.
  • Integration of ML models with domain-specific rules: We’ve combined ML models with domain-specific rules to enhance prediction precision. You can now filter anomalies based on the relationship between actual and expected values using ratio thresholds or absolute value thresholds. This allows for more tailored anomaly detection that aligns with your specific business requirements.

Ease of use

This release also includes usability enhancements.

Simplify index creation with application-based configuration templates for optimized performance and seamless integration

OpenSearch 2.17 introduces application-based configuration templates as an experimental feature. If this feature is enabled, your cluster provides a predefined set of component templates that can be used while creating indexes and index templates based on your use case. These templates are designed to optimize performance for various use cases, such as logs and metrics, and are compatible with the schema defined in the OpenSearch catalog to further ease end-to-end integration with visualization experiences. This feature can be used through the newly available opensearch-system-templates plugin.

Cost, performance, and scalability

This release introduces updates to help you improve the stability, availability, and resiliency of your OpenSearch clusters.

Enhance snapshot scalability and resilience with centralized operations and improved data distribution

In OpenSearch 2.17, we added multiple optimizations to snapshots, making them more scalable and resilient. For clusters with remote-backed storage enabled, the snapshot operation is now centralized on the cluster manager node, removing the overhead of communication with other nodes in the cluster. For non-remote-backed storage clusters, we added support for evenly distributing data to the configured snapshot store using hashed prefixes. This helps to avoid snapshot store throttling issues.

Reduce overhead and boost efficiency with remote cluster state publication

Remote cluster state publication was introduced as an experimental feature in OpenSearch 2.15 and is now generally available in OpenSearch 2.17. When this feature is enabled, the active cluster manager node uploads the cluster state to remote-backed storage and then notifies the follower nodes to download the cluster state from remote-backed storage rather than sending it over the transport layer. This reduces the memory, CPU, and communication overhead on the cluster manager node during publication.

Enhance performance and manage traffic with search-only replica shards

OpenSearch 2.17 introduces experimental mechanisms for achieving indexing and search isolation within a cluster. We’ve added a new replica shard type that is intended only to serve search traffic. The shard is isolated from the _bulk write path and not primary eligible. These shards work by using segment replication to pull new segments at the refresh interval and are compatible with both node-node and remote-backed replication sources. To experiment with this feature, set opensearch.experimental.feature.read.write.split.enabled to true in opensearch.yml. Once the setting is enabled, you can add or remove shards using the new index.number_of_search_only_replicas setting. You can optionally allocate the shards to specific hardware by using a new cluster.routing.allocation.search.replica.dedicated.include filter. This feature is still in active development, and we would appreciate your feedback.

Get started with OpenSearch 2.17

Ready to try the latest features? Head to our downloads page to download OpenSearch 2.17. For more information, see the release notes, documentation release notes, and documentation. You can also explore the new visualization options on OpenSearch Playground. As always, we welcome your feedback about this release on our community forum, and we look forward to hearing about your experiences with the latest features!