Education, Science, Technology, Innovation and Life
Open Access
Sign In

An Optimization Strategy of Shard on Elasticsearch

Download as PDF

DOI: 10.23977/amce.2019.003

Author(s)

Zhanglong Wang, Yang Pi

Corresponding Author

Zhanglong Wang

ABSTRACT

With the development of big data application, there are more demands on big data storage and retrieval. Thus, Elasticsearch, a distributed full-text search engine, has appeared, which can well meet these demands. However, Elasticsearch has the following disadvantages: First, shard settings are based on user experience and may degrade retrieval performance due to human factors. Second, the factors considered in the distribution strategy of shards are incomplete. And the last, without effective processing of concurrent access to hot index data, the average performance of each node in the Elasticsearch cluster varies greatly. In this paper, we propose a shard optimization strategy of Elasticsearch, through data and performance analysis, to obtain reasonable shard settings. After that, the shards are placed in nodes with better performance which are evaluated by the linear weighting method. Then, the optimized load balancing strategy will migrate hot shards caused by hot data to make the cluster load balanced. The experimental results show that the proposed shard optimization strategy can achieve better index retrieval performance and better cluster load.

KEYWORDS

ElasticSearch, Index, Shard, Load balancing, Mathematical modeling, Linear weighting method

All published work is licensed under a Creative Commons Attribution 4.0 International License.

Copyright © 2016 - 2031 Clausius Scientific Press Inc. All Rights Reserved.