Author(s)

Pengcheng Yao

Corresponding Author

Pengcheng Yao

ABSTRACT

The arrival of the era of big data has brought great challenges to the storage and management of massive data. Distributed storage is an extremely important solution to storage capacity pressure and cost in the era of big data. As an emerging open source distributed system, Ceph can provide three storage functions: object storage, block storage and file storage. The pseudo-random data mapping function CRUSH is used to complete the mapping of data to storage nodes, eliminating the traditional centralized metadata nodes, while system scalability has no theoretical upper limit. However, Ceph adopts a strong consistency strategy and only reads the primary node when reading objects, which results in low read-write efficiency. At the same time, Ceph does not perceive heterogeneous storage and does not make full use of SSD storage characteristics. Therefore, a combined storage strategy based on weak consistency model and multi-node read optimization is proposed to optimize Ceph storage. The system can effectively reduce the read-write delay of Ceph and improve its storage efficiency. Experiments show that compared with the original mechanism of Ceph, the proposed optimization mechanism can improve the writing efficiency by more than ten times and the reading efficiency by more than six times.

KEYWORDS

Ceph, CRUSH, Heterogeneous storage, Weak consistency, Multi-node read