Author(s)

Min Ji, Peng Fei J

Corresponding Author

Min Ji

ABSTRACT

Applying big data technology for improving the condition evaluation of power transmission and transforming equipment and solving its practical problems becomes a new challenge in power industry. For high reliable storage and rapid access of data, the data distribution strategy, data block size adjustment and the cluster network topology are studied based on hadoop. A multi-copy consistency Hash algorithm based on data correlation (CMCH) is proposed. The algorithm makes the relevant data gathering in the cluster and improves the data processing speed. Based on the CMCH algorithm and Map Reduce model, a multiple data sources map join query algorithm and multi-channel data fusion feature extraction algorithm are designed. The two algorithms are executed on our built clusters and the results show that the CMCH improves the efficiency of multiple data sources join query and multi-channel data fusion feature extraction, and the execution time is just 32% and 35% respectively comparing with standard Hadoop.

KEYWORDS

Big data, Power transmission and transformation equipment, Consistency Hash, Cloud computing