Author(s)

Yiru Zhang ¹

Affiliation(s)

¹ Cornell Tech, 2 W Loop Rd, New York, NY 10044, USA

Corresponding Author

Yiru Zhang

ABSTRACT

This article meticulously crafts and deploys a comprehensive computer network log analysis system, leveraging advanced big data analytics technologies. Following a rigorous feasibility assessment, the system's architecture was meticulously designed, encompassing robust hardware infrastructure and software components tailored for high-performance processing. The operating environment was optimized to handle massive log data, ensuring scalability and efficiency. The core lies in the five interlocking modules: user authentication ensures secure access; log collection employs distributed techniques for seamless data aggregation; association rule mining uncovers hidden patterns and anomalies through advanced algorithms; security auditing validates log integrity and identifies potential threats; while database management ensures data storage is optimized for both speed and capacity. The system's rigorous functional testing validates its ability to maintain log data integrity, uncover intricate relationships, and bolster log analysis's authenticity and reliability. This achievement not only meets the predefined objectives but also sets a benchmark for future research endeavors in the realm of network log analysis.

KEYWORDS

Big Data, Log Collection, Mapreduce, Log Analysis, P Statistics

CITE THIS PAPER

Yiru Zhang, Design and Implementation of a Computer Network Log Analysis System Based on Big Data Analytics. Advances in Computer, Signals and Systems (2024) Vol. 8: 40-46. DOI: http://dx.doi.org/10.23977/acss.2024.080607.

REFERENCES

[1] Andalib A, Babamir S M. Anomaly detection of policies in distributed firewalls using data log analysis. Journal of supercomputing, 2023, 79 (17): 19473-19514. 
[2] Muratov S Y, Muravyov S B. Framework architecture of a secure big data lake. Procedia Computer Science, 2023, 229: 39-46. DOI: 10. 1016/j. procs. 2023. 12. 005.
[3] Zouhri A, Ezzahout A, Chakouk S, et al. A Numerical Analysis Based Internet of Things (IoT) and Big Data Analytics to Minimize Energy Consumption in Smart Buildings[J].Journal of Automation, Mobile Robotics and Intelligent Systems, 2024, 18(2):46-56. DOI:10.14313/jamris/2-2024/12.
[4] Qiu X, Fan P, Xie B .The Application of Software Engineering Technology in the Era of Big Data[J].Journal of Electronics and Information Science, 2023.DOI:10.23977/jeis.2023.080205.
[5] Wang M, Lu S, Xiao S,et al.An Unsupervised Gradient-Based Approach for Real-Time Log Analysis From Distributed Systems[J].International Journal of Cooperative Information Systems, 2024, 33(02).DOI:10. 1142/ S0218843023500181.
[6] Tim Fürmann, Lechl M, Meer H D, et al. From computer systems to power systems: using stochastic network calculus for flexibility analysis in power systems[J].Energy Informatics, 2023, 6(Suppl 1).DOI:10.1186/ s42162-023 -00286-z.
[7] Tian X, Zhiyuan W U, Cao J,et al.ILIDViz:An incremental learning-based visual analysis system for network anomaly detection[J].Virtual Reality & Intelligent Hardware, 2023, 5(6):471-489.
[8] Rakib M H, Hossain S, Jahan M, et al. A Blockchain-Enabled Scalable Network Log Management System[J]. Journal of computer sciences, 2022.
[9] Yang C T, Chan Y W, Liu J C,et al. Cyberattacks detection and analysis in a network log system using XGBoost with ELK stack[J].Soft Computing, 2022, 26(11):5143-5157. DOI:10.1007/s00500-022-06954-8.
[10] Quan X .A Neural Network Classifier Intrusion Detection Vulnerability System Based on HFE-CAMMLP Intrusion Detection Model and Weblog Analysis[J].2022 4th International Conference on Frontiers Technology of Information and Computer (ICFTIC), 2022:718-721.DOI:10.1109/ICFTIC57696.2022.10075290.

Sponsors, Associates, and Links

---

• Power Systems Computation
  
  
• Internet of Things (IoT) and Engineering Applications
  
  
• Computing, Performance and Communication Systems
  
  
• Journal of Artificial Intelligence Practice
  
  
• Journal of Network Computing and Applications
  
  
• Journal of Web Systems and Applications
  
  
• Journal of Electrotechnology, Electrical Engineering and Management
  
  
• Journal of Wireless Sensors and Sensor Networks
  
  
• Journal of Image Processing Theory and Applications
  
  
• Mobile Computing and Networking
  
  
• Vehicle Power and Propulsion
  
  
• Frontiers in Computer Vision and Pattern Recognition
  
  
• Knowledge Discovery and Data Mining Letters
  
  
• Big Data Analysis and Cloud Computing
  
  
• Electrical Insulation and Dielectrics
  
  
• Crypto and Information Security
  
  
• Journal of Neural Information Processing
  
  
• Collaborative and Social Computing
  
  
• International Journal of Network and Communication Technology
  
  
• File and Storage Technologies
  
  
• Frontiers in Genetic and Evolutionary Computation
  
  
• Optical Network Design and Modeling
  
  
• Journal of Virtual Reality and Artificial Intelligence
  
  
• Natural Language Processing and Speech Recognition
  
  
• Journal of High-Voltage
  
  
• Programming Languages and Operating Systems
  
  
• Visual Communications and Image Processing
  
  
• Journal of Systems Analysis and Integration
  
  
• Knowledge Representation and Automated Reasoning
  
  
• Review of Information Display Techniques
  
  
• Data and Knowledge Engineering
  
  
• Journal of Database Systems
  
  
• Journal of Cluster and Grid Computing
  
  
• Cloud and Service-Oriented Computing
  
  
• Journal of Networking, Architecture and Storage
  
  
• Journal of Software Engineering and Metrics
  
  
• Visualization Techniques
  
  
• Journal of Parallel and Distributed Processing
  
  
• Journal of Modeling, Analysis and Simulation
  
  
• Journal of Privacy, Trust and Security
  
  
• Journal of Cognitive Informatics and Cognitive Computing
  
  
• Lecture Notes on Wireless Networks and Communications
  
  
• International Journal of Computer and Communications Security
  
  
• Journal of Multimedia Techniques
  
  
• Automation and Machine Learning
  
  
• Computational Linguistics Letters
  
  
• Journal of Computer Architecture and Design
  
  
• Journal of Ubiquitous and Future Networks