Beyond Plain Text Logs: Transforming Software Logs Representations Towards Efficient and Effective Log Management
Loading...
Authors
Yao, Kundi
Date
2024-01-30
Type
thesis
Language
eng
Keyword
Log management , Log compression , Software logging , Log analysis , AIOps , Compression algorithm , Sturctured logging , Natural language processing
Alternative Title
Abstract
Software practitioners (e.g., developers and operators) rely on log management tools such as ELK Stack, Datadog, and Splunk to collect, parse, store, and analyze large-sized log data. As of today, log management has evolved into a large and rapidly growing market worth over 2.56 billion. While log management tools are widely embraced in practice, today's log management tools still treat logs as basic text files leading to three major challenges: 1) Log management tools usually have to introduce additional parsing steps in place to transform logs into structured representations due to their limited controls over log data generation; 2) Log management tools have a limited understanding of log semantics which makes log interpretation and analysis a challenging task; 3) The limited knowledge of log characteristics and their varying usage scenarios restricts log management tools from effectively leveraging such characteristics to accommodate various log storage scenarios. These challenges pose significant roadblocks to the development of log management tools for the efficient and effective ingestion, storage, and analysis of logs.
In this Ph.D. thesis, we propose approaches to address the challenges of modern log management tools. The first part of our thesis studies the usage of structured logging, a logging mechanism that directly generates logs in a structured representation to reduce log parsing efforts. This study helps developers of log management tools to better understand the developers' actual needs of structured logging, therefore optimizing the current log generation and ingestion pipelines. Next, we explore the association between log data and its semantic knowledge that can be leveraged to facilitate log analysis activities such as failure diagnosis. Developers of log management tools may leverage our research results to enrich logs with semantic knowledge to build more effective and efficient log analysis solutions. Then we explore the unique characteristics of log data and use them to improve the current log compression techniques used in log management tools. Our proposed approach will help developers of log management tools design more efficient compression strategies to accommodate various log types and different usage scenarios, eventually improving the overall storage efficiency of log data in log management tools.
Description
Citation
Publisher
License
Queen's University's Thesis/Dissertation Non-Exclusive License for Deposit to QSpace and Library and Archives Canada
ProQuest PhD and Master's Theses International Dissemination Agreement
Intellectual Property Guidelines at Queen's University
Copying and Preserving Your Thesis
This publication is made available by the authority of the copyright owner solely for the purpose of private study and research and may not be copied or reproduced except as permitted by the copyright laws without written authority from the copyright owner.
Attribution-NonCommercial-NoDerivatives 4.0 International
ProQuest PhD and Master's Theses International Dissemination Agreement
Intellectual Property Guidelines at Queen's University
Copying and Preserving Your Thesis
This publication is made available by the authority of the copyright owner solely for the purpose of private study and research and may not be copied or reproduced except as permitted by the copyright laws without written authority from the copyright owner.
Attribution-NonCommercial-NoDerivatives 4.0 International