Luận án tiến sĩ Predictive Adaptive Parallelism của David L. Wangerin
Luận án tiến sĩ về song song thích ứng dự đoán. Đề xuất phương pháp tối ưu hiệu suất máy tính đa xử lý qua dự đoán động và phân chia công việc thích ứng.
University of California, Irvine
Information and Computer Science
Luan An
Luận án
Năm xuất bản
Số trang
145
Thời gian đọc
22 phút
Lượt xem
0
Lượt tải
0
Phí lưu trữ
40 Point
Mục lục chi tiết
Tóm tắt nội dung
I. Predictive Adaptive Parallelism Overview
Predictive Adaptive Parallelism represents a groundbreaking approach to parallel computing optimization. The dissertation introduces novel techniques for dynamic workload prediction and runtime adaptation. Traditional parallel computing systems struggle with irregular workloads and varying data characteristics. This research addresses these limitations through machine learning prediction and adaptive algorithms.
The core innovation lies in performance vector analysis. Load vectors capture computational characteristics across different program segments. These vectors enable intelligent thread management decisions at runtime. The system adapts to changing conditions without manual intervention.
Computational efficiency improves significantly through predictive scheduling. The framework analyzes instruction classes and execution patterns. Dynamic scheduling algorithms adjust resource allocation based on real-time performance data. This adaptive approach outperforms static compilation techniques.
The research demonstrates practical applications across multiple domains. Benchmark testing validates the theoretical framework. Results show substantial performance gains on heterogeneous systems. Load balancing becomes more effective through continuous monitoring and adjustment.
1.1. Core Research Problem
Parallel programs face irregularity challenges from data set variations. Performance degradation occurs when workloads deviate from expected patterns. Static compiler optimizations cannot address runtime variations. The dissertation identifies three primary sources of irregularity: data set size variations, data value dependencies, and dynamic program behavior. These factors prevent optimal resource utilization in traditional parallel computing frameworks.
1.2. Proposed Solution Framework
The Predictive Adaptive Parallelism system combines workload prediction with runtime adaptation. Performance vectors quantify computational requirements across program segments. Load vectors track instruction class distributions and execution characteristics. The framework enables dynamic thread allocation based on predicted workloads. Machine learning prediction models forecast execution patterns from historical data.
1.3. Key Innovation Areas
Three major innovations distinguish this research. First, performance vector methodology provides fine-grained workload characterization. Second, adaptive algorithms adjust parallelization strategies during execution. Third, predictive models anticipate resource requirements before execution begins. These components work synergistically to optimize computational efficiency across diverse workload conditions.
II. Performance Vector Analysis Methods
Performance vectors form the foundation of predictive adaptive parallelism. These mathematical constructs capture program execution characteristics. Each vector represents computational workload across defined instruction classes. The methodology enables quantitative comparison between program segments.
Load vector generation occurs through static program analysis. The system examines basic blocks and control flow structures. Instruction counting produces initial workload estimates. Loop structures receive special treatment for variant identification.
Vector composition includes multiple instruction classes. Memory operations, arithmetic computations, and control flow instructions each contribute. The granularity of classification affects prediction accuracy. Finer classifications improve precision but increase overhead.
Predictive models use historical vector data. Machine learning algorithms identify patterns across executions. The system forecasts future workloads based on input characteristics. Accuracy improves through continuous learning and refinement.
Runtime adaptation leverages real-time vector updates. Actual execution data refines initial predictions. The framework adjusts thread allocation dynamically. This feedback loop ensures optimal resource utilization throughout program execution.
2.1. Load Vector Construction
Load vectors quantify computational requirements for program segments. Static analysis examines source code structure and instruction sequences. The system categorizes instructions into predefined classes based on operation type. Each class receives a weight reflecting execution cost. Vector components represent expected instruction counts per class. This representation enables mathematical comparison between different parallelization strategies.
2.2. Instruction Classification System
The framework defines distinct instruction classes for accurate workload modeling. Categories include integer arithmetic, floating-point operations, memory accesses, and control flow instructions. Each class exhibits different performance characteristics on target hardware. Classification granularity balances prediction accuracy against computational overhead. The appendix details complete instruction class definitions and associated cost metrics.
2.3. Vector Based Prediction Models
Predictive algorithms transform load vectors into execution time estimates. Machine learning models correlate vector characteristics with observed performance. Training data comes from previous program executions across varying inputs. The system identifies relationships between vector components and runtime behavior. Prediction accuracy determines the effectiveness of adaptive scheduling decisions.
III. Dynamic Scheduling Adaptive Algorithms
Dynamic scheduling algorithms form the execution engine for adaptive parallelism. These algorithms make real-time decisions about thread allocation. The system balances computational load across available processors. Optimization goals include minimizing total execution time and maximizing resource utilization.
The framework supports multiple parallelism patterns. Loop level parallelism distributes iteration spaces across threads. Task level parallelism assigns independent operations to separate processors. Pipeline parallelism streams data through sequential processing stages.
Load balancing strategies adapt to workload characteristics. Homogeneous systems use equal distribution strategies. Heterogeneous systems require capability-aware allocation. The scheduler considers processor speeds and memory hierarchies.
Thread management overhead requires careful consideration. Thread creation and synchronization impose costs. The system balances parallelism benefits against coordination overhead. Adaptive algorithms adjust thread counts based on workload granularity.
Runtime adaptation responds to prediction errors. Monitoring systems detect performance deviations. The framework reallocates resources when actual workloads differ from predictions. This self-correction mechanism maintains efficiency despite imperfect forecasting.
3.1. Loop Level Parallelism Optimization
Loop parallelization represents the most common parallel computing pattern. The system analyzes loop variants to predict iteration workloads. Regular loops with constant iteration costs use simple division strategies. Irregular loops require sophisticated prediction based on data characteristics. The scheduler determines optimal iteration distribution across available threads. Performance optimization considers both computation and communication costs.
3.2. Task Level Parallelism Management
Task parallelism assigns independent program segments to separate processors. The framework identifies parallelizable tasks through dependency analysis. Performance vectors estimate computational requirements for each task. Dynamic scheduling allocates tasks based on processor availability and capability. Load balancing ensures no processor remains idle while work remains. The system handles both embarrassingly parallel and loosely coupled task structures.
3.3. Heterogeneous System Adaptation
Heterogeneous computing environments present unique scheduling challenges. Processors vary in speed, memory capacity, and architectural features. The adaptive framework characterizes each processor's capabilities. Workload distribution accounts for performance differences across resources. Capability-aware scheduling assigns larger workloads to faster processors. This approach maximizes overall system throughput on diverse hardware configurations.
IV. Runtime Adaptation Thread Management
Runtime adaptation enables continuous performance optimization during execution. The system monitors actual performance against predictions. Deviations trigger adaptive responses to maintain efficiency. This dynamic approach handles unpredictable workload variations.
Thread count adjustment represents a primary adaptation mechanism. The framework increases parallelism when workloads exceed predictions. Thread reduction occurs when overhead exceeds parallelism benefits. Adaptation decisions consider current system state and workload characteristics.
Communication pattern optimization reduces synchronization overhead. The system identifies communication bottlenecks through runtime monitoring. Adaptive algorithms restructure data exchange patterns. This optimization improves performance in communication-intensive applications.
Nested parallelism presents complex management challenges. Multiple parallel regions may execute simultaneously. The framework coordinates resource allocation across nesting levels. Hierarchical scheduling prevents resource contention between nested regions.
Performance feedback loops enable continuous improvement. Actual execution data updates predictive models. Machine learning algorithms refine predictions based on observed behavior. This learning process increases accuracy over repeated executions.
4.1. Real Time Performance Monitoring
Continuous monitoring tracks actual execution characteristics during runtime. The system measures instruction execution rates and memory access patterns. Performance counters provide hardware-level execution data. Monitoring overhead must remain minimal to avoid degrading overall performance. Collected data feeds into adaptive decision-making algorithms. Real-time analysis enables rapid response to changing workload conditions.
4.2. Adaptive Thread Allocation Strategies
Thread allocation adapts based on observed performance and predicted workloads. The framework adjusts thread counts to match computational requirements. Increasing threads improves parallelism for large workloads. Decreasing threads reduces overhead for fine-grained operations. Allocation decisions consider thread creation costs and synchronization overhead. The system finds optimal thread counts through iterative refinement.
4.3. Communication Overhead Reduction
Communication costs often dominate parallel program performance. The adaptive framework minimizes data transfer between threads. Scheduling algorithms co-locate communicating tasks when possible. Data replication strategies reduce remote memory access frequency. The system balances communication reduction against load balancing requirements. Network topology awareness improves performance on distributed systems.
V. Workload Prediction Machine Learning
Machine learning prediction enhances adaptive parallelism effectiveness. Predictive models forecast execution characteristics from program inputs. Training data comes from historical execution profiles. Model accuracy directly impacts scheduling quality.
Feature extraction identifies relevant input characteristics. Data set size represents an obvious predictive feature. Data value distributions affect execution paths in conditional code. Input complexity metrics correlate with computational requirements.
Model selection balances accuracy against computational cost. Simple linear models provide fast predictions with limited accuracy. Neural networks achieve higher accuracy but require more computation. The framework chooses models appropriate for application characteristics.
Training strategies determine model quality. Offline training uses historical execution data. Online learning adapts models during program execution. Hybrid approaches combine both strategies for optimal results.
Prediction confidence metrics guide adaptive decisions. High-confidence predictions enable aggressive optimization. Low-confidence scenarios trigger conservative strategies. The system adjusts risk tolerance based on prediction quality.
5.1. Feature Engineering for Prediction
Effective prediction requires identifying relevant input features. Data set dimensions provide primary predictive information. Value distributions indicate computational complexity for data-dependent algorithms. Historical execution patterns reveal recurring workload characteristics. Feature selection reduces dimensionality while preserving predictive power. The system automatically identifies salient features through correlation analysis.
5.2. Model Training and Validation
Training processes establish relationships between features and execution characteristics. Supervised learning uses labeled execution data from benchmark runs. Cross-validation ensures model generalization across diverse inputs. The framework partitions training data to prevent overfitting. Validation metrics assess prediction accuracy on unseen data. Continuous retraining maintains model relevance as workload patterns evolve.
5.3. Online Learning Adaptation
Online learning enables model refinement during production execution. The system updates predictions based on observed performance. Incremental learning algorithms incorporate new data without complete retraining. This approach handles workload drift and changing system conditions. Learning rates balance stability against adaptation speed. Online adaptation improves prediction accuracy for evolving application behaviors.
VI. Experimental Results Performance Analysis
Comprehensive experimental validation demonstrates framework effectiveness. Testing covers diverse applications and system configurations. Results show significant performance improvements over static approaches. The Jacobi relaxation algorithm serves as primary benchmark.
Benchmark selection represents various parallel computing patterns. Loop-intensive scientific computations test iteration distribution. Task-parallel applications validate independent workload scheduling. Pipeline applications assess streaming data processing optimization.
System configurations include homogeneous and heterogeneous platforms. Testing spans different processor counts and network topologies. Performance varies based on workload characteristics and system capabilities. Adaptive approaches show greatest benefits on heterogeneous systems.
Performance metrics quantify optimization effectiveness. Speedup measurements compare parallel versus sequential execution. Efficiency metrics assess resource utilization quality. Scalability analysis examines performance across processor counts.
Timing results validate predictive accuracy. Predicted execution times closely match observed performance. Adaptation overhead remains minimal compared to performance gains. The framework achieves near-optimal scheduling in most scenarios tested.
6.1. Jacobi Relaxation Benchmark Results
The Jacobi relaxation algorithm provides comprehensive performance testing. Experiments vary matrix dimensions from 32 to 1024 elements. Iteration counts range from single to multiple passes. Results demonstrate effective load balancing across thread counts. Timing plots show performance optimization across system configurations. The adaptive framework consistently outperforms static thread allocation strategies.
6.2. Heterogeneous System Performance
Heterogeneous platforms reveal adaptive scheduling advantages. Processor speed variations create load balancing challenges. The framework successfully distributes workload proportional to capabilities. Performance gains increase with greater system heterogeneity. Capability-aware scheduling prevents fast processors from waiting idle. Results validate the effectiveness of performance vector-based workload estimation.
6.3. Scalability and Overhead Analysis
Scalability testing examines performance across processor counts. The framework maintains efficiency as thread counts increase. Prediction overhead remains below five percent of total execution time. Adaptation costs decrease relative to benefits for larger workloads. Communication overhead grows with processor count but remains manageable. The system demonstrates good scalability characteristics for tested applications.
Tải xuống file đầy đủ để xem toàn bộ nội dung
Tải đầy đủ (145 trang)Trích đoạn nội dung luận án
Tải xuống để đọc toàn bộUNIVERSITY OF CALIFORNIA, IRVINE Predictive Adaptive Parallelism DISSERTATION submitted in partial satisfaction of the requirements for the degree of DOCTOR OF PHILOSOPHY in Information and Computer Science by David L. Wangerin Dissertation Committee: Professor Isaac D. Scherson, Chair Professor Alex Nicolau Professor Alex Veidenbaum 2006 UMI Number: 3243277 INFORMATION TO USERS The quality of this reproduction is dependent upon the quality of the copy submitted. Broken or indistinct print, colored or poor quality illustrations and photographs, print bleed-through, substandard margins, and improper alignment can adversely affect reproduction.
In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note will indicate the deletion. ® UMI UMI Microform 3243277 Copyright 2007 by ProQuest Information and Learning Company. All rights reserved.
This microform edition is protected against unauthorized copying under Title 17, United States Code. ProQuest Information and Learning Company 300 North Zeeb Road P. Box 1346 Ann Arbor, MI 48106-1346 © 2006 David L. Wangerin The dissertation of David L.
Wangerin is approved and is acceptable in quality and form for publication on microfilm: Ts lao =e = mmfftee Chair University of California, Irvine 2006 ii TABLE OF CONTENTS LIST OF FIGURES Vv LIST OF TABLES VII ACKNOWLEDGMENTS VIII CURRICULUM VITAE IX ABSTRACT OF THE DISSERTATION xX 1 FOREWORD 1 2 INTRODUCTION 3 2. Q Q Q Q Q Quy và va 10 3 SOLUTION OUTLINE 12 3. vn v VN cv VN va 14 3.4 Example of the System. cv v g k v v k k v k Và 29 43 Loop Level Parallelism.4 Multiple Program Segments .41 Nested Parallel Segments.2 Adjacent Parallel Segments .3 Concurrent Parallel Segments.5 Sensitivity of Performance Vectors.6 Task Level Parallelism.
eee eee ee 56 5 ANALYSIS OF SOLUTION 60 5.3 Jacobi Relaxation Algorithm. 0000000004 90 iil 6 PREVIOUS WORK 91 6.2 Parallelizing Compiler Techniques .3 Full Dynamic Runtime Optimizing Systems. 101 7 CONCLUSIONS AND FUTURE WORK 104 7. ee 105 BIBLIOGRAPHY 114 APPENDICES A Instruction Classes.v và xà B Parallel Python.
ee C Benchmark Set. ee v2 iv LIST OF FIGURES 2.1 Irregularity from data set size.2 Irregularity from data values.3 Choice of how to divide a parallel program.1 Program cost and time translations.2 Jacobi relaxation algorithm in HPE.3 Visual representation of two iterations of the Jacobi relaxation algorithm.4 Division of the data set for the parallel Jacobi relaxation algorithm.5 Structure of a generic thread in the parallel Jacobi relaxation algorithm.6 Pseudocode for a parallel Jacobi relaxation algorithm.7 The load vectors for each section of the parallel Jacobi relaxation al- gorithm. ng cv kg k NV kia 22 3.8 Time profile of the parallel Jacobi relaxation algorithm.1 Visualization of a performance vector with two instruction classes.2 Example of the load vector created from a single basic block.3 Basic block structure of a while loop and repeat loop.4 The load vector from a loop with a regular loop variant.5 The load vector from an irregular loop with an identifiable and pre- dictable loop variant.6 The load vector from a base metric loop.7 Basic block structure of a conditional if-then-else statement.8 Structure of an embarrassingly parallel program.9 Typical timing curve of equation (4.10 Two adjacent parallel sections with communications in the transition between sections. HQ HH ng và Tà xa 42 4.11 Time plot of two adjacent program segments.12 Two parallel sections that execute concurrently.13 Time profile of two program segments under case (1).14 Time profile of two program segments under case (2).15 Time profile of two program segments under case (3).16 Time profile to execute two program segments simultaneously while minimizing the total processing time.17 Time profile of executing three program segments simultaneously.18 Structure of a program using pipeline parallelism.1 Time plot on a fast system.
ee ee ee 61 5.2 Time plot on a medium system.3 Time plot on a slow system.4 Time plot of the same program on three different systems.9 The kernel of the 1-dimensional Jacobi relaxation algorithm.6 The client code for parallel Jacobi relaxation.7 Timing of network packet transmissions.8 Closeup of the low range of packet sizes and timing.9 Timing results for A= 32.10 Timing results for A=64.11 Timing results for A= 128.12 Timing results for A= 256.13 Timing results for A=512, 2.14 Timing results for Á = 1024.15 Timing results for A = 32, 2 iterations.16 Timing results for A = 64, 2 iterations.17 Timing results for A= 128, 2 iterations.18 Timing results for A = 256, 2iterations.19 Timing results for A = 512, 2iterations.20 Timing results for A = 1024, 2 iterations.1 Data decomposition of a matrix for a heterogeneous system. 106 vi LIST OF ‘TABLES 3.1 Machine and program cost characteristics.1 Comparison between scheduling methods for two program segments.1 Performance vectors for the three machines.2 Sensitivity of the performance vector memory value.3 Sensitivity with large memory values.4 Configuration of the test cluster.5 Instruction counts and timing results from the benchmark set.6 Performance vectors using memory-only and least squares models.7 Load vectors for the parallel Jacobi relaxation algorithm.8 Optimal number of threads for Jacobi relaxation algorithm.9 Optimal number of threads for Jacobi relaxation algorithm with 2 it- erations. vii ACKNOWLEDGEMENTS First and foremost, I would like to thank Issac Scherson for being a wonderful adviser, a great friend, and a constant source of inspiration. I first met Isaac in my under- graduate career, and it was by his suggestion that I became involved in research.
If it were not for Isaac, I would never have pursued a graduate degree. I am, and always will be, in debt to him for all of his great advice and help. I am also very grateful for the help and advice of my committee members Alex Nicolau and Alex Veidenbaum. Their feedback and critiques have been both invalu- able and through-provoking, and they have greatly increased the quality of my work.
I would like to thank everyone in my family (and soon to be family) for their constant love and support. I can’t express how much it has meant to me. I would also like to thank everyone from my research group, whose comments, insights, and help have been invaluable. In particular I would like to thank Shean McMahon and John Duselis for their hours of help with working on frustrating math- ematical problems, proof-reading papers, discussing difficult problems, and providing encouragement when I got stuck.
My education and research endeavors have been enriched by my experiences work- ing with NASA Goddard, NASA JPL, Unisys, the UC-MEXUS program, and of course UCI. In particular, I would like to thank John Dorband, Raphael Some, Mike Haken, and the great faculty of UCI. Last but not least, I would like to thank Rob Kolstad who has always given me great advice and pushed me to pursue hard and interesting problems. Vili CURRICULUM VITAE David L.
in Information and Computer Science, University of California, Irvine. 2005-2006 Software Developer, TMT Laboratories, Huntington Beach, California. 2005 Summer Researcher Fellowship, University of California, Irvine. in Information and Computer Science, University of California, Irvine.
2003-2004 Teaching Assistant, Information and Computer Science, University of California, Irvine. 2001 Internship, NASA Goddard Space Flight Center, VSEP Program, Greenbelt, Maryland. in Information and Computer Science, University of California, Irvine. 1998-2002 Co-op, Unisys Corporation, Systems Analysis, Modeling and Measurement Group, Mission Viejo, California.
Publications e David Wangerin and Isaac D. Using Predictive Adaptive Parallelism to Address Portability and Irregularity. In Proceedings of the 2005 Interna- tional Symposium on Parallel Architectures, Algorithms, and Networks (I-SPAN 2005), Las Vegas, Nevada, USA, December 2005. e David Wangerin and Isaac D.
Automatic Resource Management using an Adaptive Parallelism Environment. In Proceedings of the 2003 IEEE Inter- national Parallel and Distributed Processing Symposium (IPDPS) Workshop on Massively Parallel Processing, Nice, France, April 2003. A Modular Client-Server Discrete Event Simulator for Networked Computers. In Proceedings of the 35th Annual Simulation Symposium 2002, San Diego, California, USA, April 2002.
ix ABSTRACT OF THE DISSERTATION Predictive Adaptive Parallelism By David L. Wangerin Doctor of Philosophy in Information and Computer Science University of California, Irvine, 2006 Professor Isaac D. Scherson, Chair Parallel processing is used to increase the execution rate of programs. Since pro- cessing resources are the bottleneck for processing speed, increasing the processing rate is accomplished by adding more processing resources to the system.
However, using extra resources adds extra overhead, and the more resources that are used, the more overhead that is incurred. Optimal performance for parallel programs is achieved by finding the correct balance between computational speedup and overhead of using parallel resources. The current method of optimizing parallel programs is through extensive profiling and manual tuning of programs. This is not ideal since it is both time consuming in terms of programmer and machine time, and does not work for all programs, such as irregular programs and sequential programs executed in dynamic parallel systems.
A novel method, called predictive adaptive parallelism, is presented for automat- ically calculating the optimal number of threads for parallel programs at runtime. The method uses a combination of compile-time and run-time information to gauge the program resource requirements and target machine capabilities. Programs are x described in platform-independent load vectors, which describe the cost of executing a section of a program, and cost functions, which describe the number of times each section will be executed. When the programs are loaded onto a machine, the load vectors are translated into time costs.
As runtime metrics become known, the cost functions can be solved to give a time profile of executing the program as a function of the number of threads assigned to the program. Minimizing the cost function yields the minimal execution time of the program and thus the optimal number of threads. The method is applied to loop-level parallelism and task-level parallelism, and the techniques are shown to be effective and accurate on a cluster system. In addition, the sensitivity of the method to inaccuracy in measuring the machine capabilities is explored.
xi CHAPTER Í FOREWORD A personal anecdote: when I first entered college, I bought a dual-processor computer system. I was extremely excited about it, not the least because I had purchased the system piecemeal over the span of a year with all the money I earned from my job. I had the expectation that with a second processor, all the programs on my system would run about twice as fast. However, after I completed the system and got it running, it seemed that everything ran at about the same speed as when I only had a single processor.
In fact, after running some tests with game frame rates and other simple benchmarking tools available to me, it turned out that all the programs really were running at the same speed as with a single processor. This was both very frustrating and confusing!. The experience was not all bad, as it led me to wonder why the second processor did not add to the execution speed of typical programs. After some investigation, I found out that writing programs to use more than one processor is difficult— difficult enough that for the most part it is only done when programs require it.
This is both because the programming tools for parallel processing are somewhat primitive and because of the fact that using extra processors does not always give a speedup as expected, i., there is more to the problem than just throwing resources at it. Parallel programs need to use different data structures and figure out how to partition the data among the processors, need to use communications between processors to share data, and need to coordinate the activities of all processors to avoid problems like race conditions, deadlocks, and using old data. In addition, debugging parallel programs 1To make the experience even worse, the motherboard had a faulty capacitor that literally burned up and destroyed the motherboard a few months after I added the second processor. Needless to say, the replacement was a single processor system.
Nội dung được bảo vệ bản quyền — Tải xuống đầy đủ
Từ khóa và chủ đề nghiên cứu
Câu hỏi thường gặp
Luận án "Predictive Adaptive Parallelism - Luận án tiến sĩ" nghiên cứu về vấn đề gì?
Luận án tiến sĩ về song song thích ứng dự đoán. Đề xuất phương pháp tối ưu hiệu suất máy tính đa xử lý qua dự đoán động và phân chia công việc thích ứng.
Luận án "Predictive Adaptive Parallelism - Luận án tiến sĩ" được bảo vệ tại trường nào?
Luận án này được bảo vệ tại University of California, Irvine. Năm bảo vệ: 2006.
Luận án "Predictive Adaptive Parallelism - Luận án tiến sĩ" thuộc chuyên ngành gì?
Luận án "Predictive Adaptive Parallelism - Luận án tiến sĩ" thuộc chuyên ngành Information and Computer Science. Danh mục: Khoa Học Máy Tính.
Luận án "Predictive Adaptive Parallelism - Luận án tiến sĩ" có bao nhiêu trang?
Luận án "Predictive Adaptive Parallelism - Luận án tiến sĩ" có 145 trang. Bạn có thể xem trước một phần tài liệu ngay trên trang web trước khi tải về.
Cách tải luận án "Predictive Adaptive Parallelism - Luận án tiến sĩ" về máy như thế nào?
Để tải luận án về máy, bạn nhấn nút "Tải xuống ngay" trên trang này, sau đó hoàn tất thanh toán phí lưu trữ. File sẽ được tải xuống ngay sau khi thanh toán thành công. Hỗ trợ qua Zalo: 0559 297 239.