Stack smashing is still one of the most popular techniques for hijacking program controls. A return address of a program is the best-known target, and its previous frame pointer is the second best. Various techniques have been proposed to defeat stack smashing attacks, but most techniques need to alter compilers or require hardware support, and only few of them are developed for Windows. We discover there is a potential security risk in those schemes which dynamically allocate a return address stack to backup return addresses against stack smashing attacks. We are able to hijack the program control from applications that are protected by them via manipulating Memory Pointer Corruption Attack because they only pay attention to protect the return address stack and all neglect to protect Entry Pointer of the return address stack. In this thesis, we design a Secure Return Address Stack to protect both of them from stack smashing attacks on Windows. Moreover, we also extend our approach to instrument a DLL, a multi-thread application, and DLLs used by multi-thread applications. In contrast to previous researches, our approach properly instruments DLLs. Finally, benchmark GnuWin32 shows that the relative performance overhead of our approach is only between 3.47% and 8.59%.

Stack Smashing依然是攔截程式控制權最常用的方法之一。return address是一個程式裡最常被攻擊的目標，而previous frame pointer僅次於它。已經有許多的防禦機制被提出來抵擋Stack Smashing Attack，但是大部份都須要修改compiler或硬體支援，而且只有少數是發展在Windows上。我們發現了一個潛在的安全性問題針對於那些使用動態配置Return Address Stack來抵抗Stack Smashing Attack的方法。我們能夠攔截受這些方法保護程式的程式控制權經由操作Memory Pointer Corruption Attack，因為這些方法只注意到要保護Return Address Stack而完全忽略Return Address Stack的Entry Pointer也是須要保護的。在這篇論文裡，我們將設計一種Secure Return Address Stack來保護上述二者以防禦Stack Smashing Attack在Windows上。此外，我們還延伸我們的方法去處理一個DLL、一個multi-thread程式、多個DLL被應用於multi-thread程式。相較於先前的研究，我們的方法更完善的處理DLL問題。最後，benchmark GnuWin32顯示我們的方法只有增加3.47%到8.59%的performance overhead而已。

Table of Contents..........................................I
List of Figures..........................................III
List of Tables............................................IV
Chapter 1 Introduction.....................................1
1.1 Outline................................................1
1.2 Background.............................................4
1.3 Memory Pointer Corruption Attack.......................5
1.4 Bypass StackShield by Memory Pointer Corruption Attack.8
1.5 Organization..........................................12
Chapter 2 Approach........................................13
2.1 Overview..............................................13
2.2 Secure Return Address Stack...........................16
2.3 Instrumentation Process...............................17
2.4 Binary Disassembly and Function Analysis..............19
2.5 DLL Injection and Dynamic Binary Rewriting............20
2.5.1 DLL Injection.......................................20
2.5.2 Return Address Stack Initialization.................20
2.5.3 Dynamic Binary Rewriting............................21
2.6 Instrumenting DLLs and Multi-threading Applications...21
2.6.1 Instrumenting a DLL.................................22
2.6.2 Instrumenting a Multi-threading Application.........23
2.6.3 Instrumenting DLLs used by Multi-threading Applications..............................................25
Chapter 3 Experiment and Analysis.........................26
3.1 Micro-Benchmark.......................................27
3.2 Macro-Benchmark.......................................27
Chapter 4 Discussion......................................33
4.1 The setjmp() and longjmp() Problem....................33
4.2 The Optimization Problem..............................34
4.3 The Synchronization Problem...........................34
Chapter 5 Related Work....................................36
5.1 Static Analysis.......................................36
5.2 C Library Patches.....................................37
5.3 Kernel Patches........................................37
5.4 Compiler-Based........................................38
5.5 Hardware-Support......................................39
5.6 Binary Rewriting......................................40
5.7 Summary...............................................41
Chapter 6 Conclusion......................................42
Bibliography..............................................50
Appendix..................................................50
Chapter A Unsafe Functions in the Standard C Library......51
Chapter B Windows APIs....................................52

[1] P. Akritidis, E. P. Markatos, M. Polychronakis, and K. Anagnostakis. Stride: Polymorphic Sled Detection through Instruction Sequence Analysis. In Proceedings of the 20th IFIP International Information Security Conference, 2005.

[2] K. Avijit, P. Gupta, and D. Gupta. TIED, LibsafePlus: Tools for Runtime Buffer Overflow Protection. In Proceedings of the 13th Conference on USENIX Security Symposium, pages 45-56, 2004.

[3] K. Avijit, P. Gupta, and D. Gupta. Binary Rewriting and Call Interception for Efficient Runtime Protection against Buffer Overflows: Research Articles. John Wiley and Sons, New York, 2006.

[4] A. Baratloo, N. Singh, and T. Tsai. Transparent Run-Time Defense against Stack Smashing Attacks. In Proceedings of the Annual Conference on USENIX Annual Technical Conference, pages 251-262, 2000.

[5] A. Baratloo, T. Tsai, and N. Singh. Libsafe: Protecting Critical Elements of Stacks. http://www.research.avayalabs.com/project/libsafe/, 1999.

[6] B. Bray. Compiler Security Checks in Depth. http://msdn.microsoft.com/zh-tw/library/aa290051(en-us).aspx, 2002.

[7] Bulba and Kil3r. Bypassing StackGuard and StackShield.
http://www.phrack.org/issues.html?issue=56&id=5, 2000.

[8] S. Chen, J. Xu, N. Nakka, Z. Kalbarczyk, and R. K. Iyer. Defeating Memory Corruption Attacks via Pointer Taintedness Detection. In Proceedings of the 2005 International Conference on Dependable Systems and Networks, pages 378-387, 2005.

[9] T. Chiueh and F. Hsu. RAD: A Compile-Time Solution to Buffer Overflow Attacks. In Proceedings of the 21th International Conference on Distributed Computing Systems, pages 409-419, 2001.

[10] Y. Choi, D. Seo, and S. Sohn. A New Stack Buffer Overflow Hacking Defense Technique with Memory Address Confirmation. In Proceedings of the 4th International Conference Seoul on Information Security and Cryptology, pages 146-159, 2002.

[11] M. L. Corliss, E. C. Lewis, and A. Roth. Using DISE to Protect Return Addresses from Attack. SIGARCH Computer Architecture News, pages 65-72, 2005.

[12] C. Cowan, S. Beattie, J. Johansen, and P.Wagle. Pointguard: Protecting Pointers from Buffer Overflow Vulnerabilities. In Proceedings of the 12th Conference on USENIX Security Symposium, pages 91-104, 2003.

[13] C. Cowan, C. Pu, D. Maier, H. Hintony, J. Walpole, P. Bakke, S. Beattie, A. Grier, P. Wagle, and Q. Zhang. StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks. In Proceedings of the 7th Conference on USENIX Security Symposium, pages 63-78, 1998.

[14] C. Cowan, P. Wagle, C. Pu, S. Beattie, and J. Walpole. Buffer Overflows: Attacks and Defenses for the Vulnerability of the Decade. In Proceedings of the DARPA Information Survivability Conference and Expo, pages 119-129, 1999.

[15] J. Duraes and H. Madeira. A Methodology for the Automated Identification of Buffer Overflow Vulnerabilities in Executable Software without Source-Code. In Proceedings of the Second Latin-American Symposium on Dependable Computing, 2005.

[16] A. Ekbom and S. Ottosson. Comparative Study of Run-Time Defense against Buffer Overflows. http://www.ida.liu.se/~TDDC03/oldprojects/2005/final-projects/prj15.pdf, 2005.

[17] H. Etoh and K. Yoda. Protecting from Stack-Smashing Attacks. http://www.trl.ibm.com/projects/security/ssp/main.html, 2002.

[18] M. Frantzen and M. Shuey. StackGhost: Hardware Facilitated Stack Protection. In Proceedings of the 10th Conference on USENIX Security Symposium, pages 55-66, 2001.

[19] S. Gupta, P. Pratap, H. Saran, and S. Arun-Kumar. Dynamic Code Instrumentation to Detect and Recover from Return Address Corruption. In Proceedings of the 2006 International Workshop on Dynamic Systems Analysis, pages 65-72, 2006.

[20] E. Haugh and M. Bishop. Testing C Programs for Buffer Overflow Vulnerabilities. In Proceedings of the Network and Distributed System Security Symposium, 2003.

[21] G. Hunt and D. Brubacher. Detours: Binary Interception of Win32 Functions. In Proceedings of the 3rd Conference on USENIX Windows NT Symposium, pages 135-144, 1999.

[22] IDA Pro Project. IDA Pro v.5.3. http://www.datarescue.com/, 2008.

[23] G. S. Kc and A. D. Keromytis. e-NeXSh: Achieving an Effectively Non-Executable Stack and Heap via System-Call Policing. In Proceedings of the 21st Annual Computer Security Applications Conference, pages 286-302, 2005.

[24] D. J. Kim, T. H. Kim, J. Kim, and S. J. Hong. Return Address Randomization Scheme for Annuling Data-Injection Buffer Overflow Attacks. http://www.springerlink.com/content/fv7847765r7173h0/fulltext.pdf, 2006.

[25] Klog. The Frame Pointer Overwrite. http://www.phrack.com/issues.html?issue=55&id=8, 2000.

[26] K. Ku, T. E. Hart, M. Chechik, and D. Lie. A Buffer Overflow Benchmark for Software Model Checkers. In Proceedings of the twenty-second IEEE/ACM International Conference on Automated Software Engineering, pages 389-392, 2007.

[27] B. A. Kuperman, C. E. Brodley, H. Ozdoganoglu, T. N. Vijaykumar, and A. Jalote. Detection and Prevention of Stack Buffer Overflow Attacks. Communications of the ACM, pages 50-56, 2005.

[28] D. Larochelle and D. Evans. Statically Detecting Likely Buffer Overflow Vulnerabilities. In Proceedings of the 10th Conference on USENIX Security Symposium, pages 177-190, 2001.

[29] K. Lhee and S. J. Chapin. Type-Assisted Dynamic Buffer Overflow Detection. In Proceedings of the 11th Conference on USENIX Security Symposium, pages 81-88, 2002.

[30] Z. Lin, B. Mao, and L. Xie. LibsafeXP: A Practical and Transparent Tool for Run-Time Buffer Overflow Preventions. In Proceedings of the 2006 IEEE Workshop on Information Assurance, 2006.

[31] B. B. Madan, S. Phoha, and K. S. Trivedi. StackOFFence: A Technique for Defending against Buffer Overflow Attacks. In Proceedings of the International
Conference on Information Technology: Coding and Computing, pages 656-661, 2005.

[32] S. Nanda, W. Li, L. Lam, and T. Chiueh. BIRD: Binary Interpretation using Runtime Disassembly. In Proceedings of the International Symposium on Code Generation and Optimization, pages 358-370, 2006.

[33] D. Nebenzahl, M. Sagiv, and A. Wool. Install-Time Vaccination of Windows Executables to Defend against Stack Smashing Attacks. IEEE Transactions on Dependable and Secure Computing, 3(1), 2006.

[34] OllyDbg Project. OllyDbg v.2.0. http://www.ollydbg.de/version2.html, 2007.

[35] H. Ozdoganoglu, T. N. Vijaykumar, C. E. Brodley, B. A. Kuperman, and A. Jalote. SmashGuard: A Hardware Solution to Prevent Security Attacks on the Function Return Address. IEEE Transactions on Computers, 55(10), 2006.

[36] S. Park, Y. Han, S. Hong, H. Kim, and T. Chung. The Dynamic Buffer Overflow Detection and Prevention Tool for Windows Executables using Binary Rewriting. In Proceedings of the 9th International Conference on Advanced Communication Technology, pages 1776-1781, 2007.

[37] Y. Park and G. Lee. Repairing Return Address Stack for Buffer Overflow Protection. In Proceedings of the 1st Conference on Computing Frontiers, pages 335-342, 2004.

[38] Y. Park, Z. Zhang, and G. Lee. Microarchitectural Protection against Stack-Based Buffer Overflow Attacks. IEEE Micro, pages 62-71, 2006.

[39] M. Pietrek. An In-Depth Look into the Win32 Portable Executable File Format. http://msdn.microsoft.com/en-us/magazine/cc301805.aspx, 2002.

[40] J. Pincus and B. Baker. Beyond Stack Smashing Recent Advances in Exploiting Buffer Overruns. In Proceedings of the IEEE Symposium on Security and Privacy, pages 20-27, 2004.

[41] M. Prasad and T. Chiueh. A Binary Rewriting Defense against Stack-Based Buffer Overflow Attacks. In Proceedings of the USENIX Annual Technical Conference, pages 211-224, 2003.

[42] F. Qin, C. Wang, Z. Li, H. Kim, Y. Zhou, and Y. Wu. LIFT: A Low-Overhead Practical Information Flow Tracking System for Detecting Security Attacks. In Proceedings of the 39th Annual IEEE/ACM International Symposium on
Microarchitecture, pages 135-148, 2006.

[43] G. Richarte. Four Different Tricks to Bypass StackShield and StackGuard Protection. http://downloads.securityfocus.com/library/StackGuard.pdf, 2002.

[44] G. Richarte. Insecure Programming by Example. http://community.core-sdi.com/~gera/InsecureProgramming/, 2002.

[45] T. Romer, G. Voelker, D. Lee, A. Wolman, W. Wong, H. Levy, B. Bershad, and B. Chen. Instrumentation and Optimization of Win32/Intel Executables using Etch. In Proceedings of the USENIX Windows NT Workshop, pages 1-7,
1997.

[46] O. Ruwase and M. S. Lam. A Practical Dynamic Buffer Overflow Detector. In Proceedings of the 11th Annual Network and Distributed System Security Symposium, pages 159-169, 2004.

[47] R. Seacord. Secure Coding in C and C++: Of Strings and Integers. In Proceedings of the IEEE Symposium on Security and Privacy, pages 74-76, 2006.

[48] S. Sidiroglou, G. Giovanidis, and A. D. Keromytis. A Dynamic Mechanism for Recovery from Buffer Overflow Attacks. In Proceedings of the 8th Information Security Conference, pages 1-15, 2005.

[49] K. Skadron, P. S. Ahuja, M. Martonosi, and D. W. Clark. Improving Prediction for Procedure Returns with Return-Address-Stack Repair Mechanisms. In Proceedings of the 31st Annual ACM/IEEE International Symposium on Microarchitecture, pages 259-271, 1998.

[50] Vendicator. StackShield. http://www.angelfire.com/sk/stackshield/, 2000.

[51] J. Wilander and M. Kamkar. A Comparison of Publicly Available Tools for Dynamic Buffer Overflow Prevention. In Proceedings of the Network and Distributed System Security Symposium, pages 149-162, 2003.

[52] J. Xu. Intrusion Prevention using Control Data Randomization. In Proceedings of the IEEE International Conference on Dependable Systems and Networks, pages 25-27, 2003.

[53] J. Xu, Z. Kalbarczyk, S. Patel, and R. K. Iyer. Architecture Support for Defending against Buffer Overflow Attacks. In Proceedings of the Workshop on Evaluating and Architecting System Dependability, pages 51-62, 2002.

[54] TAN Y. and CAO Y. Method of Preventing Buffer Overflow Attacks by Intercepting DLL Functions. Journal of Beijing Institute of Technology, 14(3), 2005.

[55] D. Ye and D. Kaeli. A Reliable Return Address Stack: Microarchitectural Features to Defeat Stack Smashing. In Proceedings of the Workshop on Architectural Support for Security and Anti-Virus, pages 73-80, 2005.

[56] Y. Younan, D. Pozza, F. Piessens, and W. Joosen. Extended Protection against Stack Smashing Attacks without Performance Loss. In Proceedings of the 22nd Annual Computer Security Applications Conference, pages 429-438, 2006.