Antivirus Hacker′s Handbook

Introduction xix Part I Antivirus Basics 1 Chapter 1 Introduction to Antivirus Software 3 What Is Antivirus Software? 3 Antivirus Software: Past and Present 4 Antivirus Scanners, Kernels, and Products 5 Typical Misconceptions about Antivirus Software 6 Antivirus Features 7 Basic Features 7 Making Use of Native Languages 7 Scanners 8 Signatures 8 Compressors and Archives 9 Unpackers 10 Emulators 10 Miscellaneous File Formats 11 Advanced Features 11 Packet Filters and Firewalls 11 Self-Protection 12 Anti-Exploiting 12 Summary 13 Chapter 2 Reverse-Engineering the Core 15 Reverse-Engineering Tools 15 Command-Line Tools versus GUI Tools 16 Debugging Symbols 17 Tricks for Retrieving Debugging Symbols 17 Debugging Tricks 20 Backdoors and Configuration Settings 21 Kernel Debugging 23 Debugging User-Mode Processes with a Kernel-Mode Debugger 25 Analyzing AV Software with Command-Line Tools 27 Porting the Core 28 A Practical Example: Writing Basic Python Bindings for Avast for Linux 29 A Brief Look at Avast for Linux 29 Writing Simple Python Bindings for Avast for Linux 32 The Final Version of the Python Bindings 37 A Practical Example: Writing Native C/C++ Tools for Comodo Antivirus for Linux 37 Other Components Loaded by the Kernel 55 Summary 56 Chapter 3 The Plug-ins System 57 Understanding How Plug-ins Are Loaded 58 A Full-Featured Linker in Antivirus Software 58 Understanding Dynamic Loading 59 Advantages and Disadvantages of the Approaches for Packaging Plug-ins 60 Types of Plug-ins 62 Scanners and Generic Routines 63 File Format and Protocol Support 64 Heuristics 65 Bayesian Networks 66 Bloom Filters 67 Weights-Based Heuristics 68 Some Advanced Plug-ins 69 Memory Scanners 69 Non-native Code 70 Scripting Languages 72 Emulators 73 Summary 74 Chapter 4 Understanding Antivirus Signatures 77 Typical Signatures 77 Byte-Streams 78 Checksums 78 Custom Checksums 79 Cryptographic Hashes 80 Advanced Signatures 80 Fuzzy Hashing 81 Graph-Based Hashes for Executable Files 83 Summary 85 Chapter 5 The Update System 87 Understanding the Update Protocols 88 Support for SSL/TLS 89 Verifying the Update Files 91 Dissecting an Update Protocol 92 When Protection Is Done Wrong 100 Summary 101 Part II Antivirus Software Evasion 103 Chapter 6 Antivirus Software Evasion 105 Who Uses Antivirus Evasion Techniques? 106 Discovering Where and How Malware Is Detected 107 Old Tricks for Determining Where Malware Is Detected: Divide and Conquer 107 Evading a Simple Signature-Based Detection with the Divide and Conquer Trick 108 Binary Instrumentation and Taint Analysis 113 Summary 114 Chapter 7 Evading Signatures 117 File Formats: Corner Cases and Undocumented Cases 118 Evading a Real Signature 118 Evasion Tips and Tricks for Specific File Formats 124 PE Files 124 JavaScript 126 String Encoding 127 Executing Code on the Fly 128 Hiding the Logic: Opaque Predicates and Junk Code 128 PDF 129 Summary 131 Chapter 8 Evading Scanners 133 Generic Evasion Tips and Tricks 133 Fingerprinting Emulators 134 Advanced Evasion Tricks 136 Taking Advantage of File Format Weaknesses 136 Using Anti-emulation Techniques 137 Using Anti-disassembling Techniques 142 Disrupting Code Analyzers through Anti-analysis 144 More Anti-Anti-Anti 147 Causing File Format Confusion 148 Automating Evasion of Scanners 148 Initial Steps 149 Installing ClamAV 150 Installing Avast 150 Installing AVG 151 Installing F-Prot 152 Installing Comodo 153 Installing Zoner Antivirus 154 MultiAV Configuration 154 peCloak 158 Writing the Final Tool 160 Summary 162 Chapter 9 Evading Heuristic Engines 165 Heuristic Engine Types 165 Static Heuristic Engines 166 Bypassing a Simplistic Static Heuristic Engine 166 Dynamic Heuristic Engines 173 Userland Hooks 173 Bypassing a Userland HIPS 176 Kernel-Land Hooks 178 Summary 180 Chapter 10 Identifying the Attack Surface 183 Understanding the Local Attack Surface 185 Finding Weaknesses in File and Directory Privileges 185 Escalation of Privileges 186 Incorrect Privileges in Files and Folders 186 Incorrect Access Control Lists 187 Kernel-Level Vulnerabilities 187 Exotic Bugs 188 Exploiting SUID and SGID Binaries on Unix-Based Platforms 189 ASLR and DEP Status for Programs and Binaries 190 Exploiting Incorrect Privileges on Windows Objects 193 Exploiting Logical Flaws 196 Understanding the Remote Attack Surface 197 File Parsers 198 Generic Detection and File Disinfection Code 199 Network Services, Administration Panels, and Consoles 199 Firewalls, Intrusion Detection Systems, and Their Parsers 200 Update Services 201 Browser Plug-ins 201 Security Enhanced Software 202 Summary 203 Chapter 11 Denial of Service 207 Local Denial-of-Service Attacks 208 Compression Bombs 208 Creating a Simple Compression Bomb 209 Bugs in File Format Parsers 212 Attacks against Kernel Drivers 213 Remote Denial-of-Service Attacks 214 Compression Bombs 214 Bugs in File Format Parsers 215 Summary 215 Part III Analysis and Exploitation 217 Chapter 12 Static Analysis 219 Performing a Manual Binary Audit 219 File Format Parsers 220 Remote Services 228 Summary 233 Chapter 13 Dynamic Analysis 235 Fuzzing 235 What Is a Fuzzer? 236 Simple Fuzzing 237 Automating Fuzzing of Antivirus Products 239 Using Command-Line Tools 240 Porting Antivirus Kernels to Unix 243 Fuzzing with Wine 244 Problems, Problems, and More Problems 247 Finding Good Templates 248 Finding Template Files 250 Maximizing Code Coverage 252 Blind Code Coverage Fuzzer 253 Using Blind Code Coverage Fuzzer 254 Nightmare, the Fuzzing Suite 259 Configuring Nightmare 260 Finding Samples 262 Configuring and Running the Fuzzer 262 Summary 266 Chapter 14 Local Exploitation 269 Exploiting Backdoors and Hidden Features 270 Finding Invalid Privileges, Permissions, and ACLs 274 Searching Kernel-Land for Hidden Features 279 More Logical Kernel Vulnerabilities 285 Summary 295 Chapter 15 Remote Exploitation 297 Implementing Client-Side Exploitation 297 Exploiting Weakness in Sandboxing 297 Exploiting ASLR, DEP, and RWX Pages at Fixed Addresses 298 Writing Complex Payloads 300 Taking Advantage of Emulators 301 Exploiting Archive Files 302 Finding Weaknesses in Intel x86, AMD x86-64, and ARM Emulators 303 Using JavaScript, VBScript, or ActionScript 303 Determining What an Antivirus Supports 304 Launching the Final Payload 306 Exploiting the Update Services 307 Writing an Exploit for an Update Service 308 Server-Side Exploitation 317 Differences between Client-Side and Server-Side Exploitation 317 Exploiting ASLR, DEP, and RWX Pages at Fixed Addresses 318 Summary 318 Part IV Current Trends and Recommendations 321 Chapter 16 Current Trends in Antivirus Protection 323 Matching the Attack Technique with the Target 324 The Diversity of Antivirus Products 324 Zero-Day Bugs 324 Patched Bugs 325 Targeting Home Users 325 Targeting Small to Medium-Sized Companies 326 Targeting Governments and Big Companies 326 The Targets of Governments 327 Summary 328 Chapter 17 Recommendations and the Possible Future 331 Recommendations for Users of Antivirus Products 331 Blind Trust Is a Mistake 332 Isolating Machines Improves Protection 337 Auditing Security Products 338 Recommendations for Antivirus Vendors 338 Engineering Is Different from Security 339 Exploiting Antivirus Software Is Trivial 339 Perform Audits 340 Fuzzing 340 Use Privileges Safely 341 Reduce Dangerous Code in Parsers 342 Improve the Safety of Update Services and Protocols 342 Remove or Disable Old Code 343 Summary 344 Index 347

JOXEAN KORET is a security researcher at Coseinc, a Singapore-based security services firm. He is an in-demand speaker at international security conferences, and a well-regarded specialist in reverse-engineering, vulnerability research, and malware analysis. ELIAS BACHAALANY is a computer programmer, reverse engineer, and technical writer employed by Microsoft. An authority in multiple programming languages, database design, and web development, he played an instrumental role in the IDAPython project and IDA Pro's scripting facilities development.