Debugging Techniques in Linux
Duration: 3 days

Target Audience Application developers, and/or anyone interested in obtaining deeper insights into ekeing out the maximum from the Linux OS
Synopsis This course introduces debugging tools in Linux. Powerful utilities ubiquitous in Linux distributions, such as strace/ltrace, proc, and - the built-in debugger, gdb. We discuss how to develop and use debugging skills to deal with common bugs, such as memory corruptions, erroneous usage of pointers, multi-threaded code and race-conditions, socket bugs, and more. Special consideration is given to analyzing core dumps and crashes.
Prerequisites Knowledge of user mode programming. Familiarity with POSIX (unistd.h) is highly recommended.
Objectives
  • Use common Linux performance monitoring tools
  • Use Linux's built-in debugging tools - strace, ltrace, gdb, and others
  • Understand and effectively use GCC's myriad compiler optimizations
  • Effectively code multi-threaded programs
Exercises This course allocates plenty of time for hands-on practice.
The hands-on exercises include:
  • Debugging user mode crash dumps ("core" files)
  • Writing, profiling and debugging code
Modules
1. Introduction to Linux Architecture
2 hours
Introduction to the Linux architecture. We discuss the design and implementation of the Linux Kernel and its various subsystems, at a modular "black box" level, without going into the source code level. We discuss the Kernel architecture and tunable parameters that allow for performance optimization.
  • Linux Kernel features
    • Version differences - 2.4, 2.6 and 2.6 subversions
      • User-Mode and Kernel-Mode Architecture
        • The Linux Task Scheduler (process manager)
          • The Linux I/O Schedulers:
            • Elevator/NoOp
              • DeadLine
                • Anticipatory
                  • Complete Fair Queueing (CFQ)
                  • The /proc filesystem
                    • The /sys filesystem
                      2a. The Basics
                      2 hours
                      This module, dealing with the basic hardware and software architectures of Intel-compatible x86 and x64, serves as an introduction and sets the baseline for the debugging workshop.

                      Note: This module may be replaced by a platform-specific discussion of non-intel architectures as well. This is mostly the case where embedded Linux, on ARM or PPC architectures, is involved.

                      • Computer Architecture
                        • The OS and the CPU
                          • Exception handling
                            • Traps
                              • Interrupt handling
                              2b. Compiling, Linking and Debugging
                              2 hours
                              In-depth discussion of code at the assembly (machine code) level. Students will learn:
                              • Familiarity with assembly code
                                • GCC compiler optimizations
                                  • Using GDB - Tips & Tricks
                                    3. Processes and Threads
                                    2 hours
                                    The inside view of Linux "tasks" - processes and threads - and how the scheduler manages. Using process diagnostic tools effectively.
                                    • Processes
                                      • Daemons
                                        • Threads
                                          • Priorities and Nice
                                            • The kernel perspective - clone()
                                              • Process scheduler states
                                                • Real-Time priorities
                                                  4. Process Tracing & Hooking
                                                  2 hours
                                                  Methods for code-injection into running processes via libraries and the ptrace(2) API.
                                                  • Library calls vs. system calls - when to use which?
                                                    • Libraries
                                                      • Compiling a shared library
                                                        • Injecting libraries
                                                        • System call tracing with ptrace
                                                          • The ptrace(2) API
                                                            • System call level tracing with strace(1)
                                                              • Library call level tracing with ltrace(1)
                                                              • gprof
                                                                • valgrind
                                                                  5. Process Internals
                                                                  3 hours
                                                                  A deeper look into user mode processes - the virtual memory layout, and structure of the stack and the heap.
                                                                  • Process memory layout
                                                                    • The Stack
                                                                      • Compiler optimization
                                                                        • The Heap
                                                                          6. Memory management bugs
                                                                          4 hours
                                                                          A detailed discussion of memory allocation/management bugs, and how they manifest themselves in stack and heap corruption.
                                                                          • Stack overrun
                                                                            • Stack buffer overflows
                                                                              • Heap buffer overflows
                                                                                • Double Free bugs
                                                                                  • Uninitialized memory
                                                                                    • Use after free
                                                                                      • Memory leaks
                                                                                        7. Concurrency
                                                                                        3 hours
                                                                                        A discussion of the challenges involved with writing multi-threaded and multi-process code.
                                                                                        • Concurrency
                                                                                          • Inter-Process Communication (IPC) techniques
                                                                                            • Locking techniques
                                                                                              • Types of locks
                                                                                                • Optimizing locks
                                                                                                  • Deadlocks
                                                                                                  • GDB and multi-threaded programs
                                                                                                    8. Kernel-Level Debugging
                                                                                                    1 hours
                                                                                                    Basic knowledge of Kernel-level coding techniques offers a plethora of new debugging techniques. While this is not a full-fledged Linux Kernel course, we introduce the basic concepts for a quick-and-dirty module, to allow for the injection of code into the Kernel.
                                                                                                    • Why Kernel? The benefits of writing a module:
                                                                                                      • System call interception
                                                                                                        • Precise timing measurements
                                                                                                          • Faking system calls - Virtualizing or stress testing
                                                                                                          • Writing a basic module
                                                                                                            9. Kernel Network Debugging
                                                                                                            2 hours
                                                                                                            Simple, yet surprisingly effective techniques to debug networking (socket) code. We introduce socket level API hooking, and packet level hooking (via NetFilter).

                                                                                                            • struct sock - The Kernel implementation of sockets
                                                                                                              • struct sk_buff - Socket buffers
                                                                                                                • struct proto - the implementations of TCP and UDP
                                                                                                                  • NetFilter hooks