DTRACE BOOK PDF
The contents of this Documentation are subject to the Public Documentation License Version (the “License”); you may only use this Documentation if you . Technically-oriented PDF Collection (Papers, Specs, Decks, Manuals, etc) - tpn/ pdfs. not just a book about DTrace but a book about using it in the wild, Sun Documentation Web site in HTML and PDF format, at more than
|Language:||English, Spanish, Dutch|
|ePub File Size:||15.83 MB|
|PDF File Size:||8.54 MB|
|Distribution:||Free* [*Regsitration Required]|
book, 'DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X, and. FreeBSD', authored by Brendan. Gregg and Jim Mauro, published by Pearson/Prentice Hall. The DTrace error reporting and debugging features are described later in this book. The following diagram shows the different components of the DTrace. @brendangregg. What can Linux learn from DTrace? Primary author of the DTrace book .. ftp://myavr.info
In the above examples, we have seen a few of these probe clauses. Probe Descriptions The probe description defines what probe the clause matches. These are all of the forms where parts can be left out: provider:module:function:name For example, syscall matches all probes by the syscall provider.
The predicates are evaluated when the specified probes fire. If the predicate evaluates to non-zero, the action statements will run, similar to an if statement in C. We can list the same probe multiple times with different predicates. If multiples of them match, they will be run in the order that they appear in the D program.
Actions The actions are enclosed in curly braces. The D language is lightweight, small, and simple. D does not support control flow, such as loops or branches.
We can not define any user functions. And variable declaration is optional. This limits what we can do. Be sure to also check out the D Programming Language guide for details.
Common D Programming Patterns The following examples will give us a good feel for some of the things we can do. This example measures the accumulated time the App Store application spends inside any syscall i.
Scope of Variable D has three different scopes for variables: global, thread local, and probe clause local. A global variable such as foo or bar is visible throughout the D program. Probe clause local variables are similar to local variables in C or Swift.
They can be useful for intermediate results. In the script, we use the first probe clause to match when we enter a syscall. The in front of the name turns it into an aggregation. For syscall probes, as in our case, probefunc is the name of the system call being made.
WebKi mds 8 Wire 8 Terminal 10 com. WebKi value Distribution count 16 0 32 28 64 7 6 0 27 0 Be sure to check out the samples from the Dynamic Tracing Guide for information on how to use lquantize. In addition, they are multidimensional. We fire a statically defined probe when we resume such a task, and a second probe when it completes.
Many of these are specific to the kernel and system calls. Some of these providers are part of the original set documented in the Dynamic Tracing Guide. The dtraceEND is particularly helpful for outputting summaries when running DTrace in the quiet mode.
We used ticksec in our aggregation example above. In most cases, we should really use static probes as described below.
It lets us probe on the function entry and exit of a process. This works in most cases. Note that function entry and return are not always well defined, particularly due to tail-call optimization. For example, look at this probe: pidlibSystem:printf:return It would attach to printf, returning in the process with the process identifier PID It provides probes for Objective-C method entry and exit. Again, using static probes provides more flexibility. Since the colon symbol : is part of the DTrace probe specifier scheme, : symbols in Objective-C method names need to be replaced with a question mark,?.
They are triggered whenever data is either sent or received over the IP. The arguments arg0 to arg5 provide access to the kernel structures related to the IP packet being sent or received. This can be put together into very powerful network debugging tools.
Check the documentation for some excellent examples. These kinds of probes are called static probes in DTrace. We briefly talked about this in the first examples.
We can later change the Objective-C code without affecting the DTrace probes. Additionally, the static probes give us easy access to the arguments. We saw above how we can use this both for timing and logging.
Dynamic Tracing with DTrace SystemTap
The power of DTrace static probes is that they give us a stable interface to debug our code, and an interface that exists even in production code. When someone is seeing odd behavior, we can attach a DTrace script to the running app, even for a production build of that app. And the flexibility of DTrace allows us to use the same probes for multiple purposes. We can use DTrace as a logging tool. But it also allows us to gather detailed quantitative information about timing, network requests, etc.
The reason we can leave the probes in production code is that probes are zero cost, or to be fair, equivalent to a test-and-branch CPU instruction. It is not necessary to edit configuration file each time parameter have to be altered: run subcommand has -s option. The kernel is a fundamental part of a modern computer's operating system.
We will refer to operating system kernel as kernel in the rest of the book. Applications are using system call mechanism to access various kernel functions, and by doing that they transfer control to kernel routines. The current state of application including all variables and current program counter is called context.
Advanced Apple Debugging & Reverse Engineering – Available Now!
C supports only procedural programming, but kernel developers adopted object-oriented and even functional programming. Where can we get information on kernel? Like I said, the most reliable source of such information is source codes which contain comments. You can use cross-reference tools to navigate source codes as easy as click a hyperlink. Some of them are publicly available: like lxr.
You can create your own Introduction 11 12 cross-reference with OpenGrok tool: Of course we have to mention textual sources of information. To address these issues, system administrator should perform instrumentation which depends on the issue arisen: it could be performance statistics collection and their analysis, debug or system audit.
Two common approaches to instrumentation are sampling when you collect state of the system: values of some variables, stacks of threads, etc. Profiling is a most famous example of sampling.
[PDF Download] Solaris Performance and Tools: DTrace and MDB Techniques for Solaris 10 and
Sampling is very helpful when you do not know where issue happens, but it hardly help when you try to know why it happened. With tracing we can install a probe to that function, gather information on lists say their length and collect cumulative execution of function foo , and then cross-reference them, searching for a pattern in lists whose processing costs too much CPU time.
Over time operating system kernels have grown different methods of tracing. First one and a simplest one is counters - each time probe fires say, major page fault , they increase some counter. Module 1: Dynamic tracing tools. But if all debug printing will be enabled, you will get excessive system load. By default, most of debug printing in Solaris are disabled unless you compile a DEBUG-build, which is not publicly available.
So, generally speaking, that approaches provide very limited set of data at very limited set of tracing points. The only approach that widens that limits is kernel debugger, but because each breakpoint halts system, it cannot be used on production systems.
The answer to them are dynamic tracing which is the topic of this book. Dynamic tracing Unlike other approaches, dynamic tracing tool embeds tracing code into working user program or kernel, without need of recompilation or reboot.
Since any processor instruction may be patched, it can virtually access any information you need at any place. Solaris DTrace development was begun in , and it became part of Solaris 10 release. Along with other revolutionary Solaris 10 features, it shaken world of operating systems, and still evolve. Several ports existed, but they lacked of proper support.
The only stable port is provided in Unbreakable Enterprise Kernel by Oracle for their own Linux distribution which is not wide-spread. There were attempt to develop another clone of DTrace - DProbes, but it was a failure. It has two primary sources of information: SystemTap Language Reference to which we will reference with icon SystemTap Tapset Reference Manual to which we will reference with icon.
Of course, there is a Unix manual pages, to which we will refer with icon. SystemTap has to generate native module for each script it runs, which is huge performance penalty, so as alternative to it, Ktap is developing. Modern kernel versions has ebpf integrated, and there is experiment on using it as a platform for generating probe code, but it is far from final stage as of kernel version 4. Finally, there is a sysdig which is scriptless.
Another implementation of Linux tracing is LTTng. It had used static tracing and required kernel recompilation until version 2. As name of the book states, it describes 14 Module 1: Dynamic tracing tools. Here are the workflow of dynamic tracing systems: Userspace Kernel.
DTrace scripts are written in D do not disambiguate with D language from digital mars have extension. That probe code is loaded into kernel address space by a kernel module and patches current binary code of kernel. Probes are writing collected data to intermediate buffers that are usually lock-free, so they have small effect on kernels performance and doesn't need to switch context to a tracing program.
Separate entity called consumer reads that buffers and writes gathered data into terminal, pipe or to a file. DTrace DTrace is shipped with Solaris from version 10, no additional actions needed to install it. It also doesn't need any changes to kernel code: it relies on CTF sections, symbol tables and static tracing points that are included into Solaris Kernel binaries. The heart of DTrace is libdtrace.
There are other front-ends: trapstat 1m and lockstat 1m , but libdtrace. We will refer to dtrace 1m as DTrace further in a book. Note that because there is no special handling for string arguments, you may need duplicate quotes double-quotes needed by DTrace : dtrace -s syscalls.
Can be filtered using options -P, -m, -f or -n or using grep. If file already exists, it appends to it. Here are some useful tunables: bufsize - size of consumer buffer same as -b.
Note that consumer buffers are per-cpu. See more in Dynamic code analysis. That allows handling C preprocessor directives such as include, define, ifdef and so on.
There are some extra preprocessor-related options: 16 Module 1: Dynamic tracing tools.
What changed? Q: Calculate number of probes that are provided by fbt provider: dtrace -l -P fbg wc -l References dtrace 1m dtrace 1m Utility Options and Tunables SystemTap SystemTap is not part of Linux Kernel, so it have to adapt to kernel changes: i. Also, Linux kernels in most distributions are stripped which means that debug information in DWARF format or symbol tables are removed. Many distributions have separate packages with debug information: packages with -debuginfo suffix on RPM-based distributions, packages with -dbg on Debian-based distributions.
They have files that originate from same build the binary came from it is crucial for SystemTap because it verifies buildid of kernel , but instead of text and data sections they contain debug sections. Recent SystemTap versions have stap-prep tool that automatically install kernel debuginfo from appropriate repositories with correct versions.
SystemTap doesn't have VM in-kernel unlike DTrace and KTap , instead it generates kernel module source written in C than builds it, so you will also need a compiler toolchain make, gcc and ld. Compilation takes five phases: parse, elaborate in which tapsets and debuginfo is linked with script, translate in which C code is generated, compile and run:.
Tapsets are a helpers that are written in SystemTap language but some parts may be written in C and they are plugged during elaborate stage. Runtime is written in C and used during compile stage. Because of high complexity of preparing source code and compiling that, SystemTap is slower than a DTrace. To mitigate that issue, it can cache compiled modules, or even use compile servers.
Unlike DTrace, SystemTap has several front-end tools with different capabilities: stapio is a consumer which runs module and prints information from its buffer to a file or stdout.
It is never used directly, but called by stap and staprun tools. Note that SystemTap is very strict about version of kernel it was compiled for. Warning If stap parent is exited, than killall -9 stap won't finish stapio daemon. For example: stap -l 'scsi. The more letters you passed, the more diagnostic information will be printed.
If only one -v was passed, stap will report only finishing of each stage. Generally speaking, it allows dangerous actions.
If it already exists, SystemTap rewrites it. When SystemTap needs to resolve address into a symbol for example, instruction pointer to a corresponding function name , it doesn't look into libraries or kernel modules. Note that in case it is not provided, stap will print a warning with corresponding -d option. Q: Calculate number of probes in a syscall provider and number of variables provided by syscall.
If probes are executed for too much time or too often , it will induce performance degradation in a production system, or at least give results that are very different than from a non-traced system i. Dynamic tracing systems allocate memory for their internal memory which should be limited. That leads to a common principle for all dynamic tracing systems: add some checks before executing actual tracing.
For example, DTrace has Deadman Mechanism that detects system unresponsiveness induced by DTrace and aborts tracing, while SystemTap monitors time spent in each tracing probe. The common error messages you'll see due to that are processing aborted: Abort due to systemic unresponsiveness in DTrace and SystemTap probe overhead exceeded threshold.
Kumpulan Buku Pemrograman Gratis dalam Bahasa Inggris
Unfortunately, SystemTap is not that affective as DTrace, so probe overhead error message is a common thing. To overcome this error in SystemTap you can recompile your script with -t option to see what probes are causing overload and try to optimize them. Another resource that is limited is memory.
Memory limitations are implemented pretty simple: all allocations should be performed when script is launched and with a fixed size. Note that SystemTap still can exhaust memory if you create too many associative arrays, but this will be handled by OOM which will simply kill stap tool. Both DTrace and SystemTap limit size of strings used in scripts. Transport buffer between probes and consumer is also limited, so if you will print in probes faster than consumer can take, you will see There were NN transport failures error in 20 Module 1: Dynamic tracing tools.
The answer to that problem is simple: be less verbose, take data from buffer more frequently regulated by cleanrate tunable in DTrace or increase buffer size -b option and bufsize tunable in DTrace and -s option in SystemTap. Both DTrace and SystemTap are also using special handlers for in-kernel pagefaults, that will disable panic and handle fault if it was caused by tracing.
For example DTrace will complain with error on enabled probe ID 1 ID syscall::read:entry : invalid alignment 0x in action 1 at DIF offset 24 and continue execution, while SystemTap will print ERROR: read fault [man error::fault] at 0x a8 addr near operator at and stop tracing. Note that SystemTap provides more context than DTrace.
Demonstration scripts These scripts have errors which cause error messages described above. While system calls never change their interface due to backwards compatibility if something need to be changed, new system call is introduced , internal kernel function often do that especially if they not a public API for a drivers.
Dynamic tracing languages provide mechanisms to avoid direct use of in-kernel interface by hiding them in abstractions: Stability Data access DTrace SystemTap High translators, i. Downside of that approach is that it provides fewer information than you could access with other approaches. These options will be described in Translators and tapsets section of next module.
Linux kernel is changing faster: it has stable releases each months, and moreover, its builds are configurable, so some features present in one kernel may be disabled in another and vice versa which makes stability is much more fragile.
To overcome that, SystemTap Language has conditional compilation statements which like in C allow to disable certain paths in code. Simplest conditional compilation statements which evaluates to true if variable passed to it is present in debug information which chooses from several variables.
These options are not available in Embedded C, use traditional preprocessor there. Finally, if some probe is missing from kernel, script compilation will fail. DTrace allow to ignore such errors by passing -Z command line option. In SystemTap you may add? Notes - unless you are running Solaris 11 which was deprecated and obsoleted many of its system calls.. References Conditional compilation Stability 22 Module 1: Dynamic tracing tools.
Every script is a set of probes, and each of them binds to a certain event in kernel or application, for example dispatching of a process, parsing SQL query, etc. Each probe may have a predicate which acts as a filter of unnecessary probes, i. Each script consists of global variables declarations followed by probes, and possibly function declarations.
Conditional branch in DTrace may be emulated using predicates, and also a limited support of ternary operator? It converts received pointer to a type struct dentry, extracts string from it and returns it.
Note that there is a delay of 1 millisecond between loop cycles, but it won't be noticed due to larger buffer switching intervals. Probes Definition Probe - is a handler of kernel or application event. When probe is installed into kernel or application, so it can handle such event, we will call it attaching a probe or binding a probe. When event occurs and probe code is executing, we will say probe is firing.
To satisfy this requirements, tracing languages provide mechanisms of defining probes. Definition of SystemTap probe begins with probe keyword followed by probe name and body of probe handler. Name is a dotted-symbol sequence, where each symbol may have optional parameters in braces.
SystemTap supports wildcards in probe names or several probe names in probe clause if you need to use same handler for multiple probes. For example: probe kernel. Provider is a hint to DTrace on how to attach a probe. Different providers usually have different mechanisms of attachment.
Function and Module are relate to a code location where probe will be installed. Name and optional parameters provide meaningful names to a event which will be handled in a probe. Probe names may be combined using comma, and have multiple probes attached to same event, for example in SystemTap: probe syscall.
If DTrace or SystemTap fail to find a probe, it will abort script translation. To overcome that, use -Z option can be supplied to dtrace or question mark has to be added to a probe name in SystemTap: probe kernel. Function boundary tracing Function boundary tracing is the largest and most generic class of tracing.
Function boundary probes attach to entry point or exit hence bounds from a function. Since most functions begin with saving stack and end with retq or similar instruction, tracer simply patches that instruction, by simply replacing it to interrupt or call depending on a platform. Function probe name ends with suffix defining a point in function where probe should be attached:.After that, I started to apply code reading on my work, because sources always have most actual and full information than user documentation.
In the end of draft of this book was prepared. DTrace can detect the corruption as it cannot match the remembered stored value and the loaded value at that memory address. A probe that fires may analyze the run-time situation by accessing the call stack and context variables and evaluating expressions; it can then print out or log some information, record it in a database, or modify context variables.
Safari Books. Both DTrace and SystemTap are also using special handlers for in-kernel pagefaults, that will disable panic and handle fault if it was caused by tracing. DTrace primitives DTrace provides a serial of primitives that help to capture security-sensitive data access events.