Advanced .NET Debugging with windbg

Setup the environment

Download and install windows debugging tools: http://www.microsoft.com/whdc/devtools/debugging/default.mspx

Run windbg:

Press F6 (or File-> Attach to process) and select the process. You should see a new window with the list of dlls used by the process:

Windbg is a native debugger and is not aware of .NET CLR objects. But it is also an extensible debugger and it is possible to make him aware by loading some extensions. The one we are interested in is SOS. This is done by typing:

.loadby sos mscorwks

The command tells windbg to load sos.dll from the same directory as mscorwks.dll.

Next comes setting up the symbol files location. First the symbols of Microsoft dlls:

.sympath SRV*c:\wheretostorethefiles*http://msdl.microsoft.com/download/symbols

Next adding the symbols for the product under debug:

.sympath+ c:\LocationOfTheProgramFiles

You should see both paths at the output of the debug window.

Now we have to let our application run until the point where we want to analyze the leak. So hit F5.

Main Commands

• !finalizequeue –> Output the list of objects in the finalization queue.
• !address –summary –> Output a resume of the memory usage in the process.
• !eeheap –gc –> Output a resume of the memory used by the internal clr structures.
• !dumpheap –stat –> Output how many instances per type are in the heap.
• !dumpheap –type <TYPE> –> Output the instances of the given type in the heap.
• !dumpobj <address> –> Output the object at the given address.
• !gcroot <address> –> Output the root of the object at the given address.
• !threads –> List the clr threads.
• ~<ThreadNumber>s –> Switch to the given thread number.
• kc –> Output the native stack.
• kb –> Output the native stack including the arguments to the methods.
• !clrstack –> Output the managed stack.

A short overview of how memory management works on .NET

In short .NET memory management works by registering internally the objects that are allocated at the application roots:

• The global and static object pointers in an application.
• Any local variable/parameter object pointers on a thread's stack.
• Any CPU registers containing pointers to objects in the managed heap.

Once these objects are known their children are transverse to build a graph with the reachable objects. If an object is not reachable  then it is garbish an must be collected.

When an object is to be collected it first waits in a finalization queue. The finalization code runs in a separated thread called the finalizer thread. The actual process is a bit more complicated but there are very good articles on this topic:

Garbage Collection: Automatic Memory Management in the Microsoft .NET Framework

Garbage Collection - Part 2: Automatic Memory Management in the Microsoft .NET Framework

Detecting the leak

When I am analyzing a memory leak I first try to identify what is the operation causing the leak. This is done by observing the process memory behavior using task manager, performance counters or any other way of observing the memory usage of a process. The bigger the leak the easier it is to spot. Once the operation causing the leak is found things get simpler.

Before executing that operation I break the process and run:

!dumpheap –stat to see what objects are in the heap.

I them execute the operation and repeat the command. There will be an increase in the number of objects and memory in some of the listed objects. After repeating the process a couple of times one will find a pattern and some suspects.

I select of those suspects, maybe the one causing the biggest increase in memory and use:

!dumpheap –type <object type>

This will list all the objects of that type in the heap. After this one can use the:

!objdump <address>

This will output the object content. The contents can point to the place in the code where these objects are coming from. There is another command that is very usefull to determine that:

!gcroot <address>

This will output the stack of the root of the object.The root will point to the place in the code where the object was created. It does not point to the exact line of code but to the class and method. One must look at the code and figure out the exact point.

Them comes asking if that object should still be in memory or if it should have been collected. If it should them there is an object referencing it. To fix the problem one must change the code to make sure that reference is cleared once the object is no longer needed.

One thing that is sometimes useful is to issue:

!threads

And get the number of the thread of the finalizer and them switch to it:

~<ThreadNumber>s

Finalize issue:

!clrstack

And get the stack where the thread is. If you have a long finalize queue maybe the thread is doing something he should not and you might find there a class finalizer doing things that should not be done on finalizers.

Understanding memory management in mixed mode applications (Managed and Unmanaged).

 

All developers are familiar with the GC class and .NET and its important role in memory management. It was designed to take out the burden of allocating and releasing memory to hold objects. This is a great feature though it comes at a price; you are not in control of when memory is released. This is not an issue for most applications and most objects. The problems start when you allocate large objects.

In the references I include references to information about what is going internally in CLR but here goes a summary:

The .NET garbage collector is a generational collector. It has three generations: generation 0, generation 1, and generation 2. Generations are the logical view of the garbage collector heap.

Objects live on managed heap segments that are chunks of memory that the garbage collector reserves from the OS by calling VirtualAlloc. Large objects have more that 85kb and belong to generation 2, and are collected only in generation 2.

The collection of an older generation triggers the collection of the younger ones. When a generation 1 garbage collection happens, both generation 1 and 0 are collected.

A garbage collection occurs if one of the following conditions happens:

1. Allocation Exceeds the Generation 0 or Large Object Threshold. Most GCs happen because of allocations on the managed heap (this is the most typical case).

2. System.GC.Collect Is called.

3. System Is in Low Memory Situation. The high memory notification is sent by the OS.

The threshold is a property of each generation. Allocating objects into a generation gets the amount of memory used closer to the generation's threshold. If the threshold is exceeded a garbage collection is triggered on that generation.

The CLR clears the memory for every new object. Imagining it takes two cycles to clear 1 byte, it means it takes closely 170,000 cycles to clear the smallest large object (85kb).

When an application makes both managed and unmanaged memory allocations it is a good practice to tell the GC how much memory is being allocated by the unmanaged threads. This is done using the AddMemoryPressure API. This call will not trigger GC by itself but it will allow the GC to tune memory management.

References

http://msdn.microsoft.com/en-us/magazine/cc534993.aspx

http://social.msdn.microsoft.com/Forums/en-US/clr/thread/bfdcbad3-7405-4ef0-8457-88add656f0ca