Debugging hints & tips

I've been doing a lot of spelunking with WinDBG recently. There's a lot to know and the documentation can fall short sometimes. Tess's blog usually has enough entries that you can usually figure things out by looking at her examples and scenarios. I always find myself referring back to her blog and a bunch of other blogs which I referred to in a previous blog entry.

Primarily, I'm posting this entry as a future helper for myself. Invariably, I draw the information I need from more than one source so I am effectively collating as much of that that useful information here in one place. If that helps anyone else out along the way, then more power to the blog!

Need to get yourself set up for a WinDBG debug session? Here's what you need to do:

1) Create a directory C:\MyLocalSymbols

2) Install symbols for your operating system to C:\MyLocalSymbols - get symbols from: here

3) Install Windows Debugging Tools - get it from: here

4) Set up _NT_SYMBOL_PATH (choose properties on My Computer->Advanced->Environment Variables->New) to SRV*C:\MyLocalSymbols*http://msdl.microsoft.com/download/symbols
This will save you from having to set the path up in WinDBG and will do it globally for the operating system


How do I load the CLR debug extension SOS.DLL into WinDBG?

1) Type the following command:
.loadby sos mscorwks


What is the WinDBG command to get a list of CLR instances of a given type, e.g. System.AppDomain?

1) !dumpheap -type System.AppDomain

e.g.
0:009> !dumpheap -type System.AppDomain
Address MT Size
012611b4 790fb998 100
01261218 790fbdcc 40
012688c8 790fb998 100
0126bd28 790fbdcc 40


What is the WinDBG command to see the fields of an object?

1) Find the object you are interested and use !do addressgoeshere - (use the previous item as a starting point if you don't currently have an object address!)

In this example we are dumping an AppDomain instance found in the previous example - notice we are using the address column not the MT column!
0:009> !do 012611b4
Name: System.AppDomain
MethodTable: 790fb998
EEClass: 790c3608
Size: 100(0x64) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
MT Field Offset Type VT Attr Value Name
790f9ce8 4000184 4 System.Object 0 instance 01284ed8 __identity
7910f540 4000185 8 ....AppDomainManager 0 instance 00000000 _domainManager
790feb44 4000186 c ...ections.Hashtable 0 instance 00000000 _LocalStore
790fbdcc 4000187 10 ...em.AppDomainSetup 0 instance 01261218 _FusionStore
791094f0 4000188 14 ...y.Policy.Evidence 0 instance 00000000 _SecurityIdentity
79124318 4000189 18 System.Object[] 0 instance 00000000 _Policies
7911d470 400018a 1c ...yLoadEventHandler 0 instance 00000000 AssemblyLoad
79116ffc 400018b 20 ...solveEventHandler 0 instance 00000000 TypeResolve
79116ffc 400018c 24 ...solveEventHandler 0 instance 00000000 ResourceResolve
79116ffc 400018d 28 ...solveEventHandler 0 instance 00000000 AssemblyResolve
79116ffc 400018e 2c ...solveEventHandler 0 instance 00000000 ReflectionOnlyAssemblyResolve
791034f8 400018f 30 ....Contexts.Context 0 instance 012687c4 _DefaultContext
7911db24 4000190 34 ...ActivationContext 0 instance 00000000 _activationContext
7911dc44 4000191 38 ...plicationIdentity 0 instance 00000000 _applicationIdentity
7911dd20 4000192 3c ....ApplicationTrust 0 instance 00000000 _applicationTrust
79118234 4000193 40 ...ncipal.IPrincipal 0 instance 00000000 _DefaultPrincipal
79103b10 4000194 44 ...cificRemotingData 0 instance 01268688 _RemotingData
7910d6f8 4000195 48 System.EventHandler 0 instance 00000000 _processExit
7910d6f8 4000196 4c System.EventHandler 0 instance 00000000 _domainUnload
7911d508 4000197 50 ...ptionEventHandler 0 instance 0129d118 _unhandledException
790fe234 4000198 54 System.IntPtr 0 instance 1345952 _dummyField
7911d974 4000199 58 System.Int32 0 instance 0 _PrincipalPolicy
79105040 400019a 5c System.Boolean 0 instance 0 _HasSetPolicy


Need to drill down into a field of an object you just dumped?

1) We want to dump the field called _FusionStore of the AppDomain object we just dumped

2) Use the 'Value' column from the previous !do command for the field called _FusionStore
e.g.
0:009> !do 01261218
Name: System.AppDomainSetup
MethodTable: 790fbdcc
EEClass: 790fbcfc
Size: 40(0x28) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
MT Field Offset Type VT Attr Value Name
79124318 40001a3 4 System.Object[] 0 instance 01261240 _Entries
79100ed0 40001a4 20 System.Int32 0 instance 0 _LoaderOptimization
790fa4b0 40001a5 8 System.String 0 instance 00000000 _AppBase
7915cb08 40001a6 c ...DomainInitializer 0 instance 00000000 _AppDomainInitializer
79124318 40001a7 10 System.Object[] 0 instance 00000000 _AppDomainInitializerArguments
7915ce5c 40001a8 14 ...tivationArguments 0 instance 00000000 _ActivationArguments
790fa4b0 40001a9 18 System.String 0 instance 00000000 _ApplicationTrust
79124508 40001aa 1c System.Byte[] 0 instance 00000000 _ConfigurationBytes


How do I dump an array?

1) Notice the array in the previous item called _Entries? Use !da addressofobject

In this example we want to dump the field '_Entries':

e.g.
0:009> !da 01261240
Name: System.String[]
MethodTable: 79124318
EEClass: 7912488c
Size: 80(0x50) bytes
Array: Rank 1, Number of elements 16, Type CLASS
Element Methodtable: 790fa4b0
[0] 01261530
[1] 01261290
[2] null
[3] null
[4] 012612e4
[5] null
[6] null
[7] null
[8] null
[9] null
[10] null
[11] null
[12] null
[13] null
[14] null
[15] null


What is the WinDBG command to find the current managed exception?

1) !pe


What is the WinDBG command to find the current unmanaged exception?

1) .exr -1

Application seems to generate a lot of exceptions but only one is of concern to you?
You need to product a dump of the program that you can then debug with WinDBG. You'll need to launch the application and then attach the debugger. In a program I was debugging earlier, I was interested in System.NullReferenceException.

1) Create a directory on your hard disk called C:\Dumps

2) Create the following file called trackclr.cfg in the same directory as windbg.EXE (on my machine that was "C:\Program Files\Debugging Tools For Windows"):
<ADPlus>
<Settings>
<RunMode>CRASH</RunMode>
<Option>Quiet</Option>
</Settings>
<Exceptions>
<Option>NoDumpOnFirstChance</Option>
<Config>
<Code>clr</Code>
<Actions1>Void</Actions1>
<CustomActions1>
.loadby sos mscorwks;
!stoponexception System.NullReferenceException 4;
.if(@$t4==1)
{
.dump /ma /u c:\\Dumps\\NullRef.dmp
}
</CustomActions1>
<ReturnAction1>gn</ReturnAction1>
</Config>
</Exceptions>
</ADPlus>


3) Run the target program (the one you want to debug)

4) Attach the debugger via the following command line:
adplus -c trackclr.cfg

5) Crash dump will appear in C:\Dumps

6) Get some very useful information from the dump via the following command: !analyze -v

7) From the previous command (!analyze -v), grab the IP address e.g. FAULTING_IP: +957984d and then do a !U addressgoeshere e.g. !U 957984d
This will show you which method and the JIT'd instructions - keep an eye out for the IP itself which is highlighted by WinDBG with chevrons: >>> e.g.
>>> 0957984d 8b01 mov eax,dword ptr [ecx]

If you're getting a NullReferenceException and it's on an assembly instruction as seen in the example, then examine the machine registers with the command: r
e.g.
0:009> r
eax=01e05ac4 ebx=00000000 ecx=00000000 edx=00000096 esi=01e05ad0 edi=01dfd144
eip=0957984d esp=07dbd718 ebp=00000096 iopl=0 nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010297
0957984d 8b01 mov eax,dword ptr [ecx] ds:0023:00000000=????????

In this example, ecx contains all zeros and yet the machine instruction mov eax,dword ptr [ecx] is trying to dereference the zeros in ecx and assign that to eax - hence an AccessViolation which the CLR will translate to a NullReferenceException


You think there is an object of a certain type that is being accessed by more than one thread but can't be sure?

1) Produce a crash dump or debug live in WinDBG

2) Get a list of objects of that type [see above on how to do that]

3) Pick on one of the objects and see which threads have a reference to it with !gcroot

This example finds all the threads that have a reference to the AppDomain instance whose address we dumped earlier:

e.g.
0:009> !gcroot 012611b4
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
Scan Thread 0 OSTHread 10a8
ESP:12f010:Root:012687c4(System.Runtime.Remoting.Contexts.Context)->
012611b4(System.AppDomain)
ESP:12f318:Root:0129cf3c(NUnit.ConsoleRunner.ConsoleUi+EventCollector)->
0129d6b0(System.Runtime.Remoting.ServerIdentity)->
012687c4(System.Runtime.Remoting.Contexts.Context)
ESP:12f328:Root:01267e48(NUnit.Util.TestDomain)->
0129cf3c(NUnit.ConsoleRunner.ConsoleUi+EventCollector)->
0129d6b0(System.Runtime.Remoting.ServerIdentity)
ESP:12f32c:Root:0129cf3c(NUnit.ConsoleRunner.ConsoleUi+EventCollector)->
012611b4(System.AppDomain)
ESP:12f364:Root:0129d138(NUnit.Util.TestExceptionHandler)->
0129d118(System.UnhandledExceptionEventHandler)->
01267e48(NUnit.Util.TestDomain)->
0129cf3c(NUnit.ConsoleRunner.ConsoleUi+EventCollector)
ESP:12f36c:Root:0129cf3c(NUnit.ConsoleRunner.ConsoleUi+EventCollector)->
012611b4(System.AppDomain)
ESP:12f370:Root:01267e48(NUnit.Util.TestDomain)->
012611b4(System.AppDomain)
ESP:12f38c:Root:01267e48(NUnit.Util.TestDomain)->
012611b4(System.AppDomain)
ESP:12f394:Root:0129cf3c(NUnit.ConsoleRunner.ConsoleUi+EventCollector)->
012611b4(System.AppDomain)
ESP:12f3d8:Root:0129cf3c(NUnit.ConsoleRunner.ConsoleUi+EventCollector)->
012611b4(System.AppDomain)
ESP:12f3e0:Root:01267e48(NUnit.Util.TestDomain)->
012611b4(System.AppDomain)
ESP:12f3e4:Root:01267e48(NUnit.Util.TestDomain)->
012611b4(System.AppDomain)
Scan Thread 2 OSTHread a74
Scan Thread 4 OSTHread 938
Scan Thread 5 OSTHread 10d0
Scan Thread 8 OSTHread 258
Scan Thread 9 OSTHread ca8
Scan Thread 11 OSTHread 7f4
Scan Thread 14 OSTHread 16a8
Scan Thread 18 OSTHread 31c
Scan Thread 19 OSTHread b84
Scan Thread 20 OSTHread a18
Scan Thread 21 OSTHread 86c
Scan Thread 22 OSTHread 10c0
Scan Thread 23 OSTHread 11c4
Scan Thread 24 OSTHread 1348
Scan Thread 25 OSTHread 3e4
Scan Thread 26 OSTHread de8
DOMAIN(001489A0):HANDLE(WeakLn):8e10fc:Root:012687c4(System.Runtime.Remoting.Contexts.Context)->
012611b4(System.AppDomain)
DOMAIN(001489A0):HANDLE(Strong):8e1140:Root:012f59e8(System.Threading.Thread)->
012687c4(System.Runtime.Remoting.Contexts.Context)
DOMAIN(001489A0):HANDLE(Strong):8e1150:Root:0129d6b0(System.Runtime.Remoting.ServerIdentity)->
012687c4(System.Runtime.Remoting.Contexts.Context)
DOMAIN(001489A0):HANDLE(Strong):8e1154:Root:01284ed8(System.Runtime.Remoting.ServerIdentity)->
012611b4(System.AppDomain)
DOMAIN(001489A0):HANDLE(Strong):8e11fc:Root:012611b4(System.AppDomain)->
012611b4(System.AppDomain)
DOMAIN(001489A0):HANDLE(WeakSh):8e12d0:Root:012f59e8(System.Threading.Thread)->
012611b4(System.AppDomain)
DOMAIN(001489A0):HANDLE(Pinned):8e13fc:Root:02261010(System.Object[])->
012687c4(System.Runtime.Remoting.Contexts.Context)


Want to find out which threads are running in your application?

1) !threads

e.g.
0:009> !threads
ThreadCount: 17
UnstartedThread: 0
BackgroundThread: 15
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
0 1 10a8 00150b50 2006020 Enabled 00000000:00000000 00198f90 0 STA
2 2 a74 0015d290 b220 Enabled 00000000:00000000 001489a0 0 MTA (Finalizer)
4 3 938 0018d488 1220 Enabled 00000000:00000000 001489a0 0 Ukn
5 4 10d0 001ab710 80a220 Enabled 00000000:00000000 001489a0 0 MTA (Threadpool Completion Port)
8 5 258 06fa3c90 200b020 Enabled 00000000:00000000 00198f90 0 MTA
9 6 ca8 06fa48f8 7220 Disabled 00000000:00000000 00198f90 0 STA
11 7 7f4 06fdce90 b220 Enabled 00000000:00000000 00198f90 0 MTA
14 8 16a8 001bbcc0 180b220 Enabled 00000000:00000000 001489a0 0 MTA (Threadpool Worker)
18 9 31c 07029e80 200b220 Enabled 00000000:00000000 00198f90 0 MTA
19 a b84 0a885508 200b220 Enabled 00000000:00000000 00198f90 0 MTA
20 b a18 0a8b0208 200b220 Enabled 00000000:00000000 00198f90 0 MTA
21 c 86c 0a8b4160 200b220 Enabled 00000000:00000000 00198f90 0 MTA
22 d 10c0 0a8b49e8 200b220 Enabled 00000000:00000000 00198f90 0 MTA
23 e 11c4 0a8b5080 200b220 Enabled 00000000:00000000 00198f90 0 MTA
24 f 1348 0a8b6b40 200b220 Enabled 00000000:00000000 00198f90 0 MTA
25 10 3e4 0a8ba158 200b220 Enabled 00000000:00000000 00198f90 0 MTA
26 11 de8 0a8d0a58 200b220 Enabled 00000000:00000000 00198f90 0 MTA


Want to see the managed stack of a particular thread?

First you need to context switch to the thread of interest in WinDBG, then dump its stack

1) Context switch to a thread with this command: ~threadnumber s
e.g. from previous !threads call, pick a thread ID from the far left column from the !threads command output
~20 s

2) Dump the managed call stack for the thread currently switched in:
!clrstack


Want to see parameters and autos as part of the stack dump?

1) Use !clrstack -a


Application is hung and you want to get a crash dump?

1) adplus -hang -pn foo.exe