Failed to load data access DLL, 0x80004005 - hm

Me and some colleagues of mine recently all stumbled over the following error when analyzing .NET minidumps:
I opened a the minidump and typed !CLRStack. What I got was:

Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks___.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.

OK, so I followed the step 1 through 4...
ad 1) I'm using version 6.7.5.0 for good reasons
ad 2) Don't know what that means...
ad 3) Why should a dll be in my symbol path?!?
ad 4) The architecture is x86_32 on both machines

So I tried .cordll -ve -u -l

0:000> .cordll -ve -u -l
CLRDLL: Unknown processor type in image C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
CLR DLL status: No load attempts

The following sentence finally helped me:

If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.

So I executed:

0:000> lmv m mscorwks
start end module name
79e70000 7a3d6000 mscorwks T (pdb symbols) C:\windbg\symbols\mscorwks.pdb\6D3E0DE91A284256A48A60718DC9CDEB2\mscorwks.pdb
Loaded symbol image file: mscorwks.dll
Image path: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
Image name: mscorwks.dll
Timestamp: Fri Apr 13 09:15:54 2007 (461F2E2A)
CheckSum: 005635C7
ImageSize: 00566000
File version: 2.0.50727.832
Product version: 2.0.50727.832
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0

And...

0:000> .exepath+ C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\

Finally...

0:000> .reload
...................................................................................................................................................................................................................................................
CLRDLL: Loaded DLL C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll

And my !CLRStack worked ;-)

SOSAssist >> Toolbox

Tess mentioned in her recent post '.NET Finalizer Memory Leak: Debugging with sos.dll in Visual Studio' a tool named SOSAssis written by Ingo Rammer. I was curious, clicked on the link and looked at the screenshots.
I couldn't believe that there is a tool out there that eases .NET production debugging in such a way. If you ever walked down the objects tree via sos commands you will get tears in your eyes when you look at the Stack Objects window!!!


[click to enlarge...]

I googled for sosassist and rammer and just got 4 relevant hits?!

So I needed to update on this after my very first look at the tool as I'm totally amazed! This thing should be in the toolbox of every .NET developer.

Next I will follow up on this tool soon with more experiences.

Windbg version 6.8.4.0 is out (I'm still keeping 6.7.5.0)

Microsoft released a new version of Debugging Tools for Windows (containing Windbg). The Highlights in Version 6.8.4.0 are very roughly described. Still the integrated .NET Support that came up with version 6.7.5.0 by accident (see Pat Styles [MSFT] comment in this post) is not available.
Luckily different versions of windbg can be installed side by side (via XCopy) and I'm keeping my 6.7.5.0 for managed debugging as it works pretty well.

Symbol table tab completion

I just stumbled over a feature which is either new or I didn't realize before.
It is support of tab completion in the command window.

When I'm going to set a breakpoint I normall execute an 'x' command with a good guess about the signature:
E.g. :

x kernel32!*ToWideChar

Giving:
7c809bf8 kernel32!MultiByteToWideChar =

Then I can set the breakpoint either by:
bp 7c809bf8
or
bp kernel32!MultiByteToWideChar

Now by accident (I'm used to that from the console) I wrote just a few letters and pressed the tab key and surprise windbg did the rest ;-)

Try the following:

1.) bp ker<2*tab> gives bp kernel32!
2.) bp kernel32!Mul gives bp kernel32!MultiByteToWideChar
3.) and the breakpoint is set

Those little enhancements can significantly make your life easier.

I observed it with windbg 6.7.5.1. Please tell me if it was there before and I'm telling old stories.

Cheers,
Volker

...

First look at windbg 6.7.5.1 disappointing

As Dmitry just mentioned a new version of windbg, I downloaded and installed it. I was curious, if the new version will provide better support for managed dumps as i could see big improvements in the last version.

So I loaded my standard managed crash dump file and...
where version 6.7.5.0 delivered nicely the stack by 'k':

0:000> k *** Stack trace for last set context - .thread/.cxr resets itChildEBP RetAddr 0012f094 00db0581 Demo1!Demo1._FormDemo1.ItsNorMe()+0x44 [E:\My Projects\TechChannel\Demo1\Demo1\Form1.cs @ 36]0012f094 00db054d Demo1!Demo1._FormDemo1.ItsNeitherMe()+0x19 [E:\My Projects\TechChannel\Demo1\Demo1\Form1.cs @ 28]0012f094 7b060a6b Demo1!Demo1._FormDemo1.button1_Click(System.Object, System.EventArgs)+0x1d [E:\My Projects\TechChannel\Demo1\Demo1\Form1.cs @ 23]0012f094 7b105379 System_Windows_Forms_ni!System.Windows.Forms.Control.OnClick(System.EventArgs)+0x570012f094 7b10547f System_Windows_Forms_ni!System.Windows.Forms.Button.OnClick(System.EventArgs)+0x490012f094 7b0d02d2 System_Windows_Forms_ni!System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventArgs)+0xc30012f094 7b072c74 System_Windows_Forms_ni!System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32)+0xf20012f100 7b0815a6 System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)+0x5440012f13c 7b0814c3 System_Windows_Forms_ni!System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message ByRef)+0xce0012f19c 7b07a72d System_Windows_Forms_ni!System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)+0x2b0012f19c 7b07a706 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)+0xd0012f19c 7b07a515 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0xd60012f19c 00342124 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0x75WARNING: Frame IP not in any known module. Following frames may be wrong.0012f1c0 7e418734 0x3421240012f1ec 7e418816 user32!InternalCallWinProc+0x280012f254 7e4189cd user32!UserCallWinProcCheckWow+0x1500012f2b4 7e418a10 user32!DispatchMessageWorker+0x3060012f2c4 00f20e4e user32!DispatchMessageW+0xf0012f2e0 7b084766 CLRStub[StubLinkStub]@f20e4e0012f398 7b08432d System_Windows_Forms_ni!System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)+0x2ea

I now get this with the new version:

0:000> k *** Stack trace for last set context - .thread/.cxr resets itChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong.0012f094 7b072c74 0xdb05dc0012f100 7b0815a6 System_Windows_Forms_ni+0xa2c740012f13c 7b0814c3 System_Windows_Forms_ni+0xb15a60012f1c0 7e418734 System_Windows_Forms_ni+0xb14c30012f1ec 7e418816 user32!InternalCallWinProc+0x280012f254 7e4189cd user32!UserCallWinProcCheckWow+0x1500012f2b4 7e418a10 user32!DispatchMessageWorker+0x3060012f2c4 00f20e4e user32!DispatchMessageW+0xf0012f2e0 7b084766 0xf20e4e0012f398 7b08432d System_Windows_Forms_ni+0xb47660012f408 7b08416b System_Windows_Forms_ni+0xb432d0012f438 7b0c69fe System_Windows_Forms_ni+0xb416b0012f480 79e88f63 System_Windows_Forms_ni+0xf69fe0012f490 79e88ee4 mscorwks!CallDescrWorker+0x330012f510 79e88e31 mscorwks!CallDescrWorkerWithHandler+0xa30012f650 79e88d19 mscorwks!MethodDesc::CallDescr+0x19c0012f668 79e88cf6 mscorwks!MethodDesc::CallTargetWorker+0x200012f67c 79f084b0 mscorwks!MethodDescCallSite::Call+0x180012f7e0 79f082a9 mscorwks!ClassLoader::RunMain+0x2200012fa48 79f0817e mscorwks!Assembly::ExecuteMainMethod+0xa6

Needless to say that the 'Calls' window does show the same and there is no click on a frame that brings to directly to the sources :-(

'!DumpStack -ee' still works, but this didn't change...

Resume: I will try the see the new improvements but I will not uninstall 6.7.5.0!

Scan the full process memory for a pattern

Very often I need to scan the process memory for a specific pattern.
This can be either a pointer or a string or whatever and I want to find out, which other memory references this pointer or pattern.

Simply type ''s -d 0x00000000 L?0xffffffff ' to find a referenced pointer on a x32 architecture.

E.g.:

0:000> s -d 0x00000000 L?0xffffffff 30c5bf9c
0012b2b0 30c5bf9c 00000000 00000000 00000000 ...0............
0012b2f8 30c5bf9c 9955d404 0badf00d 3e4d1f74 ...0..U.....t.M>
0012b340 30c5bf9c 3e4d1f70 9955d450 0badf00d ...0p.M>P.U.....
0012b374 30c5bf9c 3e4d1f70 9955d49c 00000001 ...0p.M>..U.....
3e4d1f7c 30c5bf9c 00000000 00000000 00000001 ...0............
3e4d1f90 30c5bf9c 00000000 00000000 00000000 ...0............
3e4d1fd0 30c5bf9c 30c5bf9c 00000000 00000001 ...0...0........
3e4d1fd4 30c5bf9c 00000000 00000001 33522fc0 ...0........./R3

The first column lists the locations that matched the pattern.

For more information refer to windbg online help: s (Search Memory)

New must-have Windbg extension SOSEX

John Robbins latest blog post pointed me to a new Windbg extension SOSEX written by Steve Johnson. This extension greatly simplifies many tasks that are tedious to achive with original SOS extension provided by Microsoft.

Root Out Elusive Production Bugs with These Effective Techniques

Reading Matt Adams Blog latest Post brought me to an interesting article called "Root Out Elusive Production Bugs with These Effective Techniques" which I would suggest as possible starting point getting familiar with windbg.

Contents:

Debugging Tools for Windows
Using ADPlus
Debugging Symbols
First-Chance Exceptions
Unmanaged First-Chance Exceptions
Managed First-Chance Exception
Unmanaged Thread Executing Endlessly
Managed Thread Executing Endlessly
Deadlocks
Unmanaged Deadlock Application
Managed Deadlock Application
Crashing
Conclusion

New Debugging Blog hosted by the Microsoft Critical Problem Resolution (CPR) Platforms Team

Google Alerts on keyword 'windbg' delivered me an interesting new blog hosted by the Microsoft Critical Problem Resolution (CPR) Platforms Team. Especially the article 'This button doesn’t do anything!' got my interest as I needed to do nearly the same thing some days ago. This will definitely go onto by blog roll.

Getting VB6 Err Object from a dump

Once again (sigh) looking at vb6 crash dumps I found this very interesting article from Matt Adamson about Visual Basic Production Debugging. He did a great job reversing data structures used by VB6 error handling. When you need to get the VB6 Err Object information from a crash dump you should read it!

John Robbins blogged me!

John Robbins, the author of the must read book 'Debugging .NET 2.0 Applications', blogged me in his latest post :-)

I was very excited to read, that he considered my blog as 'excellent'. But I must say, that I have a different understanding about this adjective.

First I must say that Johns book is 'excellent' and every developer should read it, beacuse it's not just about fixing what is already broken, it's very focused on avoiding errors in the first place.

Secondly, if you want to read an 'excellent' blog about windbg goto Tess. This blog is 'excellent'.

Finally this might be a matter of culture. I think my blog is something away from 'excellent', but I hope some day I get near to it. I'm a european and we seem to have a different scale ;-)

Creating and analyzing minidumps in .NET production applications (continued)

In my previous post I delivered a small receipt how to create and analyze minidumps from .NET applications. Then I discovered that version 6.7.5.0 of windbg shows some nice integrations of sos. Now, I was curious if the same integration also works for crash dumps and it does:
Opening a crash dump and processing .ecxr brings me directly to the source code line, without much peeking and poking:

Only thing, I discovered: When I did a 'analyze -v' windbg crashed - reliably!

Oh Borland, oh Microsoft, oh boy

What a wonderful world would it be, if you could develop your applications just on a single frameworks. Reality looks different. Me and many others need to cope with applications constructed from MSVC + or - MFC, VB6, Borland C++ and/or Delphi and other frameworks.
At first it looks like there is a more or less seamless integration possible. Later you discover all those little nasty glitches. In this post I will cover one of those glitches I really didn't know how to solve it but were I finally found a strange solution which I want to share.

To demonstrate the problem create a small C# application like this:

namespace CLRvsBorland
{
static class Program
{
[DllImport("kernel32.dll", CharSet = CharSet.Auto)]
static extern IntPtr LoadLibrary(string lpFileName);
static void Main()
{
IntPtr res = LoadLibrary(@"CC3250MT.DLL");
//try
//{
//// Throw an exception..
//throw new Exception();
//}
//catch
//{
//// .. and just catch it
//}

Double a1 = Double.NaN;
MessageBox.Show("Doh" + a1.CompareTo(Double.NaN).ToString());
}
}
}

When you start it in the debugger, you will get the following exception at instruction 'a1.CompareTo(Double.NaN)':

System.ArithmeticException was unhandled
Message="Overflow or underflow in the arithmetic operation."
Source="mscorlib"
StackTrace:
at System.Double.CompareTo(Double value)
[...]
at System.Threading.ThreadHelper.ThreadStart()


Now remove the comments - and surprise - it works!

I don't know, how it works. I assume it has to do with the order of jit-ing.
Explanations are highly wellcome ;-)

Update: Explanation found! Please have a look into the first 2 comments.

Calling functions and methods

Great article from Raymond Chen about calling functions and methods in a windbg debugging session. Must read!

http://blogs.msdn.com/oldnewthing/archive/2007/04/27/2292037.aspx

Windbg 6.7.5.0 released

A long awaited new release of windbg is available.

http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx

The greates thing, I observed so far, is the embedded support of SOS which I didn't see in the new features list. There is no direct need to call !CLRStack or !DumpStack as the managed (along with the unmanaged) stack get's listed in the 'Calls' Window ;-)

And I couln't believe - a click on the frame brought me directly to the source code :-) :-) :-)

Only this thing that didn't work is the locals window. Then I get the message:

Integrated managed debugging does not support enumeration of local variables.See http://dbg/managed.htm for more details.

But in microsoft.public.windbg no one (at Microsoft) wanted to comment on this:

http://groups.google.com/group/microsoft.public.windbg/browse_frm/thread/85afed79ee62854f/#

Anyways - I like the new windbg.

Creating and analyzing minidumps in .NET production applications

Preface:
Identifying the source of an error in production applications can be hard task. There are simple errors that can be reproduced with a 'steps to repeat' receipt. Other errors, that are logged as one time only or sporadic are much more difficult. In order to adress them, you can spent tons of time to find a way to create the bug and possibly you will never succeed. In deep this does not mean that this defect does not exist! It is just a matter of probability until it reoccurs. So it is aimed to catch that thing at the first occurrence and gather as much information as reasonable.

How to setup:
First you must make sure your build creates pdbs for all release binaries. Those need to be checked in or sent to a symbol server.
Next thing to do, is to catch all unhandled exceptions:

AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(AppDomainUnhandledException); (*)
Application.ThreadException += new System.Threading.ThreadExceptionEventHandler(Application_ThreadException);

(*) needs
[SecurityPermission(SecurityAction.Demand,ControlAppDomain=true)]

In the exception handler, which is ideally ONE function, you need to implement the user notification and the dump generation:

private static void HandleException(Exception ex)
{
if (ex == null)
return;
// ExceptionPolicy.HandleException(ex, "Default Policy");
MessageBox.Show("An unhandled exception occurred, and the application is terminating. For more information, see your Application event log.");
CCLRDump.Dump();
Application.Exit();
}

You will notice two specialities in this function.
One is the commented 'ExceptionPolicy.HandleException(ex, "Default Policy");'. This refers to Exception Handling Application Block which can be greatly combined with this mechanism.
The other thing is "CCLRDump.Dump();" This is a wrapper class around ClrDump (© Oleg Starodumov, 2004 - 2006 ).

The implementation is fairly easy :
[C#]


[Flags]
enum MINIDUMP_TYPE {
MiniDumpNormal = 0x00000000,
MiniDumpWithDataSegs = 0x00000001,
MiniDumpWithFullMemory = 0x00000002,
MiniDumpWithHandleData = 0x00000004,
MiniDumpFilterMemory = 0x00000008,
MiniDumpScanMemory = 0x00000010,
MiniDumpWithUnloadedModules = 0x00000020,
MiniDumpWithIndirectlyReferencedMemory = 0x00000040,
MiniDumpFilterModulePaths = 0x00000080,
MiniDumpWithProcessThreadData = 0x00000100,
MiniDumpWithPrivateReadWriteMemory = 0x00000200,
MiniDumpWithoutOptionalData = 0x00000400,
MiniDumpWithFullMemoryInfo = 0x00000800,
MiniDumpWithThreadInfo = 0x00001000,
MiniDumpWithCodeSegs = 0x00002000,
MiniDumpWithoutManagedState = 0x00004000,
};

[...]
[DllImport("clrdump.dll", CharSet = CharSet.Unicode, SetLastError = true)]
private static extern Int32 CreateDump(Int32 ProcessId, string FileName,
Int32 DumpType, Int32 ExcThreadId, IntPtr ExtPtrs);

[...]

public static void Dump()
{
IntPtr pEP = System.Runtime.InteropServices.Marshal.GetExceptionPointers();
CreateDump(
System.Diagnostics.Process.GetCurrentProcess().Id,
@"C:\temp\test.dmp",
(Int32)MINIDUMP_TYPE.MiniDumpWithFullMemory,
AppDomain.GetCurrentThreadId(),
pEP);
}

Ok, now we have setup everything to catch all the nasty stuff, that can happen in our application.

Analyzing those MiniDumps

[please read this post, before moving on...]

In order to analyze those special minidumps some magic is needed. There are several blog posts, articles, books, etc. about how to deal with Dumps in managed and unmanaged manner. I will list some of them I found very supporting at the end of this post. As the most common question is: "Where is the source of ^%!&@ exception" I will deal with that:

1. Open the crash dump in Windbg
   
   
   



2. Make sure you have the correct symbol path (currently no symbol server as 6.6.7.5 has a bad bug with symbol loading - instead use C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\symbols\) and image path. Type .reload to get the correct symbols.
   
   




3. Load correct sos (Son of Strike): .loadby sos mscorwks
   
   0:000> .loadby sos mscorwks
   




4. !Threads gives you on overview of managed threads (with exceptions)
   
   0:000> !Threads
   ThreadCount: 2
   UnstartedThread: 0
   BackgroundThread: 1
   PendingThread: 0
   DeadThread: 0
   Hosted Runtime: no
   
   ID OSID ThreadOBJ State GC Context Domain Count APT Exception
   0 1 1224 001506c0 6020 Enabled 00000000:00000000 0014e808 0 STA System.NullReferenceException (0138fec8)
   2 2 153c 00156158 b220 Enabled 00000000:00000000 0014e808 0 MTA (Finalizer)
   




5. !pe will dump the last exception on the current thread. Unfortunately this will just give you the function name and not the source line - but with some effort, we can extract this information... For now remember the IP address of the function that threw (we need this later)
   
   0:000> !pe
   Exception object: 0138fec8
   Exception type: System.NullReferenceException
   Message: Object reference not set to an instance of an object.
   InnerException:
   StackTrace (generated):
   SP IP Function
   0012EFF0 00DB04AD Demo1._FormDemo1.ItsNorMe()
   0012F000 00DB0451 Demo1._FormDemo1.ItsNeitherMe()
   0012F008 00DB041D Demo1._FormDemo1.button1_Click(System.Object, System.EventArgs)
   0012F018 7B060A6B System.Windows.Forms.Control.OnClick(System.EventArgs)
   0012F028 7B105379 System.Windows.Forms.Button.OnClick(System.EventArgs)
   0012F034 7B10547F System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventArgs)
   0012F058 7B0D02D2 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32)
   0012F0A4 7B072C74 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
   0012F108 7B0815A6 System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message ByRef)
   0012F144 7B0814C3 System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)
   0012F14C 7B07A72D System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
   0012F150 7B07A706 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
   0012F164 7B07A515 System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
   




6. !DumpStack will give you full stack trace with all managed and unmanaged frames (pretty large...). In this stack search for "====> Exception cxr@" and remember the return adress (2nd adress from the beginning)
   
   0:000>
   0:000> !DumpStack
   OS Thread Id: 0x1224 (0)
   Current frame: ntdll!KiFastSystemCallRet
   ChildEBP RetAddr Caller,Callee
   [... lots of stuff ...]
   0012eff0 00db04ac (MethodDesc 0xa25aa8 +0x44 Demo1._FormDemo1.ItsNorMe()) ====> Exception cxr@12ed24
   [... lots of stuff ...]
   0012eff8 00db0451 (MethodDesc 0xa25aa0 +0x19 Demo1._FormDemo1.ItsNeitherMe()), calling 00a264d8
   0012f000 00db041d (MethodDesc 0xa25a98 +0x1d Demo1._FormDemo1.button1_Click(System.Object, System.EventArgs)), calling 00a264c4
   0012f00c 7b060a6b (MethodDesc 0x7b4a6598 +0x57 System.Windows.Forms.Control.OnClick(System.EventArgs))
   0012f020 7b105379 (MethodDesc 0x7b5ab788 +0x49 System.Windows.Forms.Button.OnClick(System.EventArgs)), calling (MethodDesc 0x7b4a6598 +0 System.Windows.Forms.Control.OnClick(System.EventArgs))
   0012f02c 7b10547f (MethodDesc 0x7b5ab798 +0xc3 System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventArgs))
   0012f050 7b0d02d2 (MethodDesc 0x7b5a59d8 +0xf2 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32))
   0012f094 7b072c74 (MethodDesc 0x7b5a5a50 +0x544 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)), calling (MethodDesc 0x7b5a59d8 +0 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32))
   0012f100 7b0815a6 (MethodDesc 0x7b5aba60 +0xce System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message ByRef)), calling (MethodDesc 0x7b5a5a50 +0 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef))
   0012f128 7b07a608 (MethodDesc 0x7b5af1c0 +0x48 System.Windows.Forms.Message.Create(IntPtr, Int32, IntPtr, IntPtr)), calling (JitHelp: CORINFO_HELP_GETSHARED_NONGCSTATIC_BASE_NOCTOR)
   0012f13c 7b0814c3 (MethodDesc 0x7b5ab7c0 +0x2b System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)), calling (MethodDesc 0x7b5aba60 +0 System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message ByRef))
   0012f144 7b07a72d (MethodDesc 0x7b5a8168 +0xd System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef))
   0012f148 7b07a706 (MethodDesc 0x7b5a8180 +0xd6 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)), calling 00a3704e
   0012f15c 7b07a515 (MethodDesc 0x7b4a7d60 +0x75 System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr))
   [... lots of stuff ...]
   0012ffb0 79011b5f mscoree!_CorExeMain+0x2c
   0012ffc0 7c816fd7 kernel32!RegisterWaitForInputIdle+0x49
   




7. Now lets unassamble the function that threw (!u ).
   You'll get an machine code listing of that function. Now simply search for the second address to remember and there it is.All you need to do is to synchronise machine code with the source code, which is much easier with .NET code compared to C++ code.
   
   
   
   0:000> !u 00DB04AD
   Normal JIT generated code
   Demo1._FormDemo1.ItsNorMe()
   Begin 00db0468, size 51
   
   
   private void ItsNorMe()
   {
   
   00db0468 57 push edi
   00db0469 56 push esi
   00db046a 50 push eax
   00db046b 890c24 mov dword ptr [esp],ecx
   00db046e 833dc82da20000 cmp dword ptr ds:[0A22DC8h],0
   00db0475 7405 je 00db047c
   *** WARNING: Unable to verify checksum for mscorlib.ni.dll
   *** ERROR: Module load completed but symbols could not be loaded for mscorlib.ni.dll
   00db0477 e8821e2e79 call mscorlib_ni+0x221e82 (792e1e82) (mscorlib_ni)
   00db047c 33f6 xor esi,esi
   00db047e 90 nop
   00db047f b9fcf91979 mov ecx,offset mscorlib_ni+0xdf9fc (7919f9fc)
   00db0484 e8931bc6ff call 00a1201c (JitHelp: CORINFO_HELP_NEWSFAST)
   00db0489 8bf8 mov edi,eax
   00db048b 8bcf mov ecx,edi
   
   List l = new List();
   
   00db048d e83ea97a78 call mscorlib_ni+0x49add0 (7955add0) (System.Collections.Generic.List`1[[System.Int32, mscorlib]]..ctor(), mdToken: 0600194d)
   00db0492 8bf7 mov esi,edi
   00db0494 8bce mov ecx,esi
   00db0496 ba01000000 mov edx,1
   00db049b 3909 cmp dword ptr [ecx],ecx
   
   l.Add(1);
   
   00db049d e8aec77a78 call mscorlib_ni+0x49cc50 (7955cc50) (System.Collections.Generic.List`1[[System.Int32, mscorlib]].Add(Int32), mdToken: 0600195e)
   00db04a2 90 nop
   
   l = null;
   
   00db04a3 33f6 xor esi,esi
   00db04a5 8bce mov ecx,esi
   00db04a7 ba02000000 mov edx,2
   00db04ac 3909 cmp dword ptr [ecx],ecx
   
   l.Add(2);
   
   00db04ae e89dc77a78 call mscorlib_ni+0x49cc50 (7955cc50) (System.Collections.Generic.List`1[[System.Int32, mscorlib]].Add(Int32), mdToken: 0600195e)
   00db04b3 90 nop
   00db04b4 90 nop
   00db04b5 59 pop ecx
   00db04b6 5e pop esi
   00db04b7 5f pop edi
   00db04b8 c3 ret
   
   
   

References:
Debugging Microsoft .NET 2.0 Applications, John Robbins
Production Debugging for .NET Framework Applications
SOS: It's Not Just an ABBA Song Anymore
A reading list for debugging, .NET, CLR, WinDBG etc
SOS Debugging Extension Online Reference
.Net exceptions - Tracking down where in the code the exceptions occurred
Back to Basics - How do I get the memory dumps in the first place? And what is SOS.dll?
A Hang Scenario, Locks and Critical Sections
.NET Hang Debugging Walkthrough
Some new SOS functions

SOS Debugging with the CLR (Part 1)

A very impressive demonstration on what can be achived with SOS. It also takes care of when to use windbg/sos and when it's just overkill.
Must read:

Jason Zander's WebLog (General Manager, .NET Framework - DevFX)

Reflector for .NET

Must have tool for .NET debugging and reverse engineering:

Reflector is the class browser, explorer, analyzer and documentation viewer for .NET. Reflector allows to easily view, navigate, search, decompile and analyze .NET assemblies in C#, Visual Basic and IL.

WinDbg tips and tricks: triple dereference

A very helpful post from Dmitry Vostokov (Crash Dump Analysis) when you want to walk pointer to pointer structures:

WinDbg tips and tricks: triple dereference

Using .NET components in an VB6 host

In order to allow correct objects cleanup on process shutdown of a VB6 executable that uses .NET components via CCWs and RCWs you must explicitly shut down .NET runtime in the unload of the project. Otherwise you can observe nasty crahses.

The easiest way to achive this is using mscoree.CorRuntimeHost.
Simply create an instance of this server an call Stop when your project unloads. With Start you can also control, when the runtime is loaded.

You need to reference mscoree.tlb and mscoree.dll needs to be registered via regasm.

How to load the correct sos.dll

In order to load the correct sos.dll for .NET debugging into WinDbg you can either do this by .load with the full path to the wanted version of sos or you can type the following:

.loadby sos mscorwks