Friday, December 04, 2009

Displaying dynamically allocated c arrays

I just analyzed a crash dump were I needed to investigate a dynamically allocated a array.

We have a declaration:

CString **ids;

the allocation

ids = new CString*[size];

Given a size of 8 you can dump the content of the string array with dt -a<size> <address> <type>:

0:000> dv ids
            ids = 0x06314ec0

0:000> dt -a8 0x06314ec0 CString*
[0] @ 06314ec0
   +0x000 m_pszData        : 0x062b7d58  "string 1"

[1] @ 06314ec4
   +0x000 m_pszData        : 0x785039c0  ""

[2] @ 06314ec8
   +0x000 m_pszData        : 0x06373958  "string 2"

[3] @ 06314ecc
   +0x000 m_pszData        : 0x06301dc8  "string 3"

[4] @ 06314ed0
   +0x000 m_pszData        : 0x785039c0  ""

[5] @ 06314ed4
   +0x000 m_pszData        : 0x785039c0  ""

[6] @ 06314ed8
   +0x000 m_pszData        : 0x785039c0  ""

[7] @ 06314edc
   +0x000 m_pszData        : 0x785039c0  ""

Friday, February 20, 2009

Windbg is out

There's no support for integrated managed debugging. Here's the link:


Highlights in Version
The following changes have been made to Debugging Tools for Windows:

• Numerous bug fixes and documentation updates
• Numerous updates to improve 1394 debugging (see relnotes.txt for details)
• support using “.process /p …” in kd -kl, so you can see user mode memory in the appropriate process context (which means user mode stacks, !peb, etc.)

Friday, February 13, 2009

Another goody: uf /c /D and the peb lock

Remember my previous post?

I was hunting a dead lock and found the good old again worth keeping.

But I didn't write about the dead lock cause itself. Now here it comes:


The main thread was waiting for the peb lock:

0:00> ~0 kb
ChildEBP RetAddr Args to Child
0012c72c 7c90e9c0 7c91901b 0000056c 00000000 ntdll!KiFastSystemCallRet
0012c730 7c91901b 0000056c 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
0012c7b8 7c90104b 0097e4c0 7c910945 7c97e4c0 ntdll!RtlpWaitForCriticalSection+0x132
0012c7c0 7c910945 7c97e4c0 ffffffff 00000000 ntdll!RtlEnterCriticalSection+0x46
0012c800 7c80e2cf ffffffff 7c80e038 00000000 ntdll!RtlAcquirePebLock+0x28
0012c868 7c80e00d 7c80e038 00020690 00000000 kernel32!BasepComputeProcessPath+0x5f
0012c8a8 7c801b83 00000000 00000000 0012d7a0 kernel32!BaseComputeProcessDllPath+0xe3
0012c908 7c801d6e 7ffdfc00 00000000 00000000 kernel32!LoadLibraryExW+0x12f
0012c91c 7c801da4 06057954 00000000 00000000 kernel32!LoadLibraryExA+0x1f
0012c938 060b4823 06057954 7c911993 03f41ea8 kernel32!LoadLibraryA+0x94


This lock was owned by managed thread number 6:

0:006> k 100
ChildEBP RetAddr 
0522f33c 7c9141e4 ntdll!RtlGetFullPathName_Ustr+0x653
0522f364 7c80b86c ntdll!RtlGetFullPathName_U+0x33
0522f37c 015da1c3 kernel32!GetFullPathNameW+0x1a
0522f3a4 79364aef CLRStub[StubLinkStub]@15da1c3
0522f8f0 793644ae mscorlib_ni!System.IO.Path.NormalizePathFast(System.String, Boolean)+0x637
017d2738 793821b0 mscorlib_ni!System.IO.Path.GetFullPathInternal(System.String)+0x2e
017d2738 793820f6 mscorlib_ni!System.Security.Util.StringExpressionSet.CanonicalizePath(System.String, Boolean)+0x90
017d2738 79381dbe mscorlib_ni!System.Security.Util.StringExpressionSet.CreateListFromExpressions(System.String[], Boolean)+0x14a
017d2738 79381d16 mscorlib_ni!System.Security.Permissions.FileIOPermission.AddPathList(System.Security.Permissions.FileIOPermissionAccess, System.Security.AccessControl.AccessControlActions, System.String[], Boolean, Boolean, Boolean)+0x6a
0178b9ec 793a2014 mscorlib_ni!System.Security.Permissions.FileIOPermission..ctor(System.Security.Permissions.FileIOPermissionAccess, System.String)+0x42
0522f978 04189669 mscorlib_ni!System.IO.FileSystemInfo.get_FullName()+0x58
0177cdbc 041892b2 XYZ.DirectoryCleanup.PopulateAllFilesList(System.Collections.Generic.List`1<System.IO.FileInfo>, System.IO.DirectoryInfo)+0x69
0522fa00 0418916f XYZ.DirectoryCleanup.CleanupThread()+0xaa
0522fa30 79407caa XYZ.DirectoryCleanup.CleanupThreadProc(System.Object)+0x67
0179439c 79373ecd mscorlib_ni!System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Object)+0x1a
0179439c 79407e18 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x81
0522fa60 79407d90 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)+0x50
1f7814b0 79e7c74b mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)+0x60
0522fa84 79e7c6cc mscorwks!CallDescrWorker+0x33
0522fb04 79f00eca mscorwks!CallDescrWorkerWithHandler+0xa3
0522fb24 79f00e75 mscorwks!DispatchCallBody+0x1e
0522fb88 79f00f03 mscorwks!DispatchCallDebuggerWrapper+0x3d
0522fbbc 79f6e39d mscorwks!DispatchCallNoEH+0x51
0522fc1c 79ef3207 mscorwks!QueueUserWorkItemManagedCallback+0x6c
0522fc30 79ef31a3 mscorwks!Thread::DoADCallBack+0x32a
0522fcc4 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0522fd00 79ef4826 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
0522fd28 79ef48cf mscorwks!Thread::ShouldChangeAbortToUnload+0x33e
0522fd40 79f6e2d8 mscorwks!ManagedThreadBase::ThreadPool+0x13
0522fda8 79f0202a mscorwks!ManagedPerAppDomainTPCount::DispatchWorkItem+0xdb
0522fdbc 79f021a0 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf
0522fe14 79f95a2e mscorwks!ThreadpoolMgr::WorkerThreadStart+0x223
0522ffb4 7c80b683 mscorwks!Thread::intermediateThreadProc+0x49
0522ffec 00000000 kernel32!BaseThreadStart+0x37

Now my question was: Does kernel32!GetFullPathNameW acquire a peb lock?

The short answer is yes (in Windows XP but not in Vista).

Here's the easy way to find out with uf /c /D (/D creates linked callee names for navigation of the call graph. Do you can click on the hyperlinks ;-) ):

0:006> uf /c /D kernel32!GetFullPathNameW
kernel32!GetFullPathNameW (7c80b852)
  kernel32!GetFullPathNameW+0x14 (7c80b866):
call to ntdll!RtlGetFullPathName_U (7c9141b1)

0:006> uf /c /D 0x7c9141b1
ntdll!RtlGetFullPathName_U (7c9141b1)
  ntdll!RtlGetFullPathName_U+0xe (7c9141bf):
    call to ntdll!RtlInitUnicodeStringEx (7c9103a5)
  ntdll!RtlGetFullPathName_U+0x2e (7c9141df):
    call to ntdll!RtlGetFullPathName_Ustr (7c913b67)

0:006> uf /c /D 0x7c913b67
ntdll!RtlGetFullPathName_Ustr (7c913b67)
  ntdll!RtlGetFullPathName_Ustr+0xa (7c913b71):
    call to ntdll!_SEH_prolog (7c90edc2)
  ntdll!RtlGetFullPathName_Ustr+0xbe (7c913c19):
    call to ntdll!RtlIsDosDeviceName_Ustr (7c9139ed)
  ntdll!RtlGetFullPathName_Ustr+0x176 (7c913c59):
    call to ntdll!RtlDetermineDosPathNameType_U (7c91399f)
  ntdll!RtlGetFullPathName_Ustr+0x180 (7c913c63):
    call to ntdll!RtlAcquirePebLock (7c91091d) <== Here you go


Doing the same on Vista shows no PebLock

0:003> uf /c /D kernel32!GetFullPathNameW
KERNEL32!GetFullPathNameW (77c60a0e)
  KERNEL32!GetFullPathNameW+0x14 (77c60a22):
    call to ntdll!RtlGetFullPathName_U (77b0ac1f)
0:003> uf /c /D 0x77b0ac1f
ntdll!RtlGetFullPathName_U (77b0ac1f)
  ntdll!RtlGetFullPathName_U+0xe (77b0ac2d):
    call to ntdll!RtlInitUnicodeStringEx (77b09b9f)
  ntdll!RtlGetFullPathName_U+0x2e (77b0ac4d):
    call to ntdll!RtlGetFullPathName_Ustr (77b0a01f)
0:003> uf /c /D 0x77b0a01f
ntdll!RtlGetFullPathName_Ustr (77b0a01f)
  ntdll!RtlGetFullPathName_Ustr+0x270 (77aaa47d):
    call to ntdll!RtlpReferenceCurrentDirectory (77b0aed5)
  ntdll!RtlGetFullPathName_Ustr+0x2a3 (77aaa497):
    call to ntdll!RtlpComputeBackupIndex (77b0af83)
  ntdll!RtlGetFullPathName_Ustr+0x2f9 (77ab0340):
    call to ntdll!RtlpReferenceCurrentDirectory (77b0aed5)
  ntdll!RtlGetFullPathName_Ustr+0x32b (77ab0366):
    call to ntdll!RtlUpcaseUnicodeChar (77ae0db2)
  ntdll!RtlGetFullPathName_Ustr+0x339 (77ab0374):
    call to ntdll!RtlUpcaseUnicodeChar (77ae0db2)
  ntdll!RtlGetFullPathName_Ustr+0x366 (77ab0386):
    call to ntdll!RtlpCheckRelativeDrive (77ab040c)
  ntdll!RtlGetFullPathName_Ustr+0x38c (77ab03ac):
    call to ntdll!RtlInitUnicodeString (77ae7e70)
  ntdll!RtlGetFullPathName_Ustr+0x3a9 (77ab03c9):
    call to ntdll!RtlQueryEnvironmentVariable_U (77ad4c19)
  ntdll!RtlGetFullPathName_Ustr+0x11e (77ad8154):
    call to ntdll!memmove (77ae8d80)
  ntdll!RtlGetFullPathName_Ustr+0x7 (77b0a026):
    call to ntdll!_SEH_prolog4_GS (77b08f60)
  ntdll!RtlGetFullPathName_Ustr+0xb0 (77b0a0c3):
    call to ntdll!RtlIsDosDeviceName_Ustr (77b09bf1)
  ntdll!RtlGetFullPathName_Ustr+0x15e (77b0a0e7):
    call to ntdll!memset (77ae90c0)
  ntdll!RtlGetFullPathName_Ustr+0x16a (77b0a0f3):
    call to ntdll!RtlDetermineDosPathNameType_Ustr (77b08ed3)
  ntdll!RtlGetFullPathName_Ustr+0x83e (77b0a27d):
    call to ntdll!RtlGetFullPathName_Ustr+0x84b (77b0a292)
  ntdll!RtlGetFullPathName_Ustr+0x141 (77b0a285):
    call to ntdll!_SEH_epilog4_GS (77b08fa8)
  ntdll!RtlGetFullPathName_Ustr+0x21b (77b0ae76):
    call to ntdll!RtlpReferenceCurrentDirectory (77b0aed5)
  ntdll!RtlGetFullPathName_Ustr+0x261 (77b0aea0):
    call to ntdll!RtlpComputeBackupIndex (77b0af83)
  ntdll!RtlGetFullPathName_Ustr+0xe9 (77b10822):
    call to ntdll!RtlpCheckDeviceName (77b2cf1c)
  ntdll!RtlGetFullPathName_Ustr+0x3f2 (77b1093d):
    call to ntdll!RtlInitUnicodeString (77ae7e70)


Many guys are offending Vista for this and that (mainly UAC stuff which sometimes really sucks) but for me this is another prove that Vista behaves more stable than previous Windows versions.

Thursday, February 12, 2009

Keeping the Pearl

I recently get a memory dump of a hanging process. So I opened it with windbg

The main thread was looking like this:

0:00> ~0 kb
ChildEBP RetAddr Args to Child
0012c72c 7c90e9c0 7c91901b 0000056c 00000000 ntdll!KiFastSystemCallRet
0012c730 7c91901b 0000056c 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
0012c7b8 7c90104b 0097e4c0 7c910945 7c97e4c0 ntdll!RtlpWaitForCriticalSection+0x132
0012c7c0 7c910945 7c97e4c0 ffffffff 00000000 ntdll!RtlEnterCriticalSection+0x46
0012c800 7c80e2cf ffffffff 7c80e038 00000000 ntdll!RtlAcquirePebLock+0x28
0012c868 7c80e00d 7c80e038 00020690 00000000 kernel32!BasepComputeProcessPath+0x5f
0012c8a8 7c801b83 00000000 00000000 0012d7a0 kernel32!BaseComputeProcessDllPath+0xe3
0012c908 7c801d6e 7ffdfc00 00000000 00000000 kernel32!LoadLibraryExW+0x12f
0012c91c 7c801da4 06057954 00000000 00000000 kernel32!LoadLibraryExA+0x1f
0012c938 060b4823 06057954 7c911993 03f41ea8 kernel32!LoadLibraryA+0x94

Sieextpub's very usefull !critlist extension showed me were whose holding the critical section:

0:000> !critlist
CritSec at 7c97e4c0. Owned by thread 6.
Waiting Threads: 0 21
CritSec at 7c97c0d8. Owned by thread 21.
Waiting Threads: 22

SOS!Threads told me that it's a managed thread from the thread pool:

0:006> !Threads
ThreadCount: 5
UnstartedThread: 0
BackgroundThread: 5
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
0 1 ea4 001886e0 4220 Enabled 00000000:00000000 0019c7b8 1 STA System.Runtime.InteropServices.SEHException (0177d128)
2 2 2c0 001a6210 b220 Enabled 00000000:00000000 0019c7b8 0 MTA (Finalizer)
3 3 d7c 001efbd0 80a220 Enabled 00000000:00000000 0019c7b8 0 MTA (Threadpool Completion Port)
4 4 894 00223760 880b220 Enabled 00000000:00000000 0019c7b8 0 MTA (Threadpool Completion Port)
6 5 650 00224310 180b220 Enabled 017d2770:017d4608 0019c7b8 0 MTA (Threadpool Worker)

Now I wanted to get the Stack Dump for thread number 6:

0:006> !CLRStack
OS Thread Id: 0x650 (6)
0522f3bc 7c913da4 [NDirectMethodFrameStandalone: 0522f3bc]
0522f3d4 79364aef
0522f8f8 793644ae
0522f900 793821b0
0522f90c 793820f6
0522f928 79381dbe
0522f950 79381d16
0522f968 793a2014
0522f980 04189669
0522f9a0 041892b2
0522fa08 0418916f
0522fa38 79407caa
0522fa3c 79373ecd
0522fa54 79407e18
0522fa68 79407d90
0522fbf8 79e7c74b [GCFrame: 0522fbf8]

Oops? OK - sometimes !DumpStack does a better job:

0:006> !DumpStack
OS Thread Id: 0x650 (6)
Current frame: ntdll!RtlGetFullPathName_Ustr+0x653
ChildEBP RetAddr Caller,Callee
0522f2d0 79e75002 mscorwks!FrameWithCookie<HelperMethodFrame>::FrameWithCookie<HelperMethodFrame>+0x1e, calling mscorwks!HelperMethodFrame::HelperMethodFrame
0522f318 7c91056d ntdll!RtlFreeHeap+0x647, calling ntdll!_SEH_epilog
0522f33c 7c9141e4 ntdll!RtlGetFullPathName_U+0x33, calling ntdll!RtlGetFullPathName_Ustr
0522f364 7c80b86c kernel32!GetFullPathNameW+0x1a, calling ntdll!RtlGetFullPathName_U
0522f37c 015da1c3 015da1c3
0522f3a4 79364aef 79364aef, calling 796c00cc
0522f3c4 79364aef 79364aef, calling 796c00cc
0522f8f0 793644ae 793644ae, calling 793644b8
0522f8f8 793821b0 793821b0, calling 79364480
0522f904 793820f6 793820f6, calling 79382120
0522f920 79381dbe 79381dbe, calling 79381fac
0522f934 79381d16 79381d16, calling 79381d54
0522f95c 793a2014 793a2014, calling 79381cd4
0522f978 04189669 04189669
0522f994 041892b2 041892b2, calling 04189600
0522fa00 0418916f 0418916f, calling 04189208
0522fa30 79407caa 79407caa
0522fa34 79373ecd 79373ecd
0522fa48 79407e18 79407e18, calling 79373e4c
0522fa60 79407d90 79407d90, calling 79407dc8
0522fa74 79e7c74b mscorwks!CallDescrWorker+0x33
0522fa84 79e7c6cc mscorwks!CallDescrWorkerWithHandler+0xa3, calling mscorwks!CallDescrWorker
0522fb04 79f00eca mscorwks!DispatchCallBody+0x1e, calling mscorwks!CallDescrWorkerWithHandler
0522fb24 79f00e75 mscorwks!DispatchCallDebuggerWrapper+0x3d, calling mscorwks!DispatchCallBody
0522fb88 79f00f03 mscorwks!DispatchCallNoEH+0x51, calling mscorwks!DispatchCallDebuggerWrapper
0522fbbc 79f6e39d mscorwks!QueueUserWorkItemManagedCallback+0x6c, calling mscorwks!DispatchCallNoEH
0522fc1c 79ef3207 mscorwks!Thread::DoADCallBack+0x32a
0522fc30 79ef31a3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3, calling mscorwks!Thread::DoADCallBack+0x2db
0522fc58 79e744d0 mscorwks!Thread::EnterRuntimeNoThrow+0x94, calling ntdll!RtlSetLastWin32Error
0522fc5c 79e744d7 mscorwks!Thread::EnterRuntimeNoThrow+0x9b, calling mscorwks!_EH_epilog3
0522fcc4 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a, calling mscorwks!Thread::ShouldChangeAbortToUnload+0x32
0522fd00 79ef4826 mscorwks!Thread::ShouldChangeAbortToUnload+0x33e, calling mscorwks!Thread::ShouldChangeAbortToUnload+0x29d
0522fd28 79ef48cf mscorwks!ManagedThreadBase::ThreadPool+0x13, calling mscorwks!Thread::ShouldChangeAbortToUnload+0x319
0522fd40 79f6e2d8 mscorwks!ManagedPerAppDomainTPCount::DispatchWorkItem+0xdb, calling mscorwks!ManagedThreadBase::ThreadPool
0522fd94 79f022c7 mscorwks!PerAppDomainTPCountList::GetAppDomainIndexForThreadpoolDispatch+0x35
0522fda8 79f0202a mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf
0522fdbc 79f021a0 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x223, calling mscorwks!ThreadpoolMgr::ExecuteWorkRequest
0522fe14 79f95a2e mscorwks!Thread::intermediateThreadProc+0x49
0522ffa0 79f95a1c mscorwks!Thread::intermediateThreadProc+0x37, calling mscorwks!_alloca_probe_16
0522ffb4 7c80b683 kernel32!BaseThreadStart+0x37

Not really better, eh?

So I launched my good old (which lives side by side with the latest windbg on my machine - thanks to xcopy) and surprise:

[click image to enlarge]


The call stack window did a wonderful job and I even got pointed to the corresponding source code.

As it is the same sos, a !DumpStack did not do the job in either.

So please Microsoft, give us back the this very very helpful feature of!
Or at least make SOS working like the stack window of

Tuesday, February 10, 2009

Thursday, December 18, 2008

Symbol Server Performance Improvement

Once you've fallen in love with your own symbol server you might discover bad performance the bigger the symbol store grows.
We've recently moved our private symbol server onto a new server and found in the symhttp.doc the following:

Normally files are placed in a symbol store with a single tier directory structure in which a single subdirectory exists to store all versions of a certain filename. Such a tree may look like this





However if you are going to store a massive amount of filenames, you may prefer to use the two-tier structure. To do this, place a file called index2.txt in the root of c:\symstore. The contents of the file are of no importance. This would result in a tree that looks like this





The added layer of directories allows you to use DFS or directory junctions to manage and split up your files.

Putting the empty index2.txt in the root is not really a task to mention, but what happens to all the files you have added over time?
As a software developer I tend to do things the lazy way. So rerunning symstore on the old configurations is something for those having killed father and mother.
Thanks to the power of LINQ and NConsoler I could write a handy tool in minutes that does the migration stuff for me (and you).


Lay back and feel the speed ;-)

[Please backup the symbol store before migration - I give no warranty!]

Monday, November 24, 2008

Windbg version is out (I'm still keeping

As some blogs (here and here) already published there is a new version of windbg out:


Here's the direct download link. Unfortunately there's still no inherent managed debugging support as we had with so I'll keep this version (until the end of days?)...



Pat Styles explains in the windbg newsgroup the background why this managed debugging feature made it's way in public and why it has been taken way very soon.
So sad it is: likely there  will be NO managed debugging support for windbg in the future. So be happy if you kept your and make an extra backup!


As last time the highlights are very roughly described so I pasting the full list of relnotes.txt here for googles convenience ;-)

I marked the features I very personally like in green

Important changes in WinDbg 6.10:

  * CE6 dump processing now works.
  * NT4 no longer supported.
  * Enable installation of the USB debug driver on 64bit versions of Windows.
  * Install USBView.exe as part of the debugger package.
  * Enable use of any 1394 cards in target machines.
  * Improve 1394 debug driver reliability, responsiveness and performance.
  * Improve error messages for specific 1394 driver installation failures.
  * Prevent loss of 1394 debug connections after host sleep or hibernate.
  * Change the 1394 debug driver to write every sent packet to the target
    regardless of whether the previously sent packet was acknowleged.
  * Ensure the installed 1394 debug driver is new enough to work properly with
    the current release of the debugger.  (Update the 1394 driver to the
    version shipped in this debugger package.)
  * Automatically reconnect to broken in target machines when doing 1394
    kernel debugging.
  * Introduce new ".dump /mA ..." option to generate user-mode minidump.
    The new behaiour is the same as ".dump /ma" except that it will ignore
    inacessable memory read failure and continue minidump generation,
    instead of bailing out immediately.
>>I've seen this behavior quite often - so thank's for that!<<
  * Add CHK/FRE information in user-mode minidump. Only dumps generated from
    the new debugger will carry this information.
  * Include breakpoint exception in user-mode minidump when the dump is
    generated after a breakpoint hits. This enables debugger to show the
    current process/thread information when debugging a dump file.
  * Fix debugger hang when writing lsass dump files.
  * Fix debugging of WOW64 minidumps containing managed code.
  * Force the WOW64 extension to convert contexts as appropriate.
  * Don't sign extend addresses of 32 bit user mode processes in the 64 bit
  * Set 64bit debugger as the postmortem debugger for WOW64 applications.
  * Disable adding user-mode breakpoints in kd when KdIgnoreUmException is set.
  * Fix escape character ("\") parsing problem in the bl command.
  * Force bu breakpoint removal for unloaded modules.
  * Add automatic IPv6 detection to the "-remote tcp:..." remote protocol.
  * Fix .sleep to work when -ddefer is used.
  * Fix .tlist command
  * Performance improvements to windbg and other debuggers.
  * Notify users when workspace corruption is detected.
  * Fix the infinite recursion with corrupt workspaces in windbg.
  * Report the correct processor family values in !cpuid for all processors.
  * Attempt to breakin only once per ctrl-c, ctrl-break instead of spinning
    in a loop on packet write failure.
  * Print the target computername in the debugger vertarget output when symbol
    information is available.
  * Tell debugger users how to break out of symbol loading, if it does not
    complete within a reasonable amount of time.
  * Display source server spew state when .srcnoisy is run. Also display the
    source server only output filter state.
  * Workaround for Vista/DWM unpainted captions.
  * Add host debugger time when remote kernel target boots and shuts down.
  * Symstore: don't truncate long file names when compressing files.
  * Symchk: Add recursive symbol search option ("/sr") to force SymChk to
    recursively search (non-symbol server) symbol paths for matching symbols.
  * Add support in symsrv for getting file pointers from an http store and
    using them as is without caching them to the default downstream store.
  * Agestore tool should not run on computers that do not update the
    last-access-time on files.
  * Many !analyze improvements.
  * Extensive debugger documentation updates.