Hi,
we have faulting application errors every ~5-10 minutes on our 2016 servers with vm tools:
Faulting application name: WmiApSrv.exe, version: 10.0.14393.0, time stamp: 0x57899ac8
Faulting module name: vmStatsProvider.dll, version: 10.0.9.55972, time stamp: 0x57426eff
Exception code: 0xc0000005
Fault offset: 0x0000000000001d9a
Faulting process id: 0x14a4
Faulting application start time: 0x01d2f499b7b2dbde
Faulting application path: C:\WINDOWS\system32\wbem\WmiApSrv.exe
Faulting module path: C:\Program Files\VMware\VMware Tools\vmStatsProvider\win64\vmStatsProvider.dll
Report Id: 42c0409f-03cd-4033-bedc-9194b77373b4
Faulting package full name:
Faulting package-relative application ID:
Tried different vmtools versions including newest 10.1.7, still the same problem.
Server has latest security and feature updates, also .net 4.7
I've tried to look at procdump, however I don't have any experience with windbg therefore I'm unable to conclude anything from the output. Some info from the dump:
AULTING_IP:
vmStatsProvider+1d9a
00007ffd`1bf41d9a ff5010 call qword ptr [rax+10h]
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00007ffd1bf41d9a (vmStatsProvider+0x0000000000001d9a)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 00007ffd241fe520
Attempt to read from address 00007ffd241fe520
PROCESS_NAME: WmiApSrv.exe
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".
EXCEPTION_PARAMETER1: 0000000000000000
EXCEPTION_PARAMETER2: 00007ffd241fe520
READ_ADDRESS: 00007ffd241fe520
FOLLOWUP_IP:
vmStatsProvider+1d9a
00007ffd`1bf41d9a ff5010 call qword ptr [rax+10h]
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
LAST_CONTROL_TRANSFER: from 00007ffd1bf428c5 to 00007ffd1bf41d9a
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [UnloadedModule_Arch_AX] from Frame:[0] on thread:[188c] ; Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD] ; Followup set based on attribute [ip_is_call_value_Arch_ax] from Frame:[0] on thread:[188c] ; Followup set based on attribute [Is_OriginalExceptionThread] from Frame:[0] on thread:[188c]
FAULTING_THREAD: ffffffffffffffff
BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ_CALL_LEAK
PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ_CALL_LEAK
DEFAULT_BUCKET_ID: INVALID_POINTER_READ_CALL_LEAK
STACK_TEXT:
00000000`00000000 00000000`00000000 fastprox!CWbemInstance::_vftable_+0x0
STACK_COMMAND: .ecxr ; ~~[188c] ; .frame 0 ; ** Pseudo Context ** ; kb
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: fastprox!CWbemInstance::_vftable_
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: fastprox
IMAGE_NAME: fastprox.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 0
FAILURE_BUCKET_ID: INVALID_POINTER_READ_CALL_LEAK_c0000005_fastprox.dll!CWbemInstance::_vftable_
BUCKET_ID: X64_APPLICATION_FAULT_INVALID_POINTER_READ_CALL_LEAK_fastprox!CWbemInstance::_vftable_
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/WmiApSrv_exe/6_2_14393_0/57899ac8/vmStatsProvider_dll/10_0_9_55972/57426eff/c0000005/00001d9a.htm?Retriage=1;
and some more:
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(688.188c): Access violation - code c0000005 (first/second chance not available)
*** ERROR: Symbol file could not be found. Defaulted to export symbols for vmStatsProvider.dll -
vmStatsProvider+0x1d9a:
00007ffd`1bf41d9a ff5010 call qword ptr [rax+10h] ds:00007ffd`241fe520=????????????????
0:004> k
# Child-SP RetAddr Call Site
00 0000001c`d977f500 00007ffd`1bf428c5 vmStatsProvider+0x1d9a
01 0000001c`d977f540 00007ffd`1bf42403 vmStatsProvider+0x28c5
02 0000001c`d977f580 00007ffd`1bf56190 vmStatsProvider+0x2403
03 0000001c`d977f5d0 00007ffd`1bf5171f vmStatsProvider!DllUnregisterServer+0x10830
04 0000001c`d977f600 00007ffd`1bf51965 vmStatsProvider!DllUnregisterServer+0xbdbf
05 0000001c`d977f640 00007ffd`378ea35f vmStatsProvider!DllUnregisterServer+0xc005
06 0000001c`d977f680 00007ffd`378d3d2d ntdll!LdrpCallInitRoutine+0x4b
07 0000001c`d977f6e0 00007ffd`378d1621 ntdll!LdrpProcessDetachNode+0xf5
08 0000001c`d977f7b0 00007ffd`3790eefc ntdll!LdrpUnloadNode+0x49
09 0000001c`d977f800 00007ffd`3790ee24 ntdll!LdrpDecrementModuleLoadCountEx+0xc4
0a 0000001c`d977f850 00007ffd`347602cd ntdll!LdrUnloadDll+0x94
0b 0000001c`d977f880 00007ffd`367f5b38 KERNELBASE!FreeLibrary+0x1d
0c 0000001c`d977f8b0 00007ffd`367f5a56 combase!CClassCache::CDllPathEntry::CFinishObject::Finish+0x28 [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 3368]
0d 0000001c`d977f8e0 00007ffd`3679f1b4 combase!CClassCache::CFinishComposite::Finish+0x56 [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 3478]
0e 0000001c`d977f910 00007ffd`3679c4cb combase!CClassCache::CleanUpDllsForProcess+0x124 [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 6969]
0f (Inline Function) --------`-------- combase!CCCleanUpDllsForProcess+0xe [d:\rs1\onecore\com\combase\objact\dllcache.cxx @ 8680]
10 0000001c`d977fb10 00007ffd`367ded00 combase!ProcessUninitialize+0x1c3 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 2236]
11 0000001c`d977fb50 00007ffd`367deb23 combase!DecrementProcessInitializeCount+0x44 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 992]
12 0000001c`d977fb80 00007ffd`367db835 combase!wCoUninitialize+0x87 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 4115]
13 0000001c`d977fbc0 00007ff7`5bd44351 combase!CoUninitialize+0x85 [d:\rs1\onecore\com\combase\class\compobj.cxx @ 3946]
14 0000001c`d977fbf0 00007ff7`5bd45472 WmiApSrv!WinRun+0x89
15 0000001c`d977fc30 00007ffd`35223dd2 WmiApSrv!WmiAdapterService::_ServiceMain+0x52
16 0000001c`d977fc60 00007ffd`37728364 sechost!ScSvcctrlThreadA+0x22
17 0000001c`d977fc90 00007ffd`379270d1 kernel32!BaseThreadInitThunk+0x14
18 0000001c`d977fcc0 00000000`00000000 ntdll!RtlUserThreadStart+0x21