VHD surfacing hang, 500 drvinst.exe processes, VMMS event 7011, VSS KB2627052 symptoms, but does not fix.

hi everyone,

recently i've noticed mounting or attaching vhd on our servers hyper-v role hang system. applies vms hypervisor itself. has seen before?

symptoms:

• appear kb2627052 hotfix not resolve issue
• 500 drvinst.exe processes spawned, , never complete (not infection)
• the affected host or vm becomes unresponsive, not affect other vms - vms continue operate, , appear unaffected when host has hung.
• remote desktop connects, no shell ever appears
• some executable work, of them
  eg: cmd.exe works, vssadmin.exe, tasklist.exe, taskkill.exe not.
• processes not die.
• rpc doesn't appear respond
• hard reset needed restore operation.
• service control manager (for vmms) error 7011 in system log
• volsnap error 10 in system log
• hyper-v-hypervisor error 8451 in hyper-v-hypervisor (under apps , services) event log - not sure if related though.

environment

• windows server 2008 r2 sp1 hyper-v role + various updates , recommended hotfixes.
• 3x vms (separate domain controller, exchange, , misc vms)
• hypervisor not joined domain
• vss image backups done host using shadowprotect (and appear work, , can restored)
• no multi-path i/o
• vhds stored on 1 4tb ntfs partition (gpt)

reproduction

• attach vhd using disk management mmc (host or vm online)
• attach vhd new drive using hyper-v manager (vm online) - no issues if attaching while offline
• changing drive letter

additional information

• applied hotfix in kb2627052 - removed volsnap error 10 system log, else remains same. (appled vms , host)
• applied hotfix in kb2661530 - bring vmms.exe latest version
• read on http://social.technet.microsoft.com/forums/en-us/dataprotectionmanager/thread/76da4778-91f3-4c2e-a1e3-066b2a67e73a/

full event log error details

vmms - service control manager - event id 7011 - "a timeout (30000 milliseconds) reached while waiting transaction response vmms service."

- <event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">  - <system>    <provider name="service control manager" guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" eventsourcename="service control manager" />     <eventid qualifiers="49152">7011</eventid>     <version>0</version>     <level>2</level>     <task>0</task>     <opcode>0</opcode>     <keywords>0x8080000000000000</keywords>     <timecreated systemtime="2012-02-29t02:27:21.108641000z" />     <eventrecordid>59349</eventrecordid>     <correlation />     <execution processid="252" threadid="2308" />     <channel>system</channel>     <computer>allhyper</computer>     <security />     </system>  - <eventdata>    <data name="param1">30000</data>     <data name="param2">vmms</data>     </eventdata>    </event>

vss - volsnap - event id 10 - "the shadow copy of volume g: took long install."

- <event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">  - <system>    <provider name="volsnap" />     <eventid qualifiers="49158">10</eventid>     <level>2</level>     <task>0</task>     <keywords>0x80000000000000</keywords>     <timecreated systemtime="2012-02-29t02:46:38.443473700z" />     <eventrecordid>59350</eventrecordid>     <channel>system</channel>     <computer>allhyper</computer>     <security />     </system>  - <eventdata>    <data />     <data>g:</data>     <binary>0000000002003000000000000a0006c000000000200100c000000000000000000000000000000000</binary>     </eventdata>    </event>

hyper-v-hypervisor - event id 8451 - "hyper-v failed creating new partition (status 11)!"

- <event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">  - <system>    <provider name="microsoft-windows-hyper-v-hypervisor" guid="{52fc89f8-995e-434c-a91e-199986449890}" />     <eventid>8451</eventid>     <version>0</version>     <level>2</level>     <task>0</task>     <opcode>0</opcode>     <keywords>0x2000000000000000</keywords>     <timecreated systemtime="2012-03-01t11:18:44.455700100z" />     <eventrecordid>2764</eventrecordid>     <correlation />     <execution processid="4" threadid="80" />     <channel>microsoft-windows-hyper-v-hypervisor-operational</channel>     <computer>allhyper</computer>     <security userid="s-1-5-18" />     </system>  - <eventdata>    <data name="partitionid">11</data>     </eventdata>    </event>

any appreciated.

this looks microsoft issue/bug, , until release fix (unlikely, or @ least not quickly), here's workaround:

elevated command prompt:

diskpart automount

check if status enabled or disabled, disable automount.

automount disable automount scrub

attach problem vhd (i did using hyper-v manager, aware following commands act on disk not vdisk, that's how vhd being shown)

find , online disk

list disk select disk x online disk attributes disk clear readonly

exit diskpart

exit

find shadow copies

diskshadow list shadows all

note volume id of affected vhd (\\?\{guid})

remove shadow copies

delete shadows volume \\?\{guid}

-or-

delete shadows all

exit diskshadow

exit

re-enable automounting if enabled

diskpart automount enable exit

this not solution, @ least can mount affected vhd without hanging servers hours.

Windows Server  >  Hyper-V