The monitoring system on one of my servers running Windows Server 2016 started sending multiple messages of the Volume Shadow Copy Service error with the Event ID 8193 and VSS as a source. The full error description in the Event Viewer (Application log) looks like this:
Volume Shadow Copy Service error: Unexpected error calling routine RegOpenKeyExW (-2147483646, SYSTEM\CurrentControlSet\Services\VSS\Diag,...). hr = 0x80070005, Access is denied. Operation: Initializing Writer Context: %MINIFYHTMLb5bdc426b817a5035bf92ebd42cdf8ae5%Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220} Writer Name: System Writer Writer Instance ID: {4f096fb4-2e00-4864-aa8f-885aa9186850}
At the same time, the server is working fine, and there were no visible problems with the services or apps. There were no errors in the list of VSS modules for the instance ID of VSS Writer.
vss list writers
VSS EventID 8193 is a known error that is generally related to the installation of the DHCP role on a server running Windows Server 2008 (or newer) resulting in the Network Service account losing the permissions on the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Diag.
The problem is related to the installation of the VSS module – DHCP Jet Writer, which is responsible for correct shadow copy creation of the DHCP service. This VSS module changes permissions on the specified registry key during installation.
To fix this error, you have to manually grant the Full Control permission for Network Service on the specified key.
- Run the Registry Editor (
regedit.exe
); - Go to the registry key HKLM\SYSTEM\CurrentControlSet\services\VSS\Diag and open its permissions (Permissions option in the context menu);
- Find Network Service in the list and assign the Full Control permissions.
Instead of editing the registry manually, it would be more correct to restore the default permissions on the registry key using a special command CLI tool – SubInACL. Download and install it (if you have not done it yet) and run the following commands:
cd “C:\Program Files (x86)\Windows Resource Kits\Tools”
subinacl.exe /Subkeyreg System\CurrentControlSet\Services\VSS\Diag /sddl=D:PAI(A;;KA;;;BA)(A;;KA;;;SY)(A;;CCDCLCSWRPSDRC;;;BO)(A;;CCDCLCSWRPSDRC;;;LS)(A;;CCDCLCSWRPSDRC;;;NS)(A;CIIO;RC;;;OW)(A;;KR;;;BU)(A;CIIO;GR;;;BU)(A;CIIO;GA;;;BA)(A;CIIO;GA;;;BO)(A;CIIO;GA;;;LS)(A;CIIO;GA;;;NS)(A;CIIO;GA;;;SY)(A;CI;CCDCLCSW;;;S-1-5-80-3273805168-4048181553-3172130058-210131473-390205191)(A;ID;KR;;;AC)(A;CIIOID;GR;;;AC)S:ARAI
In Windows Server 2008 R2, the command will look like this (according to Microsoft KB):
subinacl.exe /Subkeyreg System\CurrentControlSet\Services\VSS\Diag /sddl=O:SYG:SYD:PAI(A;;KA;;;BA)(A;;KA;;;SY)(A;;SDGRGW;;;BO)(A;;SDGRGW;;;LS)(A;;SDGRGW;;;NS)(A;CIIO;RC;;;S-1-3-4)(A;;KR;;;BU)(A;CIIO;GR;;;BU)(A;CIIO;GA;;;BA)(A;CIIO;GA;;;BO)(A;CIIO;GA;;;LS)(A;CIIO;GA;;;NS)(A;CIIO;GA;;;SY)(A;CI;CCDCLCSW;;;S-1-5-80-3273805168-4048181553-3172130058-210131473-390205191)
Then replace the permissions on the child (nested) objects in the Registry Editor. To do it, open the Diag reg key properties and click “Permissions” -> “Advanced” -> “Replace all child object permissions”.
Then just restart your Windows.
PS. When writing this article I found out that there was another VSS error with the same (!!!) EventID 8193 and the following description:
Volume Shadow Copy Service error: Unexpected error calling routine ConvertStringSidToSid(S-1-5-21-2470146651-3958396388-212345117-21232.bak). hr = 0x80070539,The security ID structure is invalid. Operation: OnIdentify event Gathering Writer Data Context: Execution Context: Shadow Copy Optimization Writer Writer Class Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f} Writer Name: Shadow Copy Optimization Writer Writer Instance ID: {5e5d68e6-9c97-4af6-a09f-bb2db4c65058}.
To make the article comprehensive, I will describe the solution of this issue as well.
This error doesn’t allow you to create system state backup or backup your Hyper-V VMs (through the Windows Server Backup), which leads to the following error: “0x80042308: The specified object was not found
”.
This problem is solved differently and related to the incorrect entry in the profile’s reg key HKLM\Software\Microsoft\Windows NT\CurrentVersion\ProfileList. The issue occurs due to VSS service (Shadow Copy Optimization Writer component) cannot find a user profile with the SID ending in .bak and returns an error. You need to delete the registry key shown in the description of the ConvertStringSidToSid event (S-1-5-21-2470146651-3958396388-212345117-21232.bak).
- Open the
regedit.exe
; - Go to the HKLM\Software\Microsoft\Windows NT\CurrentVersion\ProfileList;
- Find and delete the registry key with the .bak suffix;
- Restart your computer and try to run your backup task again.
4 comments
What monitoring system are you using on your servers? Especially for reading the event logs?
Thanks
Event logs from Windows servers are sent to Elasticsearch. There is also a Zabbix server with email alerts about critical events.
Thanks!
Do you have any blog post how you do that?
There are so many articles out there in the web about doing things but only a few about monitoring and topics like file/folder access monitoring etc.
Hello,
I managed to solve this issue with the help of Microsoft support.
The issue seems to be with the VSS metadata max filesize which is limited to 16MB. This can be increased to 32, 64 or 128 according to your needs.
RegKey SYSTEM\CurrentControlSet\Services\VSS\Settings “MaxXmlDocumentSizeMB”.
A detailed exaplanation https://www.storcom.com/sql-vss-backup-unexpected-error-calling-routine-xml-document-is-too-long/
Regards,
Dardan