Fix for KDC Kerberos issues after patching for CVE-2022–37966
TL&DR: On 17 November 2022, Microsoft released OOB patches to install on all Domain Controllers to fix this issue. Install these patches and remove any workarounds you applied on all other systems.
UPDATE 3 17–11: Microsoft is releasing Out-of-band (OOB) security updates today, November 17, 2022 for installation on all the Domain Controllers (DCs) in affected environments. This update addresses a known issue which might cause sign in failures or other Kerberos authentication issues. You do not need to install any update or make any changes to other servers or client devices in your environment to resolve this issue. If you used any workaround or mitigations for this issue, they are no longer needed, and we recommend you remove them.
UPDATE 2 - 14–11: Microsoft is aware o this problem ans currently working on a fix. They also ‘published’ a workaround. You can run the following command on the KDC servers (DCs) to suppress this issue:reg add HKLM\System\currentcontrolset\services\kdc /t REG_DWORD /v ApplyDefaultDomainPolicy /d 0 /f
UPDATE 1 - 14–11: We now have a better understanding of the bug. It seems that with the default setting for DefaultDomainSupportedEncTypes the client will request DES encryption when RC4 is NOT enabled and the KDC will reject any request containing DES cyphers. For the time being we suggest that you set DefaultDomainSupportedEncTypes to 0x3C (60 decimal)
In our original post we confused the default value of DefaultDomainSupportedEncTypes for 27 instead of 0x27 (dec vs hex). However, these two numbers have the same impact as the both turn RC4 off and DES on.
Update 3:
Out of Band patches are now available to fix the issues with this patch
Take action: OOB update to address an issue with sign in and Kerberos
authentication
Microsoft is releasing Out-of-band (OOB) security updates today,
November 17, 2022 for installation on all the Domain Controllers (DCs) in
affected environments. This update addresses a known issue which might
cause sign in failures or other Kerberos authentication issues. You do
not need to install any update or make any changes to other servers or
client devices in your environment to resolve this issue. If you used any
workaround or mitigations for this issue, they are no longer needed, and we
recommend you remove them.Update 2:
When we tried to raise a ticket for these issues on the Microsoft support portal and selected the catagroy ‘Windows Security services’ and the sub-catagory ‘Kerberos Authention’ we were presented with a pop-up with the following message:
Kerberos Authentication fails after installing November 8, 2022 update
After installing updates released on November 8, 2022 or later on
Windows Servers with the Domain Controller role, you might have issues
with Kerberos authentication. This issue might affect any Kerberos
authentication in your environment. In an environment where RC4 has
been removed as a supported encryption type for user accounts, computer
accounts, service accounts, and group managed service accounts (GMSAs),
applying the November 8, 2022 updates to a domain controller prevents
that domain controller from servicing Kerberos authentication requests.
The November 8, 2022 updates contain security hardening for Netlogon
and Kerberos.
Symptoms
You might experience one or more of the following symptoms:
Microsoft-Windows-Kerberos-Key-Distribution-Center logs Event ID 14
in the System event log of the domain controller.
While processing an AS request for target service <service>, the
account <account name> did not have a suitable key for generating a Kerberos
ticket (the missing key has an ID of 1). The requested etypes : 18 3. The
accounts available etypes : 23 18 17. Changing or resetting the password
of <account name> will generate a proper key.
In such cases, the missing key is 1, and 4 is missing from the "requested
etypes" or "accounts available etypes" lists.
Domain users might not be able to sign in. This issue might also affect
Active Directory Federation Services (AD FS) authentication.
Group Managed Service Accounts (gMSA), which Windows uses for services
such as Internet Information Services (IIS Web Server), might fail to
authenticate.
Domain users might fail to connect to Remote Desktop sessions.
Users might not be able to access shared folders on workstations and
file shares on servers.
Domain users might not be able to print, if printing requires
authentication.
Workaround
This solution provides information for two scenarios:
Work around current Kerberos authentication failures.
Test accounts before you install the updates.
Work around current Kerberos authentication failures
Important: Follow this section's instructions carefully. Serious problems
can occur if the registry is modified incorrectly. Before you start modifying
the registry, back up the registry so you can restore it in case something
goes wrong.
Before or after you install the November 8, 2022 updates on a domain
controller, temporarily set the domain controller's default domain policy
setting. To do this, follow these steps:
On the domain controller, open an administrative Command Prompt window.
At the command prompt, run the following command:
reg add HKLM\System\currentcontrolset\services\kdc /t REG_DWORD /v
ApplyDefaultDomainPolicy /d 0 /f
Test accounts before you install the updates
The following Windows PowerShell script performs an LDAP search to see if
the November 8, 2022 updates affect user, computer, service or GMSA accounts
in your environment. Run the following script at a PowerShell command prompt
in each Active Directory domain:
Get-ADObject -Filter "msDS-supportedEncryptionTypes -bor 0x18 -and -not
msDS-supportedEncryptionTypes -bor 0x7"
This script identifies any accounts in the domain that use the RC4 encryption
type. If the script identifies such accounts in your environment, follow the
steps in the previous section to work around the issue.
Mitigation
Microsoft is urgently working on a Windows Update that resolves this issue.
When it's released, the fix will be documented on Windows Release Health.
The Release Notes sections of the related Windows Update KBs will also
document the fix.
Windows release health
The Windows release health page is designed to inform you about known issues
so you can troubleshoot known issues your users may be experiencing in your
organization.
Check sections of Windows Health Release Dashboard on Docs for each
impacted OS version.
See Windows release health page in M365 portal in the Microsoft 365
admin center (available to customers with Microsoft 365 subscription).Ever since our teams applied the patches in Microsoft’s Patch Tuesday of 8 November we started to run into issues with several Windows environments.
As you can see in the screenshot below, those issues were related to Kerberos authentication which quickly set our star engineer Matthijs Wijers on the trail of the patch for CVE-2022–379666 a vulnerability that allows an attacker to elevate his privileges when RC4-HMAC is used.
It is odd that a patch for CVE-2022–37966 would affect us. Since we focus on Mission Critical applications, we are running a domain wide policy on most of our environments that forces all Kerberos authentication to use at least AES authentication (match ms-DS-SupportedEncryptionType/Network security: Configure encryption types allowed for Kerberos), so why it this effecting us?
As a first step in trouble shooting, Matthijs rolled back the patch for this CVE on the domain controllers, and it indeed caused the authentication errors to stop. This confirmed that indeed the patch for CVE-2022–37966 was to blame, but why?
A closer inspection of the logs shows that the KDC service, which is responsible for Kerberos authentication, is rejecting client requests because they are using Kerberos tickets with DES ciphers. But why? We have a domain policy that says these ciphers should not be used.
After several days of research, Matthijs figured out that if he set the registry key HKLM:\System\CurrentControlSet\Services\KDC\ DefaultDomainSupportedEncTypes to 0x3C (60 decimal) on the KDC servers (typically the domain controllers), clients stopped requesting these ciphers and issue requests in line with our setting for ms-DS-SupportedEncryptionType.
You can set this registry key with the following PowerShell command.
New-ItemProperty -Path “HKLM:\System\CurrentControlSet\Services\KDC” -Name
DefaultDomainSupportedEncTypes -Value 60 -PropertyType DWORD -ForceYou then need to restart the KDC Service and maybe need to restart any service that is currently experiencing authentication issues (like e.g. sql server service using Kerberos delegation)
So, what is the root cause?
According to KB5021131 and this linked article when this registry key is not set, its default value is (now?) 0x27 (39 dec.) which signifies that DES-CBC-CRC, DES-CBC-MD5, DES-CBC-RC4 and AES128-CTS-HMAC-SHA1–96 is supported and that RC4-HMAC, AES128-CTS-HMAC-SHA1–96 and AES256-CTS-HMAC-SHA1–96 are turned off. But our value for ms-DS-SupportedEncryptionType is 0x18 (24 dec.), which signifies that only AES128-CTS-HMAC-SHA1–96 and AES256-CTS-HMAC-SHA1–96 are supported.
It appears that this causes the clients to also use DES ciphers, but those authentication requests to (incorrectly?) be denied higher in the stack.
Bear in mind, we did not reverse engineer the patch, so we are not 100% sure if this really is the root cause. Our tests so far indicate that setting this registry key to match ms-DS-SupportedEncryptionType solves the issue.
Not all issues resolved.
While setting DefaultDomainSupportedEncTypes to 0x39 solves a lot of Kerberos authentication issues with user accounts. But computer accounts still have troubles with kerberos pre-authentication. Matthijs is still investigating. For now this most likely needs a fix from Microsoft.
It is still early in the game, so your milage may vary. But this has worked for us so far on multiple environments and we want others to benefit from this knowledge as well, because caring is sharing.
For now, Matthijs is our hero of the day.
