RSA-PSS on Windows fail to verify signature with salt len != hash len

[touilleMan]

The Windows version of rsa_pss_verify always set the salt length to the size of the hash algorithm:

oscrypto/oscrypto/_win/asymmetric.py

Line 2617 in e90100e

padding_info_struct.cbSalt = len(digest)

The issue is also present in the legacy backend for Windows:

oscrypto/oscrypto/_win/asymmetric.py

Line 2494 in e90100e

if not verify_pss_padding(hash_algorithm, hash_length, key_size, data, decrypted_signature):

oscrypto/oscrypto/_pkcs1.py

Line 185 in e90100e

def verify_pss_padding(hash_algorithm, salt_length, key_length, message, signature):

This is an issue especially given OpenSSL's default behavior is to use a salt len as big as possible

This is also the case for other libraries (for instance in RustCrypto) given there is no obligation for the salt length to be the size of the hash.

Hence valid signatures may appear invalid on Windows :'(

Minimal reproduction example

import binascii
import oscrypto.asymmetric

sign_key = """
-----BEGIN PRIVATE KEY-----
MIIEuwIBADALBgkqhkiG9w0BAQoEggSnMIIEowIBAAKCAQEArHkxb5OQOh5BiPS1
Y+EIANr7OZVyqi7fSw69mjIxjgRqnz/WjG3wuBTLcDG1vuXpu7Ojg/eADo1ZUdZL
lgSPe1mUmF8rFUW6E0j6GBUBJS4mSnqLKjjdqKPs1RjtAX3HD6juCc5un/oyeGcj
MWh1+ixuYoRgIduHpmc5kSLxpjwy2c2ZUYWR+15uKbi7Uqrn8CzmghhPF7u1t5jJ
k+mbRLU+NwqjKP+k02rrHKfsENBu+2yhugCfgUHGz51fT4N3lgbK41WB4vO54kIJ
4SaN2UDElmvJeYJVl1rPXLMVz+1ChfrIFdaVxFEF7Yn82+cgoU8ejrqejyea3/M9
5/2k0QIDAQABAoIBAD85pDozPX1YpwyGLFKDZTQvEkZPNzwUt61jp3S1rr3Rd6aO
N9+9068fjF5CEs56qN6yoSAY5Dwxa8tYw9eoL1L4CUV8KaaAK5CzQV7/oC5Zhxbp
akedlgAiq4iIvSU9TvI6Kpy0rI//n23M3TVZBlqh3AtIXJc8yzLgh1VhmnUl5Gom
i36jmW2BF1wwuqjVSKCvml/RAo2wVELwBm3WXhPxqwYGZStfonEB/MwsHGfbvFcg
MQRCClpLGKXdspam3uQvMYC0lLRED13lL1qB8pV0YFBHpNIdWlG3QYKxcwwcCoE2
OEldjRIugR9XE+t/EyQnzfkeJ3DK74h78CF7rfsCgYEAvjz8FJ85Zb2cCWCwSbBF
QR6VzSvPeHlI8z5DFFtL/nbclhUB3z12uuE2JHMwCox4HLCNYWDoOXUmWwypYJ1X
YV2VZzZyPiAuNFTQ4UHhOk6w6bChiLCjnSCet0MoV/Ukaamu49Rxq7gDy2XkIWoD
wD+9xrdD7lmP0t4YykW7IqMCgYEA6BgfUJD/SQbZkO8+YpUYC8LmFZkEg8txj7kR
wISLsLzdGxy3xh95JS96SofvfGpI5oFcMGAjX3a6vzZ2qBvQa3+IOQQWqJULdaTW
FRPPWYv9gBJDLz4SOH9pv8ZEdBbcZ5N+Y7FEBp0eTXjh1J56NonpQIRXf4ojIlvI
fXmshfsCgYBOO3gS5vPMsi/j714vv4yLXg+Oo1Cbo4zrcxRU38Kdr7XBBnyRmI4m
Bg2k6bW88M1IRxatEBQP5OxUDx3sfGf9w2V4X3yVrdgybxrDN7tupgO85oVXWATA
zjRW+wgxO7+wsDYavTfNvUvaLlmloBpQyiW5/Y2zDCPIPMuHCywM7wKBgQDdy7eZ
QYeEnRQrSkZe5UYebzl7qEhFPpUemOibBs+LrWDK+Q2yOv+FhrKiKPe2+McD6NlV
rXoAT7E06/JGwpXRNQXUHtEcd5qE6WpgqBa952bxDgLAUdwNu80uJGXkXrhwDuZ4
lL2CaIG93WhKzMvT9MVAD3iifDsJKZcWOcGiIwKBgD40p/yLSdU07L5xoJi/XhZi
lSjkwVVjelZrebI6fczcFijpIgxu0Jns21NEDQE/qPE6Mc5atCh2DenM55XXI+sa
LiSbnD7yIMXNuAS63Q+TCSDdFhTE6UHi84+1WqON8kOZrQ+2bzT014mZN+2fNXfR
jyj0su/eWkS3DmtTie1z
-----END PRIVATE KEY-----
"""
verify_key = b"""-----BEGIN PUBLIC KEY-----
MIIBIDALBgkqhkiG9w0BAQoDggEPADCCAQoCggEBAKx5MW+TkDoeQYj0tWPhCADa
+zmVcqou30sOvZoyMY4Eap8/1oxt8LgUy3Axtb7l6buzo4P3gA6NWVHWS5YEj3tZ
lJhfKxVFuhNI+hgVASUuJkp6iyo43aij7NUY7QF9xw+o7gnObp/6MnhnIzFodfos
bmKEYCHbh6ZnOZEi8aY8MtnNmVGFkftebim4u1Kq5/As5oIYTxe7tbeYyZPpm0S1
PjcKoyj/pNNq6xyn7BDQbvtsoboAn4FBxs+dX0+Dd5YGyuNVgeLzueJCCeEmjdlA
xJZryXmCVZdaz1yzFc/tQoX6yBXWlcRRBe2J/NvnIKFPHo66no8nmt/zPef9pNEC
AwEAAQ==
-----END PUBLIC KEY-----
"""

content = b"hello world\n"

signature_with_salt_len_32 = binascii.unhexlify(
"036230d01bf469685efe8b205402f7eb81305cfa13363ba07a9c77cf400a"
"41e87c532a14fbddc572b686f772ba6349840871fef7c1b473e20b138c52"
"c66b0fa72ebb84c552ccc17911c1c02134e75651860fa680f0bb87dc774a"
"9a2e32dc6bd283f52afeb888848b80f7ad3397b6d81e4820e958a17c2816"
"0de5c53a5501eb382f06317920d0fdd9bfe5219c68796313cb73a02a43d9"
"34eb5db0bdec019b56349ff5f19042e5f260949595506c7347290a876216"
"c131da6afa08e0208181f56fa07f06707f1e93e7193fc3afc9c064b55196"
"a8f9f6aa111d5fa1681ec31b43d0bfdb6de39fd0506feb2c8b7cbf332cde"
"ed75694c60404b852d062820293322ab"
)
signature_with_salt_len_200 = binascii.unhexlify(
"18133b397edad78aeb4a181a7d213f966e866ef15b6a86254ec423e1e44e"
"46fe7cf8aa7e42d577c81da1f5ffccaba7045b0e45d80866d47eb632a312"
"61d80a61b210dc64c35334c1fae211fc6b091586fdde1428d2104f20017c"
"55006e9849573b24f6a9b94ab870a6c79e10ca5d0659072639495cf8b8d7"
"e364ecbe0b2c1a3ab2fbf80abb17d61c45a8dbe14712446bd27aaa089b20"
"bd9683b628fea46e22a6a31855df71858683cd6b49b7911ca535d5c53b3e"
"2f3c4ed3913e1242933bc3e4a2e87f409e935e2536386e6e4baf3674159c"
"469d92ff775a63f4e384f524e392fc5f940744b229e512ba71f65c7845c6"
"9a49326468ce3756e421bb72e24281b1"
)

verify_key = oscrypto.asymmetric.load_public_key(verify_key)
for salt_len, signature in [(32, signature_with_salt_len_32), (200, signature_with_salt_len_200)]:
    print(f"verify signature with salt len {salt_len}")
    oscrypto.asymmetric.rsa_pss_verify(verify_key, signature, content, "sha256")

output on Linux (Ubuntu 22.04 with openssl 3.0.2):

verify signature with salt len 32
verify signature with salt len 200

output on Windows 10:

verify signature with salt len 32
verify signature with salt len 200
Traceback (most recent call last):
  File "C:\Users\gbleu\source\repos\parsec-cloud\parsec-cloud\bug.py", line 74, in <module>    
    oscrypto.asymmetric.rsa_pss_verify(verify_key, signature, content, "sha256")
  File "C:\Users\gbleu\AppData\Local\pypoetry\Cache\virtualenvs\parsec-cloud--xPPmQVg-py3.9\lib\site-packages\oscrypto\_win\asymmetric.py", line 2296, in rsa_pss_verify
    return _verify(certificate_or_public_key, signature, data, hash_algorithm, rsa_pss_padding=True)
  File "C:\Users\gbleu\AppData\Local\pypoetry\Cache\virtualenvs\parsec-cloud--xPPmQVg-py3.9\lib\site-packages\oscrypto\_win\asymmetric.py", line 2452, in _verify
    return _bcrypt_verify(certificate_or_public_key, signature, data, hash_algorithm, rsa_pss_padding)
  File "C:\Users\gbleu\AppData\Local\pypoetry\Cache\virtualenvs\parsec-cloud--xPPmQVg-py3.9\lib\site-packages\oscrypto\_win\asymmetric.py", line 2650, in _bcrypt_verify
    raise SignatureError('Signature is invalid')
oscrypto.errors.SignatureError: Signature is invalid

Possible fix

Windows' CNG api seems to force the choice of a salt length when doing signature verification

So one solution would be to re-implement the salt length detection code in oscrypto. However this doesn't seems trivial

Another simpler but less elegant solution would be to use try the verification twice (one time with salt length == hash length, another time with salt length == key size - hash length - 2). This is obviously not perfect since arbitrary length salt are not supported and invalid signature got checked twice :'(

[vxgmichel]

Note that the problem can be reproduced on linux by switching from libcrypto 3 to libcrypto 1 between signing and verifying.

Consider those two programs, sign.py and verify.py:

# sign.py

import sys
import oscrypto
base_path = sys.argv[1]
oscrypto.use_openssl(f"{base_path}/libcrypto.so", f"{base_path}/libssl.so")

import oscrypto.asymmetric
print("Libcrypto version:", *oscrypto._openssl.asymmetric.libcrypto_version_info, file=sys.stderr)


sign_key = b"""
-----BEGIN PRIVATE KEY-----
MIIEuwIBADALBgkqhkiG9w0BAQoEggSnMIIEowIBAAKCAQEArHkxb5OQOh5BiPS1
Y+EIANr7OZVyqi7fSw69mjIxjgRqnz/WjG3wuBTLcDG1vuXpu7Ojg/eADo1ZUdZL
lgSPe1mUmF8rFUW6E0j6GBUBJS4mSnqLKjjdqKPs1RjtAX3HD6juCc5un/oyeGcj
MWh1+ixuYoRgIduHpmc5kSLxpjwy2c2ZUYWR+15uKbi7Uqrn8CzmghhPF7u1t5jJ
k+mbRLU+NwqjKP+k02rrHKfsENBu+2yhugCfgUHGz51fT4N3lgbK41WB4vO54kIJ
4SaN2UDElmvJeYJVl1rPXLMVz+1ChfrIFdaVxFEF7Yn82+cgoU8ejrqejyea3/M9
5/2k0QIDAQABAoIBAD85pDozPX1YpwyGLFKDZTQvEkZPNzwUt61jp3S1rr3Rd6aO
N9+9068fjF5CEs56qN6yoSAY5Dwxa8tYw9eoL1L4CUV8KaaAK5CzQV7/oC5Zhxbp
akedlgAiq4iIvSU9TvI6Kpy0rI//n23M3TVZBlqh3AtIXJc8yzLgh1VhmnUl5Gom
i36jmW2BF1wwuqjVSKCvml/RAo2wVELwBm3WXhPxqwYGZStfonEB/MwsHGfbvFcg
MQRCClpLGKXdspam3uQvMYC0lLRED13lL1qB8pV0YFBHpNIdWlG3QYKxcwwcCoE2
OEldjRIugR9XE+t/EyQnzfkeJ3DK74h78CF7rfsCgYEAvjz8FJ85Zb2cCWCwSbBF
QR6VzSvPeHlI8z5DFFtL/nbclhUB3z12uuE2JHMwCox4HLCNYWDoOXUmWwypYJ1X
YV2VZzZyPiAuNFTQ4UHhOk6w6bChiLCjnSCet0MoV/Ukaamu49Rxq7gDy2XkIWoD
wD+9xrdD7lmP0t4YykW7IqMCgYEA6BgfUJD/SQbZkO8+YpUYC8LmFZkEg8txj7kR
wISLsLzdGxy3xh95JS96SofvfGpI5oFcMGAjX3a6vzZ2qBvQa3+IOQQWqJULdaTW
FRPPWYv9gBJDLz4SOH9pv8ZEdBbcZ5N+Y7FEBp0eTXjh1J56NonpQIRXf4ojIlvI
fXmshfsCgYBOO3gS5vPMsi/j714vv4yLXg+Oo1Cbo4zrcxRU38Kdr7XBBnyRmI4m
Bg2k6bW88M1IRxatEBQP5OxUDx3sfGf9w2V4X3yVrdgybxrDN7tupgO85oVXWATA
zjRW+wgxO7+wsDYavTfNvUvaLlmloBpQyiW5/Y2zDCPIPMuHCywM7wKBgQDdy7eZ
QYeEnRQrSkZe5UYebzl7qEhFPpUemOibBs+LrWDK+Q2yOv+FhrKiKPe2+McD6NlV
rXoAT7E06/JGwpXRNQXUHtEcd5qE6WpgqBa952bxDgLAUdwNu80uJGXkXrhwDuZ4
lL2CaIG93WhKzMvT9MVAD3iifDsJKZcWOcGiIwKBgD40p/yLSdU07L5xoJi/XhZi
lSjkwVVjelZrebI6fczcFijpIgxu0Jns21NEDQE/qPE6Mc5atCh2DenM55XXI+sa
LiSbnD7yIMXNuAS63Q+TCSDdFhTE6UHi84+1WqON8kOZrQ+2bzT014mZN+2fNXfR
jyj0su/eWkS3DmtTie1z
-----END PRIVATE KEY-----
"""

content = b"hello world\n"
sign_key = oscrypto.asymmetric.load_private_key(sign_key)
signature = oscrypto.asymmetric.rsa_pss_sign(sign_key, content, "sha256")
print(signature)

# verify.py

import sys
import oscrypto
base_path = sys.argv[1]
oscrypto.use_openssl(f"{base_path}/libcrypto.so", f"{base_path}/libssl.so")

import oscrypto.asymmetric
print("Libcrypto version:", *oscrypto._openssl.asymmetric.libcrypto_version_info, file=sys.stderr)


verify_key = b"""-----BEGIN PUBLIC KEY-----
MIIBIDALBgkqhkiG9w0BAQoDggEPADCCAQoCggEBAKx5MW+TkDoeQYj0tWPhCADa
+zmVcqou30sOvZoyMY4Eap8/1oxt8LgUy3Axtb7l6buzo4P3gA6NWVHWS5YEj3tZ
lJhfKxVFuhNI+hgVASUuJkp6iyo43aij7NUY7QF9xw+o7gnObp/6MnhnIzFodfos
bmKEYCHbh6ZnOZEi8aY8MtnNmVGFkftebim4u1Kq5/As5oIYTxe7tbeYyZPpm0S1
PjcKoyj/pNNq6xyn7BDQbvtsoboAn4FBxs+dX0+Dd5YGyuNVgeLzueJCCeEmjdlA
xJZryXmCVZdaz1yzFc/tQoX6yBXWlcRRBe2J/NvnIKFPHo66no8nmt/zPef9pNEC
AwEAAQ==
-----END PUBLIC KEY-----
"""

content = b"hello world\n"
verify_key = oscrypto.asymmetric.load_public_key(verify_key)
signature = eval(input())
result = oscrypto.asymmetric.rsa_pss_verify(verify_key, signature, content, "sha256")

See what happens when running those scripts with different combination of libcrypto:

$ python sign.py /path/to/libcrypto_v1 | python verify.py /path/to/libcrypto_v1 
Libcrypto version: 1 1 1 q
Libcrypto version: 1 1 1 q

$ python sign.py /path/to/libcrypto_v3 | python verify.py /path/to/libcrypto_v3                           
Libcrypto version: 3 0 2
Libcrypto version: 3 0 2

$ python sign.py /path/to/libcrypto_v1 | python verify.py /path/to/libcrypto_v3
Libcrypto version: 1 1 1 q
Libcrypto version: 3 0 2

$ python sign.py /path/to/libcrypto_v3 | python verify.py /path/to/libcrypto_v1
Libcrypto version: 1 1 1 q
Libcrypto version: 3 0 2
Libcrypto version: 1 1 1 q
Libcrypto version: 3 0 2
Traceback (most recent call last):
  File "/home/vinmic/repos/verify.py", line 25, in <module>
    result = oscrypto.asymmetric.rsa_pss_verify(verify_key, signature, content, "sha256")
  File "/home/vinmic/miniconda/envs/parsec/lib/python3.9/site-packages/oscrypto/_openssl/asymmetric.py", line 1160, in rsa_pss_verify
    return _verify(certificate_or_public_key, signature, data, hash_algorithm, rsa_pss_padding=True)
  File "/home/vinmic/miniconda/envs/parsec/lib/python3.9/site-packages/oscrypto/_openssl/asymmetric.py", line 1486, in _verify
    raise SignatureError('Signature is invalid')
oscrypto.errors.SignatureError: Signature is invalid

In my opinion, the bug is due to this block of code:

oscrypto/oscrypto/_openssl/asymmetric.py

Lines 1890 to 1900 in e90100e

	if libcrypto_version_info < (3, 0):
	res = libcrypto.EVP_PKEY_CTX_ctrl(
	evp_pkey_ctx_pointer,
	LibcryptoConst.EVP_PKEY_RSA,
	LibcryptoConst.EVP_PKEY_OP_SIGN | LibcryptoConst.EVP_PKEY_OP_VERIFY,
	LibcryptoConst.EVP_PKEY_CTRL_RSA_PSS_SALTLEN,
	-1,
	null()
	)
	handle_openssl_error(res)

Note how the salt length is only set if the version for libcrypto is less than 3. When libcrypto is at version 3, the default configuration is used during the signing which uses the maximum admissible length. This is not the expected behavior for oscrypto as the docs for rsa_pss_sign explicitly states:

The salt length with be the length of the hash algorithm