ECDSA verify fails on windows when R,S values both have leading zeroes.

[shane-kearns]

Initially observed that some ECDSA certificates are considered to have invalid signatures by certvalidator but which are valid according to windows certutil.exe and openssl.

The root cause is the marshalling of R, S values for calling BCryptVerifySignature here:

oscrypto/oscrypto/_win/asymmetric.py

Lines 2629 to 2649 in 1547f53

	else:
	# Windows doesn't use the ASN.1 Sequence for DSA/ECDSA signatures,
	# so we have to convert it here for the verification to work
	try:
	signature = DSASignature.load(signature).to_p1363()
	except (ValueError, OverflowError, TypeError):
	raise SignatureError('Signature is invalid')
	res = bcrypt.BCryptVerifySignature(
	certificate_or_public_key.key_handle,
	padding_info,
	digest,
	len(digest),
	signature,
	len(signature),
	flags
	)
	failure = res == BcryptConst.STATUS_INVALID_SIGNATURE
	failure = failure or res == BcryptConst.STATUS_INVALID_PARAMETER
	if failure:
	raise SignatureError('Signature is invalid')

Because oscrypto passes a 62 byte signature where a 64 byte signature is expected (in the case of NIST p256 keys), windows returns STATUS_INVALID_PARAMETER, which is treated as an invalid signature at line 2647.

The DSASignature.to_p1363 function does not know the expected key size:
https://github.com/wbond/asn1crypto/blob/b763a757bb2bef2ab63620611ddd8006d5e9e4a2/asn1crypto/algos.py#L720-L736
It can't, since the (EC)DSA signature ASN1 is just a sequence of two Integers. The context of what the signature algorithm and key size are is held at a higher level.

The required length can be determined from the public key used for verification.
e.g. certificate_or_public_key.byte_size is 32 for an ECDSA p256 key (since it's encoded as a compressed point), which is also the size of the private key and thereforre the required size for each of the R and S values.

Following script usually reproduces the problem within the 100k iterations

from asn1crypto.algos import DSASignature
from oscrypto import backend, platform
from oscrypto.asymmetric import ecdsa_sign, ecdsa_verify, generate_pair
from oscrypto.errors import SignatureError
import secrets

def check():
    public, private = generate_pair("ec", curve="secp256r1")
    data = secrets.token_bytes(128)
    signature = ecdsa_sign(private, data, "sha256")
    try:
        ecdsa_verify(public, signature, data, "sha256")
    except SignatureError:
        print("\n\nBad signature!")
        print("public key:", public.asn1.dump().hex())
        print("private key:", private.asn1.dump().hex())
        print("data:", data.hex())
        print("signature:", signature.hex())
        p1363 = DSASignature.load(signature).to_p1363()
        print(f"P1363 signature: {p1363.hex()} ({len(p1363)} bytes)")
        raise

print("testing on", platform.platform(), "with backend", backend())
for i in range(100000):
    if(i % 7800 == 0):
        print()
    if(i % 100 == 0):
        print('.', end="", flush=True)
    check()

[ArneBachmannDLR]

I can reproduce this even faster, fails 1/3 to 1/10 invocations, is maybe timing-based?

import base64, json
from oscrypto import asymmetric

pub, pri = asymmetric.generate_pair("ec", curve="secp521r1")

HASH_METHOD = "sha256"
DATA = {
  "installations": "once",
  "validity": "2025-04-10",
  "restrictions": {},
  "group": "test",
  "name": "test",
  "version": "test",
  "replaces": [],
  "sources": [],
  "commands": [],
  "links": []
}

# Sign
jb = json.dumps(DATA, skipkeys=True, allow_nan=False, indent=None, separators=(',', ':'), sort_keys=True).encode('utf-8')
signature = base64.b64encode(asymmetric.ecdsa_sign(pri, jb, HASH_METHOD)).decode('ascii')

# Verify
asymmetric.ecdsa_verify(pub, base64.b64decode(signature.encode('ascii')), jb, HASH_METHOD)

[shane-kearns]

I think because you're using the p521 curve, the chances of a leading zero are much higher (1 in 2 for R/S and 1/4 overall) rather than 1/256 each and 1/65536 overall for p256 / p384.

For patching, this is the correct marshalling:

    def fixed_p1363(signature, key_byte_size):
        """
        Dumps a signature to a byte string compatible with Microsoft's
        BCryptVerifySignature() function.

        :return:
            A byte string compatible with BCryptVerifySignature()
        """
        asn1_signature = DSASignature.load(signature)
        r = int_to_bytes(asn1_signature['r'].native, width=key_byte_size)
        s = int_to_bytes(asn1_signature['s'].native, width=key_byte_size)
        return r + s

and called like this (at line 2633 in the original code quoted):

signature = fixed_p1363(signature, certificate_or_public_key.byte_size)

The signature size should be 64 / 96 / 130 bytes, R and S should each be equal to the key size.