email package: UnicodeEncodeError in as_bytes() when folding malformed Subject header mixing raw UTF‑8 and encoded‑words

[DRSpalding]

Bug report

Bug description:

While parsing a recent spam message, my script threw a UnicodeEncodeError exception from inside the email package. I reduced the problem to a minimal test case based on the real‑world (non‑compliant) Subject header. The header mixes raw UTF‑8 characters with RFC 2047 encoded‑words, which appears to cause the parser to produce surrogateescape code points when utf8=True is enabled.
Calling as_bytes() then triggers a failure during header folding.

I think that the package is mixing raw utf-8 handling with trying to wrap a header in =?unknown-8bit? sequences when utf-8 is true but should likely do something to treat the entirety of the header line as malformed and defensively fence it all off.

Below is a complete, self‑contained Python script that reproduces the error:

# This is a real-world spam subject line, and the testcase is distilled down
# to the form you see below with simply a subject, mime-version and short body.
# 
# This script reproduces a UnicodeEncodeError in the email package
# when parsing a header containing mixed encoded-words and raw UTF-8
# under policy.default.clone(utf8=True).  NB: the email is malformed
# because in mixes raw UTF-8 with encoded-words in the same header.
#
# Expected traceback:
#
expected_traceback = r"""
Traceback (most recent call last):
  File "C:\Tmp\PyTest\email_package_corrupted.py", line 32, in <module>
    print(msg.as_bytes())
          ~~~~~~~~~~~~^^
  File "C:\Program Files\Python\Lib\email\message.py", line 208, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Lib\email\generator.py", line 117, in flatten
    self._write(msg)
    ~~~~~~~~~~~^^^^^
  File "C:\Program Files\Python\Lib\email\generator.py", line 200, in _write
    self._write_headers(msg)
    ~~~~~~~~~~~~~~~~~~~^^^^^
  File "C:\Program Files\Python\Lib\email\generator.py", line 432, in _write_headers
    self._fp.write(self.policy.fold_binary(h, v))
                   ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "C:\Program Files\Python\Lib\email\policy.py", line 207, in fold_binary
    folded = self._fold(name, value, refold_binary=self.cte_type=='7bit')
  File "C:\Program Files\Python\Lib\email\policy.py", line 228, in _fold
    return self.header_factory(name, ''.join(lines)).fold(policy=self)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "C:\Program Files\Python\Lib\email\headerregistry.py", line 253, in fold
    return header.fold(policy=policy)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Lib\email\_header_value_parser.py", line 166, in fold
    return _refold_parse_tree(self, policy=policy)
  File "C:\Program Files\Python\Lib\email\_header_value_parser.py", line 2874, in _refold_parse_tree
    last_ew = _fold_as_ew(tstr, lines, maxlen, last_ew,
                          part.ew_combine_allowed, charset, leading_whitespace)
  File "C:\Program Files\Python\Lib\email\_header_value_parser.py", line 3006, in _fold_as_ew
    encoded_word = _ew.encode(to_encode_word, charset=encode_as)
  File "C:\Program Files\Python\Lib\email\_encoded_words.py", line 222, in encode
    bstring = string.encode('ascii', 'surrogateescape')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 4-5: ordinal not in range(128)
"""


from email import policy
from email import message_from_bytes, message_from_binary_file

# Raw email with NO Content-Type header
raw = (
	b'Subject: '
	b'xbox =?UTF-8?B?4puU77iP?= \xf0\x9d\x90\xa5\xf0\x9d\x90'
	b'\x9a\xf0\x9d\x90\xac\xf0\x9d\x90\xad \xf0\x9d\x90\xab\xf0\x9d\x90'
	b'\x9e\xf0\x9d\x90\xa6\xf0\x9d\x90\xa2\xf0\x9d\x90\xa7\xf0\x9d\x90'
	b'\x9d\xf0\x9d\x90\x9e\xf0\x9d\x90\xab! \xf0\x9d\x90\x98\xf0\x9d\x90'
	b'\xa8\xf0\x9d\x90\xae\xf0\x9d\x90\xab \xf0\x9d\x90\xa5\xf0\x9d\x90'
	b'\xa2\xf0\x9d\x90\x9c\xf0\x9d\x90\x9e\xf0\x9d\x90\xa7\xf0\x9d\x90\xac'
	b'\xf0\x9d\x90\x9e \xf0\x9d\x90\xa1\xf0\x9d\x90\x9a\xf0\x9d\x90\xac '
	b'\xf0\x9d\x90\x9e\xf0\x9d\x90\xb1\xf0\x9d\x90\xa9\xf0\x9d\x90\xa2\xf0'
	b'\x9d\x90\xab\xf0\x9d\x90\x9e\xf0\x9d\x90\x9d \xf0\x9d\x90\x93\xf0\x9d'
	b'\x90\xa8\xf0\x9d\x90\x9d\xf0\x9d\x90\x9a\xf0\x9d\x90\xb2 '
	b'=?UTF-8?B?4pqg77iP?=... \xf0\x9d\x90\xb2\xf0\x9d\x90\xa8\xf0\x9d\x90'
	b'\xae\xf0\x9d\x90\xab \xf0\x9d\x90\x9c\xf0\x9d\x90\xa8\xf0\x9d\x90\xa6'
	b'\xf0\x9d\x90\xa9\xf0\x9d\x90\xae\xf0\x9d\x90\xad\xf0\x9d\x90\x9e\xf0'
	b'\x9d\x90\xab \xf0\x9d\x90\xa2\xf0\x9d\x90\xac \xf0\x9d\x90\xa2\xf0\x9d'
	b'\x90\xa7 \xf0\x9d\x90\x9d\xf0\x9d\x90\x9a\xf0\x9d\x90\xa7\xf0\x9d\x90'
	b'\xa0\xf0\x9d\x90\x9e\xf0\x9d\x90\xab\r\n'
	b'MIME-Version: 1.0'
	b'\r\n\r\n'
	b'Hello, world!')


msg = message_from_bytes(raw, policy=policy.default.clone(utf8=True))

print('This call to as_bytes() should trigger the UnicodeEncodeError')
print(msg.as_bytes())


expected_traceback = r"""
This call to as_bytes() should trigger the UnicodeEncodeError
Traceback (most recent call last):
  File "C:\Tmp\PyTest\email_package_corrupted.py", line 32, in <module>
    print(msg.as_bytes())
          ~~~~~~~~~~~~^^
  File "C:\Program Files\Python\Lib\email\message.py", line 208, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Lib\email\generator.py", line 117, in flatten
    self._write(msg)
    ~~~~~~~~~~~^^^^^
  File "C:\Program Files\Python\Lib\email\generator.py", line 200, in _write
    self._write_headers(msg)
    ~~~~~~~~~~~~~~~~~~~^^^^^
  File "C:\Program Files\Python\Lib\email\generator.py", line 432, in _write_headers
    self._fp.write(self.policy.fold_binary(h, v))
                   ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "C:\Program Files\Python\Lib\email\policy.py", line 207, in fold_binary
    folded = self._fold(name, value, refold_binary=self.cte_type=='7bit')
  File "C:\Program Files\Python\Lib\email\policy.py", line 228, in _fold
    return self.header_factory(name, ''.join(lines)).fold(policy=self)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "C:\Program Files\Python\Lib\email\headerregistry.py", line 253, in fold
    return header.fold(policy=policy)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Lib\email\_header_value_parser.py", line 166, in fold
    return _refold_parse_tree(self, policy=policy)
  File "C:\Program Files\Python\Lib\email\_header_value_parser.py", line 2874, in _refold_parse_tree
    last_ew = _fold_as_ew(tstr, lines, maxlen, last_ew,
                          part.ew_combine_allowed, charset, leading_whitespace)
  File "C:\Program Files\Python\Lib\email\_header_value_parser.py", line 3006, in _fold_as_ew
    encoded_word = _ew.encode(to_encode_word, charset=encode_as)
  File "C:\Program Files\Python\Lib\email\_encoded_words.py", line 222, in encode
    bstring = string.encode('ascii', 'surrogateescape')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 4-5: ordinal not in range(128)
"""

CPython versions tested on:

3.14

Operating systems tested on:

Windows

[a12k]

I can repro this. I took a look at the email package. There's a simple bandage you could apply, in the _fold_as_ew function I did:

try:
    to_encode_word.encode('ascii')
except UnicodeEncodeError:
    to_encode_word = to_encode_word.encode('utf-8', 'surrogateescape').decode('utf-8', 'replace')

To smoke test it, then convert those internal \udcxx surrogates and turn them back into the original raw bytes they were before parsing, then decode back to string. This works in the repl

... from email import policy
... from email import message_from_bytes, message_from_binary_file
...
... # Raw email with NO Content-Type header
... raw = (
...     b'Subject: '
   ...  b'xbox =?UTF-8?B?4puU77iP?= \xf0\x9d\x90\xa5\xf0\x9d\x90'
   ...  b'\x9a\xf0\x9d\x90\xac\xf0\x9d\x90\xad \xf0\x9d\x90\xab\xf0\x9d\x90'
   ...  b'\x9e\xf0\x9d\x90\xa6\xf0\x9d\x90\xa2\xf0\x9d\x90\xa7\xf0\x9d\x90'
   ...  b'\x9d\xf0\x9d\x90\x9e\xf0\x9d\x90\xab! \xf0\x9d\x90\x98\xf0\x9d\x90'
   ...  b'\xa8\xf0\x9d\x90\xae\xf0\x9d\x90\xab \xf0\x9d\x90\xa5\xf0\x9d\x90'
   ...  b'\xa2\xf0\x9d\x90\x9c\xf0\x9d\x90\x9e\xf0\x9d\x90\xa7\xf0\x9d\x90\xac'
   ...  b'\xf0\x9d\x90\x9e \xf0\x9d\x90\xa1\xf0\x9d\x90\x9a\xf0\x9d\x90\xac '
   ...  b'\xf0\x9d\x90\x9e\xf0\x9d\x90\xb1\xf0\x9d\x90\xa9\xf0\x9d\x90\xa2\xf0'
   ...  b'\x9d\x90\xab\xf0\x9d\x90\x9e\xf0\x9d\x90\x9d \xf0\x9d\x90\x93\xf0\x9d'
   ...  b'\x90\xa8\xf0\x9d\x90\x9d\xf0\x9d\x90\x9a\xf0\x9d\x90\xb2 '
   ...  b'=?UTF-8?B?4pqg77iP?=... \xf0\x9d\x90\xb2\xf0\x9d\x90\xa8\xf0\x9d\x90'
   ...  b'\xae\xf0\x9d\x90\xab \xf0\x9d\x90\x9c\xf0\x9d\x90\xa8\xf0\x9d\x90\xa6'
   ...  b'\xf0\x9d\x90\xa9\xf0\x9d\x90\xae\xf0\x9d\x90\xad\xf0\x9d\x90\x9e\xf0'
   ...  b'\x9d\x90\xab \xf0\x9d\x90\xa2\xf0\x9d\x90\xac \xf0\x9d\x90\xa2\xf0\x9d'
   ...  b'\x90\xa7 \xf0\x9d\x90\x9d\xf0\x9d\x90\x9a\xf0\x9d\x90\xa7\xf0\x9d\x90'
   ...  b'\xa0\xf0\x9d\x90\x9e\xf0\x9d\x90\xab\r\n'
   ...  b'MIME-Version: 1.0'
   ...  b'\r\n\r\n'
   ...  b'Hello, world!')
   ...
   ...
   ... msg = message_from_bytes(raw, policy=policy.default.clone(utf8=True))
   ...
   ... print('This call to as_bytes() should trigger the UnicodeEncodeError')
   ... print(msg.as_bytes())
...
This call to as_bytes() should trigger the UnicodeEncodeError
b'Subject: xbox \xe2\x9b\x94\xef\xb8\x8f =?unknown-8bit?b?8J2QpfCdkJrwnZCs8J2QrQ==?=\n =?unknown-8bit?b?8J2QmvCdkKzwnZCt?=\n =?unknown-8bit?b?77+98J2QrPCdkK0=?=\n =?unknown-8bit?b?77+977+98J2QrQ==?=\n =?unknown-8bit?b?77+977+977+9IPCdkKvwnZCe8J2QpvCdkKLwnZCn8J2QnfCdkJ7wnZCr?=\n =?unknown-8bit?b?77+977+98J2QnvCdkKbwnZCi8J2Qp/CdkJ3wnZCe8J2QqyE=?=\n =?unknown-8bit?b?8J2QovCdkKfwnZCd8J2QnvCdkKsh?=\n =?unknown-8bit?b?77+977+98J2QnfCdkJ7wnZCrIQ==?=\n =?unknown-8bit?b?8J2QnvCdkKsh?=\n =?unknown-8bit?b?77+98J2QqyE=?=\n =?unknown-8bit?b?77+977+9ISDwnZCY8J2QqPCdkK7wnZCr?=\n =?unknown-8bit?b?8J2QmPCdkKjwnZCu8J2Qqw==?=\n =?unknown-8bit?b?8J2QqPCdkK7wnZCr?=\n =?unknown-8bit?b?77+98J2QrvCdkKs=?=\n =?unknown-8bit?b?77+977+98J2Qqw==?=\n =?unknown-8bit?b?77+977+977+9IPCdkKXwnZCi8J2QnPCdkJ7wnZCn8J2QrPCdkJ4=?=\n =?unknown-8bit?b?77+977+977+98J2QovCdkJzwnZCe8J2Qp/CdkKzwnZCe?=\n =?unknown-8bit?b?77+977+98J2QnvCdkKfwnZCs8J2Qng==?=\n =?unknown-8bit?b?8J2Qp/CdkKzwnZCe?=\n =?unknown-8bit?b?77+98J2QrPCdkJ4=?=\n =?unknown-8bit?b?77+977+98J2Qng==?=\n =?unknown-8bit?b?77+977+977+9IPCdkKHwnZCa8J2QrA==?=\n =?unknown-8bit?b?77+977+9IPCdkKHwnZCa8J2QrA==?=\n =?unknown-8bit?b?77+98J2QmvCdkKw=?=\n =?unknown-8bit?b?77+977+98J2QrA==?=\n =?unknown-8bit?b?77+977+977+9IPCdkJ7wnZCx8J2QqfCdkKLwnZCr8J2QnvCdkJ0=?=\n =?unknown-8bit?b?77+977+977+98J2QsfCdkKnwnZCi8J2Qq/CdkJ7wnZCd?=\n =?unknown-8bit?b?77+977+98J2QovCdkKvwnZCe8J2QnQ==?=\n =?unknown-8bit?b?8J2Qq/CdkJ7wnZCd?=\n =?unknown-8bit?b?77+98J2QnvCdkJ0=?=\n =?unknown-8bit?b?77+977+98J2QnQ==?=\n =?unknown-8bit?b?77+977+977+9IPCdkJPwnZCo8J2QnfCdkJrwnZCy?=\n =?unknown-8bit?b?IPCdkJPwnZCo8J2QnfCdkJrwnZCy?=\n =?unknown-8bit?b?77+977+977+98J2QnfCdkJrwnZCy?=\n =?unknown-8bit?b?77+98J2QmvCdkLI=?=\n =?unknown-8bit?b?77+977+98J2Qsg==?=\n =?unknown-8bit?b?77+977+977+9IOKaoO+4jy4uLiDwnZCy8J2QqPCdkK7wnZCr?=\n =?unknown-8bit?b?LiDwnZCy8J2QqPCdkK7wnZCr?=\n =?unknown-8bit?b?8J2QqPCdkK7wnZCr?=\n =?unknown-8bit?b?77+98J2QrvCdkKs=?=\n =?unknown-8bit?b?77+977+98J2Qqw==?=\n =?unknown-8bit?b?77+977+977+9IPCdkJzwnZCo8J2QpvCdkKnwnZCu8J2QrfCdkJ7wnZCr?=\n =?unknown-8bit?b?77+977+98J2QqPCdkKbwnZCp8J2QrvCdkK3wnZCe8J2Qqw==?=\n =?unknown-8bit?b?77+98J2QqfCdkK7wnZCt8J2QnvCdkKs=?=\n =?unknown-8bit?b?77+977+977+98J2QrfCdkJ7wnZCr?=\n =?unknown-8bit?b?77+98J2QnvCdkKs=?=\n =?unknown-8bit?b?77+977+98J2Qqw==?=\n =?unknown-8bit?b?77+977+977+9IPCdkKLwnZCs?=\n =?unknown-8bit?b?77+9IPCdkKLwnZCs?=\n =?unknown-8bit?b?8J2QovCdkKw=?=\n =?unknown-8bit?b?77+977+98J2QrA==?=\n =?unknown-8bit?b?77+977+977+9IPCdkKLwnZCn?=\n =?unknown-8bit?b?77+9IPCdkKLwnZCn?=\n =?unknown-8bit?b?8J2QovCdkKc=?=\n =?unknown-8bit?b?77+977+98J2Qpw==?=\n =?unknown-8bit?b?77+977+977+9IPCdkJ3wnZCa8J2Qp/CdkKDwnZCe8J2Qqw==?=\n =?unknown-8bit?b?8J2QnfCdkJrwnZCn8J2QoPCdkJ7wnZCr?=\n =?unknown-8bit?b?77+977+98J2Qp/CdkKDwnZCe8J2Qqw==?=\n =?unknown-8bit?b?8J2QoPCdkJ7wnZCr?=\n =?unknown-8bit?b?77+98J2QnvCdkKs=?=\n =?unknown-8bit?b?77+977+98J2Qqw==?=\n =?unknown-8bit?b?77+977+977+9?=\nMIME-Version: 1.0\n\nHello, world!'

While that patch works, I expect it's a little "whack-a-mole" and an actual fix is to make email/_encoded_words.py "surrogate aware" so we prevent this class of bugs regardless of which submodule does it. But that's an exercise for whoever gets assigned to fix this. :)

[picnixz]

Is this allowed by RFC 6532? because you set utf8=True adn the docs say:

If False, follow RFC 5322, supporting non-ASCII characters in headers by encoding them as “encoded words”. If True, follow RFC 6532 and use utf-8 encoding for headers. Messages formatted in this way may be passed to SMTP servers that support the SMTPUTF8 extension (RFC 6531).

[a12k]

This took longer to research than I expected, but I think yes a fix would be allowed by RFC 6532. Postel's Law aside (and despite my personal disagreement in with its principle in OSS):

Section 3.6 "Use of MIME Encoded-Words" states "Agents MAY, when incorporating material from another message, convert encoded-word use to direct use of UTF-8. (emphasis mine)" By fixing this By fixing the bug we would be performing the conversion explicitly authorized by this section.

Further, Section 3.2 "Syntax Extensions to RFC 5322" states that "constructs now allow UTF-8: 1. Unstructured text, used in header fields like 'Subject:'..." which in my reading makes the raw portions of the header in the repro script valid.

Back to Section 3.6, "...encoded-words SHOULD NOT be used when generating header fields for messages employing this extension." So the implication is that prohibition is not a mandate for cpython to crash upon receiving, but a mandate to generate utf-8.

Together, my combined read is that the repro mixing raw UTF-8 and encoded words is a "generation" error by the sender (violating the "SHOULD NOT"), but RFC 6532 Section 3.6 grants the receiver (read, cpython email package) permission to resolve this by converting the encoded parts to UTF-8. So I would advocate for handling this gracefully rather than crashing.

That being said, if fixing this is a direction we want to pursue, I did some more digging into the code and believe I could create a simple PR to handle this.

[picnixz]

The question is also whether this would work with SMTP servers:

Messages formatted in this way may be passed to SMTP servers that support the SMTPUTF8 extension

Together, my combined read is that the repro mixing raw UTF-8 and encoded words is a "generation" error by the sender (violating the "SHOULD NOT"), but RFC 6532 Section 3.6 grants the receiver (read, cpython email package) permission to resolve this by converting the encoded parts to UTF-8. So I would advocate for handling this gracefully rather than crashing.

Then I read it differently. Because this would be breaking other users that correctly use this. When there is a "MAY" in an RFC, it means that we could but that's optional. It's not a bug to me, but rather a feature request which I personally don't find attractive enough. It's prone to breaking validation downstream. If we want to allow additional support, we should either create an additional policy object for that, or have a strict=True by default for the policy for which strict=False would mean "try harder".

cc @bitdancer

[a12k]

Yeah agree that RFC 6532 makes the conversion optional ("MAY"), but I don't think it justifies the current failure mode. My take is that we are crashing in violation of the email package's API contract. The default policy (raise_on_defect=False) implies that malformed input should result in a email.errors.MessageDefect, not an uncaught UnicodeEncodeError traceback.

By crashing, we currently force an implicit strict=True behavior for this specific edge case. That prevents users from inspecting the message or handling the error gracefully.

Reflecting further, my suggestion isn't that we silently fix the input, it's that we catch the encoding error, register a MessageDefect (signaling the RFC violation), and fallback to a safe encoding (e.g., treating the segment as raw UTF-8 as permitted by Section 3.6). This preserves validation capabilities (via defects) without crashing the runtime.

[bitdancer]

I fixed this in #142517 so that it correctly re-encodes the unknown bytes as unknown-8bit but leaves the utf-8 alone if utf8=True. Please re-run your tests against main.

I was not aware of the RFC 6532 discouragement of encoded words when using the SMTPUTF8 extension. I may have been when I originally wrote the code, but that was a while ago and I at least didn't remember when writing the fix. That said, SHOULD NOT is not MUST NOT, which means that smtp servers must be prepared to handle encoded words in SMTPUTF8 mode. There is also the issue of dealing with signed message attachments and rfc822 attachments that shouldn't normally be modified when resent, both of which potentially require handling encoded words even if in SMPTUTF8 mode.

I think my fix is as correct as we're going to get, but I'm open to argument. The fix hasn't been released anywhere yet, so we can still change this if we decide we want to.

However this report demonstrates a deeper problem with the current support for utf8=True. On parse those utf-8 characters should have been decoded, so that the end result of refolding that header should have been all utf8. The fix is simple (checking the policy flag and applying the correct charset on decode in the various parser functions). I'm surprised that this hasn't been reported before.

How about we change this issue to "utf8 is not correctly decoded when parsing binary messages with utf=True"? We can still discuss whether or not my fix in #142517 was correct or not, since the correct context for that discussion is with this parsing issue resolved.

[bitdancer]

Oh, and also, the other reason for this bug surfacing is the incorrect decision I made originally to make the default value of 'refold_source' be 'long' instead of 'none'. I'd love to change that default, but it would have backward compatibility implications, so I'm not sure we can.

[picnixz]

The fix hasn't been released anywhere yet, so we can still change this if we decide we want to.

Release is tomorrow I think. And the backports were already merged right?

cc @hugovk