gh-83938, gh-122476: Stop incorrectly RFC 2047 encoding non-ASCII email addresses

[medmunds]

This PR fixes gh-83938 and fixes gh-122476, which have the same underlying issue.

Email generators had been incorrectly flattening non-ASCII email addresses to RFC 2047 encoded-word format, leaving them undeliverable. (RFC 2047 prohibits use of encoded-word in an addr-spec.) This change raises a ValueError when attempting to flatten an EmailMessage with a non-ASCII addr-spec and a policy with utf8=False. (Exception: If the non-ASCII address originated from parsing a message, it will be flattened as originally parsed, without error.)

Non-ASCII email addresses are supported when using a policy with utf8=True (such as email.policy.SMTPUTF8) under RFCs 6531 and 6532.

Non-ASCII email address domains (but not localparts) can also be used with non-SMTPUTF8 policies by encoding the domain as an IDNA A-label. (The email package does not perform this encoding, because it cannot know whether the caller wants IDNA 2003, IDNA 2008, or some other variant such as UTS-46.)

• Issue: EmailMessage bad encoding for international domain #83938

• Issue: EmailMessage bad encoding for non-ASCII localpart #122476

---

📚 Documentation preview 📚: https://cpython-previews--122540.org.readthedocs.build/

[medmunds]

This is based on #81074 (comment):

we should probably be raising an error if the rendering policy does not have utf8=True and we don't have an "original source line" from parsing a message (which is the case here), rather than using the incorrect RFC2047 encoding.

Checking part.token_type == 'addr-spec' seemed like the simplest approach.

An alternative would be to introduce a new NonASCIIDomainLiteralDefect paralleling NonASCIILocalPartDefect and apply it in _header_value_parser.get_domain_literal(). And add NonASCIIAddrSpecDefect as a superclass of both. Then change _refold_parse_tree() to check any(isinstance(d, NonASCIIAddrSpecDefect) for d in part.all_defects) (and perhaps move it up with the other UnicodeEncodeError logic). (If we go this direction, PR #122477 will also need an update.)

Also, I think charset == 'unknown-8bit' is only possible in _refold_parse_tree() when the non-ASCII characters resulted from parsing an existing message: see the UndecodableBytesDefect logic just above the new code. (The added tests seem to confirm this.)

[bitdancer]

Your analysis is solid and the fix looks great. We'll need a follow on PR to have smtplib handle the new error, but that should be a trivial PR.

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be poked with soft cushions!

[medmunds]

I have made the requested changes; please review again

[bedevere-app]

Thanks for making the requested changes!

@bitdancer: please review the changes made to this pull request.

[bitdancer]

Sorry it took me so long to get back to this.

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[medmunds]

We'll need a follow on PR to have smtplib handle the new error, but that should be a trivial PR.

It looks like smtplib's send_message() already detects (some) non-ASCII mailboxes and if necessary, automatically clones a policy with utf8=True and requires SMTPUTF8. The cases handled by smtplib's current code should never get an InvalidMailboxError, so I'm not sure what would be needed.

There's a potential enhancement, but it's a (little) bit more involved. send_message()'s international detection doesn't check all address headers—e.g., an IDN reply-to would fool it. (And its from_addr and to_addrs params make it ignore the corresponding message headers.) That logic could be made more robust by serializing unconditionally with the message's own policy, and if that raises InvalidMailboxError (or already has utf8 True) falling into the SMTPUTF8-required code.

[medmunds]

I have made the requested changes; please review again

[bedevere-app]

Thanks for making the requested changes!

@bitdancer: please review the changes made to this pull request.

[github-actions]

This PR is stale because it has been open for 30 days with no activity.

[bitdancer]

Well, it turns out that this bug goes a bit deeper than apparent. addr_spec was already supposed to be prevented from being encoded, but the logic around as_ew_allowed is broken: it doesn't "look inside" the token it is encoded to make sure all of it can be encoded. The fix in this PR also has its problems, as proven by the test failures after merge with main: want_encoding can be set when what is wanted is wrapping, rather than needing encoding. Those should probably have been two separate flags...but at this point the folder is a bit of a patchwork mess. (I plan to fix that...someday).

If we fix as_eq_allowed handling I think it will make those test failures go away, as well as simplifying the logic for raising the error, but it won't be certain until we try it.

There is another test failure as well, but I think that's not a bug, but rather a change in behavior that the test needs to adapt to. I haven't proven that yet, though :)

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.