gh-46236: Add missing PyUnicode_GetDefaultEncoding() doc

[rruuaanng]

• Issue: Document PyUnicode_* API #46236

---

📚 Documentation preview 📚: https://cpython-previews--130335.org.readthedocs.build/

[rruuaanng]

cc @vstinner

[rruuaanng]

As usual, the documentation does not require a NEWS (I hope I remember correctly).

I'm aware that this function returns a utf-8. Perhaps describing the function in this way would make it more robust (at the very least, we won't need to update its documentation if we modify it in the future).

[encukou]

I'd link to sys.getdefaultencoding, copy the doc from there, and add a .. impl-detail:: that it's always utf-8.

Perhaps this should be soft-deprecated.

[rruuaanng]

Alright, something interesting happened. sys.getdefaultencoding always returns utf-8, but it's description says:

Return the name of the current default string encoding used by the Unicode implementation.

Should I modify its description? e.g., Return a "utf-8" string constant.

copy the doc from there, and add a .. impl-detail:: that it's always utf-8.

In other words, should I add .. impl-detail:: tag to sys.getdefaultencoding to point it to the PyUnicode_GetDefaultEncoding function? But here's the interesting part, it uses a completely new implementation, like the one below:

static PyObject *
get_utf8_unicode(void)
{
    _Py_DECLARE_STR(utf_8, "utf-8");
    PyObject *ret = &_Py_STR(utf_8);
    return Py_NewRef(ret);
}

Even so, do I still need to add this tag?

[encukou]

Hm, looking further, looks like the “current default string encoding used by the Unicode implementation” is what str.encode() uses. In Python 2, it was ascii; in Python 3 it's utf-8.

(click for Python 2 behaviour)

>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> u'á'.encode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 0: ordinal not in range(128)

>>> import ctypes
>>> ctypes.pythonapi.PyUnicodeUCS4_SetDefaultEncoding("utf-8")  # hack!
0

>>> sys.getdefaultencoding()
'utf-8'
>>> 'á'.encode()
'\xc3\xa1'

It might be good to clarify this in the sys.getdefaultencoding docs, e.g.:

Return "utf-8". This is the name of the default string encoding, used in methods like str.encode.

(So, UTF-8 is not a CPython implementation detail.)

[encukou]

it uses a completely new implementation

That's needed to get a Python object, as opposed to a char*

[rruuaanng]

Hmm, maybe I should open a separate PR to do it.

@encukou Please review! I hope it meets your expectations. #130353

[encukou]

I'd prefer including the sys.getdefaultencoding change here, so that the PyUnicode_GetDefaultEncoding doc can refer to it.

[encukou]

I suggest this wording, plus making lifetime explicit since we're returning a buffer.
(Currently the string is static of course, but if it'd ever become dynamic again, it'd need to be borrowed from the interpreter.)

Suggested change

	Return a ``"utf-8"`` string constant, which corresponds to the
	:func:`~sys.getdefaultencoding` function in Python.
	Return the name of the default string encoding, ``"utf-8"``.
	See :func:`sys.getdefaultencoding`.
	The returned string does not need to be freed, and is valid
	until interpreter shutdown.