Update Python 3.x installation flags

[simonsan]

Tool information

• Tool name: Python 3.x (Debug/Development)
• Add or update? Update/Add
• Desired version: Development, Debug
• Approximate size:
• If this is an add request:
  • Brief description of tool: Python is an interpreted, high-level, general-purpose programming language.
  • URL for tool's homepage: https://www.python.org/

Area for Triage:
Python, Packages

Question, Bug, or Feature?:
Feature

Virtual environments affected

• Windows Server 2016 R2
• Windows Server 2019

Can this tool be installed during the build?
Uninstall other Python versions or unset/reset PATH without any Python directories.

cmd:
IF NOT EXIST "%GITHUB_WORKSPACE%\Python38-x64-dbg\python.exe" curl --create-dirs -o "%GITHUB_WORKSPACE%\python-for-openage\python-3.8.0-amd64.exe" "https://www.python.org/ftp/python/3.8.0/python-3.8.0-amd64.exe"

%GITHUB_WORKSPACE%\python-for-openage\python-3.8.0-amd64.exe /quiet /uninstall

setup.exe /quiet Include_debug=1 Include_dev=1 Include_lib=1 Include_pip=1 PrependPath=1 CompileAll=1 InstallAllUsers=1 TargetDir=<workspace-dir>\Python38-x64-dbg\

Are you willing to submit a PR?
Yes

LINK : fatal error LNK1104: cannot open file 'python38_d.lib'
We need a Python interpreter that is shipped with libpython and header files.

So basically it could be just working as a replacement for the usual Python installations. (although I'm not sure due to the CompileAll flag that compiles .py into .pyc. Don't know really if they are left as a .py as well or if they are removed and replaced.

Since ever I need to download the Python setup and deinstall with it the same version from the CI just to install with the beforementioned commands. Maybe this could be the chance to finally spare this tasks :-/

cc: actions/setup-python#86

[maxim-lobanov]

Hello @simonsan ,
Currently, we install Python with default flags (just override installation dir): https://docs.python.org/3/using/windows.html#installing-without-ui

Am I right, that you request is just enabling Include_debug to 1. Other flags looks like already enabled

[simonsan]

Indeed, only changes would be:
Include_debug=1
CompileAll=1

I checked it and nothing gets overwritten and should be fully compatible with existing Python installations (the compiled files coexist inside the directory).

[konradpabjan]

In terms of compatibility there shouldn't be a problem. The biggest concern that I have is regards to the new size.

Locally I tested with Python 3.8 and a standard installation would report Size: 120MB and Size on Disk: 137MB. After updating the installation with the debug binaries and with the compile all option, it jumped up to Size: 221MB and Size on Disk: 246 MB

Updating the installation flags for all the 3.X versions would be a 400+ GB MB increase in size and I'm not to sure if it's worth it.

If we include this for Windows, then it would make sense to do this as well for Linux & Mac as well. It doesn't appear we're currently using --with-pydebug
https://devguide.python.org/setup/#compile-and-build

cc @brcrista

[simonsan]

@konradpabjan I don't have any problems on Linux and Mac (Brew) as they both have Package Managers that where I can add these easily.

It's as easy as:
apt-get install python3-dev

But I understand the size issue. What about having just the newest Python version of each Minor version (3.7x, 3.8.x) with these libraries to deal with this problem. That would be just two installations not all 3.x.

Update: Also I feel like 400+GiB in size is a bit much?

If we have 109 MiB per Installation more and you have 10 Installations on the Win runner that's 1 GiB more for me. Having the setups stored as well would be another GiB more, so we are talking about 2 GiB more overall here - at maximum. Or how did you calculate the 400+GiB in size? Am I overlooking something?

[simonsan]

For others, that probably need it and don't want to work around, this I do currently to get the Debug version for development with a -DPYTHON_DIR flag for CMake to ship around the Python in PATH.

I as well opened an issue in actions/setup-python#86 because I think it would be also fine, if the the debug versions (let's really just say of the last two newest major versions, 3.7x/3.8.x) could be reconfigured using actions/setup-python from the hostedtoolchache. I (and suppose others?) would be just happy, if it would be easier than this:

      - name: Cache (Python)
        uses: actions/[email protected]
        id: cache-python
        with:
          path: '${{ env.PYTHON_INSTALL_DIR }}\'
          key: '${{ runner.os }}-PythonCache-v${{ matrix.python-version }}-${{ matrix.python-platform }}-dbg'

      - name: Download Python
        uses: carlosperate/[email protected]
        id: download-python
        with:
         file-url: 'https://www.python.org/ftp/python/${{ matrix.python-version }}/python-${{ matrix.python-version }}-${{ matrix.python-platform }}.exe'
         file-name: 'python-${{ matrix.python-version }}-${{ matrix.python-platform }}.exe'
         location: '${{ runner.workspace }}\dl\'
        if: steps.cache-python.outputs.cache-hit != 'true'

      - name: Install Python debug (x64) # we need debug binaries and precompiled standard libraries
        run: |
          Start-Process -FilePath "${{ runner.workspace }}\dl\python-${{ matrix.python-version }}-${{ matrix.python-platform }}.exe" -ArgumentList "/quiet Include_debug=1 Include_dev=1 Include_lib=1 Include_pip=1 PrependPath=1 CompileAll=1 InstallAllUsers=0 TargetDir=${{ env.PYTHON_INSTALL_DIR }}\" -Verb runas -Wait
        if: matrix.arch == 'x64' && steps.cache-python.outputs.cache-hit != 'true'
        shell: pwsh

      - name: Display Python version
        run: ${{ env.PYTHON_INSTALL_DIR }}\python.exe -c "import sys; print(sys.version)"
        shell: cmd

      - name: Upgrade Python pip
        run: |
          ${{ env.PYTHON_INSTALL_DIR }}\python.exe -m pip install --upgrade pip
        shell: cmd

      - name: Install Python dependencies
        run: ${{ env.PYTHON_INSTALL_DIR }}\Scripts\pip.exe install -r ".github/workflows-support/requirements.txt"
        if: steps.cache-python.outputs.cache-hit != 'true'

[konradpabjan]

400+ MB, not GB, my bad. (246MB - 137MB = 109MB) We have 3.5, 3.6, 3,7 and 3.8 on each image so updating the flags for each would be 109 * 4 would be just over 436MB. Including both (x86 and x64) we would get around 872MB.

We have to be very careful and considerate in terms of what goes on the image because there is a limited amount of storage. Some recent issues have come up with not enough storage being available on Ubuntu due to how much software is installed (see #709) so we have to be careful. You can see in the issue that we have over 70GB of various software installed (this number is slightly different for windows). Some considerations that we follow: https://github.com/actions/virtual-environments#software-guidelines

---

I think the solution here is to maybe have something on the setup-python side since that doesn't have any impact in terms of what gets pre-installed on the image.

We have a new feature that is on the way that will allow for extra versions of python to be downloaded using setup-python. See actions/setup-python#85
Some modifications in the future could possibly be made in terms of how certain packages get installed so a user who wants the extra binaries could get them

[alepauly]

thanks @simonsan for the suggestion and @konradpabjan for the investigation! I agree that this seems something that should be handled in https://github.com/actions/setup-python which can probably be taught to serve debug versions as needed. @simonsan feel free to create an issue/PR there.