GH-34812: [Packaging][Python] Use self-hosted arm64 Linux runner instead of Travis CI for Linux arm64 wheels #34835

eitsupi · 2023-04-01T07:13:23Z

Rationale for this change

Change one of the jobs that uses Travis CI to run on GitHub Actions self-hosted runner, reducing the number of Travis CI jobs.

What changes are included in this PR?

Changes to use self-hosted arm64 linux runner for Crossbow jobs building wheels.

Are these changes tested?

Yes

Are there any user-facing changes?

No

Closes: [Packaging][Python] Use self-hosted arm64 runner instead of Travis CI for arm64 wheels #34812

Signed-off-by: SHIMA Tatsuya <[email protected]>

github-actions · 2023-04-01T07:13:41Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2023-04-01T07:14:25Z

Closes: [Packaging][Python] Use self-hosted arm64 runner instead of Travis CI for arm64 wheels #34812

github-actions · 2023-04-01T07:14:29Z

⚠️ GitHub issue #34812 has been automatically assigned in GitHub to PR creator.

eitsupi · 2023-04-01T07:19:28Z

@github-actions crossbow submit wheel-manylinux*

github-actions · 2023-04-01T07:22:01Z

Revision: b2a442c

Submitted crossbow builds: ursacomputing/crossbow @ actions-f5b646c499

Task	Status
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp311-amd64
wheel-manylinux2014-cp311-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64

eitsupi · 2023-04-01T08:00:52Z

@github-actions crossbow submit wheel-manylinux2014-cp39-arm64

github-actions · 2023-04-01T08:03:31Z

Revision: 5c06f77

Submitted crossbow builds: ursacomputing/crossbow @ actions-300f1c78f9

Task	Status
wheel-manylinux2014-cp39-arm64

Signed-off-by: SHIMA Tatsuya <[email protected]>

eitsupi · 2023-04-01T08:09:47Z

@github-actions crossbow submit wheel-manylinux2014-cp39-arm64

github-actions · 2023-04-01T08:12:26Z

Revision: a6d7db2

Submitted crossbow builds: ursacomputing/crossbow @ actions-8254ca5d1d

Task	Status
wheel-manylinux2014-cp39-arm64

Signed-off-by: SHIMA Tatsuya <[email protected]>

eitsupi · 2023-04-01T08:18:06Z

@github-actions crossbow submit wheel-manylinux2014-cp39-arm64

github-actions · 2023-04-01T08:20:32Z

Revision: 90ebfb3

Submitted crossbow builds: ursacomputing/crossbow @ actions-4cee463654

Task	Status
wheel-manylinux2014-cp39-arm64

eitsupi · 2023-04-01T08:35:19Z

@github-actions crossbow submit wheel-manylinux2014-cp39-arm64

github-actions · 2023-04-01T08:37:44Z

Revision: cac43b6

Submitted crossbow builds: ursacomputing/crossbow @ actions-8bf50b4a7c

Task	Status
wheel-manylinux2014-cp39-arm64

Signed-off-by: SHIMA Tatsuya <[email protected]>

eitsupi · 2023-04-01T13:09:37Z

@github-actions crossbow submit wheel-manylinux2014-cp39-arm64

github-actions · 2023-04-01T13:12:04Z

Revision: 778298c

Submitted crossbow builds: ursacomputing/crossbow @ actions-b19acd78ec

Task	Status
wheel-manylinux2014-cp39-arm64

eitsupi · 2023-04-01T15:01:29Z

@github-actions crossbow submit wheel-manylinux*

github-actions · 2023-04-01T15:04:11Z

Revision: 778298c

Submitted crossbow builds: ursacomputing/crossbow @ actions-e732e65c9b

Task	Status
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp311-amd64
wheel-manylinux2014-cp311-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64

kou

Should some of the jinja macros be removed in this PR as they seem to be used only in the file that will be removed in this PR?

Yes, please.

dev/tasks/python-wheels/github.linux.yml

kou · 2023-04-01T15:32:29Z

dev/tasks/python-wheels/github.linux.yml

@@ -49,6 +71,7 @@ jobs:
          archery docker run python-wheel-manylinux-test-imports
          archery docker run python-wheel-manylinux-test-unittests

+      {% if arch == "amd64" %}


Can we enable this on arm64 too?

I did not enable these in arm64 because Travis job did not seem having this, but I will try it out.

dev/tasks/tasks.yml

Signed-off-by: SHIMA Tatsuya <[email protected]>

eitsupi · 2023-04-01T17:29:02Z

@github-actions crossbow submit -g wheel

github-actions · 2023-04-01T17:32:41Z

Revision: 65eedb8

Submitted crossbow builds: ursacomputing/crossbow @ actions-4d259964be

Task	Status
wheel-clean
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-mojave-cp310-amd64
wheel-macos-mojave-cp311-amd64
wheel-macos-mojave-cp37-amd64
wheel-macos-mojave-cp38-amd64
wheel-macos-mojave-cp39-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp311-amd64
wheel-manylinux2014-cp311-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

kou

+1

ursabot · 2023-04-03T11:42:01Z

Benchmark runs are scheduled for baseline = ce94fba and contender = 34addbb. 34addbb is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Failed ⬇️0.0% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 34addbb5 ec2-t3-xlarge-us-east-2
[Failed] 34addbb5 test-mac-arm
[Finished] 34addbb5 ursa-i9-9960x
[Failed] 34addbb5 ursa-thinkcentre-m75q
[Finished] ce94fbab ec2-t3-xlarge-us-east-2
[Failed] ce94fbab test-mac-arm
[Finished] ce94fbab ursa-i9-9960x
[Failed] ce94fbab ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

…r instead of Travis CI for Linux arm64 wheels (apache#34835) ### Rationale for this change Change one of the jobs that uses Travis CI to run on GitHub Actions self-hosted runner, reducing the number of Travis CI jobs. ### What changes are included in this PR? Changes to use self-hosted arm64 linux runner for Crossbow jobs building wheels. ### Are these changes tested? Yes ### Are there any user-facing changes? No * Closes: apache#34812 Authored-by: SHIMA Tatsuya <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

use self-hosted runner for building python arm64 linux wheels

b2a442c

Signed-off-by: SHIMA Tatsuya <[email protected]>

github-actions bot added the awaiting review Awaiting review label Apr 1, 2023

eitsupi changed the title ~~[Packaging][Python] Use self-hosted arm64 runner instead of Travis CI for arm64 wheels~~ GH-34812: [Packaging][Python] Use self-hosted arm64 runner instead of Travis CI for arm64 wheels Apr 1, 2023

This comment was marked as resolved.

Sign in to view

fix python installation on arm64

a6d7db2

Signed-off-by: SHIMA Tatsuya <[email protected]>

eitsupi force-pushed the dev-python-arm64-self-hosted-runner branch from 5c06f77 to a6d7db2 Compare April 1, 2023 08:09

add /home/runner/.local/bin to PATH

90ebfb3

Signed-off-by: SHIMA Tatsuya <[email protected]>

This comment was marked as resolved.

Sign in to view

archery assumes arm64v8, not arm64

778298c

Signed-off-by: SHIMA Tatsuya <[email protected]>

eitsupi force-pushed the dev-python-arm64-self-hosted-runner branch from cac43b6 to 778298c Compare April 1, 2023 13:08

eitsupi marked this pull request as ready for review April 1, 2023 15:14

eitsupi requested review from assignUser and kou as code owners April 1, 2023 15:14

eitsupi requested a review from raulcd as a code owner April 1, 2023 15:14

This comment was marked as resolved.

Sign in to view

kou reviewed Apr 1, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes awaiting review Awaiting review and removed awaiting review Awaiting review labels Apr 1, 2023

eitsupi added 4 commits April 2, 2023 02:02

remove unused macros

9ff3c7f

Signed-off-by: SHIMA Tatsuya <[email protected]>

formatting

1b0cffd

Signed-off-by: SHIMA Tatsuya <[email protected]>

enable more tests on arm64

f93f521

Signed-off-by: SHIMA Tatsuya <[email protected]>

move some lines to macros.jinja

65eedb8

Signed-off-by: SHIMA Tatsuya <[email protected]>

github-actions bot added awaiting review Awaiting review awaiting committer review Awaiting committer review and removed awaiting review Awaiting review awaiting changes Awaiting changes labels Apr 1, 2023

eitsupi changed the title ~~GH-34812: [Packaging][Python] Use self-hosted arm64 runner instead of Travis CI for arm64 wheels~~ GH-34812: [Packaging][Python] Use self-hosted arm64 Linux runner instead of Travis CI for Linux arm64 wheels Apr 1, 2023

kou approved these changes Apr 3, 2023

View reviewed changes

kou merged commit 34addbb into apache:main Apr 3, 2023

github-actions bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Apr 3, 2023

kou mentioned this pull request Apr 3, 2023

GH-33804: [Python] Add support for manylinux_2_28 wheel #34818

Merged

eitsupi deleted the dev-python-arm64-self-hosted-runner branch April 3, 2023 12:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-34812: [Packaging][Python] Use self-hosted arm64 Linux runner instead of Travis CI for Linux arm64 wheels #34835

GH-34812: [Packaging][Python] Use self-hosted arm64 Linux runner instead of Travis CI for Linux arm64 wheels #34835

eitsupi commented Apr 1, 2023 •

edited

Loading

github-actions bot commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

This comment was marked as resolved.

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

This comment was marked as resolved.

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

This comment was marked as resolved.

kou left a comment

kou Apr 1, 2023

eitsupi Apr 1, 2023

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

kou left a comment

ursabot commented Apr 3, 2023

GH-34812: [Packaging][Python] Use self-hosted arm64 Linux runner instead of Travis CI for Linux arm64 wheels #34835

GH-34812: [Packaging][Python] Use self-hosted arm64 Linux runner instead of Travis CI for Linux arm64 wheels #34835

Conversation

eitsupi commented Apr 1, 2023 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

This comment was marked as resolved.

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

This comment was marked as resolved.

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

This comment was marked as resolved.

kou left a comment

Choose a reason for hiding this comment

kou Apr 1, 2023

Choose a reason for hiding this comment

eitsupi Apr 1, 2023

Choose a reason for hiding this comment

eitsupi commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

kou left a comment

Choose a reason for hiding this comment

ursabot commented Apr 3, 2023

eitsupi commented Apr 1, 2023 •

edited

Loading