Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added configuration option for not doing chunked upload for HTTP remotes #6117

Closed
wants to merge 8 commits into from
Closed

Added configuration option for not doing chunked upload for HTTP remotes #6117

wants to merge 8 commits into from

Conversation

scotty3005
Copy link

@scotty3005 scotty3005 commented Jun 4, 2021 •
edited
Loading

Changes to be committed:
modified: dvc/config_schema.py
modified: dvc/fs/http.py

Draft proposal for feature request iterative/dvc-http#9

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

scotty3005 and others added 4 commits June 4, 2021 16:54
@scotty3005
Added configuration option for not doing chunked upload
8fc1cae 
for HTTP remotes

Changes to be committed:
	modified:   .codecov.yml
	modified:   .coveragerc
	modified:   .dvc/.gitignore
	modified:   .dvc/config
	modified:   .github/FUNDING.yml
	modified:   .github/ISSUE_TEMPLATE/bug_report.md
	modified:   .github/ISSUE_TEMPLATE/config.yml
	modified:   .github/ISSUE_TEMPLATE/feature_request.md
	modified:   .github/PULL_REQUEST_TEMPLATE.md
	modified:   .github/dependabot.yml
	modified:   .github/mergify.yml
	modified:   .github/release-drafter.yml
	modified:   .github/workflows/packages.yaml
	modified:   .github/workflows/tests.yaml
	modified:   .gitignore
	modified:   .mailmap
	modified:   .pre-commit-config.yaml
	modified:   .pre-commit-hooks.yaml
	modified:   .pylintrc
	modified:   .zenodo.json
	modified:   CODE_OF_CONDUCT.md
	modified:   CONTRIBUTING.md
	modified:   LICENSE
	modified:   MANIFEST.in
	modified:   README.rst
	modified:   bin/dvc.bat
	modified:   dvc/__init__.py
	modified:   dvc/__main__.py
	modified:   dvc/_debug.py
	modified:   dvc/analytics.py
	modified:   dvc/api.py
	modified:   dvc/checkout.py
	modified:   dvc/cli.py
	modified:   dvc/command/__init__.py
	modified:   dvc/command/add.py
	modified:   dvc/command/base.py
	modified:   dvc/command/cache.py
	modified:   dvc/command/check_ignore.py
	modified:   dvc/command/checkout.py
	modified:   dvc/command/commit.py
	modified:   dvc/command/completion.py
	modified:   dvc/command/config.py
	modified:   dvc/command/daemon.py
	modified:   dvc/command/dag.py
	modified:   dvc/command/data_sync.py
	modified:   dvc/command/destroy.py
	modified:   dvc/command/diff.py
	modified:   dvc/command/experiments.py
	modified:   dvc/command/freeze.py
	modified:   dvc/command/gc.py
	modified:   dvc/command/get.py
	modified:   dvc/command/get_url.py
	modified:   dvc/command/git_hook.py
	modified:   dvc/command/imp.py
	modified:   dvc/command/imp_url.py
	modified:   dvc/command/init.py
	modified:   dvc/command/install.py
	modified:   dvc/command/live.py
	modified:   dvc/command/ls/__init__.py
	modified:   dvc/command/ls/ls_colors.py
	modified:   dvc/command/metrics.py
	modified:   dvc/command/move.py
	modified:   dvc/command/params.py
	modified:   dvc/command/plots.py
	modified:   dvc/command/remote.py
	modified:   dvc/command/remove.py
	modified:   dvc/command/repro.py
	modified:   dvc/command/root.py
	modified:   dvc/command/run.py
	modified:   dvc/command/stage.py
	modified:   dvc/command/status.py
	modified:   dvc/command/unprotect.py
	modified:   dvc/command/update.py
	modified:   dvc/command/version.py
	modified:   dvc/compare.py
	modified:   dvc/config.py
	modified:   dvc/config_schema.py
	modified:   dvc/daemon.py
	modified:   dvc/dagascii.py
	modified:   dvc/data_cloud.py
	modified:   dvc/dependency/__init__.py
	modified:   dvc/dependency/base.py
	modified:   dvc/dependency/param.py
	modified:   dvc/dependency/repo.py
	modified:   dvc/dvcfile.py
	modified:   dvc/env.py
	modified:   dvc/exceptions.py
	modified:   dvc/external_repo.py
	modified:   dvc/fs/__init__.py
	modified:   dvc/fs/_metadata.py
	modified:   dvc/fs/azure.py
	modified:   dvc/fs/base.py
	modified:   dvc/fs/dvc.py
	modified:   dvc/fs/fsspec_wrapper.py
	modified:   dvc/fs/gdrive.py
	modified:   dvc/fs/git.py
	modified:   dvc/fs/gs.py
	modified:   dvc/fs/hdfs.py
	modified:   dvc/fs/http.py
	modified:   dvc/fs/https.py
	modified:   dvc/fs/local.py
	modified:   dvc/fs/memory.py
	modified:   dvc/fs/oss.py
	modified:   dvc/fs/pool.py
	modified:   dvc/fs/repo.py
	modified:   dvc/fs/s3.py
	modified:   dvc/fs/ssh/__init__.py
	modified:   dvc/fs/ssh/connection.py
	modified:   dvc/fs/webdav.py
	modified:   dvc/fs/webdavs.py
	modified:   dvc/fs/webhdfs.py
	modified:   dvc/hash_info.py
	modified:   dvc/ignore.py
	modified:   dvc/info.py
	modified:   dvc/istextfile.py
	modified:   dvc/lock.py
	modified:   dvc/logger.py
	modified:   dvc/main.py
	modified:   dvc/objects/__init__.py
	modified:   dvc/objects/db/__init__.py
	modified:   dvc/objects/db/base.py
	modified:   dvc/objects/db/gdrive.py
	modified:   dvc/objects/db/local.py
	modified:   dvc/objects/db/ssh.py
	modified:   dvc/objects/errors.py
	modified:   dvc/objects/file.py
	modified:   dvc/objects/stage.py
	modified:   dvc/objects/tree.py
	modified:   dvc/output.py
	modified:   dvc/parsing/__init__.py
	modified:   dvc/parsing/context.py
	modified:   dvc/parsing/interpolate.py
	modified:   dvc/parsing/versions.py
	modified:   dvc/path_info.py
	modified:   dvc/pathspec_math.py
	modified:   dvc/progress.py
	modified:   dvc/prompt.py
	modified:   dvc/remote/__init__.py
	modified:   dvc/remote/base.py
	modified:   dvc/remote/index.py
	modified:   dvc/remote/local.py
	modified:   dvc/remote/slow_link_detection.py
	modified:   dvc/repo/__init__.py
	modified:   dvc/repo/add.py
	modified:   dvc/repo/brancher.py
	modified:   dvc/repo/checkout.py
	modified:   dvc/repo/collect.py
	modified:   dvc/repo/commit.py
	modified:   dvc/repo/destroy.py
	modified:   dvc/repo/diff.py
	modified:   dvc/repo/experiments/__init__.py
	modified:   dvc/repo/experiments/apply.py
	modified:   dvc/repo/experiments/base.py
	modified:   dvc/repo/experiments/branch.py
	modified:   dvc/repo/experiments/diff.py
	modified:   dvc/repo/experiments/executor/__init__.py
	modified:   dvc/repo/experiments/executor/base.py
	modified:   dvc/repo/experiments/executor/local.py
	modified:   dvc/repo/experiments/gc.py
	modified:   dvc/repo/experiments/ls.py
	modified:   dvc/repo/experiments/pull.py
	modified:   dvc/repo/experiments/push.py
	modified:   dvc/repo/experiments/remove.py
	modified:   dvc/repo/experiments/run.py
	modified:   dvc/repo/experiments/show.py
	modified:   dvc/repo/experiments/utils.py
	modified:   dvc/repo/fetch.py
	modified:   dvc/repo/freeze.py
	modified:   dvc/repo/gc.py
	modified:   dvc/repo/get.py
	modified:   dvc/repo/get_url.py
	modified:   dvc/repo/graph.py
	modified:   dvc/repo/imp.py
	modified:   dvc/repo/imp_url.py
	modified:   dvc/repo/init.py
	modified:   dvc/repo/install.py
	modified:   dvc/repo/live.py
	modified:   dvc/repo/ls.py
	modified:   dvc/repo/metrics/__init__.py
	modified:   dvc/repo/metrics/diff.py
	modified:   dvc/repo/metrics/show.py
	modified:   dvc/repo/move.py
	modified:   dvc/repo/params/__init__.py
	modified:   dvc/repo/params/diff.py
	modified:   dvc/repo/params/show.py
	modified:   dvc/repo/plots/__init__.py
	modified:   dvc/repo/plots/data.py
	modified:   dvc/repo/plots/diff.py
	modified:   dvc/repo/plots/template.py
	modified:   dvc/repo/pull.py
	modified:   dvc/repo/push.py
	modified:   dvc/repo/remove.py
	modified:   dvc/repo/reproduce.py
	modified:   dvc/repo/run.py
	modified:   dvc/repo/scm_context.py
	modified:   dvc/repo/stage.py
	modified:   dvc/repo/status.py
	modified:   dvc/repo/trie.py
	modified:   dvc/repo/update.py
	modified:   dvc/rwlock.py
	modified:   dvc/schema.py
	modified:   dvc/scheme.py
	modified:   dvc/scm/__init__.py
	modified:   dvc/scm/base.py
	modified:   dvc/scm/git/__init__.py
	modified:   dvc/scm/git/backend/__init__.py
	modified:   dvc/scm/git/backend/base.py
	modified:   dvc/scm/git/backend/dulwich.py
	modified:   dvc/scm/git/backend/gitpython.py
	modified:   dvc/scm/git/backend/pygit2.py
	modified:   dvc/scm/git/objects.py
	modified:   dvc/scm/git/stash.py
	modified:   dvc/stage/__init__.py
	modified:   dvc/stage/cache.py
	modified:   dvc/stage/decorators.py
	modified:   dvc/stage/exceptions.py
	modified:   dvc/stage/imports.py
	modified:   dvc/stage/loader.py
	modified:   dvc/stage/monitor.py
	modified:   dvc/stage/params.py
	modified:   dvc/stage/run.py
	modified:   dvc/stage/serialize.py
	modified:   dvc/stage/utils.py
	modified:   dvc/state.py
	modified:   dvc/system.py
	modified:   dvc/types.py
	modified:   dvc/ui/__init__.py
	modified:   dvc/ui/table.py
	modified:   dvc/updater.py
	modified:   dvc/utils/.gitignore
	modified:   dvc/utils/__init__.py
	modified:   dvc/utils/cli_parse.py
	modified:   dvc/utils/collections.py
	modified:   dvc/utils/conversions.py
	modified:   dvc/utils/diff.py
	modified:   dvc/utils/flatten.py
	modified:   dvc/utils/fs.py
	modified:   dvc/utils/html.py
	modified:   dvc/utils/http.py
	modified:   dvc/utils/humanize.py
	modified:   dvc/utils/pager.py
	modified:   dvc/utils/pkg.py
	modified:   dvc/utils/serialize/__init__.py
	modified:   dvc/utils/serialize/_common.py
	modified:   dvc/utils/serialize/_json.py
	modified:   dvc/utils/serialize/_py.py
	modified:   dvc/utils/serialize/_toml.py
	modified:   dvc/utils/serialize/_yaml.py
	modified:   dvc/utils/stream.py
	modified:   dvc/utils/table.py
	modified:   dvc/version.py
	modified:   fastentrypoints.py
	modified:   pyproject.toml
	modified:   scripts/build-requirements.txt
	modified:   scripts/build.py
	modified:   scripts/fpm/.gitignore
	modified:   scripts/fpm/after-install.sh
	modified:   scripts/fpm/after-remove.sh
	modified:   scripts/fpm/build.py
	modified:   scripts/fpm/notarize.py
	modified:   scripts/fpm/sign.py
	modified:   scripts/innosetup/.gitignore
	modified:   scripts/innosetup/addSymLinkPermissions.ps1
	modified:   scripts/innosetup/addsymlink.iss
	modified:   scripts/innosetup/build.py
	modified:   scripts/innosetup/dvc.ico.dvc
	modified:   scripts/innosetup/dvc_left.bmp.dvc
	modified:   scripts/innosetup/dvc_up.bmp.dvc
	modified:   scripts/innosetup/modpath.iss
	modified:   scripts/innosetup/setup.iss
	modified:   scripts/pyinstaller/.gitignore
	modified:   scripts/pyinstaller/build.py
	modified:   scripts/pyinstaller/entitlements.plist
	modified:   scripts/pyinstaller/hooks/hook-dvc.py
	modified:   scripts/pyinstaller/hooks/hook-dvc.system.py
	modified:   scripts/pyinstaller/hooks/hook-dvc.tree.gs.py
	modified:   scripts/pyinstaller/hooks/hook-dvc.utils.flatten.py
	modified:   scripts/pyinstaller/hooks/hook-google_compute_engine.logger.py
	modified:   scripts/pyinstaller/hooks/hook-googleapiclient.model.py
	modified:   scripts/pyinstaller/hooks/hook-pydrive2.py
	modified:   scripts/pyinstaller/hooks/hook-win32file.py
	modified:   scripts/pyinstaller/sign.py
	modified:   scripts/schema/dvc-yaml.json
	modified:   setup.cfg
	modified:   setup.py
	modified:   test_requirements.txt
	modified:   tests/__init__.py
	modified:   tests/__main__.py
	modified:   tests/basic_env.py
	modified:   tests/conftest.py
	modified:   tests/dir_helpers.py
	modified:   tests/docker-compose.yml
	modified:   tests/func/__init__.py
	modified:   tests/func/conftest.py
	modified:   tests/func/experiments/__init__.py
	modified:   tests/func/experiments/conftest.py
	modified:   tests/func/experiments/test_checkpoints.py
	modified:   tests/func/experiments/test_diff.py
	modified:   tests/func/experiments/test_experiments.py
	modified:   tests/func/experiments/test_gc.py
	modified:   tests/func/experiments/test_remote.py
	modified:   tests/func/experiments/test_show.py
	modified:   tests/func/metrics/__init__.py
	modified:   tests/func/metrics/test_diff.py
	modified:   tests/func/metrics/test_show.py
	modified:   tests/func/params/__init__.py
	modified:   tests/func/params/test_diff.py
	modified:   tests/func/params/test_show.py
	modified:   tests/func/parsing/__init__.py
	modified:   tests/func/parsing/test_errors.py
	modified:   tests/func/parsing/test_foreach.py
	modified:   tests/func/parsing/test_interpolated_entry.py
	modified:   tests/func/parsing/test_resolver.py
	modified:   tests/func/plots/__init__.py
	modified:   tests/func/plots/conftest.py
	modified:   tests/func/plots/test_diff.py
	modified:   tests/func/plots/test_modify.py
	modified:   tests/func/plots/test_show.py
	modified:   tests/func/plots/utils.py
	modified:   tests/func/remote/__init__.py
	modified:   tests/func/remote/test_index.py
	modified:   tests/func/test_add.py
	modified:   tests/func/test_analytics.py
	modified:   tests/func/test_api.py
	modified:   tests/func/test_check_ignore.py
	modified:   tests/func/test_checkout.py
	modified:   tests/func/test_cli.py
	modified:   tests/func/test_commit.py
	modified:   tests/func/test_config.py
	modified:   tests/func/test_data_cloud.py
	modified:   tests/func/test_diff.py
	modified:   tests/func/test_dvcfile.py
	modified:   tests/func/test_external_repo.py
	modified:   tests/func/test_fs.py
	modified:   tests/func/test_gc.py
	modified:   tests/func/test_get.py
	modified:   tests/func/test_get_url.py
	modified:   tests/func/test_ignore.py
	modified:   tests/func/test_import.py
	modified:   tests/func/test_import_url.py
	modified:   tests/func/test_init.py
	modified:   tests/func/test_install.py
	modified:   tests/func/test_live.py
	modified:   tests/func/test_lock.py
	modified:   tests/func/test_lockfile.py
	modified:   tests/func/test_ls.py
	modified:   tests/func/test_merge_driver.py
	modified:   tests/func/test_move.py
	modified:   tests/func/test_odb.py
	modified:   tests/func/test_remote.py
	modified:   tests/func/test_remove.py
	modified:   tests/func/test_repo.py
	modified:   tests/func/test_repro.py
	modified:   tests/func/test_repro_multistage.py
	modified:   tests/func/test_root.py
	modified:   tests/func/test_run_cache.py
	modified:   tests/func/test_run_multistage.py
	modified:   tests/func/test_run_single_stage.py
	modified:   tests/func/test_scm.py
	modified:   tests/func/test_stage.py
	modified:   tests/func/test_stage_load.py
	modified:   tests/func/test_state.py
	modified:   tests/func/test_status.py
	modified:   tests/func/test_system.py
	modified:   tests/func/test_unprotect.py
	modified:   tests/func/test_update.py
	modified:   tests/func/test_utils.py
	modified:   tests/func/test_version.py
	modified:   tests/func/utils/__init__.py
	modified:   tests/func/utils/test_fs.py
	modified:   tests/func/utils/test_html.py
	modified:   tests/pylint_plugin_disable.py
	modified:   tests/remotes/__init__.py
	modified:   tests/remotes/azure.py
	modified:   tests/remotes/base.py
	modified:   tests/remotes/gdrive.py
	modified:   tests/remotes/gs.py
	modified:   tests/remotes/hdfs.py
	modified:   tests/remotes/http.py
	modified:   tests/remotes/local.py
	modified:   tests/remotes/oss.py
	modified:   tests/remotes/s3.py
	modified:   tests/remotes/ssh.py
	modified:   tests/remotes/user.key
	modified:   tests/remotes/user.key.pub
	modified:   tests/remotes/webdav.py
	modified:   tests/remotes/webhdfs.py
	modified:   tests/remotes_env.sample
	modified:   tests/unit/__init__.py
	modified:   tests/unit/command/__init__.py
	modified:   tests/unit/command/ls/__init__.py
	modified:   tests/unit/command/ls/test_ls.py
	modified:   tests/unit/command/ls/test_ls_colors.py
	modified:   tests/unit/command/test_add.py
	modified:   tests/unit/command/test_cache.py
	modified:   tests/unit/command/test_checkout.py
	modified:   tests/unit/command/test_config.py
	modified:   tests/unit/command/test_dag.py
	modified:   tests/unit/command/test_data_sync.py
	modified:   tests/unit/command/test_diff.py
	modified:   tests/unit/command/test_experiments.py
	modified:   tests/unit/command/test_get.py
	modified:   tests/unit/command/test_get_url.py
	modified:   tests/unit/command/test_git_hook.py
	modified:   tests/unit/command/test_imp.py
	modified:   tests/unit/command/test_imp_url.py
	modified:   tests/unit/command/test_live.py
	modified:   tests/unit/command/test_metrics.py
	modified:   tests/unit/command/test_params.py
	modified:   tests/unit/command/test_plots.py
	modified:   tests/unit/command/test_repro.py
	modified:   tests/unit/command/test_run.py
	modified:   tests/unit/command/test_stage.py
	modified:   tests/unit/command/test_status.py
	modified:   tests/unit/command/test_update.py
	modified:   tests/unit/dependency/__init__.py
	modified:   tests/unit/dependency/test_dependency.py
	modified:   tests/unit/dependency/test_params.py
	modified:   tests/unit/fs/__init__.py
	modified:   tests/unit/fs/test_azure.py
	modified:   tests/unit/fs/test_base.py
	modified:   tests/unit/fs/test_dvc.py
	modified:   tests/unit/fs/test_fs.py
	modified:   tests/unit/fs/test_hdfs.py
	modified:   tests/unit/fs/test_repo.py
	modified:   tests/unit/fs/test_repo_metadata.py
	modified:   tests/unit/fs/test_s3.py
	modified:   tests/unit/fs/test_tree.py
	modified:   tests/unit/objects/__init__.py
	modified:   tests/unit/objects/db/__init__.py
	modified:   tests/unit/objects/db/test_local.py
	modified:   tests/unit/objects/test_tree.py
	modified:   tests/unit/output/__init__.py
	modified:   tests/unit/output/test_load.py
	modified:   tests/unit/output/test_local.py
	modified:   tests/unit/output/test_output.py
	modified:   tests/unit/remote/__init__.py
	modified:   tests/unit/remote/ssh/__init__.py
	modified:   tests/unit/remote/ssh/test_connection.py
	modified:   tests/unit/remote/ssh/test_pool.py
	modified:   tests/unit/remote/ssh/test_ssh.py
	modified:   tests/unit/remote/test_base.py
	modified:   tests/unit/remote/test_gdrive.py
	modified:   tests/unit/remote/test_http.py
	modified:   tests/unit/remote/test_index.py
	modified:   tests/unit/remote/test_oss.py
	modified:   tests/unit/remote/test_remote.py
	modified:   tests/unit/remote/test_remote_tree.py
	modified:   tests/unit/remote/test_slow_link_detection.py
	modified:   tests/unit/remote/test_webdav.py
	modified:   tests/unit/remote/test_webhdfs.py
	modified:   tests/unit/repo/__init__.py
	modified:   tests/unit/repo/plots/__init__.py
	modified:   tests/unit/repo/plots/test_data.py
	modified:   tests/unit/repo/plots/test_diff.py
	modified:   tests/unit/repo/test_repo.py
	modified:   tests/unit/repo/test_reproduce.py
	modified:   tests/unit/repo/test_run.py
	modified:   tests/unit/scm/__init__.py
	modified:   tests/unit/scm/test_git.py
	modified:   tests/unit/scm/test_scm.py
	modified:   tests/unit/stage/test_cache.py
	modified:   tests/unit/stage/test_loader_pipeline_file.py
	modified:   tests/unit/stage/test_run.py
	modified:   tests/unit/stage/test_serialize_pipeline_file.py
	modified:   tests/unit/stage/test_serialize_pipeline_lock.py
	modified:   tests/unit/stage/test_stage.py
	modified:   tests/unit/stage/test_utils.py
	modified:   tests/unit/test_analytics.py
	modified:   tests/unit/test_collect.py
	modified:   tests/unit/test_compare.py
	modified:   tests/unit/test_config.py
	modified:   tests/unit/test_context.py
	modified:   tests/unit/test_daemon.py
	modified:   tests/unit/test_dvcfile.py
	modified:   tests/unit/test_external_repo.py
	modified:   tests/unit/test_ignore.py
	modified:   tests/unit/test_imports.py
	modified:   tests/unit/test_info.py
	modified:   tests/unit/test_interpolate.py
	modified:   tests/unit/test_lockfile.py
	modified:   tests/unit/test_logger.py
	modified:   tests/unit/test_metrics.py
	modified:   tests/unit/test_params.py
	modified:   tests/unit/test_path_info.py
	modified:   tests/unit/test_pathspec_math.py
	modified:   tests/unit/test_plots.py
	modified:   tests/unit/test_progress.py
	modified:   tests/unit/test_prompt.py
	modified:   tests/unit/test_run.py
	modified:   tests/unit/test_rwlock.py
	modified:   tests/unit/test_state.py
	modified:   tests/unit/test_tabular_data.py
	modified:   tests/unit/test_updater.py
	modified:   tests/unit/test_version.py
	modified:   tests/unit/ui/__init__.py
	modified:   tests/unit/ui/test_console.py
	modified:   tests/unit/ui/test_pager.py
	modified:   tests/unit/ui/test_table.py
	modified:   tests/unit/utils/__init__.py
	modified:   tests/unit/utils/serialize/__init__.py
	modified:   tests/unit/utils/serialize/test_yaml.py
	modified:   tests/unit/utils/test_cli_parse.py
	modified:   tests/unit/utils/test_collections.py
	modified:   tests/unit/utils/test_conversions.py
	modified:   tests/unit/utils/test_fs.py
	modified:   tests/unit/utils/test_http.py
	modified:   tests/unit/utils/test_humanize.py
	modified:   tests/unit/utils/test_stream.py
	modified:   tests/unit/utils/test_utils.py
	modified:   tests/utils/__init__.py
	modified:   tests/utils/asserts.py
	modified:   tests/utils/httpd.py
@scotty3005
Added configuration option for not doing chunked upload
e9b7fee 
for HTTP remotes

Changes to be committed:
	modified:   .codecov.yml
	modified:   .coveragerc
	modified:   .dvc/.gitignore
	modified:   .dvc/config
	modified:   .github/FUNDING.yml
	modified:   .github/ISSUE_TEMPLATE/bug_report.md
	modified:   .github/ISSUE_TEMPLATE/config.yml
	modified:   .github/ISSUE_TEMPLATE/feature_request.md
	modified:   .github/PULL_REQUEST_TEMPLATE.md
	modified:   .github/dependabot.yml
	modified:   .github/mergify.yml
	modified:   .github/release-drafter.yml
	modified:   .github/workflows/packages.yaml
	modified:   .github/workflows/tests.yaml
	modified:   .gitignore
	modified:   .mailmap
	modified:   .pre-commit-config.yaml
	modified:   .pre-commit-hooks.yaml
	modified:   .pylintrc
	modified:   .zenodo.json
	modified:   CODE_OF_CONDUCT.md
	modified:   CONTRIBUTING.md
	modified:   LICENSE
	modified:   MANIFEST.in
	modified:   README.rst
	modified:   bin/dvc.bat
	modified:   dvc/__init__.py
	modified:   dvc/__main__.py
	modified:   dvc/_debug.py
	modified:   dvc/analytics.py
	modified:   dvc/api.py
	modified:   dvc/checkout.py
	modified:   dvc/cli.py
	modified:   dvc/command/__init__.py
	modified:   dvc/command/add.py
	modified:   dvc/command/base.py
	modified:   dvc/command/cache.py
	modified:   dvc/command/check_ignore.py
	modified:   dvc/command/checkout.py
	modified:   dvc/command/commit.py
	modified:   dvc/command/completion.py
	modified:   dvc/command/config.py
	modified:   dvc/command/daemon.py
	modified:   dvc/command/dag.py
	modified:   dvc/command/data_sync.py
	modified:   dvc/command/destroy.py
	modified:   dvc/command/diff.py
	modified:   dvc/command/experiments.py
	modified:   dvc/command/freeze.py
	modified:   dvc/command/gc.py
	modified:   dvc/command/get.py
	modified:   dvc/command/get_url.py
	modified:   dvc/command/git_hook.py
	modified:   dvc/command/imp.py
	modified:   dvc/command/imp_url.py
	modified:   dvc/command/init.py
	modified:   dvc/command/install.py
	modified:   dvc/command/live.py
	modified:   dvc/command/ls/__init__.py
	modified:   dvc/command/ls/ls_colors.py
	modified:   dvc/command/metrics.py
	modified:   dvc/command/move.py
	modified:   dvc/command/params.py
	modified:   dvc/command/plots.py
	modified:   dvc/command/remote.py
	modified:   dvc/command/remove.py
	modified:   dvc/command/repro.py
	modified:   dvc/command/root.py
	modified:   dvc/command/run.py
	modified:   dvc/command/stage.py
	modified:   dvc/command/status.py
	modified:   dvc/command/unprotect.py
	modified:   dvc/command/update.py
	modified:   dvc/command/version.py
	modified:   dvc/compare.py
	modified:   dvc/config.py
	modified:   dvc/config_schema.py
	modified:   dvc/daemon.py
	modified:   dvc/dagascii.py
	modified:   dvc/data_cloud.py
	modified:   dvc/dependency/__init__.py
	modified:   dvc/dependency/base.py
	modified:   dvc/dependency/param.py
	modified:   dvc/dependency/repo.py
	modified:   dvc/dvcfile.py
	modified:   dvc/env.py
	modified:   dvc/exceptions.py
	modified:   dvc/external_repo.py
	modified:   dvc/fs/__init__.py
	modified:   dvc/fs/_metadata.py
	modified:   dvc/fs/azure.py
	modified:   dvc/fs/base.py
	modified:   dvc/fs/dvc.py
	modified:   dvc/fs/fsspec_wrapper.py
	modified:   dvc/fs/gdrive.py
	modified:   dvc/fs/git.py
	modified:   dvc/fs/gs.py
	modified:   dvc/fs/hdfs.py
	modified:   dvc/fs/http.py
	modified:   dvc/fs/https.py
	modified:   dvc/fs/local.py
	modified:   dvc/fs/memory.py
	modified:   dvc/fs/oss.py
	modified:   dvc/fs/pool.py
	modified:   dvc/fs/repo.py
	modified:   dvc/fs/s3.py
	modified:   dvc/fs/ssh/__init__.py
	modified:   dvc/fs/ssh/connection.py
	modified:   dvc/fs/webdav.py
	modified:   dvc/fs/webdavs.py
	modified:   dvc/fs/webhdfs.py
	modified:   dvc/hash_info.py
	modified:   dvc/ignore.py
	modified:   dvc/info.py
	modified:   dvc/istextfile.py
	modified:   dvc/lock.py
	modified:   dvc/logger.py
	modified:   dvc/main.py
	modified:   dvc/objects/__init__.py
	modified:   dvc/objects/db/__init__.py
	modified:   dvc/objects/db/base.py
	modified:   dvc/objects/db/gdrive.py
	modified:   dvc/objects/db/local.py
	modified:   dvc/objects/db/ssh.py
	modified:   dvc/objects/errors.py
	modified:   dvc/objects/file.py
	modified:   dvc/objects/stage.py
	modified:   dvc/objects/tree.py
	modified:   dvc/output.py
	modified:   dvc/parsing/__init__.py
	modified:   dvc/parsing/context.py
	modified:   dvc/parsing/interpolate.py
	modified:   dvc/parsing/versions.py
	modified:   dvc/path_info.py
	modified:   dvc/pathspec_math.py
	modified:   dvc/progress.py
	modified:   dvc/prompt.py
	modified:   dvc/remote/__init__.py
	modified:   dvc/remote/base.py
	modified:   dvc/remote/index.py
	modified:   dvc/remote/local.py
	modified:   dvc/remote/slow_link_detection.py
	modified:   dvc/repo/__init__.py
	modified:   dvc/repo/add.py
	modified:   dvc/repo/brancher.py
	modified:   dvc/repo/checkout.py
	modified:   dvc/repo/collect.py
	modified:   dvc/repo/commit.py
	modified:   dvc/repo/destroy.py
	modified:   dvc/repo/diff.py
	modified:   dvc/repo/experiments/__init__.py
	modified:   dvc/repo/experiments/apply.py
	modified:   dvc/repo/experiments/base.py
	modified:   dvc/repo/experiments/branch.py
	modified:   dvc/repo/experiments/diff.py
	modified:   dvc/repo/experiments/executor/__init__.py
	modified:   dvc/repo/experiments/executor/base.py
	modified:   dvc/repo/experiments/executor/local.py
	modified:   dvc/repo/experiments/gc.py
	modified:   dvc/repo/experiments/ls.py
	modified:   dvc/repo/experiments/pull.py
	modified:   dvc/repo/experiments/push.py
	modified:   dvc/repo/experiments/remove.py
	modified:   dvc/repo/experiments/run.py
	modified:   dvc/repo/experiments/show.py
	modified:   dvc/repo/experiments/utils.py
	modified:   dvc/repo/fetch.py
	modified:   dvc/repo/freeze.py
	modified:   dvc/repo/gc.py
	modified:   dvc/repo/get.py
	modified:   dvc/repo/get_url.py
	modified:   dvc/repo/graph.py
	modified:   dvc/repo/imp.py
	modified:   dvc/repo/imp_url.py
	modified:   dvc/repo/init.py
	modified:   dvc/repo/install.py
	modified:   dvc/repo/live.py
	modified:   dvc/repo/ls.py
	modified:   dvc/repo/metrics/__init__.py
	modified:   dvc/repo/metrics/diff.py
	modified:   dvc/repo/metrics/show.py
	modified:   dvc/repo/move.py
	modified:   dvc/repo/params/__init__.py
	modified:   dvc/repo/params/diff.py
	modified:   dvc/repo/params/show.py
	modified:   dvc/repo/plots/__init__.py
	modified:   dvc/repo/plots/data.py
	modified:   dvc/repo/plots/diff.py
	modified:   dvc/repo/plots/template.py
	modified:   dvc/repo/pull.py
	modified:   dvc/repo/push.py
	modified:   dvc/repo/remove.py
	modified:   dvc/repo/reproduce.py
	modified:   dvc/repo/run.py
	modified:   dvc/repo/scm_context.py
	modified:   dvc/repo/stage.py
	modified:   dvc/repo/status.py
	modified:   dvc/repo/trie.py
	modified:   dvc/repo/update.py
	modified:   dvc/rwlock.py
	modified:   dvc/schema.py
	modified:   dvc/scheme.py
	modified:   dvc/scm/__init__.py
	modified:   dvc/scm/base.py
	modified:   dvc/scm/git/__init__.py
	modified:   dvc/scm/git/backend/__init__.py
	modified:   dvc/scm/git/backend/base.py
	modified:   dvc/scm/git/backend/dulwich.py
	modified:   dvc/scm/git/backend/gitpython.py
	modified:   dvc/scm/git/backend/pygit2.py
	modified:   dvc/scm/git/objects.py
	modified:   dvc/scm/git/stash.py
	modified:   dvc/stage/__init__.py
	modified:   dvc/stage/cache.py
	modified:   dvc/stage/decorators.py
	modified:   dvc/stage/exceptions.py
	modified:   dvc/stage/imports.py
	modified:   dvc/stage/loader.py
	modified:   dvc/stage/monitor.py
	modified:   dvc/stage/params.py
	modified:   dvc/stage/run.py
	modified:   dvc/stage/serialize.py
	modified:   dvc/stage/utils.py
	modified:   dvc/state.py
	modified:   dvc/system.py
	modified:   dvc/types.py
	modified:   dvc/ui/__init__.py
	modified:   dvc/ui/table.py
	modified:   dvc/updater.py
	modified:   dvc/utils/.gitignore
	modified:   dvc/utils/__init__.py
	modified:   dvc/utils/cli_parse.py
	modified:   dvc/utils/collections.py
	modified:   dvc/utils/conversions.py
	modified:   dvc/utils/diff.py
	modified:   dvc/utils/flatten.py
	modified:   dvc/utils/fs.py
	modified:   dvc/utils/html.py
	modified:   dvc/utils/http.py
	modified:   dvc/utils/humanize.py
	modified:   dvc/utils/pager.py
	modified:   dvc/utils/pkg.py
	modified:   dvc/utils/serialize/__init__.py
	modified:   dvc/utils/serialize/_common.py
	modified:   dvc/utils/serialize/_json.py
	modified:   dvc/utils/serialize/_py.py
	modified:   dvc/utils/serialize/_toml.py
	modified:   dvc/utils/serialize/_yaml.py
	modified:   dvc/utils/stream.py
	modified:   dvc/utils/table.py
	modified:   dvc/version.py
	modified:   fastentrypoints.py
	modified:   pyproject.toml
	modified:   scripts/build-requirements.txt
	modified:   scripts/build.py
	modified:   scripts/fpm/.gitignore
	modified:   scripts/fpm/after-install.sh
	modified:   scripts/fpm/after-remove.sh
	modified:   scripts/fpm/build.py
	modified:   scripts/fpm/notarize.py
	modified:   scripts/fpm/sign.py
	modified:   scripts/innosetup/.gitignore
	modified:   scripts/innosetup/addSymLinkPermissions.ps1
	modified:   scripts/innosetup/addsymlink.iss
	modified:   scripts/innosetup/build.py
	modified:   scripts/innosetup/dvc.ico.dvc
	modified:   scripts/innosetup/dvc_left.bmp.dvc
	modified:   scripts/innosetup/dvc_up.bmp.dvc
	modified:   scripts/innosetup/modpath.iss
	modified:   scripts/innosetup/setup.iss
	modified:   scripts/pyinstaller/.gitignore
	modified:   scripts/pyinstaller/build.py
	modified:   scripts/pyinstaller/entitlements.plist
	modified:   scripts/pyinstaller/hooks/hook-dvc.py
	modified:   scripts/pyinstaller/hooks/hook-dvc.system.py
	modified:   scripts/pyinstaller/hooks/hook-dvc.tree.gs.py
	modified:   scripts/pyinstaller/hooks/hook-dvc.utils.flatten.py
	modified:   scripts/pyinstaller/hooks/hook-google_compute_engine.logger.py
	modified:   scripts/pyinstaller/hooks/hook-googleapiclient.model.py
	modified:   scripts/pyinstaller/hooks/hook-pydrive2.py
	modified:   scripts/pyinstaller/hooks/hook-win32file.py
	modified:   scripts/pyinstaller/sign.py
	modified:   scripts/schema/dvc-yaml.json
	modified:   setup.cfg
	modified:   setup.py
	modified:   test_requirements.txt
	modified:   tests/__init__.py
	modified:   tests/__main__.py
	modified:   tests/basic_env.py
	modified:   tests/conftest.py
	modified:   tests/dir_helpers.py
	modified:   tests/docker-compose.yml
	modified:   tests/func/__init__.py
	modified:   tests/func/conftest.py
	modified:   tests/func/experiments/__init__.py
	modified:   tests/func/experiments/conftest.py
	modified:   tests/func/experiments/test_checkpoints.py
	modified:   tests/func/experiments/test_diff.py
	modified:   tests/func/experiments/test_experiments.py
	modified:   tests/func/experiments/test_gc.py
	modified:   tests/func/experiments/test_remote.py
	modified:   tests/func/experiments/test_show.py
	modified:   tests/func/metrics/__init__.py
	modified:   tests/func/metrics/test_diff.py
	modified:   tests/func/metrics/test_show.py
	modified:   tests/func/params/__init__.py
	modified:   tests/func/params/test_diff.py
	modified:   tests/func/params/test_show.py
	modified:   tests/func/parsing/__init__.py
	modified:   tests/func/parsing/test_errors.py
	modified:   tests/func/parsing/test_foreach.py
	modified:   tests/func/parsing/test_interpolated_entry.py
	modified:   tests/func/parsing/test_resolver.py
	modified:   tests/func/plots/__init__.py
	modified:   tests/func/plots/conftest.py
	modified:   tests/func/plots/test_diff.py
	modified:   tests/func/plots/test_modify.py
	modified:   tests/func/plots/test_show.py
	modified:   tests/func/plots/utils.py
	modified:   tests/func/remote/__init__.py
	modified:   tests/func/remote/test_index.py
	modified:   tests/func/test_add.py
	modified:   tests/func/test_analytics.py
	modified:   tests/func/test_api.py
	modified:   tests/func/test_check_ignore.py
	modified:   tests/func/test_checkout.py
	modified:   tests/func/test_cli.py
	modified:   tests/func/test_commit.py
	modified:   tests/func/test_config.py
	modified:   tests/func/test_data_cloud.py
	modified:   tests/func/test_diff.py
	modified:   tests/func/test_dvcfile.py
	modified:   tests/func/test_external_repo.py
	modified:   tests/func/test_fs.py
	modified:   tests/func/test_gc.py
	modified:   tests/func/test_get.py
	modified:   tests/func/test_get_url.py
	modified:   tests/func/test_ignore.py
	modified:   tests/func/test_import.py
	modified:   tests/func/test_import_url.py
	modified:   tests/func/test_init.py
	modified:   tests/func/test_install.py
	modified:   tests/func/test_live.py
	modified:   tests/func/test_lock.py
	modified:   tests/func/test_lockfile.py
	modified:   tests/func/test_ls.py
	modified:   tests/func/test_merge_driver.py
	modified:   tests/func/test_move.py
	modified:   tests/func/test_odb.py
	modified:   tests/func/test_remote.py
	modified:   tests/func/test_remove.py
	modified:   tests/func/test_repo.py
	modified:   tests/func/test_repro.py
	modified:   tests/func/test_repro_multistage.py
	modified:   tests/func/test_root.py
	modified:   tests/func/test_run_cache.py
	modified:   tests/func/test_run_multistage.py
	modified:   tests/func/test_run_single_stage.py
	modified:   tests/func/test_scm.py
	modified:   tests/func/test_stage.py
	modified:   tests/func/test_stage_load.py
	modified:   tests/func/test_state.py
	modified:   tests/func/test_status.py
	modified:   tests/func/test_system.py
	modified:   tests/func/test_unprotect.py
	modified:   tests/func/test_update.py
	modified:   tests/func/test_utils.py
	modified:   tests/func/test_version.py
	modified:   tests/func/utils/__init__.py
	modified:   tests/func/utils/test_fs.py
	modified:   tests/func/utils/test_html.py
	modified:   tests/pylint_plugin_disable.py
	modified:   tests/remotes/__init__.py
	modified:   tests/remotes/azure.py
	modified:   tests/remotes/base.py
	modified:   tests/remotes/gdrive.py
	modified:   tests/remotes/gs.py
	modified:   tests/remotes/hdfs.py
	modified:   tests/remotes/http.py
	modified:   tests/remotes/local.py
	modified:   tests/remotes/oss.py
	modified:   tests/remotes/s3.py
	modified:   tests/remotes/ssh.py
	modified:   tests/remotes/user.key
	modified:   tests/remotes/user.key.pub
	modified:   tests/remotes/webdav.py
	modified:   tests/remotes/webhdfs.py
	modified:   tests/remotes_env.sample
	modified:   tests/unit/__init__.py
	modified:   tests/unit/command/__init__.py
	modified:   tests/unit/command/ls/__init__.py
	modified:   tests/unit/command/ls/test_ls.py
	modified:   tests/unit/command/ls/test_ls_colors.py
	modified:   tests/unit/command/test_add.py
	modified:   tests/unit/command/test_cache.py
	modified:   tests/unit/command/test_checkout.py
	modified:   tests/unit/command/test_config.py
	modified:   tests/unit/command/test_dag.py
	modified:   tests/unit/command/test_data_sync.py
	modified:   tests/unit/command/test_diff.py
	modified:   tests/unit/command/test_experiments.py
	modified:   tests/unit/command/test_get.py
	modified:   tests/unit/command/test_get_url.py
	modified:   tests/unit/command/test_git_hook.py
	modified:   tests/unit/command/test_imp.py
	modified:   tests/unit/command/test_imp_url.py
	modified:   tests/unit/command/test_live.py
	modified:   tests/unit/command/test_metrics.py
	modified:   tests/unit/command/test_params.py
	modified:   tests/unit/command/test_plots.py
	modified:   tests/unit/command/test_repro.py
	modified:   tests/unit/command/test_run.py
	modified:   tests/unit/command/test_stage.py
	modified:   tests/unit/command/test_status.py
	modified:   tests/unit/command/test_update.py
	modified:   tests/unit/dependency/__init__.py
	modified:   tests/unit/dependency/test_dependency.py
	modified:   tests/unit/dependency/test_params.py
	modified:   tests/unit/fs/__init__.py
	modified:   tests/unit/fs/test_azure.py
	modified:   tests/unit/fs/test_base.py
	modified:   tests/unit/fs/test_dvc.py
	modified:   tests/unit/fs/test_fs.py
	modified:   tests/unit/fs/test_hdfs.py
	modified:   tests/unit/fs/test_repo.py
	modified:   tests/unit/fs/test_repo_metadata.py
	modified:   tests/unit/fs/test_s3.py
	modified:   tests/unit/fs/test_tree.py
	modified:   tests/unit/objects/__init__.py
	modified:   tests/unit/objects/db/__init__.py
	modified:   tests/unit/objects/db/test_local.py
	modified:   tests/unit/objects/test_tree.py
	modified:   tests/unit/output/__init__.py
	modified:   tests/unit/output/test_load.py
	modified:   tests/unit/output/test_local.py
	modified:   tests/unit/output/test_output.py
	modified:   tests/unit/remote/__init__.py
	modified:   tests/unit/remote/ssh/__init__.py
	modified:   tests/unit/remote/ssh/test_connection.py
	modified:   tests/unit/remote/ssh/test_pool.py
	modified:   tests/unit/remote/ssh/test_ssh.py
	modified:   tests/unit/remote/test_base.py
	modified:   tests/unit/remote/test_gdrive.py
	modified:   tests/unit/remote/test_http.py
	modified:   tests/unit/remote/test_index.py
	modified:   tests/unit/remote/test_oss.py
	modified:   tests/unit/remote/test_remote.py
	modified:   tests/unit/remote/test_remote_tree.py
	modified:   tests/unit/remote/test_slow_link_detection.py
	modified:   tests/unit/remote/test_webdav.py
	modified:   tests/unit/remote/test_webhdfs.py
	modified:   tests/unit/repo/__init__.py
	modified:   tests/unit/repo/plots/__init__.py
	modified:   tests/unit/repo/plots/test_data.py
	modified:   tests/unit/repo/plots/test_diff.py
	modified:   tests/unit/repo/test_repo.py
	modified:   tests/unit/repo/test_reproduce.py
	modified:   tests/unit/repo/test_run.py
	modified:   tests/unit/scm/__init__.py
	modified:   tests/unit/scm/test_git.py
	modified:   tests/unit/scm/test_scm.py
	modified:   tests/unit/stage/test_cache.py
	modified:   tests/unit/stage/test_loader_pipeline_file.py
	modified:   tests/unit/stage/test_run.py
	modified:   tests/unit/stage/test_serialize_pipeline_file.py
	modified:   tests/unit/stage/test_serialize_pipeline_lock.py
	modified:   tests/unit/stage/test_stage.py
	modified:   tests/unit/stage/test_utils.py
	modified:   tests/unit/test_analytics.py
	modified:   tests/unit/test_collect.py
	modified:   tests/unit/test_compare.py
	modified:   tests/unit/test_config.py
	modified:   tests/unit/test_context.py
	modified:   tests/unit/test_daemon.py
	modified:   tests/unit/test_dvcfile.py
	modified:   tests/unit/test_external_repo.py
	modified:   tests/unit/test_ignore.py
	modified:   tests/unit/test_imports.py
	modified:   tests/unit/test_info.py
	modified:   tests/unit/test_interpolate.py
	modified:   tests/unit/test_lockfile.py
	modified:   tests/unit/test_logger.py
	modified:   tests/unit/test_metrics.py
	modified:   tests/unit/test_params.py
	modified:   tests/unit/test_path_info.py
	modified:   tests/unit/test_pathspec_math.py
	modified:   tests/unit/test_plots.py
	modified:   tests/unit/test_progress.py
	modified:   tests/unit/test_prompt.py
	modified:   tests/unit/test_run.py
	modified:   tests/unit/test_rwlock.py
	modified:   tests/unit/test_state.py
	modified:   tests/unit/test_tabular_data.py
	modified:   tests/unit/test_updater.py
	modified:   tests/unit/test_version.py
	modified:   tests/unit/ui/__init__.py
	modified:   tests/unit/ui/test_console.py
	modified:   tests/unit/ui/test_pager.py
	modified:   tests/unit/ui/test_table.py
	modified:   tests/unit/utils/__init__.py
	modified:   tests/unit/utils/serialize/__init__.py
	modified:   tests/unit/utils/serialize/test_yaml.py
	modified:   tests/unit/utils/test_cli_parse.py
	modified:   tests/unit/utils/test_collections.py
	modified:   tests/unit/utils/test_conversions.py
	modified:   tests/unit/utils/test_fs.py
	modified:   tests/unit/utils/test_http.py
	modified:   tests/unit/utils/test_humanize.py
	modified:   tests/unit/utils/test_stream.py
	modified:   tests/unit/utils/test_utils.py
	modified:   tests/utils/__init__.py
	modified:   tests/utils/asserts.py
	modified:   tests/utils/httpd.py
Merge branch 'master' of https://github.com/scotty3005/dvc
432b8bf
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
7fa3f3b 
for more information, see https://pre-commit.ci
@scotty3005 scotty3005 mentioned this pull request Aug 16, 2022
Open
karajan1001
karajan1001 suggested changes Jun 5, 2021
View reviewed changes
dvc/fs/http.py Outdated
@@ -159,6 +160,17 @@ def chunks(fobj):
if response.status_code not in (200, 201):
raise HTTPError(response.status_code, response.reason)

def _upload_fobj_not_chunked(self, fobj, to_info):
response = self.request(self.method, to_info.url, files={"file": fobj})
if response.status_code not in (200, 201):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicated code with what in _upload_fobj_chunked maybe we can move them to _upload_fobj

dvc/fs/http.py
@@ -128,7 +129,7 @@ def _head(self, url):
# Sometimes servers are configured to forbid HEAD requests
# Context: https://github.com/iterative/dvc/issues/4131
with self.request("GET", url, stream=True) as r:
if r.ok:
if r.ok or r.status_code == 404:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this cause some problems when the server was actually not found (404). Some additional check are needed.

@karajan1001
Copy link
Contributor

karajan1001 commented Jun 5, 2021 •
edited
Loading

Thanks for your contributions, so this PR is related to iterative/dvc-http#9 ? I recommend You can add a link to it.

And I'm curious why it shows 513 file changes when they actually change nothing, never see this before, we need to solve it.

karajan1001
karajan1001 reviewed Jun 5, 2021
View reviewed changes
Copy link
Contributor

@karajan1001 karajan1001 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And, some tests are needed for this PR.

@scotty3005
Copy link
Author

Hi, thanks for the feedback. I am willing to work on implementing some tests where needed. However I would be glad if I could have some help as it is not clear to me how to test remotes.

@efiop
Copy link
Contributor

efiop commented Jun 7, 2021

I also wonder if https://github.com/intake/filesystem_spec/blob/master/fsspec/implementations/http.py is already able to do what you need. We are migrating to fsspec and will be considering replacing our http implementation with their soon, so might be a good time to give it a shot.

@isidentical
Copy link
Contributor

It seems like open() on HTTPFileSystem is read-only;
https://github.com/intake/filesystem_spec/blob/61ec18fb0a77d3a524a6b2d72340af3748c5bf2a/fsspec/implementations/http.py#L279-L280

@efiop
Copy link
Contributor

efiop commented Jul 18, 2021 •
edited
Loading

@scotty3005 Note that your PR now includes perm changes to a lot of files, which obviously shouldn't be there. There is something wrong with your environment, most likely.

Also looks like there is something wrong with commit authorship, likely your email was not added to https://github.com/settings/emails

Regarding the contents of the PR, looks like we need to dive deeper to properly review this. That will probably happen once we get to migrating httpfs to fsspec.

@scotty3005
Copy link
Author

scotty3005 commented Jul 19, 2021 •
edited
Loading

Hey @efiop, yes I noted the issue with the changes. I do not what happen. W.r.t. the authorship I believe it is because I worked from two different machines, and forgot to set the local user name and email.
In the meanwhile that you guys do the migration, would you prefer to close this and me to resubmit a cleaner PR (correcting the changes and authorship)? Or if there is any other way I can help, I will be happy to do so.

@scotty3005 scotty3005 requested a review from a team as a code owner July 19, 2021 21:57
@scotty3005 scotty3005 requested a review from pmrowla July 19, 2021 21:57
@efiop
Copy link
Contributor

efiop commented Jul 19, 2021

@scotty3005 As @karajan1001 noted, the current logic will cause problems for other servers, so we need more research here to see if we can actually detect this and handle the fallback properly. Unfortunately, I would probably recommend closing this PR for now and going through the research to implement proper logic. 🙁

@scotty3005
Copy link
Author

Hey @efiop, sure I understand. I will close this and keep maintaining the fork for my particular use case. Thanks for the time you guys dedicated to this issue. I am looking forward to contribute to DVC in other occasions.

@scotty3005 scotty3005 closed this Jul 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@karajan1001 karajan1001 karajan1001 requested changes

@pmrowla pmrowla Awaiting requested review from pmrowla

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@scotty3005 @karajan1001 @efiop @isidentical