Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataMovement Migration Guide: Packaging, types, API comparison tables #47890

Merged
merged 1 commit into from
Jan 22, 2025
Merged

DataMovement Migration Guide: Packaging, types, API comparison tables #47890

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 77 additions & 9 deletions sdk/storage/Azure.Storage.DataMovement/MigrationGuide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ Familiarity with the legacy data movement client library is assumed. For those n
- [Package and namespaces](#package-and-namespaces)
- [Authentication](#authentication)
- [Type structure](#type-structure)
- [TransferManager](#transfermanager)
- [Storage resources and providers](#storage-resources-and-providers)
- [TransferContext → TransferOptions](#transfercontext--transferoptions)
- [Migration samples](#migration-samples)
- [Additional information](#additional-information)

Expand All @@ -21,7 +24,7 @@ Version 12 of the Data Movemenet library inherits all the benefits of the versio

TODO

### Core client libraries
### Core client library migration benefits

To understand why we created our version 12 client libraries, you may refer to the Tech Community blog post, [Announcing the Azure Storage v12 Client Libraries](https://techcommunity.microsoft.com/t5/azure-storage/announcing-the-azure-storage-v12-client-libraries/ba-p/1482394) or refer to our video [Introducing the New Azure SDKs](https://aka.ms/azsdk/intro).

Expand Down Expand Up @@ -52,6 +55,79 @@ dotnet add package Azure.Storage.DataMovement.Blobs

Note the separation of the Data Movement package from the Blob Storage Data Movement package. Packages to other storage services can be added or removed from your installation.

### Type Structure

This section to discuss the following as important structural API changes:

- Changes to the `TransferManager`
- New `StorageResource` type and storage resource providers
- Replacing `TransferContext` with `TransferOptions`

#### TransferManager

The `TransferManager` is still the core type that handles transfers in Data Movement, though its API has been completely rewritten.
It is no longer a static type, though **we recommend maintaining a singleton instance** for optimal usage.

All transfer methods for upload, download, and copy, for both files and directories, have been replaced with a single instance method:
```csharp
Task<TransferOperation> StartTransferAsync(
StorageResource sourceResource,
StorageResource destinationResource,
TransferOptions transferOptions = default,
CancellationToken cancellationToken = default)
```
The appropriate upload, download, or copy operation will be performed based on the `StorageResource` instances passed. E.g., a `sourceResource` pointing to a local file and a `destinationResource` pointing to an Azure blob will perform an upload. More information on `StorageResource` can be found in the following section on [storage resources and providers](#storage-resources-and-providers).

The following table is a quick reference of where to find functionality from legacy `TransferManager` APIs. This **does not** imply fully identical behavior between libraries.

| Operation | Legacy `TransferManager` API | Modern API
| --------- | ---------- | ---------- |
| Upload, Download, Copy | `UploadAsync`, `UploadDirectoryAsync`, `DownloadAsync`, `DownloadDirectoryAsync`, `CopyAsync`, `CopyDirectoryAsync` | `TransferManager.StartTransferAsync`
| Limit concurrent transfers | `Configurations.ParallelOperations` | `TransferManagerOptions.MaximumConcurrency` (applied at TransferManager construction)
| Limit concurrent listings | `Configurations.MaxListingConcurrency` | Not implemented
| Limit memory usage | `Configurations.MaximumCacheSize` | Not implemented
| Set chunking size | `Configurations.BlockSize` | `TransferOptions.MaximumTransferChunkSize` (applied at TransferManager construction)
| Alter user agent string | `Configurations.UserAgentPrefix` | Not implemented

#### Storage Resources and Providers

The modern library introduces a new and unified `StorageResource` type. While the legacy library directly used types like `CloudBlob` and `CloudFile` in its API, the modern library uses `StorageResource` to define transfer source and destination. Storage resources can be further divided into two subtypes: **items** and **containers**. Examples of items are block blobs and local files. Examples of containers are blob containers and local folders.

**Important:** a transfer **must** be between two items or two containers. An exception will be thrown when attempting to transfer an item to a container and vice versa.

`StorageResource` instances are obtained through providers. Providers are often scoped to a single storage service and will have unique APIs to acquire `StorageResource` instances as well as properly authenticate them. Here is an example using providers to create an upload to an Azure blob. Further examples can be found in our [migration samples](#migration-samples).

```csharp
LocalFilesStorageResourceProvider files = new();
BlobsStorageResourceProvider blobs = new(myTokenCredential);
TransferManager transferManager = new TransferManager();

TransferOperation transferOperation = await transferManager.StartTransferAsync(
sourceResource: files.FromFile(sourceLocalPath),
destinationResource: blobs.FromBlob(destinationBlobUri));
```

Providers can also be registered with the `TransferManager` at construction as a mechanism to resume in case of a failure. (TODO link to resume sample).

#### TransferContext &rarr; TransferOptions

In the legacy library, `TransferContext` was an object with per-transfer specifications. In the modern library, `TransferOptions` is its replacement.

Not all functionality exposed by `TransferContext` is currently supported in the modern Data Movement library.

The following table is a quick reference of where to find functionality from legacy `TransferContext` APIs. This **does not** imply fully identical behavior between libraries.

| Configuration | Legacy `TransferContext` API | Modern API
| ------------- | ---------------------------- | ----------
| Checkpointing | `TransferCheckpoint` provided at `TransferContext` construction | `TransferCheckpointStoreOptions` provided at `TransferManagerOptions` construction.
| Observe checkpointing | `LastCheckpoint` | Not implemented
| Client request ID | `ClientRequestId` | Not implemented
| Logging | `LogLevel` | **No longer per-transfer**. `TransferManagerOptions.Diagnostics`.
| Overwrite | `ShouldOverwriteCallbackAsync` | `TransferOptions.CreationPreference`
| Progress Handling | `ProgressHandler` | `TransferOptions.ProgressHandlerOptions`
| Transfer Updates | `FileTransferred`, `FileSkipped`, `FileFailed` | `TransferOptions.ItemTransferCompleted`, `TransferOptions.ItemTransferFailed`, `TransferOptions.ItemTransferSkipped`, `TransferOptions.TransferStatusChanged`
| Attributes | `SetAttributesContextAsync` | TODO

### Authentication

#### Azure Active Directory
Expand All @@ -68,14 +144,6 @@ TODO

TODO

### Type Structure

TODO TransferManager changes

TODO StorageResource, Container and Item

TODO Providers

## Migration Samples

TODO
Expand Down
Loading