[RFC] Remove explicit resources/types for alternate Azure API versions

[mjeffryes]

Hi Folks!

We're exploring options to significantly shrink the Azure Native SDK size in the next major version. The proposal below would reduce SDK size by about 75% for all languages by removing typed resources for alternate API versions from the default SDK.

Please have a look at the document below and feel free to share your thoughts with us.

Thanks!

Context

Pulumi's Azure Native provider is based on the Open API specifications of Azure Resource Manager published by Microsoft. Pulumi generates resources automatically from those API specifications to maximize completeness and fidelity with the Azure platform. Every Azure API endpoint defines a set of API versions that it accepts and breaking changes may occur between API versions. Pulumi currently selects a default Azure API version to use for each resource, but also generates alternative resources based on the specs for alternate Azure API versions supported on these endpoints.

A recurring pain point we've heard from users is that the size of the SDKs for this provider causes problems for their tools; eg. slowing down LSPs to the point they become unusable. The SDKs also grow between major releases as we add types for each new azure API version. Eventually we encounter language package managers limitations (eg max number of files, or max bundled size) that force us to cut a new major version where we prune a large number of resources for explicit Azure API versions in order to continue publishing.

A significant portion of the bulk of the Azure Native SDKs is the resources and types for explicit Azure API versions. However, these resources are used infrequently in practice so most users are not benefiting from a sizable portion of the current SDK.

Goals

• Reduce the size of the Azure Native SDK significantly for most users
• Retain the option for users to manage resources at an explicit Azure API version when needed

Proposal

We propose to release a new major version of the Azure Native provider that does not generate resources for explicit API versions. Instead, this provider would offer two options for accessing alternative API versions: 1) a generic Resource that allows untyped access to any API version and 2) an option to generate local SDKs that use an alternate API version.

Implications:

1. All Azure Native SDK users would benefit from a significantly smaller SDK
2. Users who need to access an API version other than the default would need to adopt a new approach for these resources; either:
   1. Using the generic Resource type with an untyped property map for resource specific properties, or
   2. Using pulumi package add to generate a local SDK for the resources they need to use with an alternate API version

Expected SDK Size Impact

	Current	Proposed	Reduction
.Net	73134 files (74.19 MB)	16890 files (~17MB)	76.9%
JS/TS	57230 files (398 MB)	13880 files (~97MB)	75.7%
Python	20735 files (104.6 MB)	4926 files (~25MB)	76.3%

Code samples

Creating a storage account with the generic resource

In typescript:

const resourceResource = new azure_native.resources.Resource("acc", {
    resourceProviderNamespace: "Microsoft.Storage",
    resourceType: "storageAccounts",
    apiVersion: "2022-09-01",
    resourceGroupName: resourceGroup.Name,
    kind: "StorageV2",
    sku: {
        name: "Standard_LRS",
    },
    Properties: {
      "isSftpEnabled": "true"
    }
});

In go:

_, err = resources.NewResource(ctx, "acc", &resources.ResourceArgs{
    ResourceProviderNamespace: pulumi.String("Microsoft.Storage"),
    ResourceType:              pulumi.String("storageAccounts"),
    ApiVersion:                pulumi.String("2022-09-01"),
    ResourceGroupName:         resourceGroup.Name,
    Kind:                      pulumi.String("StorageV2"),
    Sku: &resources.SkuArgs{
        Name: pulumi.String("Standard_LRS"),
    },
    Properties: &map[string]string {
      "isSftpEnabled": "true"
    }
})

Creating a storage account using a local SDK

1. Generate a local package that uses a specific API version:

   pulumi package add azure-native storage v20240101

2. Use this local package in the program

import * as resources from "@pulumi/azure-native/resources";
import * as storage_v20240101 from "@pulumi/azure-native_storage_v20240101";

// Create an Azure Resource Group
const resourceGroup = new resources.ResourceGroup("resourceGroup");

// Create an Azure resource (Storage Account)
const storageAccount = new storage_v20240101.storage.v20240101.StorageAccount("sa", {
	resourceGroupName: resourceGroup.name,
	sku: {
    	name: storage_v20240101.storage.v20240101.SkuName.Standard_LRS,
	},
	kind: storage_v20240101.storage.v20240101.Kind.StorageV2,
});

Migration

A significant consideration in the user experience is how users can switch between default and explicit versions. There are two basic migration scenarios to consider:

1. Moving from the default version to an explicit API version. The user may want to switch to an explicit version from the default version to access the features of a newer API version, or to avoid moving to a new API version when default versions change in the provider.

2. Moving from an explicit API version to the default version. The user may want to revert back to the default version when it meets their needs to benefit from better type checking.

During the migration to the new major version there are two other variations to consider:

3. Moving from a resource with explicit API in v2 to the default version in v3
4. Moving from a resource with explicit API in v2 to the generic resource for explicit API versions in v3

All of the above migrations will require code changes to match the new type of resource the user would like to use. However, it should be possible to migrate state automatically in most of these scenarios using a combination of aliases and default state migrations:

• The default version resources in v3 will include aliases that allow them to read state from explicit versions from v2.
• The default version resources in v3 will include an alias and state upgrade from the generic resource type in v3 to allow reading the state and matching the properties to the structured type.
• The generic resource in v3 will include aliases for all default resources in v3 as well as aliases for all explicit versions in v2. It will also have a state upgrade to generalize the properties.

With these aliases and state upgrades, we should not generate any property diffs in the migration scenarios above when the targeted API version is the same. (However, there will be diffs when changing the API version used if the new version has different properties than the previous version.)

Alternatives/Future work

While we expect that the changes proposed above will be a significant improvement for most users of this provider, we will still Consider splitting up the package by namespace · pulumi-azure-native/3124 to give users even more control over how much of the Azure API they import in their programs.

[JasonWhall]

Hi @mjeffryes - Speaking as a C# user of the Azure Native Provider, I don't generally see the problems as an end user of the package size and much prefer the current interface of using namespaced API versions with properties as it remains strongly typed and easier to develop against.

I don't have any strong opinion on this change personally as it's generally quite rare to use a non-default API version. Here are some examples of where I've had to deviate and would be affected by this change:

Policy and governance contention with Azure resource implementation

Currently there are some security controls required that can only be achieved by using a preview version of an API. For example
Event Grid Topics only expose the Minimal TLS version in a preview version of their API

Leveraging new product features/capability

There are certain resources that make new features available in Preview versions of their API, even if the resource shape hasn't changed. We currently leverage this for things like Machine Learning Services where there are lots of rapid changes introduced.

Breaking changes or incompatibility

Some breaking changes may be introduced to how a resource is created/deployed that may need to be opted-out of until ready. Previously this has been handled by controlling the major version of the Azure Native provider used, but also pinning to the namespaced version if needed for longer periods of time. An example of this previously was Azure Container Apps have made changes to their hosting model to support workload profiles and consumption which required pinning for a period of time until ready to migrate.

Having an interface as you've described feels a lot like the AzApi provider, which coming from using Terraform was useful as "an escape hatch" when the primary provider didn't meet the needs, but required lots of referring back to documentation and other examples which was closer to just working with the REST API directly and felt more like ARM 🙈.

I assume other alternatives have been considered i.e. packages per resource type or a package for the default API versions and separate for "all other versions?

[realrubberduckdev]

Hi @mjeffryes

I agree with @JasonWhall that package size doesn't seem like that big an issue from an end user's perspective.
Myself and other Pulumi users I know of have not hit the issues described in the context section of this post.

Losing the strongly typed nature of resources, mainly for dotnet developers will be a big setback. Having a mix of strongly typed in one case and not strongly typed in another makes development more complex.

As suggested above, I would also prefer package per resource type or package per API version. Follows one pattern but is split into multiple packages.

[Jacse]

In Typescript this is a really big problem, rendering the provider almost useless. Because of the huge type files our developer's language servers go into a continuous crash loop and we've had to basically build our own version just exporting the default api versions to even be able to use it. Even then, language server will frequently take 6-8GBs of memory which is completely crazy.

There's a large number of issues in the history pointing at this (e.g. #932, #1997) and I've heard from multiple people that they've gone so far as to publish the individual resources as their own packages.

I think the best of both worlds solution is to have a small default package azure-native which only contains default/current versions and then publish individual packages for other versions if needed.

[mjeffryes]

Thanks for the feedback so far! It's especially helpful to learn about the specific resources where you've needed to select a different API version. (@thomas11 maybe we can check if these are resources where we're already looking to bump the default version in v3?) From what that Pulumi Cloud users have shared with us, we believe about 5% of current resources are deployed with explicit versions. (However usage is very concentrated by user, so most users are using few or no explicit versions.)

Also interesting to hear that size has maybe not been as big of concern for users of the .NET SDK.

I assume other alternatives have been considered i.e. packages per resource type or a package for the default API versions and separate for "all other versions?

Yes, we've considered these options (and we may still consider releasing per namespace packages in the future for further granularity). Unfortunately, we're not set up well to release multiple SDKs for a provider at the moment, so it's not something we could do right away.

I think the best of both worlds solution is to have a small default package azure-native which only contains default/current versions and then publish individual packages for other versions if needed.

We agree and are investigating options for the "publish individual packages for other versions if needed" part of that equation.

[thomas11]

@thomas11 maybe we can check if these are resources where we're already looking to bump the default version in v3?

We'll bump the default version for almost all resources, so we should be good.

I think the examples listed in this issue are:

• EventGrid: bumped to 2025-02-15
• MachineLearningServices: bumped to 2024-10-01

[aangelisc]

I'd love to see this as an interim solution with the longer term solution being split packages per namespace. The TS language server frequently crashes using the Azure native provider (this is further aggravated when building multi-cloud Pulumi projects).

I have previously used some preview versions of APIs for resources like Event Hub clusters and I think having the ability to continue to do so would be great but on balance I'd much prefer having a better developer experience first.

[mjeffryes]

Thanks for all the feedback so far!

Based on what we've heard, we decided to spend a little more time explore using parameterization to generate SDKs for services at explicit API versions· pulumi-azure-native/3997 in order to retain an option for strongly typed use of explicit versions while still shrinking the SDK size. And it looks like it works!

I've updated the RFC text above to reflect this, but to summarize:
Our release branch for the next major version now supports an option to generate a "local package" for any resources where you'd like to use an explicit version. For example to create a storage account using v20240101 you would:

1. Generate a local package that uses a specific API version:

   pulumi package add azure-native storage v20240101

2. Use this local package in the program

import * as resources from "@pulumi/azure-native/resources";
import * as storage_v20240101 from "@pulumi/azure-native_storage_v20240101";

// Create an Azure Resource Group
const resourceGroup = new resources.ResourceGroup("resourceGroup");

// Create an Azure resource (Storage Account)
const storageAccount = new storage_v20240101.storage.v20240101.StorageAccount("sa", {
	resourceGroupName: resourceGroup.name,
	sku: {
    	name: storage_v20240101.storage.v20240101.SkuName.Standard_LRS,
	},
	kind: storage_v20240101.storage.v20240101.Kind.StorageV2,
});

[mjeffryes]

Another update: We now have a release candidate published that implements the above plan: v3.0.0-alpha.2 (npm, PyPI, NuGet)

Please do try it out and give us your feedback if you have a chance!

[aangelisc]

🥳 thank you for the update @mjeffryes! Will give it a try 😊

[aangelisc]

FYI @mjeffryes I just tried this out and I'm seeing some missing resources such as VirtualNetworkLink and PrivateZone when trying to import like this in the NodeSDK (I've checked master and these files exist):

import {
  PrivateZone,
  VirtualNetworkLink,
} from "@pulumi/azure-native/network";