Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add data import/export commands #156

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+196 −0
Open

feat: add data import/export commands #156

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions cmd/data/data.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
package data

import (
"github.com/spf13/cobra"
data_export "github.com/zitadel/zitadel-tools/cmd/data/export"
data_import "github.com/zitadel/zitadel-tools/cmd/data/import"
)

// Cmd represents the data root command
var Cmd = &cobra.Command{
Use: "data",
Short: "Import/Export data",
}

func init() {
issuer := Cmd.PersistentFlags().String("issuer", "", "issuer of your ZITADEL instance (in the form: https://<instance>.zitadel.cloud or https://<yourdomain>)")
api := Cmd.PersistentFlags().String("api", "", "gRPC endpoint of your ZITADEL instance (in the form: <instance>.zitadel.cloud:443 or <yourdomain>:443)")
insecure := Cmd.PersistentFlags().Bool("insecure", false, "disable TLS to connect to gRPC API (use for local development only)")
Comment on lines +16 to +18
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel that these 3 command line options are quite redundant.

For example:

  1. If the issuer is http://localhost:8080, you already know the API endpoint is localhost:8080 and the connection is insecure.
  2. If the issuer is https://<instance>.zitadel.cloud, you already know the API endpoint is <instance>.zitadel.cloud:443 and the connection is secure.

For this example you'll need to parse the URL of the issuer and imply the the API configuration from there.

The reverse can also be applied:

  1. A --host flag can point to the API and will become part of the issuer
  2. The --insecure flag toggles https to http in front of the issuer. The API port toggles between 443 and 80 by default.
  3. A --port flag sets / overrides the port suffix to the host for the issuer and API.

In the last example, if one would only set <instance>.zitadel.cloud, the default will create issuer https://<instance>.zitadel.cloud and API endpoint <instance>.zitadel.cloud:443.

For user friendlyness I would want to stick to one of the above solutions, you may pick one.

keyPath := Cmd.PersistentFlags().String("key", "", "path to the JSON machine key")

Cmd.MarkPersistentFlagRequired("issuer")
Cmd.MarkPersistentFlagRequired("api")
Cmd.MarkPersistentFlagRequired("key")

Cmd.AddCommand(data_import.Cmd(issuer, api, insecure, keyPath))
Cmd.AddCommand(data_export.Cmd(issuer, api, insecure, keyPath))
}
81 changes: 81 additions & 0 deletions cmd/data/export/export.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
package data_export

import (
"context"
"log"
"os"

"github.com/spf13/cobra"
"github.com/zitadel/zitadel-go/v2/pkg/client/admin"
"github.com/zitadel/zitadel-go/v2/pkg/client/middleware"
"github.com/zitadel/zitadel-go/v2/pkg/client/zitadel"
pb "github.com/zitadel/zitadel-go/v2/pkg/client/zitadel/admin"
"google.golang.org/protobuf/encoding/protojson"
)

// Cmd represents the export command
var Cmd = func(issuer *string, api *string, insecure *bool, keyPath *string) *cobra.Command {
var dataPath string

cmd := &cobra.Command{
Use: "export",
Short: "Export data from an instance",
Run: func(cmd *cobra.Command, args []string) {
exportData(*issuer, *api, *insecure, *keyPath, dataPath)
},
}

cmd.Flags().StringVar(&dataPath, "data", "", "path to the file where exported data will be written")
cmd.MarkFlagRequired("data")

return cmd
}

func exportData(issuer string, api string, insecure bool, keyPath string, dataPath string) {
opts := []zitadel.Option{
zitadel.WithJWTProfileTokenSource(middleware.JWTProfileFromPath(keyPath)),
}

if insecure {
opts = append(opts, zitadel.WithInsecure())
}

client, err := admin.NewClient(
issuer,
api,
[]string{zitadel.ScopeZitadelAPI()},
opts...,
)

if err != nil {
log.Fatalln("failed to create admin client:", err)
return
}

defer func() {
if err := client.Connection.Close(); err != nil {
log.Fatalln("failed to close client connection:", err)
}
}()

resp, err := client.ExportData(context.Background(), &pb.ExportDataRequest{})

if err != nil {
log.Fatalln("failed to export data:", err)
return
}

data, err := protojson.Marshal(resp)

if err != nil {
log.Fatalln("failed to marshal data:", err)
return
}

err = os.WriteFile(dataPath, data, 0644)

if err != nil {
log.Fatalln("failed to write data file:", err)
return
}
}
86 changes: 86 additions & 0 deletions cmd/data/import/import.go
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the exported json file, the import fails with:

go run main.go data import --api=localhost:8080 --insecure --issuer=http://localhost:8080 --key=275218352408933106.json --data=export.json
2024/07/09 17:29:23 failed to unmarshal data: proto: (line 1:2): unknown field "orgs"
exit status 1

That's because the Request has a Oneof field that has to be populated first. I'll make a suggestion below in the file.

Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
package data_import

import (
"context"
"log"
"os"

"github.com/spf13/cobra"
"github.com/zitadel/zitadel-go/v2/pkg/client/admin"
"github.com/zitadel/zitadel-go/v2/pkg/client/middleware"
"github.com/zitadel/zitadel-go/v2/pkg/client/zitadel"
pb "github.com/zitadel/zitadel-go/v2/pkg/client/zitadel/admin"
"google.golang.org/protobuf/encoding/protojson"
)

// Cmd represents the import command
var Cmd = func(issuer *string, api *string, insecure *bool, keyPath *string) *cobra.Command {
var dataPath string

cmd := &cobra.Command{
Use: "import",
Short: "Import data to an instance",
Run: func(cmd *cobra.Command, args []string) {
importData(*issuer, *api, *insecure, *keyPath, dataPath)
},
}

cmd.Flags().StringVar(&dataPath, "data", "", "path to the file containing data to import")
cmd.MarkFlagRequired("data")

return cmd
}

func importData(issuer string, api string, insecure bool, keyPath string, dataPath string) {
opts := []zitadel.Option{
zitadel.WithJWTProfileTokenSource(middleware.JWTProfileFromPath(keyPath)),
}

if insecure {
opts = append(opts, zitadel.WithInsecure())
}

client, err := admin.NewClient(
issuer,
api,
[]string{zitadel.ScopeZitadelAPI()},
opts...,
)

if err != nil {
log.Fatalln("failed to create admin client:", err)
return
}

defer func() {
if err := client.Connection.Close(); err != nil {
log.Fatalln("failed to close client connection:", err)
}
}()

data, err := os.ReadFile(dataPath)

if err != nil {
log.Fatalln("failed to read data file:", err)
return
}

var req pb.ImportDataRequest

err = protojson.Unmarshal(data, &req)

if err != nil {
log.Fatalln("failed to unmarshal data:", err)
return
}

resp, err := client.ImportData(context.Background(), &req)
Comment on lines +68 to +77
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the propper message type for unmarshall like this:

Suggested change
var req pb.ImportDataRequest
err = protojson.Unmarshal(data, &req)
if err != nil {
log.Fatalln("failed to unmarshal data:", err)
return
}
resp, err := client.ImportData(context.Background(), &req)
var reqOrgs pb.ImportDataOrg
err = protojson.Unmarshal(data, &reqOrgs)
if err != nil {
log.Fatalln("failed to unmarshal data:", err)
return
}
resp, err := client.ImportData(context.Background(), &pb.ImportDataRequest{
Data: &pb.ImportDataRequest_DataOrgs{
DataOrgs: &reqOrgs,
},
Timeout: "5m", // TODO: from command line argument!
})

Timeout is a required field and will need to be parsed from the command line arguments as well.


if err != nil {
log.Fatalln("failed to import data:", err)
return
}

log.Println("Success: ", resp.Success)
log.Println("Errors: ", resp.Errors)
}
2 changes: 2 additions & 0 deletions cmd/root.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (
"github.com/spf13/cobra"

"github.com/zitadel/zitadel-tools/cmd/basicauth"
"github.com/zitadel/zitadel-tools/cmd/data"
"github.com/zitadel/zitadel-tools/cmd/jwt"
"github.com/zitadel/zitadel-tools/cmd/migration"
)
Expand All @@ -33,4 +34,5 @@ func init() {
rootCmd.AddCommand(jwt.Cmd)
rootCmd.AddCommand(basicauth.Cmd)
rootCmd.AddCommand(migration.Cmd)
rootCmd.AddCommand(data.Cmd)
}
Loading