Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: use jsonnet render engine in go binary #54

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Duologic
Copy link
Member

@Duologic Duologic commented Sep 20, 2023

This PR removes the Go implementation of the render engine and replacing it by simply invoking the jsonnet code from doc-util in a Jsonnet VM.

This'll get rid of the diverging behavior in rendering engines and reduce toil on maintenance while still providing a binary for those that prefer that.

This also removes the option show the Package data as JSON, I believe they mostly existed to debug the Extract and Transform functions of the go code.

Considering the low number of lines (~100), I've also dropped everything in main.go.

Fixes #35
Replaces #37
Replaces #39


This PR takes a very similar approach as #39 but far more destructive as it doesn't intend to maintain the Go version.

In #39, @yorinasub17 suggests that it hits a max-stack-frames on (tf-libsonnet/hashicorp-aws), I was not able to reproduce that with this PR. Performance wise it was terribly slow, I've ran the same code with jrsonnet (rust version) and that was magnitudes faster (~40x) and didn't burn up my CPU. So perhaps it would be beneficial to use the jrsonnet C bindings here instead of the native Go implementation (next PR?).

➜ time go run . -o b -J vendor ./vendor/github.com/tf-libsonnet/hashicorp-aws/5.x/main.libsonnet
go run . -o b -J vendor   500.89s user 8.70s system 167% cpu 5:03.56 total

➜ time jrsonnet -S -J vendor -c -m d --exec "(import 'doc-util/main.libsonnet').render(import 'github.com/tf-libsonnet/hashicorp-aws/5.x/main.libsonnet')"
jrsonnet -S -J vendor -c -m d --exec   13.19s user 0.84s system 99% cpu 14.158 total

➜ time jsonnet -S -J vendor -c -m d --exec "(import 'doc-util/main.libsonnet').render(import 'github.com/tf-libsonnet/hashicorp-aws/5.x/main.libsonnet')"
jsonnet -S -J vendor -c -m d --exec   473.37s user 10.30s system 164% cpu 4:53.24 total

I've also resolved the ./render.libsonnet import with a simple strings.ReplaceAll on the main.libsonnet, embedding the whole thing.

return nil
}

if err := root.Execute(); err != nil {
log.Fatalln(err)
}
}

// importer wraps jsonnet.FileImporter
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part is copied from pkg/docsonnet/load.go.

Comment on lines +59 to +68
for k, v := range output {
fullpath := filepath.Join(*dir, k)
dir := filepath.Dir(fullpath)
if err := os.MkdirAll(dir, os.ModePerm); err != nil {
return err
}
if err := os.WriteFile(fullpath, []byte(v), 0644); err != nil {
return err
}
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part is copied from pkg/render/render.go.

@Duologic
Copy link
Member Author

While running this against k8s-libsonnet, it surfaces a few changes that should be considered, I'll circle back to this when I have bit more time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update embedded jsonnet code.
1 participant