fix: parallel plan and apply also in a single workspace (rebased)

[plentydone]

what

@finnag did all the real work here. I just rebased and regenerated mocks. I'm new to this codebase, but I've reviewed the code and it seems reasonable to me, and I've reviewed all the commits between October and today and I see no logical conflicts. If in fresh review any changes are needed, I'm happy to do so.

From the original PR description:

- Add more thorough locking around Clone() calls, covering all of these phases:
   Am I on the right commit
   Merge with upstream
   Clone if necessary
- Reduce the number of remote git operations when planning or applying in parallel
   Clean up the Clone() method, split into Clone() and MergeAgain()

For parallel mode to work, you must either set the environment variable `TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE` to something, or check in your .hcl files. Otherwise terraform cannot run in parallel.

why

The Clone call had several race conditions where it could miss clones or delete the working directory under running processes causing failures.

tests

I ran make test-all fmt lint

references

This is just #3670 rebased

[plentydone]

I'm not sure what the etiquette is around signing off on this as the committer given that I just rebased someone else's work from their own MR.

[jamengual]

I'm not sure what the etiquette is around signing off on this as the committer given that I just rebased someone else's work from their own MR.

just sign it with yours

[jseiser]

While we wait on this to get merged, we wanted to test running it locally. Are the images built, ever pushed to ghcr?

docker pull ghcr.io/runatlantis/atlantis:pr-5264-debian                                                         
Error response from daemon: manifest unknown

[jamengual]

While we wait on this to get merged, we wanted to test running it locally. Are the images built, ever pushed to ghcr?

docker pull ghcr.io/runatlantis/atlantis:pr-5264-debian                                                         
Error response from daemon: manifest unknown

sha256:5f2412560425761d177fab5e1e878e3d9e08c0bce7a248d65eff88efde593b84

[jseiser]

sha256:5f2412560425761d177fab5e1e878e3d9e08c0bce7a248d65eff88efde593b84

Any chance you can paste syntax on this? Everything Im attempting is apparently wrong.

[jseiser]

 docker pull ghcr.io/runatlantis/atlantis@sha256:5f2412560425761d177fab5e1e878e3d9e08c0bce7a248d65eff88efde593b84                                                                                                                            
Error response from daemon: manifest unknown   

[jamengual]

We are looking into it

…

On Tue, Jan 28, 2025, 6:51 a.m. Justin Seiser ***@***.***> wrote: docker pull ***@***.***:5f2412560425761d177fab5e1e878e3d9e08c0bce7a248d65eff88efde593b84 Error response from daemon: manifest unknown — Reply to this email directly, view it on GitHub <#5264 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3ERFOI3Q7K6RSEKCV66L2M6KO7AVCNFSM6AAAAABVWNZJR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJZGIZDMMZXGQ> . You are receiving this because your review was requested.Message ID: ***@***.***>

[jseiser]

Any update on this?

[jseiser]

@jamengual

We just had an almost 2 hours long production deployment, so we would really love to be able to test this image, but it still doesnt seem to be built?

Thanks,

[jamengual]

@jamengual

We just had an almost 2 hours long production deployment, so we would really love to be able to test this image, but it still doesnt seem to be built?

Thanks,

I just updated the branch, it should build the new image in a few minutes.

[lukemassa]

FYI There were a few issues with the PR; firstly an additional callsite to Clone() added since this was written that I adjusted to expect the new number of results, secondly a test was failing, also due to the change in return values from Clone. I just lending a hand fixing up the PR

[jseiser]

Just wanted to report. We were able to deploy this image

ghcr.io/runatlantis/atlantis:dev-debian-cf0b3f3

We then created a scaled down reproduction of our monorepo. Each bottom folder you see, is a state.

└── fake
    ├── global
    │   └── user
    └── us-gov-east-1
        ├── deploys
        │   ├── f1
        │   │   ├── child
        │   │   └── parent
        │   └── f2
        │       ├── child
        │       └── parent
        └── ops
            └── network

We created a simple atlantis.yml

version: 3
automerge: true
autodiscover:
  mode: disabled
delete_source_branch_on_merge: true
parallel_plan: true
parallel_apply: true
abort_on_execution_order_fail: true
projects:
  - name: fake/global/user
    dir: fake/global/user
    execution_order_group: 1

  - name: fake/us-gov-east-1/ops/network
    dir: fake/us-gov-east-1/ops/network
    execution_order_group: 2

  - name: fake/us-gov-east-1/deploys/f1/parent
    dir: fake/us-gov-east-1/deploys/f1/parent
    execution_order_group: 3

  - name: fake/us-gov-east-1/deploys/f2/parent
    dir: fake/us-gov-east-1/deploys/f2/parent
    execution_order_group: 3

  - name: fake/us-gov-east-1/deploys/f1/child
    dir: fake/us-gov-east-1/deploys/f1/child
    execution_order_group: 4

  - name: fake/us-gov-east-1/deploys/f2/child
    dir: fake/us-gov-east-1/deploys/f2/child
    execution_order_group: 4

Things worked perfectly. On initial deployment, the plans happened in order, but would fail since we are calling the state of of 1 into order group 2 etc. Once things were deployed, creating a 2nd MR to update all the tags happened in the proper order. We also did not get any errors related to multiple states trying to access providers etc.

This would be a huge workflow improvement for our organization.

[jamengual]

could we ahh documentation for this and maybe some examples on how to use it?