Add polling to wait for post plan tasks to complete in remote backend

[Maed223]

Description

Original Problem

It was seen that in some instances (Jira here) that using the remote backend with certain post plan tasks configured would cause the CLI to unceremoniously exit before confirmation could be given to apply runs in certain cases. This would occur due to these post plan tasks being in a running state when the backend checks if the run is confirmable.

To reproduce this behavior, set up a workspace (backed by a remote backend) with an OPA policy configured. When you perform an apply run, you'll see that the CLI exits before confirmation can be given (whether explicitly by the user or using auto-apply). Interestingly enough, I saw that enabling cost estimation avoided this erroneous behavior– since the processing of the cost estimate gave enough time for the OPA policy to complete before the confirmability of the run was checked.

Solution

Make the plan operation in the remote backend aware of post plan tasks, and have it wait for their completion before proceeding. The additions here are mostly copied from the analogous cloud backend methods, though I purposely omitted many of the features that you find in the cloud backend. The only text streamed to the console here would be notifications in the event of a task failing, or needing manual override– but the ability to do the override is not present here as it is in the cloud backend.

Target Release

1.12.x

CHANGELOG entry

The remote backend now notifies the user if post plan tasks have failed and/or need manual override.

• This change is user-facing and I added a changelog entry.
• This change is not user-facing.

[github-actions]

Changelog Warning

Currently this PR would target a v1.12 release. Please add a changelog entry for in the .changes/v1.12 folder, or discuss which release you'd like to target with your reviewer. If you believe this change does not need a changelog entry, please add the 'no-changelog-needed' label.