Merge #785

785: Add comprehensive support for remote docker. r=Emilgardis a=Alexhuszagh

Add comprehensive support for remote docker.

This supports the volume-based structure, and uses some nice optimizations to ensure that only the desired toolchain and cargo items are copied over. It also uses drops to ensure scoped deletion of resources, to avoid complex logic ensuring their cleanup.

It also supports persistent data volumes, through `cross-util`. In order to setup a persistent data volume, use:

```bash
cross-util volumes create
```

This will create a persistent data volume specific for your current toolchain, and will be shared by all targets. This volume is specific for the toolchain version and channel, so a new volume will be created for each toolchain.

Make sure you provide your `DOCKER_HOST` or correct engine type to ensure these are being made on the remote host. Then, run your command as before:

```bash
CROSS_REMOTE=1 cross build --target arm-unknown-linux-gnueabihf
```

Finally, you can clean up the generated volume using:

```bash
cross-util volumes remove
```

A few other utilities are present in `cross-util`:
- `volumes list`: list all volumes created by cross.
- `volumes remove`: remove all volumes created by cross.
- `volumes prune`: prune all volumes unassociated with a container.
- `containers list`: list all active containers created by cross.
- `containers remove`: remove all active containers created by cross.
- `clean`: clean all temporary data (images, containers, volumes, temp data) created by cross.

The initial implementation was done by Marc Schreiber, schrieveslaach.

A few more environment variables exist to fine-tune performance, as well as handle private dependencies.
- `CROSS_REMOTE_COPY_REGISTRY`: copy the cargo registry
- `CROSS_REMOTE_COPY_CACHE`: copy cache directories, including the target directory.

Both of these generally lead to substantial performance penalties, but can enable the use of private SSH dependencies. In either case, the use of persistent data volumes is highly recommended.

Fixes #248.
Fixes #273.
Closes #449.

Co-authored-by: Alex Huszagh <[email protected]>

Author: bors[bot]

CHANGELOG.md

@@ -12,6 +12,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
 - #803 - added `CROSS_CUSTOM_TOOLCHAIN` to disable automatic installation of components for use with tools like `cargo-bisect-rustc`
 - #795 - added images for additional toolchains maintained by cross-rs.
 - #792 - added `CROSS_CONTAINER_IN_CONTAINER` environment variable to replace `CROSS_DOCKER_IN_DOCKER`.
+- #785 - added support for remote container engines through data volumes through setting the `CROSS_REMOTE` environment variable. also adds in utility commands to create and remove persistent data volumes.
 - #782 - added `build-std` config option, which builds the rust standard library from source if enabled.
 - #678 - Add optional `target.{target}.dockerfile[.file]`, `target.{target}.dockerfile.context` and `target.{target}.dockerfile.build-args` to invoke docker/podman build before using an image.
 - #678 - Add `target.{target}.pre-build` config for running commands before building the image.

Cargo.lock

@@ -43,6 +43,10 @@ serde_json = "1"
 serde_ignored = "0.1.2"
 shell-words = "1.1.0"
 const-sha1 = "0.2.0"
+ctrlc = { version = "3.2.2", features = ["termination"] }
+directories = "4.0.1"
+walkdir = { version = "2", optional = true }
+tempfile = "3.3.0"
 
 [target.'cfg(not(windows))'.dependencies]
 nix = { version = "0.24", default-features = false, features = ["user"] }

Cargo.toml

@@ -0,0 +1,77 @@
+use std::fs;
+
+use super::containers::*;
+use super::images::*;
+use clap::Args;
+
+#[derive(Args, Debug)]
+pub struct Clean {
+    /// Provide verbose diagnostic output.
+    #[clap(short, long)]
+    pub verbose: bool,
+    /// Force removal of images.
+    #[clap(short, long)]
+    pub force: bool,
+    /// Remove local (development) images.
+    #[clap(short, long)]
+    pub local: bool,
+    /// Remove images. Default is a dry run.
+    #[clap(short, long)]
+    pub execute: bool,
+    /// Container engine (such as docker or podman).
+    #[clap(long)]
+    pub engine: Option<String>,
+}
+
+impl Clean {
+    pub fn run(self, engine: cross::docker::Engine) -> cross::Result<()> {
+        let tempdir = cross::temp::dir()?;
+        match self.execute {
+            true => {
+                if tempdir.exists() {
+                    fs::remove_dir_all(tempdir)?;
+                }
+            }
+            false => println!(
+                "fs::remove_dir_all({})",
+                cross::pretty_path(&tempdir, |_| false)
+            ),
+        }
+
+        // containers -> images -> volumes -> prune to ensure no conflicts.
+        let remove_containers = RemoveAllContainers {
+            verbose: self.verbose,
+            force: self.force,
+            execute: self.execute,
+            engine: None,
+        };
+        remove_containers.run(engine.clone())?;
+
+        let remove_images = RemoveImages {
+            targets: vec![],
+            verbose: self.verbose,
+            force: self.force,
+            local: self.local,
+            execute: self.execute,
+            engine: None,
+        };
+        remove_images.run(engine.clone())?;
+
+        let remove_volumes = RemoveAllVolumes {
+            verbose: self.verbose,
+            force: self.force,
+            execute: self.execute,
+            engine: None,
+        };
+        remove_volumes.run(engine.clone())?;
+
+        let prune_volumes = PruneVolumes {
+            verbose: self.verbose,
+            execute: self.execute,
+            engine: None,
+        };
+        prune_volumes.run(engine)?;
+
+        Ok(())
+    }
+}