# Testing Dev Container Features A built-in testing framework for Features is in active development. This command helps you iterate on [self-authored Features](https://containers.dev/implementors/features-distribution/). The `test` command utilizes the CLI's `build` and `exec` commands to test Features in your local source tree. The command will look at the `target` path for mirrored `src` and `test` directories ([example](https://github.com/devcontainers/features). Without any additional arguments, the `test` command will auto-generate a test for each Feature (pulling the source code directly from `src/`), and exec `test//test.sh` inside of the running container. For the test to pass, the container must (1) build the generated dev container and start successfully, and (2) execute the `test.sh` with a success (zero) exit code. Note that auto-generated tests will execute the given Feature with default options. There are additional 'modes' that run an additional set of tests per Feature if the required assertion script is present. The table below highlights other modes: | Mode | Use Case | Details | CLI Flag | | ---- | ----------- | ---------------- | ----- | | Auto-generated | Auto-generates a barebones dev container with (1) the target Features, (2) the `--base-image`, and the (3) `--remote-user`) | Quick way to assert the default behavior of a Feature | `--skip-autogenerated` | Scenarios | Define more complicated test scenarios. Use a scenario to test Feature options or >1 Feature in a container. | [Details](#scenarios)| `--skip-scenarios` | Duplicate Tests | A mode that will generate a dev container installing the same Feature twice with different options | [Details](#duplicate-style-tests)| `--skip-duplicated` The source code of the sub-command is [here](../../src/spec-node/featuresCLI/test.ts). An example of the command being used in CI can be [found in the `devcontainers/feature-starter` repo](https://github.com/devcontainers/feature-starter/blob/main/.github/workflows/test.yaml) and the [`devcontainers/features` repo](https://github.com/devcontainers/features). For more information on the `test` command, run `devcontainer features test --help`. An example project structure can be found below. ``` . ├── README.md ├── src │ ├── dotnet │ │ ├── devcontainer-feature.json │ │ └── install.sh │ ├── oryx │ │ ├── devcontainer-feature.json │ │ └── install.sh | ├── ... │ │ ├── devcontainer-feature.json │ │ └── install.sh ├── test │ ├── _global │ │ ├── scenarios.json │ │ └── some_test_scenario.sh │ ├── dotnet | | ├── duplicate.sh │ │ └── test.sh │ ├── oryx | | ├── scenarios.json | | ├── install_dotnet_and_oryx.sh │ | └── test.sh | ├── ... │ │ └── test.sh ... ``` To run all the `dotnet` related tests from a repo structured above, the command would be: ```bash devcontainer features test -f dotnet --base-image ubuntu ``` ## Scenarios Scenarios are an additional mode that augments the auto-generated test (that is asserted with the `test//test.sh` script). Scenarios are snippets of `devcontainer.json` configuration. The scenario is a JSON object, where the key is the test name, and the object is a `devcontainer.json`. > The following example references the [`oryx` Feature](https://github.com/devcontainers/features/tree/2d89dc301ed834d74b07350c7d8567eee78f966d/test/oryx). The following `scenarios.json` defines a single test scenario named `install_dotnet_and_oryx`. The scenario will install the `dotnet` and `oryx` Features in the target repo with the provided options. ##### test/oryx/scenarios.json ```json { "install_dotnet_and_oryx": { "image": "ubuntu:focal", "features": { "dotnet": { "version": "6", "installUsingApt": "false" }, "oryx": {} } } } ``` The test command will build a container with the config above, and then look for a `.sh` test file with the same name. The test will pass if the container builds successfully and the `install_dotnet_and_oryx.sh` shell script exits will a successful exit code (0). ##### test/install_dotnet_and_oryx.sh ```shell #!/bin/bash set -e # Import test library for `check` command source dev-container-features-test-lib check "Oryx version" oryx --version check "Dotnet is not removed if it is not installed by the Oryx Feature" dotnet --version # Install platforms with oryx build tool check "oryx-install-dotnet-2.1" oryx prep --skip-detection --platforms-and-versions dotnet=2.1.30 check "dotnet-2-installed-by-oryx" ls /opt/dotnet/ | grep 2.1 .... .... # Replicates Oryx's behavior for universal image mkdir -p /opt/oryx echo "vso-focal" >> /opt/oryx/.imagetype mkdir -p /opt/dotnet/lts cp -R /usr/local/dotnet/current/dotnet /opt/dotnet/lts cp -R /usr/local/dotnet/current/LICENSE.txt /opt/dotnet/lts cp -R /usr/local/dotnet/current/ThirdPartyNotices.txt /opt/dotnet/lts .... .... # Report result reportResults ``` ### Providing additional build files (Dockerfile, lifecycle scripts, etc...) The pattern below can be used to provide additional files to the container build. The `test` command will look for a `test//` directory, and copy the contents into the hidden, intermediate `.devcontainer` folder used for the build. This can be used to provide additional files to the container, such as a `Dockerfile` or lifecycle scripts. For example, given a scenario defined in a Feature's `scenarios.json`: ```jsonc // ... "frowning_with_a_dockerfile": { "build": { "dockerfile": "Dockerfile" }, "features": { "smile": { "shouldFrown": true } } } // ... ``` The following file structure can be used to correctly "wire in" the Dockerfile for the build of this scenario: ``` . ├── src │ └── smile │ ├── devcontainer-feature.json │ └── install.sh └── test └── smile ├── frowning.sh ├── frowning_with_a_dockerfile <------------ new folder │ └── Dockerfile <----- gets copied over to generated .devcontainer ├── frowning_with_a_dockerfile.sh ├── scenarios.json ├── smiling.sh └── test.sh ``` The intermediate `.devcontainer` folder will look similar to this. Notice that the Dockerfile will be correctly evaluated since it is placed into the `.devcontainer` folder. A full example is codified as a [test in this repository](https://github.com/devcontainers/cli/tree/de6be379622554c5c15fa5a02403006575e087b3/src/test/container-features/example-v2-features-sets/dockerfile-scenario-test). ``` /tmp/devcontainercli/container-features-test/1667949592814 $ tree -a . ├── .devcontainer │ ├── devcontainer.json │ ├── Dockerfile │ └── smile │ ├── devcontainer-feature.json │ └── install.sh │ ├── dev-container-features-test-lib ├── frowning.sh ├── frowning_with_a_dockerfile │ └── Dockerfile ├── frowning_with_a_dockerfile.sh ├── scenarios.json ├── smiling.sh └── test.sh ``` > NOTE: The flags `--global-scenarios-only`, `--skip-scenarios`, and `--skip-autogenerated` can be passed to run a subset of tests. ### Global Scenarios The `test/_global` directory is a special directory that holds scenario tests not tied to a specific Feature. This directory is useful for scenarios that broadly tests several Features in a given repository. The `--global-scenarios-only` can be passed to only run the global scenarios. ### Duplicate-style Tests When executing the command without the `--skip-duplicated` flag, each Feature with a `duplicate.sh` will generate a test installing a given Feature twice (with different options). This is useful for asserting that a Feature can be installed multiple times without conflict (is idempotent). Additionally, the options used for each distinct Feature are passed into the assertion script, should that be useful to write an assertion. For example, the `dotnet` Feature above provided a `duplicate.sh`, therefore the test command will generate a dev container test case installing `dotnet` twice. The generated dev container could look something like this: #### `devcontainer.json` ```jsonc { "image": "ubuntu", "features": { "./dotnet": { "version": "5", "installUsingApt": "false" }, "./dotnet-0": {} // Default } } ``` #### `duplicate.sh` ```shell #!/bin/bash set -e # Optional: Import test library source dev-container-features-test-lib # The values of the randomized options will be set as environment variables. if [ -z "${VERSION}" ]; then echo "Version of dotnet to install from randomized Feature not set!" exit 1 fi if [ -z "${INSTALLUSINGAPT}" ]; then echo "Boolean to install using apt from randomized Feature not set!" exit 1 fi # The values of the default options will be set as environment variables. if [ -z "${VERSION__DEFAULT}" ]; then echo "Version of dotnet to install default Feature not set!" exit 1 fi if [ -z "${INSTALLUSINGAPT__DEFAULT}" ]; then echo "oolean to install using apt from default Feature not set!" exit 1 fi check "randomized version of dotnet installed" bash -c "dotnet --list-sdks | ${VERSION}" check "default version of dotnet installed" bash -c "dotnet --list-sdks | ${VERSION__DEFAULT}" # ... # ... # Report result reportResults ``` ## dev-container-features-test-lib The `dev-container-features-test-lib` is convenience helper [defined in the CLI](https://github.com/devcontainers/cli/blob/1910ca41015c627b884ddd69ebc52d1e8cdd8cf0/src/spec-node/featuresCLI/utils.ts#L59) that adds several bash functions to organize test asserts. Note that using this libary **is not required**. #### `check