Introducing Devshell: like virtualenv, for every language

STATUS: unstable

It should not take more than 10 minutes from the time you clone a repo and can start contributing.

Unfortunately, an unbounded amount of time is usually spent installing build dependencies on the system. If you are lucky, it's a pure $LANG project and all it takes is to install that language and its dedicated package manager. On bigger projects it's quite common to need more than one language to be installed. The side-effect of that is that it creates silos withing companies, and less contributors in the open-source world.

It should be possible to run a single command that loads and makes those dependencies available to the developer.

And it should keep the scope of these dependencies at the project level, just like virtualenv.

These are the goals of this project.

Getting started

This project has a single dependency; Nix. It will be used to pull in all other dependencies. It can be installed by following the instructions over there: https://nixos.org/download.html#nix-quick-install

Now that's done, got to your project root and create an empty devshell.toml.

There are different ways to load that config depending on your preferences:

Nix (non-flake)

Add another file called shell.nix with the following content. This file will contain some nix code. Don't worry about the details.

{ system ? builtins.currentSystem }:
let
  src = fetchTarball "https://github.com/numtide/devshell/archive/main.tar.gz";
  devshell = import src { inherit system; };
in
devshell.fromTOML ./devshell.toml

NOTE: it's probably a good idea to pin the dependency by replacing main with a git commit ID.

Now you can enter the developer shell for the project:

$ nix-shell
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
these 4 derivations will be built:
  /nix/store/nvbfq9h68r63k5jkfnbimny3b35sx0fs-devshell-bashrc.drv
  /nix/store/ppfyf9zv023an8477hcbjlj0rbyvmwq7-devshell.env.drv
  /nix/store/8027cgy3xcinb59aaynh899q953dnzms-devshell-bin.drv
  /nix/store/w33zl180ni880p18sls5ykih88zkmkqk-devshell.drv
building '/nix/store/nvbfq9h68r63k5jkfnbimny3b35sx0fs-devshell-bashrc.drv'...
building '/nix/store/ppfyf9zv023an8477hcbjlj0rbyvmwq7-devshell-env.drv'...
created 1 symlinks in user environment
building '/nix/store/8027cgy3xcinb59aaynh899q953dnzms-devshell-bin.drv'...
building '/nix/store/w33zl180ni880p18sls5ykih88zkmkqk-devshell.drv'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
🔨 Welcome to devshell

[general commands]

  menu - prints this menu

[devshell]$

Nix Flakes

For users of nix flakes, a default template is provided to get you up and running.

nix flake new -t "github:numtide/devshell" project/

cd project/

# enter the shell
nix develop # or `direnv allow` if you want to use direnv

Find templates/gettingStartedExample in this repository for a working example of the additional configuration below: env, packages, and serviceGroups.

Adding environment variables

Environment variables that are specific to the project can be added with the [[env]] declaration. Each environment variable is an entry in an array, and will be set in the order that they are declared.

Eg:

[[env]]
name = "GO111MODULE"
value = "on"

There are different ways to set the environment variables. Look at the schema to find all the ways. But in short:

  • Use the value key to set a fixed env var.
  • Use the eval key to evaluate the value. This is useful when one env var depends on the value of another.
  • Use the prefix key to prepend a path to an environment variable that uses the path separator. Like PATH.

Adding new commands

Devshell also supports adding new commands to the environment. Those are displayed on devshell entry so that the user knows what commands are available to them.

In order to bring in new dependencies, you can either add them to devshell.packages or as an entry in the [[commands]] list (see TOML docs). Commands are also added to the menu so they might be preferable for discoverability.

As an exercise, add the following snippet to your devshell.toml:

[[commands]]
package = "go"

Then re-enter the shell with nix-shell/nix develop/direnv reload. You should see something like this:

$ nix-shell
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
these 4 derivations will be built:
  /nix/store/nvbfq9h68r63k5jkfnbimny3b35sx0fs-devshell-bashrc.drv
  /nix/store/ppfyf9zv023an8477hcbjlj0rbyvmwq7-devshell.env.drv
  /nix/store/8027cgy3xcinb59aaynh899q953dnzms-devshell-bin.drv
  /nix/store/w33zl180ni880p18sls5ykih88zkmkqk-devshell.drv
building '/nix/store/nvbfq9h68r63k5jkfnbimny3b35sx0fs-devshell-bashrc.drv'...
building '/nix/store/ppfyf9zv023an8477hcbjlj0rbyvmwq7-devshell-env.drv'...
created 1 symlinks in user environment
building '/nix/store/8027cgy3xcinb59aaynh899q953dnzms-devshell-bin.drv'...
building '/nix/store/w33zl180ni880p18sls5ykih88zkmkqk-devshell.drv'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
🔨 Welcome to devshell

[general commands]

  menu - prints this menu
  go   - The Go Programming language

[devshell]$

Now the go program is available in the environment and can be used to develop Go programs. This can easily be adapted to any language.

Similarly, you could also add go to the packages list, in which case it would not appear in the menu:

[devshell]
packages = [
  "go"
]

Finding packages

Check out the Nix package repository.

Note that it is also possible to use specific versions for some packages - e.g. for NodeJS, search the repo & use like this:

[[commands]]
package = "nodejs-17_x" # https://search.nixos.org/packages?type=packages&query=nodejs
name = "node"
help = "NodeJS"

Or another example:

[devshell]
packages = [
  "python27", # 2.7
  "python311", # 3.11
]

Adding background services

Many projects need background services to be running during development or to run tests (e.g. a database, caching server, webserver, ...). This is supported in devshell through the concept of service groups.

A service group defines a collection of long-running processes that can be started and stopped.

An example service group could be configured like this:

[serviceGroups.database]
description = "Runs a database in the backgroup"
[serviceGroups.database.services.postgres]
command = "postgres"
[serviceGroups.database.services.memcached]
command = "memcached"

This will add two commands to the devshell: database:start and database:stop. database:start starts the defined services in the database group in the foreground and shows their output. database:stop can be executed in a different shell to stop the processes (or press Ctrl-C in the main shell).

Wrapping up

devshell is extensible in many different ways. In the next chapters we will discuss the various ways in which it can be adapted to your project's needs.

Continuous Integration setup (CI)

Traditionally, the CI build environment has to be kept in sync with the project. If the project needs make to build, the CI has to be configured to have it available. This can become quite tricky whenever a version requirement changes.

With devshell, the only dependency is Nix. Once the devshell is built, all the dependencies are loaded into scope and automatically are in sync with the current code checkout.

General approach

The only dependency we need installed in the CI environment is Nix.

Assuming that the shell.nix file exists, the general approach is to build it with nix to get back the entrypoint script. And then executed that script with the commands.

For example, let's say that make is being used to build the project.

The devshell.toml would have it as part of its commands:

[[commands]]
package = "gnumake"

All the CI has to do, is this: nix-shell --run "$(nix-build shell.nix)/entrypoint make".

  1. $(nix-build shell.nix)/entrypoint outputs a path to the entrypoint script
  2. nix-shell --run sets the required environment variables for the entrypoint script to work.
  3. The entrypoint script is executed with make as an argument. It loads the environment.
  4. Finally make is executed in the context of the project environment, with all the same dependencies as the developer's.

Hercules CI

Hercules CI is a Nix-based continuous integration and deployment service.

Build

If you haven't packaged your project with Nix or if a check can't run in the Nix sandbox, you can run it as an effect.

ci.nix

let
  shell = import ./shell.nix {};
  pkgs = shell.pkgs;
  effectsSrc =
    builtins.fetchTarball "https://github.com/hercules-ci/hercules-ci-effects/archive/COMMIT_HASH.tar.gz";
  inherit (import effectsSrc { inherit pkgs; }) effects;
in
{
  inherit shell;
  build = effects.mkEffect {
    src = ./.;
    effectScript = ''
      go build
    '';
    inputs = [
      shell.hook
    ];
  };
}

Replace COMMIT_HASH by the latest git sha from hercules-ci-effects, or, if you prefer, you can bring effects into scope using another pinning method.

Run locally

The hci command is available in nixos-21.05 and nixos-unstable.

devshell.toml

[[commands]]
package = "hci"

Use hci effect run. Following the previous example:

hci effect run build --no-token

Shell only

To build the shell itself on x86_64-linux:

ci.nix

{
  shell = import ./shell.nix {};

  # ... any extra Nix packages you want to build; perhaps
  # pkgs = import ./default.nix {} // { recurseForDerivations = true; };
}

system

If you build for multiple systems, pass system:

import ./shell.nix { inherit system; };

GitHub Actions

Add the following file to your project. Replace the <your build command> part with whatever is needed to build the project.

.github/workflows/devshell.yml

name: devshell
on:
  push:
    branches:
      - master
  pull_request:
  workflow_dispatch:
jobs:
  build:
    strategy:
      matrix:
        os: [ ubuntu-20.04, macos-latest ]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v2
      - uses: cachix/install-nix-action@v12
      - uses: cachix/cachix-action@v8
        with:
          name: "<your cache here>"
          signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
      - run: |
          source "$(nix-build shell.nix)"
          <your build command>

TODO

Add more CI-specific examples.

Extending devshell

When the base modules that are provided by devshell are not enough, it is possible to extend it.

Extra modules

All the devshell.toml schema options that are Declared in: the extra/ folder in the schema documentation are loaded on demand.

In order to load an extra module, use the <name> in the import section. For example to make the locale options available, import locale:

devshell.toml:

imports = ["locale"]

Make sure to add this at the first statement in the file.

Now that the module has been loaded, the devshell.toml understands the locale prefix:

imports = ["locale"]

[locale]
lang = "en_US.UTF-8"

From a nix flake you would import it like

imports = ["${devshell}/extra/locale.nix"];

Building your own modules

Building your own modules requires to understand the Nix language. If this is too complicated, please reach out to the issue tracker and describe your use-case. We want to be able to support a wide variety of development scenario.

In the same way as previously introduced, devshell will also load files that are relative to the devshell.toml. For example:

imports = ["mymodule.nix"]

Will load the mymodule.nix file in the project repository and extend the devshell.toml schema accordingly.

Options

commands

Add commands to the environment.

Type:

list of (submodule)

Default value:

[ ]

Example value:

[
  {
    help = "print hello";
    name = "hello";
    command = "echo hello";
  }

  {
    package = "nixpkgs-fmt";
    category = "formatter";
  }
]

Declared in:

commands.*.package

Used to bring in a specific package. This package will be added to the environment.

Type:

null or (package or string convertible to it)

Default value:

null

Declared in:

commands.*.category

Set a free text category under which this command is grouped and shown in the help menu.

Type:

string

Default value:

"[general commands]"

Declared in:

commands.*.command

If defined, it will add a script with the name of the command, and the content of this value.

By default it generates a bash script, unless a different shebang is provided.

Type:

null or string

Default value:

null

Example value:

''
  #!/usr/bin/env python
  print("Hello")
''

Declared in:

commands.*.help

Describes what the command does in one line of text.

Type:

null or string

Default value:

null

Declared in:

commands.*.name

Name of this command. Defaults to attribute name in commands.

Type:

null or string

Default value:

null

Declared in:

devshell.packages

The set of packages to appear in the project environment.

Those packages come from https://nixos.org/NixOS/nixpkgs and can be searched by going to https://search.nixos.org/packages

Type:

list of (package or string convertible to it)

Default value:

[ ]

Declared in:

devshell.packagesFrom

Add all the build dependencies from the listed packages to the environment.

Type:

list of (package or string convertible to it)

Default value:

[ ]

Declared in:

devshell.interactive.<name>.deps

A list of other steps that this one depends on.

Type:

list of string

Default value:

[ ]

Declared in:

devshell.interactive.<name>.text

Script to run.

Type:

string

Declared in:

devshell.load_profiles

Whether to enable load etc/profiles.d/*.sh in the shell. Type:

boolean

Default value:

false

Example value:

true

Declared in:

devshell.meta

Metadata, such as 'meta.description'. Can be useful as metadata for downstream tooling.

Type:

attribute set of anything

Default value:

{ }

Declared in:

devshell.motd

Message Of The Day.

This is the welcome message that is being printed when the user opens the shell.

You may use any valid ansi color from the 8-bit ansi color table. For example, to use a green color you would use something like {106}. You may also use {bold}, {italic}, {underline}. Use {reset} to turn off all attributes.

Type:

string

Default value:

''
  {202}🔨 Welcome to devshell{reset}
  $(type -p menu &>/dev/null && menu)
''

Declared in:

devshell.name

Name of the shell environment. It usually maps to the project name.

Type:

string

Default value:

"devshell"

Declared in:

devshell.prj_root_fallback

If IN_NIX_SHELL is nonempty, or DIRENV_IN_ENVRC is set to '1', then PRJ_ROOT is set to the value of PWD.

This option specifies the path to use as the value of PRJ_ROOT in case IN_NIX_SHELL is empty or unset and DIRENV_IN_ENVRC is any value other than '1'.

Set this to null to force PRJ_ROOT to be defined at runtime (except if IN_NIX_SHELL or DIRENV_IN_ENVRC are defined as described above).

Otherwise, you can set this to a string representing the desired default path, or to a submodule of the same type valid in the 'env' options list (except that the 'name' field is ignored).

Type:

null or ((submodule) or non-empty string convertible to it)

Default value:

{
  eval = "$PWD";
}

Example value:

{
  # Use the top-level directory of the working tree
  eval = "$(git rev-parse --show-toplevel)";
};

Declared in:

devshell.prj_root_fallback.eval

Like value but not evaluated by Bash. This allows to inject other variable names or even commands using the $() notation.

Type:

null or string

Default value:

null

Example value:

"$OTHER_VAR"

Declared in:

devshell.prj_root_fallback.name

Name of the environment variable Type:

string

Declared in:

devshell.prj_root_fallback.prefix

Prepend to PATH-like environment variables.

For example name = "PATH"; prefix = "bin"; will expand the path of ./bin and prepend it to the PATH, separated by ':'.

Type:

null or string

Default value:

null

Example value:

"bin"

Declared in:

devshell.prj_root_fallback.unset

Whether to enable unsets the variable. Type:

boolean

Default value:

false

Example value:

true

Declared in:

devshell.prj_root_fallback.value

Shell-escaped value to set Type:

null or string or signed integer or boolean or package

Default value:

null

Declared in:

devshell.startup.<name>.deps

A list of other steps that this one depends on.

Type:

list of string

Default value:

[ ]

Declared in:

devshell.startup.<name>.text

Script to run.

Type:

string

Declared in:

env

Add environment variables to the shell.

Type:

list of (submodule)

Default value:

[ ]

Example value:

[
  {
    name = "HTTP_PORT";
    value = 8080;
  }
  {
    name = "PATH";
    prefix = "bin";
  }
  {
    name = "XDG_CACHE_DIR";
    eval = "$PRJ_ROOT/.cache";
  }
  {
    name = "CARGO_HOME";
    unset = true;
  }
]

Declared in:

env.*.eval

Like value but not evaluated by Bash. This allows to inject other variable names or even commands using the $() notation.

Type:

null or string

Default value:

null

Example value:

"$OTHER_VAR"

Declared in:

env.*.name

Name of the environment variable Type:

string

Declared in:

env.*.prefix

Prepend to PATH-like environment variables.

For example name = "PATH"; prefix = "bin"; will expand the path of ./bin and prepend it to the PATH, separated by ':'.

Type:

null or string

Default value:

null

Example value:

"bin"

Declared in:

env.*.unset

Whether to enable unsets the variable. Type:

boolean

Default value:

false

Example value:

true

Declared in:

env.*.value

Shell-escaped value to set Type:

null or string or signed integer or boolean or package

Default value:

null

Declared in:

extra.locale.package

Set the glibc locale package that will be used on Linux Type:

package

Default value:

"pkgs.glibcLocales"

Declared in:

extra.locale.lang

Set the language of the project Type:

null or string

Default value:

null

Example value:

"en_GB.UTF-8"

Declared in:

git.hooks.enable

Whether to enable install .git/hooks on shell entry. Type:

boolean

Default value:

false

Example value:

true

Declared in:

git.hooks.applypatch-msg.text

Text of the script to install Type:

string

Default value:

""

Declared in:

git.hooks.commit-msg.text

Text of the script to install Type:

string

Default value:

""

Declared in:

git.hooks.fsmonitor-watchman.text

Text of the script to install Type:

string

Default value:

""

Declared in:

git.hooks.post-update.text

Text of the script to install Type:

string

Default value:

""

Declared in:

git.hooks.pre-applypatch.text

Text of the script to install Type:

string

Default value:

""

Declared in:

git.hooks.pre-commit.text

Text of the script to install Type:

string

Default value:

""

Declared in:

git.hooks.pre-merge-commit.text

Text of the script to install Type:

string

Default value:

""

Declared in:

git.hooks.pre-push.text

Text of the script to install Type:

string

Default value:

""

Declared in:

git.hooks.pre-rebase.text

Text of the script to install Type:

string

Default value:

""

Declared in:

git.hooks.prepare-commit-msg.text

Text of the script to install Type:

string

Default value:

""

Declared in:

language.c.compiler

Which C compiler to use Type:

package or string convertible to it

Default value:

"pkgs.clang"

Declared in:

language.c.includes

C dependencies from nixpkgs Type:

list of (package or string convertible to it)

Default value:

[ ]

Declared in:

language.c.libraries

Use this when another language dependens on a dynamic library Type:

list of (package or string convertible to it)

Default value:

[ ]

Declared in:

language.go.package

Which go package to use Type:

package or string convertible to it

Default value:

<derivation go-1.22.5>

Example value:

pkgs.go

Declared in:

language.go.GO111MODULE

Enable Go modules Type:

one of "on", "off", "auto"

Default value:

"on"

Declared in:

language.hare.package

Which Hare package to use Type:

package or string convertible to it

Default value:

<derivation hare-0.24.2>

Example value:

pkgs.hare

Declared in:

language.hare.thirdPartyLibs

List of extra packages (coming from hareThirdParty) to add Type:

list of (package or string convertible to it)

Default value:

[ ]

Example value:

[ hareThirdParty.hare-compress ]

Declared in:

language.hare.vendoredLibs

List of paths to add to HAREPATH Type:

list of string

Default value:

[ ]

Example value:

[ ./vendor/lib ]

Declared in:

language.perl.package

Which Perl package to use Type:

package or string convertible to it

Default value:

<derivation perl-5.38.2>

Example value:

pkgs.perl538

Declared in:

language.perl.extraPackages

List of extra packages (coming from perl5XXPackages) to add Type:

list of (package or string convertible to it)

Default value:

[ ]

Example value:

[ perl538Packages.FileNext ]

Declared in:

language.perl.libraryPaths

List of paths to add to PERL5LIB Type:

list of string

Default value:

[ ]

Example value:

[ ./lib ]

Declared in:

language.ruby.package

Ruby version used by your project Type:

package or string convertible to it

Default value:

"pkgs.ruby_3_2"

Declared in:

language.ruby.nativeDeps

Use this when your gems depend on a dynamic library Type:

list of (package or string convertible to it)

Default value:

[ ]

Declared in:

language.rust.enableDefaultToolchain

Enable the default rust toolchain coming from nixpkgs Type:

boolean

Default value:

"true"

Declared in:

language.rust.packageSet

Which rust package set to use Type:

attribute set

Default value:

"pkgs.rustPlatform"

Declared in:

language.rust.tools

Which rust tools to pull from the platform package set Type:

list of string

Default value:

[
  "rustc"
  "cargo"
  "clippy"
  "rustfmt"
]

Declared in:

serviceGroups

Add services to the environment. Services can be used to group long-running processes.

Type:

attribute set of (submodule)

Default value:

{ }

Declared in:

serviceGroups.<name>.description

Short description of the service group, shown in generated commands

Type:

null or string

Default value:

null

Declared in:

serviceGroups.<name>.name

Name of the service group. Defaults to attribute name in groups.

Type:

null or string

Default value:

null

Declared in:

serviceGroups.<name>.services

Attrset of services that should be run in this group.

Type:

attribute set of (submodule)

Default value:

{ }

Declared in:

serviceGroups.<name>.services.<name>.command

Command to execute.

Type:

string

Declared in:

serviceGroups.<name>.services.<name>.name

Name of this service. Defaults to attribute name in group services.

Type:

null or string

Default value:

null

Declared in:

services.postgres.package

Which version of postgres to use Type:

package or string convertible to it

Default value:

"pkgs.postgresql"

Declared in:

services.postgres.createUserDB

Create a database named like current user on startup. This option only makes sense when setupPostgresOnStartup is true.

Type:

boolean

Default value:

true

Declared in:

services.postgres.initdbArgs

Additional arguments passed to initdb during data dir initialisation.

Type:

list of string

Default value:

[
  "--no-locale"
]

Example value:

[
  "--data-checksums"
  "--allow-group-access"
]

Declared in:

services.postgres.setupPostgresOnStartup

Whether to enable call setup-postgres on startup. Type:

boolean

Default value:

false

Example value:

true

Declared in:

Extra options

_module.args

Additional arguments passed to each module in addition to ones like lib, config, and pkgs, modulesPath.

This option is also available to all submodules. Submodules do not inherit args from their parent module, nor do they provide args to their parent module or sibling submodules. The sole exception to this is the argument name which is provided by parent modules to a submodule and contains the attribute name the submodule is bound to, or a unique generated name if it is not bound to an attribute.

Some arguments are already passed by default, of which the following cannot be changed with this option:

  • {var}lib: The nixpkgs library.

  • {var}config: The results of all options after merging the values from all modules together.

  • {var}options: The options declared in all modules.

  • {var}specialArgs: The specialArgs argument passed to evalModules.

  • All attributes of {var}specialArgs

    Whereas option values can generally depend on other option values thanks to laziness, this does not apply to imports, which must be computed statically before anything else.

    For this reason, callers of the module system can provide specialArgs which are available during import resolution.

    For NixOS, specialArgs includes {var}modulesPath, which allows you to import extra modules from the nixpkgs package tree without having to somehow make the module aware of the location of the nixpkgs or NixOS directories.

    { modulesPath, ... }: {
  imports = [
    (modulesPath + "/profiles/minimal.nix")
  ];
}

For NixOS, the default value for this option includes at least this argument:

  • {var}pkgs: The nixpkgs package set according to the {option}nixpkgs.pkgs option.

Type:

lazy attribute set of raw value

Declared in:

Env vars

This section describes a few environment variables that can influence the behaviour of devshell.

DEVSHELL_NO_MOTD=1

When that variable is set, devshell will not display the menu that is executed on entrypoint.

TODO

  • How to add a new dependency
  • Using with CI
  • Extending devshell
  • Contributing