virtualenv, for every language

STATUS: unstable

It should not take more than 10 minutes from the time you clone a repo and can start contributing.

Unfortunately, an unbounded amount of time is usually spent installing build dependencies on the system. If you are lucky, it's a pure $LANG project and all it takes is to install that language and its dedicated package manager. On bigger projects it's quite common to need more than one language to be installed. The side-effect of that is that it creates silos withing companies, and less contributors in the open-source world.

It should be possible to run a single command that loads and makes those dependencies available to the developer.

And it should keep the scope of these dependencies at the project level, just like virtualenv.

These are the goals of this project.

Getting started

This project has a single dependency; Nix. It will be used to pull in all other dependencies. It can be installed by following the instructions over there: https://nixos.org/download.html#nix-quick-install

Now that's done, got to your project root and create an empty devshell.toml.

There are different ways to load that config depending on your preferences:

Add another file called shell.nix with the following content. This file will contain some nix code. Don't worry about the details.

{ system ? builtins.currentSystem }:
let
  src = fetchTarball "https://github.com/numtide/devshell/archive/master.tar.gz";
  devshell = import src { inherit system; };
in
devshell.fromTOML ./devshell.toml

NOTE: it's probably a good idea to pin the dependency by replacing master with a git commit ID.

Now you can enter the developer shell for the project:

$ nix-shell
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
these 4 derivations will be built:
  /nix/store/nvbfq9h68r63k5jkfnbimny3b35sx0fs-devshell-bashrc.drv
  /nix/store/ppfyf9zv023an8477hcbjlj0rbyvmwq7-devshell.env.drv
  /nix/store/8027cgy3xcinb59aaynh899q953dnzms-devshell-bin.drv
  /nix/store/w33zl180ni880p18sls5ykih88zkmkqk-devshell.drv
building '/nix/store/nvbfq9h68r63k5jkfnbimny3b35sx0fs-devshell-bashrc.drv'...
building '/nix/store/ppfyf9zv023an8477hcbjlj0rbyvmwq7-devshell-env.drv'...
created 1 symlinks in user environment
building '/nix/store/8027cgy3xcinb59aaynh899q953dnzms-devshell-bin.drv'...
building '/nix/store/w33zl180ni880p18sls5ykih88zkmkqk-devshell.drv'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
🔨 Welcome to devshell

[general commands]

  menu - prints this menu

[devshell]$

Flakes

For users of nix flakes, a default template is provided to get you up and running.

nix flake new -t "github:numtide/devshell" devshell

cd devshell

# enter the shell
nix develop # or `direnv allow`

Adding environment variables

Environment variables that are specific to the project can be added with the [[env]] declaration. Each environment variable is an entry in an array, and will be set in the order that they are declared.

Eg:

[[env]]
name = "GO111MODULE"
value = "on"

There are different ways to set the environment variables. Look at the schema to find all the ways. But in short:

  • Use the value key to set a fixed env var.
  • Use the eval key to evaluate the value. This is useful when one env var depends on the value of another.
  • Use the prefix key to prepend a path to an environment variable that uses the path separator. Like PATH.

Adding new commands

Devshell also supports adding new commands to the environment. Those are displayed on devshell entry so that the user knows what commands are available to them.

In order to bring in new dependencies, you can either add them to devshell.packages or to the commands list. Commands are also added to the menu so they might be preferable for discovery.

As an exercise, add the following snippet to your devshell.toml:

[[commands]]
package = "go"

Then re-enter the shell with nix-shell. You should see something like this:

$ nix-shell
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
these 4 derivations will be built:
  /nix/store/nvbfq9h68r63k5jkfnbimny3b35sx0fs-devshell-bashrc.drv
  /nix/store/ppfyf9zv023an8477hcbjlj0rbyvmwq7-devshell.env.drv
  /nix/store/8027cgy3xcinb59aaynh899q953dnzms-devshell-bin.drv
  /nix/store/w33zl180ni880p18sls5ykih88zkmkqk-devshell.drv
building '/nix/store/nvbfq9h68r63k5jkfnbimny3b35sx0fs-devshell-bashrc.drv'...
building '/nix/store/ppfyf9zv023an8477hcbjlj0rbyvmwq7-devshell-env.drv'...
created 1 symlinks in user environment
building '/nix/store/8027cgy3xcinb59aaynh899q953dnzms-devshell-bin.drv'...
building '/nix/store/w33zl180ni880p18sls5ykih88zkmkqk-devshell.drv'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
🔨 Welcome to devshell

[general commands]

  menu - prints this menu
  go   - The Go Programming language

[devshell]$

Now the go program is available in the environment and can be used to develop Go programs. This can easily be adapted to any language.

Similarly, you could also add go to the packages list, in which case it would not appear in the menu:

[devshell]
packages = [
  "go"
]

devshell is extensible in many different ways. In the next chapters we will discuss the various ways in which it can be adapted to your project's needs. to find of the configuration options available.

Continuous Integration setup (CI)

Traditionally, the CI build environment has to be kept in sync with the project. If the project needs make to build, the CI has to be configured to have it available. This can become quite tricky whenever a version requirement changes.

With devshell, the only dependency is Nix. Once the devshell is built, all the dependencies are loaded into scope and automatically are in sync with the current code checkout.

General approach

The only dependency we need installed in the CI environment is Nix.

Assuming that the shell.nix file exists, the general approach is to build it with nix to get back the entrypoint script. And then executed that script with the commands.

For example, let's say that make is being used to build the project.

The devshell.toml would have it as part of its commands:

[[commands]]
package = "gnumake"

All the CI has to do, is this: $(nix-build shell.nix) make.

  1. nix-build shell.nix outputs a path to the entrypoint script
  2. The entrypoint script is executed with make as an argument. It loads the environment.
  3. Finally make is executed in the context of the project environment, with all the same dependencies as the developer's.

Hercules CI

Hercules CI is a Nix-based continuous integration and deployment service.

Build

If you haven't packaged your project with Nix or if a check can't run in the Nix sandbox, you can run it as an effect.

ci.nix

let
  shell = import ./shell.nix {};
  pkgs = shell.pkgs;
  effectsSrc =
    builtins.fetchTarball "https://github.com/hercules-ci/hercules-ci-effects/archive/COMMIT_HASH.tar.gz";
  inherit (import effectsSrc { inherit pkgs; }) effects;
in
{
  inherit shell;
  build = effects.mkEffect {
    src = ./.;
    effectScript = ''
      go build
    '';
    inputs = [
      shell.hook
    ];
  };
}

Replace COMMIT_HASH by the latest git sha from hercules-ci-effects, or, if you prefer, you can bring effects into scope using another pinning method.

Run locally

The hci command is available in nixos-21.05 and nixos-unstable.

devshell.toml

[[commands]]
package = "hci"

Use hci effect run. Following the previous example:

hci effect run build --no-token

Shell only

To build the shell itself on x86_64-linux:

ci.nix

{
  shell = import ./shell.nix {};

  # ... any extra Nix packages you want to build; perhaps
  # pkgs = import ./default.nix {} // { recurseForDerivations = true; };
}

system

If you build for multiple systems, pass system:

import ./shell.nix { inherit system; };

GitHub Actions

Add the following file to your project. Replace the <your build command> part with whatever is needed to build the project.

.github/workflows/devshell.yml

name: devshell
on:
  push:
    branches:
      - master
  pull_request:
  workflow_dispatch:
jobs:
  build:
    strategy:
      matrix:
        os: [ ubuntu-20.04, macos-latest ]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v2
      - uses: cachix/install-nix-action@v12
      - uses: cachix/cachix-action@v8
        with:
          name: "<your cache here>"
          signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
      - run: |
          source "$(nix-build shell.nix)"
          <your build command>

TODO

Add more CI-specific examples.

Extending devshell

When the base modules that are provided by devshell are not enough, it is possible to extend it.

Extra modules

All the devshell.toml schema options that are Declared in: the extra/ folder in the schema documentation are loaded on demand.

In order to load an extra module, use the <name> in the import section. For example to make the locale options available, import locale:

devshell.toml:

imports = ["locale"]

Make sure to add this at the first statement in the file.

Now that the module has been loaded, the devshell.toml understands the locale prefix:

imports = ["locale"]

[locale]
lang = "en_US.UTF-8"

Building your own modules

Building your own modules requires to understand the Nix language. If Nix this is too complicated, please reach out to the issue tracker and describe your use-case. We want to be able to support a wide variety of development scenario.

In the same way as previously introduced, devshell will also load files that are relative to the devshell.toml. For example:

imports = ["mymodule.nix"]

Will load the mymodule.nix file in the project repository and extend the devshell.toml schema accordingly.

commands

Add commands to the environment.

Default value:

[]

Type: list of submodules

Example value:

{"_type":"literalExample","text":"[\n  {\n    help = \"print hello\";\n    name = \"hello\";\n    alias = \"echo hello\";\n  }\n\n  {\n    help = \"used to format nix code\";\n    package = pkgs.nixpkgs-fmt;\n  }\n]\n"}

Declared in:

commands.*.package

Used to bring in a specific package. This package will be added to the environment.

Type: null or package or string convertible to it

Declared in:

commands.*.category

Set a free text category under which this command is grouped and shown in the help menu.

Default value:

"general commands"

Type: string

Declared in:

commands.*.command

If defined, it will add a script with the name of the command, and the content of this value.

By default it generates a bash script, unless a different shebang is provided.

Type: null or string

Example value:

"#!/usr/bin/env python\nprint(\"Hello\")\n"

Declared in:

commands.*.help

Describes what the command does in one line of text.

Type: null or string

Declared in:

commands.*.name

Name of this command. Defaults to attribute name in commands.

Type: null or string

Declared in:

devshell.packages

The set of packages to appear in the project environment.

Those packages come from https://nixos.org/NixOS/nixpkgs and can be searched by going to https://search.nixos.org/packages

Default value:

[]

Type: list of package or string convertible to its

Declared in:

devshell.interactive.<name>.deps

A list of other steps that this one depends on.

Default value:

[]

Type: list of strings

Declared in:

devshell.interactive.<name>.text

Script to run.

Type: string

Declared in:

devshell.motd

Message Of The Day.

This is the welcome message that is being printed when the user opens the shell.

Default value:

"\u001b[38;5;202m🔨 Welcome to devshell\u001b[0m\n$(type -p menu &>/dev/null && menu)\n"

Type: string

Declared in:

devshell.name

Name of the shell environment. It usually maps to the project name.

Default value:

"devshell"

Type: string

Declared in:

devshell.startup.<name>.deps

A list of other steps that this one depends on.

Default value:

[]

Type: list of strings

Declared in:

devshell.startup.<name>.text

Script to run.

Type: string

Declared in:

env

Add environment variables to the shell.

Default value:

[]

Type: list of submodules

Example value:

{"_type":"literalExample","text":"[\n  {\n    name = \"HTTP_PORT\";\n    value = 8080;\n  }\n  {\n    name = \"PATH\";\n    prefix = \"bin\";\n  }\n  {\n    name = \"XDG_CACHE_DIR\";\n    eval = \"$PRJ_ROOT/.cache\";\n  }\n]\n"}

Declared in:

env.*.eval

Like value but not evaluated by Bash. This allows to inject other variable names or even commands using the $() notation.

Type: null or string

Example value:

"$OTHER_VAR"

Declared in:

env.*.name

Name of the environment variable

Type: string

Declared in:

env.*.prefix

Prepend to PATH-like environment variables.

For example name = "PATH"; prefix = "bin"; will expand the path of ./bin and prepend it to the PATH, separated by ':'.

Type: null or string

Example value:

"bin"

Declared in:

env.*.value

Shell-escaped value to set

Type: null or string or signed integer or boolean

Declared in:

extra.locale.package

Set the glibc locale package that will be used on Linux

Default value:

pkgs.glibcLocales

Type: package

Declared in:

extra.locale.lang

Set the language of the project

Type: null or string

Example value:

"en_GB.UTF-8"

Declared in:

git.hooks.enable

Whether to enable install .git/hooks on shell entry.

Default value:

false

Type: boolean

Example value:

true

Declared in:

git.hooks.applypatch-msg.text

Text of the script to install

Default value:

""

Type: string

Declared in:

git.hooks.commit-msg.text

Text of the script to install

Default value:

""

Type: string

Declared in:

git.hooks.fsmonitor-watchman.text

Text of the script to install

Default value:

""

Type: string

Declared in:

git.hooks.post-update.text

Text of the script to install

Default value:

""

Type: string

Declared in:

git.hooks.pre-applypatch.text

Text of the script to install

Default value:

""

Type: string

Declared in:

git.hooks.pre-commit.text

Text of the script to install

Default value:

""

Type: string

Declared in:

git.hooks.pre-merge-commit.text

Text of the script to install

Default value:

""

Type: string

Declared in:

git.hooks.pre-push.text

Text of the script to install

Default value:

""

Type: string

Declared in:

git.hooks.pre-rebase.text

Text of the script to install

Default value:

""

Type: string

Declared in:

git.hooks.prepare-commit-msg.text

Text of the script to install

Default value:

""

Type: string

Declared in:

language.c.compiler

Which C compiler to use

Default value:

pkgs.clang

Type: package or string convertible to it

Declared in:

language.c.includes

C dependencies from nixpkgs

Default value:

[]

Type: list of package or string convertible to its

Declared in:

language.c.libraries

Use this when another language dependens on a dynamic library

Default value:

[]

Type: list of package or string convertible to its

Declared in:

language.go.package

Which go package to use

Default value:

"go-1.15.6"

Type: package or string convertible to it

Example value:

{"_type":"literalExample","text":"pkgs.go"}

Declared in:

language.go.GO111MODULE

Enable Go modules

Default value:

on

Type: one of "on", "off", "auto"

Declared in:

language.rust.packageSet

Which rust package set to use

Default value:

pkgs.rustPlatform

Type: attribute set

Declared in:

language.rust.tools

Which rust tools to pull from the platform package set

Default value:

["rustc","cargo","clippy","rustfmt"]

Type: list of strings

Declared in:

service.postgres.package

Which version of postgres to use

Default value:

pkgs.postgresl

Type: package or string convertible to it

Declared in:

TODO

  • How to add a new dependency
  • Using with CI
  • Extending devshell
  • Contributing