Introducing Devshell: like virtualenv, for every language
STATUS: unstable
It should not take more than 10 minutes from the time you clone a repo and can start contributing.
Unfortunately, an unbounded amount of time is usually spent installing build dependencies on the system. If you are lucky, it's a pure $LANG project and all it takes is to install that language and its dedicated package manager. On bigger projects it's quite common to need more than one language to be installed. The side-effect of that is that it creates silos withing companies, and less contributors in the open-source world.
It should be possible to run a single command that loads and makes those dependencies available to the developer.
And it should keep the scope of these dependencies at the project level, just like virtualenv.
These are the goals of this project.
Getting started
This project has a single dependency; Nix. It will be used to pull in all other dependencies. It can be installed by following the instructions over there: https://nixos.org/download.html#nix-quick-install
Now that's done, got to your project root and create an empty devshell.toml
.
There are different ways to load that config depending on your preferences:
Nix (non-flake)
Add another file called shell.nix
with the following content. This file will
contain some nix code. Don't worry about the details.
{ system ? builtins.currentSystem }:
let
src = fetchTarball "https://github.com/numtide/devshell/archive/master.tar.gz";
devshell = import src { inherit system; };
in
devshell.fromTOML ./devshell.toml
NOTE: it's probably a good idea to pin the dependency by replacing
master
with a git commit ID.
Now you can enter the developer shell for the project:
$ nix-shell
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
these 4 derivations will be built:
/nix/store/nvbfq9h68r63k5jkfnbimny3b35sx0fs-devshell-bashrc.drv
/nix/store/ppfyf9zv023an8477hcbjlj0rbyvmwq7-devshell.env.drv
/nix/store/8027cgy3xcinb59aaynh899q953dnzms-devshell-bin.drv
/nix/store/w33zl180ni880p18sls5ykih88zkmkqk-devshell.drv
building '/nix/store/nvbfq9h68r63k5jkfnbimny3b35sx0fs-devshell-bashrc.drv'...
building '/nix/store/ppfyf9zv023an8477hcbjlj0rbyvmwq7-devshell-env.drv'...
created 1 symlinks in user environment
building '/nix/store/8027cgy3xcinb59aaynh899q953dnzms-devshell-bin.drv'...
building '/nix/store/w33zl180ni880p18sls5ykih88zkmkqk-devshell.drv'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
🔨 Welcome to devshell
[general commands]
menu - prints this menu
[devshell]$
Nix Flakes
For users of nix flakes, a default template is provided to get you up and running.
nix flake new -t "github:numtide/devshell" project/
cd project/
# enter the shell
nix develop # or `direnv allow` if you want to use direnv
Adding environment variables
Environment variables that are specific to the project can be added with the
[[env]]
declaration. Each environment variable is an entry in an array, and
will be set in the order that they are declared.
Eg:
[[env]]
name = "GO111MODULE"
value = "on"
There are different ways to set the environment variables. Look at the schema to find all the ways. But in short:
- Use the
value
key to set a fixed env var. - Use the
eval
key to evaluate the value. This is useful when one env var depends on the value of another. - Use the
prefix
key to prepend a path to an environment variable that uses the path separator. LikePATH
.
Adding new commands
Devshell also supports adding new commands to the environment. Those are displayed on devshell entry so that the user knows what commands are available to them.
In order to bring in new dependencies, you can either add them to
devshell.packages
or as an entry in the [[commands]]
list (see TOML docs). Commands are also added to the
menu so they might be preferable for discoverability.
As an exercise, add the following snippet to your devshell.toml
:
[[commands]]
package = "go"
Then re-enter the shell with nix-shell
/nix develop
/direnv reload
. You should see something like this:
$ nix-shell
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
these 4 derivations will be built:
/nix/store/nvbfq9h68r63k5jkfnbimny3b35sx0fs-devshell-bashrc.drv
/nix/store/ppfyf9zv023an8477hcbjlj0rbyvmwq7-devshell.env.drv
/nix/store/8027cgy3xcinb59aaynh899q953dnzms-devshell-bin.drv
/nix/store/w33zl180ni880p18sls5ykih88zkmkqk-devshell.drv
building '/nix/store/nvbfq9h68r63k5jkfnbimny3b35sx0fs-devshell-bashrc.drv'...
building '/nix/store/ppfyf9zv023an8477hcbjlj0rbyvmwq7-devshell-env.drv'...
created 1 symlinks in user environment
building '/nix/store/8027cgy3xcinb59aaynh899q953dnzms-devshell-bin.drv'...
building '/nix/store/w33zl180ni880p18sls5ykih88zkmkqk-devshell.drv'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
🔨 Welcome to devshell
[general commands]
menu - prints this menu
go - The Go Programming language
[devshell]$
Now the go
program is available in the environment and can be used to
develop Go programs. This can easily be adapted to any language.
Similarly, you could also add go to the packages list, in which case it would not appear in the menu:
[devshell]
packages = [
"go"
]
Finding packages
Check out the Nix package repository.
Note that it is also possible to use specific versions for some packages - e.g. for NodeJS, search the repo & use like this:
[[commands]]
package = "nodejs-17_x" # https://search.nixos.org/packages?type=packages&query=nodejs
name = "node"
help = "NodeJS"
Or another example:
[devshell]
packages = [
"python27", # 2.7
"python311", # 3.11
]
Wrapping up
devshell is extensible in many different ways. In the next chapters we will discuss the various ways in which it can be adapted to your project's needs. to find of the configuration options available.
Continuous Integration setup (CI)
Traditionally, the CI build environment has to be kept in sync with the
project. If the project needs make
to build, the CI has to be configured to
have it available. This can become quite tricky whenever a version requirement
changes.
With devshell, the only dependency is Nix. Once the devshell is built, all the dependencies are loaded into scope and automatically are in sync with the current code checkout.
General approach
The only dependency we need installed in the CI environment is Nix.
Assuming that the shell.nix
file exists, the general approach is to build it
with nix to get back the entrypoint script. And then executed that script with
the commands.
For example, let's say that make
is being used to build the project.
The devshell.toml
would have it as part of its commands:
[[commands]]
package = "gnumake"
All the CI has to do, is this: nix-shell --run "$(nix-build shell.nix)/entrypoint make"
.
$(nix-build shell.nix)/entrypoint
outputs a path to the entrypoint scriptnix-shell --run
sets the required environment variables for the entrypoint script to work.- The entrypoint script is executed with
make
as an argument. It loads the environment. - Finally make is executed in the context of the project environment, with all the same dependencies as the developer's.
Hercules CI
Hercules CI is a Nix-based continuous integration and deployment service.
Build
If you haven't packaged your project with Nix or if a check can't run in the Nix sandbox, you can run it as an effect.
ci.nix
let
shell = import ./shell.nix {};
pkgs = shell.pkgs;
effectsSrc =
builtins.fetchTarball "https://github.com/hercules-ci/hercules-ci-effects/archive/COMMIT_HASH.tar.gz";
inherit (import effectsSrc { inherit pkgs; }) effects;
in
{
inherit shell;
build = effects.mkEffect {
src = ./.;
effectScript = ''
go build
'';
inputs = [
shell.hook
];
};
}
Replace COMMIT_HASH by the latest git sha from hercules-ci-effects
,
or, if you prefer, you can bring effects
into scope using another pinning method.
Run locally
The hci
command is available in nixos-21.05
and nixos-unstable
.
devshell.toml
[[commands]]
package = "hci"
Use hci effect run
. Following the previous example:
hci effect run build --no-token
Shell only
To build the shell itself on x86_64-linux
:
ci.nix
{
shell = import ./shell.nix {};
# ... any extra Nix packages you want to build; perhaps
# pkgs = import ./default.nix {} // { recurseForDerivations = true; };
}
system
If you build for multiple systems, pass system
:
import ./shell.nix { inherit system; };
GitHub Actions
Add the following file to your project. Replace the <your build command>
part with whatever is needed to build the project.
.github/workflows/devshell.yml
name: devshell
on:
push:
branches:
- master
pull_request:
workflow_dispatch:
jobs:
build:
strategy:
matrix:
os: [ ubuntu-20.04, macos-latest ]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v2
- uses: cachix/install-nix-action@v12
- uses: cachix/cachix-action@v8
with:
name: "<your cache here>"
signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
- run: |
source "$(nix-build shell.nix)"
<your build command>
TODO
Add more CI-specific examples.
Extending devshell
When the base modules that are provided by devshell are not enough, it is possible to extend it.
Extra modules
All the devshell.toml
schema options that are Declared in:
the extra/
folder in the schema documentation are loaded on demand.
In order to load an extra module, use the <name>
in the import section. For
example to make the locale
options available, import locale
:
devshell.toml
:
imports = ["locale"]
Make sure to add this at the first statement in the file.
Now that the module has been loaded, the devshell.toml
understands the
locale
prefix:
imports = ["locale"]
[locale]
lang = "en_US.UTF-8"
From a nix flake you would import it like
imports = ["${devshell}/extra/locale.nix"];
Building your own modules
Building your own modules requires to understand the Nix language. If this is too complicated, please reach out to the issue tracker and describe your use-case. We want to be able to support a wide variety of development scenario.
In the same way as previously introduced, devshell will also load files that
are relative to the devshell.toml
. For example:
imports = ["mymodule.nix"]
Will load the mymodule.nix
file in the project repository and extend the
devshell.toml
schema accordingly.
commands
Add commands to the environment.
Default value:
[]
Type: list of submodules
Example value:
{"_type":"literalExpression","text":"[\n {\n help = \"print hello\";\n name = \"hello\";\n command = \"echo hello\";\n }\n\n {\n package = \"nixpkgs-fmt\";\n category = \"formatter\";\n }\n]\n"}
Declared in:
commands.*.package
Used to bring in a specific package. This package will be added to the environment.
Type: null or package or string convertible to it
Declared in:
commands.*.category
Set a free text category under which this command is grouped and shown in the help menu.
Default value:
"general commands"
Type: string
Declared in:
commands.*.command
If defined, it will add a script with the name of the command, and the content of this value.
By default it generates a bash script, unless a different shebang is provided.
Type: null or string
Example value:
"#!/usr/bin/env python\nprint(\"Hello\")\n"
Declared in:
commands.*.help
Describes what the command does in one line of text.
Type: null or string
Declared in:
commands.*.name
Name of this command. Defaults to attribute name in commands.
Type: null or string
Declared in:
devshell.packages
The set of packages to appear in the project environment.
Those packages come from https://nixos.org/NixOS/nixpkgs and can be searched by going to https://search.nixos.org/packages
Default value:
[]
Type: list of package or string convertible to its
Declared in:
devshell.interactive.<name>.deps
A list of other steps that this one depends on.
Default value:
[]
Type: list of strings
Declared in:
devshell.interactive.<name>.text
Script to run.
Type: string
Declared in:
devshell.load_profiles
Whether to enable load etc/profiles.d/*.sh in the shell.
Default value:
false
Type: boolean
Example value:
true
Declared in:
devshell.meta
Metadata, such as 'meta.description'. Can be useful as metadata for downstream tooling.
Default value:
{}
Type: attribute set of anythings
Declared in:
devshell.motd
Message Of The Day.
This is the welcome message that is being printed when the user opens the shell.
You may use any valid ansi color from the 8-bit ansi color table. For example, to use a green color you would use something like {106}. You may also use {bold}, {italic}, {underline}. Use {reset} to turn off all attributes.
Default value:
"{202}🔨 Welcome to devshell{reset}\n$(type -p menu &>/dev/null && menu)\n"
Type: string
Declared in:
devshell.name
Name of the shell environment. It usually maps to the project name.
Default value:
"devshell"
Type: string
Declared in:
devshell.startup.<name>.deps
A list of other steps that this one depends on.
Default value:
[]
Type: list of strings
Declared in:
devshell.startup.<name>.text
Script to run.
Type: string
Declared in:
env
Add environment variables to the shell.
Default value:
[]
Type: list of submodules
Example value:
{"_type":"literalExpression","text":"[\n {\n name = \"HTTP_PORT\";\n value = 8080;\n }\n {\n name = \"PATH\";\n prefix = \"bin\";\n }\n {\n name = \"XDG_CACHE_DIR\";\n eval = \"$PRJ_ROOT/.cache\";\n }\n {\n name = \"CARGO_HOME\";\n unset = true;\n }\n]\n"}
Declared in:
env.*.eval
Like value but not evaluated by Bash. This allows to inject other
variable names or even commands using the $()
notation.
Type: null or string
Example value:
"$OTHER_VAR"
Declared in:
env.*.name
Name of the environment variable
Type: string
Declared in:
env.*.prefix
Prepend to PATH-like environment variables.
For example name = "PATH"; prefix = "bin"; will expand the path of ./bin and prepend it to the PATH, separated by ':'.
Type: null or string
Example value:
"bin"
Declared in:
env.*.unset
Whether to enable unsets the variable.
Default value:
false
Type: boolean
Example value:
true
Declared in:
env.*.value
Shell-escaped value to set
Type: null or string or signed integer or boolean or package
Declared in:
extra.locale.package
Set the glibc locale package that will be used on Linux
Default value:
pkgs.glibcLocales
Type: package
Declared in:
extra.locale.lang
Set the language of the project
Type: null or string
Example value:
"en_GB.UTF-8"
Declared in:
git.hooks.enable
Whether to enable install .git/hooks on shell entry.
Default value:
false
Type: boolean
Example value:
true
Declared in:
git.hooks.applypatch-msg.text
Text of the script to install
Default value:
""
Type: string
Declared in:
git.hooks.commit-msg.text
Text of the script to install
Default value:
""
Type: string
Declared in:
git.hooks.fsmonitor-watchman.text
Text of the script to install
Default value:
""
Type: string
Declared in:
git.hooks.post-update.text
Text of the script to install
Default value:
""
Type: string
Declared in:
git.hooks.pre-applypatch.text
Text of the script to install
Default value:
""
Type: string
Declared in:
git.hooks.pre-commit.text
Text of the script to install
Default value:
""
Type: string
Declared in:
git.hooks.pre-merge-commit.text
Text of the script to install
Default value:
""
Type: string
Declared in:
git.hooks.pre-push.text
Text of the script to install
Default value:
""
Type: string
Declared in:
git.hooks.pre-rebase.text
Text of the script to install
Default value:
""
Type: string
Declared in:
git.hooks.prepare-commit-msg.text
Text of the script to install
Default value:
""
Type: string
Declared in:
language.c.compiler
Which C compiler to use
Default value:
pkgs.clang
Type: package or string convertible to it
Declared in:
language.c.includes
C dependencies from nixpkgs
Default value:
[]
Type: list of package or string convertible to its
Declared in:
language.c.libraries
Use this when another language dependens on a dynamic library
Default value:
[]
Type: list of package or string convertible to its
Declared in:
language.go.package
Which go package to use
Default value:
"go-1.16.13"
Type: package or string convertible to it
Example value:
{"_type":"literalExpression","text":"pkgs.go"}
Declared in:
language.go.GO111MODULE
Enable Go modules
Default value:
on
Type: one of "on", "off", "auto"
Declared in:
language.rust.enableDefaultToolchain
Enable the default rust toolchain coming from nixpkgs
Default value:
true
Type: boolean
Declared in:
language.rust.packageSet
Which rust package set to use
Default value:
pkgs.rustPlatform
Type: attribute set
Declared in:
language.rust.tools
Which rust tools to pull from the platform package set
Default value:
["rustc","cargo","clippy","rustfmt"]
Type: list of strings
Declared in:
services.postgres.package
Which version of postgres to use
Default value:
pkgs.postgresql
Type: package or string convertible to it
Declared in:
services.postgres.createUserDB
Create a database named like current user on startup.
This option only makes sense when setupPostgresOnStartup
is true.
Default value:
true
Type: boolean
Declared in:
services.postgres.initdbArgs
Additional arguments passed to
Default value:
["--no-locale"]
Type: list of strings
Example value:
["--data-checksums","--allow-group-access"]
Declared in:
services.postgres.setupPostgresOnStartup
Whether to enable call setup-postgres on startup.
Default value:
false
Type: boolean
Example value:
true
Declared in:
Env vars
This section describes a few environment variables that can influence the behaviour of devshell.
DEVSHELL_NO_MOTD=1
When that variable is set, devshell will not display the menu that is executed on entrypoint.
TODO
- How to add a new dependency
- Using with CI
- Extending devshell
- Contributing