Install NixOS on GCE

NOTE: as of March 2020 this page is not accurate and many changes have taken place in NixOS and in GCE. There may be useful information here but following the instructions will not lead to a happy running VM instance on GCE. Hopefully someone can figure out how to fix it because there are updates in the source code. You may want to use nixos-generators to create google cloud images.

This is a recipe for creating a NixOS machine on Google Compute Engine (GCE) which is part of Google Cloud Platform. This tutorial assumes you have already set up and account and project under Google Cloud Platform.

Preparing a NixOS image

To bootstrap the process, first start any Linux instance and log into it.

1. From the Google Developers Console select Compute - Compute Engine - VM instances
2. Press "Create an Instance" or "New instance" button
3. Click on "Show advanced options"
4. Select a ZONE close to you and any MACHINE TYPE (f1-micro is adequate)
5. Select any linux image such as centos-7
6. Make sure "Enable Compute Engine service account." is checked
7. Under COMPUTE select "Read Write"
8. Press "Create" button and wait until the VM is ready
9. From the Google Developers Console select Compute - Compute Engine - VM instances press the "SSH" button on your new instance (usually named "instance-1") to log in

Next we will create a NixOS image. A source for NixOS images can be found the google storage bucket named gs://nixos-images.

$ gsutil ls -l gs://nixos-images
 256556736  2014-12-17T10:51:00Z  gs://nixos-images/nixos-14.10pre-git-x86_64-linux.raw.tar.gz
 290985235  2014-12-19T12:45:58Z  gs://nixos-images/nixos-14.12.542.4c9ef9f7-x86_64-linux.raw.tar.gz

You can find more up-to-date images at gs://nixos-cloud-images, but the bucket itself is not public- the objects inside it are.

This means that gsutil ls gs://nixos-cloud-images will yield an AccessDeniedException, but the following works:

$ gsutil ls -l gs://nixos-cloud-images/nixos-image-18.09.1228.a4c4cbb613c-x86_64-linux.raw.tar.gz
 344473043 2018-11-16T09:16:51Z  gs://nixos-cloud-images/nixos-image-18.09.1228.a4c4cbb613c-x86_64-linux.raw.tar.gz
TOTAL: 1 objects, 344473043 bytes (328.52 MiB)

You can find a full list of available nixos-cloud-images at <nixpkgs/nixos/modules/virtualisation/gce-images.nix>.

In this tutorial we will use gs://nixos-images/nixos-14.12.542.4c9ef9f7-x86_64-linux.raw.tar.gz and follow the documentation on how to create an image from a .tar.gz file.

$ gcloud compute images create nixos-14125424c9ef9f7-x86-64-linux --source-uri gs://nixos-images/nixos-14.12.542.4c9ef9f7-x86_64-linux.raw.tar.gz
$ gcloud compute images describe nixos-14125424c9ef9f7-x86-64-linux
...
status: READY

You can now log out of your VM and delete the instance

1. $ logout
2. From the Google Developers Console select Compute - Compute Engine - VM instances check your instance and then press the "Delete" button at the top of the page.

Starting a NixOS image

Now that the image is made we can start as many NixOS VMs as we like.

(Strongly Recommended) Generate SSH host keys

Before beginning we should generate unique SSH host keys for our new VM so that when will be able to authenticate our SSH connection. The example below generates keys /dev/shm/ssh_host_ecdsa_key and /dev/shm/ssh_host_ecdsa_key.pub files.

$ ssh-keygen -N '' -C '' -t ecdsa -f /dev/shm/ssh_host_ecdsa_key
Generating public/private ecdsa key pair.
Your identification has been saved in /dev/shm/ssh_host_ecdsa_key.
Your public key has been saved in /dev/shm/ssh_host_ecdsa_key.pub.
The key fingerprint is:
92:2a:e9:28:1a:cd:43:71:31:36:f2:8e:6e:fa:13:c4 
The key's randomart image is:
+--[ECDSA  256]---+
|  . =            |
|   + +           |
| .. o            |
|  E=   .         |
| .o . o S        |
| =.. . .         |
|. O..            |
|.*.o             |
|*.o.             |
+-----------------+

Take note of your unique key fingerprint.

Create your VM

Now we are ready to create a NixOS VM.

1. From the Google Developers Console select Compute - Compute Engine - VM instances
2. Press "Create an Instance" or "New instance" button
3. Click on "Show advanced options"
4. (Strongly Recommended) Under METADATA, create a Key called ssh_host_ecdsa_key and copy the contents of your /dev/shm/ssh_host_ecdsa_key to the corresponding Value if you generated SSH host keys.
5. (Strongly Recommended) Under METADATA, create a Key called ssh_host_ecdsa_key_pub and copy the contents of your /dev/shm/ssh_host_ecdsa_key.pub to the corresponding Value if you generated SSH host keys.
6. Under SSH KEYS, copy the contents of your desktop computer's ~/.ssh/id_rsa.pub file into the box that says "Enter entire key data"
7. Under BOOT SOURCE, make sure "New disk form image" is selected
8. Under IMAGE, select your nixos image, e.g. "nixos-14125424c9ef9f7-x86-64-linux".
9. Set any additional options as you see fit.
10. Press the "Create" button and wait until the VM is ready

Connect to your VM

Notwithstanding the username associated with your SSH, you must log in as root to the NixOS machine. Also, you will need to ssh in from your computer's terminal. Below replace XXX.XXX.XXX.XXX with the EXTERNAL IP address listed for your VM instance. (Strongly Recommended) Verify the ECDSA key fingerprint is the same as the one you generated.

$ ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=ask root@XXX.XXX.XXX.XXX
The authenticity of host '130.211.149.218 (130.211.149.218)' can't be established.
ECDSA key fingerprint is 92:2a:e9:28:1a:cd:43:71:31:36:f2:8e:6e:fa:13:c4.
Are you sure you want to continue connecting (yes/no)? yes
Enter passphrase for key '.../.ssh/id_rsa': 

[root@instance-1:~]#

Once you are logged into your NixOS machine, I recommend that you create an user account for yourself with administrator privileges.

1. chmod u+w /etc/nixos/configuration.nix
2. nano -w /etc/nixos/configuration.nix
3. Add the follow to the configuration:

security.sudo.wheelNeedsPassword = false;
users.extraUsers.<your-username> = {
  createHome = true;
  home = "/home/<your-username>";
  description = "<your-name>";
  group = "users"; 
  extraGroups = [ "wheel" ];
  useDefaultShell = true;
  openssh.authorizedKeys.keys = [ "<contents of your ~/.ssh/id_rsa.pub>" ];
};

After you save this file run nixos-rebuild switch --upgrade. Once that is complete reboot and log back in with your user account. (Strongly Recommended) Again verify the ECDSA key fingerprint is the same as the one you generated. If you plan to keep this instance running for a long time you may removed the -o UserKnownHostsFile=/dev/null option to SSH to add the host to your ~/.ssh/known_hosts file, however be advised that IP addresses for GCE instances are often reused, so when you create and destroy instances you may end up with stale keys in your known_hosts file.

$ ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=ask <your-username>@XXX.XXX.XXX.XXX
The authenticity of host '130.211.149.218 (130.211.149.218)' can't be established.
ECDSA key fingerprint is 92:2a:e9:28:1a:cd:43:71:31:36:f2:8e:6e:fa:13:c4.
Are you sure you want to continue connecting (yes/no)? yes
Enter passphrase for key '.../.ssh/id_rsa': 

[<your-username>@instance-1:~]$

At this point you may want to snapshot this image and use this snapshot to make future VMs. You should also delete the /dev/shm/ssh_host_ecdsa_key and /dev/shm/ssh_host_ecdsa_key.pub files from your home machine if you generated them.

Bootstrapping a NixOS image from the build of your choice

You normally can use a preexisting NixOS image, such as the ones found in gs://nixos-images and gs://nixos-cloud-images. Sometimes the images there haven't been updated in a long time, or you need a newer one to work around a bug. If you need to make a new one, here's how. Perform these steps from any Linux machine.

You'll need setup the credentials for Google Cloud (via gsutil config or similar- if you have a GCE service account and have created an associated access key, the following is nice and quick: readlink -f ./relative/path/to/key | gsutil config -e, though it will prompt you for permission to lock down the file permissions on the key if they currently allow others to read the key- Note: the service account must be configured with a Role that allows it to write to your storage bucket), with the Cloud Storage and Compute Engine APIs enabled. You'll also need a Storage bucket. These steps will assume you've already made a bucket and it's named gs://example. Prepare a local copy of the nixpkgs repository in the state you want to build from. If you want to build a released version, this means checking out one of the release branches from the nixpkgs-channels repository. Make sure you haven't left any unwanted local changes in it. These examples assume you've checked it out at /home/example/nixpkgs-clean.

$ BUCKET_NAME=example /home/example/nixpkgs-clean/nixos/maintainers/scripts/gce/create-gce.sh

This will create an image and upload it to the bucket example