NixOS on ARM
ARM support for NixOS is a work-in-progress, but is progressing quickly.
The support varies depending on the architecture and the specific boards. The way the ARM integration is built into NixOS is by making generic builds the first-class citizens; as soon as there is upstream support for the board in the kernel and the bootloader, NixOS should work once updated to these versions. It is still possible, when needed, to build and use a customised bootloader and kernel for specific boards[reference needed]. At this moment in time (early 2018) only AArch64 is planned for full support upstream. Though, neither armv6l or armv7l are being ignored, fixes are worked on and approved as needed; what's missing is support and upstream builds being maintained in binary form. Additional resources (and some sd card images) can be found courtesy of illegalprime.
Supported devices
Upstream (NixOS) supported devices
NixOS has support for these boards using AArch64 architecture on the nixpkgs-unstable and nixpkgs-19.09 channel.
Manufacturer | Board | SoC | ISA | CPU | RAM | Storage |
---|---|---|---|---|---|---|
Raspberry Pi Foundation | Raspberry Pi 3 | Broadcom BCM2837 | AArch64 / ARMv7 | 4× Cortex-A53 @ 1.2 GHz | 1 GB | SD/microSD |
Community supported devices
The installation images by @dezgeg should work on the following devices; follow the links for more details, like installation instructions and device-specific notes.
Manufacturer | Board | SoC | ISA | CPU | RAM | Storage |
---|---|---|---|---|---|---|
ASUS | Tinker Board | Rockchip RK3288 | ARMv7 | 4× Cortex-A17 | 2 GB | microSD |
Banana Pi | Banana Pi | Allwinner A20 | ARMv7 | 2× Cortex-A7 | 1 GB | SD, SATA |
Banana Pi M64 | Banana Pi M64 | Allwinner A64 | ARMv8 | 4× Cortex-A53 | 2 GB | microSD, 8GB eMMc |
BeagleBoard.org | BeagleBone Black | TI AM335x | ARMv7 | 1× Cortex-A8 @ 1 GHz | 512 MB | 4 GB eMMC, microSD |
Firefly | AIO-3399C | Rockchip RK3399 | AArch64 | 2× Cortex-A72 @ 2.0 GHz, 4x Cortex-A53 @ 1.5 Ghz | 2/4 GB | 8/16 GB eMMC, microSD |
Hardkernel | ODROID-HC1 & ODROID-HC2 | Samsung Exynos 5422 | ARMv7 | 4 × Cortex-A15 @ 2GHz, 4 × Cortex-A7 @ 1.4GHz | 2 GB | microSD |
Hardkernel | ODROID-C2 | Amlogic S905 | AArch64 | 4 × Cortex-A53 @ 1.5GHz | 2 GB | eMMC, microSD |
Libre Computer | ROC-RK3399-PC | Rockchip RK3399 | AArch64 | 2× Cortex-A72 @ 2.0 GHz, 4x Cortex-A53 @ 1.5 Ghz | 4 GB | eMMC, microSD, NVMe |
Linksprite | pcDuino3 Nano | Allwinner A20 | ARMv7 | 2× Cortex-A7 @ 1 GHz | 1 GB | 4 GB NAND, microSD, SATA |
FriendlyElec | NanoPC-T4 | Rockchip RK3399 | AArch64 | 2× Cortex-A72 @ 2.0 GHz, 4x Cortex-A53 @ 1.5 Ghz | 4 GB | 16 GB eMMC, microSD, NVMe |
NVIDIA | Jetson TK1 | Tegra K1/T124 | ARMv7 | 4× Cortex-A15 @ 2.3 GHz | 2 GB | 16 GB eMMC, SD, SATA |
Orange Pi | Orange Pi One | Allwinner H3 | ARMv7 | 4× Cortex-A7 @ 1.2 GHz | 512 MB | microSD |
Orange Pi | Orange Pi PC | Allwinner H3 | ARMv7 | 4× Cortex-A7 @ 1.6 GHz | 1 GB | SD/microSD |
Orange Pi | Orange Pi Zero Plus2 (H5) | Allwinner H5 | AArch64 | 4× Cortex-A53 @ 1.2 GHz | 1 GB | SD/microSD + 8GB eMMC |
PINE64 | PINE A64-LTS | Allwinner R18 | AArch64 | 4× Cortex-A53 @ ? GHz | 2 GB | microSD & eMMC |
PINE64 | Pinebook | Allwinner A64 | AArch64 | 4× Cortex-A53 @ ? GHz | 2 GB | microSD & eMMC |
PINE64 | Pinebook Pro | Rockchip RK3399 | AArch64 | 2× Cortex-A72 @ 2.0 GHz, 4x Cortex-A53 @ 1.5 Ghz | 4 GB | microSD & eMMC |
PINE64 | ROCK64 | Rockchip RK3328 | AArch64 | 4x Cortex-A53 @ 1.5 GHz | 1/2/4 GB | microSD/eMMC |
PINE64 | ROCKPro64 | Rockchip RK3399 | AArch64 | 2× Cortex-A72 @ 2.0 GHz, 4x Cortex-A53 @ 1.5 Ghz | 2/4 GB | microSD/eMMC |
Raspberry Pi Foundation | Raspberry Pi | Broadcom BCM2835 | ARMv6 | 1× ARM1176 @ 700 MHz | 256 MB / 512 MB | SD/microSD |
Raspberry Pi Foundation | Raspberry Pi 2 | Broadcom BCM2836 | ARMv7 | 4× Cortex-A7 @ 900 MHz | 1 GB | SD/microSD |
Raspberry Pi Foundation | Raspberry Pi 3 | Broadcom BCM2837 | AArch64 / ARMv7 | 4× Cortex-A53 @ 1.2 GHz | 1 GB | SD/microSD |
Raspberry Pi Foundation | Raspberry Pi 4 | Broadcom BCM2711 | AArch64 / ARMv7 | 4× Cortex-A53 @ 1.5 GHz | 1-8 GB | microSD |
Toshiba | AC100 (mini laptop) | Tegra 2 250 (T20) | ARMv7 | 2× Cortex-A9 @ 1 GHz | 512 MB | 8–32 GB eMMC, SD |
Wandboard | Wandboard Solo/Dual/Quad | Freescale i.MX6 | ARMv7 | 1×/2×/4× Cortex-A9 @ 1000 MHz | 512 MB / 1 GB / 2 GB | microSD, SATA |
Special Devices
It is possible to emulate an ARM platform with QEMU.
Manufacturer | Board | SoC | ISA | CPU | RAM | Storage |
---|---|---|---|---|---|---|
QEMU | QEMU | — | ARMv7 | up to 8 | up to 2 GB | Anything QEMU supports |
Installation
Getting the installer
For aarch64 it is possible to download images from Hydra (20.09) and (unstable). On the page click on the latest successful build to get a download link under build products. If the image has the extension .zst
, it will need to be decompressed before writing to installation device. Use nix-shell -p zstd --run "unzstd <img-name>.img.zstm"
to decompress the image.
Installation steps
The installation images come in two flavors: sd-image-armv6l-linux.img
is built for the ARMv6 architecture and it comes with the Raspberry Pi kernel. sd-image-armv7l-linux.img is built for the ARMv7 architecture and comes with the mainline multiplatform ARMv7 kernel (multi_v7_defconfig). Make sure you download the correct image for your board!
The .img files can be directly written to a microSD/SD card (minimal recommended size: 4 GB) using dd. The SD card needs to be unmounted first. For more information, check NixOS and for mac
The base images are configured to boot up with a serial TTY ( RX/TX UART ) @ 115200 Baud. That way you not necessarily have to have a HDMI Display and keyboard.
Note: I (kalbasit) had to replace console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0
with console=ttyS1,115200n8
in /boot/extlinux/extlinux.conf
to make the serial console work on my Rasberry-Pi 3.
Continue with #NixOS installation & configuration.
Binary cache
AArch64
Thanks to @grahamc and Packet, the official NixOS Hydra builds a full set of binaries (available on https://cache.nixos.org) for the AArch64 architecture on the nixpkgs-unstable channel. Starting with the 18.03 release the official Hydra instance will also build the stable channel .
armv6l and armv7l
A binary cache, containing a subset of several channels, is hosted at https://app.cachix.org/cache/thefloweringash-armv7 .
Additional caches
- https://arm.cachix.org/ -
arm.cachix.org-1:5BZ2kjoL1q6nWhlnrbAl+G7ThY7+HaBRD9PZzqZkbnM=
(updated 2020-08-10: /nix/store/lxi1hq2p0fx5wr123q9yyfmh3ngqnrq5-nix-2.4pre20200721_ff314f1 nixpkgs: 96069f7d890b90cbf4e8b4b53e15b036210ac146) (2020-08-11: NixOS image build uploaded)
Build your own image
You can customize image by using the following snippet.
# save as sd-image.nix somewhere
{ ... }: {
imports = [
<nixpkgs/nixos/modules/installer/cd-dvd/sd-image-aarch64.nix>
];
# put your own configuration here, for example ssh keys:
users.extraUsers.root.openssh.authorizedKeys.keys = [
"ssh-ed25519 AAAAC3NzaC1lZDI1.... username@tld"
];
}
Then build with:
$ nix-build '<nixpkgs/nixos>' -A config.system.build.sdImage -I nixos-config=./sd-image.nix
Note that this requires a machine with aarch64. You can however also build it from your laptop using an aarch64 remote builder as described in Distributed build or ask for access on the community aarch64 builder.
Compiling through QEMU
It is also possible to compile for aarch64 on your non-aarch64 local machine, or a remote builder, by registering QEMU as a binfmt wrapper for the aarch64 architecture. This wrapper uses emulation and will therefore be slower than comparable native machines.
To enable the binfmt wrapper on NixOS, add the following to configuration.nix
{
boot.binfmt.emulatedSystems = [ "aarch64-linux" ];
}
If you are building on non-NixOS machine with QEMU binfmt wrapper configured, you will want to configure nix daemon to let it know that it can build for aarch64. Add the following line to /etc/nix/nix.conf
:
extra-platforms = aarch64-linux arm-linux
If you want to build just one specific package, use this:
nix-build '<nixpkgs/nixos>' -A pkgs.theRequiredPackage --argstr system aarch64-linux -I nixos-config=/path/to/target/machine/nixos/config/copy
(the last option should not be required on NixOS machines)
Installer image with custom uboot
nixos-aarch64-images provides a mechanism to modify the official NixOS installer with custom uboot bootloader required for different boards. This method does not require qemu or native arm, since the existing hydra image/pre-build uboot binaries are used.
Board-specific installation notes
Depending on the board, some additional preparation steps might be needed to make the SD card bootable on your device. All of the board-specific installation notes are now found on their respective pages.
Enable UART
If you try to use UART to log on NixOS, you might hang on the line "Starting kernel ...". To enable UART, you will need to add at the end of the line that contains loglevel4
in the file /extlinux/extlinux.conf
the text:
/extlinux/extlinux.conf
console=ttyAMA0,115200n8
/extlinux/extlinux.conf
console=ttyS0,115200n8
The actual device (ttyAMA0
, ttyS0
, ttyS1
) will depend on the hardware.
NixOS installation & configuration
The installation image is actually a MBR partition table plus two partitions; a FAT32 /boot and a ext4 root filesystem. The image is designed such that it's possible to directly reuse the SD image's partition layout and "install" NixOS on the very same SD card by simply replacing the default configuration.nix and running nixos-rebuild. Using this installation method is strongly recommended, though if you know exactly what you're doing and how U-Boot on your board works, you can use nixos-install as usual. To help with the SD card installation method, the boot scripts on the image automatically resize the rootfs partition to fit the SD card on the first boot.
Use this as a template:
/etc/nixos/configuration.nix
{ config, pkgs, lib, ... }:
{
# NixOS wants to enable GRUB by default
boot.loader.grub.enable = false;
# Enables the generation of /boot/extlinux/extlinux.conf
boot.loader.generic-extlinux-compatible.enable = true;
# !!! If your board is a Raspberry Pi 1, select this:
boot.kernelPackages = pkgs.linuxPackages_rpi;
# !!! If your board is a Raspberry Pi 3, select not latest (5.8 at the time)
# !!! as it is currently broken (see https://github.com/NixOS/nixpkgs/issues/97064)
boot.kernelPackages = pkgs.linuxPackages;
# !!! Otherwise (even if you have a Raspberry Pi 2), pick this:
boot.kernelPackages = pkgs.linuxPackages_latest;
# !!! This is only for ARMv6 / ARMv7. Don't enable this on AArch64, cache.nixos.org works there.
nix.binaryCaches = lib.mkForce [ "https://app.cachix.org/cache/thefloweringash-armv7" ];
nix.binaryCachePublicKeys = [ "thefloweringash-armv7.cachix.org-1:v+5yzBD2odFKeXbmC+OPWVqx4WVoIVO6UXgnSAWFtso=" ];
# !!! Needed for the virtual console to work on the RPi 3, as the default of 16M doesn't seem to be enough.
# If X.org behaves weirdly (I only saw the cursor) then try increasing this to 256M.
# On a Raspberry Pi 4 with 4 GB, you should either disable this parameter or increase to at least 64M if you want the USB ports to work.
boot.kernelParams = ["cma=32M"];
# File systems configuration for using the installer's partition layout
fileSystems = {
# Prior to 19.09, the boot partition was hosted on the smaller first partition
# Starting with 19.09, the /boot folder is on the main bigger partition.
# The following is to be used only with older images.
/*
"/boot" = {
device = "/dev/disk/by-label/NIXOS_BOOT";
fsType = "vfat";
};
*/
"/" = {
device = "/dev/disk/by-label/NIXOS_SD";
fsType = "ext4";
};
};
# !!! Adding a swap file is optional, but strongly recommended!
# swapDevices = [ { device = "/swapfile"; size = 1024; } ];
}
Note: the default configuration.nix will contain something like imports = [ <nixos/modules/installer/cd-dvd/sd-image-armv7l-multiplatform.nix> ];
do not include that in your final installation or you will experience interesting problems. It is only for building the installation image!
First rebuild on ARMv6 and ARMv7
To make the unsupported ARM experience slightly less painful, the config template adds nixos-arm.dezgeg.me
as a binary cache, which contains a small subset of packages on the unstable channel (though a caution for US users: the server hosting them is physically located in Finland). Note that the binary cache isn't enabled on the prebuilt images, so enable it via the command line when building for the first time:
nixos-rebuild switch --fast --option binary-caches http://nixos-arm.dezgeg.me/channel --option binary-cache-public-keys nixos-arm.dezgeg.me-1:xBaUKS3n17BZPKeyxL4JfbTqECsT+ysbDJz29kLFRW0=%
Resizing the boot partition
It is possible that you run out of disk space on the boot partition after some system upgrades. To resize the boot partition:
- (If not already done, boot once to trigger the initial partition resizing)
- Backup the files currently stored in the boot partition
- Repartition and make sure to delete and then recreate the boot partition (fat32, primary, label:
NIXOS_BOOT
) - Copy the files from the backup back to the new boot partition
Disable use of /boot partition
The /boot partition is quite small on the NixOS on ARM images. This means that few generations can be kept, and nixos-rebuild
can often fail due to lack of space.
Here are quick instructions to disable use of the (small) /boot partition. The disk used in the following example is /dev/mmcblk1 which may differ depending on use of SD card and the particular device.
# umount /boot
Comment out the fileSystems."/boot" entry from configuration.nix
# $EDITOR /etc/configuration.nix
Use fdisk or cfdisk to remove the bootable flag from the FAT32 partition, and set it for the ext4 partition
# fdisk -l /dev/mmcblk1 Disk /dev/mmcblk1: 58.2 GiB, 62537072640 bytes, 122142720 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x2178694e Device Boot Start End Sectors Size Id Type /dev/mmcblk1p1 * 16384 262143 245760 120M b W95 FAT32 /dev/mmcblk1p2 262144 122142567 121880424 58.1G 83 Linux # echo -e 'a\n1\na\n2\nw' | fdisk /dev/mmcblk1 Welcome to fdisk (util-linux 2.31.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): Partition number (1,2, default 2): The bootable flag on partition 1 is disabled now. Command (m for help): Partition number (1,2, default 2): The bootable flag on partition 2 is enabled now. Command (m for help): The partition table has been altered. Syncing disks. # fdisk -l /dev/mmcblk1 Disk /dev/mmcblk1: 58.2 GiB, 62537072640 bytes, 122142720 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x2178694e Device Boot Start End Sectors Size Id Type /dev/mmcblk1p1 16384 262143 245760 120M b W95 FAT32 /dev/mmcblk1p2 * 262144 122142567 121880424 58.1G 83 Linux
Then rebuild the system.
$ nixos-rebuild switch
You may want to verify the presence of the /boot/extlinux/extlinux.conf file.
Details about the boot process
On NixOS, all ARM boards are expected to use U-Boot as the firmware and bootloader. NixOS uses U-Boot's Generic Distro Configuration Concept as the mechanism to communicate boot information (such as path to kernel zImage, initrd, DTB, command line arguments). For a quick TL;DR about the generic distro configuration support: U-Boot is scripted to scan all attached storage devices & partitions and look for a file named /extlinux/extlinux.conf
or /boot/extlinux/extlinux.conf
(which will be generated by NixOS, just like /boot/grub/grub.cfg
is generated on PCs). The partition needs to have its "bootable" flag set.
U-Boot also provides an interactive shell and the generation selection menu (just like GRUB). However, support for input or display devices varies greatly, depending on the board. Details for what the boards support in relationship to the boot process are detailed in their respective pages.
Porting NixOS to new boards
The easiest way
Assuming upstream u-boot supports the board through a defconfig, it is possible possible to build u-boot using the cross-compiling architecture from an x86_64 host. Here's a sample use.
# Assuming you're in a recent nixpkgs checkout
$ nix-shell \
-I "nixpkgs=$PWD" \
-p 'let plat = pkgsCross.aarch64-multiplatform; in plat.buildUBoot{defconfig = "orangepi_zero_plus2_defconfig"; extraMeta.platforms = ["aarch64-linux"]; BL31 = "${plat.armTrustedFirmwareAllwinner}/bl31.bin"; filesToInstall = ["u-boot-sunxi-with-spl.bin"];}'
For armv7 and armv6 pkgsCross.arm-embedded
should work, this is available in the unstable channel (19.03 and following) by setting -I "nixpkgs=/path/to/new-nixpkgs-checkout
.
This should build whatever is needed for, and then build u-boot for the desired defconfig, then open a shell with the build in $buildInputs
. Do note that this particular invocation may need more changes than only the defconfig if built for other than allwinner boards.
Here's an example command, for allwinner boards, on how to write to an SD card.
$ sudo dd if=$buildInputs/u-boot-sunxi-with-spl.bin of=/dev/sdX bs=1024 seek=8
The easy way
(if you're lucky)
If your board is an ARMv7 board supported by multi_v7_defconfig and you have access to U-Boot on the board, getting sd-image-armv7l-linux.img
to boot is the easiest option:
- If you're lucky and your U-Boot build comes with the extlinux.conf support built in, the image boots out-of-the-box. This is the case for all (upstream) Allwinner and Tegra U-Boots, for instance.
- Otherwise, you can get the boot information (path to kernel zImage, initrd, DTB, command line arguments) by extracting
extlinux.conf
from the boot partition of the image, and then attempt to boot it via the U-Boot shell, or some other mechanism that your board's distro uses (e.g.uEnv.txt
).
Building u-boot from your NixOS PC
Assuming
- Your board is supported upstream by u-boot or there is a recent enough fork with
extlinux.conf
support. - You do not have nix setup on an ARM device
- Your nix isn't setup for cross-compilation
It is still possible to build u-boot using tools provided by NixOS.
In the following terminal session, replace orangepi_pc_defconfig
with the appropriate board from the configs folder of u-boot.
$ nix-shell -E 'with import <nixpkgs> {}; stdenv.mkDerivation { name = "arm-shell"; buildInputs = [git gnumake gcc gcc-arm-embedded dtc]; }' $ git clone git://git.denx.de/u-boot.git $ cd u-boot # We're checking out a version from before the use of `binman`. # The dtc package is 1.4.2, which does not include `pylibftd`. # Furthermore, I do not know how to package the library so it would be # available in the python interpreter, making binman happy. $ git checkout v2017.03 $ make -j4 CROSS_COMPILE=arm-none-eabi- orangepi_pc_defconfig $ make -j4 CROSS_COMPILE=arm-none-eabi-
The name of the final file will change depending on the board. For this specific build, and most Allwinner builds, the file will be named u-boot-sunxi-with-spl.bin
.
You can flash this file to boot device with
dd if=u-boot-sunxi-with-spl.bin of=/dev/sdX bs=1024 seek=8
Note: This mailing list contains a patch which may help some builds: https://lists.denx.de/pipermail/u-boot/2016-December/275664.html
The hard way
Alternatively/if all else fails, you can do it the hard way and bootstrap NixOS from an existing ARM Linux installation.
Contributing new boards to nixpkgs
- Add a new derivation for your board's U-Boot configuration, see for example ubootJetsonTK1 in
all-packages.nix
. - If your board's U-Boot configuration doesn't use the
extlinux.conf
format by default, create a patch to enable it. Some C hacking skills & U-Boot knowledge might be required. For some pointers, see this patch to enable it on the Versatile Express. - Make a pull request, also containing the board-specific instructions. Ping @dezgeg for review and for building & hosting the U-Boots at http://nixos-arm.dezgeg.me/installer.
Support
All ARM platforms are experimental. Only AArch64 platforms are currently being worked on for eventual support from NixOS.
There is a dedicated IRC Channel for the upstream effort on freenode, #nixos-aarch64.
Resources
Subpages
The following is a list of all sub-pages of the NixOS on ARM topic.
- Allwinner GPT Installation
- Apple Silicon Macs
- Banana Pi
- Banana Pi BPI-M5
- Banana Pi M64
- BeagleBone Black
- Clockworkpi A06 uConsole
- Firefly AIO-3399C
- Jetson TK1
- LS1046A
- Libre Computer AML-S905X-CC-V2
- Libre Computer ROC-RK3328-CC
- Libre Computer ROC-RK3399-PC
- NanoPC-T4
- NanoPi-R6C
- Novena
- ODROID-C2
- ODROID-HC1
- ODROID-HC4
- OLIMEX Teres-A64
- Orange Pi 5
- Orange Pi 5 Plus
- Orange Pi One
- Orange Pi PC
- Orange Pi Zero2 H616
- Orange Pi Zero Plus2 H5
- PINE64 Pinebook
- PINE64 Pinebook Pro
- PINE64 ROCK64
- PINE64 ROCKPro64
- PINE A64-LTS
- PcDuino3 Nano
- QEMU
- Radxa ROCK5 Model A
- Radxa ROCK5 Model B
- Raspberry Pi
- Raspberry Pi 3
- Raspberry Pi 4
- Raspberry Pi 5
- Scaleway C1
- Tinker Board
- Toshiba AC100
- UEFI
- Wandboard