Talk: Nvidia

I have a few issues with the By making a FHS user env section

first of some characters get replaced by html entities which makes it annoying to copy the example and I'm not sure how to avoid that when editing the page, specifically <nixpkgs>. Strangely I can't recreate this behavior here.

Furthermore I changed the file like so:

cuda-fsh.nix

{ pkgs ? import &lt;nixpkgs&gt; {} }:

let fhs = pkgs.buildFHSUserEnv {
        name = "cuda-env";
        targetPkgs = pkgs: with pkgs;
               [ git
                 gitRepo
                 gnupg
                 autoconf
                 curl
                 procps
                 gnumake
                 gcc7
                 utillinux
                 m4
                 gperf
                 unzip
                 cudatoolkit
                 linuxPackages.nvidia_x11
                 libGLU
                 libGL
		 xorg.libXi xorg.libXmu freeglut
                 xorg.libXext xorg.libX11 xorg.libXv xorg.libXrandr zlib 
		 ncurses5
		 stdenv.cc
		 binutils
                ];
          multiPkgs = pkgs: with pkgs; [ zlib ];
          runScript = "bash";
          profile = ''
                  export CUDA_PATH=${pkgs.cudatoolkit}
                  export PATH=$CUDA_PATH:$PATH
                  # export LD_LIBRARY_PATH=${pkgs.linuxPackages.nvidia_x11}/lib
		  export EXTRA_LDFLAGS="-L/lib -L${pkgs.linuxPackages.nvidia_x11}/lib"
		  export EXTRA_CCFLAGS="-I/usr/include"
            '';
          };
in pkgs.stdenv.mkDerivation {
   name = "cuda-env-shell";
   nativeBuildInputs = [ fhs ];
   shellHook = "exec cuda-env";
}

adding gcc7 since it otherwise nvcc wouldn't work and adding

export PATH=$CUDA_PATH:$PATH

since otherwise cicc wouldn't be found

Also LIBGLU_combined has been removed since 20.03, see https://github.com/NixOS/nixpkgs/pull/73261#issue-339663290

moritz: libGLU_combined should be replaced with libGL and libGLU. I'm fairly new with all this, so I don't dare to edit any wikipages as of yet.

I couldn't make this work successfully until I unset CUDA_HOME: that was conflicting with CUDA_PATH and making CUDA devices undiscoverable. --Akiross (talk) 16:08, 11 February 2021 (UTC)

reverse PRIME with an integrated amdgpu and discrete nvidia card is working as of nvidia driver version 470 beta, with improved functionality in 495. it should work similarly for an intel integrated device.

configuration.nix

{ config, pkgs, ... }:
{
  # do not include "amdgpu" here or the nvidia driver will not get loaded correctly.
  services.xserver.videoDrivers = [ "nvidia" ];  
  
  hardware.nvidia = {
    # this will work on the 470 stable driver as well but I had issues getting the 
    # external monitors to be recognized unless I made sure the monitors were already 
    # on during the nvidia driver's initialization. 
    # see: https://github.com/NixOS/nixpkgs/pull/141685 for the 495 beta driver.
    package = config.boot.kernelPackages.nvidiaPackages.beta;
   
    # make sure power management is on so the card doesn't stay on when displays 
    # (and presumably power) aren't connected.
    powerManagement = {
        enabled = true;
        
        # note: this option doesn't currently do the right thing when you have a pre-Ampere card.
        # if you do, add "nvidia.NVreg_DynamicPowerManagement=0x02" to your kernelParams.
        # for Ampere and newer cards, this option is on by default.
        finegrained = true;
    };

    # ensure the kernel doesn't tear down the card/driver prior to X startup due to the card powering down.
    nvidiaPersistenced = true;
    
    # the following is required for amdgpu/nvidia pairings.
    modesetting.enable = true;
    prime = {
      offload.enable = true;

      # Bus ID of the AMD GPU. You can find it using lspci, either under 3D or VGA
      amdgpuBusId = "PCI:5:0:0";

      # Bus ID of the NVIDIA GPU. You can find it using lspci, either under 3D or VGA
      nvidiaBusId = "PCI:1:0:0";
    };
  };

  # now set up reverse PRIME by configuring the NVIDIA provider's outputs as a source for the 
  # amdgpu. you'll need to get these providers from `xrandr --listproviders` AFTER switching to the 
  # above config AND rebooting.
  services.xserver.displayManager.sessionCommands = ''
    ${pkgs.xorg.xrandr}/bin/xrandr --setprovideroutputsource NVIDIA-G0 "Unknown AMD Radeon GPU @ pci:0000:05:00.0"
  '';
}

you'll need to restart your display manager session one more time after setting up the xrandr provider output source. if you still don't see the external displays attached to the NVIDIA card, it probably means the providers weren't both initialized when the command was run. you'll need to move setting up the output provider source to a later point, such as after window manager start up.

this set up does carry one negative and it's that there's unavoidable high CPU usage by the X server while monitors are connected in this way. see the bug report here: https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-470/+bug/1944667. swapping to discrete mode (and disabling prime render offloading), with the NVIDIA card set up as the primary output does away with this high usage, at the cost of having the discrete card powered on even once the external displays are disconnected. hopefully this is resolved in future versions of the NVIDIA driver.