View source for Future of Nix and NixOS

There are some hard-to-solve feature-requests for Nix and NixOS, which hopefully will be solved in the future.

== Space efficiency of the Nix store ==

* [https://github.com/NixOS/nixpkgs/issues/89380 Investigate deduplication to reduce storage and transfer nixpkgs#89380]

Possible solutions:

* Store the Nix store on a filesystem with reflink support: Btrfs, XFS
* Store the Nix store on a ZFS filesystem with block-level deduplication. This seems to give poor results [https://www.reddit.com/r/NixOS/comments/e3t3q5/zfs_dedup_on_nixstore_is_it_worth_it/].
* Use a deduplication tool (TODO how do they work? could they be integrated into Nix?)
** [https://bup.github.io/ bup]
** [https://borgbackup.readthedocs.io/en/stable/ borg]
** [https://attic-backup.org/ attic]

=== Network efficiency of system upgrades ===

Currently, fetching a new version of a package from a [[Binary Cache]] will fetch all files of the new version, without any deduplication, causing lots of network traffic. Ideally, fetching a new version would fetch only the difference between old version and new version.

* [https://github.com/NixOS/nixpkgs/issues/89380 Investigate deduplication to reduce storage and transfer nixpkgs#89380]

Possible solutions:

* [https://github.com/flokli/nix-casync flokli/nix-casync]
** https://discourse.nixos.org/t/nix-casync-a-more-efficient-way-to-store-and-substitute-nix-store-paths/16539
** https://alternativebit.fr/posts/nixos/future-of-nix-substitution/
* [https://github.com/folbricht/desync folbricht/desync] - alternative implementation of casync with higher performance

See also:

* [https://github.com/chromium/chromium/blob/main/courgette/description.md#adjust courgette] - lightweight incremetal updates for Google Chrome
** https://www.chromium.org/developers/design-documents/software-updates-courgette/ 

== Distributed Binary Cache ==

Currently, binary caches are centralized for better performance.

* [https://github.com/NixOS/nix/issues/859 Nix and IPFS nix#859]
* https://github.com/obsidiansystems/ipfs-nix-guide/blob/master/tutorial.md
* https://github.com/NixIPFS/nixipfs-scripts

== Memory usage of Nix evaluation ==

* [https://github.com/NixOS/nix/issues/7308 Evaluating nixpkgs takes a huge RAM nix#7308]

Possible solutions:

* [[Swap]]
* TODO remote evaluation?

== Intellisense for the Nix language ==

Intellisense for Nix is part of the bigger goal [https://discourse.nixos.org/t/making-nixos-more-user-friendly-whats-needed/25567 Make Nix more user friendly].

* [https://github.com/oxalica/nil oxalica/nil] - NIx Language server, an incremental analysis assistent for writing in Nix.
** [https://github.com/oxalica/nil/issues/9 Go to Definition for attributes of nixpkgs nil#9]
** [https://github.com/oxalica/nil/issues/14 Type system and attributes analysis nil#14]

Keywords: static analysis, dynamic analysis, incremental evaluation

See also:

* [[NixOS configuration editors]]
* [https://github.com/nix-community/nix-init nix-community/nix-init] - Generate Nix packages from URLs
* [https://github.com/NixOS/rfcs/pull/109 NixOS RFC 109: Nixpkgs Generated Code Policy]

=== Intellisense for embedded code fragments in Nix expressions ===

For example, the <code>buildPhase</code> of a <code>stdenv.mkDerivation</code> contains Bash code.

== Integration with large package sets ==

The evaluation of nixpkgs must work offline, with a static set of inputs. Other package managers send dynamic queries to servers for dependency resolution. To integrate such package managers into nixpkgs, we need an offline version of this dependency resolution, to get the file URLs and checksums of all dependencies.

* [https://discourse.nixos.org/t/status-of-lang2nix-approaches/14477 Status of lang2nix approaches]

=== Number of packages ===

Number of modules per language, based on [https://libraries.io/platforms libraries.io] and [http://www.modulecounts.com/ modulecounts.com]

* NPM: 4M packages
* Maven: 500K packages
* Go: 500K packages
* PyPi: 500K packages
* Cargo: 100K packages

Note: On average, every package has about 10 releases (10 different versions).

For reference, <code>nixpkgs</code> has about 70K packages per https://repology.org/

=== NodeJS ===

==== Machine-level store ====

Currently, NPM packages are installed by copying all files to a <code>node_modules</code> folder in the Nix store. Ideally, every NPM package would be stored only once in the Nix store, and symlinked into the <code>node_modules</code> folder, where it is used.

The problem is, nodejs would follow all symlinks, so it would escape the <code>node_modules</code> folder and break the resolution of sub-dependencies. Package managers like <code>pnpm</code> use hardlinks to solve this problem, but hardlinks cannot be stored in the Nix store.

Possible solutions:

* [https://github.com/milahu/nodejs-hide-symlinks nodejs-hide-symlinks], a LD_PRELOAD library, which stops the symlink-resolution before the Nix store, and makes symlinks appear as regular files to nodejs
* file-deduplication in the Nix store

=== Python ===

* [https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741 Nixpkgs’s current development workflow is not sustainable] - maintaining Python packages in nixpkgs is difficult, because a Python environment can have only one version of each package.
* [https://github.com/DavHau/mach-nix DavHau/mach-nix] - dependencies are resolved with an offline snapshot of the pypi package set.
* [https://github.com/nix-community/dream2nix nix-community/dream2nix] - online resolution of dependencies (TODO verify), so this is not compatible with nixpkgs.

=== Cargo ===

Cargo package (for the Rust language) are stored in the Github repository [https://github.com/rust-lang/crates.io-index rust-lang/crates.io-index], so its trivial to fetch a snapshot of all Cargo packages.