VMware
From NixOS Wiki
Troubleshooting
Guest issues
This section covers issues you might have running NixOS as a guest VM in VMware Workstation or similar.
Cannot change resolution in KDE Plasma ISO
If you are unable to change resolution in a guest VM while booted into the NixOS Plasma graphical installer ISO...
This appears to be an issue where if KScreen is running as a service it will instantly revert to the default resolution [1].
Disabling the KScreen 2 Service under Background Services allows resolution settings to apply properly.
If you continue to have issues, running journalctl should print a system log with debug info which might help in resolving the issue.