Anatomy of the FreeBSD NVIDIA driver

A tour through the open source portions of NVIDIA for FreeBSD

FreeBSD has support for NVIDIA graphics cards in the form of a proprietary package in the ports tree. It actually works quite well for most use cases, but it's still missing a lot of very important bits. Two of the most egregious omissions are Vulkan and CUDA. I originally wanted to use Vulkan, but had to resort to using EGL/GL due to NVIDIA's lack of support.

My notes here are sort of a random collection of things I remember while porting nvidia-drm.ko.

Kernel Modules

• nvidia.ko

  • The majority of the driver implementation
  • Supported very well. Used by many.

• nvidia-modeset.ko

  • KMS (Kernel Modesetting) support. This is used to start x11, change vt's, and wake the screen up after suspending.
  • Also supported well (but is missing some functions nvidia-drm.ko needs)

• nvidia-uvm.ko

  • Unified Virtual Memory for CUDA
  • Not supported, but in theory shouldn't be too difficult.
    • CUDA libraries are missing, so there's no reason to port this

• nvidia-drm.ko

  • DRM (Direct Rendering Manager)
  • Ported by Y.T.
    • Used by nobody

In the FreeBSD driver's source tree, it looks like this:

nvidia-driver/
|
 -- src/
     |
      -- nvidia/
      -- nvidia-modeset/
      -- nvidia-drm/ (if you use my repository)

Abstractions

NVIDIA tries to keep as much code shared between operating systems as possible. (With the exception of DRM only in linux, and suspend/resume only working well in windows, and ...) Part of this abstraction is done using a nvKms structure containing implementation specific function pointers:

const struct NvKmsKapiFunctionsTable* const nvKms;

struct NvKmsKapiFunctionsTable {

 const char *versionString;

 /*!
  * Enumerate the available physical GPUs that
  * can be used with NVKMS.
  */
 NvU32 NVKMS_KAPI_CALL (*enumerateGpus)(nv_gpu_info_t *gpuInfo);

 /*!
  * Allocate an NVK device using which you can
  * query/allocate resources on GPU and do modeset.
  */
 struct NvKmsKapiDevice* NVKMS_KAPI_CALL (*allocateDevice)
 (
     const struct NvKmsKapiAllocateDeviceParams *params
 );

 ... etc.
}

Each operating system's driver implements its own version of these functions. For example, in the FreeBSD driver nvKms->allocateDevice will call nvidia_open_dev.

Installing From Source

Compiling and installing is very simple. NVIDIA clearly made a small attempt to make the driver ready for the ports tree. Let me be clear, it is in no way ready for the ports tree and you should not install from source. Compiling and installing is as easy as:

make install clean

The catch is there is no make uninstall, nor did I find any way to remove it from my system. This makes you have to uninstall by hand like the cavemen of old. If the library names changed between versions, such as versions > 400, you have to manually track down all the libraries and remove them. Seriously, just stick to the package even if it is out of date.