Writing tests for Nvim dev-test

Writing tests for Nvim

Nvim has a powerful yet simple test framework. It's approximately 7x better than whatever you use at work.

Each test starts a new Nvim process (10-30ms) which is discarded after the test finishes. You assert stuff using t.eq() and screen:expect() (which automatically waits as needed). That's pretty much it.

External consumers (plugins, …) can use the test framework dev-test-plugin. Improvements are tracked in: https://github.com/neovim/neovim/issues/34592

Test framework

Tests are broadly divided into unit tests (test/unit/), functional tests (test/functional/) and old tests (test/old/testdir/).

You can learn [Lua concepts 15 minutes](https://learnxinyminutes.com/docs/lua/), see also lua-guide. Use any existing test as a template to start writing new tests, or see dev-quickstart.

Tests are run by the /cmake/RunTests.cmake script using the repo-local Lua test harness in /test/harness.lua. For some failures, ./build/nvim.log (or $NVIM_LOG_FILE) may provide insight.

Depending on the presence of binaries (e.g., xclip) some tests will be skipped.

The Lua harness runs all selected spec files in one nvim -l process, then restores a baseline before each file. For each suite iteration, the file boundary baseline is captured after --helper is loaded, so helper-provided modules and defaults persist across files. Helper files are preload-only: they may require modules, set defaults, and register suite-end cleanup, but they do not define tests or hooks.

The harness restores _G, package.loaded, package.preload, arg, and the process environment with the same shallow file boundary model that busted uses.

Test Layout

Running tests dev-run-test

EXECUTING TESTS

To run all tests (except "old" tests):

make test

To run only _unit_ tests:

make unittest

To run only _functional_ tests:

make functionaltest

To run functional tests in parallel (used in CI):

cmake --build build --target functionaltest-parallel -j2
cmake --build build --target functionaltest-summary

LEGACY TESTS

To run all legacy Vim tests:

make oldtest

To run a _single_ legacy test file you can use either:

# Specify only the test file name, not the full path.
make oldtest TEST_FILE=test_syntax.vim

or:

make test/old/testdir/test_syntax.vim

DEBUGGING TESTS

Then, in another terminal:

gdb -ex 'target remote localhost:7777' build/bin/nvim

If $VALGRIND is also set it will pass --vgdb=yes to valgrind instead of starting gdbserver directly.

See nvim_argv in test/functional/testnvim.lua.

DEBUGGING LUA TEST CODE

Debugging Lua test code is a bit involved. Get your shopping list ready, you'll need to install and configure:

The setup roughly looks like this:

┌─────────────────────────┐
│ nvim used for debugging │◄────┐
└─────────────────────────┘     │
           │                    │
           ▼                    │
  ┌─────────────────┐           │
  │ local-lua-debug │           │
  └─────────────────┘           │
          │                     │
          ▼                     │
     ┌─────────┐                │
     │   nlua  │                │
     └─────────┘                │
          │                     │
          ▼                     │
     ┌───────────┐              │
     │ test-case │              │
     └───────────┘              │
          │                     │
          ▼                     │
  ┌────────────────────┐        │
  │ nvim test-instance │        │
  └────────────────────┘        │
    │   ┌─────┐                 │
    └──►│ osv │─────────────────┘
        └─────┘

With these installed you can use a configuration like this:

local dap = require("dap")


local function free_port()
  local tcp = vim.loop.new_tcp()
  assert(tcp)
  tcp:bind('127.0.0.1', 0)
  local port = tcp:getsockname().port
  tcp:shutdown()
  tcp:close()
  return port
end


local name = "nvim-test-case" -- arbitrary name
local config = {
  name = name,

  -- value of type must match the key used in `dap.adapters["local-lua"] = ...` from step 2)
  type = "local-lua",

  request = "launch",
  cwd = "${workspaceFolder}",
  program = {
    command = "nlua",
  },
  args = {
    "-l",
    "test/runner.lua",
    "--helper=test/functional/preload.lua",
    "--lpath=build/?.lua",
    "--lpath=?.lua",

    -- path to file to debug, could be replaced with a hardcoded string
    function()
      return vim.api.nvim_buf_get_name(0)
    end,

    -- You can filter to specific test-case by adding:
    -- '--filter="' .. test_case_name .. '"',
  },
  env = {
    OSV_PORT = free_port
  }
}

-- Whenever the config is used it needs to launch a second debug session that attaches to `osv`
-- This makes it possible to step into `exec_lua` code blocks
setmetatable(config, {

  __call = function(c)
    ---@param session dap.Session
    dap.listeners.after.event_initialized["nvim_debug"] = function(session)
      if session.config.name ~= name then
        return
      end
      dap.listeners.after.event_initialized["nvim_debug"] = nil
      vim.defer_fn(function()
        dap.run({
          name = "attach-osv",
          type = "nlua", -- value must match the `dap.adapters` definition key for osv
          request = "attach",
          port = session.config.env.OSV_PORT,
        })
      end, 500)
    end

    return c
  end,
})

You can either add this configuration to your dap.configurations.lua list as described in :help dap-configuration or create it dynamically in a user-command or function and call it directly via dap.run(config). The latter is useful if you use treesitter to find the test case around a cursor location with a query like the following and set the --filter property to it.

(function_call
  name: (identifier) @name (#any-of? @name "describe" "it")
  arguments: (arguments
    (string) @str
  )
)

Limitations:

Filtering tests dev-filter-test

FILTER BY NAME

Tests can be filtered by setting a pattern of test name to TEST_FILTER or TEST_FILTER_OUT.

it('foo api',function()
  ...
end)
it('bar api',function()
  ...
end)

To run only test with filter name:

TEST_FILTER='foo.*api' make functionaltest

To run all tests except ones matching a filter:

TEST_FILTER_OUT='foo.*api' make functionaltest

FILTER BY FILE

To run a _specific_ unit test:

TEST_FILE=test/unit/foo.lua make unittest

or

cmake -E env "TEST_FILE=test/unit/foo.lua" cmake --build build --target unittest

To run a _specific_ functional test:

TEST_FILE=test/functional/foo.lua make functionaltest

or

cmake -E env "TEST_FILE=test/functional/foo.lua" cmake --build build --target functionaltest

To _repeat_ a test:

TEST_ARGS="--repeat=100 --no-keep-going" TEST_FILE=test/functional/foo_spec.lua make functionaltest

or

cmake -E env "TEST_FILE=test/functional/foo_spec.lua" cmake -E env TEST_ARGS="--repeat=100 --no-keep-going" cmake --build build --target functionaltest

FILTER BY TAG

Tests can be "tagged" by adding # before a token in the test description.

it('#foo bar baz', function()
  ...
end)
it('#foo another test', function()
  ...
end)

To run only the tagged tests:

TEST_TAG=foo make functionaltest

NOTE:

Writing tests dev-write-test

GUIDELINES

WHERE TESTS GO

Tests in /test/unit and /test/functional are divided into groups by the semantic component they are testing.

Fixing tests dev-fix-test

FIXING HARNESS WARNINGS

> Nvim session T123 took 2000 milliseconds to exit > This indicates a likely problem with the test even if it passed!

This may indicate a leak, because Nvim waits on uv handles before exiting. Example: https://github.com/neovim/neovim/pull/35768

FIXING LINT FAILURES

make emmylua-check runs EmmyLua on the runtime Lua code, using .emmyrc.json. The checker is downloaded automatically. Enable more checks incrementally by removing codes from diagnostics.disable and fixing the resulting diagnostics.

If an EmmyLua warning must be ignored, specify the warning code. Example:

---@diagnostic disable-next-line: unused

https://github.com/EmmyLuaLs/emmylua-analyzer-rust

Ignore the smallest applicable scope (e.g. inside a function, not at the top of the file).

Testing plugins dev-test-plugin

Plugins (and CI) can use the test harness without cloning the Nvim repo:

make test downloads the harness modules to .deps/, from the Nvim git ref NVIM_TEST_VERSION (default "master"). Use a ref that matches the Nvim under test.

Configuration dev-test-config

(TODO: clean this up, too many variables and some of them are not used anymore.)

Test behaviour is affected by environment variables. Currently supported (Functional, Unit, Benchmarks) (when Defined; when set to _1_; when defined, treated as Integer; when defined, treated as String; when defined, treated as Number; !must be defined to function properly):