Testing

Tip

This document is a stub and is awaiting more detail.

Venator supports two test suites: the local test suite, and Complement. Currently, the local test suites should be preferred, as they are more tailored to Venator than complement is. Complement depends on a lot of functionality not currently implemented in Venator, so a lot of tests there are skipped. However, Complement provides a more comprehensive and agnostic framework that may catch compliance issues.

Local tests

To run local tests, you just need to run this command in the project root:

VENATOR_TEST_DB_URI='postgresql://venator:venator@localhost:5432/venator?sslmode=disable' GOEXPERIMENT=jsonv2 go test -count 1 ./...

You must make sure a database is available to be connected to at VENATOR_TEST_DB_URI; you can start one with ./dev/start-db.sh (which uses docker to spawn a disposable postgresql server).

Tests inside of the /tests directory are end-to-end tests. This means these tests may take a lot longer than the standard unit tests littered around the codebase. In this case, you must ensure 127.0.0.1:8008, 127.0.0.1:8080, 127.0.0.1:8443, and 127.0.0.1:8448 are all available addresses that are not already in use.

Complement

To run complement, you MUST have docker available. You can then just use the script ./dev/complement.sh. The complement script will then:

1. Download complement to $COMPLEMENT_ROOT (default: ./complement)
2. Attempt to locate a rootless docker socket, falling back to a rooted docker socket.
3. Builds the complement base OCI image ($COMPLEMENT_BASE_IMAGE_PRESENT, default: codeberg.org/timedout/venator:complement). Note that providing the environment variable will SKIP automatic building.
4. Runs the complement tests with a 10-minute timeout, outputting raw JSON results to ./complement.json, and “pretty-printing” results as they come in to stdout.
5. If gotestfmt is present in $PATH, interprets ./complement.json into ./complement.log.
6. Otherwise, runs a basic jq query to show test names and their associated FAIL/PASS label, outputting to ./complement.log.

There is a mandatory 5-second wait when you run the script to remind you that complement is not currently utilised for coverage.

Running with a different complement repository location

You can change where complement is run from by providing $COMPLEMENT_ROOT before complement.sh. For example, to use a local fork, you might do something like this:

COMPLEMENT_ROOT=$HOME/projects/complement ./dev/complement.sh

$COMPLEMENT_ROOT should be the root path, NOT the path including /tests.

If this path does not exist, the script will clone the latest main branch of complement to it.

Increasing log verbosity

Since complement runs use a pre-built configuration file (located at ./contrib/complement/config.yaml), it is not possible to override the configuration on the fly. Since Venator’s trace and even debug logs are very noisy, the minimum level during complement runs is set to info.

To increase the verbosity to trace, set PASS_VENATOR_COMPLEMENT_TRACING to 1:

PASS_VENATOR_COMPLEMENT_TRACING=1 ./dev/complement.sh

Running in complement mode without docker

If for some reason you want to run in “complement mode” without Docker, there are a few things you need to do:

1. $VENATOR_COMPLEMENT MUST be 1 (this is baked into the Dockerfile).
2. $SERVER_NAME MUST be set.
3. /venator/$SERVER_NAME.crt MUST exist (/ as in literal filesystem root).
4. /venator/$SERVER_NAME.key MUST exist (/ as in literal filesystem root).
5. /venator/config.yaml MUST exist (/ as in literal filesystem root).
   1. You will need to manually specify database.url in this case.

Caution

“Complement mode” is INSECURE, intentionally so. Do NOT run complement mode outside a test environment. Ideally, don’t let it be exposed to anything other than your lo interface.