Improve Programmatic Usage

[saibotsivad]

What I did

I've got a bunch of tests that I want to run in a specific order, e.g. "create the user" -> "log in as user" -> "log out as user" (where each of those has a bunch of tests).

Each of those tests is in its own file, then I created a test runner so I can manage state easier.

The test files all export a default function that takes test (the suite) and assert (for convenience):

// tests/*.js
export default (test, assert) => {
	test('example test', () => {
		assert.is(true, true);
	});
}

The custom test runner sets up state, and manually controls the suite ordering:

// runner.js
import { exec, suite } from 'uvu'
import * as assert from 'uvu/assert'

const scenarios = [
	'tests/aaa.js',
	'tests/ccc.js',
	'tests/bbb.js'
]

const run = async () => {
	for (const scenario of scenarios) {
		const test = suite(scenario)
		const run = await import(scenario)
		run.default(test, assert)
		test.run()
	}
	return exec()
}

run()
	.then(() => { console.log('done') })
	.catch(error => { console.error('error', error) })

Then I execute the runner simply node ./runner.js

What I expect

When I execute the runner, normal uvu output should list each suite:

$ node ./runner.js

 tests/aaa.js  •   (1 / 1)
 tests/ccc.js  •   (1 / 1)
 tests/bbb.js  •   (1 / 1)

  Total:     3
  Passed:    3
  Skipped:   0
  Duration:  0.29ms

What actually happens

If I try reordering the tests, they are all run in order, except always the first one is skipped and prints this instead:

$ node ./runner.js

function () { [native code] }
 tests/ccc.js  •   (1 / 1)
 tests/bbb.js  •   (1 / 1)

  Total:     2
  Passed:    2
  Skipped:   0
  Duration:  0.29ms

Other information

$  node -v
v16.0.0

$  npm -v
7.12.1

$  hostinfo 
Mach kernel version:
	 Darwin Kernel Version 20.4.0: Fri Mar  5 01:14:02 PST 2021; root:xnu-7195.101.1~3/RELEASE_ARM64_T8101

[saibotsivad]

I have a really ugly workaround, where I insert a fake suite before calling the others:

const run = async () => {
	suite('hack').run()
	for (const scenario of scenarios) {

This still prints out the function line, but doesn't really matter of course. So it's ugly, but it works for now. 🤷

[lukeed]

Hey, so uvu isn't currently really meant for programmatic scheduling. It can done somewhat easily, but it relies on replicating some internals. This will be enhanced in a later version (API tbd)

First, you need to use uvu@next which has some fixes for the internals we're going to use. I think 0.4.x accidentally lost these ... I forget now.

Then here is your updated runner.js script:

// runner.js
import * as assert from 'uvu/assert'

const scenarios = [
  './tests/foo.js',
  './tests/bar.js',
]

const run = async () => {
  globalThis.UVU_DEFER = 1
  const uvu = await import('uvu')

  for (const scenario of scenarios) {
    const test = uvu.suite(scenario)

    // manually replicate uvu global state
    const count = globalThis.UVU_QUEUE.push([scenario])
    globalThis.UVU_INDEX = count - 1

    const run = await import(scenario)
    run.default(test, assert)
    test.run()
  }
  return uvu.exec()
}

run()
  .then(() => { console.log('done') })
  .catch(error => { console.error('error', error) })

This gives you the following output:

If you don't want to have the FILE > suites summary hierarchy (I assume you don't here), then this is your run function instead:

const run = async () => {
  globalThis.UVU_DEFER = 1
  globalThis.UVU_QUEUE = [[null]] // not a typo
  const uvu = await import('uvu')

  for (const scenario of scenarios) {
    const test = uvu.suite(scenario)
    const run = await import(scenario)
    run.default(test, assert)
    test.run()
  }
  return uvu.exec()
}

Producing this:

[saibotsivad]

Ah, that's great, thanks!

I actually did exactly what you had, except I set the globalThis after the import instead of before. 🤦

I think this will work well enough for me. It's kind of reaching into the internals more than I'd like of course, but it'll be fine until there's an API sorted out.

(Note: it's actually working on [email protected] so I don't need @next yet.)

[lukeed]

Cool! I'll have to go back & see what the actual differences in @next are atm.

Will leave this issue open for now, but track it a bit differently. Thanks

[saibotsivad]

Just a note to myself / whoever else stumbles across this: if you don't bail (i.e. uvu.exec(true)) then the way to get the exit status code is using process.exitCode afterward, e.g.:

const run = async () => {
  // ... any of the forms @lukeed wrote
  return uvu.exec()
}
// ... where you call `run`
await run()
// after the promise resolves
if (process.exitCode) {
  // one or more tests failed
} else {
  // all tests passed
}

[lukeed]

Just an update:

I'm working on this now, which is effectively a rewrite of uvu. It will allow you to call a new run() export which will provide a JSON report of the results. I don't have anything concrete yet (I started on this last night) but it will touch on multiple open issues: #14, #38, #52, #60, #80, #130

[mdbetancourt]

Could we have a way to feed uvu with tests? so that we can use vite to provide uvu with only the files that changed.

[jmcdo29]

Another option I figured out for running uvu programmatically is as follows:

import { parse } from 'uvu/parse';
import { run } from 'uvu/run';

const runUvu = async () => {
  const uvuCtx = await parse(join(process.cwd(), 'test'), /\.spec\.ts$/, {
    ignore: [/(index)/],
  });
  await run(uvuCtx.suites, { bail: false });
};

This is pretty much mimicking what the CLI does. Works out well as you can see here

[SokratisVidros]

Just an update:

I'm working on this now, which is effectively a rewrite of uvu. It will allow you to call a new run() export which will provide a JSON report of the results. I don't have anything concrete yet (I started on this last night) but it will touch on multiple open issues: #14, #38, #52, #60, #80, #130

@lukeed Any updates on this ☝️ ?

I use uvu to test Cloudflare workers with Miniflare and I need a way to tell if the suite runs successfully or not so that the worker can return a fetch response accordingly.

The setup is similar to https://github.com/cloudflare/miniflare-typescript-esbuild-jest.

Thanks.