Analysis updated 2026-05-18
Run this in CI to verify Playwright end to end tests actually fail when they should.
Catch newly added tests that lack a real, working assertion.
Track intentionally skipped tests with an owner and expiration date instead of letting them linger.
Check whether an assertion truly depends on the network data it claims to verify.
| vladyslavdmitriiev/playwright-mutation-gate | 0labs-in/vision-link | alfons-fhl/cursor-plan2api | |
|---|---|---|---|
| Stars | 4 | 4 | 4 |
| Language | TypeScript | TypeScript | TypeScript |
| Setup difficulty | easy | moderate | moderate |
| Complexity | 3/5 | 3/5 | 3/5 |
| Audience | developer | developer | developer |
Figures from each repo's GitHub metadata at analysis time.
Requires Node 20 or higher and an existing Playwright test setup already configured in the project.
This is a testing tool for teams that write end to end tests with Playwright, the browser automation framework. Its purpose is to catch a specific problem: tests that pass but do not actually check anything meaningful, sometimes called hollow assertions. The idea is simple. You mark the main assertion in each test with a comment, then run this tool as part of your CI pipeline. It takes that marked assertion, flips it to expect the opposite result, and reruns the test in a temporary copy. If the test still passes after the assertion was reversed, that means the original assertion was never really testing anything, and the tool fails the build to flag it. If the test fails once reversed, that is a good sign, since it shows the assertion was actually connected to real behavior. The tool reports each test as killed, meaning the assertion is doing real work, survived, meaning it is not, or cannot mutate, meaning the assertion could not be safely reversed. By default, any test without a marker also fails the check, though this can be turned off while a team is still adding markers to an older test suite. There is also a system for deliberately skipped tests, where a comment can record an owner and an expiration date so the exception does not linger forever unnoticed. A newer, optional mode goes further by rewriting the actual data a test's assertion depends on as it travels over the network, rather than just flipping the assertion itself. This checks whether an assertion is truly tied to the data it claims to verify, rather than a cached or coincidentally correct value. It ships as an npm package, requires Node version 20 or higher plus an existing Playwright setup, and includes a sample GitHub Actions workflow for running it in CI. Cost scales with the number of marked assertions, so the README suggests running it only on changed test files per pull request, or as a full nightly sweep instead of on every commit.
A CI tool that flips each test's key assertion to prove it can actually fail, catching tests that pass without checking anything real.
Mainly TypeScript. The stack also includes TypeScript, Playwright, Node.js.
Details are shown in a LICENSE file linked from the README, but the license type itself is not stated in the text provided.
Setup difficulty is rated easy, with roughly 30min to a first successful run.
Mainly developer.
This repo across BitVibe Labs
Verify against the repo before relying on details.