Testing
All of it, in the order CI runs it:
All of it, in the order CI runs it:
make ts # the type check, which CI runs first and gates the rest on
make test # everything this machine can run
make test-quick # unit and Node suites onlyThe type check
make ts builds the TypeScript packages and then checks everything that reads
them: the three host packages, and the demo apps in e2e/. The apps are the half
worth having. They call basalt's published types the way an app does, so a prop
renamed in src/ and not in the apps fails here and nowhere else.
In CI it is a job of its own, the TypeScript, on the pinned React Native, and
the three build jobs wait on it. Before that it ran inside each of them, which
meant one bad declaration failed nine shards after nine builds. See
docs/ci-performance.md.
It can disagree with your machine, and CI is the one that is right.
build_ts.sh resolves react-native to the first of three files it finds, and
which one that is changes what is declared: a checkout with generated
types_generated/ pulls in React Native's hand-written globals, and a release
checkout falls back to ReactNativeApi.d.ts, which declares none. To get CI's
answer here, move the generated types aside:
mv "$RN_DIR/packages/react-native/types_generated" /tmp/aside && make ts
mv /tmp/aside "$RN_DIR/packages/react-native/types_generated"Not npx tsc. There is an unrelated package on npm called tsc, and from a
directory with no local TypeScript npx fetches that one, which prints a banner
and exits 0. It reported a clean e2e/ while checking nothing at all.
The version can disagree too, and that is the sharper trap. The check is
only as good as the checkout it runs against, and the two differ on purpose:
scripts/react-native.pin is a release, because applications pin releases, and
development happens against main. So a type that exists only on main passes
here and fails the CI job that gates every other job.
Which happened on 2026-10-09, and took three tries to pin down because the
Flow types and the TypeScript ones disagree. main declares
backgroundSize in both and its C++ reads that raw prop, falling back to
experimental_backgroundSize. On v0.87.1 the Flow types declare only the
prefixed spelling -- and ReactNativeApi.d.ts, which is the file a release
checkout is actually type-checked against, declares neither. So both
spellings failed the pinned job, twenty seconds in, while both passed here.
The lesson in it is which file to read: not Libraries/StyleSheet/StyleSheetTypes.js,
which is Flow and is what a search finds first, but the .d.ts that
build_ts.sh resolves to -- and at the pinned tag rather than in the checkout.
e2e/views.tsx now writes those three through one cast, with the reason beside
it, and backlog/upstream.md carries the omission.
build_ts.sh now prints which React Native it is checking against whenever that
is not the pin. It is a warning rather than an error, because working against
main is the normal case -- the point is that the next red is obvious rather than
mysterious.
Running one thing
The whole suite is about a quarter of an hour per host, and most changes want
one part of it. Everything here narrows, and each narrows the same way: a
substring of a name, and --list to see the names.
The make targets above are these with the arguments filled in, and they pass
BASALT_BUILD_DIR through, so make test BUILD=build-087 tests the tree it
built. Narrowing is what the scripts are for, and is below.
scripts/test_all.sh --list # the step names
scripts/test_all.sh --only unit # steps whose name contains "unit"
scripts/test_all.sh --host gtk --only end # the other host, end-to-end only
python3 scripts/integration_test.py --list # scenario names
python3 scripts/integration_test.py -k window # one scenario
python3 scripts/integration_test.py -k window -k menu # or several
python3 scripts/integration_test.py --platform macos -k hover # and which host
./build/basalt_gtk_tests --list # test names
./build/basalt_gtk_tests limits # tests whose name contains "limits"
./build/basalt_gtk_tests window switch # or severalA filter that matches nothing says so and exits non-zero, rather than reporting
that everything passed. That is not hypothetical: the unit binaries used to
accept an argument and ignore it, so basalt_gtk_tests hover ran all 185 tests
and printed a pass, which looks exactly like a filter that worked.
It builds nothing but the demo bundle, and skips what is not built or not installed rather than failing, so on a machine with one host it runs that host's suites and says which it left out. CI's jobs and that script are two spellings of one list; a suite added to one belongs in the other.
The pieces, for when one of them is the thing being worked on. The Linux ones
need a display, because everything in them is a GTK program, and CI runs them on
Linux, which is where they matter: see .github/workflows/ci.yml.
./build/basalt_gtk_tests # unit, Linux
./build/basalt_appkit_tests # unit, on a Mac
./build/basalt_win32_tests.exe # unit, on Windows
make bundle # a production bundle to test against
make e2e # end to end, on whichever host is built
make compare-hosts # every host built, same app, diffed
make compare # every app in e2e/, through every host
node --test scripts/test_cli.js # run-linux, run-macos and run-windows
node scripts/check_includes.js # a header used and not included, and a
# spelling one toolchain lacks
node scripts/check_backlog.js # the backlog's counts against its entries
node scripts/check_support.js # the support page, from the scraped prop list
python3 scripts/scrape_props.py # what React Native declares, from the pinThe last two need nothing built, and run in both CI jobs. check_includes.js
exists because a header that reached <cstdint> through windows.h took the
Linux job down for twenty-three commits and could not be seen from the machine the
work was on; it reads text, so it cannot fail on one platform and pass on
another.
It does two things now, for the same reason. The second is a short list of
spellings that one of the three toolchains does not have, M_PI first among
them: MSVC defines the POSIX math constants only behind _USE_MATH_DEFINES,
which core/cmake/ReactNativeCore.cmake sets for React Native's targets and
deliberately not for this project's, so a shared header using it compiled on the
two hosts its author could build and took the Windows job down. A file that
mentions _WIN32 is skipped, having already thought about the question.
Those spellings are all unused in the tree on purpose, which means nothing here
would notice if the rules stopped matching. So the guard has its own tests,
node --test scripts/test_check_includes.js, which feed text through the rules
directly and run before it in both places.
check_support.js is the third, and the only one that reads React Native.
website/docs/support.mdx has one row per attribute, and the attributes are not
this repository's to list: scripts/scrape_props.py --from-pin reads
BaseViewProps, AccessibilityProps and TextAttributes out of the pinned
release and writes docs/react-native-props.json, and the statuses live beside
it keyed by Struct.prop. So a prop added upstream arrives as a row with no
status and fails the check until somebody says what this platform does with it,
which a hand-written table cannot do: a missing row looks exactly like a row
nobody filled in.
It also checks the statuses against the code -- a host claiming a prop has to
read it, and a host disclaiming one has to not -- which found two wrong cells the
day it was written, one of them accessibilityValue claiming Windows support
that was never implemented. The scraper needs the network and CI does not run it;
what CI runs is the renderer's --check against the committed inventory.
check_backlog.js is the same shape of guard pointed at prose. docs/BACKLOG.md
is a table of areas and how much is open in each, every file repeats its own
count in an **Open (n):** header, and both are maintained by hand in a
repository where a commit routinely strikes one entry and adds another. Counting
them for the first time on 2026-10-08 found accessibility.md claiming two with
four open, wrong for an unknown number of commits, and the commit that found it
had nearly shipped a wrong count of its own.
The rule it applies is the one these files are written to: an entry is closed when its own text is struck, and a struck note inside an entry leaves the entry open. It also has its own tests, for the reason above and because the shapes that matter -- the wrapped entry, the struck note, the prose file with no list -- are easier to feed as text than to find.
integration_test.py picks whichever host it finds and runs the same scenarios
against it, because they are about React and Fabric rather than about a
toolkit, --platform overrides the guess. Every scenario runs on every
platform; the packager the two development-mode ones need is started through
scripts/metro.js, in Node, because a shell script is not a way to start one
on Windows.
Read the first line of the output before believing a pass. It names the host
that actually ran, and the guess is first-found rather than this machine's own:
on a Mac with both hosts built, a bare run says on linux and exercises GTK.
Chasing a macOS-only failure that way produces a green run that proves nothing
about macOS, which has happened. Pass --platform macos when the host is the
point.
And --platform linux on a Mac is not Linux. It is the GTK host built
against this machine's C library, so anything whose behaviour comes from libc
rather than from GTK behaves as macOS, not as glibc. A green GTK run here is a
statement about the GTK code and not about the platform under it.
That is not a hypothetical. backtrace_symbols_fd, which the crash handler uses,
formats a frame differently in each of the three, and only one of the three can
be seen from here:
macOS 0 basalt_appkit 0x0000000104e8c040 _ZN6basalt... + 156
glibc build/basalt_gtk(+0x875a5c)[0x55cef4472a5c]
Windows 0 00007FF621EE0094A leading index on two and none on glibc; 0x on two and absent on the third.
A scenario matching the shape of the one visible here passed locally on both
hosts and failed on CI twice, once per platform, against a handler that was
printing a perfectly good stack each time. It counts the lines between the marker
and the blank line now, which is the one thing all three agree on.
On a headless machine, run either under a virtual display:
xvfb-run -a ./build/basalt_gtk_tests
xvfb-run -a scripts/integration_test.pybasalt_gtk_tests exits 77 rather than failing when it cannot reach a display, which
is the convention for "skipped".
Screenshotting a GTK window on macOS
screencapture -l <window-id> wants a CGWindow id, which nothing in the shell
exposes. A five-line Swift program over CGWindowListCopyWindowInfo, filtered
by kCGWindowOwnerPID, produces one. Compile it once with swiftc: swift <file> interprets slowly enough to miss the window's first frame.
basalt_appkit_tests needs neither a display nor a window: nothing in it is presented.
It is built only on a Mac, and CMake omits the target everywhere else.
basalt_win32_tests needs neither either, and unlike the other two it makes
real pictures while not needing them: Direct2D renders into a WIC bitmap with no
window, no device and no display connection. That is why the rendering
assertions this project has wanted since GTK live there and nowhere else. Built
only on Windows; CMake omits the target elsewhere.
| File | What it pins down |
|---|---|
native/tests/test_win32_view.cpp | The view layer as a tree. Tags and frames, insert indices, that a Remove detaches without destroying, that an Insert reparents, that destroying a view leaves nothing pointing at it, that zIndex does not touch the child list, and that describeTree is byte-for-byte what the other two hosts print. |
native/tests/test_win32_paint.cpp | What reached the pixels. Placement, opacity as a subtree layer, clipping under overflow, a quarter turn about the centre and in the right direction, zIndex paint order, the scroll offset, and the corner radius. Each is something a tree dump cannot show. |
native/tests/test_win32_hittest.cpp | Picking. Depth, sibling order, zIndex, misses, scroll offsets, hidden views, and that a press follows a transform, which this platform has to invert itself, having no toolkit picking to inherit. |
native/tests/test_win32_text.cpp | DirectWrite measurement. Wrapping, numberOfLines, that a bigger font measures bigger, that alignment does not change the measured size, and two regression tests carried over from the other hosts: that font sizes are absolute rather than points, and that the default ellipsize mode does not collapse a wrapping paragraph. Plus one they cannot make: that the glyphs actually move when the alignment does. |
native/tests/test_win32_image.cpp | The four resize modes, against a real bitmap and real pixels rather than against the destination rect: cover that scales correctly and forgets to clip looks right in a rect and wrong on a screen. Also the WIC decode, round-tripped through the snapshot encoder. |
native/tests/test_win32_accessibility.cpp | The UI Automation mapping, asked the way a screen reader asks it: build the provider and read property ids. Roles, the group fallback, that a plain <View> stays out of the tree, labels and hints, the tri-state states, and that a role can change after mount, which GTK cannot do, because a GtkAccessible role is construct-only. |
native/tests/test_win32_mounting.cpp | The mutation walk, through the real Win32MountingManager. Deliberately the same nine questions asked of GTK and AppKit, in the same order with the same tags, so a divergence in the shared walk fails here rather than in an app. Built only when the build was pointed at a React Native. |
native/tests/test_win32_imageloader.cpp | The image loader. Decoding a file URI, the cache answering without going back to disk, the three ways a load can fail, and that a decoded image survives the loader that produced it. Built only when the build was pointed at a React Native, because the fetch is core/ImageBytes.cpp. |
Watching an event go past
native/tests/EventRecorder.h is how a unit test sees that an event fired.
Neither suite could: the managers ask eventEmitterForTag for an emitter and
the hand-built shadow views carry none, so what the text-input tests pinned
was, in their own words, "that the round trip did not throw".
A stub emitter cannot fix that, TextInputEventEmitter contains the word
virtual zero times, so the recorder builds the real thing and listens
underneath it. EventDispatcher takes a std::function<bool(const RawEvent &)> and consults it before the logger and before the queue; returning true
stops the default dispatch, so no RuntimeScheduler ever has to run.
What it sees is the event's type and its position in the order. Payload
values need a jsi::Runtime to read, so what a value ends up as stays the
end-to-end suite's question.
build/basalt_gtk_tests: the unit suite
Everything reachable without a JavaScript runtime. No React, no Metro, no
Hermes: mutations are hand-built, the way mount_harness_gtk builds them.
| File | What it pins down |
|---|---|
native/tests/test_mounting.cpp | The mutation walk. Create/Insert/Remove/Delete/Update ordering, insert indices, nesting, that Remove detaches without destroying, that Delete unregisters, and that a stray Insert for a deleted tag is ignored rather than fatal. |
native/tests/test_view.cpp | The widget layer. Frames, child order, that a scroll offset moves children, that measure reports zero so GTK never competes with Yoga, and that dispose unparents children. |
native/tests/test_text.cpp | Pango measurement. Wrapping, numberOfLines, that a bigger font measures bigger, and two regression tests: that font sizes are absolute rather than points, and that the default ellipsize mode does not collapse a wrapping paragraph to one line. |
native/tests/test_hittest.cpp | Hit testing. Depth, sibling order, misses, that it follows a scroll offset, and that hitSlop grows a target without moving it. |
native/tests/test_gtk_paint.cpp | What reached the pixels, through tests/GtkPixels.h: placement, opacity as a subtree layer, clipping under overflow, zIndex paint order, the scroll offset, corner radii, and a border inside the box and over the children. The same eleven questions test_win32_paint.cpp asks of Direct2D, in the same order. Each is something a tree dump cannot show. |
native/tests/test_image.cpp | The image loader. Decoding, the cache answering synchronously, and the three ways a load can fail. |
native/tests/test_textinput.cpp | The controlled-value loop. That applying a prop is not reported back as typing, that the caret survives a prop arriving mid-word, that a command carrying a stale eventCount is dropped, and that the GtkText peer is allocated inside the content inset. Props are built through React Native's own RawProps parser, because the fields that matter are const and only reachable that way. Two tests watch the events rather than the widget, through native/tests/EventRecorder.h. |
build/basalt_appkit_tests: the macOS suite
Seventeen files, no window and no toolkit initialisation. The binary prints its own total, which was 824 on 2026-10-14; the lead here said "two files, sixteen tests" for a long time after it was neither, which is what a count kept by hand in prose does.
Four of them carry an argument about how they test, and those are described after this table. These are the rest:
| File | What it pins down |
|---|---|
native/tests/test_appkit_text.mm | The Core Text layer, against RnTextLayout directly: whether a paragraph measures and truncates the way React Native means it to. |
native/tests/test_appkit_textinput.mm | The controlled-value loop, the same questions test_textinput.cpp asks of GTK. |
native/tests/test_appkit_borders.mm | What a border is drawn as, against a real bitmap: four filled edges or one dashed stroke. A view that remembered dashed and painted solid would pass any test that asked it what it holds. |
native/tests/test_appkit_image.mm | How an <Image> fills its frame. The four resize modes are what two platforms are most likely to disagree about while both looking plausible. |
native/tests/test_appkit_scrollview.mm | What a scroll view does that needs no display: props arriving as mutations, and the offset that results. |
native/tests/test_appkit_scrollbar.mm | The overlay scrollbar, which on this host has to be drawn over a content view that covers it. |
native/tests/test_appkit_selection.mm | Selecting a paragraph with a pointer: the same three questions the GTK suite asks, so that the two hosts agree. |
native/tests/test_appkit_key_events.mm | An NSEvent as a key combination, which is the half of key handling that is this host's alone and where the silent failures live. |
native/tests/test_appkit_droptarget.mm | Which view a drag is offered to. Added because a change that stopped reading the nativeID marker altogether left the whole suite green. |
native/tests/test_appkit_titlebar.mm | The title bar's metrics, which are the whole contract src/TitleBar.tsx has with the host. |
native/tests/test_appkit_titlebar_drag.mm | Which parts of an app-drawn header drag the window: the same four questions test_titlebar_drag.cpp asks of GTK, in the same order, because the rule in core/TitleBarRegions.h is shared and the answers have to match. |
native/tests/test_appkit_fonts.mm | A font loaded while the app is running, which is the seam expo-font arrives through, and whether the name the app chose keeps working. |
native/tests/test_appkit_services.mm | The clipboard and URL halves of the platform-services seam. |
native/tests/test_appkit_skia.mm | That Skia renders rather than merely links; see the Skia section below. |
native/tests/test_appkit_view.mm is against RnAppKitView alone, with no Fabric and no
React Native in it. It pins the tag and frame, the child order including
insertion into the middle, that removing a view that is not yours is a no-op,
that background colour, opacity, clipping and corner radius reach the CALayer,
that the background is in sRGB rather than the display's space, and that
describeTree emits the same text the GTK side does.
native/tests/test_appkit_accessibility.mm asserts the AppKit role, not the
React Native name: the name is what describeTree reports and what the
cross-platform diff compares, and whether the mapping onto NSAccessibility
actually happened is a platform question. The GTK suite's
test_accessibility.cpp asserts the same thing about GtkAccessibleRole.
native/tests/test_appkit_input.mm is hit testing and the touch state machine.
It covers what test_win32_hittest.cpp does, including that a press follows a
transform: a translation, a quarter turn, and a scale: 0 that has no
inverse and is skipped. Hit testing is a pure function of the view tree, so it
needs no mouse, no window server, and no permission to synthesise an event, which matters, because a
real click on macOS means CGEvent and accessibility permission an automated
run does not have. Unlike the GTK equivalent none of it needs a window:
gtk_widget_pick skips unmapped widgets, so those tests show a window and pump
the frame clock, and RnAppKitHitTest walks frames and answers straight away.
native/tests/test_appkit_mounting.mm is against AppKitMountingManager, and is
deliberately the same questions native/tests/test_mounting.cpp asks of GTK, in
the same order, with the same tags and frames. Since both managers now share
their mutation walk, most of what these test is whether that sharing still holds.
Built only when the build was pointed at a React Native; without one the view
tests still run.
The one worth naming is mac_view_is_flipped_so_frames_are_top_left. AppKit's
origin is bottom-left and every frame Fabric produces is top-left, so a child at
y=32 in a 420-tall parent belongs 32 from the top; unflipped it would sit at 228
and the layout would look plausible and be wrong.
native/appkit/mount_harness_appkit.mm is the mounting equivalent: the same two
transactions gtk/mount_harness_gtk.cpp runs, driven through the real mounting
manager with no JavaScript anywhere. BASALT_SNAPSHOT_DIR=/tmp/out ./build/mount_harness_appkit writes mount-1.png and mount-2.png and prints the
tree after each; with no directory set it opens a window and applies the second
transaction two seconds in, the way the GTK harness does.
scripts/compare_all.sh runs every app in e2e/ through every host that is
built and prints where the platforms stand. Windows against Linux in WSL:
hosts: windows linux (linux in Ubuntu-24.04)
views identical, frames included
press identical, frames included
scroll identical, frames included
image identical, frames included
text identical, frames ignored
a11y identical, frames ignored
input identical, frames ignored
appearance identical, frames included
blob identical, frames included
modules identical, frames included
probe identical, frames included
index identical, frames ignored
the 2 hosts agree on all 12 appsAn app that only passes with frames ignored is one whose layout depends on text measurement. Everything else (tree shape, strings, colours, roles, flags) is still compared there.
scripts/compare_hosts.sh is the single-app version, and the one that cannot
run in CI. It runs e2e/views.tsx (a real React app made only of <View>) through every host that is built, and checks two things that fail differently:
that the view trees match, and that each host's log reports the Platform.OS
its bundle was built for. A bundle built for the wrong platform can render
perfectly and be wrong about everything Platform.OS guards.
It takes an app name rather than a bundle path per platform, and finds
build/<app>.<platform>.jsbundle.js for each host it is going to run. Three
paths on a command line was one too many.
It needs at least two hosts, so it runs on a developer's machine rather than on CI, where each host only exists on its own side. On a Mac with GTK installed, build the bundles first:
scripts/bundle.sh --platform linux --entry views --out views.linux.jsbundle
scripts/bundle.sh --platform macos --entry views --out views.macos.jsbundle
scripts/compare_hosts.shOn Windows the second host is the GTK one, running inside WSL. Ubuntu 24.04, because the GTK host needs GTK 4.10 and 22.04 ships 4.6, installed beside any distro already there:
wsl --install -d Ubuntu-24.04 --no-launch
wsl -d Ubuntu-24.04 -u root -e bash scripts/wsl_setup.sh
BASALT_COMPARE_WSL=Ubuntu-24.04 scripts/compare_all.shwsl_setup.sh installs what the Linux CI job installs, clones this checkout
onto the distro's own filesystem; building across /mnt/c is too slow to be
worth it, and builds the host there the way CI does. So it compares the
committed state: rerun it after committing. compare_all.sh builds any
missing Linux bundles inside the distro, and runs the host under Xvfb rather
than WSLg, because Xvfb is what CI uses.
It also registers something to open https with. Linking.canOpenURL asks
GIO whether anything handles the scheme, a bare distro has nothing, and
e2e/modules.tsx then fails a check on Linux that a GitHub runner, which has a
browser, passes, which is exactly what the first comparison through WSL
reported as a difference.
BASALT_COMPARE_TAP="x,y;x,y" forwards taps to every host; they read the same
variable, so the input path is compared too, not just the initial render.
e2e/press.tsx is the app for that, and needs a longer quit than the default
since the first tap is at 1500ms:
BASALT_COMPARE_TAP="400,100;400,100;400,100" BASALT_COMPARE_QUIT_AFTER_MS=5000 \
scripts/compare_hosts.sh press BasaltPresse2e/scroll.tsx scrolls itself through scrollTo on a ref, which is how both
hosts can be driven identically from one file: a wheel has to be injected per
platform and a command does not, and is the only test of the command path:
scripts/bundle.sh --platform linux --entry scroll --out scroll.linux.jsbundle
scripts/bundle.sh --platform macos --entry scroll --out scroll.macos.jsbundle
BASALT_COMPARE_QUIT_AFTER_MS=3000 scripts/compare_hosts.sh scroll BasaltScrollFor e2e/text.tsx and e2e/image.tsx, frames have to be ignored:
scripts/bundle.sh --platform linux --entry text --out text.linux.jsbundle
scripts/bundle.sh --platform macos --entry text --out text.macos.jsbundle
BASALT_COMPARE_IGNORE_FRAMES=1 scripts/compare_hosts.sh text BasaltTextPango over the system sans and Core Text over San Francisco are different shapers over different fonts, so the same paragraph is a few points taller on one than the other and every frame below it shifts. Demanding equality there would mean the check could never be turned on for text at all. What is still compared is everything that must match: the tree shape, the strings, the colours, the clip and opacity flags.
As of phase 23 that comparison differs by exactly one thing, GTK emits
role=label on a paragraph and macOS emits nothing, because macOS has no
accessibility yet.
native/appkit/demo_layout_appkit.mm is the visual half of the view layer. It carries the same boxes at
the same coordinates as the GTK demo_layout_gtk, so the two can be compared
directly, and BASALT_SNAPSHOT=out.png ./build/demo_layout_appkit renders them
offscreen to a PNG. BASALT_DUMP_TREE=1 prints the tree instead: useful, but
it would print the same thing whether or not the flip works, which is what the
snapshot is for.
The harness
native/tests/TestHarness.h, about sixty lines, no dependencies, shared by all
three. TestHarness.cpp is the runner and knows about no toolkit; each suite
brings its own main: tests/main_gtk.cpp initialises GTK, and the macOS one
sits at the bottom of test_appkit_view.mm. Its
assertions are EXPECT, EXPECT_EQ and EXPECT_NEAR rather than CHECK*
because glog (which React Native pulls in almost everywhere) already defines
CHECK and CHECK_EQ. Those win the preprocessor silently, and a failing
assertion then aborts the process instead of being reported, taking every later
test with it. That happened; hence the naming.
Hit tests put their widgets in a real window. gtk_widget_pick skips widgets
that are not mapped, and a widget is only mapped inside a shown window.
Allocation then happens in the frame clock's layout phase, so the tests build
the tree first, show the window, and spin the main loop until a frame has
actually gone through. Draining the context without waiting is not enough: with
no frame due it returns immediately and the children are still unallocated.
scripts/integration_test.py: the end-to-end suite
Runs the real host against the real Metro bundle, injects taps, and asserts on the widget tree the host writes on the way out.
This is what BASALT_DUMP_TREE exists for. The path this project exists to
provide (JavaScript, React, Fabric, the mounting manager, GTK) can only be
exercised by running the whole thing, and before the dump the only way to check
the result was to look at a screenshot.
What the scenarios are is not listed here, and deliberately: SCENARIOS at the
bottom of scripts/integration_test.py is the list, and every one of them has a
docstring saying what it asserts and why that is the thing worth asserting. A
copy in this file was five entries long while the suite had twenty, which is
what a hand-maintained list of something the code already states turns into.
What is worth writing down is what is not in the code.
BASALT_SNAPSHOT=<path> writes the window as a PNG, on all three hosts, on
the way out and beside the tree dump. Windows had it from the start, since
Direct2D renders into a WIC bitmap with no window and no display; GTK and
AppKit grew it on 2026-10-10, each through the renderer its own suite already
uses -- gsk_renderer_render_texture with a NULL surface, and
RnAppKitWriteSnapshot -- so none of the three needs a display server or a
screenshot permission.
BASALT_SNAPSHOT_SCALE=2 doubles the Windows render, and is for pictures
rather than for tests. Win32Snapshot pins the render target to 96 dpi so a
snapshot is the size the frames say whatever the machine is set to, which is
what every pixel assertion depends on; the scale multiplies both the bitmap
and the dpi, so the drawing is identical and the output is twice as many
pixels. Unset, or outside [1, 4], it is 1 and nothing changes. CI uses it for
the window in the README, because macOS and Linux are photographed at 2x and
a 96 dpi Windows shot beside them is visibly the soft one.
BASALT_WINDOW_CAPTURE=<path> photographs the live window on Windows,
through PrintWindow with PW_RENDERFULLCONTENT, which asks DWM for what it
composited. That is the picture with the things the offscreen pass cannot
see in it: a <TextInput>, whose peer is a child window; a <Canvas>, which
holds the last frame delivered to the real window; and the caption buttons,
which DWM draws over the app's own header.
Use the render for a test and the photograph for a picture somebody will
look at. The render needs no window, is the size the frames say and scales;
the photograph needs a real window and is the size that window actually is,
so BASALT_SNAPSHOT_SCALE does not apply to it. Both can be asked for in one
run, which is what CI does.
It exists because a tree dump says a view has a colour, a frame and a transform, and cannot say that any of it reached the screen. The two text bugs found on 2026-10-10 were both of that kind, and both were found by looking at a picture of React Native's own LogBox rather than by any assertion.
What a picture cannot show, which is worth knowing before one is read as a
bug. On macOS the snapshot is [root.layer renderInContext:], and Core
Animation composites nothing in that path: a boxShadow, a mixBlendMode and
an <ActivityIndicator color> -- which is a CIFalseColor filter on the
indicator's layer -- are all absent from the file while being perfectly visible
on screen. GTK and Windows render their own node trees and have no such gap.
The tree dump is what those three are asserted through instead.
The suite can read the picture back: Snapshot in scripts/integration_test.py
decodes a PNG with the standard library alone -- no Pillow and no numpy on three
CI images -- and answers at(x, y). a snapshot has each view's colour where its frame says is what that is for, and the pairing is the point: the dump says
where React put a view, the picture says what is actually painted there, and
either half alone passes while the other is wrong.
They read coordinates from each app's own layout. Most tap fixed points:
e2e/index.tsx for the demo, and its own file for each of the apps written for a
scenario. Change a demo's spacing and the coordinates need changing too. The
alternative, searching the dumped tree for a button by its label and tapping its
centre, would be more robust and is worth doing; the reason it has not been is
that a wrong coordinate fails loudly and immediately.
Several apps set fixed heights on their labels for this reason. Text
measurement is the one thing Pango, Core Text and DirectWrite will never agree
on, so an app whose rows are positioned by measured text has different
coordinates on each host. A height on a label is what keeps one set of taps
working on all three.
Fast Refresh is the only scenario that runs in dev mode, which is why it
exists: "development still works" was being taken on trust, and a wrong claim
about it reached the README. It starts its own Metro on port 8099, edits
e2e/index.tsx while the app is on screen, and restores the file after the host
has exited, not before, because putting the original back while the app still
has a Metro connection triggers a second refresh that undoes the edit before the
tree is dumped, which looks exactly like Fast Refresh being broken. A kill -9
of the test would leave the demo edited; nothing softer will.
The edit does not work in CI. Metro on a GitHub runner never notices one:
the file changes on disk with a fresh mtime, a newly requested bundle still
carries the old text, and Metro's log is empty of complaint. fs.watch sees the
same edit on the same runner, the inotify limits are already generous, both sides
run the same Node, and the upstream Watchman build reports "watcher": "inotify"
over exactly the root Metro was given and changes nothing. It passes on
developer machines, on macOS and on Linux.
BASALT_SKIP_FAST_REFRESH therefore drops the edit, not the scenario. It
used to drop the whole thing, which meant no machine other than somebody's
laptop checked dev mode at all, and what CI cannot do is one specific half of
it. Everything before the edit is the host's own code: dev mode, the dev server
helper, the websocket, the DevSettings TurboModule, and a bundle fetched from
Metro rather than the release one sitting on disk. Two bugs have already lived
in that half: the host read its Metro entry from an argument nothing passed,
and every platform reported a dev script URL naming linux, and both would
have been caught with no edit made. When the variable is set the scenario passes
with a note saying what it did not check, which is the point: a scenario that
quietly checks less than its name says is worse than one that skips outright.
Input: real events, and where they are not
The end-to-end suite delivers taps one of two ways, and says which at the top of its output.
real: xdotool moves the pointer and clicks, so the event goes through
the X server and GDK exactly as a person's would. This is the only mode that
exercises event delivery itself. Chosen automatically when DISPLAY is set and
xdotool is installed.
injected, BASALT_TEST_TAP enters at the touch dispatcher, skipping the
window system. The fallback where a real event cannot be synthesised, which is
both of the other desktops: macOS needs accessibility permission an automated
run does not have, and Windows needs SendInput, which moves the real cursor
and so cannot run beside anything else on the machine.
Force either with --input real or --input injected.
Typing splits the same way. In real mode xdotool type sends key events
through the X server, so GDK and the input method see them. In injected mode
BASALT_TEST_TYPE reaches whatever field has focus: through GtkEditable on
Linux, the field editor on macOS, and real WM_CHAR messages on Windows, which
is the closest of the three; what it skips there is the keyboard driver and
nothing above it.
Windows is the exception to the exception in one place. A real click on a
<TextInput> never reaches the touch dispatcher at all, because the field's
peer is a child window and USER32 routes the click to it; so BASALT_TEST_TAP
focuses the field under the point as well as dispatching the touch. That step
exists only for the injected path.
Hover has no real mode on any of the three. BASALT_TEST_HOVER takes the same
"x,y;x,y" as BASALT_TEST_TAP and moves the pointer without pressing it; a
negative point is the pointer leaving the surface. There is no real-event
equivalent because a real hover means moving the machine's actual cursor onto
the window and leaving it there, which takes the pointer away from whoever is
using the machine, xdotool mousemove included.
Keyboard focus is injected everywhere too, for the same reason.
BASALT_TEST_FOCUS takes actions separated by ;: tab, shift-tab,
activate and escape, and enters at each host's focus manager. A real Tab
needs a window the display server considers focused, which an automated run does
not reliably have on any of the three; what the instrument skips is the delivery
of the keystroke and nothing above it.
escape and devmenu are the odd ones in that list and are there on purpose:
neither is a focus action. escape closes the topmost <Modal>, and devmenu
opens React Native's developer menu, the thing Ctrl+D (Cmd+D on macOS) opens.
This is the instrument for "a key was pressed and nothing on the window has to
be focused for it to arrive", which is what both of those are.
BASALT_TEST_FONT_SCALE is the odd one in a different direction: it is a
setting rather than an event, and it stands in for something two of the three
desktops do not have. GTK reads a real text scale from
GtkSettings:gtk-xft-dpi, which is what GNOME's "Large Text" moves; macOS
publishes no scalar at all, and Win32 would need WinRT for the one Windows
publishes. So allowFontScaling and maxFontSizeMultiplier could otherwise
only be tested where the machine running the suite happened to be set to large
text. The variable goes in where the platform's own scale goes, so nothing above
it is skipped, and because nothing is scheduled for it, it is deliberately not
in core/TestSettle.h's list; scripts/test_harness.py records that with its
reason.
A popup menu is the second thing an automated run cannot get past, and for a
sharper reason than a dialog: on macOS popUpMenuPositioningItem runs the
menu's own tracking loop on the main thread, so a menu nobody dismisses stops
the process where it stands. BASALT_TEST_MENU answers it: an entry index, or
dismiss: exactly as BASALT_TEST_DIALOG answers an alert, and with the same
limit: only the presentation is skipped, and whatever the entry does still
happens.
A file dialog is the third, and the one with the most behind it: what an app
does with a path cannot be reached without a path. BASALT_TEST_FILE_DIALOG
answers one: cancel, or a list of paths separated by : (; on Windows,
where a path starts C:\ and a colon would cut every one of them in two).
A tap may name a window: BASALT_TEST_TAP="134,110;134,110@3" taps in the app's
own window and then in the window whose surface is 3. Without it a second window
could not be reached at all: each has its own touch dispatcher, and the app's
would happily hit-test a tree that is not on screen and report a press on
whatever was at those coordinates. A point with no @ means the app's own
window, so every spec written before windows existed still means what it did.
BASALT_TEST_SECONDARY_TAP takes the same spec as BASALT_TEST_TAP and clicks
the other button. Its own variable rather than a suffix on a point, because the
two assert opposite things, one presses what it lands on and the other must
not, and a scenario that mixed them in one string would be harder to read than
to write.
BASALT_TEST_CLOSE_WINDOW takes a surface id and closes that window the way a
person would, its own close button rather than the app asking. The two are
different paths through the host, and only that one can leave the host holding a
record whose window is gone while the app still believes it is open.
It is also the only thing an app's interception ever refuses, which is what
makes it the instrument for onCloseRequest: an app closing its own window is
not asking anybody. 1 names the app's own window, and a host whose main window
refuses to close still has to shut down when BASALT_QUIT_AFTER_MS says so:
on Windows that meant the quit timer stopping going through WM_CLOSE, because
otherwise the app would have refused the harness too and the failure would have
been a hang rather than a test.
A locked screen is not a broken host
Leave the macOS suite running long enough for the Mac to lock itself and two scenarios fail, neither of them with a message about screens:
click a TextInput with a real mouse and see it focus
could not find the host window after 20 seconds
scrollTo({animated: true}) moves rather than jumps
no scroll offsets were reported at allA locked session realises no windows. The host still runs, still renders and
still dumps a tree, but its accessibility window list is empty, so the point to
click cannot be computed; and CADisplayLink does not fire, because there is
nothing on a screen to be in step with, so an animated scrollTo arrives with
no offsets in between. The second one is the dangerous shape: no offsets is
exactly what a host that dropped the animated flag would report.
Both scenarios now ask, and skip with the reason. screen_is_locked() reads
CGSSessionScreenIsLocked out of ioreg -n Root -d1, and answers false
anywhere it cannot be sure, a skip that fires wrongly being worse than a
failure. CI is never locked, so nothing is lost there.
What this cost before it was written down: two scenarios chased as regressions, including one that was blamed on a commit that had only touched text.
The suite is a background app, and macOS treats those differently
A host the harness launches does not get focus on a developer's Mac: the terminal running the suite keeps it. CI's macOS runner has nothing else on screen, so there the host does become active, and that difference hid two real bugs for as long as this suite has existed.
NSApp.keyWindow and NSApp.mainWindow are both nil for an app that is not
active. Two things asked only those two:
windowBounds()answered 0x0, and because the bounds cache is primed once at startup and rewritten only on a move or a resize, an app that was launched behind another window believed it was 0x0 for its whole life.getBounds()returned zeros andsetSizesilently did nothing.performMenuRolesent through[NSApp sendAction:to:nil], which walks the responder chain from the key window's first responder, so apasterole reached nothing.
Both now go through appWindow() (appkit/AppKitAppWindow.h), which is GTK's
rule: prefer the active window, fall back to the first visible one that is not a
panel. Both are covered by unit tests that work because the suite is a
background app, which is the useful half of this: basalt_appkit_tests has no
key window either, so a test that passes there has exercised the case a
developer's machine is always in.
The lesson for the next one: a scenario that passes on CI's Mac and fails on yours is not automatically flaky. It may be the only machine where the app is not frontmost.
A real click lands where the window server says, not where the test aimed
click a TextInput with a real mouse and see it focus posts a CGEvent at a
screen point, computed from the host window's position. The window server then
delivers that click to whichever window is topmost at that point, which is not
necessarily the window the point came from. An editor or a Finder window over
that corner of the screen takes the click instead, the field under test stays
unfocused, and the scenario reports the feature broken.
That is measured, not reasoned. With another application's window placed over the field the scenario failed 3 of 3 runs; without it, 0 of 5. It is why the scenario failed once on a developer Mac in a run whose only change was a text shadow.
So the scenario raises the host before it measures it: frontmost on the
process and AXRaise on the window, then half a second for the raise to reach
the window server. With the raise in place the same covering window produces 3
of 3 passes. The raise is not what focuses the field, which is worth stating
because it would be a comfortable way to make the scenario pass for the wrong
reason: raise the window and post no click, and the scenario still fails.
The host is also addressed by pid rather than by name. every process whose name contains "basalt" matches basalt_appkit_tests and any host left over from an
earlier scenario, and item 1 of that list is whichever the window server
answered with first. The Linux path had the same shape, xdotool search --name basalt-core taking the first match, and now prefers the window whose
_NET_WM_PID is the host's. Neither bug was the cause of the failure above:
every host opens a 900x700 window at the same default position, so the measured
rectangle is identical and the click lands in the same place either way, which
a run with a leftover host confirmed by passing. They are fixed because the
next scenario that moves a window would have been debugging the wrong thing.
Skia, where the tests do not run in CI
Two suites, for two renderers, and neither runs in CI because the package is borrowed from the app rather than vendored and CI has no app. That is a real gap and is the price of not vendoring a 300MB dependency; the rest of the suite is unaffected either way, and with nothing pointed at, none of it is compiled.
Which suite a build gets is decided by what it was pointed at, and the two are mutually exclusive by construction:
# 2.x, Ganesh over Metal: tests/test_appkit_skia.mm, six tests.
cmake -B build-skia2 -DBASALT_SKIA=<app>/node_modules/@shopify/react-native-skia
# 3.x, Graphite over Dawn: basalt-skia's three, nine tests in all.
cmake -B build-skia3 -G Ninja \
-DBASALT_SKIA_DIR=<app>/node_modules/react-native-skia \
-DBASALT_SKIA_LIBS=<app>/node_modules/react-native-skia-graphite-apple-macos \
-DBASALT_SKIA_DAWN=<app>/node_modules/react-native-webgpu-dawnThe three variables of the second are defaults, not requirements: basalt-skia
looks for all three in the app's node_modules on its own, and they are here
because the only app on this machine with a 3.x install is kino, which is not
where this repository's build runs. BASALT_SKIA is the 2.x path and
cmake/Skia.cmake refuses a 3.x install by version, so pointing both at the
same install is not a thing that can happen; scripts/test_skia_cmake.js is
that refusal's own test and does run in CI, since it needs no Skia.
Both were run against this repository's AppKit suite on 2026-10-11, separately: 786 tests with 2.14.0 and 789 with 3.4.0, all passing in both. The counts differ because the suites are different files, not because one skipped anything.
The canvas, and what a test of one can see. test_skia_canvas.mm drives
the AppKit peer with a real RnAppKitView, a real CAMetalLayer and a real
Dawn swapchain, which the 2.x canvas could never be tested with. Two of its
assertions were vacuous when first written and the sabotage pass is what said
so, which is worth recording because both failures are invisible from reading
the test:
- Points and pixels are the same number here.
RNSkWindowSurface::attachtakes pixels where 2.x took points, and the scale is 1 in a test -- it comes fromcurrentDeviceMetrics(), which answers from what a host reported, and there is no host. So passingframe.sizestraight through passed every assertion. The multiplication is now a function,skiaCanvasPixelSize, called with the density the test chooses; the one line that feeds the real density into it is still not covered and cannot be until something can set the scale. CAMetalLayerignores adrawableSizeof zero and keeps the last one, so attaching a zero-sized layout -- the case the peer guards, because Yoga hands out zero sizes before the first real layout -- looked identical from the layer. The surface's own size is what says, andskiaCanvasAttachedSizeexists for that.
And a frame, which is the claim none of the rest makes.
skia_canvas_draws_a_picture_and_reads_it_back goes the way an app goes:
SkiaApi builds a picture in JavaScript, SkiaViewApi.setJsiProperty pushes
it to the view by id, and SkiaViewApi.makeImageSnapshot reads back what the
view rendered. The same three calls e2e/skia.tsx makes, with no app and no
window, in the Hermes runtime the module test already needed.
Green over the left half of a 32x24 canvas and red over the right, sampled at
(8,12) and (24,12): two rectangles rather than one fill, so that "drew
something" and "drew it where it was asked to" are separate claims. Verified
falsifiable by swapping the two samples, which prints the colours the other
way round, and by removing the peer's registerSkiaView, which turns it into
No Skia View currently available. -- the exception an app would get.
What the 3.x suite proves that is new. skia_module_installs_skias_javascript_api
builds a Hermes runtime, calls install through the module's own property the
way JavaScript does, and reads SkiaApi back off the global object -- which is
the only thing that can say Skia's whole JavaScript API arrived, because
arriving is all install does. And
skia_context_makes_a_surface_that_draws is now a GPU round trip: the
canvas records, the recording is submitted, the texture is shared as an image
and read back. It found out it had become a GPU surface by peekPixels
answering false, which is worth knowing as a symptom -- a Graphite surface has
no CPU pixels to peek, and the test that used to peek them passed for a year
against raster.
Dawn gets a Metal device on a machine with no window and no display, which is why that round trip can run at all, and would run in CI if CI had an app.
What they assert is pixels, not objects. RNSkiaModule: Skia installed says the
bindings were constructed, and that is a long way from a pixel: a Metal device
that fails to create, a GrDirectContext that comes back null, a surface wrapping
a texture nothing can read, each installs perfectly well and then draws nothing.
So a known shape goes into an offscreen surface and the pixels come back out,
which is also the path an app rendering frames for export takes.
Verified falsifiable by drawing the rectangle somewhere else, which failed both halves at once:
expected inside.describe() == "rgba(255,0,0,255)" but got rgba(0,0,0,0)
expected outside.describe() == "rgba(0,0,0,0)" but got rgba(255,0,0,255)"Drew something" and "drew it where it was asked to" are separate claims, and
that is the pair that tells them apart. Corners rather than edges, for the reason
test_appkit_image.mm gives: antialiasing puts a boundary half a pixel either
way and pinning it fails on a future Skia for no useful reason.
A sabotage that does not build is not a passing test
Breaking the thing a test covers and watching it fail is the only way to know the test covers it. The hazard is that the two ways a sabotage can end look identical in a grep: the test failed, or the sabotage never ran. Three times on 2026-10-12:
| what was broken | how the sabotage died |
|---|---|
| the Win32 registry's includes | -Werror=unused-parameter on a name the gutted branch stopped using |
| the Skia TurboModule's factory name check | the same, one argument over |
localModulesMissingDesktop | TypeScript lost missing's inferred type once nothing was pushed to it, so dist/ kept the good build and the suite passed on it |
The third is the nastiest, because the suite genuinely passed: scripts/test_cli.js
requires the built package, a failed build_ts.sh leaves the previous dist/
in place, and the test then exercises code with no sabotage in it at all. The
first run of the Skia CLI sabotages did exactly this three times over and
proved nothing.
Two habits make it visible. Keep the sabotage compiling -- change a condition's threshold rather than deleting its body, so every variable stays used and every type stays inferable -- and read the build's exit status rather than only the test's output. A sabotage that prints no failing test and no error is the case to look at twice.
Knowing whether CI actually passed
scripts/ci_status.py says what each job last decided, which is not what gh run list shows. gh shows the latest run; when that run was cancelled by a
newer push there is no verdict in it, and nothing distinguishes "not tested" from
"tested and fine"; a cancelled job is not a failed one.
That cost five runs once. The Linux job was cancelled on five consecutive pushes to main while Windows failed on every one of them, and what exposed it was a sixth push adding a macOS job, whose failure was then assumed to be new. Five red runs read as four cancellations and a mystery.
build and test success ace5dd3 2026-09-26T06:25:27Z
the AppKit host success ace5dd3 2026-09-26T06:25:27Z
the whole Windows stack success ace5dd3 2026-09-26T06:25:27ZEach row is the newest run that job reached a verdict in, and a row gains <- and N newer run(s) never finished, so this is older than it looks when there are
later runs it said nothing in. That count is the blind spot, and it is the number
that would have exposed the day above.
Shards fold together (the AppKit host (shard 2) is the AppKit host) and a
job counts as decided for a run only when every shard decided. Three green
shards and one cancelled is not a tested commit. It exits non-zero when any job's
last verdict was a failure, so it works as a check and not only a report.
The root cause is fixed too: cancel-in-progress is now
${{ github.ref != 'refs/heads/main' }}, so main never loses a result and
branches still free their runners. scripts/test_ci_status.py checks the logic
against that five-run history, because a bug in a tool like this points the same
comfortable way as the problem it was written for.
near is a macro, and it is in <minwindef.h>
A test helper called near compiles on both desktops and fails on Windows with
error: expected unqualified-id, because near and far are legacy no-op
macros the Windows headers still define. The declaration becomes
bool (float, float).
It does not fail in every file, which is what makes it confusing: it fails in
the ones whose includes reach windows.h, which
<react/renderer/attributedstring/AttributedString.h> does. Three core tests
spell it near today and compile, and test_text_shadows.cpp did not. Call it
closeTo. It cost a CI round trip, which is the only reason it is written down
here.
Windows, on a machine you own
The Windows job is the one nothing here can check. No development machine in this project runs Windows, so a Windows-only file is written, pushed, and answered fifteen minutes later, and if it is wrong, that is fifteen minutes per attempt. It is also the least of the problem: the Windows implementation of the subprocess capability passed on its first run, and the two things that failed around it were a test that assumed my machine and a missing include.
So this is worth setting up when there is Windows work in flight, and not otherwise.
scripts/setup_windows_runner.ps1, run as administrator on the Windows
machine, enrols it under the label basalt-windows and installs the runner as
a service. It checks first and installs nothing: Visual Studio with the C++
workload and C++ Clang tools for Windows (the core is built with clang-cl,
not cl: see docs/PORTING.md), vcpkg with VCPKG_INSTALLATION_ROOT pointing at
it, Git for Windows for the bash that runs bootstrap.sh, cmake, ninja and node.
A machine missing any of them is told all of them at once.
Enrolling a machine does not move the job to it. That is a separate step:
gh variable set WINDOWS_RUNNER --body basalt-windows # use it
gh variable delete WINDOWS_RUNNER # go back to GitHub's imageTwo decisions rather than one, because a self-hosted runner that is switched off does not fail a job: it queues it, for the job's six-hour timeout, and a run that never reports looks exactly like a run that is still going. Deleting one variable is a fix that can be made from a phone; taking a label out of a workflow and pushing it is not.
What it buys is that ccache, vcpkg's binaries and the vendored third_party
survive on local disk between runs rather than being restored from a cache, so a
warm run is minutes rather than a quarter of an hour. What it costs is that the
machine's state is now part of the build: a green run on a machine with something
left over from last month is not the same evidence as a green run on a fresh
image. When a Windows failure looks impossible, unset the variable and let
GitHub's image answer.
Sharding
--shard I/N runs the Ith of N shards of the end-to-end suite, and
BASALT_COMPARE_SHARD=I/N does the same for compare_all.sh. CI runs three of
each, on the same index, so a shard does a third of both.
Why three, what each job actually costs, and the ccache fault that hid inside those numbers are recorded in docs/ci-performance.md, which is in the repository but not on the site. None of it is needed to run a shard.
The crash reports a passing run leaves on macOS
A full run leaves a crash report named after the host, and it is a test.
test_crash_handler sets BASALT_TEST_CRASH, which makes installCrashHandler
raise the signal on purpose, because a handler that has never run is a guess.
macOS reports any process that dies on a signal, so every run of the end-to-end
suite writes one .ips into ~/Library/Logs/DiagnosticReports. There is no way
to opt a process out of that reporting, so this is the note instead.
Telling it from a real one takes one line of the report's faulting thread:
raise <- basalt::(anonymous namespace)::crashIfAsked <- basalt::installCrashHandlerThat is the deliberate one, and it happens before a window exists. A real crash
is anywhere else. One report per run, named after whichever host ran: measured
by running the scenario on each, which leaves a basalt_appkit report and a
basalt_gtk one. Reading the stack out of a report, rather than opening the
Console:
python3 - <<'EOF'
import glob, json, os
reports = os.path.expanduser("~/Library/Logs/DiagnosticReports/basalt*.ips")
for path in sorted(glob.glob(reports)):
header, body = open(path).read().split("\n", 1)
head, report = json.loads(header), json.loads(body)
images = report.get("usedImages", [])
thread = report["threads"][report.get("faultingThread", 0)]
print(head["timestamp"], head["app_name"],
report.get("exception", {}).get("signal"))
for frame in thread["frames"][:10]:
image = images[frame["imageIndex"]] if "imageIndex" in frame else {}
print(" ", image.get("name", "?"), frame.get("symbol", ""))
EOFWorth doing after a session of work rather than never: every one of the
twenty-four reports on this machine on 2026-10-08 was either that deliberate
crash or a bug that the same day's commits fixed -- nineteen of them the
use-after-free in ~GtkFocusManager, which the GTK suite's own crash handler had
already named, and two from GSK assertions that the box shadow and
accessibilityLabelledBy work turned into clamps and a different call. The
check is cheap and it is the only place a crash that happens after a test's
assertions pass would show up.
Test instruments, and what they cannot reach
Dragging out has no automated coverage at all, and cannot have: once a
drag begins the system owns it, and no instrument can put a file manager on the
other end to receive the drop. What the suite does prove is that a drag source
is inert when no view is marked: every press, scroll and context-menu
scenario passes with one attached, which is worth knowing and is not the same
as proving a drag works. The verification is e2e/drop.tsx's green row and a
person dragging it somewhere.
An instrument goes on all three hosts, or the scenario that uses it skips
where it is missing. Written down because it has been got wrong twice, the
same way both times: BASALT_TEST_QUIT and then BASALT_TEST_DROP were added
to GTK and AppKit, the scenario was written against those two, and Windows
failed for a reason that had nothing to do with what the scenario tests. The
failure is especially misleading: "the inner target was not told" reads as a
broken feature rather than a missing instrument, and the feature was fine.
BASALT_TEST_DROP takes x,y:path and reports a file dropped at that point.
It enters below the toolkit, the way BASALT_TEST_TAP does and for the same
reason: a real drag needs a source outside the process, and no test can conjure
one. So what it exercises is the hit test, the walk up to the nearest marked
ancestor, the accept check, the event and React's half; every part that could
be wrong about which view is told and what it is told, and not
GtkDropTarget or NSDraggingDestination themselves. Worth stating, because a
scenario that looks like it drives a real drag and does not is worse than one
that admits it.
BASALT_TEST_QUIT is the same idea one level up: it asks the application to
quit the way a person does (Cmd-Q on macOS, the session ending on Linux) so a quit an app refuses can be driven from a script. A count rather than a
flag, because one ask cannot show both halves: a refusal is proved by the
process still being there afterwards, and the agreement that follows is proved
by it going before its own timer.
That the harness's shutdown is exempt is not a detail. BASALT_QUIT_AFTER_MS
quits AppKit by calling terminate:, which arrives at the very handler an app
uses to refuse, so the demo for this feature would have refused the harness,
and every scenario running it would have hung until its own timeout and
reported something other than what it was testing. The same trap as the
Windows one above, two years of API apart. GTK and Win32 need no exemption:
their session signals are not on the path g_application_quit and
PostQuitMessage take.
BASALT_QUIT_WHEN_SETTLED=<ms> ends a run when there is nothing left to wait
for, rather than on a budget chosen before the process started.
No scenario opts into it today, and the reason is worth reading before
re-enabling any. A settle waits for the first mount and for scripted input to
be delivered. It cannot wait for the round trip to JavaScript, and most
scenarios assert on what JavaScript logged, so the host quits while the
evidence is still in flight. Alert.alert failed on CI with "the alert's
callback never reached JavaScript"; the hover scenario failed locally with a
truncated event list. The opt-in list was built by opting everything in, running
the suite and keeping what passed, which is unsound: passing four times does not
establish the absence of a race. It was trimmed twice before the method rather
than the entries was accepted as the problem, and the suite went back to 445
seconds from 298. Those 147 seconds were not real.
The right shape is per scenario, and the tool already exists: wait for the
line the scenario is about to assert on, then write BASALT_TEST_QUIT_FILE. That
is what stop_host does for Fast Refresh. It is correct rather than
probabilistic, and it is faster than a settle, because it ends at the evidence
rather than at a fixed delay after input. Settled means
the first mount has been applied and any scripted input has been delivered, plus
a short settle for React's next commit. BASALT_QUIT_AFTER_MS stays as the
backstop, so nothing can get slower and a host that never settles still ends.
Measured on the whole macOS suite: 445s to 210s, twice, to the second. The
developer menu scenario went from 39.5s to 8.2s, test_controls from 43.1s, the
application menu from 7.6s to 1.0s.
The first mount is recorded in MountingWalk::applyMutations, which is the one
place all three hosts mount through, so this is one code site rather than
three, for the reason core/TestQuitFile.h gives about shared spellings. Only
the first mount counts: taking the latest would push the settle out on every
commit, and an app that re-renders steadily would never settle at all.
The first version saved nothing on the scenarios it was written for. It
measured 2.96s against a 3s budget, because the hosts report their scripted
schedule whether or not any input was asked for, and that total is never zero,
it starts at 1500ms so a tap has a window to land on. Core ignores an empty
schedule now, which is what took the no-input case to 1.4s. Deciding whether a
schedule is real needs a list of the instruments that make one, and that list
must be scheduled instruments: BASALT_TEST_MENU and BASALT_TEST_DIALOG look
like they belong and do not, because they answer something when it appears rather
than being laid out on a timer. That error was caught by the check in
scripts/test_harness.py, which holds the list against what the three host mains
actually read, in both directions.
Which scenarios opt in is a list of scenarios that have been run, not of
scenarios that looked fine. Every one was opted in at once and the suite run;
seven failed, and all seven for the same reason, work after the last input with
nothing scheduling it. An animation is time. A refusal followed by an agreement
is two events deliberately separated by time. A trace update that takes itself
down again is measuring a decay. A settle cannot stand in for a duration that is
the thing under test, and those keep their budgets. The list and each exclusion's
failure text live in scripts/integration_test.py.
Off entirely in real input mode. A settle accounts for what the host
scheduled, and real input is driven from outside the process: the harness waits
four seconds for a window and then moves the pointer with xdotool, having set
none of the BASALT_TEST_* variables. Nothing is requested, so the host settles
at the first mount and has quit before the click lands. Linux CI is the only
place that mode runs, which is why every local run on a Mac passed and all three
Linux shards failed on the first push. The rule underneath is the exclusion list's
rule one step further out: a settle cannot stand in for a duration it cannot
see, and it cannot see another process. So the saving is a macOS and Windows
saving; Linux CI keeps its budgets.
Guessing is only safe because getting it wrong fails loudly: everything opted in
asserts on the dumped tree or a logged line, so quitting early produces a missing
assertion rather than a quiet success. BASALT_NO_SETTLE=1 turns the whole thing
off, which is how the saving was measured and how to tell "this scenario is
broken" from "this scenario needed longer than it was given".
The suite also prints each scenario's duration now. Its cost is the thing most often being worked on here and was only ever visible as a total, which is why every estimate in this area had been a guess.
BASALT_TEST_KEY presses a key: "m;j;q;z+meta", presses separated by ; and
modifiers by +. Parsed once in core/TestKeys.h rather than in three hosts,
because three parsers is three chances to disagree quietly about what z+meta
means: the wrong answer is a combination that matches nothing, which reads as a
shortcut that did not fire rather than as a bad instrument.
It enters after the platform translation, not before it, so what it does not
exercise is the NSEvent, keyval and WM_KEYDOWN mapping, which is the half
most likely to be wrong, and is unit-tested per host instead (twenty-seven tests
across the three). A scenario passing here says the registry, the focus path and
the round trip to JavaScript work. It says nothing about whether pressing the left
arrow produces "ArrowLeft".
On all three hosts, which is the rule below rather than a coincidence.
BASALT_TEST_QUIT_FILE is how the harness ends a run at a moment it picks,
rather than on a budget fixed before launch. The host polls for the path and
shuts down as soon as it exists, through exactly the path BASALT_QUIT_AFTER_MS
takes, so the tree is dumped and the exit code is the app's. Existence is the
whole message; nothing is read from the file, which is why there is no race
between creating it and seeing it.
It exists because the portable way to end a host used to be a signal, and there
is no portable signal: Windows has no SIGTERM and TerminateProcess cannot be
handled. The scenario that needed it is Fast Refresh, whose length is not known
in advance: an edit reaches the window in a second on a warm laptop and in up
to two and a half minutes on a cold CI machine, so no fixed budget is both quick
and safe.
Polled at 250ms rather than watched, on the timer each host already runs for
BASALT_QUIT_AFTER_MS. A file watcher is three APIs with three failure modes on
network and virtual filesystems, and a quarter-second poll cannot fail to be
delivered. Measured end to end on GTK and AppKit: 0.24s from writing the file to
the process exiting 0 with its tree on disk.
The skip path asserts that quitting this way wrote the widget tree, not merely that the process ended. Without that line CI proves the poll fires and says nothing about the half the instrument was added for, since the only scenario that reads the tree after a quit-file shutdown is the edit path, and the edit path cannot run on a CI machine. It costs a stat and it is the only place the whole chain is checked by a machine rather than by hand.
A host that does not answer within 30 seconds fails the scenario rather than being quietly killed. It was quietly killed at first, and the consequence was immediate: the first Windows run reported green while proving nothing, because a dead poll and a working one are indistinguishable when the fallback is silent. Verifying it fired meant grepping a CI log for the shared line, and the grep matched a compiler diagnostic quoting the header rather than a host logging it. So the check moved into the harness, where it cannot be forgotten.
Not refusable, for the reason the timed quit is not, see below. The variable
name, the poll interval and the log line all live in core/TestQuitFile.h so the
three hosts cannot disagree about them; BASALT_QUIT_AFTER_MS elapsed; quitting
had to be retrofitted onto Windows after a scenario asserted on a string only two
hosts printed, and scripts/test_quit_file.cpp asserts the shared spellings
rather than trusting them.
BASALT_DUMP_TREE writes every window, each under a --- window <n> ---
header, for the same reason it already appended the error inspector: they are
separate trees on screen, and nesting one inside another would say something
untrue.
A menu bar cannot be reached by any of them, because a menu cannot be opened
without a person. BASALT_DUMP_MENU writes the menu the platform actually
installed instead, read back from AppKit or from the HMENU rather than from
the description that was sent, so it says a <Menu> became a real menu with the
shortcuts the platform attached to its roles. It is written even when it is
empty, because "this platform has no menu bar" is what a test on Linux asserts.
The wheel has its own: BASALT_TEST_SCROLL takes "x,y,lines" triples
separated by ;, a wheel over a point, in surface-root coordinates, positive
lines scrolling down, which is the direction contentOffset reads. Each host
enters it at a different depth, deliberately: Windows sends a real
WM_MOUSEWHEEL through the real window procedure, macOS builds a real NSEvent
and hands it to the view its own hit test found, and Linux enters at the scroll
manager where GtkEventControllerScroll would. It is also the only way to reach
<RefreshControl>, whose gesture on a desktop is a wheel that keeps asking to go
up after the list has already reached its top.
A modal dialog is the one thing no instrument could reach and no person is
present for, so BASALT_TEST_DIALOG answers it instead of showing it: a button
index, or dismiss for the last button. It covers Alert.alert and the share
picker on every host, and it is the one place this project deliberately does not
drive the real thing: see native/core/TestDialog.h, which sets out why
(Apple's own addUIInterruptionMonitor is documented as not firing on recent
iOS versions, and reaching into the dialog through accessibility needs a
permission an automated run does not have).
What it skips is the presentation and nothing else. The request is built by the same module from the same JavaScript, the answer travels back through the same callback, and on the desktops whose share picker is built from a clipboard and a mail client, a scripted "Copy" really copies, which is what the end-to-end suite asserts on.
Without it, a run that opens a dialog hangs rather than failing: on macOS a
sheet that is up stops [NSApp terminate:] outright. That is a real bug rather
than a testing inconvenience (a person pressing Cmd-Q would see the same) so
the host now closes any open sheet before it tries to quit.
That cuts the other way too, and it is worth knowing before a tree looks wrong:
a window that opens under someone's cursor is hovered, before either host has
drawn anything. e2e/hover.tsx is left out of scripts/compare_all.sh for that
reason, and the end-to-end suite asserts the order events arrive in rather than
that nothing else happens.
Animations the suite drives itself
An animated <Image> does not animate on its own. The view advances by a number
of milliseconds its caller passes, and the mounting manager is what starts a
clock, so a test says "a hundred and twenty milliseconds later" and renders.
That split exists because of a real failure rather than a preference. The first
version drove itself from the GTK frame clock, and renderView spins the main
loop to get a frame out of a widget, so every pixel assertion found the
animation several frames past where it left it: four samples of the same GIF
reported red, red, red, red with the frame changing underneath. Both hosts are
split the same way, even though an AppKit timer is inert in a suite that never
runs its run loop.
Fonts loaded at runtime, and the font map on a Mac
Three of the four font tests in each host's suite register a font the machine
already has, under a second name of the app's choosing, which is the mapping
expo-font needs and all a portable test can arrange. The fourth registers one
the machine does not have, because that is the case a loader exists for, and
it gets there without a fixture font: tests/RenamedFontFile.h copies a real
file and renames the family inside it, one byte per occurrence of the name, so
no offset in the file moves and no checksum has to be recomputed.
The GTK suite stops early for that fourth test on a Mac, and it says so
rather than failing. Homebrew's Pango answers
pango_cairo_font_map_get_default with a PangoCairoCoreTextFontMap, which
never consults fontconfig, so a font added to fontconfig is registered and
invisible. Running the GTK host next to the AppKit one is a comparison
arrangement rather than a target, and the Linux fact is proved on Linux.
Expo, and the apps that need it
Two of the apps in e2e/ import an Expo package, notifications.tsx imports
expo-notifications, and this directory is not an npm package, so there is
nothing for Metro to resolve them against. Both halves have to be pointed at an
app that does have them installed:
cmake -B build -DBASALT_EXPO_MODULES_CORE=/path/to/app/node_modules/expo-modules-core
BASALT_EXPO_APP=/path/to/app scripts/bundle.sh --entry notifications --out notifications.jsbundle
BASALT_EXPO_APP=/path/to/app scripts/integration_test.pyThe same app for both, so that Expo's C++ and its JavaScript are the same version, the same reason the host builds against the app's React Native. Without them the host has no Expo at all and the scenario skips saying so, which is what a plain checkout and every CI job do.
e2e/fonts.tsx needs the host's half and not the app's: it reaches
expo.modules.ExpoFontLoader through the global rather than importing
expo-font, so only the build flag matters and no package is borrowed.
cmake -B build-expo -G Ninja -DRN_DIR=../react-native/packages/react-native \
-DBASALT_EXPO_MODULES_CORE=/path/to/app/node_modules/expo-modules-core
cmake --build build-expo --target basalt_appkit
scripts/bundle.sh --prod --platform macos --entry index.tsx --out main.jsbundle \
--build-dir build-expo
scripts/integration_test.py --build-dir build-expo -k "font loaded at runtime"That is how it was first run, against Expo 57 and a host built on React Native main, on 2026-10-09.
Notifications, and the daemon that is never there
GtkNotifications.cpp sends Notify over the session bus, and neither a
developer's Mac nor a CI runner has a desktop's notification daemon listening
for it. So the suite brings its own: basalt_notification_stub owns
org.freedesktop.Notifications, answers Notify with an id and prints what it
was asked to show, and scripts/integration_test.py starts it on a session bus
of its own for the length of one scenario.
dbus-daemon directly rather than dbus-run-session: on macOS the latter
insists on launchd's socket and fails with "DBUS_LAUNCHD_SESSION_BUS_SOCKET is
empty". brew install dbus is enough on a Mac; a Linux runner has it already.
Without dbus-daemon the scenario still runs and asserts the other half: that
the host reports why it cannot send, which is also what macOS and Windows
report on every machine, daemon or not.
Running on Linux
The project targets Linux and is developed on a Mac, so everything here should be checked on Linux before it is believed. A VM is enough:
brew install lima
limactl start --name=basalt <a template with vz, 8 cpus, 16GiB, 120GiB>
limactl shell basaltApple Virtualization rather than QEMU: the guest is aarch64 like the host, so it runs at native speed. Building React Native's core and Hermes under emulation would take hours.
Inside, on Ubuntu 24.04:
sudo apt install build-essential clang cmake ninja-build pkg-config git curl \
libgtk-4-dev libpango1.0-dev libglib2.0-dev libgoogle-glog-dev \
libboost-dev libboost-regex-dev libfmt-dev libdouble-conversion-dev \
libgflags-dev libssl-dev libcurl4-openssl-dev
# Node 24: React Native needs ^22.13 || ^24.3 || >= 26, and 24.04 ships 18.
curl -fsSL https://deb.nodesource.com/setup_24.x | sudo -E bash - && sudo apt install nodejsThen the usual bootstrap, cmake, build.
Rendering, on Wayland
A real compositor, headless, with no GPU:
sudo apt install sway grim mesa-vulkan-drivers libgl1-mesa-dri
printf 'output HEADLESS-1 resolution 1400x1000\ndefault_border none\n' > /tmp/sway.conf
XDG_RUNTIME_DIR=/run/user/$(id -u) WLR_BACKENDS=headless WLR_RENDERER=pixman \
sway -c /tmp/sway.conf &
WAYLAND_DISPLAY=wayland-1 GSK_RENDERER=gl ./build/basalt_gtk build/main.jsbundle.js
grim shot.pngTwo renderer choices, for two different layers, and they are not the same one.
WLR_RENDERER=pixman is the compositor's. Without a GPU, wlroots' default
fails with drmGetDevices2 failed and sway starts but draws nothing.
GSK_RENDERER=gl is GTK's, and it must not be cairo. GTK 4.14's cairo
renderer draws a transformed widget subtree unrotated and in the wrong colour;
the demo's rotated card comes out as a flat pink square. The GL renderer gets it
right, and works headless through Mesa's software rasteriser, so there is no
reason to reach for cairo. This cost an hour of suspecting the transform code,
which is fine on macOS and fine here under GL.
Input, on X11
Real pointer events need a seat with input devices, which a headless Wayland
compositor does not have: sway sees no inputs, and swaymsg seat … cursor
silently does nothing. Getting one means logind, a session and uinput, which
is a lot of machinery for one assertion. X11 gives the same guarantee for far
less:
sudo apt install xvfb xdotool
Xvfb :99 -screen 0 1400x1000x24 &
DISPLAY=:99 GDK_BACKEND=x11 GSK_RENDERER=cairo scripts/integration_test.pyThat is a genuine event through the X server and GDK. The wheel path checks out
too: five xdotool click 5 notches move the content exactly 265 points, which
is the 53-point step in GtkScrollViewManager times five.
Dragging a window, which no suite can do
<TitleBar.DragRegion> says which parts of an app-drawn header move the
window, and three things about it are tested in three different places,
because they fail differently:
- Which points drag is a question about the view tree, and each host
answers it with its own walk:
test_titlebar_drag.cpp,test_win32_titlebar.cppandtest_appkit_titlebar_drag.mmask the same four questions in the same order, for the reason the paint tests do. - That a press on one is taken is the half that can be written and never
called, which is what macOS did until 2026-10-19: the rule existed in
core/TitleBarRegions.h, two hosts applied it and the third never asked.a_press_on_a_drag_region_moves_the_window_instead_of_pressingintest_appkit_input.mmis the test that fails if the call goes away. - That the window actually moves is nobody's test. It needs a real
pointer, and a real pointer on macOS means
CGEventand accessibility permission an automated run does not have.e2e/titlebar.tsxexists to be dragged by hand, andscripts/integration_test.py -k "title bar"runs the same app to check the arithmetic around it: the style the host reports, the metrics reaching a render, and the header being as tall as the metric says.
What is still not covered
Wayland input. Rendering is verified on Wayland and input on X11, but not both at once. Closing it needs a compositor with a real seat: a desktop session in the VM, or a physical Linux machine.
Rendering. All three hosts assert on pixels now, each with no window on
screen and no reference image: Direct2D into a WIC bitmap
(test_win32_paint.cpp), Core Graphics into a CGBitmapContext through the
view's own drawRect: (test_appkit_image.mm and two others), and GSK into a
GdkTexture (test_gtk_paint.cpp, over tests/GtkPixels.h). The three ask the
same eleven questions in the same order, because the answers are supposed to be
the same picture.
That gap is what let GTK's cairo renderer mangle every transform in the demo without a single test noticing. What is still not asserted is the thing a reference image would catch and these cannot: whether a glyph is the right shape. Font rendering differs enough between machines that reference images would not travel, so the pixel suites ask about colour, placement, clipping and order -- all of which are exact -- and text is asserted by where its ink lands rather than by what it looks like.
One caveat lives in the GTK helper and it logs which renderer answered: the GL
one is what an application uses and the cairo one is the fallback on a machine
without GL, and they do not agree about transforms. So the GTK pixel suite sticks
to questions both answer the same way, and transforms are checked through
gtk_widget_compute_point.
Threading. The mounting manager asserts it is on the main thread, and text measurement takes a mutex, but nothing exercises the JS thread and the main thread concurrently.
Nothing, as of the day this repository went public. All three hosts are
built and tested on every push: Linux and Windows have been for a while, and
macOS joined when standard runners became free and the ten-times billing
multiplier stopped being a reason to leave it out. The macOS job also runs
compare_all.sh, which no other runner can, because no other has both hosts
installed.
This paragraph used to say "CI is Linux only ... macOS is covered by whoever is working; it is not covered by a machine", which was already wrong about Windows and stopped being true of macOS on the same day. Left here as the answer to the obvious question rather than deleted.
CI
.github/workflows/ci.yml. Installs the dependencies, builds, and runs both
suites under Xvfb with xdotool present, so the end-to-end suite is in real
input mode.
It builds against a pinned React Native commit, in scripts/react-native.pin.
This project tracks main, which is right for development and wrong for CI: a
build that follows main turns every upstream change into a red tick on an
unrelated pull request. A separate drift job builds against main weekly and
is allowed to fail, so upstream churn is noticed without blocking anything.
Three caches carry the cost: React Native's node_modules and third_party/
(which holds the Hermes build) are keyed on the pin, and ccache on the commit
with a rolling fallback.
They are the difference between a 42 minute run and a 2 minute one.
Cold, a run installs React Native's dependencies (949MB of node_modules),
builds Hermes, and compiles 400-odd objects of React Native core. Warm, it
restores 304MB of third_party, hits ccache, and goes straight to the tests.
Two things about those caches that are easy to lose an hour to:
- The
third_partykey hashes the whole ofbootstrap.sh, so editing a comment in it costs a Hermes rebuild. That is deliberate: over-invalidating is the safe direction, and the script changes rarely now. - A cache saved on a branch is invisible to
main. GitHub scopes caches to the branch that created them, and shares them only downwards, to that branch's descendants. So the first run after merging a branch is cold all over again, and only then do the caches land somewhere every later run can reach. actions/cachewill not save a path that escapes the workspace. It says nothing about it either; the cache simply never appears. React Native is therefore cloned toreact-native-src/inside the checkout rather than beside it. Getting this wrong is quiet until the daythird_partyhits: then bootstrap skipsyarn installbecause codegen is already there, and bundling fails for want ofnode_modules.