Performance Optimization

What started as an investigation into slow TUI rendering became a four-month rewrite of the rendering stack: CPU usage down 97%, memory allocations down 99.7%.

Final Results

Benchmarks run 5 times per commit, values are mean ± std.

Metric	Phase 1: Basic Caching (5646a10)	Phase 2: Optimized Bubble Tea (2988f71)	Phase 3: Initial zeroterm (e2b5802)	Phase 4: Final (808a739)	Improvement
Duration	93.81 ± 3.99s	91.45 ± 1.67s	90.86 ± 2.55s	89.76 ± 2.08s	-4.3%
CPU Samples	51.94 ± 4.52s	10.15 ± 0.21s	2.36 ± 0.17s	1.42 ± 0.09s	-97.3%
CPU Utilization	55.29 ± 2.65%	11.10 ± 0.27%	2.59 ± 0.13%	1.58 ± 0.07%	-97.1%
Total Alloc	17,649.10 ± 969.73 MB	4,388.47 ± 76.51 MB	432.80 ± 17.84 MB	44.23 ± 4.87 MB	-99.7%
Total Alloc (objects)	34,584,210 ± 1,660,990	4,072,698 ± 523,252	2,829,148 ± 168,550	273,916 ± 49,438	-99.2%

Benchmarks run on the e2e deploy test with real Nix builds. Commit hashes in parentheses mark each phase’s endpoint.

goos: linux
goarch: amd64
cpu: AMD Ryzen 9 5900HX with Radeon Graphics
ran: 2026-06-23

Why This Was Necessary: Go’s Runtime and GC

The four phases were not arbitrary refactors. They were a sequence of attacks on specific Go runtime behaviors that Bubble Tea’s string-heavy design triggered. Go strings are immutable: every concatenation, split, or truncation allocates a new string on the heap. Bubble Tea uses strings as the intermediate representation for all rendering, so every frame produced thousands of short-lived allocations that immediately became garbage. The Phase 1 baseline measured 34.5 million object allocations per e2e run. CPU profiles showed most time in runtime.mallocgc, not in application code.

Because Go’s GC is non-generational, there is no shortcut for short-lived objects. Every allocation contributes to heap growth, and when the heap doubles, a GC cycle scans the entire live object graph. At 34.5 million objects, cycles fired back to back. Reducing allocations has a compounding effect: fewer allocations means less heap growth, which means fewer GC cycles, which means less CPU spent scanning and pruning. The drop to 273,916 objects in Phase 4 was a 126x reduction that collapsed GC overhead proportionally, not just linearly.

The custom engine attacks this from multiple angles. sync.Pool reuses backing arrays across frames so they never become garbage. A contiguous []byte buffer with []int line offsets replaces []string, turning per-line allocations into zero-copy slices. buf[:0] resets the length while retaining capacity, so the same backing array survives across frames. Scratch buffers use fixed-size arrays (var pfxBuf [1024]byte) to stay stack-allocated via escape analysis. unsafe.Slice in the stringbyte package shares string backing memory as []byte without copying, eliminating the standard []byte(s) allocation on every read-only access in the hot path. Double-buffered frame diffing via bytes.Equal ensures only changed lines produce terminal output, bounding both rendering work and I/O by the delta, not the frame size. See Custom TUI Engine for the full architecture.

The trade-off is that eliminating allocations shifts the bottleneck. In the final phase, CPU profiles show runtime.memmove as the largest non-syscall node (10-26% of samples depending on the run; copying bytes between buffers to assemble terminal output), with unicode grapheme width calculation via rivo/uniseg (since replaced by a custom rune-width implementation), memeqbody (the bytes.Equal calls for frame diffing), and GC overhead each in the low single digits. None of these are realistically optimizable further in Go: the memory copies are the minimum required to assemble terminal output, unicode width is a fundamental requirement for correct layout, and frame diffing is already O(changed lines).

Why a custom engine

Plenty of TUIs never had this problem: their output is append-only, while Panix renders a tree of nested viewports that can shrink.

Append-only output keeps rendering simple. Lines already on screen never move, so layout is just appending at the bottom, and a frame diff only ever touches the end of the screen. Naive string rendering is good enough for that.

In Panix, each running command has a viewport showing its live output. The viewport grows while the command runs, and when the command finishes it can collapse to nothing, removing its node from the tree. Everything below shifts up, the parent’s height changes, and so on through the tree. With dozens of machines updating at once, this happens constantly, and rarely at the bottom of the screen.

Other deployment tools avoid the problem by showing only the last line of Nix output, or no intermediate logs at all. We show the full tree, so shrinking layouts had to be fast.

The beginning (naive approach)

We had a working TUI built on Bubble Tea. It was fine at first, but with the introduction of e2e tests it became obvious that Panix’s single thread was maxing out and making the TUI unresponsive.

The renderer was doing far too much work. Every frame re-rendered every component from scratch, even when nothing had changed: stats table, footer, viewports, phase status, build logs.

Phase 1: Basic Caching

The initial solution was hash-based caching: compute an FNV hash of each component’s content, skip the render when the hash matches the previous frame. The stats table alone saw a 96.7% CPU reduction. Then we applied the same pattern to the footer, viewports, phase status, and build logs. The trade-off was minor: animated spinners in the stats table had to go, since they require constant re-rendering by definition.

The key insight from this phase was simple: the app was spending over 90% of its CPU time on TUI, re-rendering content that hadn’t changed. Caching would fix that almost entirely.

Phase 2: Optimized Bubble Tea Components

With caching in place, the next target was the components themselves. The viewport got improved cache hits by tracking whether only appended content changed. The tree renderer was rebuilt with node reuse based on xpath. A custom tui.style package replaced Bubble Tea’s string-heavy styling with pre-rendered ANSI sequences. Benchmarks were added for all TUI packages to measure progress.

These were meaningful improvements, but they were optimizations on top of a framework that re-renders entire views every frame. CPU dropped to 11% and allocations to 4.4 GB, but it was clear that the framework itself was the ceiling.

Phase 3: Initial zeroterm

The major change was building a custom TUI rendering engine from scratch (PR #2). This was the single most impactful decision in the entire optimization effort.

zeroterm was built because Bubble Tea, despite being an excellent framework, re-renders entire views every frame with no built-in support for incremental rendering and using strings as an intermediate in all operations. Bubble Tea has to split, combine, append or cut strings at almost every step, making it a heavy burden on the Go garbage collector. For a deployment tool that monitors dozens of machines simultaneously and updates in real time, this meant the renderer was doing O(n) work every frame even when only a single line changed.

The entire pkg/tui/ and pkg/buffer/ packages were built from scratch (see Custom TUI Engine for the full architecture). CPU dropped from 11% to 2.6%, and allocations fell from 4.4 GB to 433 MB.

Phase 4: Final

With the zeroterm architecture in place, the final phase was about rewriting the components built on top of it to fully leverage zero-allocation patterns.

PR #7 rewrote the viewport (scrollable content buffer, horizontal border cache), rebuilt the tree renderer, and optimized the style system. PR #9 rewrote build_logs to use the contiguous buffer directly, added a stringbyte package for zero-allocation string/byte conversions, optimized atomicslice, and improved xpath caching. The stats table and phase flow components were also reworked to minimize per-frame allocations.

These changes brought allocations from 433 MB down to 44 MB and CPU to 1.6%.

Lessons Learned

Measure before you optimize. Without building a benchmark, we would have been guessing. Several times, an “optimization” turned out to make things worse, and the benchmarks caught it immediately.
Caching is the biggest lever. Most TUI components don’t change between frames, and detecting that is cheap.
Allocations compound. No single allocation was the problem. But when every component allocates on every frame, and you’re rendering 60 times per second with dozens of log outputs, it adds up to gigabytes. Eliminating allocations one type at a time: pooled buffers, callback iteration, pre-computed values, eventually brought total allocations from 17.6 GB down to 44 MB.
Sometimes you need to build your own. Bubble Tea is a good framework, but it wasn’t designed for the kind of high-frequency, incremental rendering that Panix needed. Building a custom engine was a significant investment, but more than 200x performance improvement justified it.
Cache invalidation is hard. One mistake, and cache won’t work, or may produce visual artifacts due to being stale or improperly reused.