feat: implement in-house grapheme cluster-aware rendering system (#103)
* feat: implement in-house grapheme cluster-aware rendering system
Replace rune-based text iteration and width measurement with a proper
grapheme cluster model throughout the rendering pipeline.
Core changes:
- New grapheme.go: in-house cluster state machine (zero external deps)
- nextCluster/nextClusterBytes iterate by grapheme cluster
- ClusterAdvance shared state machine handles: combining marks (Mn/Me/Mc),
ZWJ sequences, regional-indicator pairs, variation selectors (VS16),
and emoji skin-tone modifiers
- helper functions: clusterCount, clusterRuneStarts,
clusterEndAfterInsert, snapRuneToClusterStart, runeIndexToDisplayCol
- Cell struct: new Combining string field for multi-rune clusters
- newClusterCell() stores base rune inline, clones remaining runes
- cellGlyph() reconstructs full glyph from Rune+Combining
- Buffer: setCluster method replaces setRune internals; SetString,
SetStringGradient, String, StringTrimmed iterate by cluster
- stringWidth measures by cluster width instead of per-rune sum
- wrapText/wrapSpans: hard word breaks iterate by cluster
- border titles: draw by cluster
- element rendering: renderClippedElement, truncateText,
renderTextContent iterate by cluster
- ANSI output: render_element.go, terminal_ansi.go, mock_terminal.go
write c.Combining after c.Rune
- stream_writer: WriteStyled/WriteGradient track columns by cluster
- inline scanner: consumes full grapheme clusters
- Input/TextArea: cursor movement snaps to cluster boundaries
(insertChar, backspace, delete, moveLeft/Right all cluster-aware)
Fixes: multi-rune emoji (flags, ZWJ families, skin tones), decomposed
accented characters, keycap sequences now render correctly as single
glyphs with proper width. Word wrapping never breaks inside a cluster.
Tests: new grapheme_test.go (segmentation, width, byte scanner sync,
rune starts), new grapheme_render_test.go (buffer, gradient, wrap,
render tree, cursor tests), new cell_glyph_test.go (test helper)
* fix: write Combining in MockTerminal String/StringTrimmed methods
MockTerminal's output methods were missing the Combining field when
reconstructing cell glyphs, which would silently drop multi-rune cluster
content in tests that use MockTerminal for assertions.
* fix: make Input scrolling column-based and cluster-aware
Change scrollPos from storing a rune index to storing a display column
so that CJK/emoji (width 2) and multi-rune grapheme clusters are
correctly scrolled.
- scrollPos is now a display column, not a rune index.
- ensureCursorVisible converts cursor rune index to display column via
runeIndexToDisplayCol and compares against scrollPos (columns).
- displayText segments text into grapheme clusters, then selects the
visible slice starting at the scroll column and filling the viewport.
The cursor is inserted as a 1-column cluster at the correct visual
position among the cluster list.
- textToClusters helper segments a string into []displayCluster.
- viewportText slices the cluster list by scroll column and visible
width, returning the visible text string.
- backspace and delete now rely on ensureCursorVisible for scroll
math instead of ad-hoc rune-index arithmetic.
* fix: correct runeIndexToDisplayCol for multi-rune clusters
The function was returning column after a multi-rune cluster when the
rune index fell inside it (e.g. inside a ZWJ family emoji or decomposed
accent). Added proper snap-to-cluster-start logic: if the cluster has
multiple runes and the rune index is inside it (not at its boundary),
return the cluster's starting column. For single-rune clusters the
boundary check is correct because clusterRunes == 1 means runeAt+1
crossing runeIdx is the boundary, not the interior.
* test: add comprehensive edge case tests for grapheme cluster system
New test file grapheme_edge_test.go covers:
- nextCluster edge cases: empty, single/two ascii, ascii+emoji, CJK,
ZWJ sequences, lone RI, leading combining mark, ZWNJ between chars,
skin tone modifier, VS15 text presentation selector
- nextClusterBytes edge cases: nil, single byte, two bytes, flag cluster
- clusterCount: empty, ascii, flag, ZWJ family, mixed, accent, CJK
- buffer.SetString overlap: overwriting continuation cells
- buffer.SetStringGradient with mixed-width clusters
- Input combining mark insert: cursor lands after combined cluster
- Input ZWJ sequence insert: cursor after the joined cluster
- buffer String/StringTrimmed: round-trip preserving multi-rune clusters
- viewportText: starting from cluster boundaries and beyond-all
- segmentStyledRunes: multi-rune cluster takes base rune's style;
flag across styled spans
- newClusterCell: empty string, width 0, single CJK
- snapRuneToClusterStart: inside/across multi-rune clusters
- runeIndexToDisplayCol: all cluster types and positions including
inside multi-rune clusters
* test: update test buffer read-back helpers to emit Combining field
Backport from experiment/cell-rune-combining branch. Test helper
functions that reconstruct cell content from the buffer now write
c.Combining after c.Rune so multi-rune grapheme clusters (flags, ZWJ
families, decomposed accents) are not silently truncated in test
assertions.
Also fix buffer_test.go byte-index bug: rune(tt.text[0]) takes the
first byte of a multi-byte string instead of the first rune. Changed
to []rune(tt.text)[0].
Files:
- buffer_test.go: multi-byte safe first-rune assertion
- element_integration_render_test.go: extractBufferLine + row builder
- mount_layout_test.go: extractBufferText
- textarea_test.go: renderedRows helper
* style: gofmt fix for mount_layout_test.go
* fix: optimize cluster helpers to O(pos) and rename clusterEndAfterInsert
Addresses PR review feedback:
1. Rename clusterEndAfterInsert → clusterEnd: The old name implied
insert semantics, but the function is also used in delete/move
handlers where no insertion occurs. The new name is unambiguous.
2. Rewrite clusterEnd and snapRuneToClusterStart to scan clusters
incrementally instead of building the full clusterRuneStarts array
(O(N)). They now stop as soon as they pass the target rune index,
making them O(pos) instead of O(N). For a cursor near the start
this is essentially O(1); for cursor-at-end it's the same O(N) as
before but with half the allocations (no []int slice).
3. runeIndexToDisplayCol was already O(pos) and unchanged.
* fix: make wrapInlineStyledRows grapheme-cluster-aware
The ANSI-styled inline wrapping path was still processing text rune-by-rune.
This meant multi-rune grapheme clusters (flags, ZWJ families, decomposed
accents) in PrintAboveStyled output were:
- Measured as the sum of their code points (e.g. a flag was 4 columns instead of 2)
- Potentially split across line breaks mid-cluster
Fix: extract a nextClusterWidth helper that consumes a full grapheme cluster
starting from a byte position, skipping over ANSI escape sequences (which
are transparent to grapheme boundaries). The main loop now writes the full
cluster text as one unit.
Note: ANSI sequences between the base rune and trailing combining marks are
treated as cluster boundaries (the style change breaks the cluster). This is
an extremely rare edge case (a style reset between an emoji and its skin tone
modifier) and the behavior is safer than crossing an ANSI boundary.
wrapInlineVisualRows (the non-ANSI path) was already cluster-aware.
* fix: remove double base-rune decode in wrapInlineStyledRows
The previous fix made wrapInlineStyledRows cluster-aware by adding a
nextClusterWidth helper, but the outer loop was still pre-decoding the
base rune and advancing i before calling nextClusterWidth. This caused
nextClusterWidth to consume the *next* character as its base, pairing
characters together and halving the width tracking.
Fix: remove the pre-decode from the outer loop. nextClusterWidth now
handles the full decode starting from the current position, including
newline detection (returning 0 so the caller can flush).
* fix: clusterEnd guard off-by-one and truncateText O(n²) string concat
Issue 1: clusterEnd used runeAt >= clusterStartRuneIdx which only fires
after runeAt has been advanced past the cluster. For a non-boundary
position inside a multi-rune cluster (e.g. insertChar after inserting
a combining mark), this caused the function to skip the containing
cluster entirely and return the end of the next cluster instead.
Fix: use runeAt+clusterRunes > clusterStartRuneIdx (same guard as
snapRuneToClusterStart), checked before advancing runeAt.
Issue 2: truncateText used truncated += cluster in the loop, causing
O(n²) string allocations. Replaced with strings.Builder.
* fix: wrapInlineStyledRows ? overflow and fix ZWJ test
Issue 1: When a cluster was wider than the entire line width, the code
wrote '?' into the row before checking if the row was already full.
If col == width (exactly full), this produced a row one char too wide.
Fix: normalize cw=1 and run the standard col+cw > width pre-flush check
before writing '?', matching the old code's pattern and the non-ANSI path.
Issue 2: TestInput_ZWJInsert had want = "a\u200d\U0001F469" which
was missing the man emoji (U+1F468), and the want variable was discarded
with _ = want. The test only checked for the presence of each character
but not their order or full content. Fix: correct the expected string to
"a\U0001F468\u200D\U0001F469" and assert with a direct comparison.
* fix: nextClusterWidth ZWJ init, clusterEnd doc, add wrapInlineStyledRows tests
Issue 3: nextClusterWidth initialized lastWasZWJ from the base rune
(isZWJ(r)), diverging from clusterAdvance which always starts
lastWasZWJ = false. For degenerate input with ZWJ as the first code
point, the two functions would disagree on cluster width. Fix:
initialize lastWasZWJ = false to match clusterAdvance.
Issue 2: clusterEnd docstring said 'must be at a cluster boundary'
but insertChar calls it with a position inside a multi-rune cluster
after inserting a combining mark. Fix: docstring now correctly
describes both cases.
Issue 4: Added tests for wrapInlineStyledRows covering:
- Plain ASCII (no ANSI)
- ANSI sequences around text
- Multi-rune clusters (flags, CJK) with and without ANSI
- Oversized cluster replacement with '?' at various widths
* refactor: move clusterRuneStarts to test file, document nextClusterWidth
Issue 1: clusterRuneStarts was defined in production code (grapheme.go)
but only called from tests. Moved to grapheme_test.go as a test helper.
The callers that previously used it (snapRuneToClusterStart, clusterEnd)
were rewritten to use O(pos) incremental walkers in a prior commit.
Issue 2: nextClusterWidth duplicates the clusterAdvance extension loop
logic (combining marks, ZWJ, RI pairs) with the addition of ANSI-break
handling. Extracting a shared helper is impractical due to the tight
coupling with ANSI scanning. Added a doc comment directing maintainers
to update both functions if extension rules change.
* fix: scroll boundary bisecting wide clusters loses cursor
When ensureCursorVisible computed scroll = cursorCol - visible + 1, the
result could land inside a 2-column cluster (e.g. CJK at col 1). The
old viewportText snapped to that cluster's left boundary, consuming an
extra column and pushing the cursor out of the visible window.
Two changes:
1. ensureCursorVisible now snaps the candidate scroll column up to the
nearest cluster-boundary column via snapColToClusterStart, so the
viewport always starts on a valid cluster boundary.
2. viewportText's startIdx loop now skips clusters whose column range
straddles scroll (instead of including them). If scroll falls inside
a cluster, that cluster is treated as scrolled off the left edge.
* style: gofmt fix for grapheme_test.go
* fix: flaky TestWatch_ExitsWhenStopChCloses timing race
The test sent 'second' to the watcher's input channel after closing
stopCh. Depending on goroutine scheduling, the watcher could pick up
'second' from the channel before the stopCh case was selected.
Fix: close both channels (stopCh and the watcher's ch). The goroutine
exits via either <-stopCh or the closed channel's ok=false, and no
more values can arrive on ch.
* fix: inline comments, typo, and dead parameter in viewportText
Issue 1: Inline comments in nextClusterWidth said ANSI sequences are
'skipped' and 'transparent to grapheme boundaries', but the code
breaks on them (treating them as boundaries). Fixed both comments to
accurately describe the break behavior.
Issue 2: 'AN SI' typo fixed to 'ANSI'.
Issue 3: viewportText declared _ int as its second parameter (cursor
rune-index), but never used it. All scroll logic is driven by
ensureCursorVisible + scrollPos. Removed the dead parameter from
viewportText and both call sites.
* refactor: simplify runeIndexToDisplayCol, deduplicate text.Get() calls
Issue 1: The inner if in runeIndexToDisplayCol checked clusterRunes > 1
and runeAt+clusterRunes >= runeIdx && runeAt < runeIdx to return col
early, but both branches (return col and break + loop exit) produced
the same value. Simplified to a single break with a clearer comment.
Issue 2: input.go clampCursorPos called inp.text.Get() twice (once for
max and once for snapRuneToClusterStart). Cached to local variable.
Issue 3: Same double t.text.Get() in textarea.go clampCursorPos.
* fix: VS15 handling, comment clarity, extract shared clusterExtendUpdateWidth
Issue 1: VS15 (U+FE0E, text-presentation selector) was not handled.
For emoji bases in emojiWideRanges, VS15 should narrow the cluster
width to 1. Added isVS15 check in clusterAdvance, and extracted
clusterExtendUpdateWidth as a shared helper used by both clusterAdvance
and nextClusterWidth so they stay in sync.
Issue 2: 'Fall through so the normal pre-flush check runs' — misleading
in Go since the block ends with 'continue'. Reworded to describe the
actual behavior.
Issue 3: nextClusterWidth duplicated clusterAdvance's extension logic.
Extracted clusterExtendUpdateWidth, eliminating the duplicate VS16/VS15
checks and ensuring both paths agree on width decisions.
* feat: export NextCluster, ClusterCount, StringWidth
Add public wrappers for the three grapheme utility functions so
external consumers can use them:
- NextCluster(s) — iterate grapheme clusters
- ClusterCount(s) — count user-perceived characters
- StringWidth(s) — measure display width in terminal cells
* style: gofumpt formatting — blank line between one-line wrapper and function
* feat: separate ClusterCount from internal, add ClusterRuneCount
ClusterCount now has its own implementation wrapping NextCluster
(not delegating to clusterCount). The internal clusterCount delegates
to ClusterCount.
ClusterRuneCount counts total Unicode code points by walking clusters
via NextCluster and summing each cluster's rune count. Useful for
callers that need both cluster counts and rune counts in sync.
* revert: remove ClusterRuneCount — redundant with utf8.RuneCountInString
ClusterRuneCount walked clusters and summed rune counts, but produces
the same result as utf8.RuneCountInString at higher cost. External
callers can use utf8.RuneCountInString for rune counts and ClusterCount
for cluster counts.
* feat: add ClusterRuneCount — convenience wrapper avoiding utf8 import
External consumers that already import this package for grapheme
functionality can use ClusterRuneCount instead of importing
unicode/utf8 separately for RuneCountInString. It walks clusters
via NextCluster and sums each cluster's rune count.
* feat: add NextClusterRunes — cluster iterator with rune count
Returns (cluster, width, size, runes) in one call, avoiding a separate
utf8.RuneCountInString for callers that need both the cluster text and
its rune length.
* fix: use baseRuneWidth in nextClusterWidth to match clusterAdvance
nextClusterWidth used max(RuneWidth(r), 1) while clusterAdvance uses
baseRuneWidth(r). For current wide-range tables both return identical
values for non-zero-width runes, but the inconsistency means the
shared clusterExtendUpdateWidth helper's VS15 check (which uses
baseRuneWidth to decide whether to narrow) could disagree with the
initial width in nextClusterWidth. Using baseRuneWidth keeps both
paths aligned with the same source of truth.
* fix: expand emojiWideRanges to cover BMP emoji (0x00A9-0x3299)
The emojiWideRanges list started at 0x1F004 and missed all BMP emoji
in the 0x2000-0x27BF range (Miscellaneous Symbols, Dingbats, etc.).
Characters like ⭐ (U+2B50), ☕ (U+2615), ✨ (U+2728), ❤ (U+2764) were
assigned width 1 instead of 2, causing text wrapping to calculate wrong
break positions and content to overflow past borders.
Added 27 new ranges covering all Extended_Pictographic code points
below 0x1F000 based on the Unicode emoji data that runeseg references.
The largest single addition is 0x2600-0x27BF covering misc symbols and
dingbats. Added regression tests for the specific emoji from the issue.
* fix: remove 4 redundant emojiWideRanges entries already covered by eastAsianWideRanges
U+3030, U+303D (within eastAsian 0x2E80-0x303E), U+3297, U+3299
(within eastAsian 0x3040-0xA4CF) are already width 2 via the
eastAsianWideRanges check in RuneWidth. Having them in emojiWideRanges
too is redundant and makes the list harder to audit.
* fix: replace over-broad 0x2600-0x27BF with precise Extended_Pictographic ranges
The previous fix used a single range 0x2600-0x27BF covering the entire
Miscellaneous Symbols + Dingbats block, which incorrectly assigned
width 2 to non-emoji characters like ★ (U+2605), ☏ (U+260F), etc.
Replaced with the precise 60+ sub-ranges from Unicode's
Extended_Pictographic property (verified against runeseg's data).
Also restored 0x2934-0x2935 and 0x2B05-0x2B55 ranges that were
accidentally dropped.
Verified: ✨⭐☕✅❤☀⚡ all width 2; non-emoji like ★☏☐ all width 1.
* test: add RuneWidth emoji range validation test
TestRuneWidth_EmojiRangeValidation verifies:
- 138 specific emoji code points return width 2
- 100+ non-emoji characters in the same Unicode blocks return width 1
- No overlaps between emojiWideRanges and eastAsianWideRanges
This catches regressions where a range is too broad (flagging non-emoji
as wide) or too narrow (missing actual emoji), without relying on
external references like runeseg for truth.
* style: gofumpt formatting — align comments in emojiWideRanges
* style: gofumpt formatting for cell_test.go
* fix: address 3 review comments — runeVal doc, snapCol comments, fallback comment
Issue 1: runeVal field comment now clarifies it stores the base rune
only; callers should use bytes() for the full cluster glyph.
Issue 2: snapColToClusterStart function doc and call-site comment used
contradictory 'snaps up' / 'snap down' language. Made both neutral
('advances to the next cluster boundary at or after col').
Issue 3: segmentStyledRunes fallback base := rs[0] is unreachable but
misleading to debuggers. Added comment explaining why it's safe.
* style: gofumpt formatting for inline_scan.go
* fix: restrict emojiWideRanges to Emoji_Presentation=Yes code points
BMP emoji with Emoji_Presentation=No (text-default) like ✔, ✨, ❤, ☀
are width 1 without VS16. Our previous expanded ranges assigned them
width 2 unconditionally, which caused a 1-column offset on modern
terminals like Ghostty and Kitty that follow the Unicode spec.
Now only includes:
- Emoji_Presentation=Yes BMP emoji (26 specific ranges from Unicode data)
- All SMP emoji (U+1Fxxx — all emoji-default)
BMP emoji that are text-default get width 1 from baseRuneWidth and
width 2 only when clusterAdvance detects VS16 (U+FE0F) via the
clusterExtendUpdateWidth helper.
* fix: align emojiWideRanges exactly with Unicode Emoji_Presentation=Yes
Split over-broad range 23E9-23F3 into precise sub-ranges matching
the Unicode spec: only 23E9-23EC, 23F0, 23F3 have Emoji_Presentation=Yes
(23ED-23EE, 23EF, 23F1-23F2 are Extended_Pictographic but text-default).
Added missing Emoji_Presentation=Yes entries discovered during cross-check:
- 2705 (✅ check mark button)
- 2728 (✨ sparkles)
- 2753-2755 (❓❔❕ marks)
- 2795-2797 (➕➖➗ math)
- 27B0 (➰ curly loop)
- 2B1B-2B1C (⬛⬜ squares)
Updated tests to verify the corrected ranges.
* style: gofumpt formatting for cell_test.go
* fix: move ensureCursorVisible out of viewportText, rename snapCol
Issue 1: viewportText called ensureCursorVisible on every invocation,
including from the unfocused displayText path. This meant scrollPos
was adjusted to track the cursor even when the cursor glyph isn't
shown. Fixed by moving ensureCursorVisible into the focused call site
in displayText. The unfocused path preserves the previously-set scroll
position.
Issue 2: snapColToClusterStart snapped forward (to the next cluster
boundary), not backward (to the current cluster's start). Renamed to
snapColToNextClusterBoundary to accurately describe the behavior.
* fix: nextClusterWidth newline sentinel, newClusterCell doc
Issue 1: nextClusterWidth returned 0 for both EOF and newline, but
left *pos unchanged for newlines, requiring the caller to peek at
text[i] to distinguish them. Changed to advance *pos past the newline
so the caller can unconditionally flush on cw == 0 without peeking.
Issue 2: newClusterCell's width==0 guard silently drops text, but
RuneWidth never returns 0 for printable characters. Added doc comment
clarifying that width==0 is only for explicit continuation-cell writes.
* fix: remove duplicated doc comment in clusterEnd body
* fix: cursorRowCol display-column arithmetic, comment clarity, string pinning
Issue 1: cursorRowCol and posFromRowCol used rune-count column
arithmetic (currentCol++ per rune), but wrapText now wraps on display
columns. For multi-rune clusters (e.g. ZWJ family: 7 runes, 2 cols),
the threshold comparison currentCol > utf8.RuneCountInString(lines[...])
was off by the cluster's rune/display ratio. Rewrote both functions to
iterate by grapheme cluster via NextClusterRunes, tracking display
columns and comparing against stringWidth for wrap detection.
Issue 2: Updated ASCII fast path comment to clarify that continuation
bytes (0x80-0xBF) also force the full decode path, and that malformed
UTF-8 (ASCII + raw continuation byte) is handled correctly.
Issue 3: segmentStyledRunes returned cluster text as sub-slices of the
per-word concatenated string, pinning it alive. Added strings.Clone
on the cluster text, matching the pattern newClusterCell already uses.
* fix: cursorRowCol returns rune index, not display column
cursorRowCol was changed to return a display-column col value, but all
callers (lineWithCursor, moveUp, moveDown, moveEnd) use col as a rune
index. This broke wide-character navigation:
- lineWithCursor: runes[col] with col=2 for '你好' (2 runes) panicked
- moveUp/moveDown: clamping display-col against rune-count was wrong
- moveEnd: targetCol=runeCount passed to posFromRowCol was wrong
Fix: cursorRowCol tracks both displayCol (for wrap boundary detection)
and currentRuneCol (returned as col, a rune index for callers).
posFromRowCol similarly tracks displayCol for wrap detection but
matches targetCol (a rune index) against currentCol (rune index).
* style: gofumpt formatting
* fix: block cursor at hard-newline line end no longer splits clusters
When a display-full line ends with a hard newline, the block cursor
overlays the last cell using . For a multi-rune
grapheme cluster (flag, ZWJ family), this splits the cluster in half.
Replaced with which snaps
back to the cluster boundary, preserving cluster integrity.
Adds regression test with US flag emoji (2 rune cluster) at a
display-full line end with block cursor overlay.ST33L · 23d · 2a4aa99 · parent 4418221 · 26 files +1929 -4