## Problem
The `ansi`, `base16`, and `base16-256` syntax themes are designed to
emit ANSI palette colors so that highlighted code respects the user's
terminal color scheme. Syntect encodes this intent in the alpha channel
of its `Color` struct — a convention shared with `bat` — but
`convert_style` was ignoring it entirely, treating every foreground
color as raw RGB. This caused ANSI-family themes to produce hard-coded
RGB values (e.g. `Rgb(0x02, 0, 0)` instead of `Green`), defeating their
purpose and rendering them as near-invisible dark colors on most
terminals.
Reported in #12890.
## Mental model
Syntect themes use a compact encoding in their `Color` struct:
| `alpha` | Meaning of `r` | Mapped to |
|---------|----------------|-----------|
| `0x00` | ANSI palette index (0–255) | `RtColor::Black`…`Gray` for 0–7,
`Indexed(n)` for 8–255 |
| `0x01` | Unused (sentinel) | `None` — inherit terminal default fg/bg |
| `0xFF` | True RGB red channel | `RtColor::Rgb(r, g, b)` |
| other | Unexpected | `RtColor::Rgb(r, g, b)` (silent fallback) |
This encoding is a bat convention that three bundled themes rely on. The
new `convert_syntect_color` function decodes it; `ansi_palette_color`
maps indices 0–7 to ratatui's named ANSI variants.
| macOS - Dark | macOS - Light | Windows - ansi | Windows - base16 |
|---|---|---|---|
| <img width="1064" height="1205" alt="macos-dark"
src="https://github.com/user-attachments/assets/f03d92fb-b44b-4939-b2b9-503fde133811"
/> | <img width="1073" height="1227" alt="macos-light"
src="https://github.com/user-attachments/assets/2ecb2089-73b5-4676-bed8-e4e6794250b4"
/> |
|
|
## Non-goals
- Background color decoding — we intentionally skip backgrounds to
preserve the terminal's own background. The decoder supports it, but
`convert_style` does not apply it.
- Italic/underline changes — those remain suppressed as before.
- Custom `.tmTheme` support for ANSI encoding — only the bundled themes
use this convention.
## Tradeoffs
- The alpha-channel encoding is an undocumented bat/syntect convention,
not a formal spec. We match bat's behavior exactly, trading formality
for ecosystem compatibility.
- Indices 0–7 are mapped to ratatui's named variants (`Black`, `Red`, …,
`Gray`) rather than `Indexed(0)`…`Indexed(7)`. This lets terminals apply
bold/bright semantics to named colors, which is the expected behavior
for ANSI themes, but means the two representations are not perfectly
round-trippable.
## Architecture
All changes are in `codex-rs/tui/src/render/highlight.rs`, within the
style-conversion layer between syntect and ratatui:
```
syntect::highlighting::Color
└─ convert_syntect_color(color) [NEW — alpha-dispatch]
├─ a=0x00 → ansi_palette_color() [NEW — index→named/indexed]
├─ a=0x01 → None (terminal default)
├─ a=0xFF → Rgb(r,g,b) (standard opaque path)
└─ other → Rgb(r,g,b) (silent fallback)
```
`convert_style` delegates foreground mapping to `convert_syntect_color`
instead of inlining the `Rgb(r,g,b)` conversion. The core highlighter is
refactored into `highlight_to_line_spans_with_theme` (accepts an
explicit theme reference) so tests can highlight against specific themes
without mutating process-global state.
### ANSI-family theme contract
The ANSI-family themes (`ansi`, `base16`, `base16-256`) rely on upstream
alpha-channel encoding from two_face/syntect. We intentionally do
**not** validate this contract at runtime — if the upstream format
changes, the `ansi_themes_use_only_ansi_palette_colors` test catches it
at build time, long before it reaches users. A runtime warning would be
unactionable noise.
### Warning copy cleanup
User-facing warning messages were rewritten for clarity:
- Removed internal jargon ("alpha-encoded ANSI color markers", "RGB
fallback semantics", "persisted override config")
- Dropped "syntax" prefix from "syntax theme" — users just think "theme"
- Downgraded developer-only diagnostics (duplicate override, resolve
fallback) from `warn` to `debug`
## Observability
- The `ansi_themes_use_only_ansi_palette_colors` test enforces the
ANSI-family contract at build time.
- The snapshot test provides a regression tripwire for palette color
output.
- User-facing warnings are limited to actionable issues: unknown theme
names and invalid custom `.tmTheme` files.
## Tests
- **Unit tests for each alpha branch:** `alpha=0x00` with low index
(named color), `alpha=0x00` with high index (`Indexed`), `alpha=0x01`
(terminal default), unexpected alpha (falls back to RGB), ANSI white →
Gray mapping.
- **Integration test:**
`ansi_family_themes_use_terminal_palette_colors_not_rgb` — highlights a
Rust snippet with each ANSI-family theme and asserts zero `Rgb`
foreground colors appear.
- **Snapshot test:** `ansi_family_foreground_palette_snapshot` — records
the exact set of unique foreground colors each ANSI-family theme
produces, guarding against regressions.
- **Warning validation tests:** verify user-facing warnings for missing
custom themes, invalid `.tmTheme` files, and bundled theme resolution.
## Test plan
- [ ] `cargo test -p codex-tui` passes all new and existing tests
- [ ] Select `ansi`, `base16`, or `base16-256` theme and verify code
blocks render with terminal palette colors (not near-black RGB)
- [ ] Select a standard RGB theme (e.g. `dracula`) and verify no
regression in color output
## Problem
The TUI diff renderer uses hardcoded background palettes for
insert/delete lines that don't respect the user's chosen syntax theme.
When a theme defines `markup.inserted` / `markup.deleted` scope
backgrounds (the convention used by GitHub, Solarized, Monokai, and most
VS Code themes), those colors are ignored — the diff always renders with
the same green/red tints regardless of theme selection.
Separately, ANSI-16 terminals (and Windows Terminal sessions misreported
as ANSI-16) rendered diff backgrounds as full-saturation blocks that
obliterated syntax token colors, making highlighted diffs unreadable.
## Mental model
Diff backgrounds are resolved in three layers:
1. **Color level detection** — `diff_color_level_for_terminal()` maps
the raw `supports-color` probe + Windows Terminal heuristics to a
`DiffColorLevel` (TrueColor / Ansi256 / Ansi16). Windows Terminal gets
promoted from Ansi16 to TrueColor when `WT_SESSION` is present.
2. **Background resolution** — `resolve_diff_backgrounds()` queries the
active syntax theme for `markup.inserted`/`markup.deleted` (falling back
to `diff.inserted`/`diff.deleted`), then overlays those on top of the
hardcoded palette. For ANSI-256, theme RGB values are quantized to the
nearest xterm-256 index. For ANSI-16, backgrounds are `None`
(foreground-only).
3. **Style composition** — The resolved `ResolvedDiffBackgrounds` is
threaded through every call to `style_add`, `style_del`, `style_sign_*`,
and `style_line_bg_for`, which decide how to compose
foreground+background for each line kind and theme variant.
A new `RichDiffColorLevel` type (a subset of `DiffColorLevel` without
Ansi16) encodes the invariant "we have enough depth for tinted
backgrounds" at the type level, so background-producing functions have
exhaustive matches without unreachable arms.
## Non-goals
- No change to gutter (line number column) styling — gutter backgrounds
still use the hardcoded palette.
- No per-token scope background resolution — this is line-level
background only; syntax token colors come from the existing
`highlight_code_to_styled_spans` path.
- No dark/light theme auto-switching from scope backgrounds —
`DiffTheme` is still determined by querying the terminal's background
color.
## Tradeoffs
- **Theme trust vs. visual safety:** When a theme defines scope
backgrounds, we trust them unconditionally for rich color levels. A
badly authored theme could produce illegible combinations. The fallback
for `None` backgrounds (foreground-only) is intentionally conservative.
- **Quantization quality:** ANSI-256 quantization uses perceptual
distance across indices 16–255, skipping system colors. The result is
approximate — a subtle theme tint may land on a noticeably different
xterm index.
- **Single-query caching:** `resolve_diff_backgrounds` is called once
per `render_change` invocation (i.e., once per file in a diff). If the
theme changes mid-render (live preview), the next file picks up the new
backgrounds.
## Architecture
Files changed:
| File | Role |
|---|---|
| `tui/src/render/highlight.rs` | New: `DiffScopeBackgroundRgbs`,
`diff_scope_background_rgbs()`, scope extraction helpers |
| `tui/src/diff_render.rs` | New: `RichDiffColorLevel`,
`ResolvedDiffBackgrounds`, `resolve_diff_backgrounds*`,
`quantize_rgb_to_ansi256`, Windows Terminal promotion; modified: all
style helpers to accept/thread `ResolvedDiffBackgrounds` |
The scope-extraction code lives in `highlight.rs` because it uses
`syntect::highlighting::Highlighter` and the theme singleton. The
resolution and quantization logic lives in `diff_render.rs` because it
depends on diff-specific types (`DiffTheme`, `DiffColorLevel`, ratatui
`Color`).
## Observability
No runtime logging was added. The most useful debugging aid is the
`diff_color_level_for_terminal` function, which is pure and fully
unit-tested — to diagnose a color-depth mismatch, log its four inputs
(`StdoutColorLevel`, `TerminalName`, `WT_SESSION` presence,
`FORCE_COLOR` presence).
Scope resolution can be tested by loading a custom `.tmTheme` with known
`markup.inserted` / `markup.deleted` backgrounds and checking the diff
output in a truecolor terminal.
## Tests
- **Windows Terminal promotion:** 7 unit tests cover every branch of
`diff_color_level_for_terminal` (ANSI-16 promotion, `WT_SESSION`
unconditional promotion, `FORCE_COLOR` suppression, conservative
`Unknown` level).
- **ANSI-16 foreground-only:** Tests verify that `style_add`,
`style_del`, `style_sign_*`, `style_line_bg_for`, and `style_gutter_for`
all return `None` backgrounds on ANSI-16.
- **Scope resolution:** Tests verify `markup.*` preference over
`diff.*`, `None` when no scope matches, bundled theme resolution, and
custom `.tmTheme` round-trip.
- **Quantization:** Test verifies ANSI-256 quantization of a known RGB
triple.
- **Insta snapshots:** 2 new snapshot tests
(`ansi16_insert_delete_no_background`,
`theme_scope_background_resolution`) lock visual output.
Summary
- map csharp/c-sharp aliases to the existing C# syntax in the highlight
matcher
- ensure the extension list and tests include .cs and the new aliases so
coverage stays accurate
Testing
<img width="543" height="266" alt="image"
src="https://github.com/user-attachments/assets/e6c8a42f-649c-4c30-b574-421b4287534c"
/>
## Summary
- order bundled and custom themes together by name while keeping entries
stable across platforms
- update the theme fixture names and tests to assert case-insensitive
ordering
## Summary
Adds syntax highlighting to the TUI for fenced code blocks in markdown
responses and file diffs, plus a `/theme` command with live preview and
persistent theme selection. Uses syntect (~250 grammars, 32 bundled
themes, ~1 MB binary cost) — the same engine behind `bat`, `delta`, and
`xi-editor`. Includes guardrails for large inputs, graceful fallback to
plain text, and SSH-aware clipboard integration for the `/copy` command.
<img width="1554" height="1014" alt="image"
src="https://github.com/user-attachments/assets/38737a79-8717-4715-b857-94cf1ba59b85"
/>
<img width="2354" height="1374" alt="image"
src="https://github.com/user-attachments/assets/25d30a00-c487-4af8-9cb6-63b0695a4be7"
/>
## Problem
Code blocks in the TUI (markdown responses and file diffs) render
without syntax highlighting, making it hard to scan code at a glance.
Users also have no way to pick a color theme that matches their terminal
aesthetic.
## Mental model
The highlighting system has three layers:
1. **Syntax engine** (`render::highlight`) -- a thin wrapper around
syntect + two-face. It owns a process-global `SyntaxSet` (~250 grammars)
and a `RwLock<Theme>` that can be swapped at runtime. All public entry
points accept `(code, lang)` and return ratatui `Span`/`Line` vectors or
`None` when the language is unrecognized or the input exceeds safety
guardrails.
2. **Rendering consumers** -- `markdown_render` feeds fenced code blocks
through the engine; `diff_render` highlights Add/Delete content as a
whole file and Update hunks per-hunk (preserving parser state across
hunk lines). Both callers fall back to plain unstyled text when the
engine returns `None`.
3. **Theme lifecycle** -- at startup the config's `tui.theme` is
resolved to a syntect `Theme` via `set_theme_override`. At runtime the
`/theme` picker calls `set_syntax_theme` to swap themes live; on cancel
it restores the snapshot taken at open. On confirm it persists `[tui]
theme = "..."` to config.toml.
## Non-goals
- Inline diff highlighting (word-level change detection within a line).
- Semantic / LSP-backed highlighting.
- Theme authoring tooling; users supply standard `.tmTheme` files.
## Tradeoffs
| Decision | Upside | Downside |
| ------------------------------------------------ |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-----------------------------------------------------------------------------------------------------------------------
|
| syntect over tree-sitter / arborium | ~1 MB binary increase for ~250
grammars + 32 themes; battle-tested crate powering widely-used tools
(`bat`, `delta`, `xi-editor`). tree-sitter would add ~12 MB for 20-30
languages or ~35 MB for full coverage. | Regex-based; less structurally
accurate than tree-sitter for some languages (e.g. language injections
like JS-in-HTML). |
| Global `RwLock<Theme>` | Enables live `/theme` preview without
threading Theme through every call site | Lock contention risk
(mitigated: reads vastly outnumber writes, single UI thread) |
| Skip background / italic / underline from themes | Terminal BG
preserved, avoids ugly rendering on some themes | Themes that rely on
these properties lose fidelity |
| Guardrails: 512 KB / 10k lines | Prevents pathological stalls on huge
diffs or pastes | Very large files render without color |
## Architecture
```
config.toml ─[tui.theme]─> set_theme_override() ─> THEME (RwLock)
│
┌───────────────────────────────────────────┘
│
markdown_render ─── highlight_code_to_lines(code, lang) ─> Vec<Line>
diff_render ─── highlight_code_to_styled_spans(code, lang) ─> Option<Vec<Vec<Span>>>
│
│ (None ⇒ plain text fallback)
│
/theme picker ─── set_syntax_theme(theme) // live preview swap
─── current_syntax_theme() // snapshot for cancel
─── resolve_theme_by_name(name) // lookup by kebab-case
```
Key files:
- `tui/src/render/highlight.rs` -- engine, theme management, guardrails
- `tui/src/diff_render.rs` -- syntax-aware diff line wrapping
- `tui/src/theme_picker.rs` -- `/theme` command builder
- `tui/src/bottom_pane/list_selection_view.rs` -- side content panel,
callbacks
- `core/src/config/types.rs` -- `Tui::theme` field
- `core/src/config/edit.rs` -- `syntax_theme_edit()` helper
## Observability
- `tracing::warn` when a configured theme name cannot be resolved.
- `Config::startup_warnings` surfaces the same message as a TUI banner.
- `tracing::error` when persisting theme selection fails.
## Tests
- Unit tests in `highlight.rs`: language coverage, fallback behavior,
CRLF stripping, style conversion, guardrail enforcement, theme name
mapping exhaustiveness.
- Unit tests in `diff_render.rs`: snapshot gallery at multiple terminal
sizes (80x24, 94x35, 120x40), syntax-highlighted wrapping, large-diff
guardrail, rename-to-different-extension highlighting, parser state
preservation across hunk lines.
- Unit tests in `theme_picker.rs`: preview rendering (wide + narrow),
dim overlay on deletions, subtitle truncation, cancel-restore, fallback
for unavailable configured theme.
- Unit tests in `list_selection_view.rs`: side layout geometry, stacked
fallback, buffer clearing, cancel/selection-changed callbacks.
- Integration test in `lib.rs`: theme warning uses the final
(post-resume) config.
## Cargo Deny: Unmaintained Dependency Exceptions
This PR adds two `cargo deny` advisory exceptions for transitive
dependencies pulled in by `syntect v5.3.0`:
| Advisory | Crate | Status |
|----------|-------|--------|
| RUSTSEC-2024-0320 | `yaml-rust` | Unmaintained (maintainer
unreachable) |
| RUSTSEC-2025-0141 | `bincode` | Unmaintained (development ceased;
v1.3.3 considered complete) |
**Why this is safe in our usage:**
- Neither advisory describes a known security vulnerability. Both are
"unmaintained" notices only.
- `bincode` is used by syntect to deserialize pre-compiled syntax sets.
Again, these are **static vendored artifacts** baked into the binary at
build time. No user-supplied bincode data is ever deserialized. - Attack
surface is zero for both crates; exploitation would require a
supply-chain compromise of our own build artifacts.
- These exceptions can be removed when syntect migrates to `yaml-rust2`
and drops `bincode`, or when alternative crates are available upstream.
- introduce RenderableItem to support both owned and borrowed children
in composite Renderables
- refactor some of our gnarlier manual layouts, BottomPane and
ChatWidget, to use ColumnRenderable
- Renderable and friends now handle cursor_pos()
Refactor trust_directory to use ColumnRenderable & friends, thus
correcting wrapping behavior at small widths. Also introduce
RowRenderable with fixed-width rows.
- fixed wrapping in trust_directory
- changed selector cursor to match other list item selections
- allow y/n to work as well as 1/2
- fixed key_hint to be standard
before:
<img width="661" height="550" alt="Screenshot 2025-10-09 at 9 50 36 AM"
src="https://github.com/user-attachments/assets/e01627aa-bee4-4e25-8eca-5575c43f05bf"
/>
after:
<img width="661" height="550" alt="Screenshot 2025-10-09 at 9 51 31 AM"
src="https://github.com/user-attachments/assets/cb816cbd-7609-4c83-b62f-b4dba392d79a"
/>
Also, simplify the streaming behavior.
This fixes a number of display issues with streaming markdown, and paves
the way for better markdown features (e.g. customizable styles, syntax
highlighting, markdown-aware wrapping).
Not currently supported:
- footnotes
- tables
- reference-style links
i'm not yet convinced i have the best heuristics for what to highlight,
but this feels like a useful step towards something a bit easier to
read, esp. when the model is producing large commands.
<img width="669" height="589" alt="Screenshot 2025-09-03 at 8 21 56 PM"
src="https://github.com/user-attachments/assets/b9cbcc43-80e8-4d41-93c8-daa74b84b331"
/>
also a fairly significant refactor of our line wrapping logic.
#### Summary
- Emit a “Proposed Command” history cell when an ExecApprovalRequest
arrives (parity with proposed patches).
- Simplify the approval dialog: show only the reason/instructions; move
the command preview into history.
- Make approval/abort decision history concise:
- Single line snippet; if multiline, show first line + " ...".
- Truncate to 80 graphemes with ellipsis for very long commands.
#### Details
- History
- Add `new_proposed_command` to render a header and indented command
preview.
- Use shared `prefix_lines` helper for first/subsequent line prefixes.
- Approval UI
- `UserApprovalWidget` no longer renders the command in the modal; shows
optional `reason` text only.
- Decision history renders an inline, dimmed snippet per rules above.
- Tests (snapshot-based)
- Proposed/decision flow for short command.
- Proposed multi-line + aborted decision snippet with “ ...”.
- Very long one-line command -> truncated snippet with “…”.
- Updated existing exec approval snapshots and test reasons.
<img width="1053" height="704" alt="Screenshot 2025-09-03 at 11 57
35 AM"
src="https://github.com/user-attachments/assets/9ed4c316-9daf-4ac1-80ff-7ae1f481dd10"
/>
after approving:
<img width="1053" height="704" alt="Screenshot 2025-09-03 at 11 58
18 AM"
src="https://github.com/user-attachments/assets/a44e243f-eb9d-42ea-87f4-171b3fb481e7"
/>
rejection:
<img width="1053" height="207" alt="Screenshot 2025-09-03 at 11 58
45 AM"
src="https://github.com/user-attachments/assets/a022664b-ae0e-4b70-a388-509208707934"
/>
big command:
https://github.com/user-attachments/assets/2dd976e5-799f-4af7-9682-a046e66cc494
Wait for newlines, then render markdown on a line by line basis. Word wrap it for the current terminal size and then spit it out line by line into the UI. Also adds tests and fixes some UI regressions.
