# MinimojiKit — heads as native views A Swift package (links from iOS 17; the views want iOS 18, macOS 15) that plays a **baked minimoji head** — as a RealityKit entity tree (`MinimojiKit`), or in its own `MTKView` (`MinimojiMetal`, below) — and drives it with a port of the engine's animator, so a head is a SwiftUI view: transparent, in the layout, in a list row or a chat bubble. Nothing web, no canvas. `MinimojiService` talks to the avatar service for you; `MinimojiEditor` is the editor's pure model with the catalogue and the option pictures bundled; `MinimojiEditorUI` is the editor itself, as a SwiftUI view an app drops into a sheet; `MinimojiLive` guesses a mouth shape from a live call's audio. ## Quick start (another app) The package is published from this monorepo to **`https://github.com/hypersocialinc/MinimojiKit`** (private; read-only — the source is `packages/swift/MinimojiKit` here, `pnpm kit:publish` splits and tags it). Tags are `v..`: the major *is* the bake version, because a `BAKE_VERSION` bump stops every bake a consumer cached from matching. ```yaml # xcodegen project.yml packages: MinimojiKit: url: https://github.com/hypersocialinc/MinimojiKit from: "9.4.0" targets: YourApp: dependencies: - package: MinimojiKit product: MinimojiKit # HeadDocument, FaceLife, the RealityKit views - package: MinimojiKit product: MinimojiMetal # MinimojiMetalHeadView — a head as an ordinary view - package: MinimojiKit product: MinimojiService # AvatarService — the URL contract as one call - package: MinimojiKit product: MinimojiEditor # the editor's model: catalogue, rules, thumbnails - package: MinimojiKit product: MinimojiEditorUI # MinimojiEditorView — the editor as a view - package: MinimojiKit product: MinimojiEngine # LocalBaker — the engine on the device, for the editor's previews ``` The package's platform floor is iOS 17 so an app on 17 can link it; every view is `@available(iOS 18, *)`, so gate the head and the editor with `if #available(iOS 18, *)` and show the 2D picture below it. The model, the service and the caches run on 17. Get a **publishable key** at [minimoji.com/developers](https://www.minimoji.com/developers) (it may bake heads, cut editor pieces, and generate avatars, so it can live in the app; saving avatars requires a secret key on your backend), configure the service once with the deployment's **`.convex.site`** origin (`docs/avatar-pipeline.md`, "The contract"), then: ```swift import MinimojiMetal import MinimojiService let service = AvatarService(base: URL(string: "https://.convex.site")!, key: "mk_pub_…") // an avatar document (the same JSON the web edits) → a head, baked on first ask anywhere, // cached on this device by content key from then on let doc: [String: Any] = ["seed": 4821, "head": "round", "skin": "brown", "hair": ["style": "afro", "color": "ink"]] let head = try await service.head(for: doc) // HeadDocument MinimojiMetalHeadView(document: head, expression: "happy").frame(width: 96, height: 96) // store a KEY against a user, not a document: the same document is always the same key. // Host-provided backend client (not a MinimojiKit API): your backend authenticates // the user and calls POST /avatars with its secret key. Never embed that key in iOS. let saved = try await yourBackend.saveAvatar(doc) // response includes avatarKey, glb, png, json // …and every surface that never mounts a 3D view shows the 2D picture AsyncImage(url: service.renderURL(for: saved.avatarKey, size: 256)) // a viewer of your own (or another platform) loads the bake by URL: a 302 to the CDN object let url = service.headURL(forKey: saved.avatarKey, detail: 0.4) // a FRIEND's head: you stored their key, never their document — no key, no bake let theirs = try await service.head(forKey: theirKey) MinimojiMetalHeadView(document: theirs).frame(width: 96, height: 96) ``` ### The editor ```swift import MinimojiEditorUI @State var doc: [String: Any] = me.minimojiDoc ?? MinimojiEditorView.starter() @State var head: HeadDocument? // the last baked preview, for a Save screen MinimojiEditorView(service: service, doc: $doc, head: $head) .minimojiEditorTheme(MinimojiEditorTheme(page: …, card: …, ink: …, dim: …, accent: …)) // or the default // on Save: hand `doc` to your backend, which POSTs /avatars with its secret key ``` The native demo in `apps/minimoji-ios` uses this same view from the local package. Its wrapper owns navigation, shuffle and the document, registers bundled heads with `service.remember`, and passes bundled fragment sources into the editor. It has no separate editor or composition implementation. The demo wrapper persists accepted edits locally; persistence remains the host app's job. `MinimojiEditorView` is the prototype's editor lifted whole: the head on a card (tap for the mood ring, hold to talk, the gutters cycle moods), the rail of categories with the group dividers and the jump menu, the option tiles pre-rendered on the mannequin and tinted with the chosen colours, the colour dock, the constraint refusals up front, and the head's spoken reactions. Colour edits are draw-time overrides with no bake. Structural picks **compose on the phone**: the first bake is the anchor, the service cuts each category's pieces for it as categories open (`POST /packs/category`), and a pick with a piece in hand is a decode and a swap — `AvatarPartPack` picks the deltas, `HeadComposer` assembles the same `HeadDocument` a full bake gives (held to the engine by the Kit's parity fixtures). A pick with no piece yet falls back to `service.head(for:sharingEyeColor:)`, and the next likely picks are prefetched on whichever path they take. `localParts: false` turns the track off; `bundledSources:` seeds the pack with a pack the app ships. `head(for:)` expands the catalogue's defaults, asks `POST /avatars/bake` once per distinct document (concurrent asks share the request), remembers the CDN URL on disk, and decodes through `HeadCache`. Pass `sharingEyeColor: true` only if you draw the eye colour yourself with `ColorOverrides.eyes`: it pins `eyes.color` to ink so every eye colour shares one bake. `prefetch(_:)` warms the next likely pick; `categoryPack(anchor:current:field:)` is the editor's on-demand pieces. Errors are `AvatarService.Fail`: `.rejected()` for a `400` (the request as sent will not work), `.limited(retryAfterMs:, )` for a `429`, `.service()` for any other `{error}`, and `.reply(status:body:)` for an answer that is not the contract. #### A Minimoji from a selfie or a description `service.generate(photo:prompt:current:)` asks `POST /avatars/generate` for a document: from a photo (any format; the Kit downscales it to a 1024 px long edge and JPEG-encodes it before upload), from a sentence, or both (the photo leads, the words adjust). With `current`, an edit by description: only what the words ask for changes. It returns `Generated { doc, notes }`, `notes` being one sentence naming what could not be honoured, or empty. Nothing is saved: hand `doc` to the editor or to `save(_:)`. Usually a few seconds, so show a spinner that says so; the service checks every answer, retries a miss once with the correction, and falls back to a stronger model, all inside the one call. A publishable key is enough. The service API is also available for custom flows. Photos are processed by the model provider and are not stored by Minimoji. #### Importing, built in ```swift MinimojiEditorView(service: service, doc: $doc, head: $head, imports: .kit) // .kit(camera: true) enables camera capture ``` The avatar card carries an **orange camera** (bottom left) and **AI edit** (bottom right); neither opens a sheet. **AI edit** is an inline composer: the parts panel slides away, the stage grows, three suggestion chips derived from the document sit under it (never what the head already wears), and a field docks on the keyboard with the camera as its leading button. A chip or ↑ sends the words with `current` and the seed lock; the head wears the hologram while the edit is in flight, then **whirls** into the answer (`Whirl`, the port of the web's `swap.js`) with **Undo | Keep** under the stage. The sentence stays in the field. Holding AI edit dictates and sends on release. **A selfie is always a new face.** The camera turns the stage into the viewfinder (front camera, mirrored, a face oval, Library, shutter, flip, ✕); the shutter is the send. A request with a photo carries **no `current` and no `lockedPaths`**, from anywhere in the editor, so the service answers from its one-call decisions rung. While it reads the face the stage shuffles — the old head spins through three others (`shuffleHeads:` from the host, or three the device builds when the service has a local builder, below) and lands on the new face with the boing — then **Before** (press to peek), "From your selfie" and **Retake | Keep**. No face found returns the stage to the camera with the reason as its hint. Without a camera the button opens the photo library and the pick is the send. ```swift MinimojiEditorView(service: service, doc: $doc, head: $head, imports: .kit(camera: true), shuffleHeads: cast.map(\.document), // optional: the heads the shuffle turns through proposalOpen: $stageBusy) // true while the camera, the read or a new face owns the stage ``` Hide your Cancel/Save while `proposalOpen` is true; the Kit's own Keep, Undo, Retake and ✕ are the only exits then. Reduce Motion crossfades instead of whirling. A separate creation screen can opt into a one-time fresh request: ```swift MinimojiEditorView(service: service, doc: $newDraft, head: $newHead, imports: .kit(camera: true), creatingAvatar: true, creationSource: .selfie) ``` Initialize `newDraft` independently, for example with `MinimojiEditorView.starter()`. `creationSource: .selfie` opens the stage camera on arrival; `.description` opens the composer with the keyboard up and its first request omits the starting document. In the demo, **Settings → Replace avatar…** offers both. Supply `onReplacement` to persist the accepted document after Keep; it runs only for acceptance, never cancellation, and hosts handle save errors. The review sheet with **Revise**, paging and **Accept** remains only for the parts panel's **Remix this category · 3 looks**, the one place the Kit asks for three candidates at once; it refines the visible proposal in place with bounded recent context (the service accepts at most 500 UTF-16 code units of prompt text). Only **Accept** or **Keep** applies a matching rendered proposal, as one undoable edit; closing preserves the editor draft and history. Host Save remains separate. Photos are downscaled to a 1024 px long edge before upload. Editor tools appear inside the avatar card by default, including when embedded without a navigation bar. Hosts may pass `toolsInToolbar: true` to place them in their navigation toolbar; `onShuffle` adds a host action to that menu. Slow manual updates keep the current head visible with the same hologram and background-free terminal text at bottom-left. Tint and scan lines fade together over 0.4 seconds. Reduce Motion freezes scan/type animation and makes the fade immediate. Floating controls use native Liquid Glass on iOS 26, with an ultra-thin material fallback on older supported iOS. Hosts using `.none` still open on the first manual category. The stage camera requires `.kit(camera: true)`, a nonempty host `NSCameraUsageDescription`, and available camera hardware. Otherwise the camera button opens the library. Voice requires nonempty `NSMicrophoneUsageDescription` and `NSSpeechRecognitionUsageDescription` strings in the host's plist; the mic is hidden without them. Permission is requested only when the mic is tapped or AI edit is held. Dictation uses on-device recognition. The composer's mic fills editable text for explicit Send; holding the avatar-card button sends on release. Unsupported devices/locales and denied permissions leave typing available. Capture stops when leaving the editor or backgrounding the app. Hosts retain ownership of consent copy and persistence; the editor never saves a photo or audio recording. Hosts shipping a starting bake can pass `bundledHead` and a lazy `bundledSourcesProvider`. The Kit remembers that head before the first preview, resolves packs, and warms adjacent eye shapes. When selecting a different starting avatar, update `doc` and `documentID` together; the Kit resets the session. No host `.task` gate or session UUID is needed. ```swift MinimojiEditorView(service: service, doc: $doc, head: $head, bundledHead: selected.head, documentID: selected.id, bundledSourcesProvider: { AvatarPartPack.bundledSources(subdirectory: "parts/\(selected.id)") }, imports: .kit(camera: true)) ``` For custom UI, use `imports: .callback { source in … }`. It only invokes the host callback; the host captures input, generates, replaces `doc`, clears `head`, `bundledHead` and bundled sources, and changes `documentID` (or the view's `.id`) after success. `imports: .none` is the default and hides import entries. Legacy `onImport` and `onImportSelfie` remain available; when supplied they take precedence over `imports`. See [the selfie service specification](../../../docs/roadmap/selfie-photo-to-editable-minimoji.md) for the generation contract. ### On-device previews (`MinimojiEngine`, 9.2) `LocalBaker` is the same JavaScript engine the service bakes with, as a generated bundle (`minimoji-engine.js`) run in a `WKWebView` on iOS and in `JavaScriptCore` on macOS. Attach it to the service you hand the editor and picker taps and AI proposals are built where the tap is, instead of waiting for `POST /avatars/bake` and a 2 MB download. Attach once, from a long-lived place (the demo's `Service.deviceAvatar`): ```swift import MinimojiEditorUI // builder() is the editor's import MinimojiEngine import MinimojiService enum Service { static let loggedAvatar: AvatarService? = AvatarService(base: URL(string: "https://.convex.site")!, key: "mk_pub_…") private static let baker = LocalBaker() static let deviceAvatar: Result = Result { guard var service = loggedAvatar else { return nil } service.localBuilder = try baker.builder() return service } /// the service to hand the editor: with the device builder when the service /// bakes the same version (or has not answered in 3 s), else the server path static func editorService() async -> AvatarService? { guard let remote = loggedAvatar else { return nil } let matches: Bool? = await withTaskGroup(of: Bool?.self) { group in group.addTask { (try? await remote.capabilities()).map { $0.bakeVersion == LocalBaker.bakeVersion } } group.addTask { try? await Task.sleep(for: .seconds(3)); return nil } let first = await group.next() ?? nil group.cancelAll() return first } if matches == false { return remote } return (try? deviceAvatar.get()) ?? remote } } // in the view that shows the editor @State private var editorService: AvatarService? // … .task { editorService = await Service.editorService() } ``` The `deviceAvatar` lines are the demo's (`Service.deviceAvatar`); `editorService()` adds the attach rules below, which the demo does not run. Show a spinner until it answers (at most about 3 s). A builder that fails to start falls back to the service without one, which takes the server path; the demo instead shows "The device engine could not start: …" in place of the editor. `builder()` throws when the bundled engine's bake version does not match the editor's catalogue, and starts WebKit's process in the background so the first tap does not pay its load. Keep one baker for the service's lifetime. **When to attach.** Only when the service's `GET /capabilities` answers a `bakeVersion` equal to `LocalBaker.bakeVersion`: otherwise the device would preview heads the service bakes differently. If capabilities does not answer within 3 s (or fails), attach anyway; the version is in the package and a slow network should not cost the previews. Both rules are Bunch's; the Kit does not run them for you, `editorService()` above does. **What builds on the device, what still goes to the server.** With a builder attached, every `service.head(for:)` builds locally: picker taps (a colour paints at once; a structural pick highlights at once and keeps the current head on screen until the new one is ready, with no spinner), AI and selfie proposals, and the shuffle's starter heads. Opening a category warms its valid options one at a time, and overlapping taps share the build in flight. Saving (`POST /avatars`, through your backend), a friend's head by key (`head(forKey:)`), the CDN URLs (`headURL`, the `.png` renders) and every host without a builder take the server path as before. **On failure, today.** A runtime failure is shown by the editor with a retry; it is **not** sent to the server instead. A build timeout, a server fallback and a "still building" line are the open card "On-device builds fall back to the server, and say they are waiting". **Memory.** The in-memory head cache grows from 24 to at most 48 decoded heads when building locally (roughly 3 MB a decoded head, an estimate, not a profile). WebKit's content process is the number to watch on a phone, where iOS caps it below the app's own limit. Measured on macOS and the iPhone 17 Pro simulator (iOS 26.5), ten builds of one document at detail .4 (in the monorepo: `docs/roadmap/the-avatar-engine-runs-on-the-phone/measurements.md`): | runtime | where | JIT | engine load | p50 | p95 | |---|---|---|---:|---:|---:| | `JSContext` | macOS, `swift test` (the phone's proxy) | none | 14 ms | 794 ms | 876 ms | | `JSContext` | iOS simulator | yes | 16 ms | 97 ms | 157 ms | | `WKWebView` | iOS simulator | yes | 557 ms | 126 ms | 180 ms | | either | iPhone | | | not yet measured | | The simulator flatters `JSContext`: its JavaScriptCore has a JIT and a phone's does not (a third-party app may not map executable memory; only WebKit's content process keeps its JIT), which is why `.automatic` is the `WKWebView` on iOS. In the demo, from the `/avatars/generate` answer to the proposal ready (measured on 9.2's review sheet): **18 ms** for a head already warmed, **515 ms** for a cold build, against **3.9 s** through the server. A device build matches the server's bake in nodes, materials and `extras`, with the last digit of a few coordinates different (the system's `Math.sin` is not V8's); nothing keys on the bytes across runtimes. ### Talking: mouth-only, with optional accents A talking head animates its mouth without marks beside the face by default. Lip shapes, viseme timing, expression and head motion are unchanged. Hosts can explicitly choose `.arcs` for sound rings or `.lines` for manga chatter dashes: ```swift MinimojiMetalHeadView(document: head, talking: isSpeaking) // mouth-only MinimojiMetalHeadView(document: head, talking: isSpeaking, talkStyle: .arcs) // opt in ``` Optional arcs draw over hair and turn with the head, light one to three rings with loudness, and fade within about 0.4 s of silence. Below 48 pt they are thicker and two rings at most. `ColorOverrides.talk` / `talkOpacity` tint them. The existing baked accent nodes remain available; this default needs no rebake. Existing callers that explicitly select `.arcs` or `.lines` keep that choice. Without real audio the loudness is read off whatever moves the mouth (`talking`, visemes, `mouth`). A host with real audio passes a level (0…1). It drives only the rings, never the jaw or the gaze, and nil releases it at once. Choose the way in by where the level lives: - **It is already SwiftUI state** (an editor's mic waveform, a published meter): pass the `voiceLevel:` prop. The view keeps it fresh every frame, but it only sees a new value when your view's body re-renders. - **It comes from an SDK that doesn't publish changes** (LiveKit's `participant.audioLevel`, an `AVAudioEngine` tap): don't pass it as a prop. A prop re-reads it only when something else re-renders the tile, so the arcs freeze. Hand the view an `AvatarPerformanceInput` source instead. The renderer samples it every frame with no SwiftUI churn. Or call `FaceLife.setVoiceLevel(_:)` each frame if you drive the animator yourself. ```swift @MainActor final class ParticipantVoice: AvatarPerformanceInput { let participant: Participant init(_ participant: Participant) { self.participant = participant } func performanceSnapshot(at hostTime: TimeInterval) -> AvatarPerformanceSnapshot { // only the level: look and mouth stay nil, so the head keeps its own AvatarPerformanceSnapshot(voiceLevel: participant.isSpeaking ? Double(participant.audioLevel) : nil) } } // hold ONE per participant (a stable identity), not one per body pass MinimojiMetalHeadView(document: head, talking: participant.isSpeaking, performance: voice) ``` While a live source names a level, it wins over the prop. A source that names none (a beat, a look) leaves the arcs to the prop. ### Listening Listening to music is a **state, not a face**: it composes with whatever expression the head wears — a sad head nods sadly — and sets nothing on it. Both head views take it beside `expression`: ```swift MinimojiMetalHeadView(document: head, expression: "sad", listening: Listening(tempo: 120, mouth: .hum, intensity: 1)) ``` The head nods on the beat of `tempo` (a chin dip every beat, a sway every two bars, on a clock an emote pauses and hands back in phase), three notes rise off the crown, and the mouth does what `mouth` says — `.closed`, `.hum`, `.sing`, or `.lipSync` where you feed `mouth:` from audio as the demo's `Mic` does — unless `talking` or a fed mouth wins. `intensity` (0…1) scales the nod and the notes so a list tile bops less than the hero. `life.drop()` (or `FaceState(event: .drop)`) adds a bigger nod without leaving the state. Floating single and joined notes keep their gentle independent cadence. Headphones are **worn, not saved**: `ListeningWear.headphones(on:catalog:)` in `MinimojiEditor` derives `identity ⊕ {extra: headphones}` under the catalogue's rules and says in `applied` whether they went on (a beanie wearer keeps the beanie); show `worn` through your part pack or the service, and leave the identity document alone. The status chip is yours to draw — `Listening.label` plus the track — the Kit never renders text. ### Thinking and presence (chat heads) Two more states for a chat: `thinking:` while a friend types, and `presence:` for whether they are there at all. Both head views (and `.minimojiHead(...)` / the stage's slots) take them beside `expression`: ```swift MinimojiMetalHeadView(document: friend, expression: "neutral", talking: isSpeaking, thinking: friendIsTyping, presence: isOnline ? .online : .asleep(), frameRate: isSpeaker ? .full : .reduced) ``` - **`thinking: true`** is the typing state: the `thinking` face (one brow up, the mouth pulled aside, the eyes up and aside) plus a loop — a gentle tilt, a glance every 1.5–3 s — with the mouth held shut. `"thinking"` is also a plain face, so it works as an `expression` or an emote. **Talking always owns the mouth:** turn `talking` on and the jaw takes over at once while the tilt fades out in about 0.3 s, so you don't have to flip both on the same frame. - **`presence:`** is per head and yours to set; the Kit never infers it. - `.online` (default) is the idle, with a subtle breath (the same one the web head has). - `.asleep()` is offline, not the snoring `sleeping` face: eyes and mouth shut, a slow breath, the head drooped partway, no Zs. No bake change. - `.asleep(z: true)` lies the head down with upright Zs. The Zs are plates only a head baked with **full (avatar-style) eyes and not `lite`** carries; anywhere else `z: true` just lies the head down. - `.party(tempo: 120)` is a small bounce and squash on the beat (a `performance`'s `beat` drives it too). With `listening` also set, listening wins the beat. - **Wake-up is built in.** Leaving `.asleep` for anything — `thinking`, `talking`, `.online` or `.party` — plays about 0.6 s of choreography: a blink opens the eyes, the head lifts on a spring, the brows jolt, then it hands off (a party bounce fades in once the head is up). Don't script sleeping → surprised → neutral yourself. - **`frameRate:`** (`MinimojiMetalHeadView`) is `.full` (60 fps, default), `.reduced` (30) or `.low` (15). Motion runs on the frame's `dt`, so a head keeps its speed at any rate; it only draws fewer frames. Run the speaker at `.full` and the heads beside it at `.reduced`. - **Small heads exaggerate on their own.** Under 48 pt (the view's shorter side, `FaceLife.smallHeadPoints`) the view sets `FaceLife.emphasis` to 1.6, which enlarges the thinking face's brow raise, mouth offset and gaze bias so they still read at 34 pt. Driving `FaceLife` yourself, set `life.emphasis = FaceLife.emphasis(forPoints:)`. - **Lighter bakes for small heads.** Below about 64 pt, fetch the head with `detail=.25&lite=1`. `lite` drops the effect plates — which is also where the asleep Zs live, so keep a full bake for any head that uses `.asleep(z: true)`. Which to use at which size (a tile at 3x is 64, 128 or 256 px). Every head inside the app animates live, including under 34 pt; stills are only for surfaces that can't run a live head. | Head size | Use | Why | |---|---|---| | ≥ 64 pt | a live head, `.full` | features and loops read | | 34–63 pt (a typing peek at 34 pt) | a live head, `.reduced`, lite bake; no fallback to the still | a raised brow and a side mouth are subtle; motion helps them read | | < 34 pt (a 28 pt thread avatar) | a live head in the lightest mode: `.low` (15 fps), lite bake | in-app heads stay live; this band costs the most, so it runs the cheapest | | notifications, widgets | the still (`GET /avatars/.png?face=`) | these surfaces can't run a live head | How thinking reads live at 34 pt and at 28 pt, and whether the small-size emphasis should grow further below 34 pt, is **pending on-device measurement (see PR)**. ### Emote bursts: a sticker round a mood change (9.10) At 28 pt a face change is a few pixels of brow and mouth. So when a `MinimojiMetalHeadView`'s `expression` changes, the view also pops that mood's sticker effect around the head and moves the head with it, once: hearts out from behind it on happy, a star burst on joy, HA stickers on laugh, a rain cloud on the hair on sad (heavier, with tear streams, on crying), flames behind and the anger mark on the temple on angry and furious, `!` and a bolt on surprised, a thought bubble on thinking, rising Zs on sleeping. It is on by default; `bursts: false` turns it off. ```swift MinimojiMetalHeadView(document: head, expression: mood) // bursts on a change MinimojiMetalHeadView(document: head, expression: mood, bursts: false) // the face alone ``` - **Only on a change.** Not on the view's first expression (a row of heads appearing does not all go off), not when the same expression is set again, not for `neutral` or `idle` (those clear a burst that is playing), not under Reduce Motion (the face still changes) and not while `isActive` is false (nothing plays later on resume). A new expression while a burst plays replaces it. `emotes: false` does not stop the sticker. - **Short.** Each burst plays once, about 1.8 to 3.5 s, then the layer is idle and draws nothing. While one plays it draws at the display rate, whatever the view's `frameRate:`. - **It spills past the frame.** The stickers are measured in the skull as drawn, centred on it, and reach up to 1.5 head widths from its centre, so they draw outside the view's bounds (behind the head and in front of it). A host that clips the head (`.clipShape`, a `ScrollView`'s clipping, a masked cell) clips the burst too. Neighbouring heads overlap it: a burst behind one head draws under the next head along. - The head's move (a pop on every change; a shake for angry, a jump for surprised, a droop for sad) is a SwiftUI transform on the view; the animator and the face are untouched. `MinimojiHeadView` (RealityKit) and the web twin have no bursts. ### Gestures on cue: the Kit plays, the host cues (9.4) A head's 45 automatic mood performances (three per face, in three motion languages) used to play only when the expression changed. A host can now cue one, once, over whatever face the head wears, and the face does not change: a double take when a reply lands, a nod when a card arrives, a chuckle on a knock. **9.16.0:** the catalogue adds the explicit `.headShake` cue: two small, tapering turns saying no, returning to rest in 1.85 seconds. Call `life.perform(.headShake)` or send `HeadCue(.gesture(.headShake), generation: cues)`. It uses the existing head transform, preserves the current expression, and respects gaze ownership and Reduce Motion. It is never selected automatically on a mood change; the original 45 mood tracks and their selection stay intact. ```swift @State private var cue: HeadCue? = nil @State private var cues = 0 MinimojiMetalHeadView(document: head, expression: "happy", cue: cue, onGestureEnd: { gesture, cancelled in /* chain the next one here */ }) // on the moment cues += 1 cue = HeadCue(.gesture(.doubleTake), generation: cues) // a named track, any face's cue = HeadCue(.forMood("laugh"), generation: cues) // one of laugh's three, never the last one twice ``` - `HeadGesture` is a generated enum over the track ids (`.doubleTake`, `.sunnyNod`, `.bigDoubleLaugh`, …; the raw value is the id the service's picker answers), with `face` (the face it was authored for), `duration` and `name`. Any gesture plays on any face: a happy head doing neutral's `.tinyDoubleTake` is the point. - **A cue is a value.** The view fires it once per distinct `HeadCue`: a SwiftUI rebuild with the same state replays nothing, a bumped `generation` fires the same gesture again, nil clears. `MinimojiHeadView` and `.minimojiHead(…)` take the same `cue:` and `onGestureEnd:`. - **The face never changes.** A cue sets the performance slot and nothing else: no `expression`, no blink, no jolt. A cue while a performance plays replaces it from rest, and a replaced *cue* ends `cancelled: true` (the emote's own one-shots never report); a face change picking its own one-shot cancels a cue the same way (unless `emotes: false`, below). One shot only: no loop, no queue. Chain from `onGestureEnd`, which always comes on the main actor from the frame tick: a finish on the frame the track ends, a cancel on the frame after the cue or expression that replaced it (never inside the SwiftUI update that changed the view). A cue on a head that is off screen, parked (`isActive: false`) or not yet built plays when the head is next drawn; a stage slot that goes away ends its cue cancelled. - **Reduce Motion.** The views already pass `performances: false` under it; a cue then moves nothing and `onGestureEnd` still comes on the next frame, so a chain advances. Same for an explicit `performances: false`. - **Cues without the automatic emote (9.6).** `emotes: false` stops only the one-shot a new `expression` picks for itself: the face still changes, a cue still plays, and a face change no longer cuts a playing cue short. It is for a host that cues its own gestures (a game's verdicts, a singing app). `performances: false` and Reduce Motion still freeze everything. - **Over a host's beat (9.6).** While a `performance:` source hands over a `beat`, a one-shot (a cue or an emote) rides over the nod and the nod keeps the host's beat; the two add. A free-running listener (no host beat) still pauses its nod under a one-shot and takes it back in phase, as before. The beat counts as the host's until it stops: the snapshot driver sends `setBeat(nil)` for you; a host calling `FaceLife.setBeat` directly must send that nil itself when its clock goes away. - Underneath, `FaceLife.perform(_:)` / `perform(for:)` / `perform(_ cue:)` and `onPerformanceEnd` are public for a host that owns a `FaceLife`; a document swap carries a running cue and its hook along (`continueMotion`). - **The picker's gesture.** `AvatarService.pickExpression(for: text, gesture: true)` asks the same decision call one more question and answers `ExpressionPick.gesture`, the `HeadGesture` to cue beside the picked face (nil when none fits, there is no pick, or the service named a track this build does not know), and `gestureWord` (nod, double-take, tilt, bob, laugh, shake, lean, glance). Without the flag nothing changes. - The web twin: `cue={{ gesture: 'double-take', generation }}` or `{ mood: 'laugh' }` and `onGestureEnd` on `MinimojiHead` (an avatar change ends a running cue cancelled rather than carrying it, the one place the twins differ); `life.perform` in the engine; `packages/engine/dev/cue.html` to try it. ### Performance input: a host plays the head An instrument or a game does not tell the head a mood, it **plays** it: a finger on a grid aims the gaze, a held note holds the mouth open, the audio engine's beat drives the nod, a musical event blinks. That input changes every frame, so it does not go through SwiftUI props. Hand the view ONE object and the renderer samples it on the main actor right before each frame's tick: ```swift import MinimojiKit import MinimojiMetal @MainActor final class JamPerformance: AvatarPerformanceInput { var touch: CGPoint? // plain state, set by your gesture; never @Published per frame var transport: Transport // your audio clock private var crossing = BeatCrossing(every: 2), dropEvery = BeatCrossing(every: 8) private var blinks = 0, drops = 0 // bump `drops` on a pluck, too func performanceSnapshot(at hostTime: TimeInterval) -> AvatarPerformanceSnapshot { guard transport.isPresentable else { return .released } // stopped AND idle, stale, torn down let beat = transport.beat(at: Date()) // YOUR clock; nil when stopped or stale if crossing.sample(beat, generation: transport.generation) { blinks += 1 } if dropEvery.sample(beat, generation: transport.generation) { drops += 1 } return AvatarPerformanceSnapshot( look: touch.map { SIMD2(Double($0.x) * 2 - 1, -(Double($0.y) * 2 - 1)) }, // screen y is down, the head's is up mouth: voice.sample(dt: dt, target: touch == nil ? nil : rowOpen, held: heldSeconds, vowel: touch.map { Double($0.x) }), // a SingingVoice: see below beat: beat, transportGeneration: transport.generation, blinkSequence: blinks, dropSequence: drops) } } MinimojiMetalHeadView(document: head, emotes: false, listening: transport.isRunning ? Listening(tempo: transport.bpm, notes: false) : nil, performance: jam, isActive: scenePhase == .active && isOnScreen) ``` | snapshot field | what it does | |---|---| | `isActive` | `false` (or `.released`) when the host is stopped, stale or parked: gaze, mouth and beat are let go at once | | `look` | −1…1, +x right, **+y up**; `nil` hands the gaze back to the head's own glances | | `mouth` | each 0…1. `energy` above 0 makes the mouth **sung**: it opens bigger (about 1.3× taller and 1.25× wider, growing down from the upper lip; teeth and tongue keep their size), the round and open plates are size-matched where they swap so a vowel slide morphs, and the brows lift, the eyes widen and the head stretches tall with it (0, the default, is plain lip-sync and moves nothing else). `SingingVoice` turns "this row, held this long" into such a mouth: the note lands with a pop, holds, and after 0.8 s trills with the jaw, the lips and the energy together, so the eyes and brows flutter too; an optional `vowel` (0 a tight round "oo" … 1 a wide "ah", eased) lets a pad's columns morph the lips; and past `blissAfter` (2 s) on one note it sets `bliss`, which shuts the eyes into the happy arc behind a blink until the note changes or ends. Frame-rate independent. Sampled every frame, so `setMouth`'s 0.3 s TTL never closes a held note; `nil` releases that frame. The TTL stays as the fail-safe for callers who feed `mouth:` by prop | | `beat` | the host's **absolute** beat. It replaces the listening nod's clock for that frame, before the nod is sampled, so tempo changes, loops and seeks need no bookkeeping. `nil` free-runs on `Listening.tempo`, as it always has | | `transportGeneration` | bump on a seek, a restart or a pad switch: the nod eases back in over a quarter second instead of snapping to the new phase | | `blinkSequence` | an event counter. The head blinks once per change, however many frames repeat the number; a blink under way absorbs a new one; nothing is replayed after a stop, a park or a resume | | `dropSequence` | an event counter, like `blinkSequence`: once per change the head does the listening **drop**, a nod up to 1.9× bigger and a burst of notes (if `notes` is on) for 1.2 s. Default 0; nothing is replayed after a stop, a park or a resume; none under Reduce Motion. It is `FaceLife.drop()`, so it **does nothing unless the head is listening**: pass `listening:` (with `listening` nil a drop is dropped, not saved for later) | Rules of the road: - **Ownership.** While a source is installed it owns gaze, mouth, beat, blinks and drops; the `look:` and `mouth:` props are ignored. `expression`, `listening` (tempo, mouth style, intensity, notes), `colors` and the document stay yours. Remove the source and the props speak again. - **The clock is yours.** Answer `beat` from the audio engine, not from `hostTime` deltas; `hostTime` is the frame's `CACurrentMediaTime()`, for interpolation. When your clock is not good, return `beat: nil` **and** pass `listening: nil`: a missing beat alone keeps the free-running nod, which is what existing listeners rely on. - **Singing is not talking.** Leave `talking` false, or its random jaw competes with the notes. Leave `performances` on: under your beat a one-shot rides over the nod and never pauses it (9.6). Pass `emotes: false` if a face change should not play its own emote, and cue the gestures you want (`cue:`, a nod on a verdict) yourself. - **Speech is talking.** A recorded voice (a voice status, a voice note) drives the mouth through the same snapshot with a `SpeechVoice` (8.1): see below. There `talking: true` is right: a fed mouth suppresses the random jaw, and `talking` adds only the head's small bob and brow wiggle. - **Notes are a separate choice.** `Listening(notes: false)` keeps the nod and hides the floating notes, and the camera keeps its ordinary framing (fade-out included) instead of shrinking the head for notes nobody sees. - **`isActive: false`** pauses the `MTKView` (no frames, no sampling) and lets go of the host's input; coming back resets the frame delta and takes a current sample. `drift: false` was never a pause switch. - **Reduce Motion** under a source: no beat-driven nod, no notes, no blink events; gaze and mouth stay, because they are direct feedback to a touch. - **Bad numbers are made safe.** NaN and infinities become 0 (or no beat), ranges are clamped, before anything reaches a transform. - Keep `performanceSnapshot` cheap: no document loads, no allocation, no network. Identity is loaded once; the snapshot only moves it. `MinimojiHeadView` (RealityKit) and `.minimojiHead(...)` take the same `performance:`; the stage samples it in its per-frame pass and releases it while the head is off screen. The harness's **Keys** tab (`-keys`) is a working example (`apps/swift-harness/Sources/PerformancePreview.swift`); the web twin is `packages/engine/dev/perform.html` over the same contract (`createPerformanceDriver` in the engine, `performance` on `MinimojiHead`). ### Speech: a recorded voice plays the mouth A voice status or a voice note has a clock of its own (the player's), a loudness envelope and, with on-device speech recognition, words with times. `SpeechVoice` (8.1) turns "this loud, this shape" into a plain lip-sync mouth every frame; the host owns the clock and the timed stream, the Kit owns the easing: ```swift @MainActor final class ClipVoice: AvatarPerformanceInput { let clip: VoiceClip // its envelope (~20 Hz) and a viseme per span, stored with the clip let player: AVAudioPlayer private var voice = SpeechVoice() // keep it; it eases between frames private var last: TimeInterval? func performanceSnapshot(at hostTime: TimeInterval) -> AvatarPerformanceSnapshot { guard player.isPlaying else { last = nil; return AvatarPerformanceSnapshot(mouth: voice.sample(dt: 1 / 60, viseme: nil, level: 0)) } let dt = last.map { hostTime - $0 } ?? 1 / 60; last = hostTime let t = player.currentTime // YOUR clock, not hostTime let level = clip.envelope.level(at: t) // 0…1; interpolate the 50 ms grid if you can let viseme = clip.viseme(at: t) // .rest between words while the clip plays return AvatarPerformanceSnapshot(mouth: voice.sample(dt: dt, viseme: viseme, level: level), voiceLevel: level) } } MinimojiMetalHeadView(document: head, talking: player.isPlaying, performance: clipVoice) ``` | | | |---|---| | `Viseme` | `.rest` (between words, and any letter without a shape), `.open` (ah), `.round` (oh, oo), `.wide` (ee, ih), `.shut` (m, b, p). Each sets the lips' width and roundness and caps how far the loudness may open the jaw; `.round` is the only shape past round 0.5, so a stream never dithers the mouth plate | | `SpeechVoice.sample(dt:viseme:level:)` | the loudness opens the jaw (`floor` 0.12 + level × `gain` 1.1, under the shape's cap) at `attack` 26/s, closes at `release` 14/s, quick enough that the dip between two words reads as a shut mouth (`.shut` closes at the attack rate), and the lips glide to the shape at `lips` 12/s: smooth, and alike, at 30, 60 and 120 Hz. `energy` and `bliss` are 0: nothing else on the face moves. Below `quiet` (0.03) the voice is silence | | the release | a `nil` viseme still follows the loudness (no words known); once quiet and shut with a `nil` viseme the sample is `nil` and the mouth is the head's again. Hand `.rest` over through a clip's silences so the head's own talk jaw stays off, `nil` only once the clip ends or pauses | | the arcs | pass the same loudness as `voiceLevel`; with none the arcs read it off the mouth's opening | Bad numbers are quiet: a NaN or infinite level is 0. Letters to visemes is the host's call (a, o/u, e/i, m/b/p, else `.rest`), as is snapping word timings to the envelope's gaps. **The speaking look** (8.1) is the animator's, so every host gets it from the same `mouth`: each shape is its own plate (the wide, shallow `ee` plate for `.wide`, the round one for `.round`, the open one otherwise, the resting lips when shut); the teeth and the tongue never stretch with the jaw, they keep their size, hang from the upper lip and sit in the jaw, go behind round lips and wide ones respectively, and hide in a barely parted mouth; the chin dips a little with each syllable's loudness (off under Reduce Motion, with the performances); a phrase that ends blinks once; and the eyes stay on the viewer while the head speaks. A sung mouth (`energy` above 0) keeps its own look. **Emotional talking** (8.1): speech is a layer on top of the expression, never a replacement, so a happy head and a sad head look different mid-sentence at 48 pt. The voice still chooses the plate and how open it is; the face reshapes it through its talk-only keys (nothing changes at rest): `talkSX` / `talkSY` scale the plate (happy wider and a touch up, laugh much wider, joy wide and up, sad and crying narrower and lower, worried flatter, angry and furious squarer, surprised and scared taller and narrower, dead a slack little plate), `talkY` slides it, `talkX` / `talkRoll` give smug its smirk (thinking keeps its own `mouthX` / `mouthRoll`). The reshaping grows in with the jaw (none at the lip line, all of it by a third open), so the swap from the mood's lip line to the plate never pops. `talkTeeth` biases the furniture rule: angry and furious bare the teeth on every open shape and keep the tongue down (a snarl), happy, joy and laugh show more teeth, sad, crying and worried keep them behind the lower lip. And a syllable that hits (the jaw opening past a quarter, with a 250 ms cooldown so it never flutters) lands a **stress accent**: one short eased pulse (60 ms in, 180 ms out) that flicks the brows, widens the eyes a touch and nods the head, its size how far the jaw gets in the first 60 ms and its direction the mood's (`talkBrow`: happy, joy and surprised lift, angry and furious press down, sad and crying barely move, dead not at all). Between pulses the brows and eyes sit exactly where the expression put them; the brows never follow the loudness itself, and no word changes the expression. The accent rides the performances switch (off under Reduce Motion); the mouth's reshaping does not. In a gap the mouth is the expression's own. A sung mouth keeps its look on every face. ### Live speech: shapes guessed from the audio (`MinimojiLive`, 8.4) A live voice has no transcript: speech recognition lags the sound. So for a call, `LiveVisemes` (its own product, `MinimojiLive`) guesses the shape from the audio itself. The host hands over the PCM it already plays and feeds the answer to the same `SpeechVoice`: ```swift import MinimojiLive let visemes = LiveVisemes(sampleRate: 48_000) // one per voice (per participant), kept // wherever your voice SDK hands you a remote track's PCM (mono Float32), on its own queue: func render(pcm: UnsafeBufferPointer) { visemes.process(pcm, at: CACurrentMediaTime()) // the time the buffer ARRIVED is all it needs } // in each head's performance source, on the main actor: func performanceSnapshot(at hostTime: TimeInterval) -> AvatarPerformanceSnapshot { let (viseme, level) = visemes.now(at: hostTime) // a pure read: every head of this voice gets the same answer return AvatarPerformanceSnapshot(mouth: voice.sample(dt: dt, viseme: viseme, level: level), voiceLevel: viseme == nil ? nil : level) } ``` | | | |---|---| | what it is | a port of the engine's analyser (`createLiveVisemeTimeline`, held to it by a recorded trace): five frequency bands and the loudness every 33 ms of sound, read by a fixed set of rules into `.open`, `.round`, `.wide`, `.shut` or `.rest`. On-device, `Accelerate` only; nothing is recorded or sent | | what it is **not** | lip-readable. It is a band-pattern guess, not phoneme recognition: against scripted clips it agreed with the letters on 27% of frames where chance was 25% (29% read at the delay that suits it best), and `.shut` (m/b/p) it hardly ever says. What it gives is a more varied mouth than loudness alone (another mouth plate on about a quarter of frames through Opus at 48 kbit/s) | | the speaker's volume | the rules read loudness as well as the spectrum, so the volume is evened out first: a running level of the voice is held to a fixed one by a gain of a quarter to eight, quick for a voice just met and slow once it knows the speaker; silence leaves it alone, and one loud bump moves it little. Measured on clips: at half volume 87% of shaped frames keep their shape and `.open` is still said (10% of speaking frames, against 14% at full volume and none without the evening); at a quarter, 63% and 5%. The `level` answered is the evened loudness | | the host supplies the audio | the Kit opens no microphone and no audio session, and `MinimojiLive` depends on no voice SDK. Analyse where the sound is heard (the listener's phone from the remote track, the speaker's own head from the local mic) and sync is by construction | | `process(_:at:)` | the SDK's delivery queue: any thread, one call at a time, a buffer of any size at the `sampleRate` given to `init`. It allocates nothing, in a debug build either (a test counts every malloc on the delivering thread). Buffers may arrive late or several at once: a frame's time comes from counting samples off an anchor, the least-delayed arrival of the last second, never from when its buffer landed. An arrival more than 0.25 s past its predicted time is a gap (DTX, a muted track): the analyser starts over and re-anchors. `reset()` (any thread) forgets the voice | | `now(at:)` | the shape and loudness current `playoutOffset` ago, on the clock `process` was stamped with. It moves nothing, so a tile, a header row and a pill can all read one voice; each head keeps its **own** `SpeechVoice`, which eases. `(nil, 0)` before any sound and about a third of a second after it stops arriving, which releases the mouth; delivered silence reads `.rest` | | `playoutOffset` | seconds to read behind the host clock, when your callback runs ahead of the speaker (a jitter buffer still to play). Settable at any time; measure it on a real call. It also rides out a stalled queue: a read that far back is not waiting on a late buffer | | the delay | from a voice starting to the mouth being open: 28 ms at the median, within 45 ms nine times in ten. To the analyser naming a shape: 48 ms, within 98. To the mouth plate being that shape's, through `SpeechVoice`'s easing of the lips: 110 ms at the median (measured on clips with on-time 10 ms buffers, read at 60 Hz, no `playoutOffset`; a late buffer or an offset adds its own) | **Which mouth source to use.** The Kit cannot pick this for you: the host supplies the audio and knows whether words came with it. | the voice | feed `SpeechVoice` | |---|---| | live sound, no transcript yet: a call, and a voice status while it is being recorded (the mic's PCM into `LiveVisemes`) | the audio's shapes, `LiveVisemes.now(at:)` | | a recording with timed words (a voice status played back) | the transcript's visemes, as before. They follow the words and catch closed lips (m/b/p); the audio guess does neither | | a recorded clip with no transcript | `LiveVisemes` on the audio as it plays | | no audio at all | nothing, or `viseme: nil` with a loudness if you have one | Two sharing rules: one `LiveVisemes` per voice, read by any number of heads; one `SpeechVoice` per head. Measured on synthetic voices and read clips through Opus, never on a recorded call: `node packages/engine/dev/viseme-eval.mjs` in the monorepo prints the tables and takes a captured remote track (`--pcm`). ## How it fits together ``` avatar JSON ──(Node: buildGloss + GLTFExporter)──▶ billie.glb ──▶ HeadAsset.load ──▶ HeadDocument packages/engine/dev/bake.mjs GLB.swift meta · parts · extras │ FaceLife (gface.js, ported) ◀── makeFaceNodes() ◀──┘ │ update(t, dt): every plate's transform + visibility, the head's offsets HeadEntity (RealityKit) ── apply() each frame ──▶ MinimojiHeadView (RealityView) ``` - **The builder is not ported.** The head's identity — ~5k lines of procedural geometry — stays in JS and is baked once per document (`node packages/engine/dev/bake.mjs avatar.json out/ [detail]`, default detail .4 ≈ 1.75 MB / ~51k triangles; `lite: true` drops the effect plates). The bake carries the animator's contract as glTF `extras` and the placement meta on the root. - **The animator is ported**, branch for branch (`FaceLife.swift` ↔ `packages/engine/src/gloss/gface.js`): blink, gaze (yours or its own glances), the two critically-damped springs, the fifteen faces and their blend, shape swaps behind the lids, the mouth set and its furniture, talk, visemes, parametric mouth, idle drift, every effect plate, the head's offsets. Its randomness is injected, and a **parity test** holds it to the JS: `dev/face-trace.mjs` records the JS animator on a seeded `Math.random` through 900 scripted frames; the Swift port must reproduce every plate to 2e-4 (`Tests/…/FaceLifeParityTests.swift`). - `HeadEntity` is the RealityKit half: `MeshResource`s from the bake's arrays (generate them off-main with `generateMeshes(for:)`), `PhysicallyBasedMaterial`s from the glTF materials, `apply()` writes the animator's answer to the entities. - `MinimojiStageView` is the one `RealityView` of a window on the RealityKit path — at the root, never torn down. An orthographic camera in points, a key + fill + a small gradient IBL, and a mount per mark: each `.minimojiHead` puts a probe `UIView` under its view, and every rendered frame the stage asks each probe for its window and its rect (`convert(_:to:)`), places the head over it at the web's `fitScale`, builds new ones off-main, parks departed ones by document for instant reuse, and shares meshes per document. `MinimojiHeadView` is a mark plus drag-to-look / tap-to-talk. - `FaceLifeSystem` steps every head and the stage each rendered frame, through a query with `updatingSystemWhen: .rendering`. ## Footprint (measured, 9.12) What the package adds to an app, measured on `main` at 9.12: a Release build for iOS (arm64, `__TEXT` + `__DATA` of each target's object files, before the linker's dead-stripping, so an upper bound) plus the resources each target bundles. Link only the products you use: the thumbnails are the editor's, the engine is the editor's previews, the stickers are Metal's. | Product | Code | Bundled resources | What the resources are | |---|---:|---:|---| | `MinimojiKit` | 855 KB | — | the model, `FaceLife`, the RealityKit views | | `MinimojiMetal` | 276 KB | 500 KB | 13 emote-burst stickers, 256 px PNG | | `MinimojiService` | 175 KB | — | the URL contract and the caches | | `MinimojiLive` | 20 KB | — | visemes from PCM | | `MinimojiEditor` | 185 KB | 1.5 MB | 236 option thumbnails (256 px palette PNG, ~6 KB each) + the 56 KB catalogue | | `MinimojiEngine` | 59 KB | 652 KB | the JS engine bundle, three.js r160 included (~500 KB of it; 188 KB over the wire gzipped) | | `MinimojiEditorUI` | 1.4 MB | — | the editor view, on the five above | So, roughly: - **Play a head** (`MinimojiKit` + `MinimojiMetal` + `MinimojiService`): about **1.3 MB of code and 0.5 MB of stickers, ≈ 1.8 MB installed**; a few hundred KB less after dead-stripping and App Store compression. - **Ship the editor** (everything): **≈ 6 MB installed**, half of it code. The thumbnails were 7.9 MB until 9.12 quantised them to a 256-colour palette in the generator (`gen-editor-thumbs.mjs`, `compress`); they are flat renders of one grey mannequin, so nothing shows at the tile's 64 pt. The next lever, if one is ever needed, is fetching them from the service on first open, which would take the whole Kit to about 4 MB. - **Per head, at run time**: a bake at the default detail is **1.5–2.3 MB on the wire** (`.glb`, cached on the device by content key, fetched once per distinct document) and about 3 MB decoded in memory. - **Source**: 70 Swift files, ~12,400 lines, no third-party dependencies; `MinimojiLive` links Accelerate, `MinimojiEngine` links JavaScriptCore and WebKit (system frameworks, already on the device). For comparison, the web engine (`@minimoji/engine`) is 176 KB minified / 67 KB gzipped on its own, 677 KB / 195 KB with three.js. To re-measure: `xcodebuild -scheme MinimojiKit-Package -destination 'generic/platform=iOS' -configuration Release CODE_SIGNING_ALLOWED=NO build`, then `size` over each `.build/Objects-normal/**/*.o`. The developers page (`apps/web/app/developers/Front.tsx`, the footprint band) quotes these figures: change them together. ## API | | | |---|---| | `HeadAsset.load(url) / load(data:)` | `.glb` → `HeadDocument` (async variant decodes off-main) | | `FaceLife(document:gaze:drift:performances:emotes:random:)` | the runtime: randomly selected non-repeating emote performances (`emotes: false` / `emotesEnabled` turns off the face change's own pick, cues still play), `lookAt`, `setFace`, `talk`, `setViseme`, `setMouth`, `listen`, `drop`, `think`, `setPresence`, `emphasis`, `applyState` (incl. `thinking` / `presence`), `update(t:dt:) → HeadOffsets` | | `Presence` · `HeadFrameRate` | `.online / .asleep(z:) / .party(tempo:)`; `.full / .reduced / .low` (60 / 30 / 15 fps) | | `Listening(tempo:mouth:intensity:notes:)` | the listening state — a nod on the beat of `tempo` bpm, notes rising, over any face; `mouth` `.closed / .hum / .sing / .lipSync`, `intensity` 0…1; `notes: false` keeps the nod and hides the notes; `Listening.label` for the host's chip | | `AvatarPerformanceInput` · `AvatarPerformanceSnapshot` | a host playing the head: one main-actor object, sampled right before every frame's tick, answering gaze, held mouth, absolute beat + transport generation, and blink and drop sequences | | `SingingVoice` · `MouthParams(energy:bliss:)` | an expressive sung mouth from a held note (attack, hold, trill, a vowel, eyes closing on a long hold); `energy` brings the brows, eyes and head along, `bliss` shuts the eyes behind a blink | | `SpeechVoice` · `Viseme` (8.1) | a plain lip-sync mouth for a recorded voice: the host's loudness (0…1) opens the jaw, its viseme (`.rest / .open / .round / .wide / .shut`) shapes the lips, eased, frame-rate independent; `nil` viseme and quiet releases | | `LiveVisemes(sampleRate:playoutOffset:)` (MinimojiLive, 8.4) | a `Viseme` and a loudness guessed from a live voice's PCM for `SpeechVoice`: `process(_:at:)` on the SDK's delivery queue, `now(at:)` a pure read on the main thread, `playoutOffset`, `reset()` | | `AvatarPerformanceDriver` · `BeatCrossing` | what applies a snapshot to a `FaceLife` (for a renderer of your own), and a no-backlog "every N beats" detector for the host's side | | `FaceLife.setBeat(_:generation:)` · `blink()` · `isBlinking` | the animator's side of the same: the host's beat for the coming frame, an explicit blink event | | `ListeningWear.headphones(on:catalog:)` (MinimojiEditor) | the worn document (`identity ⊕ {extra: headphones}` under the avatar rules) and `applied`; the identity is never changed | | `FaceLife.faceIds` | `idle neutral happy laugh sad angry furious surprised worried scared dead smug joy sleeping crying thinking` | | `HeadGesture` · `HeadCue` · `GestureEndHandler` (9.4) | the authored one-shots by name, including explicit `.headShake` (since 9.16.0); generated `face`, `duration`, `name`; a cue value for the views (`.gesture(_)` / `.forMood(_)` + `generation`, fired once per distinct value); what `onGestureEnd:` / `FaceLife.onPerformanceEnd` call with (the gesture, `cancelled`) | | `FaceLife.perform(_:)` · `perform(for:)` · `performanceFace` | a performance on cue over the face as it is (never changes `face`); one of a mood's three, no immediate repeat; which face's track is playing | | `ExpressionPick.gesture` · `gestureWord` · `pickExpression(for:gesture:)` (MinimojiService, 9.4) | the picker's gesture beside the face: a `HeadGesture` to cue (nil when none fits or unknown to this build) and the word it read | | `HeadEntity(document:meshes:gaze:drift:performances:emotes:random:)` | entities + a `FaceLife`; `tick(t:dt:)` | | `voiceLevel:` (7.1, every head view) | a host's live loudness 0…1 when it is **already SwiftUI state**: the sound arcs follow it, kept fresh each frame, nil releases at once; a live `performance:` source that names its own level wins. A level from an SDK that doesn't publish changes (LiveKit, `AVAudioEngine`) goes through a `performance:` source's `voiceLevel` or `FaceLife.setVoiceLevel(_:)`, or the arcs freeze between re-renders | | `TalkStyle` · `FaceLife.talkStyle` · `setVoiceLevel(_:)` · `compact` | the talking look: `.mouth` (default), optional `.arcs` or `.lines`; a host's loudness 0…1 (a 0.3 s TTL, like a viseme); the small-head look. `talkMeter` reads the lit ring count | | `MinimojiHeadView(document:expression:look:talking:rate:drift:performances:emotes:listening:performance:thinking:presence:talkStyle:)` | the SwiftUI view; honors Reduce Motion and also accepts an explicit performance opt-out, and `emotes: false` to keep cues without the automatic emote (the stage's `.minimojiHead(…)` takes it too); `MinimojiMetalHeadView` takes the same, plus `isActive:` to pause its render loop, `frameRate:` to draw at 30 or 15 fps and `bursts:` (default true) for the emote bursts round a mood change | ## Tests ```bash swift test --package-path packages/swift/MinimojiKit # or, from the root: pnpm test:swift xcodebuild -scheme MinimojiKit -destination 'generic/platform=iOS Simulator' build pnpm kit:publish # split + push + tag hypersocialinc/MinimojiKit ``` `Sources/MinimojiEditor/Resources/` (the catalogue, the thumbnails) is generated by `pnpm avatar:sync` / `pnpm avatar:thumbs` and tracked, so the package builds without Node or Chrome; `pnpm avatar:check` fails when it is stale. Fixtures: `head-small.glb` (a buzz-cut head with no accessories at detail .2 — the smallest bake that still has skin, hair, both eyes and every mouth plate) and `face-trace.json` (the parity trace; regenerate with `node packages/engine/dev/face-trace.mjs` whenever `gface.js` changes — the engine's own test suite checks the committed fixture still matches), and `performance-trace.json` (a host playing the head through `ginput.js` at 30/60/120 Hz with a held note, a seek, a stall, garbage input, a stale clock and a parked view; the scenario is data, so `PerformanceParityTests` replays the file; regenerate with `node packages/engine/dev/performance-trace.mjs`). `Tests/MinimojiLiveTests/Fixtures/viseme-trace.json` is `LiveVisemes`' parity trace: 16-bit PCM at 48 and 16 kHz, every `process` call with its arrival time (on time, in bursts, across a gap) and every `now` read with the engine's answer; regenerate with `node packages/engine/dev/viseme-trace.mjs` whenever `gvisemes.js` changes. ## Why it is shaped this way (measured, not guessed) Each simpler shape was tried on the iPhone 17 simulator and failed a frame-diff or a log: - **RealityKit renders on demand.** A head driven from `SceneEvents.Update` froze on its first frame — the scene only ticks when RealityKit itself is animating something (Apple DTS: "behaving as expected"). The sanctioned fix is a `System` querying with `updatingSystemWhen: .rendering`. - **One `RealityView` per head** ticked (logged ~40 fps each) but only the first view on screen redrew; **one per screen** left a fresh instance black on a tab switch — the Simulator renders one RealityView properly. - **RealityView's `update` closure** is not called reliably on iOS: a head built after a tab switch stayed off screen for the whole visit. - **SwiftUI preferences across a `TabView`** did not reach the root, and a hidden tab's rows fired `onAppear` again with no `onDisappear`. Only the UIKit window and frame can be trusted for where a mark is — and during a tab transition the outgoing tab is still in the window, so a probe also has to be unhidden and unfaded up its chain and on the tab bar controller's selected tab, or its heads ghost over the new tab. ## Known gaps (the spike's edges) - **Bars**: the stage overlays the whole window, so a head scrolled under a nav or tab bar draws over its blur (the web's `surfaces` problem). - **A device**: nothing here has run on a physical iPhone; the Simulator's single-RealityView limit is what forced the single stage, and a device may also be fine with more. - **Sheen**: RealityKit's sheen is energy-conserving and takes its tint out of the base colour, so the engine's rosy skin sheen rendered as a teal face at full strength; it is applied at a quarter. The clearcoat reads as on the web. - The sleeping Z's depth-test overlay is not ported (the furious flush is: `FaceLife.glow`, on the skull's emissive in both renderers). - `PhysicallyBasedMaterial` / entities must be made on the main thread (RealityKit asserts); only mesh generation goes off it. - One stage per window means a head cannot appear inside a `.sheet` or another presentation; that is what `MinimojiMetal` is for. ## MinimojiMetal — the other renderer (spike) The same `HeadDocument` + `FaceLife`, drawn by our own `MTKView` instead of RealityKit: `HeadRenderer` (pipelines and GPU meshes shared per process, a PBR with clearcoat + the studio gradient as environment, MSAA 4×, premultiplied transparency) and `MinimojiMetalHeadView`, a head as an ordinary view — one per list row, inside a sheet, under a panel — with a 60 fps loop that is ours: no on-demand rendering, no stage, no probes. ```swift import MinimojiMetal MinimojiMetalHeadView(document: doc, expression: "happy", talking: true).frame(width: 96, height: 96) ``` Heads default to `interactionMode: .decorative`: idle animation, expressions, external `look`, talking, and audio-driven mouths still run, but the head passes taps and drags through to its parent. Use this mode in `List` and `ScrollView` rows and inside buttons. Give the **parent label** a content shape when the whole row should be tappable: ```swift List(people) { person in Button { select(person) } label: { HStack { MinimojiMetalHeadView(document: person.head, interactionMode: .decorative) .frame(width: 96, height: 96) Text(person.name) Spacer() } .contentShape(Rectangle()) } .buttonStyle(.plain) } ``` For editor or standalone previews, opt into `interactionMode: .interactive`: ```swift MinimojiMetalHeadView(document: doc, interactionMode: .interactive) .frame(width: 280, height: 280) ``` Interactive heads capture drag-to-look and tap-to-talk. Releasing a drag restores the supplied `look` (or autonomous gaze when nil). Changing modes clears transient drag/talk state without restarting the renderer. Existing callers that relied on these gestures must explicitly opt in; `MinimojiEditorView` and the harness Stage already do so. Parent editor gestures may override taps and long presses as before. The offline simulator regression suite in `apps/swift-harness/InteractionTests` verifies scrolling starting over a head, parent-button taps, idle animation, mode changes, and dragging in the actual editor; see the [harness README](../../../apps/swift-harness/README.md). ### Optional face framing and soft viewport edges The default `framing: .bounds` is unchanged. Pass `framing: .face` to keep face size and center independent of hair and accessory bounds; tall hair/hats may crop. Visible listening notes still widen the camera to contain their path. `Listening(notes: false)` retains the normal face framing. This option is currently specific to the Metal renderer. ```swift MinimojiMetalHeadView(document: head, framing: .face) .frame(width: 128, height: 128) .modifier(MinimojiSoftEdges()) // Use the same framing for a cached still. Framing is not part of avatar identity. let stillHead = MetalHead(document: head, framing: .face) let texture = try renderer.render(stillHead, width: 128, height: 128) ``` `MinimojiSoftEdges` is an opt-in SwiftUI modifier: apply it once after sizing both the live view and the host's cached image view. It fades the outer 9% and uses a material blur there; the center stays sharp. Reduce Transparency removes the material blur while retaining the alpha fade. Metal textures do not bake in this UI treatment. There is no second animated head, but material and mask compositing add GPU work; measure the host's actual scrolling/list workload on device. A still matches the live camera and pivot, but its facial pose only matches when the same animation state is supplied. For the built-in editor (including Bunch's editor integration), configure its nested primary, compact and AI proposal previews together: ```swift MinimojiEditorView(service: service, doc: $avatar, head: $head, imports: .kit, avatarPresentation: .init(framing: .face, softEdges: true)) ``` `avatarPresentation` defaults to bounds framing and no soft edges. Each avatar viewport applies the modifier once after sizing; toolbars, controls, status text and the rest of the editor remain outside the fade. Do not wrap the whole editor in `MinimojiSoftEdges`. This also covers retained/loading and revised AI proposals. Hosts with their own rendered-image cache must include framing in that cache's key (or invalidate old bounds-framed images). GLB/document/geometry caches are unchanged. No rebake, schema migration or bake-version change is needed. `HeadRenderer.render(_:width:height:)` draws a head into a texture headlessly — the pixel tests in `Tests/MinimojiMetalTests` run on the Mac's GPU (`MINIMOJI_SHOTS=` writes what they rendered). Verified on the iPhone 17 simulator: ten heads alive in a `List`, a talking head inside a `.sheet` over the stage. The harness's toolbar menu (or `-metal`) picks the renderer. Still to do: match the web's gloss more closely (the key highlight and the sky term read a touch hot), metallic (carried, unread — no head is metal), the sleeping Z's overlay, and a device. If the renderer cannot be made (no device, a shader that does not compile) every view shows the error instead of an empty square, and `SharedRenderer.error` says why.