Cats by the Window

A Small World Behind the Screen

Cats by the Window, developed as Hiding Cat / 窗里有猫, turns the monitor into a window onto a miniature world. Lean left, look from above, or move closer to reveal cats behind furniture, then click the visible part of a cat to collect it. Five places—the living room, bedroom, study, cat café, and park—each contain five discoveries.

I developed the project independently through an AI-assisted workflow. My focus was connecting the interaction, art pipeline, and rendering: the perspective needs to feel stable, the scenery needs to create useful occlusion, and each hiding place needs to remain discoverable after an art change.

Work in progress. This case study reflects the implementation and project captures through October 13, 2026.

Making the Screen Behave Like a Window

The core interaction uses head-coupled perspective. A normal RGB webcam estimates the viewer's eye position; an asymmetric projection matrix then reveals different parts of the room as the viewer moves. Camera rotation stays fixed, so the screen plane remains anchored while objects at different depths produce motion parallax.

I update both the projection matrix and its inverse from the same eye position. This keeps rendering and mouse picking consistent: if furniture visually covers a cat, it also blocks the click. Moving the camera alone or orbiting around the room would break the fixed-window effect.

Calibration relates the visible game width and viewing distance to scene units. The tracker uses the midpoint between the eye corners for lateral movement, and the change in eye separation for approximate forward and backward movement. Separate One Euro filters stabilize each axis without making ordinary head movement feel excessively delayed.

Camera-mode capture showing the fixed room opening and the paper eye mirror. This project QA image uses a test video input to exercise the real tracking model.

This is a calibrated approximation from a single camera. Large head rotations, camera placement, lighting, and individual proportions affect accuracy. The experience provides motion parallax on an ordinary display; it does not produce separate stereoscopic images for the two eyes.

Keeping Tracking Responsive and Optional

MediaPipe inference runs locally in a dedicated Web Worker. The main thread continues rendering while transferable video frames are processed, with at most one inference frame in flight to avoid a growing queue. GPU initialization falls back to CPU when needed, and background tabs stop sending frames.

Camera use is an explicit choice. Players can instead select manual exploration and use dragging, touch, or the keyboard to move the viewpoint. Losing tracking holds the last view; changing to manual mode stops the camera tracks and terminates the inference worker. The small eye mirror communicates position and tracking loss without displaying the player's camera image.

The application does not upload camera frames or retain face features. Progress stays in the browser, and the interface supports English and Chinese. These decisions keep the prototype practical to try while preserving the intended physical interaction for players with a webcam.

Authoring Hiding Places in Blender

The living room establishes the procedural scene vocabulary in Three.js. The bedroom, study, café, and park are authored through Python tools running in Blender via MCP, then exported as GLBs. Shared builders create furniture, frames, plants, and paper details while preserving editable Blender source files.

Bedroom gameplay capture showing layered furniture and several partially obscured hiding places. The model, material response, and click occlusion are evaluated together in the browser.

Each scene carries five authored markers with a cat slot ID, position, scale, pose, and hiding-place description. The runtime reads those markers directly from the exported scene and instantiates the matching characters. Moving a hiding place in Blender therefore updates the game without maintaining a second coordinate list in TypeScript.

The material contract is similarly small: named paper materials in the GLB are replaced with the shared runtime paper-fiber material. This lets different scenes use one consistent rendering treatment while retaining their own geometry and palette.

Tactile Art with Functional Depth

The art direction began as an Ink & Paper miniature theater: cream paper, muted green, warm coral, printed details, and visible material thickness. Furniture edges, cutout leaves, window recesses, and layered scenery all contribute actual depth, which makes them useful for the perspective mechanic.

The characters evolved from layered paper shapes through a faceted version to smoother 3D cats. Earlier versions remain archived with comparison views. The current library contains nine characters in five poses, so a cat can curl into a small recess, peek around an edge, or sit behind a larger object without forcing every hiding place to use the same silhouette.

Character review showing the loaf, curled-sleeping, and peeking poses. These are actual model previews from the project's character review tool.

Blender exports head, tail, chest, ear, and forelimb control nodes alongside a blink morph target. Runtime code schedules breathing, glances, ear motion, tail movement, and the response to being found. Pose-specific behavior keeps resting cats grounded, and reduced-motion preferences return them to a static pose.

The park extends the same material language into an outdoor layout with a bridge, fountain, planting, and layered hiding spaces.

Checking That the Puzzle Still Works

Art changes can accidentally expose a cat from the starting view or make it impossible to reach. I use visibility scans across the allowed viewpoints to check the same exported GLBs that the game loads. Browser checks then click all 25 cats, verify that furniture blocks hidden clicks, and exercise completion, level changes, and independent saved progress.

A static bounding-box tree narrows the furniture candidates for picking, but the final intersection still uses the actual mesh triangles. This preserves the relationship between visible geometry and interaction instead of introducing invisible blockers around detailed furniture.

Tracking checks use moving test imagery with the real MediaPipe model rather than fabricated landmark outputs. They cover calibration, movement direction, depth changes, tracking loss, and camera cleanup. Those checks establish repeatable behavior; they do not replace hands-on evaluation of latency and comfort across real cameras and lighting conditions.

The result is a playable five-level prototype and a reusable content workflow in which scene composition, character pose, projection, and picking can be reviewed together. Further work centers on broader device testing and refining how naturally the window responds to each player's movement.