Building 3D Worlds with Voice & Agents.
How we co-authored a live 3D sandbox game in plain spoken English.
Voice to Scene in Real Time.
- Live Scene Evaluation: Gemini returns Three.js code blocks that run directly against the active WebGL scene while the game keeps running.
- Frame-Loop Hooks: A global
window.customTicksarray lets generated code animate meshes and check distances every frame. - Go Proxy: A lightweight Go router handles API calls, cleans payloads, and enforces rate limits.
- Fixed JSON Schema: Every response conforms to
{ say, js }so spoken feedback and 3D updates land together.
Meet Gemma.
My daughter Gemma is five. She built the entire 3D game by talking to Antigravity.
- Zero syntax: She didn't write a single line of JavaScript, HTML, or CSS.
- Spoken specs: Talked directly to the agent to shape worlds, speed boosts, space flight, and rules.
- Physics opinions: Insisted the car go higher in space and feel lighter on the Moon.
"I want it to do whatever I want."
Gemma's session was a nonstop stream of ideas: new physics, new colors, and animal shapes on the fly.
- Rapid loops: From a police car to space flight, Moon gravity, and balloons that pop on impact.
- The laptop problem: Antigravity ran in the IDE on my laptop, so every new idea locked me out of my own computer.
"Why don't we put Gemini directly inside the game?"
Move the loop out of my IDE and into the running game through an on-screen voice HUD. She got her own tool. I got my laptop back.
Click each step to trace a spoken command into the running Three.js scene.
Capture intent
Web Speech API turns what she says into live subtitle text.
Write the code
Gemini returns a fixed JSON schema: a spoken reply and Three.js code.
Run it live
The game evaluates the code against the live scene while she keeps playing.
"Turn the car into a tiger."
The actual code Gemini wrote and injected into the running game when Gemma asked for tiger mode.
while (window.carGroup.children.length > 0) {
window.carGroup.remove(window.carGroup.children[0]);
}
// 2. Build tiger body
const body = new THREE.Mesh(
new THREE.BoxGeometry(1.6, 1.0, 3.2),
new THREE.MeshStandardMaterial({ color: 0xffaa00 })
);
window.carGroup.add(body);
// 3. Add stripes along the back
for (let i = -1; i <= 1; i++) {
const stripe = new THREE.Mesh(...);
window.carGroup.add(stripe);
}
Making Things Happen.
Changing shapes wasn't enough. Gemma wanted behaviors, so we let Gemini push functions into the render loop.
- window.customTicks: Runs generated logic on every frame.
- Proximity checks: Triggers events when the player drives near an object.
- Canvas textures: Draws 2D signs like "CAR WASH" onto 3D billboards on the fly.
Let's Play.
Talk to the game and watch Gemini rewrite the 3D scene while you drive.
Play Smoosh Patrol →
From Writing Code to Directing Intent.
A five-year-old programmed a 3D game by talking. For those of us who build software for a living, the work shifts to reviewing what agents wrote and setting the boundaries.