Ustwo Visual Engine Documentation
Complete guide to using Ustwo Visual Engine — a real-time visual performance tool built for live shows, installations, and creative experimentation. Learn about all features, controls, and the technology behind it.
Overview
Ustwo Visual Engine is a browser-based visual performance tool that combines real-time 3D graphics with audio analysis and motion tracking. It's designed for VJs, visual artists, and anyone who wants to create reactive visual experiences.
20+ Visual Scenes
From particle systems and geometric shapes to generative shaders and interactive installations.
Audio Reactive
Connect microphone or audio files to sync visuals with music frequencies in real-time.
Motion Control
Use your camera to control scenes with body movements and hand gestures via AI.
Live Camera
Real-time webcam input with multiple visual styles — displacement, particles, silhouette and more.
Post Effects
30+ GPU-powered effects — bloom, glitch, fluid distortion, pixel sort, audio displacement and more. Searchable!
HD Recording
Record performances in high quality WebM format (VP9, 25 Mbps, 60fps) with audio.
Matter.js Physics
2D physics engine with bouncing balls, stacking boxes, ragdolls and more. Audio-reactive gravity!
Quick Start Guide
Get up and running with Ustwo Visual Engine in under 2 minutes.
Step 1: Choose a Scene
Click on any scene icon in the left panel, or use keyboard shortcuts 1 through 0 to quickly switch between the first 10 scenes. Use arrow keys ← → to navigate through all available scenes.
Step 2: Enable Audio Input
Click the microphone icon in the bottom toolbar to enable audio reactivity. You can use:
- Microphone — Capture ambient sound or music from speakers
- Audio File — Load an MP3/WAV file for synced playback
- System Audio — Route computer audio (requires virtual audio cable)
Step 3: Customize Parameters
Double-click on a scene to open its control panel. Each scene has unique parameters you can adjust in real-time — speed, color, intensity, and more.
Step 4: Add Effects
Scroll down in the left panel to find Post FX effects. Click to enable/disable effects like bloom, chromatic aberration, film grain, and more.
Step 5: Go Fullscreen
Press F or click the fullscreen button to enter presentation mode. Press H to hide the UI for a clean visual output.
💡 Pro Tip
Use Space to pause/resume animation. Great for finding the perfect frame when recording or taking screenshots.
User Interface
Understanding the layout will help you work faster and more efficiently.
Top Bar
- Logo — Click to return to default state
- Mode Tabs — Switch between THREE.JS and MAPPING modes
- Status Indicators — FPS counter, Audio status, Motion status
Left Panel — Scenes & Effects
- Scene Grid — Visual icons for all available scenes
- Post FX — Post-processing effects (bloom, vignette, etc.)
Right Panel — Controls
- Audio Controls — Microphone, file upload, volume
- Camera Controls — Motion tracking settings
- Global Parameters — Speed, intensity, color palette
Bottom Toolbar
- Record Button — Start/stop video recording
- Fullscreen — Enter fullscreen mode
- Settings — Additional options
Visual Scenes
Ustwo Visual Engine includes 20+ unique visual scenes, each with its own aesthetic and customizable parameters. Scenes are organized into categories.
3D Geometry Scenes
Generative Shader Scenes
Shader Scenes
Simulation Scenes
Media Scenes
Interactive Scenes
Matter.js Physics Engine NEW
The Matter.js mode adds 2D physics simulation to your visuals. Create interactive physics-based scenes with gravity, collisions, and constraints.
🎮 How to Access
Switch to Matter.js mode by clicking the "MATTER.JS" tab in the top navigation bar. This replaces the Three.js canvas with a 2D physics world.
Available Physics Scenes (17 total)
Physics + Post FX
Matter.js scenes can be combined with Three.js post-processing effects! The physics canvas is composited with the effect chain, so you can add bloom, glitch, chromatic aberration, and more to your physics scenes.
Kokeshi Choir
The Kokeshi Choir is an interactive audio-visual installation featuring 5 Japanese-style dolls that sing when you press keys or move in front of the camera.
Playing the Choir
Use your keyboard to play the dolls like a musical instrument:
| Key | Voice | Pitch Offset | Position |
|---|---|---|---|
| A | Soprano | 0 (root) | Far left |
| S | Mezzo | +4 (major 3rd) | Left |
| D | Alto | +5 (perfect 4th) | Center |
| F | Tenor | -12 (octave down) | Right |
| G | Bass | +7 (perfect 5th) | Far right |
Motion Control
When camera is enabled, you can control the dolls by moving left/right. The screen is divided into 5 horizontal zones, each triggering a different doll.
Voice Samples
🎤 About the Voice Samples
The choir voices were generated using ElevenLabs AI with the Sound Effects feature. Each voice was created with prompts like:
- "female soprano choir voice singing long sustained ah vowel, ethereal, pure tone"
- "female alto voice singing sustained note, dark rich timbre, choir style"
- "male bass voice deep sustained choir note, rich low tone, gentle"
Samples are stored in choir_voices/ as WAV files (44.1kHz, stereo).
Technical Implementation
The audio playback uses the Web Audio API with AudioBufferSourceNode for sample playback. Key features:
- Pitch Shifting — Uses
playbackRateto shift pitch by semitones - Looping — Samples loop seamlessly while key is held
- Smooth Envelope — Fade in (0.15s) and fade out (1.5s exponential)
- Recording Support — Audio routed to MediaStreamDestination for video recording
Visual Elements
- Doll Textures — PNG images mapped to PlaneGeometry
- Animated Mouth — CircleGeometry that scales when singing
- Smoke Effect — Particle system using texture billboards
- Vignette — Shader-based gradient fade at bottom
Audio Reactivity
Ustwo Visual Engine analyzes audio input in real-time to drive visual parameters. This creates a synchronized audio-visual experience.
Audio Sources
- Microphone — Click mic icon to capture from default input device
- Audio File — Upload MP3/WAV/OGG files for playback
- Line Input — Connect external audio interface
How It Works
Audio Parameters
| Parameter | Description | Range |
|---|---|---|
| Bass | Low frequencies (20-250 Hz) | 0.0 - 1.0 |
| Mid | Mid frequencies (250-4000 Hz) | 0.0 - 1.0 |
| High | High frequencies (4000-20000 Hz) | 0.0 - 1.0 |
| Volume | Overall loudness (RMS) | 0.0 - 1.0 |
| Beat | Beat detection (onset) | 0 or 1 |
Motion Control
Use your webcam to control visuals with body movements and hand gestures. Motion tracking runs locally in the browser using AI models.
Enabling Motion Control
- Click the camera icon in the toolbar
- Allow browser access to webcam
- Wait for "MOTION" indicator to turn green
- Move within camera view to control visuals
Tracking Modes
⚡ Performance Note
Motion tracking uses significant CPU/GPU resources. For best performance, use a modern device and close unnecessary browser tabs.
Post-Processing Effects
Post-processing effects add the finishing touches to your visuals. These GPU-based effects are applied after the 3D scene is rendered, allowing for cinematic looks and creative distortions.
Available Effects
New Effects NEW
Searching Effects
Use the search field at the top of the Post FX panel to quickly filter effects. Type any part of an effect name (e.g., "glitch", "blur", "pixel") and the list will instantly filter to show only matching effects.
Combining Effects
Multiple effects can be enabled simultaneously. They are applied in order, so the combination can create unique looks. Popular combinations:
- Cinematic — Bloom + Film Grain + Vignette
- Cyberpunk — Chromatic + Glitch + Bloom
- Dreamy — Bloom + Trail + Vignette
- Retro — Pixelate + Film Grain + RGB Shift
- Psychedelic — Kaleidoscope + Trail + Chromatic
⚡ Performance
Each effect adds GPU processing overhead. On lower-end devices, limit active effects to 2-3 for smooth performance. Bloom is the most demanding effect.
Keyboard Shortcuts
Master these shortcuts to control Ustwo Visual Engine like a pro.
Scene Control
| Shortcut | Action |
|---|---|
| 1 - 9, 0 | Switch to scene 1-10 |
| ← → | Previous / Next scene |
| Space | Pause / Resume animation |
View Control
| Shortcut | Action |
|---|---|
| F | Toggle fullscreen |
| H | Hide / Show UI |
Recording
| Shortcut | Action |
|---|---|
| R | Start / Stop recording |
Audio
| Shortcut | Action |
|---|---|
| M | Mute / Unmute audio |
Kokeshi Choir
| Shortcut | Action |
|---|---|
| A | Play Soprano (leftmost doll) |
| S | Play Mezzo |
| D | Play Alto (center doll) |
| F | Play Tenor |
| G | Play Bass (rightmost doll) |
Recording & Export
Ustwo Visual Engine can record your performances as high-quality video files with synchronized audio.
How to Record
- Click the record button (●) in the bottom toolbar, or press R
- The button turns red and shows elapsed time
- Perform your visuals
- Click again to stop — video downloads automatically
Recording Specifications
| Property | Value |
|---|---|
| Format | WebM |
| Video Codec | VP9 |
| Audio Codec | Opus |
| Video Bitrate | 25 Mbps |
| Frame Rate | 60 FPS |
| Resolution | Canvas size (usually 1920×1080) |
Technical Implementation
// Capture canvas as stream
const canvasStream = canvas.captureStream(60);
// Add audio if available
const audioDestination = audioCtx.createMediaStreamDestination();
analyser.connect(audioDestination);
// Combine video + audio
const combinedStream = new MediaStream([
...canvasStream.getVideoTracks(),
...audioDestination.stream.getAudioTracks()
]);
// Create recorder
const recorder = new MediaRecorder(combinedStream, {
mimeType: 'video/webm;codecs=vp9,opus',
videoBitsPerSecond: 25000000
});
📁 File Location
Recordings are saved to your browser's default download folder as
vj-recording-YYYY-MM-DDTHH-MM-SS.webm
Converting to Other Formats
WebM files can be converted to MP4 or other formats using tools like:
- FFmpeg —
ffmpeg -i input.webm -c:v libx264 output.mp4 - HandBrake — Free GUI video converter
- CloudConvert — Online conversion service
Settings & Configuration
Fine-tune Ustwo Visual Engine to match your setup and preferences. Settings are accessible via the gear icon (⚙) in the toolbar or on individual items.
Audio Settings
Configure how audio input affects the visuals.
| Setting | Description | Range |
|---|---|---|
| Sensitivity | How responsive visuals are to audio input | 0 - 100% |
| Smoothing | Reduces jitter in audio response | 0 - 0.99 |
| Frequency Range | Which frequencies to focus on (bass, mid, treble) | Low / Mid / High / Full |
| Beat Detection | Enable automatic beat sync | On / Off |
Motion Settings
Configure motion tracking behavior when using webcam input.
| Setting | Description | Range |
|---|---|---|
| Tracking Mode | Which body part to track | Full Body / Hands / Face |
| Smoothing | Reduces jitter in motion data | 0 - 1 |
| Scale X/Y | Amplify or reduce motion range | 0.1 - 3.0 |
| Dead Zone | Minimum movement to register | 0 - 0.5 |
| Mirror | Flip the camera horizontally | On / Off |
Scene-Specific Settings
Each scene has its own customizable parameters. Double-click on a scene to open its control panel. Common parameters include:
- Speed — Animation playback rate
- Scale — Size of visual elements
- Color — Primary or accent colors
- Intensity — Effect strength
- Audio Link — Which audio band affects the parameter
Global Preferences
| Setting | Description |
|---|---|
| Auto-play Audio | Start audio file playback automatically |
| Show FPS | Display frames per second counter |
| High DPI Mode | Render at retina resolution (uses more GPU) |
| Background Color | Default scene background when not overridden |
Saving & Loading Presets
Your settings and scene configurations can be saved as presets:
- Export Project — Save all settings to a JSON file
- Import Project — Load a previously saved project
- Browser Storage — Settings persist in local storage automatically
💡 Pro Tip
Create multiple presets for different venues or music styles. You can quickly switch between them during a performance.
Technology Stack
Ustwo Visual Engine is built entirely with web technologies, requiring no plugins or installations. Here's what powers it under the hood.
Three.js
3D Rendering
WebGL 2.0
GPU Graphics
GLSL
Custom Shaders
Web Audio API
Audio Analysis
TensorFlow.js
AI Pose Detection
MediaPipe
Hand Tracking
MediaRecorder
Video Export
ES Modules
Modern JavaScript
Three.js & WebGL
Three.js is the primary 3D rendering library, providing a high-level API over WebGL for creating and displaying 3D graphics.
Core Components
- Scene — Container for all 3D objects, lights, and cameras
- Camera — PerspectiveCamera with adjustable FOV
- Renderer — WebGLRenderer with antialiasing and alpha
- Geometries — PlaneGeometry, SphereGeometry, custom BufferGeometry
- Materials — MeshStandardMaterial, MeshBasicMaterial, ShaderMaterial
Render Loop
function animate() {
requestAnimationFrame(animate);
// Update time
const time = performance.now() * 0.001;
// Update scene objects
sceneManager.update(time, {
audio: audioData,
motion: motionData,
palette: currentPalette
});
// Render
renderer.render(scene, camera);
}
Performance Optimizations
- Object Pooling — Reuse geometries and materials
- Frustum Culling — Don't render off-screen objects
- LOD — Level of detail for distant objects
- Instancing — Efficient rendering of many similar objects
Web Audio API
The Web Audio API handles all audio processing — from microphone input and file playback to FFT analysis and synthesis.
Audio Graph
// Audio context and nodes
AudioContext
├── MediaStreamSource (microphone)
│ └── AnalyserNode (FFT)
│ └── Destination (speakers)
│
├── AudioBufferSource (samples)
│ └── GainNode (volume)
│ └── MediaStreamDestination (recording)
│
└── OscillatorNode (synthesis)
└── BiquadFilter (formants)
└── GainNode (envelope)
FFT Analysis
Frequency analysis uses a 2048-sample FFT, providing 1024 frequency bins. Data is smoothed with a time constant of 0.8 for stable visuals.
Kokeshi Audio Synthesis
The Kokeshi Choir uses sample-based playback with pitch shifting:
- Samples loaded as AudioBuffer via fetch() + decodeAudioData()
- Pitch shifting via playbackRate (2^(semitones/12))
- Looping enabled for sustained notes
- Exponential fade out for natural release
TensorFlow.js & MediaPipe
AI-powered motion tracking runs entirely in the browser using pre-trained models from Google's TensorFlow and MediaPipe teams.
MoveNet (Pose Detection)
- Model — MoveNet SinglePose Lightning
- Input — 192×192 pixel video frames
- Output — 17 keypoints with confidence scores
- Speed — ~30 FPS on modern hardware
- Backend — WebGL for GPU acceleration
MediaPipe Hands
- Model — MediaPipe Hand Landmarker
- Output — 21 landmarks per hand
- Features — Pinch detection, rotation, gesture recognition
// Initialize pose detector
const detector = await poseDetection.createDetector(
poseDetection.SupportedModels.MoveNet,
{
modelType: poseDetection.movenet.modelType.SINGLEPOSE_LIGHTNING
}
);
// Detect poses
const poses = await detector.estimatePoses(videoElement);
GLSL Shaders
Custom GLSL shaders power many visual effects, from post-processing to generative art scenes. Shaders run on the GPU for maximum performance.
Post-Processing Effects
- Bloom — Glow effect using brightness threshold + blur
- Chromatic Aberration — RGB channel separation
- Vignette — Darkened edges
- Film Grain — Noise overlay for cinematic look
- Pixelate — Retro pixel effect
Example: Vignette Shader
// Fragment shader for soft vignette
uniform float fadeTop;
uniform float fadeBottom;
varying vec3 vWorldPos;
void main() {
float y = vWorldPos.y;
float alpha = 0.0;
if (y < fadeBottom) {
alpha = 1.0; // Solid black
} else if (y < fadeTop) {
alpha = 1.0 - smoothstep(fadeBottom, fadeTop, y);
}
gl_FragColor = vec4(0.0, 0.0, 0.0, alpha);
}
Generative Techniques
- Simplex Noise — Smooth procedural noise for organic effects
- FBM (Fractal Brownian Motion) — Layered noise for clouds/nebula
- Ray Marching — 3D rendering technique for complex shapes
- Domain Warping — Distorting coordinates for trippy effects