Video to digital twin, with no CAD
This project turns a publicly available news video of a food bank into a working digital twin in three steps, using tools that exist today: a world model reconstructs space from the video, Blender builds the full 3D facility, and the twin runs in NVIDIA Omniverse and the browser on live data from AWS. No CAD drawings, laser scans or site visits were used.
Why it matters
A digital twin usually starts with CAD drawings or a laser survey, then weeks of modelling before any data is connected. Here the building comes from footage that already existed. World models fill in metric depth, a script fills in the facility, and OpenUSD lets the same scene run in a browser, in Omniverse and in Blender. The cloud spend to capture and model a whole facility is in the tens of dollars (see Cost), so the real work moves to connecting data, which is where a twin earns its keep.
What is real today
| Part | Status | Details |
|---|---|---|
| World-model captures (3 zones) | Ran | World Labs Marble 2, 200 credits. Pipeline |
| Photoreal splats trained from the video (4 zones) | Ran | Local 3D Gaussian Splatting, 0 credits. Pipeline |
| Blender facility model | Ran | scripts/build_facility.py writes GLB and USDZ. Pipeline |
| AWS backend (S3, API Gateway, Lambda) | Live | Deployed in us-east-2. /telemetry, Data on AWS |
| Browser twin on GitHub Pages | Live | Reads the AWS API. Open it |
| Omniverse on an AWS GPU (EC2 g6.xlarge) | Rendered | Seven runs on real AWS GPU instances, about $1.34. Runs 6 and 7 rendered the designed facility in RTX, with the forklift driving in run 7; the splats rendered blank. Learnings |
| Real sensors on a food bank floor | Design only | Designed and priced, not installed. Cost |
The data behind the twin is simulated, and every API payload says so with simulated: true. See Provenance for what is captured, designed, published, estimated and simulated.
Representative, not a replica. The twin is a representative food bank inspired by the Central Texas Food Bank, built from a publicly available news tour by 25 News KXXV. Neither organisation was involved in this work or has endorsed it. Cost figures are directional: public list prices and one individual AWS account; enterprise discounts and quotas may differ.
Pages
- Pipeline: from video frames to world-model captures, trained splats and the Blender facility.
- Architecture: the AWS reference architecture (deployed), Azure and Google Cloud designs, and the contracts between steps.
- Data on AWS: the deployed API, its JSON contract, and the real-floor sensor path.
- Visualize: browser viewer, Omniverse on a GPU, Blender.
- Omniverse learnings: every GPU run, what failed, what fixed it, and a recipe that starts Kit on AWS.
- Run it: commands to reproduce every step.
- Cost: measured cost of this build and what a full facility might cost.
- Provenance: where every part of the twin comes from.