Artificial Assemblies Seminar

Program: Augmented Reality Pavilion
Completed: 2024
First Exhibited: December 18th, 2024
Software Used: p5.js, Stable Diffusion, ComfyUI
Location: The site of the user
Role: Designer

Your Pavilion empowers people to use a simple interface on their phones to design their own architectural interventions in any space that they wish. From just an image of a space—be it an intriguing alleyway, a large room, or an expansive landscape—and a text prompt and sketch prompt, users of this program are empowered to design interventions or pavilions that are personal and unique, and which they can virtually inhabit via augmented reality (AR).

User Experience

Your Pavilion currently exists as a p5.js sketch that can be accessed from a laptop running a ComfyUI script. The sketch connects to a phone’s camera via a third-party application, and the user can sketch their desired pavilion design on top of the camera view rendered on the laptop. The user also submits a prompt describing in words what their imagined pavilion looks like. Your Pavilion also can run on a smartphone directly, but since smartphones cannot run ComfyUI its outputs currently must be saved to a Google Drive that the laptop running ComfyUI must access.

Inputs and Workflow

Once the user has pressed a button to finish their drawing and prompt, the p5.js sketch downloads four different files that are used as inputs in the ComfyUI script to generate the 2D and 3D representations of the pavilion: the prompt as a text file, the sketch rendered with an outline but no fill, the sketch rendered with a fill but no outline, and the background image the user sketched over. The ComfyUI script takes these four inputs and uses them to generate an image of the pavilion that shows it as part of the original scene that the user photographed. From there, this output image is used as an input in the second portion of the ComfyUI script which makes a high-resolution, textured 3D GLB file from the 2D image.

Process at Work

The filled sketch is used primarily to mask out the part of the photographed surroundings that will hold the pavilion. The sketch outline guides the prompt to include details drawn in by the user as part of the final render. Once the 2D render is finished, the background is removed from the render and the resulting image is input into an algorithm run by Tripo AI to turn it into a GLB mesh.

Augmented Reality

Since the ComfyUI workflow’s 3D output is a GLB file, it can be directly viewed in
Augmented Reality (AR) on Android devices. Apps such as XR Viewer can be used to view the files, allowing people to pan around and see their pavilion from any angle.

Sample Pavilions