Latency Budgets for Interactive Walls: Tracing the Delay from Camera to Projector

Latency Is an End-to-End Property

Latency is the elapsed time between an event occurring in the installation space and the corresponding visible response. On an interactive wall, this may be the interval between a visitor raising a hand and a projected object moving in response. The delay is distributed across several components: camera exposure, frame delivery, image processing, pose estimation, application logic, rendering, graphics buffering, projector synchronization and display response. Each stage contributes to the final experience.

The important distinction is between the performance of an individual component and the responsiveness of the complete installation. A tracking model may process an image quickly while waiting for the next camera frame. A renderer may produce frames at a high rate while the display introduces synchronization delay. Optimizing one stage does not automatically improve the whole system. A useful latency budget assigns a maximum contribution to each stage and identifies which delays can be controlled through configuration.

The Camera and Capture Pipeline

Camera frame rate determines how frequently the system receives new observations. At 30 frames per second, a new frame is nominally available approximately every 33.3 milliseconds. At 60 frames per second, that interval falls to approximately 16.7 milliseconds. This does not mean the complete system has those latency values. Exposure time, sensor readout, internal buffering and transmission can add further delay before the application receives usable image data.

Exposure settings also influence responsiveness. A longer exposure can introduce motion blur when visitors move quickly, reducing the quality of pose estimation and potentially requiring additional processing. Automatic exposure can change behavior as lighting conditions shift. Camera interfaces and drivers may buffer frames, meaning that the application receives an older image even when its processing speed appears healthy. Configure capture settings deliberately and confirm that the application processes current frames instead of accumulating an increasingly delayed queue.

• Record the configured camera frame rate.
• Check exposure and readout behavior.
• Inspect driver and application buffering.
• Avoid processing stale frames when newer observations are available.

Tracking, Application Logic and Rendering

The computer vision stage converts image data into useful information such as person positions, pose landmarks or movement vectors. Processing time depends on image resolution, model complexity, the number of participants and the available computing resources. A more complex model may improve recognition in difficult scenes while consuming more time per frame. Reducing input resolution or limiting the number of active participants can lower computational demand, but these changes may also reduce spatial detail.

Application logic introduces another potential source of delay. Data may pass through separate processes, message queues or network connections before reaching the rendering engine. Unnecessary serialization, excessive logging and blocking operations can create irregular pauses. Rendering then adds its own workload, including simulation updates, geometry processing, shaders and image composition. A system should be designed to avoid waiting for every optional operation before displaying the next frame. Critical interaction data should have a predictable path to the renderer.

Projector Synchronization and the Visible Response

The projector is the final stage of the response chain, but its role is often underestimated. Display refresh rate determines when a newly rendered frame can become visible. Synchronization settings can prevent tearing, although they may introduce additional waiting or frame queuing. Projector processing features, scaling and image correction can also affect the display path. The exact behavior varies by hardware and configuration, so the projector's technical documentation is essential when estimating the final delay.

Consider a simplified example in which a camera and projector both operate at 60 Hz. Their nominal frame intervals are approximately 16.7 milliseconds, but a captured event may arrive just after the camera's sampling point and wait almost a full frame before the next observation. A similar alignment issue can occur at display refresh. These timing effects are in addition to tracking and rendering work. A fast application can therefore still feel delayed because of the position of the event within the capture and display cycles.

• Confirm the projector's actual input and output refresh settings.
• Check image scaling and processing features.
• Review vertical synchronization and frame queue behavior.
• Keep the rendering resolution appropriate to the installation.

Building a Practical Latency Budget

A latency budget turns a general performance target into component-level limits. For example, an illustrative target of 100 milliseconds might allocate 20 milliseconds to capture and delivery, 30 milliseconds to tracking, 10 milliseconds to application logic, 20 milliseconds to rendering and 20 milliseconds to display timing. These figures are planning allocations, not guaranteed hardware performance. Actual contributions depend on the camera, software, computer, graphics configuration and projector.

Measure the complete path as well as the individual stages. Software timestamps can identify when a frame enters capture, when tracking completes and when rendering submits a new image. However, a timestamp at the end of rendering does not establish when light leaves the projector. End-to-end optical measurement requires a method that observes both the physical event and the visible response, such as a synchronized high-speed recording or an appropriate photodiode arrangement. Account for clock synchronization and instrument precision when interpreting results.

• Set an end-to-end response target.
• Allocate time limits to every pipeline stage.
• Record average latency and high-percentile delays.
• Distinguish software completion from visible projection.
• Repeat measurements under expected occupancy and lighting conditions.

Latency Spikes and Installation Failsafes

Average latency can hide the interruptions that visitors notice most. A system may respond smoothly for several seconds and then pause when multiple people enter the camera view, a background task consumes processing capacity or a shader becomes unusually expensive. These intermittent delays are often described through latency percentiles. The 95th percentile, for example, represents a value at or below which 95 percent of recorded observations fall. Tracking this value alongside the average provides a better view of occasional slowdowns.

A resilient interactive wall needs a defined response to overload. If tracking becomes too slow, the application can discard stale frames, reduce optional visual effects or switch to a simplified interaction model. If the camera disconnects, the installation can move to an autonomous visual sequence. If the projector stops receiving frames, the application should avoid continuing to accumulate obsolete work. Every fallback should have a clear activation condition, a visible operational state and an authorized recovery procedure.

• Normal mode: maintain full interactive tracking and rendering.
• High-load mode: reduce visual complexity and nonessential processing.
• Stale-frame mode: discard outdated observations and use the newest available frame.
• Tracking-delay mode: reduce individual responsiveness and preserve stable visual content.
• Camera-disconnection mode: transition to an autonomous loop.
• Display-failure mode: stop dependent interaction processing and alert the responsible operator.

Commissioning and Long-Run Reliability

Latency can change after an installation opens. Background software updates, changes to camera exposure, increased visitor density, thermal limits and altered projector settings can affect timing. A commissioning document should record the approved camera configuration, processing resolution, application version, graphics settings, projector refresh rate and intended operating mode. These details give technicians a reference when performance changes. Store the configuration with the installation documentation and restrict changes to personnel authorized to maintain the system.

For an installation expected to run eight hours a day, include temperature and resource monitoring in the operating plan. A computer that performs well when cool may reduce its clock speed under sustained thermal load. Inspect ventilation, dust accumulation and cable connections, and ensure that restart procedures restore the correct application state. The practical next step is to create a pipeline diagram, assign a time allowance to each component and establish the end-to-end response target before final hardware procurement.

Questions readers ask