Summary

Volumetric video is a technology that captures three-dimensional (3D) scenes from multiple viewpoints simultaneously, often including people or objects, to create a lifelike, immersive experience for viewers. It uses a combination of cameras, sensors, and software to capture and process data from multiple perspectives, resulting in a 3D representation of the captured scene. Volumetric video enables viewers to move around and interact with the virtual environment, providing a more immersive and engaging experience than traditional video. The uncompressed data generated by volumetric video is very large and is dependent on the number of objects (usually people) present in a scene.  The uncompressed data rates range from 600 megabits to over one terabit per second. However, techniques have been developed to compress this data down to 30 to 80 megabits per person (or object) present in a scene.  Many of these techniques are company-proprietary and often tied to a particular capture solution. 

 

The Volumetric Format Alliance, a non-profit standards consortium, has published the Volumetric Player Format Specification.  The Specification was developed in collaboration with a number of the companies engaged in volumetric capture, compression, and large-scale compression technologies, with the objective of creating a Specification that defines interoperable techniques for the capture and compression of large-scale scenes of over 100 people within a range of 9 megabits to 120 megabits.  The Specification outlines methods for connecting a volumetric playback application to a backend server so that volumetric data can be streamed, including details of how the stream is organized and how the playback application can communicate with the server. 

 

The Volumetric Player Format Specification covers the important aspects of volumetric video processing. Key chapters include over 125 definitions that define volumetric compression, streaming, decompression, and rendering.  It also sets basic “trick-play” requirements for all volumetric player applications, establishes fundamental standards for HTTP streaming from a server to a device, and defines mesh and patch-based compression standards.  The Specification is available to members of the Volumetric Format Association and summaries of the Specification and the Dictionary of Terms may be found at:  https://volumetricformat.org. 

 

The VFA believes that Volumetric media is an important next step.  The industry has pushed video resolution to its logical conclusion, 5G is allowing mobility at high definition, and we see interactivity through Volumetric as a logical next step.  Sports, entertainment, gaming, and Metaverse applications will be enhanced with interoperable Volumetric video technologies.  However, no single industry segment can advance the technology as Volumetric relies on cameras, GPUs, data transport, compression algorithms, controllers, and displays.  Standards setting bodies such as VFA are central to helping the industry collaborate and evolve.

 

Table of Contents

 

1. General

2. About This Book

3. Scope

4. References

5. Terminology and Conventions

6. Arithmetic-Operators

7. Numerical-Notation

8. Definitions

9. Server Side Application Organization System

9.1. Introduction

9.2. Overview

9.2.1. Presentation Data Structure

9.2.2. Presentation Data Hierarchy

9.3. The Player Application

9.3.1. Application Structure

9.3.2. Application Directory Structure

9.3.3. Presentation Data

9.3.4. Media playlist, timeline map and variable fields map

9.3.5. Menu structure

9.3.6. Media playlist

9.3.7. Media streaming

9.3.8. Download

9.4. Player system

9.5. Application management

9.5.1. Timeline map setup

9.5.2. Timeline Map to Media Playlist Connection:

9.5.3. Live Streaming Timeline Maps:

9.5.4. MENU SETUP

9.5.5. MENU MANAGEMENT

9.5.6. TIMELINE MAP AND VARIABLE FIELD MAP MANAGEMENT

9.6. Section 5

9.6.1. Navigation Management

9.6.2. Data Access Manager

9.6.3. Data Cache

9.6.4. PRESENTATION ENGINE

9.6.5. USER INTERFACE ENGINE

9.7.1. OUTPUT MANAGEMENT

9.7.2. USER INTERFACE DEVICES

9.7.3. Look Direction:

9.7.4. Move Direction:

9.7.5. Pointer and Pointer Action:

9.7.6. AV RENDERERS

9.7.7. AV Main Data:

9.7.8. AV Supplemental Data:

9.7.9. AV OUTPUT

1. Volumetric Player Data

10.1. Volumetric Audio Video Container File Definition

10.1.1. Introduction

10.1.2. Volumetric Audio Video (VAV) MPEG-2 Transport Stream

10.2. Connection conditions between VAV streams

10.3. Introduction

10.3.1. General constraints on Primary elementary streams

10.3.2. General constraints on H.265 video streams

1. Mesh-Encoding

11.1. Mesh Atom Encoding Process

1. Mesh Encoding Specification

12.1. Overview

12.2. Mesh Elements

12.3. Compression Techniques

12.3.1. Temporal compression

12.3.2. Vertex Delta encoding

12.3.3. Linear Blended Decomposition encoding

12.3.4. Mesh domain compression

12.3.5. Sequence Bytestream compression

12.3.6. Extending Data

12.4. Mesh Box (‘mesh’)

12.4.1. Layout

12.4.2. Semantics

12.4.3. Option Flags

12.4.4. How to use child boxes

12.5. Segment Table Box (‘segt’)

12.5.1. Layout

12.5.2. Semantics

12.5.3. Segment Box (‘sgmt’)

12.6. Segments and Interframe Data

12.6.1. Base frame data

12.6.2. Layout

12.6.3. Semantics

12.6.4. How to interpret child boxes

12.7. Vertex Delta Encoding Interframe Box (‘vtxd’)

12.7.1. Layout

12.7.2. Semantics

12.7.3. How to interpret child boxes

12.8. Linear Blended Skinning Decomposition Encoding Interframe Box (‘lbsd’)

12.8.1. Layout

12.8.2. Semantics

12.8.3. How to interpret child boxes

12.9. Skinning DATA

12.9.1. Layout

12.9.2. Semantics

12.10. User Vertex Property (‘uvpb’) and (‘uvpd’)

12.10.1. Layout

12.10.2. Semantics

12.11. Bytestream Compressed Segment (‘bcsg’)

1. Scene-Data Encoding

13.1. Introduction

1. Patch-Encoding

14.1. Introduction

14.2. Section 1

14.2.1. General Data Organization

14.3. Section 2

14.3.1. Volumetric Zones

14.4. BLENDING IMAGE RESOLUTION ACROSS VOLUMETRIC ZONES

14.5. PATCH PURPOSE

14.6. PATCH DATA

14.7. PATCH GENERATION

14.8. PATCH SIZING

14.9. ATLAS

14.10. ATLAS ORGANIZATION

14.11. Section 3

14.12. Section 4

14.12.1. Encoding

14.12.2. TRANSPORT STREAM

1. GLOSSARY OF TERMS