Immersive VFX Challenges & Solutions - Immersive VFX is more complex than traditional VFX due to stereo lenses, high frame rates (90 fps), and 8K resolution per eye [2] - Even simple VFX elements can have a significant impact in immersive environments if executed well [3] - Previsualization in 180 degrees is crucial for planning various aspects of production, potentially justifying budget through cost savings [4][5] - Data capture on set (HDRIs, photogrammetry, LiDAR scans, etc) is essential for adding elements, cleanup, and ensuring spatial awareness in stereo [6][7] - Camera tracking accuracy is critical due to the realistic stereoscopic depth in immersive experiences [9] - Reviewing VFX work in Apple Vision Pro is necessary at all stages to avoid discrepancies between 2D monitors and the final immersive output [10] DaVinci Resolve/Fusion Workflow for Immersive Video - DaVinci Resolve's Fusion tab offers tools for immersive VFX, including Immersive Patcher, LatLong Patcher, and Panomap [15][18] - The Immersive Patcher converts lens space images to rectilinear images for easier paint and compositing work, with adjustable angle of view [20][21] - The Panomap tool converts between different formats, such as immersive to lat-long, useful for stabilization and CG render mapping [23][24] - Fusion allows previewing work in the Apple Vision Pro headset by combining left and right eyes into layers using a combiner node [26][27] - DaVinci Resolve can pass through the ILPD metadata into EXRs, maintaining camera information throughout the VFX pipeline [74] - Spherical Stabilizer in Fusion can stabilize shaky camera footage by converting immersive space to lat-long space [86][87]

Immersive Video Post-Production Workflow - The immersive post-production phases are similar to traditional film or television projects, with a focus on creative aspects due to the new Blackmagic workflow simplifying technical considerations [1][2] - Apple Immersive Video emphasizes viewer presence, requiring post-production to protect and intentionally alter this sense [3] - Reviewing footage in Daily's Envision Pro and Resolve is crucial for identifying the best moments and addressing capture issues that could interfere with presence [4][5] - Apple immersive video enabled cameras, like the Blackmagic camera, address common immersive issues such as bad 3D sync and inaccurate world scale, simplifying the media workflow [6] Creative and Technical Considerations - Reviewing shots involves both monitor and Vision Pro to identify issues that can detract from presence, as issues must be assessed in Vision Pro [7][8] - Editorial cuts in immersive video are like transporting the viewer, requiring careful attention to timing, pacing, and information delivery to avoid viewer overload [9][10][11][12] - Editors consider z-space and viewer comfort when cutting, using techniques like dip to black to reset depth cues and guide attention [13][14] - Spatial audio is crucial for a convincing sense of presence, requiring mixers to ensure all expected sounds are audible, even if not the focus [15][16][17][18] - Visual effects in immersive video must adhere to the sense of presence, with elements like title graphics needing to exist in 3D space and integrate with the environment's light [23][24][25][26][27] Finishing and Color Grading - The finishing process addresses distractions from presence, such as image noise and compression artifacts, which are particularly noticeable in immersive video [28][29][30][31] - Creative color grading aims for an idealized reality, pushing the limits of authenticity while maintaining a realistic feel [31][32][33] - Color grading should be done with the experience of being inside the image in mind, utilizing live preview in Resolve to get the feeling just right [34] Key Takeaway - All creative and technical decisions should be made in Vision Pro to accurately assess the sense of presence [35][36][37]

Immersive Media Production - Key Takeaways - Immersive experiences aim to capture the feeling of being part of an event, not just observing it [5] - Access to prime locations is crucial for capturing immersive experiences, requiring negotiation and navigating legacy workflows [8] - Collaboration with artists is essential to understand their vision and design coverage accordingly [9][10] Technical Aspects of Immersive Capture - Stage mapping and choreography analysis are vital for strategic camera placement [10][13] - Previsualization in 3D environments like Vision Pro helps refine camera angles and movement [14] - Stabilized cameras and dynamic rigs like Steadicams and Spidercams are necessary to match the energy of live events [16][17] Overcoming Traditional Limitations - Traditional long lenses and camera pit placements are not effective for immersive capture [6][7] - Immersive capture requires breaking down the wall between artist and fan [10] - The goal is to make the audience feel like they are on stage with the performers [6]

Immersive Filmmaking Production - Key Learnings - Immersive filmmaking expands storytelling horizons, placing viewers inside the story and changing the future of filmmaking [2] - Traditional crew positions, budgets, and filming schedules are similar, but resource allocation and time management differ due to the new technology [7] Pre-Production & Development - More time and resources are needed in development and pre-production to familiarize the crew with the new technology through camera workshops and demos using Vision Pro [11] - Previs is crucial for producers to understand filming tools needed based on camera positions and blocking, aiding in environment construction [12][13] Set Design & Lighting - Set design requires extreme attention to detail due to the immersive camera's acuity, necessitating authentic materials and increased resources [15][16] - Built-in lighting within the set helps maintain authenticity and reduces visual effects cleanup costs in post-production due to the wide 180-degree field of view [17] - Location shooting requires extra mindfulness of weather conditions and unobstructed fields of view to avoid motion discomfort [18][19] Talent & Footage Review - Wardrobe, makeup, and actor performance details must be authentic due to the camera's acuity and 1:1 world scale [20] - Reviewing footage in Apple Vision Pro is essential for accurate shot assessment during production [21] - Multiple Vision Pro devices are needed on set, requiring a dedicated Vision Pro manager to maintain and manage the devices [23][24] Camera Systems & Integration - Integrating additional camera systems, like traditional cinematic 3D rigs, can be challenging and requires early camera testing to understand their needs and integration into the immersive production workflow [26][27]

Hello everyone, I'm Anton Tammi. Today I'm happy to share the benefits of previsualization that I learned when directing the first Apple Immersive music video, Open Hearts for The Weeknd. I've directed music videos for a decade now.Like many in my generation, I started with independent music videos surrounded by like minded colleagues, learning directing. With international community of filmmakers. We shared knowledge and tips with each other both offline and online.Eventually, my portfolio grew and big mus ...

SpatialGen Platform Overview - SpatialGen is designed for spatial computing, handling multi-terabyte files and streaming unique video formats [3] - The platform supports visually lossless formats like ProRes and Resolve release exports [3] - SpatialGen offers a free macOS app for uploading ProRes files, AIM bundles, and Final Cut zip files, supporting 2D and 3D spatial videos including Apple Immersive Video [5] - Users can upload videos via browser, tagging video characteristics, adding audio tracks and subtitles [5] - SpatialGen allows real-time updates to audio and subtitles without downtime, with options for setting resolutions and bitrates or using auto-configured options [6] Apple Immersive Video Streaming - Apple Immersive Video streams are fully automated, with Spatial Audio building dynamic metadata [7] - Digital rights management (DRM) is supported to prevent piracy, including Apple's Fairplay DRM [7][8] - SpatialGen provides HLS links for Apple Immersive Video, enabling distribution on Vision Pro, multiple Apple platforms, and custom apps [8][9] - SpatialGen's API allows integration into any app or website, with a published Streams endpoint for testing before launch [9] Performance and Quality - SpatialGen uses quality metrics like VMAF, aiming for scores over 90, and has achieved significant improvements in VMAF scores even at low bitrates like 30mbps [14][15] - SpatialGen monitors and embeds developments around Foveation into the platform [15][16] Use Cases and Applications - SpatialGen is trusted by brands and creatives, powering experiences on Vision Pro [10] - The platform is leveraged for enterprise use cases, including pilot training and defense workflows [11] - SpatialGen has contracts with the United States Air Force and is used by Immersive India to reduce costs [12]

Colorfront's Solutions for Apple Immersive Video - Colorfront aims to empower storytellers by providing tools that push the boundaries of immersive video creation [1] - Colorfront's software runs natively on macOS and is optimized for Apple Silicon, ensuring top performance [3] - Colorfront Onset Dailies delivers a streamlined, color-accurate workflow, ensuring what is seen on set matches the final experience on Apple Vision Pro [4] - Colorfront Transcoder ensures every frame is delivered with precision, meeting the highest industry standards for quality and performance [5] Technology and Industry Recognition - Colorfront's color science and media processing are built on the Colorfront Engine, trusted by major productions [6] - Colorfront's innovations have been recognized with scientific and technical awards from the Academy of Motion Picture Arts and Sciences and the Television Academy [7] - Colorfront's tools are used by Apple TV Studios, Disney, Sony Pictures, Company 3, and Picture Shop [8] - Colorfront supports Apple's content creation, powering immersive experiences with high-quality workflows, as seen in titles like Metallica [9] Workflow and Performance Enhancements - Colorfront captures, displays, and carries forward rich metadata, preserving critical information throughout the workflow [13] - Colorfront supports AIV for instant playback in Apple Vision Pro and ProRes AIM for offline editorial workflows [16] - Colorfront's distributed rendering allows workflows to scale across multiple systems, improving performance by up to 300% [17][19] - Colorfront integrates the MainConcept Immersive Encoding Library to enhance video quality and performance for Apple Immersive Video [20]

Spatial Audio Acquisition - Industry emphasizes recording Ambisonics spatial audio from the camera's perspective, especially in documentary filmmaking, to capture the entire sonic environment [7] - Industry suggests using mono recordings in addition to spatial recordings for dialog, prioritizing individual mics due to the wide field of view in immersive experiences [13][14] - Industry highlights the availability of various ambisonic microphones with similar form factors to regular shotgun microphones, offering options for different needs and budgets [10] Sound Editorial - Sound editorial involves assembling and layering dialog, sound effects, and music, similar to standard post-production [15] - Industry recommends using Foley and Ambiances to build immersion by focusing on small, everyday sounds [19][20] - Industry suggests compiling and assembling all sound sources, including multiple spatial mics and individual mono ISOs, during sound editorial [18] Spatial Mix - Industry advises anchoring sounds to visual elements using Apple Spatial Audio instead of relying on a center channel, as the audience is inside the scene [26] - Industry recommends mixing on headphones to ensure the mix translates well with regards to perspective and dynamic range [30] - Industry stresses the importance of reviewing work on Apple Vision Pro to accurately judge distance, perspective, and sound positioning [31]

Camera & Equipment Preparation - For expedition shoots, under 100W travel-safe, stackable V-mount batteries are essential for power [4] - 16 terabyte media modules are recommended for prioritizing the moment, but long download times should be planned for [7] - Overpacking media and power is recommended to avoid limitations in the field [8] Monitoring Solutions - Internal LCD wing is an excellent mobile monitoring option for quick deployments [10] - Single eye monitoring is fantastic for heli work, gimbal techs, crane techs, and underwater operations [12] - Dual I monitoring is important for checking lens consistency and avoiding issues like bugs on the lens [12] Aerial Equipment & Considerations - Heavy lift drones like Alta X have limited battery time (approximately 5 minutes) in cold weather [16] - Planning for VFX costs is necessary to remove blades and booms from final footage when using Alta X [17] - Cablecam allows for intimate shots without disturbing wildlife [18] Dailies & Post-Production - Dailies are essential for confirming the frame, especially since the composition can only be accurately assessed in a Vision Pro [20] - A small slimline Mac Pro with multiple raids is ideal for confirming shots in the field, while a cart with multiple Mac studios is suitable for faster backup and processing power on land [21]

Good morning. I'm Deep and I'm the lead immersive audio architect at Apple. This year, our team publicly released new end to end immersive audio technology involving a new audio format, the Apple Spatial Audio Format or ASAF, and a new codec called APAC. Today, I'm excited to give you an overview of this new format and the new codec. I'll also dive into the workflow a little bit and tooling to create content in this new format.Let's explore the format a little bit. The format allows content with both HOA ob ...