# Autoshot Overview
Source: https://docs.lightcraft.pro/autoshot/autoshot-overview
Autoshot is the desktop or laptop counterpart to Jetset. It works as a connecting piece between traditional 2D or 3D digital content creation tools like Blender, Unreal, After Effects, Nuke, Maya, etc. and Jetset.
## The Main Loop
Jetset and Autoshot work with these tools in a well-defined ‘loop’. The loop consists of 4 basic steps:
* Place scene locators (empty 3D objects named sceneloc\_\*) in the original scene in locations that will be matched to specific points in the live action shoot
* Export a subset of the main scene as a USD file set, including the scene locators, convert the USD files into a single packaged ‘blocking proxy’ USDZ file, and push the USDZ file to Jetset over the local network using Autoshot
* Load the proxy USDZ file into Jetset, pick a live action origin, and pick the desired scene locator from the scene.
* Shoot a take with a real time preview of the live action and CG interaction
* Pull the take data back to the desktop or laptop wirelessly using Autoshot, and pick which shot take you want to process.
* Still in Autoshot, choose which target application you want to create a shot package for, and the color space/gamut of the footage.
* Hit Run, and Autoshot will extract an image sequence from the shot takes, package up the camera tracking data, and write out a script file that will directly import the completed shot package into the original 3D app, or other 2D & 3D applications used in the production pipeline.
This whole process can be run in a couple of minutes, and is the core design of Jetset & Autoshot.
# Output Directory Structure
Source: https://docs.lightcraft.pro/autoshot/directory-structure
Autoshot has a flexible system to create an output directory structure to match your VFX pipeline.
## Default Output Directory
Note -- this was originally created to match the ETC 'Pathways' folder structure, but it's a well structured file and folder naming convention.
### Definitions
```
{SEQ} Sequence number e.g. PTH_001
{SHOT-CODE} Shot code composed of sequence + shot number e.g. PTH_001_170
```
### Directories
```
/{SHOT-CODE}
/3D
/BlenderFiles
/BlenderRender
/v00
/Editorial
/Plate
/v01
/Proxy
/v01
```
### Example: PTH\_002\_210
```
/PATHWAYS/
/PTH_002_210/
/3D/
/BlenderFiles/
/PTH_002_210_v001.blend
/PTH_002_210_v002.blend
/BlenderRender
/SLP_v001/
/PTH_002_210_SLP_v001.mp4 (Slap Comp)
/BRN_v001/
/PTH_002_210_BRN_v001_01001.EXR (Blender Render)
/PTH_002_210_BRN_v001.mp4
/Editorial/
/Plate (ACEScg EXR image sequence from camera original)
/v001
/PTH_002_210_PLT_v001_01001.exr
/Proxy (sRGB JPEG Downsample of Plate)
/v001
/PTH_002_170_PRX_v001_01001.jpg
/PTH_002_170_PRX_v001.mp4
```
# Folder Setup Panel
Source: https://docs.lightcraft.pro/autoshot/folder-setup-panel
The Folder Setup panel shows the locations of the main directories used by Autoshot.
![]()
## Project Directory
The Project directory is the parent project folder. A new folder is typically created for each separate project or client.
Autoshot will use this folder to create 3 subfolders:
* Takes
* Assets
* Sequences
### Takes
The Takes folder is where Autoshot will sync takes to. The takes directory structure mirrors the way files are stored on Jetset.
A separate dated folder is created for each individual day of shooting; this folder contains the day’s takes.
Underneath that folder, a Camera folder is created. This will be named A, B, C, D, etc. depending on which camera the Jetset app was set to.
Inside the Camera folder are subfolders for:
* Scans (if created)
* Images (if 2D background images were added to the scene, they are stored here for later use)
* Metadata (contains the JSON files where the tracking data and shot metadata are stored)
* Videos (contains the iOS camera original, real time composite, and depth videos for each take)
### Assets
The Assets folder is generated for an optional location for project assets, such as USD and USDZ exports
### Sequences
The Sequences folder is where Autoshot will place all data and files generated as part of processing takes for use in post production.
Sequences are stored by the date of the take processed, with subdirectories showing the individual takes from that day that are processed.
Inside the individual take directories are subdirectories that contain the actual generated files:
* aimattes: Autoshot-generated 8 bit greyscale images, written in PNG RLE format.
* blend: generated Blender files
* camEXR: EXR files extracted from the iOS camera original video
* camPNG: PNG files extracted from the iOS camera original video
* cinecamEXR: EXR files extracted from the cine camera original. These are ACEScg format and DWAA compressed by default, with ZIP compression optional
* c4d: generated Python file for C4D import
* nuke: generated Python files for Nuke import
* syntheyes: generates Sizzle script for Syntheyes import
* maya: generated .ma file for Maya import
## Cine Camera Directory
The Cine Camera directory is where Autoshot will search to find camera original files, such as BRAW, ProRes, or log .mp4 files.
Autoshot can directly read ProRes and .mp4 files, so if camera original files are in these formats, proxies are not needed.
## Cine Proxies Directory
The Cine Proxies directory is where Autoshot will search for matching proxy files for the cases where it cannot read the camera original files directory (BRAW and other RAW file formats.)
In this case, Autoshot needs both the Cine Camera directory and the Cine Proxies directory to be pointed to directories containing the files.
When the ‘Scan’ button is pressed next to the Cine Camera directory, Autoshot will search through either the camera originals, or in the case of RAW file formats the proxies folder.
### Detecting ID and Timing
It will load each take, and search through it to detect the black and white ID/timing markers that are displayed on the Digital Slate browser at the beginning of every Jetset take.
It uses this information to match a given Cine take to the correct Jetset take, and to calculate the correct cine offset timing for the tracking data.
### Image Sequence Extraction
Then, when a sequence is processed, and the camera original is a ProRes, .mp4, or .mov log file, Autoshot will extract EXR files directly from the same camera original file it found ID and timing from, and convert these frames to ACEScg formatted EXR files.
If the camera original file is in a RAW file format, such as BRAW, the files in the Cine Camera directory will be used for the actual EXR extraction, as they contain the full captured color information from the cine camera.
## Hotkeys/Shortcuts
**Shift-Scan** will erase the stored table of take file to cine video file matches, and re-scan the cine video files.
**Shift-Browse** on the Project Folder Browse button will pull up a list of the most recent project folders.
# Pasting File Paths on the Mac
Source: https://docs.lightcraft.pro/autoshot/paste-file-path-mac
Use this trick to paste file paths on the Mac. This is useful when running scripts in SynthEyes and other tools.
After running Autoshot to generate a SynthEyes (or other) script, you’ll need to point SynthEyes to the correct directory to run that script. You can click the highlighted file line in Autoshot to open up a Finder window that is set to the correct directory, and then select and copy the file path that is at the bottom of the Finder window.
So — how do you then paste a file path on the Mac?
The answer (from user JPTX1292!) is to use Command+Shift+G anywhere inside the Finder file browser, or whenever a 3rd party software uses the Finder window to retrieve a file!
Here’s the sequence in SynthEyes:
![]()
![]()
![]()
# Sync Panel
Source: https://docs.lightcraft.pro/autoshot/sync-panel
The Sync panel is used to identify, select, and sync files from Jetset devices on the local network.
![]()
## Jetset Clients List
The ‘Jetset Clients’ panel lists all Jetset devices found on the local network. This happens automatically, or whenever the Refresh button to the right of the panel is clicked. (The iOS devices must be actively running Jetset to be detected.)
Users can click to highlight on individual detected Jetset clients.
## Current Project
The Project text box below will show the name of the project that the currently selected Jetset client has active. This way, one can avoid accidentally synchronizing takes from one project into another’s working folders.
The 'Sync To' box directly below the 'Project Name' shows the name of the currently selected Autoshot project folder. Jetset will check to make sure that the 'Sync To' directory name has the same name as the currently selected 'Project Name' folder, and show a red warning if the names don't match as a courtesy to the user.
This won't stop you, however from syncing the files if you click the Sync button.
## Manual Connection
If the Jetset devices aren’t detected automatically, it’s possible to manually enter in the IP address of the device into the IP address line, and then click Connect. The Jetset device should then be listed in the client list.
## Open Web Page
The Open Web Page button will open a browser window pointed to the IP address of the currently selected Jetset client. This is usually faster and more convenient than manually typing in the IP address into the browser address bar.
## Sync Date Selection
The Sync Date dropdown lets you pick which date of recorded takes that you wish to synchronize. You can also choose ‘Sync All’, to synchronize all of the takes of a given project to the current Autoshot project directory.
## Sync Button
The Sync button next to the Sync Date dropdown triggers the actual take sync process. Takes are synced by comparing the files in the local project folder with the files in the selected Jetset device; this way only missing files are synced, saving time.
## Sync Options
* Match File Size checks to see that the copied file has exactly the same number of bits as the source file. This should be checked by default. The only reason to uncheck it is when syncing to a Google Drive device. Google Drive pads the file sizes by 512 byte intervals, so that the copied file is actually slightly larger in size than the source file.
* Verbose sends detailed debugging information for the file cloning process to the console. This can be useful when debugging file transfers.
# Take Selection Panel
Source: https://docs.lightcraft.pro/autoshot/take-selection-panel
The Take Selection panel lets users pick which take they will be processing..
![]()
### Day and Take
The Day dropdown shows the list of the shooting days from the current project in Autoshot.
The Take dropdown shows the list of the available takes from that shooting day.
The Open button next to the Take dropdown will be a frequently used button. This opens up a file explorer to the Autoshot-created take subdirectory in the current project folder hierarchy.
This is very useful for directly jumping to the various generated scripts and files that Autoshot generates for a given take.
### Take USDZ and Sceneloc
The Take USDZ line shows which USDZ file was loaded when the currently selected take was active.
The Sceneloc line shows which scene locator was active when the currently selected take was active.
### Cine Focal Length/Angle
These are only active when a Jetset Cine take is loaded.
Cine Focal length shows the calibrated focal length of the currently selected take. This is based on the horizontal field of view angle shown to the right combined with the entered sensor width at the time of calibration.
The Angle field to the right of the Cine Focal Length field shows the HFOV value calculated during the Jetset Cine lens calibration. This is an actual value determined directly from the lens and sensor width parameters.
### Cine Calibration, Solve/Resolve, and GUI Check
The Cine Calibration line shows the filename of the Cine lens calibration that was loaded when the current take was recorded.
The Solve/Resolve button is used to calculate the actual 3D cine lens position offset, field of view, and lens distortion from the captured frames of a Jetset Cine calibration. If this calculation has already been done, the button will show Resolve instead of Solve. The Solve/Resolve process uses the integrated COLMAP solver (included in the Autoshot Tools install) to perform this calculation.
The GUI check button can be used after a 3D solve has been completed. This will open up the COLMAP GUI so that you can check where the various camera positions were found in the solve, as well as the 3D point cloud from all the matched 3D points in the solve.
The COLMAP GUI button is just used as a debugging tool, and isn’t a necessary step to running Cine takes like the Solve button is.
### Cine Footage Matched
The Cine Footage Match line shows the name of the cine video take that matches the current Jetset take. This can be determined automatically via digital slate matching or timecode matching.
If the automated methods don't work, the take can be manually selected with the *Manual Pick* button.
### Cine Time Offset
The Cine Time Offset panel shows the time offset (in seconds) from the beginning of the current cine video clip in the 'Cine Footage Match' line to the beginning of the currently selected Jetset take.
This is calculated either with the flashed markers of the digital slate, or with the automated timecode match.
Note that the timecode based match is not as accurate as the digital slate match; for this reason we recommend clicking the 'Refine Offset' button after doing a take match based on timecode.
The 'Refine Offset' button will trigger a quick optical flow analysis between the cine footage and the Jetset video; this data will be used to refine the Cine Time Offset.
Using Cine Time Offset is not recommended for digital slate matches as the slate is more accurate.
# After Effects
Source: https://docs.lightcraft.pro/compositing/after-effects
Autoshot has an AE exporter that can automatically assemble rendered frames and camera originals into a composition.
## Setup and Script Export
In Autoshot:
* Under Run Values/Program, choose ‘Others’
* Set the Clip In and Clip Out frames to specify which frames you want to extract.
* The After Effects workflow does not yet use scene locators, so you don’t need to specify which one.
* Set the camera manufacturer and color space of the source clip. All Autoshot EXR files are automatically converted to ACEScg.
* You can specify an AI matte processor if you want mask shapes for the human figures in the footage.
* Next, hit Save and Run.
![]()
Autoshot will then run the EXR file extraction and write out a .jsx file into the /ae directory under the created sequence directory.
To find this directory, click the ‘Open’ button next to the selected take in the Take dropdown selector. This will open a File Explorer to the correct directory.
To run this script in After Effects, go to File -> Scripts -> Run Script and navigate to the /ae directory above, then choose the generated .jsx file.
![]()
This will create a new composition with the footage and tracked camera loaded.
![]()
## Verifying Project and Footage color
This is a good start, but we want to verify that the color is correct. By default, the generated AE script will set the After Effects project to ACES color management. You can verify this with File -> Settings:
![]()
Finally, we’ll right click on the EXR sequence under the Project panel and choose Interpret footage -> Main. Pick the Color tab and then select scene\_linear: ACES – ACEScg:
![]()
Now the displayed color in After Effects should be correct. Please note that there is a Keylight effect applied to the footage by default.
![]()
# Multi-Peel in SynthEyes
Source: https://docs.lightcraft.pro/compositing/multi-peel-syntheyes
This is a great way to automated your multiple feature detection and ‘peeling’ process, using a Synthia script developed by Matt Merkovich.
## Synthia Natural Language
Synthia is a natural language interface; it provides a great way to automate SynthEyes tasks for tracking artists that don’t want to deal with the technical details of Sizzle or Python, the other scripting languages that SynthEyes supports.
To get an idea of how easy it is to automate tasks, here’s the script:
***
//Multi-Peel MATTMERK Style
say ““.
say “Don't multitask while running: check email, twitter, etc. 😉“.
say “It will grab window focus and crash the script“.
say ““.
features room.
make current camera’s small blip density 0.008.
make current camera’s big blip density 0.004.
make current camera’s small blip size 8.
make current camera’s big blip size 16.
make current camera’s maximum tracker count number of frames\*2.2.
click and wait for “blips all frames” button.
//wait 2 seconds.
click and wait for “peel All” button.
//wait 2 seconds.
clear all blips.
make current camera’s small blip size 12.
make current camera’s big blip size 24.
click and wait for “blips all frames” button.
//wait 2 seconds.
click and wait for “peel All” button.
//wait 2 seconds.
clear all blips.
make current camera’s small blip size 16.
make current camera’s big blip size 32.
click and wait for “blips all frames” button.
//wait 2 seconds.
click and wait for “peel All” button.
//wait 2 seconds.
clear all blips.
make current camera’s small blip size 20.
make current camera’s big blip size 40.
click and wait for “blips all frames” button.
//wait 2 seconds.
click and wait for “peel All” button.
//wait 2 seconds.
clear all blips.
***
## Usage Instructions
* Download the Multi-Peel script from [https://lightcraft.pro/downloads/](https://lightcraft.pro/downloads/)
* Select Script->User Script Folder in SynthEyes to open the User Scripts folder.
* Copy the Multi-peel\_MATTMERK\_Style.sia file from Downloads into the User Scripts folder.
* Pick File->Find New Scripts in SynthEyes.
* The script can now be run from the Script menu; it will appear alphabetically in the list.
# Nuke
Source: https://docs.lightcraft.pro/compositing/nuke
Autoshot has an AE exporter that can automatically assemble rendered frames and camera originals into a composition.
## Setup and Script Export
In Autoshot:
* Under Run Values/Program, choose ‘Others’
* Set the Clip In and Clip Out frames to specify which frames you want to extract.
* Make sure to select the Scene Locator that was used in the take to have the 3D tracked camera and scene scan mesh show up in the correct position in Nuke’s 3D coordinate space.
* Set the camera manufacturer and color space of the source clip. All Autoshot EXR files are automatically converted to ACEScg.
* You can specify an AI matte processor if you want mask shapes for the human figures in the footage.
* Next, hit Save and Run.
![]()
Autoshot will then run the EXR file extraction and write out a python file into the /nuke directory under the created sequence directory.
To find this directory, click the ‘Open’ button next to the selected take in the Take dropdown selector. This will open a File Explorer to the correct directory.
To run this script in Nuke, go to File ->Run Script and navigate to the /nuke directory above, then choose the generated .py file.
![]()
The script will generate a node group. Nuke Non-Commercial and Nuke Indie are limited to 10 nodes per Python script run, but the essential nodes are all there:
* EXR loader for the camera footage EXRs in ACEScg format
* EXR loader for the Undistort STMap generated by Jetset Cine’s lens calibration
* A loader for the 8 bit grey scale AI Mattes (if generated)
* STMap node to undistort the camera footage
* An Axis node with the position and orientation of the chosen Scene Locator
* A ReadGeo node to read the scanned mesh in, and a Wireframe node to set it to wireframe.
* A Camera node with the animated track associated with the camera frames
* A Scanline Render node to see the combined 2D/3D join
* A Write node pre-configured to write to the /compEXR and /compJPG directories for saved EXR and JPG proxy renders
## Verifying Project and Footage Color
Double clicking the camera original footage node will bring up the Loader panel, where you can confirm that the footage is being correctly interpreted as ACEScg:
![]()
Hit L on the keyboard to play through the footage and verify the tracking match:
The automatic Jetset Cine track is not a sub-pixel track, but it’s workable for many shots that don’t have a lot of ground contact.
For shots with ground contact, in our next tutorial we’ll work through a method in Nuke to refine our camera solve without losing our 3D spatial coordinates.
# Syntheyes to Unreal
Source: https://docs.lightcraft.pro/compositing/syntheyes-unreal
Use SynthEyes to refine Jetset tracks for import to Unreal.
## Setup and Script Export
In Autoshot:
* Under Run Values/Program, choose ‘Others’
* Set the Clip In and Clip Out frames to specify which frames you want to extract.
* Make sure to select the Scene Locator that was used in the take to have the 3D tracked camera and scene scan mesh show up in the correct position in Nuke’s 3D coordinate space.
* Set the camera manufacturer and color space of the source clip. All Autoshot EXR files are automatically converted to ACEScg.
* You can specify an AI matte processor if you want mask shapes for the human figures in the footage.
* Next, hit Save and Run.
![]()
Autoshot will then run the EXR file extraction and write out a Sizzle script file into the /syntheyes directory under the created sequence directory.
# Tracking Markers
Source: https://docs.lightcraft.pro/compositing/tracking-markers
Use these guidelines to set up tracking markers for quick tracking refinement and easy compositing.
## Tracking Markers
![]()
Jetset and Jetset Cine can already record high quality tracking data in real time, so why are tracking markers useful, and when should we use them?
### Tracking Refinement
Jetset Pro and Jetset Cine have a unique ability to record 3 matching sets of data:
1. Per-frame 3D position and orientation of the cine camera, including the offset between the iPhone lens and the cine camera lens
2. An accurate lens calibration of either the iPhone camera (Jetset Pro) or the attached cine camera lens (Jetset Cine.)
3. A centimeter-accurate 3D scan of the set in the direction of the shot that is automatically coordinate matched to the camera position and orientation from #1.
This makes it possible to refine or retrack a shot very quickly, in minutes instead of hours or days.
There are 2 major techniques that both use the same basic information:
1. [Feature-based tracking, such as SynthEyes](https://docs.lightcraft.pro/tutorials/tracking-compositing/tracking-refinement-syntheyes)
This method uses a tracking program like SynthEyes to detect trackable points in the 3D scene, and then uses the initial real time Jetset tracking data to 'project' 3D trackers onto the scanned mesh from the scene. This creates 3D survey data that constrains the SynthEyes solver to the same position, orientation, and field of view that the original Jetset shot had! This massively simplifies an otherwise very difficult post production tracking refinement task.
2. [Optical flow based tracking, such as GeoTracker](https://docs.lightcraft.pro/tutorials/tracking-compositing/geotracker-blender)
This method uses GeoTracker's optical flow analysis tool to measure the individual pixel motions of the camera original footage, and then uses the Jetset-scanned scene geometry to automatically set alignment keyframes and automate the tracking process inside Blender or Nuke.
### Marker Color
The tracking marker color should be more or less the same color as the background screen (green or blue), but just different enough that the post tracking software can detect corner features. See the above image for a good example of a tape color that is different enough to be detected.
### Marker Shapes
Good results can be obtained with either round markers or markers that are cut into strips to create corner features. The important thing is that the tracking software has to be configured to detect either 'dots/circles' or 'corners'.
Our SynthEyes script is automatically configured to set SynthEyes to detect corner features, so we recommend tape patterns like the one above.
Note that the tape stripes are not all identical orientations; this is done so that the 'search patterns' in the tracking software don't accidentally 'jump' from one marker to the next.
### Marker Coverage
Marker coverage is driven by 2 separate goals:
1. You typically want quite a few markers in the shot, so that you can detect enough points for the refinement solve to work, but:
2. You don't want your actor's hair to cross the markers, as it makes keying more difficult.
The usual answer is to add quite a few markers to the wall (and/or floor) of the greenscreen area, and then remove / reposition any markers that are directly behind the actors for a particular shot to avoid keying problems. The markers can be rapidly put back roughly where they were for the next shot.
It's not important that the markers go back exactly where they were. The marker detection process happens on a per-shot basis. You can see an example of an automated marker detection script used in [this tutorial.](https://docs.lightcraft.pro/tutorials/tracking-compositing/fixing-misaligned-shot)
If you have a moving camera shot where the actor needs to cross the markers, there are some clever advanced keying workflows that can automatically remove the tape marks from the background, but the tape mark color needs to be similar to the background color as mentioned above for this to work.
# Blender Direct Export to USDZ
Source: https://docs.lightcraft.pro/create-usd/blender/blender-export-usdz
Blender can directly compress texture files and export to USDZ, saving a step in the process.
## Texture Compression
As seen in the [Autoshot Blender Round Trip](/tutorials/blender-workflows/autoshot-blender-round-trip) tutorial, Blender can export to a standard USD file directory with full resolution textures.
In the tutorial, we then used Autoshot to compress those 2K/4K original textures down to 512x512, and then wrapped the USD and texture files into a USDZ file ready to be pushed to Jetset.
However, Blender can also directly write to USDZ while compressing textures, which can save a step.
![]()
To do this, just use a .usdz file extension, and set the USDZ Texture Downsampling value to 512 or another small value.
Blender will compress the textures to that value and write a USDZ file with the compressed textures directly.
# Creating with Megascans
Source: https://docs.lightcraft.pro/create-usd/blender/creating-with-megascans
Megascans are fantastic 3D assets available for free on Fab.com. They come pre-configured for Unreal, but our free Blender add-on imports them automatically into Blender.
## Fab.com and Megascans
![]()
Fab.com includes the fantastic Quixel Megascans collection. These assets are available for free, and can be downloaded in either FBX or Unreal format.
The 'FBX' download has the materials and objects separated into individual folders with either a set of texture maps (for the materials) or a FBX file and a set of texture maps (for the 3D objects.)
## Lightcraft Batch Importer
![]()
Lightcraft's [Batch Importer](https://github.com/LightcraftTechnology/Batch-Import-BlenderAssets) can directly import all of the assets in the unzipped FBX download from Fab.com.
## Tutorials
![]()
See [this tutorial](/tutorials/blender-workflows/import-megascans-blender) for details, or [this great tutorial by Spencer Bell](https://lightcraft.pro/user-stories/spencer-bell/) for an example of a complete workflow including adding scene locators.
# Holdout Mattes
Source: https://docs.lightcraft.pro/create-usd/blender/holdout-mattes
How to use 3D scene geometry to create _holdout mattes_ for use in Jetset.
## Use Cases
'Holdout Mattes' are used to reveal the live action camera image in the middle of a CG scene. They are frequently created to match the size and shape of an existing piece of the physical set. Then Jetset will 'hold out' that portion of the CG imagery and let the live action imagery be displayed in the composite.
## Defining in Blender
Holdout mattes are defined in Blender by assigning the 'holdout\_jetset' prefix to the 'mesh data' of an object inside Blender.
Please note that the mesh data is located *inside* the normal object name! In the example below, the mesh object 'Suzanne' has an internal mesh datablock called 'holdout\_jetset1'.
![]()
The use of the 'mesh' datablock is so that the object itself can have a normal name, while the underlying mesh datablocks use the 'holdout\_jetset\*' tag.
Holdout matte objects are exported along with the rest of the Blender USD scene.
## Use in Jetset
For a Holdout Matte to be active in Jetset, the background must be set to 'Camera':
![]()
Once this is set and the USDZ scene is loaded, the holdout object will reveal the camera through the CG image:
![]()
# Object Locators
Source: https://docs.lightcraft.pro/create-usd/blender/object-locators
How to use object locators to specify objects in the scene that can be moved interactively in Jetset.
## Object Locators
Object locators are empty 3D nulls that Jetset uses to let users interactively move objects in the scene.
Object locators use the prefix 'objectloc\_' followed by a user-specified name, for example **objectloc\_Actor\_Bob**.
No spaces are allowed in object locator names.
## Adding Object Locators
In Blender, select the object you wish to add an object locator to.
Set Blender's 3D cursor to the object position with Shift-S, and then choose 'Cursor to Selected'.
Then hit Shift-A to Add, and select Empty and Arrows.
F2 to rename the empty, and type 'objectloc\_My\_Custom\_Name.'
Select the Blender object that you wish to move with the object locator, then Ctrl-LMB to select the object locator. Move your cursor into the main viewport area, then press Ctrl-P to Parent the object. Choose 'Keep Transform' to avoid moving the object or the object locator.
The result should look like this in the Outliner.
![]()
Export to USDZ as per usual practice.
## Using Object Locators
In Jetset, after importing the USDZ file, navigate to the 'Object Move' tab. Jetset will parse the list of objectloc\_\* nulls in the current USDZ file, and provide a list.
![]()
Highlight the object you wish to move, in this case 'cube'.
You can then drag the cube around on the horizontal plane with your finger.
Tap the visibility icon to show or hide the object.
Tap the rotation icon to interactively rotate the object.
Tap the Z icon to interactively move the object up or down.
Click OK when done moving the objects to return to the main screen.
Object locator positions are stored on a per-take basis in the take metadata, and can be viewed with the 'i' button in Autoshot next to the currently selected take.
### Optical Origin Marker Detection
While in the Object panel, if Jetset sees one of the [Lightcraft origin markers](https://docs.lightcraft.pro/equipment/stage-checklist#vertical-origin-target-stand), the 'Align to Marker' button will light up.
If the user taps that button, the currently selected Object Locator will align to the optical marker at its origin.
This is particularly useful for rapidly locating CG characters and objects within a scene.
# Recommended Video Formats
Source: https://docs.lightcraft.pro/create-usd/blender/recommended-video-formats
Jetset can load movie files in the Image Model plane. Here are the supported video formats.
## Suggested Formats
1. H.264 (AVC)
* Container: MP4 or MOV
* Profiles/Levels:
* High Profile at level 4.2 is most common for high‐quality streaming and video textures.
* Other profiles: Baseline or Main profiles may be used if lower decoding complexity is desired, but High Profile is recommended for best quality on iPhone 12+.
2. HEVC (H.265)
* Container: MP4 or MOV
* Profiles/Levels:
* Main 10 Profile is recommended for improved compression and 10‑bit color support (ideal for richer color and dynamic range).
* Levels supported on iPhone 12 typically go up to around level 5.1.
# Scene Locators
Source: https://docs.lightcraft.pro/create-usd/blender/scene-locators
How to use scene locators to specify shooting locations and virtual motion scales.
## Scene Locators
Scene locators are empty 3D nulls that Jetset uses to navigate through larger scenes, and to rapidly link a live action location to matching virtual location.
Scene locators use the prefix 'sceneloc\_' followed by a user-specified name, for example **sceneloc\_Front\_Door**.
No spaces are allowed in scene locator names.
## Adding Scene Locators
In Blender, set the 3D cursor with Shift-RMB.
Then hit Shift-A to Add, and select Empty and Arrows.
F2 to rename the empty, and type 'sceneloc\_My\_Custom\_Name.'
**[See this tutorial for details.](https://share.descript.com/view/uysDMh48xCq?t=431.151)**
## Scene Locator Scale
Scene locator scales should usually be 1.0. This way, 1 meter of motion by the user will translate to 1 meter of motion in the virtual scene.
If a scene locator has a non-zero scale, the Jetset motion will be amplified by the scale amount.
For example, a scene locator with an XYZ scale of 25 would result in one meter of motion by the operator translating to 25 meters of motion in the virtual scene.
This will break live action tracking, but can be useful for simulating cranes or drone camera moves in a pure CG environment.
Scene locators with custom scales should be named appropriately with the specified scale, for example sceneloc\_street\_crane\_25x.
# Applying AI Mattes in Unreal
Source: https://docs.lightcraft.pro/create-usd/unreal/applying-ai-mattes-unreal
Appy AI generated rotomattes to the Unreal Material Keyer generated with the level sequences by Autoshot.
User CptGabok wrote this great update to using the AI generated rotomattes with the Unreal Material Keyer generated with the level sequences by Autoshot.
1. Change the material type for the CamMaterialTemplate to Masked
2. Use the AIMatteTexture Media Texture as the Opacity Mask within the CamMaterialTemplate
3. Add a Media Track to the Level Sequence
4. Select the AIMatteMediaSource as the Media Source for the new Media Track
5. Select the AIMatteTexture as the Media Texture for the AIMatteMediaSource
Then you should see the rotoscoped footage within Unreal Engine playing back as you scrub the timeline.
# Installing LONET Plugins
Source: https://docs.lightcraft.pro/create-usd/unreal/installing-lonet
Jetset uses the LONET-2 protocol to send real time tracking information to Unreal and other real time rendering engines.
## Downloading Plugins
To download pre-compiled plugins for your version of Unreal, follow [this link:](https://loledvirtual.com/documentation/software-downloads/unreal-plugin-releases)
## Installing Plugins
Once you have downloaded the zip file, unzip it, and place the LONET plugin directory into the Engine/Plugins/VirtualProduction directory your local installation of Unreal. Your path will look something like D:\Epic\_Games\UE\_5.3\Engine\Plugins\VirtualProduction.
Now restart Unreal, and enable the LONET2 -Live Link plugin.
# Object Locators
Source: https://docs.lightcraft.pro/create-usd/unreal/object-locators
How to use object locators to specify objects in the Unreal scene that can be moved interactively in Jetset.
## Object Locators
Object locators are empty 3D nulls that Jetset uses to let users interactively move objects in the scene.
Object locators use the prefix 'objectloc\_' followed by a user-specified name, for example **objectloc\_Actor\_Bob**.
No spaces are allowed in object locator names.
## Adding Object Locators
In Unreal, it's not possible to parent static meshes to the Actor components that we used for scene locators. Instead, we will simply name a static mesh directly, and parent any other meshes needed to that named object locator.
Select a static mesh that you wish to use as an object locator.
F2 to rename the empty, and type 'objectloc\_My\_Custom\_Name.'
The result should look like this in the Outliner.
![]()
Export to USD and convert to USDZ as per [usual practice.](https://docs.lightcraft.pro/tutorials/unreal-workflows/export-usd-unreal-plugin)
## Using Object Locators
In Jetset, after importing the USDZ file, navigate to the 'Object Move' tab. Jetset will parse the list of objectloc\_\* nulls in the current USDZ file, and provide a list.
![]()
Highlight the object you wish to move, in this case 'cube'.
You can then drag the object around on the horizontal plane with your finger.
* Tap the visibility icon to show or hide the object.
* Tap the rotation icon to interactively rotate the object.
* Tap the Z icon to interactively move the object up or down.
* Click OK when done moving the objects to return to the main screen.
Object locator positions are stored on a per-take basis in the take metadata, and can be viewed with the 'i' button in Autoshot next to the currently selected take.
### Optical Origin Marker Detection
While in the Object panel, if Jetset sees one of the [Lightcraft origin markers](https://docs.lightcraft.pro/equipment/stage-checklist#vertical-origin-target-stand), the 'Align to Marker' button will light up.
If the user taps that button, the currently selected Object Locator will align to the optical marker at its origin.
This is particularly useful for rapidly locating CG characters and objects within a scene.
### Object Locator Saves
Object locator positions are stored at the moment of recording a take in the take's JSON metadata.
The object locator metadata can be seen in Autoshot with the 'info' button next to the selected take after [syncing takes to your PC/Mac.](https://docs.lightcraft.pro/autoshot/sync-panel)
# Fixing Autoshot Crashes
Source: https://docs.lightcraft.pro/debugging/autoshot-crashes
Fixing Autoshot Crashes.
If you run into the situation where Autoshot crashes when launching, it may have corrupted a specific file that Autoshot uses when starting up.
We’re still tracking down what causes this, so below is a set of steps to take to both fix the problem, and give us a file to look at to debug it.
The culprit is usually the Autoshot.yaml file. In Windows, this is located in
> C:\Users\YOUR\_USERNAME\AppData\Local\Autoshot\autoshot.yaml.
To fix this, the basic idea is to just rename that file to some other name, and send us the renamed file. Autoshot will re-make the autoshot.yaml file the next time it runs.
## Windows Explorer
One way to do this is simply to navigate in Windows Explorer to the directory, then right click on the autoshot.yaml file and rename it.
## Powershell
You can also open Windows Powershell (type power into the Windows search bar), and then enter these 2 commands:
> cd \$env:USERPROFILE\AppData\Local\Autoshot\
> Move-Item -Path .\autoshot.yaml -Destination .\autoshot\_backup.txt
Then, you can email us the “autoshot\_backup.txt” file, and we can help track down this problem.
# Blender Autoshot Errors
Source: https://docs.lightcraft.pro/debugging/blender-errors
Enabling Blender console log details in Autoshot.
When running a take in Autoshot, you may see an error that looks like:
> Return Code from Blender is 3 -- there was an error creating the blender render file.
By default, the console log doesn’t show all of the Blender console output, as it can be very long, but for cases like this it’s useful to see the complete output.
## Clear Console
![]()
We’ll first clear the console to remove text from previous runs, and make the debugging more straightforward. Click Clear on the lower right hand corner of the Autoshot window to clear the console.
## Enable All Blender Output
![]()
Next, we’ll enable seeing all of the Blender output by checking the ‘See All Blender Output’ checkbox under the Options panel in the center of the UI.
## Run Take and Copy Console text
![]()
Finally, we’ll click Save and Run to run the take, and then Copy when the take stops at an error. This will load the entire contents of the Console into the copy buffer.
## Save Console Text and Send File
Open up a text editor like Notepad or Notepad++ on Windows, or TextEdit on the Mac, and past the contents into the text editor.
Save the file as a text file, and then either post it on the forum or send it to [support@lightcraft.pro](mailto:support@lightcraft.pro).
# Exporting Takes for Debugging
Source: https://docs.lightcraft.pro/debugging/exporting-takes-debugging
Autoshot has a built-in take debugging system that makes it straightforward to send the complete dataset for a take.
## Export Take Zip
![]()
File -> Export Take Zip will write out the complete dataset of the selected take, including the captured Jetset camera original, live composite, depth video, and tracking metadata original files into a ZIP file archive that can easily be loaded into Autoshot.
We’ll use the same tool to debug the take that you’re using to work with it.
If the selected take is a Cine take, it will also write out the cine camera original file into the archive. This can become large, so write the zip file into a drive with sufficient space.
## Export Take Zip with Source Blend
Use this option if you’re working with a Blender scene file that you want to include in the zip archive. This is useful when debugging how a specific take aligns into the target blend file.
Autoshot will add whatever Blender file is currently pointed to by the ‘Scene Blend File’ line in the Run Values section.
## Send Take Via Email Link
Once you’ve zipped up the take, send the resulting (large) zip file to [support@lightcraft.pro](mailto:support@lightcraft.pro) as a link to a file download service like Dropbox, Google Drive, etc.
A take zip file can easily exceed a gigabyte in size, so they can’t be emailed directly.
## Opening Take Zip Files
You can use this to send take zips back and forth to your colleagues, as well. After you download a take zip file, unzip it to a hard drive (with enough space), then point the Autoshot Project Folder line to the Project folder inside the unzipped folder tree.
Autoshot will automatically recognize the take day and selected take (as there is only one take day and take in a take zip file!) and select them in the Take Selection section. You can then process the take normally.
# Jetset Crash Tracing
Source: https://docs.lightcraft.pro/debugging/jetset-crash-tracing
If you encounter a Jetset crash, especially one that recurs, we would appreciate your help in tracking it down!
## Share with App Developers
Inside your iPhone/iPad’s Settings menu, browse to Privacy & Security -> Analytics & Improvements -> Share With App Developers, and enable the switch:
![]()
You’ll need to enable this, and then restart the app before the log will start to be saved.
## Saving Log Files
Inside the Jetset app, go to Main Menu -> Settings & Support -> Settings, enable ‘Save log file in SPOOL folder’, and click Done.
![]()
Then, Jetset will start saving to (in the Files app) On My iPhone/Jetset/INTERNAL/SPOOL/jetset.log.
Note — don’t enable saving the log file in production; this is for debugging purposes only.
## Sending the Log Files
After you’ve enabled the above, and Jetset crashes, we still need to receive it!
To send it, navigate to the jetset.log file in the Files app (On My iPhone/Jetset/INTERNAL/SPOOL/jetset.log), and send it to [support@lightcraft.pro](mailto:support@lightcraft.pro) with a quick note on what was happening when Jetset crashed.
# Unreal Console Log
Source: https://docs.lightcraft.pro/debugging/unreal-console-log
How to capture the contents of the Unreal Console Log for debugging.
Autoshot uses Python scripting to generate the shot level sequences for regular shots, and the Composure composite for the Live Render Preview workflow.
After the user pastes the command string into the CMD bar and executes it, the results of each individual Python command are written into the Unreal Console Log. This provides a really useful trace for debugging.
## Example Video
To capture the Unreal console log text for an Autoshot run, please use this video as a guideline.
Note that before running the Autoshot script, the user right clicks on the Console log and clears it; this is because there can be thousands of lines of text from previous commands, but we only care about what the console result of the Autoshot run is.
After copying the Console Log text, users can paste it directly into the Lightcraft forums for debugging, or (if it’s long) into a separate text file and attach it to a forum post.
## Use the CMD Console
An easy mistake to make is to set the Unreal console setting to Python instead of CMD before pasting the Autoshot script. The Autoshot script is designed to work with the Unreal command (CMD) window:
![]()
# USD and USDZ Files
Source: https://docs.lightcraft.pro/debugging/usd-usdz
USDZ is the file format read by Jetset. When debugging a USDZ file, it’s important to understand a little about the structure of USD and USDZ files.
## USD Overview
USD is an acronym for Universal Scene Description, a file format created by Pixar for their animated movies, and later open sourced. When a USD ‘file’ is exported from Blender, Unreal, or other 3D applications, the result can be either a single USD file, or a top level file that then references in individual USD ‘asset’ files that are located in subdirectories.
Texture files are also exported separately into a subdirectory.
When Autoshot’s USD Model folder is pointed at a given directory, it will list the available top level USD files in that directory. When the user clicks ‘Make USDZ’ in Autoshot, the program will query the top level USD file for all the referenced USD and texture files in the subdirectories, and then assemble a USDZ file.
A USDZ file is just a folder containing a top level USD file and its referenced dependencies that has been zipped up with the standard ZIP archive, and renamed with a .usdz suffix. That’s all it is!
You can rename a .usdz file to a .zip file, and then unzip it into a directory to see how the original USD file structure was put together.
## Zipping USD Folders for debugging
This structured file system makes USD exports straightforward to debug and fix. The key thing to remember is that we’ll need not just the top level USD file, but the underlying /Textures and /Assets folders (if any), and their contents.
The easy way to get all this correct is to export from Blender, Unreal, etc. into a folder dedicated to this export, and then zip up the whole folder to send to us.
# Exporting OTIO with Avid
Source: https://docs.lightcraft.pro/editorial/avid/avid-export-otio
Use these techniques to correctly name shots in the Avid timeline for export to the OpenTimelineIO format.
To export OTIO files, select a sequence and choose Tools-> List Tool.
In the List Tool window, click on the dropdown menu for Output format and choose 'OTIO'.
[Reference](https://resources.avid.com/SupportFiles/attach/WhatsNew_MediaComposer_v24.6.pdf)
# Camera Checklist
Source: https://docs.lightcraft.pro/equipment/camera-checklist
This checklist covers recommended cine camera accessories for use with Jetset Cine.
## Accsoon SeeMo
### SeeMo 4K
![]()
[Accsoon SeeMo 4K HDMI Adapter](https://a.co/d/gciNr8d)
The SeeMo 4K will work with any camera that can output HDMI video.
4K video is not needed for calibration, as the Jetset Cine calibration process will work fine with 1080p video. The SeeMo 4K has an integrated SD card slot, but unfortunately the write speed to the SD card is quite slow for storing takes.
### SeeMo Pro
![]()
[Accsoon Seemo Pro SDI/HDMI to USB C](https://a.co/d/8X16VPa)
The SeeMo Pro can work with either SDI or HDMI video at 1080p resolution, and is constructed from aluminum for a more stage-durable configuration.
## HDMI Cables
The SeeMo does not ship with the HDMI cable that you will need to connect your camera to the SeeMo. The SeeMo uses a standard HDMI port, so the cables will either be:
* [HDMI to HDMI](https://a.co/d/7hk9IXO)
* [HDMI to Mini HDMI](https://a.co/d/1ymuTE0)
* [HDMI to Micro HDMI](https://a.co/d/b91FV95)
## Batteries
![]()
[SmallRig 10500mAh Replacement USB-C Battery](https://a.co/d/6MSAbn4)
These batteries are some of the largest capacity NP-F compatible batteries made, and can be directly charged with a high current USB-C charger.
## Battery Plate
![]()
[Kastar NP-F Battery Adapter Plate](https://a.co/d/bNYMBQd)
Even if you are not running Jetset Cine, if you are using Jetset in production you need external power and cooling.
This NP-F battery plate can screw to the side of the Smallrig iPhone cages, and holds a NP-F battery with a USB-C and USB-A power output.
## Cooling
![]()
[Black Shark Magnetic Cell Phone Cooler 4Pro](https://a.co/d/dJQ6dVt)
A cooler is required for production use. We specifically recommend the most recent BlackShark coolers due to their powerful cooling and reliable thermal connection.
When the iPhone gets too hot, the tracking performance is severely degraded and you may see sliding and drifting problems. The solution is to use an active cooling system like these.
These coolers can be powered with a USB-C 5V cable directly from the Accsoon SeeMo.
## Tentacle Sync E
![]()
[Tentacle Sync E MKII (Single Set)](https://a.co/d/0fdaWUwp)
The Tentacle Sync is a device that can generate synchronized timecode for multiple production cameras.
In addition, Jetset Cine can connect directly to it via Bluetooth, so that the recorded tracking data has timecode matched to the cine camera footage.
This can be used in 2 ways:
* As a real time data synchronization reference, for sending live camera tracking data to external rendering engines like Unreal. In this case, the embedded timecode in the tracking data is used inside Unreal to automatically synchronize with incoming live video data. Unreal supports AJA and Blackmagic SDI capture boards where the video signal can have embedded timecode.
* In post production, Autoshot will soon be able to use timecode embedded in the Jetset tracking data to automatically match Jetset takes with the corresponding cine camera takes, and to measure the timing offset between these two files. This is currently achieved with Jetset’s digital slate, using flashing ID markers, but embedded timecode is a more robust method that can do faster matching.
The Tentacle Sync devices can also jam sync to other sources of timecode, so they are compatible with a production running on Deity timecode systems.
## HDMI to HDSDI Converter
![]()
[Blackmagic Design Micro Converter Bidirectional SDI/HDMI](https://a.co/d/cxPW4Hc)
To use synchronized rendering with Unreal and Jetset Cine, the timecode that is injected into the camera by the Tentacle Sync E device has to go into a SDI capture card such as an AJA or a Blackmagic Decklink.
Cameras like the Blackmagic Pocket Cinema Camera that have an HDMI output need to convert the HDMI signal to HDSDI without losing the embedded timecode.
Most HDMI to HDSDI conversion devices won’t preserve timecode, but the above unit will, and is what we use in our testing.
# iPhone Hardware
Source: https://docs.lightcraft.pro/equipment/iphone-hardware
Each new version of the iPhone brings new capabilities. Find out which will work best for Jetset.
## iOS Hardware Families
ARKit, the underlying real time tracking technology that Jetset uses to measure camera motion, has been in iOS devices since 2017, but the capabilities of iOS devices have changed enormously during the development of Jetset.
The hardware capabilities largely break down into 3 ‘families’ of devices that can run Jetset, but **here's the key piece: use the newest 'Pro' hardware you can.**
The LiDAR devices in the newer iPhone Pro and iPad Pro units is a major tracking performance increase, and the newer units have far better GPUs and cooling than units from just a few years back.
* iPhone XR/XS, iPhone 11 Pro — these phones work, but their lack of LiDAR sensor and low GPU memory means they have model limit sizes and can’t do the 3D scanning that is really critical for Jetset Cine.
* iPhone 12/13/14 Pro — these phones have LiDAR which makes them a good fit for Cine. They can load mid sized models (Max USDZ file size: 150MB-250MB)
* iPhone 15 Pro Max and iPad Pro — these devices have powerful GPUs and good cooling. They also mark the introduction of the USB-C port for power and data. This enables the use of a USB-C to Ethernet cable for data offtake, which is 10x faster than the previous Lightning to Ethernet adapter, and is 20x faster than wifi. (Max USDZ file size: 300-500MB)
* iPhone 16 Pro Max - Improved power, cooling, and tracking over the iPhone 15 Pro Max.(Max USDZ file size -- 400-600 MB)
* iPhone 17 Pro Max - each new generation jsu
The differences come down to heat, LiDAR sensor presence/absence, and the size of 3D model that the GPU can render.
### Heat
Using Jetset for longer periods of time will heat up the device. The iPhone XR/XS, 11, 12, 13, and 14 all handle heat incrementally better, but an external cooler is still needed to use the phone in any sort of production scenario.
We recommend the [BlackShark](/equipment/camera-checklist#cooling) coolers, which can be powered by a USB-C connector.
Even when operating the iPhone without a Cine rig, you can use a [cage with a battery plate](/equipment/camera-checklist#battery-plate) to hold a NP-F battery. This enables power and cooling for the iPhone.
# Rigging Checklist
Source: https://docs.lightcraft.pro/equipment/rigging-checklist
This checklist covers recommended rigging equipment for Jetset and Jetset Cine.
## Overview
Jetset Cine uses an iPhone connected to a cine camera to accurately calibrate the cine lens optics and 3D offset from the iPhone.
For this to be accurate, a few goals are critical:
* Rigidity: once the phone and lens are calibrated to each other, moving the iPhone accidentally will disrupt the calibration. You want this to stay locked.
* Light weight: to balance the rig on a lightweight gimbal like a RS2 or RS3, the weight needs to be kept to a minimum.
* Cooler clearance: when running an iPhone Magsafe cooler (recommended), the iPhone cage can’t get in the way of the cooler mount.
* We’ll list some of the rigging setups we use from different manufacturers below.
### Camera Cage
![]()
[Camera Cage for Blackmagic Design Pocket Cinema Camera](https://www.smallrig.com/smallrig-cage-for-blackmagic-design-pocket-cinema-camera-4k-2203.html)
Blackmagic cage shown, but substitute specific cage for your camera.
### iPhone Cage
![]()
[Mobile Video Cage for iPhone 15 Pro Max](https://www.smallrig.com/smallrig-Mobile-Video-Cage-for-iPhone-15-Pro-Max-4391.html)
iPhone 15 Pro Max cage shown, but substitute specific cage for your camera.
### Magic Arm With Arri 3/8-16 mount
![]()
[Rosette Magic Arm (11") with 3/8”-16 ARRI Locating Screw](https://www.smallrig.com/Rosette-Magic-Arm-11-inch-with-ARRI-Locating-Screw-4900.html)
This magic arm has an Arri 3/8-16 thumbscrew on one side, and a standard 1/4-20 thumbscrew on the other end. The Arri side connects to one of the 3/8-16 mounts on the Smallrig cages, and provides much more gripping power. The 1/4-20 side screws into the threaded holes on the iPhone cage.
An example of the mounted setup can be seen here:
![]()
### 15mm Rail Block
![]()
[Super Lightweight 15mm-Railblock](https://www.smallrig.com/smallrig-super-lightweight-15mm-railblock-2pcs-pack-2061.html)
An alternative and more rigid way to mount the iPhone to the cage is with rail blocks mounted to both the iPhone cage and camera cage, and then connected with 15mm carbon fiber matte box rails.
This can take a bit more work to set up but is pretty bulletproof once rigged. The stock screw length is just a little long, so use a 1/4″ washer between the cage and the rails on either end to space them out and secure a tight fit.
### 1/4" Washers
![]()
[100PCS 1/4" Black Stainless Steel Flat Washers](https://a.co/d/5etoMr7)
Use these washers to take up the small amount of slack left when assembling the Smallrig cages to the rail block mounts.
### Carbon Fiber Rods
![]()
[Carbon Fiber Rod (15mm)](https://www.smallrig.com/15mm-carbon-fiber-rod-30cm-12inch-2pcs-851.html)
These 15mm carbon fiber rails fit the rail blocks shown above and are lightweight and rigid.
### Rail and Rod Rigging example
![]()
# Stage Checklist
Source: https://docs.lightcraft.pro/equipment/stage-checklist
This checklist covers recommended stage and network pieces for use with Jetset.
## Network
### Travel Router
![]()
[GL.iNet GL-MT3000](https://a.co/d/gDbEB5J)
Jetset & Autoshot use network connectivity heavily to transfer files, detect devices, and remotely trigger the digital slate.
Even if the location you are shooting at has an existing wifi service, it’s a good idea to bring your own wifi system so that you can maintain control of how your connectivity works.
Travel routers such as the Beryl AX3000 shown above are made to work ‘under’ an existing wifi network, or to link to a user’s iPhone to provide a link to external Internet.
They generate their own set of IP addresses, so that you can set up and test your Jetset iOS device, digital slate tablet, and Autoshot PC/Mac at your home or office, and then take the entire setup onto a stage shoot or on location and know that the IP addresses of the various devices will stay the same.
The travel router is very small, and can be located near your shoot. This removes a huge fraction of the connectivity problems that can otherwise cause a lot of problems under stage conditions.
Finally, it has direct Ethernet cabling to connect to the Autoshot PC/Mac and to an Ethernet network adapter on the iPhone. This can dramatically speed up transferring takes, especially in the case of the iPad Pro or iPhone 15 Pro models, which have a USB-C interface (and thus can transfer at gigabit Ethernet speeds with the appropriate network adapter.)
### USB-C Network adapter
![]()
[Belkin Connect](https://a.co/d/d514FUX)
Wifi works well for general connectivity with Jetset, but when transferring large amounts of takes it can be slow. For iPad Pro or iPhone 15 Pro models with a USB-C interface, it’s possible to transfer takes over a network far more quickly using a USB-C to Ethernet interface.
This can be directly plugged into the iPhone during times when the camera is not in direct use, and the gigabit Ethernet connection is over 10x faster than the speeds available with Lightning based connectors.
In a test case of sending a 360MB Gaussian splat over wifi vs. a USB-C to Ethernet connection, the wifi averaged about 30 Mbps, while the direct wired connection averaged over 800 Mbps! That’s the difference between transferring files in seconds instead of minutes.
## General
### Matte Cover for iPad
![]()
[Mothca Matte Glass Screen Protector for iPad](https://a.co/d/8GwLR7U)
Jetset has a built-in ‘digital slate’ which is simply a web page hosted by the Jetset app that can be viewed on a browser on the same LAN (we usually test with Chrome and Safari.)
When a Jetset take is rolled, the digital slate will display a series of black and white markers. For correct automatic take matching and timing, the markers need to be seen by both the iPhone/iPad running Jetset, and the cine camera.
These markers are then optically detected in post production using Autoshot, and used to automatically match cine footage with Jetset takes.
For this to work correctly, the black and white sections of the markers need to be readable and distinguishable from each other. A glossy surface on the iPad or laptop, coupled with bright overhead stage lights, can cause a bright reflection to appear in the recorded Jetset and cine images that prevents the detection algorithms from working.
To help prevent this, we recommend using an anti-glare filter on the tablet, laptop, or screen that is being used as the digital slate. The one above is what we have tested with, but almost any anti-glare covering should help prevent bright reflections off the glass.
### Vertical Origin Target Stand
![]()
[Display Poster Stand](https://a.co/d/68nWg5I)
This stand is lightweight and clamps the new 18″x24″ vertical origin optical targets easily for fast placement in front of the camera.
Use this with the new remote control origin setting tools in the Jetset digital slate/remote browser control web page.
### Lightcraft Optical Origin Markers
![]()
Lightcraft optical origin markers are shown in the [Introduction to Jetset](https://docs.lightcraft.pro/tutorials/jetset/introduction-jetset) tutorial, and are available on the [Downloads](https://lightcraft.pro/downloads/) page.
There are 2 sizes: letter and 16x24. These correspond to sizes that can be easily printed or ordered on Canva.com.
### Digital Green/Blue
![]()
[1 Gallon Digital Green® Paint](https://digitalgreenscreen.com/collections/digital-series-paint/products/1-gallon-digital-green-paint)
Digital Green/Blue Paint has the strongest difference between the green and red/blue channels. If you can paint your backdrop, use this according to the instructions. It requires a specific Kilz white primer to get the full intensity, but it’s absolutely worth it.
# Frequently Asked Questions
Source: https://docs.lightcraft.pro/faq/faq
Check these frequently-asked questions first!
## Help System
Search [Lightcraft's Documentation](https://docs.lightcraft.pro/)
# Embedding Audio Timecode
Source: https://docs.lightcraft.pro/jetset-cine/embedding-audio-timecode
If your cine camera does not accept direct timecode embedding, use this utility from Tentacle to convert timecode recorded in an audio channel into directly embedded timecode.
Many DSLR cameras lack a dedicated timecode port for reading timecode from the Tentacle Sync device. However, you can simply record the Tentacle timecode as audio into an audio channel of the cine camera, and then use this simple utility to embed the recorded audio timecode directly into the video file.
[https://tentaclesync.com/downloads/tentacle-timecode-tool-windows](https://tentaclesync.com/downloads/tentacle-timecode-tool-windows)
# Focal Length Override
Source: https://docs.lightcraft.pro/jetset-cine/focal-length-override
Use the Focal Length Override feature to handle very long zoom lenses that will not calibrate correctly with the default Jetset Cine optical feature based calibration system.
![]()
The existing Jetset Cine optical lens calibration system works well up to S35 equivalent focal lengths of about 70-85mm. Beyond that, the visible field of view of the cine lens is too small with respect to the Jetset lens, and the number of optical matches isn’t high enough to make optical lens calibration work well.
For focal lengths above this, we provide a Focal Length Override capability in Jetset Cine. This uses the next highest calibration you can achieve as a starting point, for example a 70mm prime in the same family.
You would load that calibration to get the offset and entry pupil correct, and then enter both the base calibration (70mm) and the override focal length (e.g. 100mm).
This provides a simplified approach, but it should get you fairly close as long as the base lens is close to the override lens.
This override should carry through into Autoshot and the connected post production tools.
# Production Checklist
Source: https://docs.lightcraft.pro/jetset-cine/production-checklist
Use this production checklist to ensure correct on-set calibration, preview, and capture with Jetset Cine and cinematic compositing.
## Production Setup
### Creating Jetset projects
Start off the process by creating a Jetset project. You'll want to determine:
1. Project storage (iCloud, App, or User-Defined)
2. Project name
3. Project prefix (up to 6 characters that will be placed in front of every file generated by Jetset for the project)
4. Camera letter (A Camera is usually the main camera)
For more details, see the [Introduction to Jetset](https://docs.lightcraft.pro/tutorials/jetset/introduction-jetset) tutorial, and check the 'Project Storage' chapter.
### Configuring and Calibrating
[Rigging Reference](https://docs.lightcraft.pro/tutorials/jetset-cine/jetset-cine-rigging)
[Calibration Reference](https://docs.lightcraft.pro/tutorials/jetset-cine/jetset-cine-blackmagic-braw)
1. Rigging:
Make sure to rig the iPhone in the same location that you plan to shoot with, as the lens calibrations will measure the exact offset between the iPhone and the cine camera lens.
2. Matching Tentacle Sync Timecode:
Verify that the Tentacle Sync timecode going into Jetset and the cine camera are matching.
3. Calibrate Cine Lenses:
Next, calibrate the cine lenses one at a time, processing each calibration with Autoshot as per the reference video.
4. Save the lens files with a unique description (e.g. project\_camera\_date\_focallength.)
### Set Origin and Scan
[Jetset Origin and Scanning Reference](https://share.descript.com/view/AqtB2jCnKio)
1. Add practical elements:
Before scanning or setting your origin, add all of the practical elements to the scene. The iPhone tracking will reference natural features in the scene, so you want to add all the non-moving physical elements that will be in the scene.
2. Reset Tracking and get anchor points:
In the Jetset Origin tab, click 'Reset' and move the iPhone all around the shooting area to capture as many of the natural features in the scene as possible.
3. Set Origin:
Now tap on one of the horizontal detected planes to set your tracking origin at the point you want your scene locator to be snapped to.
4. Jetset Scan:
Change to the Scan tab and click Start to start scanning the set. For the Jetset scan, you only care about the immediate area that will be in the shot.
5. Polycam scan:
Using a separate iPhone, it's a good idea to get an 'overall' scan of the set, including the greenscreen and the Jetset origin marker. This scan can be used in post production to augment the Jetset scans for challenging shots.
6. Enable Cinematic Compositing:
Turn on the Cinematic Compositing switch in Jetset, and verify that the origin and scan are still correctly located in the incoming cine video.
### Keying and Matting
[Keyer Reference](https://share.descript.com/view/AqtB2jCnKio)
1. In the main menu, choose the Keyer panel and pick the type of keyer in use (green in this example.)
2. Click the colored circle to toggle Jetset into the color select mode
3. Click and drag on the desired green color to use as the background reference color. The keyer will work in real time so you can see the effects of choosing different colors as you drag around. Release to select the color.
4. To set the garbage mattes, click the 'Start' button under the 3D mattes.
5. Move around the greenscreen to get a clear yellow outline on the vertical plane of the greenscreen. Click the '+' button to add a 3D matte.
6. Aim toward the floor, detect the outline of the greenscreen while the floor is detecting, and click the '+' button to set a floor matte.
7. Click the 'Edit' button and drag the silver balls around to make the 3D garbage matte best fit the greenscreen edge.
8. Click 'OK' to finish and exit.
### Take recording
[Jetset Cine with Digital Slate](https://docs.lightcraft.pro/tutorials/jetset-cine/jetset-cine-blackmagic-braw)
1. Check timecodes on BMD camera and Jetset
2. Enter any desired take info on digital slate (notes, updating scene/take, etc.)
3. Put digital slate in front of Jetset and cine camera
4. Roll cine camera
5. Roll Jetset
6. Capture take
7. Cut cine camera
8. Cut Jetset
## Post Production Operations
### Transfer takes
[Autoshot Transfer Reference](https://docs.lightcraft.pro/tutorials/jetset-cine/jetset-cine-blackmagic-braw)
1. In Autoshot, create a project folder with a matching name to the Jetset project.
2. Make sure Jetset is running, on the same network as Autoshot, and has the correct project loaded.
3. In the Autoshot Takes tab, pick the current day, and click 'Sync' to sync the takes.
4. Transfer the cine footage to a local directory on the Mac/PC, and point the 'Cine Source' to that directory.
5. Make proxies if shooting BRAW footage, and point the 'Cine Proxy' folder to the directory containing the proxies.
### Editing
1. Edit with either the timecode-matched Superdailies or the camera original footage/proxies.
2. Add a marker to each clip to name the shot (usually prefix\_scene\_shot, e.g. Pathways scene 1 shot 10 = PTH\_001\_010)
3. Export the timeline using OpenTimelineIO
### Processing Shots (Timecode Match)
1. Point the Cine Source directory at each individual shooting media shooting day, and set the Take Selection Day to match the shooting date.
2. Click 'Scan' to rapidly scan all the cine files in that directory and read the timecode. The Jetset takes will automatically be matched.
3. Use the Shots->Load Timeline to load the OpenTimelineIO file. This will automatically populate the shots with the named shots from the timeline.
4. Load the generated CSV file created from the previous step to see the list of shots, their shooting days and takes, and their USDZ files used.
5. Select the shooting day and take of one of the shots.
6. Configure the Run Values to link the blend file if used, and specify the manufacturer and LUT of the source files (e.g. Sony and S-Log3)
7. Select Shots->Quick Render to render the single shot. Open up the transcode file.
8. Select Shots->Quick Render and choose the 'All of same USDZ' choice to load all of the shots that used the same USDZ file as the currently selected shot.
9. Click the Pick Color button and pick a greenscreen color from the transcode file.
10. Click *Render* to batch render all of the shots.
### Processing Shots (RAW)
1. Generate proxies from the RAW files using the vendor's preferred proxy generation tool. The proxy files should have an identical name except for the .mov or .mp4 suffix.
2. As before, point the Take Selection day to match the shooting day of the Cine file directory (Cine files are frequently stored on a per-day basis.)
3. Point the Cine Source directory at the directory with the RAW files. This will enable the Cine Proxy directory line.
4. Point the Cine Proxy directory at the directory with the proxy files, and click Scan.
5. Configure the run values as above; with RAW files there is no need to set the manufacturer and LUT of the file.
### Processing Shots (Digital Slate Match)
1. As before, set the Take date to match the Cine source files to be scanned, and configure the Cine Source directory as above for either log or RAW files.
2. When clicking Scan, check the 'Use Markers' to use machine vision to search the cine files and the Jetset files for a match, and to set the time offset. Note: this can be quite slow compared to the timecode based matching.
### Manual Pick for Unmatched Cine Shots
1. If a take does not automatically match to a Cine file, and you know which file it should match to, you can click 'Manual Pick' and select the Cine Source file. (In the case of RAW files, pick the proxy file.)
2. If the Cine Offset cannot be manually calculated, use [the manual method](https://docs.lightcraft.pro/tutorials/jetset-cine/setting-cine-offset-manually) to set the Cine Offset so the take can be processed.
### Verifying shots
1. Open up the resulting rendered preview in the //preview folder to verify that the keying and tracking is correct.
### Refining Cine Offset
For shots using a timecode match, the Cine Offset time can be imprecise. To better align a shot in time, you can select the shot in question, and click the 'Refine Offset' button next to the Cine Time Offset number. An optical flow analysis will run on the cine footage and the Jetset footage, and the flow differences minimized. Then the shot can be re-run separately with the 'Save and Run' command.
# Jetset Cine Tracking Quality
Source: https://docs.lightcraft.pro/jetset-cine/reference-cine-tracking
This clip is a good reference for the tracking quality that we generally see from a Jetset Cine clip processed without additional tracking refinement.
This clip is a good reference for the tracking quality that we generally see from a Jetset Cine clip processed without additional tracking refinement. There is enough detail in the practical floor for Jetset to get a decent tracking lock.
This video shows the tracking data and scan data imported into Nuke with our Autoshot Nuke importer and rendered.
As you can see, it’s reasonably well lined up overall, but has small jitters in the track that don’t work for a shot with ground contact. For shots like this, we recommend refining the track using our [SynthEyes](/tutorials/tracking-compositing/tracking-refinement-syntheyes) or [GeoTracker](/tutorials/tracking-compositing/geotracker-blender) tracking refinement workflow (tutorial coming very soon.)
## Ground Contact Math
In truth, most real time tracking systems will have difficulties getting ground contact tracking accurate to sub-pixel levels, for simple reasons of optics. This chart helps explain why, using the equation:
Ground Contact Misalignment = Distance \* tan(Sensor Misalignment Angle)
## Calculations chart
| Sensor Misalignment Angle (deg) | Distance to Floor (cm) | Ground Misalignment (cm) |
| ------------------------------- | ---------------------- | ------------------------ |
| 0.1 | 200 | 0.35 |
| 0.2 | 300 | 1.05 |
| 0.3 | 400 | 2.09 |
| 0.4 | 500 | 3.49 |
| 0.5 | 600 | 5.24 |
The Distance to Floor represents how far a ray of light would travel from a visible spot on the floor to the camera lens. As you can see, very small angular misalignments in the tracking sensor result in considerable tracking misalignments as measured on the actual ground.
## Lightcraft's Approach
Lightcraft’s solution to this problem is to use a fast, lightweight real time tracking system (an iPhone or iPad), and then refine the track (when necessary for the shot!) using the actual visible points on the ground.
This is a ‘closed loop’ engineering technique, instead of the ‘open loop’ method that relies completely on perfect tracking alignment at the tracking sensor. Perfection is hard to achieve in production!
Not all shots need this refinement technique. This shot below is an example of a shot that works with the default Jetset Cine tracking data. Since there is no visible CG/live action contact point, the real time tracking data is accurate enough for the shot.
# Configuring iCloud (Optional)
Source: https://docs.lightcraft.pro/setup/configuring-icloud
If you plan to use iCloud to store your takes, follow this configuration guide. Note that most production groups will not use iCloud, and will instead use the App Storage or User-Defined storage choices described in [Creating Jetset Projects](/setup/create-project).
## Enable iCloud
![]()
If this error appears, Jetset is trying to find the iCloud storage. iCloud Drive must be enabled for this to work.
To enable iCloud Drive, go to Settings->User account -> iCloud. Make sure iCloud Drive is On.
![]()
When you click the iCloud Drive On button, it will take you to the iCloud Drive panel.
![]()
If you click **Apps Syncing to iCloud Drive,** you will see Jetset listed:
![]()
## Enable iCloud for Jetset
Jetset uses the user’s iCloud storage to write takes to. We use iCloud storage as until recently it was the only place to write files to in iOS that wouldn’t be deleted if the app was removed.
Find this in Settings->User account (the first button on top of the Settings menu) -> iCloud -> Apps Using iCloud -> Show All, and make sure that the slider next to Jetset is enabled:
![]()
## Browsing iCloud in Files Apps
If you are still seeing the ‘No iCloud’ error, try opening your Files app in iOS and browsing to the iCloud area. This can frequently catch any remaining problems. If you can browse to the iCloud top directory, Jetset should be able to write to local iCloud storage on the iOS device.
# Configuring Travel Routers
Source: https://docs.lightcraft.pro/setup/configuring-travel-router
This is a quick start guide to using the GL-MT3000 Net Travel Routers for stage operations.
## Power
The GLi Net routers use a standard 5V/3A USB-C power adapter.
## Connect
Connect your Mac/PC via Ethernet. The GLi Net router has a default IP address ofhttp\://192.168.8.1.
Choose your language.
There is no default password for the Admin panel. You have to set your own.
## Configure Internet
The most likely configuration methods you will use are as a Repeater or Tethering.
### Repeater
This method connects to an existing wireless network. Choose a SSID from the drop-down list and enter its password. Toggle *Remember* on and then *Join*.
### Tethered
This method uses another iPhone connected to the outside internet over cellular over a USB cable.
Connect the iPhone to the USB port of the router. Click 'Trust' to continue.
Go to iPhone->Settings->Personal Hotspot and enable 'Allow Others to Join'.
On another computer or iPhone, connect to the router, go to the web Admin panel, choose 'Internet' and click 'Connect'.
It will show connected information at the top of your phone screen.
More details can be found in the GLi Net [online manual](https://docs.gl-inet.com/router/en/3/setup/gl-mt1300/first_time_setup/).
# Creating Jetset Projects
Source: https://docs.lightcraft.pro/setup/create-project
Setting up Jetset project location, name, and prefix
## Project Panel
Access Jetset's Project panel from the gear menu -> Project.
![]()
## Project Location
To choose the top level project location, click the 'Path' bar with the up and down arrows.
Jetset projects can be stored in 1 of 3 locations:
1. App Storage (stored inside the Jetset app memory)
2. iCloud Storage (stored in a part of the iOS local storage that is copied to iCloud automatically)
3. User-Defined (user-selected folder in the iOS Files system)
### App Space
App Storage is completely local. It's not copied to any cloud data, which is good for IP-sensitive projects, but the downside is that the take storage is deleted if you uninstall the Jetset app!
### iCloud
iCloud storage is the default. Takes are automatically backed up to the user's iCloud account if it has storage space. This should be avoided for IP-sensitive projects.
### User-Defined
This option will let you select a location anywhere that the iOS file picker will select, including external SD card readers.
## Creating Projects
Once the project location is set, click the + button to create a new project. It will default to the name 'MyProject' with the prefix 'myproj'.
### Renaming Projects and Prefixes
Once a project is created, click the 'pencil/edit' button and then tap on the project name to rename it. Project names may not include spaces.
Click the 'edit' button again and tap on the 'prefix', which is the six character string that will be the first part of every file created for this Jetset project.
## Setting Camera
Jetset's 'Camera' setting by default is set to 'A'. To change this, click the 'A' and enter in the camera name. Only uppercase letters are allowed.
## Saving
Once the path, project name, prefix, and camera letter are set, click 'Save' to save these settings.
The current project name will be displayed in the middle lower section of the Jetset main display.
# Digital Slate/Remote Control
Source: https://docs.lightcraft.pro/setup/digital-slate
Jetset has an integrated web server that enables external devices like laptops or tablets to work as a digital slate, tested with Chrome and Safari.
## Anti-Glare Covers
Tablet and notebook screens are by default highly glossy, and reflect the environement. This can cause problems with the automatic ID marker detection in Autoshot.
To prevent this, use a [standard anti-glare cover](/equipment/stage-checklist#matte-cover-for-ipad) applied to the screen.
## Connecting
To connect, you can either manually enter the IP address of the Jetset device you’re connecting to, or click the Open Webpage button in Autoshot:
![]()
## Displayed data
![]()
The digital slate display is color coded.
* Fields that are colored grey can only be changed on the Jetset app.
* Fields that are colored gold can be directly edited on the slate, and the edit will automatically transfer to the Jetset app.
The following data is displayed on the digital slate:
* TOD TC: time of day timecode. This updates once per second (due to browser limitations) so the Frames part of the display stays at :00.
* Lens: this shows the current simulated focal length of the Jetset device.
* Camera, Roll, Scene, Take: current take configuration.
* UUID: the current unique 10 digit hex identifier that is created for each individual Jetset take.
* Lens File: when using Jetset Cine, this shows the current lens calibration file loaded for the take.
* LOC: user defined field. Usually INT Day, EXT Night, etc.
* DIR: director
* DP: directory of photography
* NICK: the six digit (or less) ‘prefix’ defined for the project. This should be identical across all project files for rapid sorting.
* DATE: current shooting date
* NOTE: space for user entered notes.
## Embedding Slate Data in Take JSON
![]()
The custom fields can be used to enter name and contact information on a per-take basis. For example, in the above case the director is named John, the DP is Ringo, and the note references [support@lightcraft.pro](mailto:support@lightcraft.pro).
![]()
This data is stored in the \{takeID}\_zl\_meta.json file located in the /Takes/Data/CamA/TRACKING directory.
The JSON line of data for this take then includes these fields:
"webDP":"RINGO",
"webNOTE":"[SUPPORT@LIGHTCRAFT.PRO](mailto:SUPPORT@LIGHTCRAFT.PRO)",
"webDIR":"JOHN",
These can easily be searched with standard text searches.
# Windows Network
Source: https://docs.lightcraft.pro/setup/windows-network
Set the Windows network configuration as Private on the network properties if using standard Windows Defender setup. You want whatever is friendly to printer sharing.
![]()
# Zoom Connection
Source: https://docs.lightcraft.pro/setup/zoom-connection
Jetset includes a direct Zoom integration that is accessed via a link sent to the Jetset user’s iOS device..
## Making Links
To create a link, you can go to the [Jetset Link Generator](https://lightcraft.pro/jetset-link-generator) page.
![]()
### Zoom Connections
To create a link that embeds a Zoom call connection, add the Zoom Meeting ID and Passcode from the Zoom meeting info panel (click the green shield icon in the upper right hand corner.)
![]()
### Zoom Meeting Details
The first time the client opens a Jetset link with Zoom meeting parameters, a Share Screen button will appear on the main screen in the upper right. Clicking on the Share button will open a panel and join the client into the Zoom meeting.
Clicking on the Broadcast button in the panel will start the screen share into the Zoom meeting.
You can hide the Share button from the UI by opening the link jetset:://unsharescreen.
Jetset’s Zoom integration is video only; this is by design so that a user can link directly into a Zoom session right next to the Jetset camera without causing audio interference in the Zoom call.
The passcode has to be the 6-7 digit number in the panel, as opposed to the encrypted value sent in a traditional Zoom link.
### Cloud File URL
The link can optionally contain the address to a USDZ file that can be accessed with a Web link (Dropbox, Google Drive, etc.)
Just paste the link into the Cloud File URL line, and enter the model file name that you wish the USDZ file to be saved as in the Jetset /models directory. (Optional -- if empty it will use the original model filename.)
### Generate Link
Click 'Generate Link' to create a jetset:// link. This can be emailed or sent to other users, and clicking the link on their device will open up Jetset and enable the screen sharing system or automatically load the linked USDZ model.
Note that some email systems like GMail won't correctly recognize these links. Fortunately, the Link Generator page can also create QR codes, which always work.
### Generate QR code
Clicking 'Generate QR Code' will create a QR code that is identical to the generated jetset:// link above.
This QR code is activated by viewing it in the standard iOS Camera app. A small yellow button will appear; click this yellow button and Jetset will open up wiht the screen share and/or USDZ file load
## Link Format Reference
The link has the format jetset:// followed by optional parameters.
The Zoom meeting link has the paramters meeting=\&pass=
This can be used without or without an automatic model open.
Make sure to not have any spaces in the meetingID.
Without a model as jetset://zoomlink?meeting=\&pass=
Example: jetset://zoomlink?meeting=88071557827\&pass=111111
With a model as jetset://usdz?url=\&meeting=\&pass=
The meeting room may need to be configured to accept unauthenticated users.
# Arranging objects
Source: https://docs.lightcraft.pro/spark/arranging-objects
Position, rotate, and scale objects in your scene using transform tools and the outliner
## Transform gizmo
Use the transform gizmo to move, rotate, and scale objects directly in the 3D canvas. Switch between modes using the toolbar buttons in the upper right of the 3D canvas, or use keyboard shortcuts:
* **Move** (**Ctrl+1**) — Drag along an axis to reposition the object.
* **Rotate** (**Ctrl+2**) — Drag a ring to rotate around an axis.
* **Scale** (**Ctrl+3**) — Drag a handle to resize along an axis.
These shortcuts apply to both USD assets and cameras.
## Numeric editing
For precise placement, edit transform values numerically in the properties panel. Enter exact position, rotation, and scale values for the selected object.
## Keyframes
To animate an object's position over time, select it and toggle **Keyframes** on in the asset settings panel. With Keyframes enabled, scrub the timeline to your start point, position the asset, scrub forward, and reposition — each move writes a keyframe automatically.
Spark Story does not enable Keyframes automatically. If you move an asset with Keyframes off, the change will not persist as animation.
## Locators
Locators are named reference points in 3D space. Use them to:
* Mark positions where physical objects will be placed on set.
* Align scenes between Spark Story and Jetset.
* Create anchor points for consistent object placement across scenes.
* Jump between locations in large scenes using Lightcraft Spark on iPhone or iPad.
To add a locator from the browser, click **Add** in the Outliner, then **Locators**, and click the position in the 3D scene. Name each locator descriptively (e.g., "Kitchen," "Hilltop," "Stage Left").
In Lightcraft Spark on iPhone or iPad, open the locators menu and tap a locator to teleport the virtual camera to that point.
See [scene locators in Blender](/create-usd/blender/scene-locators) and [object locators in Blender](/create-usd/blender/object-locators) for creating locators in your 3D pipeline.
## Outliner
The outliner panel shows the full hierarchy of objects in your scene. Use it to select, rename, reorder, and organize objects. Drag items in the outliner to reparent them within the hierarchy.
# Cameras and shots
Source: https://docs.lightcraft.pro/spark/cameras-and-shots
Set up shot cameras, record camera moves, and manage keyframes in the scene editor
## Understanding shots
Each scene can contain one or more shots. A shot represents a specific camera setup — its position, orientation, movement, and duration. By default, each scene starts with one shot tied to its script heading.
## Viewing shot cameras
Shot cameras appear in the 3D canvas as camera icons with a frustum showing their field of view. Select a shot camera to see its framing in the viewport.
## Creating shots from the script panel
To create a shot with script-line attribution:
1. Open the **Script** tab on the right side of the 3D canvas.
2. Highlight the line or block of script this shot covers.
3. Click **Create Shot**.
![]()
The Create Shot dialog opens with these fields:
* **Name** — Prefilled with a sequential number (e.g., "Shot 23"). Override for a more descriptive name.
* **Preset** — Choose a shot type preset.
* **Lens** — Select from real lens models. Pick the one you plan to shoot on so what you see in Spark Story matches what you get on set.
* **Objects** — Select which objects the shot covers.
* **Description** — Optional notes about the shot.
![]()
You can also create a shot from the Outliner with **Add > Shot**, but this skips script-line attribution and breaks coverage tracking.
Click in the 3D scene to place your camera. Press **Ctrl+F** to enter full-screen camera preview, and press it again to exit.
## Recording camera movement
Use fly mode to navigate through your scene and record the camera path in real time. The recorded movement is captured as keyframes on the timeline.
Toggle **Keyframes** on for the camera before moving it. Without Keyframes enabled, camera moves will not persist as animation on playback.
## Setting manual keyframes
For precise camera animation, set keyframes manually:
1. Move the playhead to the desired time on the timeline.
2. Position the camera in the viewport.
3. Set a keyframe to lock the camera's position and rotation at that frame.
## Camera targeting
To follow a moving target, select the camera, toggle Keyframes on, then choose one of these targeting modes:
* **Aim At** — The camera rotates to track the target while staying in place. Select the target object from the dropdown. Add a keyframe at the start of the move — without Keyframes on, Aim At appears to work in the moment but quietly loses its target on playback.
* **Parent** — The camera physically rides with the target (useful for vehicle interiors or dolly moves). The camera inherits the target's movement.
![]()
## Camera frustum visualization
The camera frustum shows the visible area from the camera's perspective. Use it to check framing and ensure objects are within the shot.
## Editing shot duration
Adjust the duration of a shot by modifying its range on the timeline. Longer durations give the camera more time to complete its movement.
# Collaboration
Source: https://docs.lightcraft.pro/spark/collaboration
Work together on scripts with real-time editing, presence indicators, and version history
## Real-time co-editing
Multiple team members can edit the same script simultaneously. Changes appear in real time across all connected browsers.
## Presence indicators
When collaborators are active in a script, you can see their cursors and selections highlighted in the editor. Each collaborator is assigned a distinct color.
## Publishing a version
Publishing a version creates a named snapshot of the script that the team can reference. Published versions serve as checkpoints — useful for table reads, production meetings, or locking a draft.
## Reverting to a previous version
If you need to roll back changes, you can revert the script to any previously published version. Reverting replaces the current working draft with the selected version.
## Version history
The version timeline shows all published versions in chronological order. Select any version to view its contents or revert to it.
# Common pitfalls
Source: https://docs.lightcraft.pro/spark/common-pitfalls
Solutions for frequent issues in Spark Story and Lightcraft Spark
## Camera or asset moves break on playback
You forgot to toggle **Keyframes** on for the asset or camera before moving it. The interface does not warn you. Select the object, enable Keyframes in the asset settings panel, then redo the move.
## Composited actor looks giant or doll-sized
The splat scale is wrong. Return to the Location Settings panel, drop in the six-foot reference figure, and rescale. Every camera distance and composited element downstream depends on correct scale.
## Teammate sees a different script version
You did not publish the latest version. Open the version sidebar and click **Publish** on the version you want everyone working from.
## Teammate cannot see your environment
You used **Load Locally** instead of **Upload** when importing the location file. Load Locally keeps the file on your machine for the current session only — collaborators opening the same scene will not see it. Re-import with Upload so the file writes to the production library.
## Scene loads in Spark Story but fails on iPhone or iPad
Your splat exceeds the roughly 5 million splat cap on Lightcraft Spark for iPhone or iPad. Downsample the splat for the phone and keep the full-resolution version for the browser and Jetset.
## Pre-animated USD sits frozen on iPhone or iPad playback
You dropped the animation clip on the timeline but did not toggle **Keyframes** on for that object. Spark Story may render it correctly in the browser while the iOS app does not.
## Lightning bolt does not engage
Tap it again. The handshake between Lightcraft Spark and Spark Story occasionally needs a second attempt.
## Driving across a large scene is slow
Drop scene locators at every action point and jump between them instead of flying. See [arranging objects](/spark/arranging-objects) for creating locators in the browser, or add them directly in Lightcraft Spark.
## Splat chokes the browser
If a splat exceeds roughly 7 GB, keep a downsampled version for previs and reserve the full-resolution version for Jetset on set.
## AI video tool output looks wrong
Download the MP4 from the shot list. AI video tools like Seedance produce cleaner output from a real Spark Story shot than from a text prompt, because the camera move is already locked and consistent.
## Keying needs green screen or blue screen
Lightcraft Spark compositing is AI matting only. Route through [Jetset](/spark/exporting-to-jetset) for green-screen or blue-screen keying workflows.
# Compositing and recording
Source: https://docs.lightcraft.pro/spark/compositing-and-recording
Record takes with live AI compositing in Lightcraft Spark on iPhone or iPad
Lightcraft Spark can composite a real performer into your virtual scene in real time using AI matting — no green screen required for previs.
## Enter compositing mode
Tap the third button on the left side of the shooting view to enable compositing mode. Lightcraft Spark uses AI matting to key the performer against the virtual background.
![]()
## View modes
Use the toggles to switch between four views while compositing:
* **Live composite** — The performer keyed over the virtual background.
* **Camera feed** — Raw iPhone or iPad camera footage.
* **Depth map** — Per-frame depth information.
* **Matte channel** — The AI-generated matte used for keying.
![]()
## Record a take
1. Frame the shot with the performer in position.
2. Tap **Record**.
3. Run the action.
4. Tap **Stop**.
5. Name and save the take.
![]()
The take appears in your shot list in the browser within seconds, ready for the team to watch and comment on.
## Output files
Compositing-mode takes write five files to the iPhone or iPad for every take. Find them in the Files app under the Spark folder:
| File prefix | Contents |
| ----------------- | --------------------------------------- |
| `cam_*` | Raw iPhone or iPad camera footage. |
| `comp_*` | The live composite the operator framed. |
| `depth_*` | Per-frame depth map. |
| `tracking_*.json` | 3D camera tracking data. |
| `meta_*.json` | Take metadata. |
## Offline behavior
Lightcraft Spark uses the same real-time database as Spark Story. A shot recorded while the phone is offline saves locally and syncs to Spark Story the next time the phone reconnects.
Lightcraft Spark compositing uses AI matting only. For green-screen or blue-screen keying, use Jetset instead. See [exporting to Jetset](/spark/exporting-to-jetset) for how to send your scene to Jetset for on-set production.
# Core concepts
Source: https://docs.lightcraft.pro/spark/core-concepts
Key terminology and concepts used throughout Spark Story
## Organization
A team workspace that contains members and productions. Each organization has its own set of roles and permissions. You can belong to multiple organizations.
## Production
A project container within an organization — typically one per film, episode, or commercial. Productions hold scripts, scenes, and shot lists.
## Script
A Fountain-format screenplay tied to a production. Scene headings in a script automatically generate scenes that you can build out in the scene editor.
## Scene
A 3D workspace generated from a script heading. Each scene has its own canvas, timeline, outliner, and properties panel where you arrange assets, set up lighting, and position cameras.
## Shot
A camera take within a scene. Shots define a camera position, movement, and duration. You can record camera moves in fly mode or set manual keyframes.
## Shot list
A curated collection of shots for planning or review. Shot lists help you organize the shots you need across scenes and share them with your team.
## Scene file
An imported 3D asset used within a scene. Supported formats include USD, USDZ, PLY, and GLB.
## Locator
A named reference point in 3D space. Locators help align objects between Spark Story and Jetset, and serve as anchors for positioning assets on set. See [locators in Blender](/create-usd/blender/scene-locators) for details on creating them in your 3D workflow.
# Creating shot lists
Source: https://docs.lightcraft.pro/spark/creating-shot-lists
Build and organize shot lists for production planning and review
## What shot lists are for
Shot lists help you organize the shots you need to capture across your production. Use them for planning shoot days, running through sequences in review meetings, or preparing call sheets.
## Creating a new shot list
Create a shot list from within a production. Give it a name that reflects its purpose — for example, "Day 1 exteriors" or "Chase sequence."
## Adding shots
Add shots from any scene in the production to your shot list. Each entry references the original shot in its scene, so changes to the shot (camera position, duration) are reflected in the list.
## Reordering shots
Drag shots within the list to set a custom order. You can arrange them by shooting schedule, story order, or any sequence that fits your workflow.
## Multiple shot lists
You can run multiple shot lists in parallel — for example, a director's list, a DP's list, and a second-unit list. Spark does not force one canonical list.
## View modes
Switch between two view modes:
* **Card grid** — Displays shots as visual cards with preview thumbnails.
* **List** — Displays shots in a compact list format with key details.
## Downloading previews
Click **Download Preview** on any shot to export an MP4. This is useful for sharing outside Spark Story or as input for AI video tools, where a real camera move produces cleaner output than a text prompt.
## Compositing-mode take files
Takes recorded in compositing mode on Lightcraft Spark also write five files to the iPhone or iPad. Find them in the Files app under the Spark folder:
| File prefix | Contents |
| ----------------- | --------------------------------------- |
| `cam_*` | Raw iPhone or iPad camera footage. |
| `comp_*` | The live composite the operator framed. |
| `depth_*` | Per-frame depth map. |
| `tracking_*.json` | 3D camera tracking data. |
| `meta_*.json` | Take metadata. |
See [compositing and recording](/spark/compositing-and-recording) for the full recording workflow.
# Exporting to Jetset
Source: https://docs.lightcraft.pro/spark/exporting-to-jetset
Export scenes from Spark Story to Jetset for on-set virtual production
## Workflow overview
The Spark-to-Jetset workflow bridges pre-production and production. Scenes designed in Spark Story can be exported and loaded directly onto an iPhone or iPad running Jetset, bringing your planned shots to the physical set.
## USDZ export
Export a scene as a USDZ file to package all 3D assets, materials, and scene data into a single file that Jetset can load. The export process generates an optimized file sized for mobile performance.
## QR codes
After exporting, Spark Story generates a QR code for the scene. Scan the QR code with Jetset on your iPhone or iPad to load the scene directly — no manual file transfer required.
## What transfers to Jetset
The following elements carry over from Spark Story to Jetset:
* 3D assets and materials
* Camera positions and shot setups
* Locators and reference points
* Scene hierarchy and object relationships
## Jetset vs Lightcraft Spark
Use **Lightcraft Spark** for previs and live blocking. Use **Jetset** when a cinema camera is on set:
| Capability | Lightcraft Spark | Jetset |
| ---------------------------------- | ---------------- | ------ |
| AI matting | Yes | Yes |
| Green-screen keying | No | Yes |
| Blue-screen keying | No | Yes |
| Cinema camera coupling (e.g., RED) | No | Yes |
| Photogrammetry | No | Yes |
| Tracking data export | Yes | Yes |
## Next steps on set
Once the scene is loaded in Jetset, you can begin tracking and compositing on set. See the Jetset documentation for next steps:
* [Jetset introduction](/tutorials/jetset/introduction-jetset)
* [Creating a project](/setup/create-project)
* [Scene locators in Blender](/create-usd/blender/scene-locators)
# Importing locations
Source: https://docs.lightcraft.pro/spark/importing-locations
Add 3D locations to your scenes in supported formats
## Supported file formats
Spark Story supports the following 3D file formats for locations:
* **USD** — Universal Scene Description
* **USDZ** — Compressed USD package
* **PLY** — Polygon file format (commonly used for Gaussian splats and photogrammetry scans)
* **SPLAT** — Compressed 8-bit splat format
* **SPZ** — Compressed Gaussian splat format
## Adding location files
From the Outliner panel, click **Add**, then **File**. The Assets Library opens with filter tabs for Location, Object, and Audio. To bring in a new environment, click **Upload** in the upper right and select your file. To reuse something already in the production, click an existing asset tile.
### Upload vs Load Locally
* **Upload** writes the file to your production library. Every teammate in the production gains access to it.
* **Load Locally** loads a file from your local drive for the current session only. Collaborators opening the same scene will not see it.
Use Load Locally for quick personal tests. Use Upload for anything the team needs.
## Orientation and scale
### Fixing orientation
After importing, fix orientation in the Location Settings panel:
* **Invert Up (Flip 180°)** — Toggle if the splat loaded upside down.
* **Z-Up (Blender/3ds Max)** — Toggle if the splat came out of Blender or 3ds Max with the wrong up axis.
* **Position and Rotation fields** — Type values into the X, Y, Z fields for finer correction.
![]()
### Setting scale
Click **Add Reference** in the Scale Reference section. Two methods are available:
* **Ruler tool** — Click two points on a feature whose real-world size you know (for example, the top of a doorway and the floor). Spark displays the measured distance in your chosen unit. If the reading is wrong, type the correct real-world dimension into the field and Spark rescales the splat to match.
* **Six-foot human figure** — Drop it into the scene and compare the splat against a known human height.
![]()
### Scale controls
Once a reference is added, the scale controls become active. The row reads **x0.1**, **-1**, **-0.1**, numeric field, **+0.1**, **+1**, **x10**. Use x0.1 and x10 for large jumps, the +/-1 and +/-0.1 buttons for fine steps, or type a value directly into the field.
Scale errors cascade. Every camera distance, lens choice, and composited actor downstream depends on scale being correct. Getting it right at upload saves rework later.
### Collision mesh
Optionally, add a collision mesh by clicking **Select GLB file**. The collision mesh defines physics collision boundaries for the location.
## Splat size guidance
Spark Story handles very large splats. Files up to 7 GB have rendered in the browser with no hard maximum confirmed. Lightcraft Spark on iPhone or iPad caps at roughly 5 million splats per scene — if your splat exceeds that count, the scene will not load on the phone. Keep a downsampled version for mobile previs.
## File caching
Imported files are cached for faster loading on subsequent visits. Large assets may take a moment to load the first time they are added to a scene.
# Importing objects
Source: https://docs.lightcraft.pro/spark/importing-objects
Add 3D files to your scenes in supported formats and manage scene files
## Supported file formats
Spark Story supports the following 3D object file formats:
* **USD** — Universal Scene Description
* **USDZ** — Compressed USD package
## Adding object files
Add 3D files to your scene through the outliner panel. Imported files appear in the scene hierarchy and can be positioned, rotated, and scaled using the transform tools.
## File caching
Imported files are cached for faster loading on subsequent visits. Large assets may take a moment to load the first time they are added to a scene.
## Preparing assets for Spark Story
For best results, export USD objects with the MaterialX and USD Preview Surface materials when available.
# Importing scripts
Source: https://docs.lightcraft.pro/spark/importing-scripts
Upload existing screenplays in Fountain or Final Draft format to Spark Story
## Uploading an existing .fountain or Final Draft .fdx file
If you already have a screenplay in `.fountain` or Final Draft `.fdx` format, you can upload it directly. The script editor parses the file and generates scenes from your scene headings automatically.
# Importing Unreal scenes
Source: https://docs.lightcraft.pro/spark/importing-unreal-scenes
Export a scene from Unreal Engine as a Gaussian splat and USD file for use in Spark Story
Export a scene from Unreal Engine as both a Gaussian splat (for the environment) and a USD file (for foreground mesh geometry), then bring them into Spark Story.
## 1. Separate foreground elements
Identify which objects in your scene need to stay as mesh geometry — anything with fine detail the splat might miss, or elements you may want to shoot through (railings, stairs, furniture).
1. In the Unreal Editor, open the **Layers** panel.
2. Add a new layer and name it **Mesh**.
3. Drag the foreground elements you want to keep as mesh geometry into this layer (e.g., tables, chairs, railings, staircases).
## 2. Add scene locators
Scene locators mark positions in the scene where you want to place actors or quickly jump to during previs.
1. Add an empty actor to the scene.
2. Rename it using the naming convention `Scene_Lock_01` (replace `01` with a descriptive name or number).
3. Position it where you want the first locator.
4. **Alt+drag** the actor to duplicate it and create additional locators at other key positions.
5. Add the scene locators to your **Mesh** layer so they export with the foreground elements.
## 3. Install the USD exporter plugin
1. Go to **Edit > Plugins**.
2. Search for **USD Importer** (this plugin handles both import and export).
3. Enable the plugin.
4. Restart Unreal Engine.
## 4. Export as USD
1. Select the **Mesh** layer (double-click it to select all its contents, including the scene locators).
2. Go to **File > Export Selected**.
3. Choose **USD** from the file type dropdown.
4. In the USD export settings, enable **Bake Materials**.
5. Expand the material settings menus and remove any generated elements you don't need.
6. Click **Export**.
## 5. Render a panoramic 360 image
Render a single panoramic frame from Unreal Engine to use as the source for your Gaussian splat.
1. Go to **Edit > Plugins** and enable **Additional Render Passes**.
2. Add a **Camera** actor to the scene and position it in the center of your environment.
3. Open the **Sequencer**, drag your camera in, and set the in and out points so it renders a single frame.
4. Go to the **Media Render Queue** for the sequence.
5. In the render settings, delete the default **Deferred Rendering** pass.
6. Go to **Settings > Panoramic Rendering** and enable **Allocate History Per Pane**. Without this, the render will come out too dark.
7. Add any anti-aliasing settings you need.
8. Set the output resolution to **4096 x 2048** (2:1 aspect ratio).
9. Render the frame.
## 6. Convert to a Gaussian splat with World Labs
1. Open [World Labs](https://www.worldlabs.ai/) and choose **2D Input**.
2. Drag and drop your panoramic render into World Labs.
3. World Labs generates a Gaussian splat from the image, preserving the lighting from your Unreal scene.
4. Download the result as a **PLY** file.
## 7. Import into Spark Story
Bring both the Gaussian splat and the USD foreground elements into Spark Story.
1. In Spark Story, click **Add** and choose **Location**.
2. Upload your PLY Gaussian splat file. This becomes the environment background with baked lighting.
3. Click **Add** again and choose **Object**.
4. Import your USD or USDZ file containing the foreground elements (tables, chairs, staircase, etc.).
The final scene combines mesh detail for foreground elements with a Gaussian splat background that preserves the original Unreal Engine lighting.
# Introduction
Source: https://docs.lightcraft.pro/spark/introduction
What Spark Story is, where it fits in the Lightcraft ecosystem, and who this documentation is for
## What is Spark Story?
Spark Story is a collaborative pre-production tool for virtual production. It lets teams write scripts, build 3D scenes, plan shots, and export everything to set — all from a web browser.
## Where Spark Story fits in the ecosystem
Spark Story is the starting point of the Lightcraft virtual production pipeline:
1. **Spark Story** — Write, design scenes, and plan shots in pre-production.
2. **Jetset** — Capture camera tracking and composites on set using an iPhone or iPad.
3. **Autoshot** — Bring tracked data into Blender, Unreal, After Effects, or Nuke for post-production.
Scenes built in Spark Story export directly to Jetset, carrying over assets, camera positions, and locators.
## High-level workflow
A typical Spark Story workflow follows these steps:
1. **Write** — Create or import a Fountain or Final Draft formatted screenplay.
2. **Build scenes** — Arrange 3D assets, set up lighting, and position cameras for each scene heading.
3. **Plan shots** — Record camera moves, organize shot lists, and share previews with your team.
4. **Export to set** — Generate USDZ files and QR codes so Jetset can load your scene on an iPhone or iPad.
## Who this documentation is for
This documentation is for anyone using Spark Story, including directors, cinematographers, producers, and virtual production supervisors. No prior 3D experience is required — the guides start from account creation and build up to advanced scene editing.
# iPhone or iPad camera controls
Source: https://docs.lightcraft.pro/spark/iphone-camera-controls
Navigate scenes and configure camera settings in Lightcraft Spark on iPhone or iPad
Lightcraft Spark provides two ways to move through a scene: touch controls and AR mode. Both send 3D coordinates (not video) back to Spark Story, which is why live camera sharing works across cities without lag.
## Touch navigation
Use the on-screen controls to move through your scene:
* **Finger swipe** — Look around by dragging on the viewport.
* **Joystick** — Move forward, backward, and strafe using the on-screen joystick.
* **Height control** — Rise and lower the camera.
* **Speed control** — Adjust movement speed to cover large scenes quickly or move precisely in tight spaces.
![]()
## AR mode
Tap the second button on the left side of the shooting view to enter AR mode. In AR mode, physically moving the phone drives the virtual camera. Dial the speed up to cover a large scene from your chair.
![]()
## Camera settings
Open camera settings to configure the virtual camera:
* **Lens** — Select from real lens models. Pick the one you plan to shoot on so what you see in Spark matches what you get on set.
* **T-stop** — Set the aperture for depth-of-field reference.
* **Focus** — Adjust focus distance.
* **Camera body** — Choose a camera body to match your production setup.
* **Zoom** — Pinch or scroll to zoom in and out.
![]()
## Scene locators
If your scene has locators set up in the browser, open the locators menu in Lightcraft Spark and tap a locator to jump directly to that point. This is especially useful in large scenes where driving across the environment is slow.
![]()
See [arranging objects](/spark/arranging-objects) for how to create locators in the browser.
# Lightcraft Spark setup
Source: https://docs.lightcraft.pro/spark/lightcraft-spark-setup
Install Lightcraft Spark on your iPhone or iPad and connect to your Spark Story scenes
Lightcraft Spark is the iOS companion to Spark Story. It turns your iPhone or iPad into a virtual camera inside the same scenes you build in the browser. Both apps share the same real-time database — move an asset in the browser and the phone sees it, move it on the phone and the browser sees it.
## Prerequisites
* An iPhone or iPad with [Lightcraft Spark installed via TestFlight](https://testflight.apple.com/)
* The same account you use for Spark Story in the browser
* A 3D scene already set up in Spark Story
Lightcraft Spark is currently distributed through TestFlight. App Store availability is forthcoming.
## Sign in
1. Open Lightcraft Spark on your iPhone or iPad.
2. Sign in with the same Google or email account you use in the browser. There is no separate phone login — your organization, productions, scripts, scenes, and shot lists follow your account.
## Navigate to a scene
1. Select your organization from the home screen.
2. Open your production.
3. Navigate to the script scene, then tap into the 3D scene you want to work in.
![]()
The scene loads with all assets, locations, and shots that exist in Spark Story.
## Splat size limits
Lightcraft Spark on iPhone or iPad caps at roughly 5 million splats per scene. If your splat exceeds that count, the scene will not load on the phone. Keep a downsampled version for mobile previs and the full-resolution version for the browser and for Jetset on set.
## Next steps
* [iPhone or iPad camera controls](/spark/iphone-camera-controls) — Navigate scenes and configure camera settings on your phone.
* [Live camera sharing](/spark/live-camera-sharing) — Broadcast your camera view to the team in real time.
* [Compositing and recording](/spark/compositing-and-recording) — Record takes with live AI compositing.
# Lights and environment
Source: https://docs.lightcraft.pro/spark/lights-and-environment
Add lights to your scene and configure environment presets for realistic previews
## Adding lights
Add lights to your scene from the outliner. Spark Story supports three light types:
* **Directional** — Simulates distant light (like sunlight) that illuminates the entire scene uniformly.
* **Spot** — Casts a focused cone of light. Useful for highlighting specific areas.
* **Point** — Emits light in all directions from a single point.
## Light properties
Select a light to edit its properties in the properties panel:
* **Intensity** — Controls how bright the light is.
* **Color** — Sets the light's color.
* **Angle** — For spot lights, controls the width of the cone.
## Environment presets
Environment presets provide background lighting and reflections based on real-world settings. Choose from presets such as apartment, city, studio, and others to set the overall mood of your scene.
## Environment intensity
Adjust the environment intensity to control how strongly the preset affects scene lighting. Lower values produce subtler ambient light; higher values make the environment more prominent.
## Lighting and Gaussian splats
Lighting presets and scene lights affect USD assets only. They do not relight Gaussian splats. A splat's baked lighting carries through unchanged in both Spark Story and Lightcraft Spark on iPhone or iPad.
# Live camera sharing
Source: https://docs.lightcraft.pro/spark/live-camera-sharing
Broadcast your iPhone or iPad camera view to teammates in the browser in real time
Live camera sharing lets one teammate drive a virtual camera on their iPhone or iPad while everyone else watches the framing update in their browser — regardless of location.
## How it works
Spark transmits 3D coordinates, not video. This means a teammate in another city can drive the camera without lag, and anyone in the same scene can see where the broadcaster is positioned.
## Start broadcasting
1. In Lightcraft Spark, tap the **lightning bolt** button at the top of the shooting view.
2. The corresponding icon in the browser changes from blue to orange, signaling that the phone is broadcasting.
![]()
If the lightning bolt does not engage on the first tap, tap it again. The handshake occasionally needs a second attempt.
## Follow the broadcaster
1. In the browser, click the orange lightning bolt icon.
2. Choose **Follow \[user name]** from the dropdown.
3. Your browser viewport now mirrors the broadcaster's camera in real time.
![]()
The broadcaster also appears as a gray camera inside the 3D scene, visible to everyone else in the session.
![]()
## Limitations
* Only one teammate can broadcast a live camera at a time. Multi-user simultaneous broadcast is on the roadmap.
* The broadcaster must have Lightcraft Spark open and connected to the same scene.
# Organizations
Source: https://docs.lightcraft.pro/spark/organizations
Create and manage organizations, invite team members, and configure roles
## Creating an organization
To create a new organization, select **Create Organization** from the dashboard and enter a name. You become the admin of any organization you create.
## Inviting team members
Invite collaborators to your organization by email. Invited members receive a notification and can join immediately if they already have a Spark Story account.
## Roles and permissions
Organizations support the following roles:
* **Admin** — Full access to organization settings, member management, and all productions.
* **Member** — Access to productions within the organization. Cannot manage organization settings or members.
## Leaving an organization
You can leave an organization at any time from the organization settings. If you are the only admin, you must assign another admin before leaving.
# Productions
Source: https://docs.lightcraft.pro/spark/productions
Create productions, navigate the dashboard, and manage your projects
## Creating a new production
From your organization page, select **New Production** and enter a name. Productions act as containers for scripts, scenes, and shot lists.
## Production dashboard overview
The production dashboard displays all productions in your organization. You can switch between grid and list views and sort by name or date.
## Navigating scripts and scenes
Within a production, use the sidebar to move between:
* **Scripts** — Open the script editor to write or review screenplays.
* **Scenes** — Access the scene editor for any scene generated from script headings.
## Deleting a production
Admins can delete a production from the production settings. Deleting a production removes all associated scripts, scenes, and shot lists. This action cannot be undone.
# Quickstart
Source: https://docs.lightcraft.pro/spark/quickstart
Create an account, set up your first organization and production, and explore the basics of Spark Story
## Prerequisites
* Google Chrome on a recent Mac or PC (other browsers are not supported)
* A Google account
* Invitation or whitelist access to Spark Story
## Sign in for the first time
Navigate to the Spark Story web app and sign in with your Google account. If you have been invited to an existing organization, you will see it after signing in.
## Create your first organization
Organizations are team workspaces that contain members and productions. Create one to get started:
1. Select **Create Organization** from the dashboard.
2. Enter a name for your organization.
3. Invite team members by email (you can also do this later).
## Create a production
Productions are project containers — one per film, episode, or commercial.
1. Open your organization.
2. Select **New Production**.
3. Give the production a name and optional description.
## Quick walkthrough
Once your production is created, try this end-to-end flow:
1. **Upload a script** — Add a `.fountain` file or create a blank script in the script editor.
2. **Open a scene** — Scene headings in your script automatically generate scenes. Select one to open the scene editor.
3. **View in 3D** — Navigate the 3D viewport to see your scene. Import assets, position objects, and set up a camera.
## Getting help inside Spark Story
The **Help** button at the bottom of Spark Story opens an in-app assistant. Type a question about Spark Story, Lightcraft Spark, Jetset, or Autoshot, and it returns guidance pulled from current Lightcraft documentation.
## Next steps
* [Core concepts](/spark/core-concepts) — Learn the key terminology used throughout Spark Story.
* [Writing scripts](/spark/writing-scripts) — Dive deeper into the script editor and Fountain format.
* [Scene overview](/spark/scene-overview) — Explore the scene editor layout and controls.
# Scene overview
Source: https://docs.lightcraft.pro/spark/scene-overview
Understand the scene editor layout, viewport controls, and panel workflow
## Four-panel layout
The scene editor is organized into four main panels:
* **3D canvas** — The main viewport where you view and interact with your scene.
* **Timeline** — Controls playback, animation clips, and audio tracks.
* **Outliner** — Lists all objects in the scene (assets, cameras, lights, locators).
* **Properties** — Displays transform, material, and other settings for the selected object.
## Browser requirements
Spark Story renders Gaussian splats in the browser via WebAssembly. Google Chrome on a recent Mac or PC is required. Other browsers are not supported.
## Canvas toolbar
Four controls sit in the upper left of the 3D canvas:
* **Gear** — Opens the Camera panel for move speed, look speed, boost speed, and transform size.
* **Compass** — Toggles the scene gizmo on or off.
* **Square** — Toggles the ground plane on or off.
* **Bug** — Toggles Show Debug Info, which overlays the current frame rate in the upper left of the canvas.
![]()
## Viewport controls
Navigate the 3D canvas using fly mode. Fly mode lets you move through the scene freely, similar to a first-person camera. Use **W**, **A**, **S**, **D** to move and **Q**, **E** to lower and raise the camera.
## Selecting objects
Click an object in the 3D canvas or the outliner to select it. The selected object's properties appear in the properties panel, where you can edit transforms, visibility, and other settings.
## Panel workflow
Panels can be resized to fit your working style. A typical workflow involves selecting objects in the outliner, positioning them in the 3D canvas, and fine-tuning values in the properties panel.
# Script outline and 3D scene list
Source: https://docs.lightcraft.pro/spark/script-outline-and-3d-scene-list
How script headings automatically generate 3D scenes in Spark Story
## Auto-generated script outline
When you write a scene heading in a script (for example, `INT. WAREHOUSE - NIGHT`), Spark Story automatically creates a script outline on the left side, and a corresponding list of 3D scenes on the right.
## Navigating the script outline
Clicking on the script outline at the left automatically navigates to that section of the script, and displays the list of 3D scenes that are associated with that script scene.
## 3D Scene list sidebar
The 3D scenelist will be empty at first. Each 3D scene is a workspace where you can import assets, set up lighting, and plan camera shots.
## Navigating from script to 3D scene editor
Click the *+ Create a New 3D Scene* button in the upper right hand corner of the Scripts page to create a new blank 3D scene. Spark will suggest a name based on the script scene title, but you can change the 3D scene name to better describe it.
Select any 3D scene in the sidebar to open it in the 3d scene editor. You can switch between the script editor and 3D scene editor at any time without losing your place.
# Shot previews
Source: https://docs.lightcraft.pro/spark/shot-previews
View shot previews, add comments, and share annotated shots with your team
## Auto-captured previews
Preview images are automatically captured during scene playback. Each shot in a shot list includes a thumbnail generated from the camera's perspective at the current playhead position.
## Full-screen preview
Select a shot in a shot list to view its preview at full size. Full-screen previews give you a clearer look at framing, composition, and asset placement.
## Comments and annotations
Add comments and annotations directly to shots. Use these to leave feedback, note blocking positions, flag issues, or document creative direction for your team.
## Sharing shot lists
Share shot lists with team members within your organization. Shared lists let everyone review the planned shots and add their own comments.
# Timeline
Source: https://docs.lightcraft.pro/spark/timeline
Control playback, manage animation clips, and work with audio tracks in the scene editor
## Interface overview
The timeline sits below the 3D canvas and includes:
* **Playhead** — Shows the current time position. Drag it to scrub through the scene.
* **Ruler** — Displays time markers for reference.
* **Tracks** — Rows for animation clips, audio files, and shot ranges.
## Playback
Use the play, pause, and scrub controls to preview your scene. Playback runs all active animation clips and audio tracks simultaneously.
### Keyboard shortcuts
| Shortcut | Action |
| -------------------------------------- | ------------------------------------------------------------ |
| **Spacebar** | Play and pause. |
| **Shift+Up** | Jump the playhead to the start of the timeline. |
| **Left / Right arrows** | Step the playhead backward and forward. |
| **Ctrl+scroll** (hover over timeline) | Zoom in or out on the time span. |
| **Shift+scroll** (hover over timeline) | Pan left or right when only part of the timeline is visible. |
## Animation clips
Animation clips contain keyframed movement for objects and cameras. Each clip appears as a bar on the timeline that you can move, trim, or extend.
### Pre-animated USD assets
If you import a USD asset that already contains animation (a rigged character, for example):
1. Open the **Outliner** panel.
2. Expand the **Object** group, then expand the asset.
3. Expand its **Animation** node.
4. Click and drag the animation clip down onto the timeline, placing it at the spot where the object's row lives. The clip displays its duration (e.g., "20.8s") so you can place it precisely.
5. Toggle **Keyframes** on for that object.
![]()
Without Keyframes enabled, pre-animated USD assets may render correctly in the browser but sit frozen on playback in Lightcraft Spark on iPhone or iPad.
## Audio tracks
Add audio files to the timeline for reference during shot planning. Audio tracks display waveforms so you can visually sync camera movement or scene events to sound cues.
## Shot ranges
Shot ranges appear on the timeline as highlighted sections, each corresponding to a shot in the scene. Adjusting a shot range changes the shot's start and end time.
# Writing scripts
Source: https://docs.lightcraft.pro/spark/writing-scripts
Create, upload, and edit screenplays in Fountain format within Spark Story
## Creating a blank script
Open a production and select **New Script** to create a blank screenplay. Spark Story uses Fountain syntax for screenplay formatting — scene headings, dialogue, action lines, and transitions are all supported.
## The script editor interface
The script editor includes:
* **Toolbar** — Formatting controls for scene headings, dialogue, action, and transitions.
* **Formatting palette** — Quick-access panel for common Fountain elements.
* **Sidebar** — Lists all scenes generated from your script headings.
## Scene headings
Scene headings (also called slug lines) structure your production. Each heading that begins with `INT.`, `EXT.`, or similar prefixes automatically creates a corresponding scene in the scene editor.
Write headings in standard Fountain format:
```
INT. STUDIO - DAY
EXT. ROOFTOP - NIGHT
```
See [script outline and 3D scene list](/spark/script-outline-and-3d-scene-list) for details on how headings become editable 3D scenes.
# Autoshot Blender Round Trip
Source: https://docs.lightcraft.pro/tutorials/blender-workflows/autoshot-blender-round-trip
A complete round trip workflow from Blender to Jetset and back
## Video Tutorial
# Transcription
Today we're going to be going through a complete Blender Autoshot Jetset round trip. We're going to install a new set of tools for Autoshot and Autoshot Tools under Windows, and as well as the Blender add on, then we're going to take this really nice Polycam scan, uh, donated by Conrad Curtis and his team. And we're going to pull that into Blender, export it to Jetset, and then take a shoot, a take, and then bring it back into Blender.
And so we'll have a complete round trip of, uh, building a model, shooting with it, and then bringing the tracking data all the way back. Uh, through Autoshot and back into Blender. Along, this time along with the green screen key.
## Downloads
Alright, well let's, uh, let's get going. First we're going to do all the downloads under the lightcraft.Pro downloads page. We're going to first download Autoshot, and Autoshot is the primary post production tool that we use.
We're also going to install Autoshot Tools. Autoshot Tools is a much larger download. It has the parts of the download that don't change very much. Those are the big AI models and some of the larger DLLs that we use to accelerate the GPU.
We split Autoshot and Autoshot Tools on Windows so that way Autoshot can be updated rapidly without a big download. On a Mac, it's all part of one thing, as it turns out to be more compact on the Mac.
We're going to download the Blender add on. And that is compatible with the Blender 4 series of Blender releases. Then we'll make sure we have Blender installed.
## Installing Autoshot and Tools
Okay. So those have finished downloading. First of all, we can install Autoshot and accept the agreement. Click next. If you get a little blue warning then go ahead and accept it. Right, finish.
And we're going to install Autoshot tools. It's again, if we get a blue warning, click more info, click run anyway.
And the defaults are all fine for these.
## Installing Blender Add-On
Then finally we're going to open up Blender and I'm using Blender 4. 1, just the standard release.
And I'm just going to do the standard practice for installing add ons, going to go to edit preferences, add ons, install, going to go to my Autoshot Blender. And it's going to install, click the install the add on. Again, we leave it as a zip file. All right, and let's make sure that it has installed. There we go, like Craft Technology Autoshot. Make sure that's checked. All right, and let's exit Blender to make sure that that takes.
That's an important step. Sometimes you have to exit Blender to make sure that the add on is correctly installed.
## Setting Autoshot Project Folder
Next what we're going to do is we're going to actually start up Autoshot.
And I'll full screen this, make it a little bit more clear. And the first thing we're going to want to do is we're going to want to set the Autoshot project folder. And this is the, the folder that all of the take information is going to be downloaded into.
And it's also the folder that the sequences, when we start to generate sequence information, on processing takes, it's going to be in there. It's just going to be the standard folder under which all the information for a given project is going to be automatically put. This would be a different folder for each client or each project.
And we can set it by clicking Browse. In this case, I already made a folder in my D drive. Blender Tutorial. And I'm going to click Accept.
## Project Folder File Structure
It's important to note, as soon as you create one of these folders, you click Open.
Blender Tutorial. We can see that it has created an assets, footage and sequences folder. Footage is where the footage will be stored when we synchronize it from Jetset. Sequences are where it'll generate sequences, again, when we process them later. And assets is just a folder that we can use as a convenience folder for putting some of our USD files when we export.
So we'll come back to that in a little bit. But this gives you a starting idea of how we do our folder structures and our data storage. All right, so now that we've picked our project folder, let's go download a file from Polycam.
## Importing Polycam Scan
Okay, so here's a nice, uh, Polycam scan. This is from a project Conrad Curtis did. One of our colleagues, and it's a really nice scan of, uh, a, trench model. We want to bring this into, into Blender first, so we can add some scene locators and then we're going to export it to Jetset.
So first of all, we're going to download it from Polycam. This is using the Polycam free version. So the only option is GLTF. That'll work fine. Click export. All right. So it's exported.
Now let's go, let's open up our Blender application yet again.
And hit A to select everything, X to delete. We're going to do file import, and we're going to import a glTF. And we're going to go to our, our freshly downloaded file. And we're going to import, and load in the file. And we'll click, uh, Z, and let's take a look at our material preview.
Okay,
there is our model. And, uh, it looks like it's about the correct size, but let's just check. Let's do Shift A and add a mesh, and we're going to add a cube. That cube is, uh, 2 meters by 2 meters by 2 meters, so it looks like we have about a 3 foot high trench.
That looks correct. Alright, let's delete the cube.
## Setting Materials to Emissive
First of all, let's do some material definition fixes, uh, because the way this is imported, it won't work quite right in Jetset, so let's fix that.
So we're going to look at it, this in the shading tab. And we're going to want to actually, do some shader changes here to make it more of an emissive color there rather than a standard base material that was expect to be lit because inside the phone, we don't really have much of a lighting system.
So we're going to open up the emissive and change our color to here, change our strength to one and change our base color to zero and going to go to slot two and again, we're going to change our base color to emission and set our strength to one and our base color down to zero. And we're going to go to slot three and once again, open up our emission, change our base color, replug it to a emission, set the strength to one and our color base color down to zero.
So what we've done now is we've changed all of the materials to be a purely emissive material. And that way when we click render, it doesn't really change the appearance.
## Decimating Mesh
The second thing we're going to want to do is we're going to take a look at this mesh, and by default that is a very, that's a pretty heavy mesh.
And that's You know, for the newer phone models that can probably happen. I'm testing with an iPhone 12 pro max. That's a little bit older and that's going to be a bit too much mesh for that. If we look at our statistics and go over here and that is wait for it. Uh, 1. 4 million triangles.
Okay. So that's a little bit much. So let's go over here and click our modifier tab and get to add a modifier. And we're just going to use the decimate modifier. And we're going to do a ratio of a 0. 1 and tell it to decimate. It will take a second to do that.
Okay. And we can check to make sure my material still looks about the same. That's fine. There's going to be a few little holes in the mesh, but that's okay for our preview purposes.
## Adding Scene Locators
Once we've done that, we need to add some scene locators and scene locators are, uh, how we navigate through 3d scenes in Jetset.
And all it is, it's an empty null with a special name. An empty 3D null with a special name. So I'm going to set the 3D cursor in Blender by holding down my Shift key and right clicking here. And I'm going to hit Shift A to add. I'm going add an empty. Uh, and there's our empty. Now we're going to name it.
Let's rename it. We're going to call it scene loke and plank 01.
## sceneloc\_\* prefix
And the important part here is the scene locator name -- the S C E N E L O C prefix underscore. When we export the scene to USD and we read that into Jetset, Jetset is going to look through the scene for all the empties that are named with this sceneloc underscore, you know, whatever name, and it's going to make a list of those.
That list is going to show up in the scene locator list in Jetset. So when you click that list in Jetset, it's going to snap this location that we pick and this orientation. In the 3D scene to whatever you have picked in your live action view in Jetset.
We're gonna rotate this a little bit. Uh, let's, let's, first of all, let's check which way the axes are pointing.
So while it's highlighted, we're going to go to our object menu and we'll click viewport display and then click show our axes. And. They're a little bit buried in the 3D model, so we're just going to click in front. So we can see that our x axis is pointing to the right. That's, that's fine.
I'm going to click R and Z, and that lets me rotate this, this minutely. And I'm just going to align it up with the, up with the plank. Uh, and for, for the uses of this, that's a little bit off. Try that again. And we can look at it from a top view if we wanted to get it really, really exact. That's fine. And what I'm doing here is I'm just picking something convenient that will be a flat surface in the model that we may want to match to our live action environment.
In this case, our live action floor. So I picked a plank that we're going to lock to the live action floor. And I may want to pick another scene locator. In this case, I'll just pick, Uh, the spot in this corner. And once again, I'm going to shift, right, click and set my 3d cursor going to do shift a and add an empty.
And I'm going to make sure that we can see its axes. And this time the X axis is pointing this way. I actually want to point it a little bit, a little bit over to our right. So I'm going to look at the top view, which I did with a seven hotkey. And I'm going to type R for rotate Z around the Z axis, and then I can mouse my way around.
And align it with the edge of the trench.
And that lets me, uh, put my x axis, uh, over to the right.
## Scene Locator Orientation In Jetset
And what's going to happen, and we'll see this when we drop this into Jetset, is when you pick a scene locator, it will automatically snap the view of the scene to whatever orientation and position that the scene locator that we set inside Blender.
And so we better, better name it with our special prefix. So again, it's going to be S C E N E L O C underscore. We'll call this corner one. All right. So we've now got a plank and a corner.
## Exporting to USD
I think we're ready to export. So we're going to go to File, Export, we're going to go to Universal Scene Description, and , I'm just going to pick the same Blender tutorial that we used going to go to assets.
So I'm going to make another new folder. I'm just going to call this USD And it'll default to trencho1. usdc, that's fine, that's the compressed version of USD. And, uh, all these defaults, uh, we can leave at their default level. If we are exporting an animated scene, we'd need to check this, or if we were doing something more unusual, we might have to change a couple of these modifications.
But right now, the defaults will work fine. I'm going to click export. And we're going to see it export. Down here at the bottom is our progress bar.
## Saving Blend File
Let's also go ahead and let's save our Blender file. Go to File, Save As, and we can go to the same Blender tutorial project top level folder that we've been using.
And we'll just go to assets, and just make a make a folder here and we'll just call it blender good place to put blender files and we can say a trench 01. blend there
## Autoshot USD to USD
Now we're going to switch over to Autoshot. And we're going to go to the models tab in Autoshot. And this part of Autoshot is dedicated to processing USD files to make them into USDZ files so that they'll work with Jetset.
Jetset requires a USDZ file, which is just a set of USD files that it's wrapped up into an archive and packaged with textures. That's all it is.
By default, Blender is going to export a USD file. So we're going to set our USD model folder. I'm going to click this.
And we're going to set it into our Blender tutorial and our assets. And we'll pick the USD file. And we'll click select. There we go. And we can see our Trench01.
This is our source, and the USDZ is going to be our target. So we're going to click set and we're going to go up, up a file and we can just make another file folder.
I'm going to call this USDZ, ..
## Image Texture Compression
Here's another thing we'll have to decide. The default file, and I'm going to go and look at this in Blender so you can see how big the default file is. I'm going to click on this, and there's our image 2, the image that we're working with.
I'm going to go over here and open up image 2. And, image 2 is big. Alright, so, just to get an idea of how big it is, I'm going to right click and hold down my cursor on the resolution. So that is, uh, image 2 is an 8K file. You can see 8, 000 pixels by 8, 000 pixels.
With a newer phone, that might be okay. I'm testing with an iPhone 12 Pro Max. So we're going to have to cut down the textures quite a bit. So I'm going to set the texture size max. You could have it as original if you have a very new phone that can handle the 8K texture. For the most part, we're going to want to downsample our textures pretty heavily to get them into the phone.
And so, Autoshot has a tool that lets us do that. And I'm going to downsample it to about 512. And I'm going to click Make USDZ. And it's going to process the USD file tree as well as the textures. And this exit value zero tells you that it's finished successfully. And we can now see a trench01. usdz file in our USDZ model folder.
## Pushing Files to Jetset
The next thing we're going to want to do is actually push this to Jetset. Now, we can't do this right now because up here in our Jetset link area, we don't actually see any Jetset devices. Uh, however, if I turn on a Jetset device,
then all of a sudden we can see it show up here . The Jetset devices are automatically detected under our Jetset link folder. And here you can see the, the, uh, phone that I'm using. And this only will detect the devices when Jetset is actually running.
You can only push, uh, files to Jetset when Jetset is actually on, which makes sense. We can see that it's currently set to the project name as Blender Tutorial. I created the Blender Tutorial project on the phone before I started recording this.
And next we're going to click Push USDZ to Jetset. Over here we can see the progress. And the overall model is about 10 megabytes, so it's going to push over pretty quickly. And in general you want to keep the, the model size, you know, modest. So this iPhone 12 Pro Max, can handle models up to around 100, 150 megabytes.
Sometimes it'll, it'll start to have some frame drop issues when it gets hot and when it has a big model. So I've, uh, decimated the model and cut down the textures a bit to make sure it fits. So if you're, if you're seeing a lot of dropped frames in your models, Uh, just, you know, decimate and, and cut down the, cut down the textures and you should be fine.
## Loading Model in Jetset
Okay, so now we're gonna move over to Jetset. And, uh, as we noted before, we have our project set to Blender tutorial. And so that way when we pushed our USDZ file over to Jetset, it knew where to put it, it knew to put it in the model subdirectory of the Blender Tutorial, uh, project. So we can now click our model, uh, menu item, click open, and there's our trend and there's our trench.
So we're going to open up our trench.
## Setting Jetset Origin and Scene Locator
We're going to be shooting in a little portable green screen here, and we're going to go through the steps to set up and align things. So the first thing we do always is we select our, select our origin, click origin, and we're going to click reset. And we're going to start, uh, we'll start to start a new map.
It's going to initialize our session. It's going to detect our floor plane. You can see the natural features in the floor that it's detecting. And we're going to click on the grid for the floor plan. We're going to drag it over here.
And that's a, that's a, we'll just pick a spot to align it, to align this. Okay.
We can slide our Ghost slider up and down to see the mix between the live action and the 3D world. And, so we can look down here, and we're going to want to make sure that we pick our Plank 01. Remember that we made a scene locator in our Blender file, brought it over, and now Jetset can detect those scene locators.
So if we click corner, it would go to the corner. We're going to click plank, it's going to go to the plank, and we can see that that is now aligned correctly with the spot on the origin of the, on the floor that we defined as our origin. We can click okay.
## Scanning Location
Next we're going to want to scan the scene. We're going to go to our set tab, and scanning is available with Jetset Pro and Cine, and it's phenomenally useful if you're going to later on do subpixel tracking refinement.
If you can, you always want to do a scan of the physical area before you're recording. So we're going to click start, and you can see immediately. It's using the onboard LiDAR scanner to very, very rapidly map some geometry onto the surrounding area. There's a little chair over there, there's a little floor, back wall there.
Okay, and so it does a very quick, kind of a rough scan of the the the area and we can come up a little bit closer and it'll get a bit more fine grained.
So now we can click stop when we have enough scan and we're actually going to change it and hide it Uh, so we can click okay.
## Setting Keyer
And finally we're going to click set and i'm going to pick our green screen color with our keyer and click okay And we're going to set our matting to green There we go.
Now, right now, all of a sudden, you can see our green screen key is working. We're going to want to set our 3D garbage mats. So we're going to go back to set, and we're going to go to our infinite green screen. I'm going to click start. And what it's doing is it's detecting the vertical plane or the horizontal plane of the green screen, and it makes a perimeter of the detected edge of the green screen.
So we can actually go ahead and click add a plus sign to add a mat, come down here and add another plus sign. And come down here and detect the floor plane, add a plus sign here.
Okay, so now what we have is we have an infinite green screen mat. Uh, so that when we're in front of the green screen, Um, I'll put my hand in here, then it's going to key correctly on the green screen, and then if we go off it, right now it's going to stop at an edge there.
## Recording a Take
And so then we can see our, uh, see our, uh, full view of the, of the trench. Okay. So once we've set our origin, we've scanned, we set our, uh, set our green screen area and then let's actually go ahead and shoot. So I'm just going to click roll and then I can come in and do a little move on the green screen.
Come back around. All right, I'm going to cut. All right. We'll do one more roll, hit roll, and they're going to come down like we're running through the trench walls. There we go. Hit cut.
## Reviewing Takes for Dropped Frames
Now, what we're going to do is, uh, go to the review tab.
And with the review tab, we can see that for every shot, it's recorded both the camera original with our portable green screen, and it's recorded the real time composite. So if we can click play, we can see the actual shot that we just did and review it. Okay. See if we like it. We can full screen it if we want to click full screen.
There we go. So we can look at it full screen and see if we like the shot. All right. It looks like good and we'll go back to the normal display and there we go.
And what we want to look for in review is we want to make sure we haven't skipped a lot of frames. So over here we can see that we duplicate, duplicated zero frames. That means that the phone didn't overheat and the GPU didn't overrun. And, uh, we had zero dropped frames. So that means we'll have really good tracking.
## Dashboard Warning Lights
The things that we're watching while we're shooting is our dashboard lights that are down in the lower right hand corner. Uh, there are three dashboard lights. You can see the temperature gauge is warming up a little bit. Uh, but there is also a navigation. Which is the compass that's toggling between red, uh, between green and yellow there.
So it's tracking pretty well. And then the GPU, which is the tachometer. And right now that's also green. So it means that we're shooting pretty well. You can see the motion is pretty smooth, uh, but that the phone starting, starting to warm up a bit, uh, because it is an iPhone 12. Uh, 12 Pro with, uh, without a cooler, and it's going to get warm over time.
All right, so let's, uh, let's go pull this take back over.
## Syncing Takes in Autoshot
Here we are back in, uh, in Autoshot. And instead of the Models tab, now we're going to go back over to the Takes tab.
Here we can see we still have Jetset running, so in our In our sync, uh, area, we can see that the iPhone is, is there and is recognized. And we can see that our project name that it's currently set to is Blender Tutorial.
## Project Folder Verification
And Autoshot actually checks to see if the project name here matches the project folder name here.
If it doesn't, it'll throw up a little red warning here. You can keep going, it's just telling you to watch out because it's trying to help you to not synchronize your takes into the wrong project. You can ignore that warning if you want to.
## Sync Selection
And here we have, uh, a couple of things. Jetset stores takes, uh, on a per day basis. So for every shooting day, it's going to make a new folder, and that's going to be a complete folder that we can synchronize with all the takes of the day.
We can also just Sync All, but usually you just want to sync a one day's worth of, of takes.
When we click sync, this is going to pull the data, the takes from Jetset, and it's going to put them inside the project folder inside the takes directory.
## Sync Info
And if you want more information, you can just mouse over this and this will show you exactly what's going to be stored there.
So you can see that it's going to store the footage assets and sequences in the project folder.
We have a couple other things here that you can look up in the documentation if you need more information. Check sizes just makes sure that the file sizes are exact from the original to the source because it's doing a clone operation, and that helps make sure that your files are correct.
You'd only leave that unchecked if you're trying to synchronize to a Google Drive shared folder . Google Drive does some weird things. So for now, we're just going to sync to a normal hard drive, and we're going to leave that checked.
I'm going to click Sync.
And over here we can see it's actually pulling the files over. It's pulling them over Wi Fi, so it's going to do a reasonably quick job.
## Select Day and Take
The next thing we're going to do is, here we have our shooting days. This is take selection, and this is referencing what we have synchronized in our project folder. So we have synchronized one day of takes, so if we pull down our day, we have the only day that we've shot, which is today, on the project.
And on takes, if we pull that down, we can see a whole list of takes, because we shot several takes today, and we can pick any of them. So we can pick, for example this last take that we worked with, or the second to last take that we worked with.
## Take Information Button
And we can click over here, the I button, and this opens up a little information window, so you can see thumbnails of the take, so you have an idea of what was on the take.
And here's some information about the take, uh, some of the details of it. So, we had normal tracking, and the mapping was mapped. Thermal was fair. It's getting a little bit warm, but it wasn't causing huge problems. And in general, it looks like it's fine. Uh, we're recording at 1280 by 720. And that's, that's about it. Can exit that.
## Take Model and Scene Locator
And here we have the take USDZ. So what this shows is that when this was being recorded the USDZ file that we had loaded was called trench\_01, and this is really helpful when you're trying to go back on a shoot and figure out what happened, where, you know, you always know this is embedded in the take metadata, you know which take USDZ was used to record a given take.
And we know which takes scene locator. It was sceneloc plank one, right? We remember that was selected. And so we, in our take metadata, we know that that's automatically set.
And this was the equivalent iOS focal length. That's just a super 35 millimeter equivalent for the iOS camera. The default Jetset camera is pretty wide.
And Cine Calibration is off because this wasn't a Jetset take. Likewise, the Cine Footage Match, that's all grayed out because this was just a normal Jetset take and not a Jetset Cine take.
So there's no Cine Footage to match.
## Picking the Blender Executable
Down here the program we're using as Blender. We have a choice between Blender, Unreal, and several others. Right now we're going to pick Blender.
We're going to click pick executable and I'm going to click here and we're going to take this in this case to C:\Program Files\Blender Foundation\Blender4.1 and then double click on the Blender. exe. And what that's doing is telling Autoshot where your Blender executable is, which we're going to use that in just a moment.
## Picking Blender Scene File
The scene blend file, likewise, we're going to click set. And remember we put that into the Blender tutorial and then assets and then in our Blender file trench01.blend.
Something interesting is happening over here. If we, if we look down here, what it's doing is it's retrieving the scene locators from trench 01. blend. And that means that in the background, it's actually a running Blender very briefly and quietly in the background to open up that file and to make a list of the scene locators that are available in that scene.
## Picking Scene Locators
And here are the possible scene locators to use, sceneloc corner 01 and Plank 01. And what it did, is it actually read the original Blender scene to give us a list to work from. By default it'll match what was in the take scene locator, Plank01.
So let's do that. But we can also override it. If we wanted to pick the corner, we can override that here. So we'll click Plank01.
## Created Blend File Types
What Autoshot is going to do when we hit run is it's actually going to reference that original blend file, and it's going to make a new blend file.
It will not overwrite the, the original Blender scene file because that, that needs to stay untouched because you might be changing that all the time.
What Autoshot is going to do is it's going to make a new Blender file And that Blender file will contain the camera tracking data and the frames, and it will reference the original Blender file in one of a couple of different ways.
## Append Scene
The easiest way is just to use Append Scene. What that means is that it will append --it'll open up that blend file and it'll copy all of the information in that original Blender file in trench01. blend, into our new Blender file and then add the collection that has the camera tracking data onto it.
Now, if you have a large Blender original scene file, and they can be multiple gigabytes, and you have a lot of shots, you don't want to copy that information into each shot because you're gonna burn up all your file space.
## Link Scene
In that case, you can use the link scene. And what this will do is it will, again, reference that Blender original file, but instead of copying the data, it's going to link that Blender original scene file as a background scene in your newly generated Blender file.
That's very efficient and it's a good production methodology when you have dozens of shots. In our case, we can just append the scene.
## Empty Comp Starter
The third option is to use the empty comp starter. And that actually does not append any of the data from the original trench file.
It actually makes an empty comp starter, and that's usually used if you're going to just be doing compositing, and you don't want to be dealing with the, the large scale original file.
In our case, we're just going to pick the append scene. We want to use this pretty simply.
## Camera Media
And we can either use the original file which is the MP4 file that recorded, or we can generate a PNG file sequence, or we can pick EXRs. In this case, I'm just going to use the original file. Blender is one of the few programs that can work with an MP4 file and have everything come out right.
Most of the other post production software tools really need to be using an image sequence, but Blender actually works pretty well with the camera original.
It looks like we had 388 frames in there. Since we know it's just a Jetset take we know what the default gamma and color are, and we're not going to use an AI roto model. We would need to use an EXR or PNG to use that. But it's sometimes very useful if you want to use AI Roto, and we'll leave these at our defaults.
Then up here under options, most of these you can leave alone. There's a couple of useful things to remember.
## Import LiDAR Scan from Set
One is that here where it's going to automatically import the LIDAR scan from the set. This USDZ file is the original LIDAR scan that the phone took of the living room. And this checkbox means that it's going to go into the Blender file.
## Camera Image Plane
We're also going to tell it to add the camera image plane, and that will have the camera original imagery, you know, the video show up in our scene. So we're going to leave that. Those are all checked by default.
## Running the Sequence
We're going to leave them there and we're going to click save and run. And it's going to cook for a bit.
Okay. So here, here is our newly created blend file.
## Sequence File Location
And to see where it was created, we're going to go back to Autoshot, and next to our take dropdown, we're going to click open and what this does, this is a very important button because this gives you a shortcut to open your file explorer to the file inside our project directory.
Here's our Blender tutorial, here's sequences, and here is the date and here are all the sequences that we've processed from this date. And so very quickly you can get confused pretty fast of, Which sequence was I working on? They all have an individual hex ID, but sometimes it's easier to just go over here and click open and open up a file explorer, and it'll take you right to the top directory for this particular processed take, and we can see in here that it contains all the different pieces of the processed files that are generated by Autoshot for this take.
Source data has copied over the original shot files, the video files, as well as the tracking data from the, the original shot.
Likewise, we can look into the blend directory and that is where our newly generated blend file is. You can see that it has the project prefix, the current take and the unique take ID of the take that we just picked and is with a suffix called render, because we usually use these files for rendering,
## Blender File Contents
Back to Blender, here is our newly created file. I can go ahead and hit play and you can see that it has. The track shot in it as we're going to come in and let's, let's back up to see a little bit of what, what all is going on here. So I'm going to, I'm going to mouse click and rotate back.
## Scene Collection
I'm going to hide these one at a time so we can see it. The scene collection was appended from our original scene. So if I turn that off, we're left with. The collection that has the the camera tracking information in and the scan collection.
## Scan Collection
So here's the scan collection. This has the scan information from the original scene so you can see The shape of the, the green screen wall. I'm going to turn off the the camera tracking collection so you can see the shape of the the green screen, the opening next to it, and a little bit of the chair leg over here.
And this is all from the original scan.
## Camera Collection
And then we drop in the, the camera tracking onto it. Here's our tracked camera. So as we're moving through the shot it's moving as well. And there is a image plane that has the contents of the live action image.
## Material Keyer
You'll see that it's doing a bit of a partial key. Let's take a look over at our shader network. I'm going to shading tab and we're going to click on that plane to highlight it. And you can see something pretty cool here, which is we have by default a, a nice little material color keyer here. And it's set to a default green level that works okay. But we want to actually set it to the actual green level that we have shot.
So we're going to hold down shift, control and click onto our camera original tab. This runs a wire directly from our camera original tab over to our material output. So we can see the true color of the image. And we can see here that the color space is an sRGB image, which is correct. And we're going to click our key or color, grab our eyedropper and then click on there.
And now we're going to, again control shift, click on our principled shader and Voila! So now we have a nice clean key, and we can go back to our layout, and here we have it. Now we can see the keyer working. So if I click zero to to frame our, our, our scene then we have our, our green screen key and we can extract people from there, if we have people in the scene.
## Combining Collections
And now we can layer on another piece. So here's our camera and here's our scan and we can add in our scene.
Okay. So there's our scene. And so frequently what you'll want to do is you may want to use your scan for reference, but then we can turn it off for now.
Okay, so here we can see as we move through the shot, you can see that the live action plate is connected pretty well to the CGI plate. And the keyer is working. If we had someone in there, then they'd be keyed out.
## Adjusting the Camera Tracking Origin
And then we want to look at a couple other useful things. One is that we're going to look at how the hierarchy of the camera track is constructed.
So we have a camera with motion, but it's parented to this shim. And this shim is a 3D shim. Let's, let's look at that real quick, quick in the 3D space. Alright, so here's our, here's our, our 3D shim. This is located exactly where the scene locator is, but it's actually, it's not this, it's not the same as the scene locator.
So the interesting thing now is we can move it. If we frame up with zero and we have our, our, our shim located, if we hit GX, you know, and we can actually move our camera track, uh, back and forth in the scene.
And that we can actually, and then, then wherever we put it, it'll still track, find the scene. So we can actually adjust, uh, where our camera tracking origin is after we shoot, which is, which is quite useful.
## Adjusting Image Plane Scale/Distance
So the other useful thing here is here's our camera and we look at our image plane, going to hide the scene for a second to make, make it more clear. Here's our image plane and our image plane shows the live action plate that is part of this. So as we move along, and it's set at a distance that's automatically set from the camera by the onboard LIDAR sensor if you have an iPhone Pro or an iPad Pro, or if you have a non Pro iPhone, then it's going to do a rough optical approximation.
But either way, this plane is set at a distance that is automatically measured by the iPhone or if you go into Autoshot, you can also set the depth manually to a, a given distance.
And this case one meter. But for this take, it was set automatically. But what happens when you want to adjust that? So the neat thing is that we can actually go to our. our plane, click our object and right click on our scale and we're going to clear all of our keyframes and now the interesting thing is if we hit s for the scale you'll see that our frustrum is made so that it is parented around the same Ian Hubert style origin of the camera so we can actually adjust the image plane further and backward from the camera and then, you know, as, as you move around.
Then it'll just stay there, or you can keyframe the locations. And Hubert does that a lot in his tutorials, is he keyframes the depth of the camera, and we wanted to enable it, so, that's how it works by, by default. So we can turn back on our scene collection.
## Fast Viewport Render
Okay. Uh, and now, we got, here's our scene collection, and let's say if we wanted to, uh, render a quick take of that. We can go hit N. And we have a tool in Autoshot that lets us do a fast viewport, uh, render. And it's just going to render what you can see through the viewport. So we can click fast viewport, and it'll render it actually quite quickly, several frames per second.
## Rendered Files Location
Okay, we'll set our layout back to 3D viewport. So that generated a preview. So let's go see where the preview is. Let's go to Autoshot. Once again, we're going to open up our handy dandy Take window opener. And we're going to go to Preview. And here is our rendered preview in mp4 format and we can look at and see how it tracks through the scene.
Now this is with viewport render settings, so it's not a finish quality. So if you want a higher quality shot, then you'll go, uh, we're going to click standard render, render settings. And what this will have done is we're going to look here and we're going to look at our output and it'll have set it to, uh, the correct, uh, sub directory we want to put it to, which is cg renders.
And it's going to render it out as an open EXR multi layer file. You'll use that if you want to render high quality frames.
## Render File Summary
But the key thing here is that Autoshot has generated this, this Blender render file with all of the settings up front. So that when you, when you want to render something, it's going to the right place with the right name.
It has the correct scene referenced in, it has the mesh and the scan referenced in, if you want to use the scan and the mesh. And everything is aligned correctly to the same origin that we set with our original live action origin, and then matched to our scene locator.
So everything lines up by default, and that's the key to going fast, is to have everything show up where it's supposed to be.
# Convert Unreal 5.4+ to Blender
Source: https://docs.lightcraft.pro/tutorials/blender-workflows/convert-unreal-5-4-blender
Unreal 5.4+ has an integrated and updated USD plugin can now handle converting most scenes to USD.
## Video Tutorial
# Transcription
In this video we're going to be going through our Unreal to Blender pipeline
using the Unreal native USD importer and exporter. It's much improved after Unreal 5.4 and above.
And so now it can actually export assets that are good enough that we can actually run them through the converter, and it brings along most of the items in a given scene, even a fairly complex one like this atmospheric house. So we'll go through the process and that way we can see how to do this now that the Omniverse plugin is no longer being updated.
## USD Plugin Setup
First of all, we're going to need to make sure that our plugins type in USD make sure that our USD importer is enabled. We're going to be exporting, but it turns out that the importer and the exporter for USD are in the same plugin. So that's fine.
## USD Export Setup
And next we're just going to go file export all, and we'll just go to our USD exports.
I already made a folder and we're just going to remember to pick USD and we'll pick a name. Default name is fine. There we go. And go ahead and save. And make sure to save this in a separate folder. And we're just going to leave all these pieces in place. We're going exporting to Blenders. So the Z axis is fine for the up.
We're going to actually add a couple things to make sure that it helps out the exporter. We're going to export our actor folders which will help because that way it exports some of the different USD meshes in a separate folder for each actor. And we're also going to bake our materials and instead of 1 28 by 1 28, we're going to go 2 0 4 8 by 2 0 4 8. Okay. That should be fine.
Most of the rest of these things we can we'll leave as default and let's go ahead and export.
So Unreal will start up exporting and it may take a little while to go through it. This current way of exporting will actually take up a lot less disk space than the old one. The old one this house exported to a hundred gigabytes of USD information and the new one is just a few gigabytes of information.
We'll let this go through an export just as we've done previously. And then we're going to come back in with Blender and we're going to start importing the USD file.
Okay, we have completed exporting so let's go over to Blender and start our import process.
## Create Asset File
And we're using Blender 4.4. It should work with four three as well. All right, so we're going to start off and let's make sure we have our correct add-on.
So add-ons. We need an add-on at version 120 or above. All right, which we have. So we can hit A to select everything X to delete, just to give ourselves a clean scene. Click N to open Autoshot. And we're going to unroll our Unreal to Blender and we're going to pick our USD asset that we just generated.
## Setting up Asset Viewer
So we built that in USD exports. There we go. And we will pick our house, worn night Lumen one. All right. I'll click accept. All right, and let's move this up a little bit 'cause we're going to want to actually see the process. So we'll drag that up again and we will hover our mouse over here and hit shift F1 a couple times to toggle over to the the asset viewer.
And here we can see we actually have some assets from another Unreal to Blender conversion. So we don't need to worry about those right here. We're going to change the view to just the current file. And so that way we don't see any extra assets.
Okay. And I'm going to open up my system console window over here just so we can see what's going on. I'll move it out of the way.
## Making Assets
All right, I'm just going to click make assets and it's going to start working on making our assets in the browser.
I'll move this over here. So it's going to import all the different USD assets.
## Rendering Asset Thumbnails
Okay, and here you can see. Is going through it has generated all the assets and what it's going through now is it's going through and rendering all these different assets and rendering the previews for all of these. So you can see it's going through and individually rendering all the different pieces of it. It takes a little while to do this and you want to make sure not to save the file until these previews are done. Otherwise, it will not go through and complete the previews so you, you can go through and you can see where it's generating previews.
At any given time, there's all the books.
Quite a few assets in this particular scene.
All right. And so when we can scroll up and down, we can see that all of our assets have rendered.
## Pack Textures
We can go ahead and go to file external data pack resources. There we go. It's just to check and hit save. All right, so we have now saved our assets and we're going to hit Control N for New and once again in A X to delete everything and make for a clean scene go to Autoshot.
And Unreal to Blender.
## Making the Scene
Once again, we're going to pick our same USD file. But now we're actually going to pick our asset file, because this next step is going to reconstruct our scene from that asset file. So now we can actually go over and make scene. And so we'll bring our window over here so you can see all the different processes of it importing the different USD positions, and then it's going to repopulate the scene.
Instead of from the original USD file, it's going to repopulate it from our generated Blender assets. So then we have a correctly constructed Blender scene without any duplicated textures or objects or materials, but with all the different assets loaded in at the original places, they were in the Unreal scene.
And it'll take a second to link all the assets. Okay. So here is our atmospheric house, and we can of course click Z and go into our material preview. going to take a little while. It's a big scene.
There we go. Here is our scene, and we can go into our scene and see that the interiors have been created correctly.
All right. And we're in our material preview, so everything looks a little bit bright. So let's let's go over into our cycles engine. And we'll go to GPU and it'll hit de-noise.
There we go. So there we can see our complete scene has is working now
and is brought through the materials and textures largely intact.
All right. And the lighting level's a little bit different 'cause we translated it from a Unreal over into Blender. But as you can see, as the path tracer works through it, it'll go through and generated a more detailed rendering of it. And you can see that our materials have largely come through intact.
There's certainly some tweaks, but most of it's here. All right.
# Convert Unreal to Blender
Source: https://docs.lightcraft.pro/tutorials/blender-workflows/convert-unreal-scenes-blender
Use this workflow to convert entire Unreal scenes to Blender, using USD and Blenders integrated asset system.
## Video Tutorial
# Transcription
Alright, I am excited to show you a clean new way of importing Unreal scenes into Blender. Instead of starting from scratch in Blender, we can get something from the Unreal Marketplace or a custom created Unreal scene. People have tried to export between software packages before, and it usually doesn't work well.
The two big problems were that the material conversion didn't work well, and the asset systems from one 3D world don't usually come across to another one. And two new things make this work. One is that we're using NVIDIA's Omniverse and their MDL material definition language to do accurate materials conversion.
This simply didn't exist before, and the material conversion just didn't work. And the second thing we're doing is to approach this in two stages. We're going to export USD out of Omniverse into a standardized file format. And then our Custom Blender Asset Importer processes that USD file into Blender formatted assets, making full use of blender's new asset browser so that we don't duplicate texture and geometry memory. And then we can work in a standard blender Asset workflow.
## Exporting to USD with Omniverse
I'm using Unreal 5.2 and I've already installed an NVIDIA's omniverse launcher and the Omniverse Unreal connector for Unreal 5.2.
So we're gonna convert this free polar sci-fi facility linked to in the comments. so first of all, we're just going to do a File, Export All, pick USD, and we'll, we'll go to our USD exports directory and I created a directory called polar sci fi. I'm going to hit save and that will pop up our exporter window.
## Modular Export
And what we're going to want to do is first, we need to pick modular export. What this does is it separates each individual Unreal asset into its own separate directory in the USD file tree. And our asset importer is actually going to look at each separate directory.
So this, this helps keep things clean. It makes for a larger export, but that's only temporary since when we go back into Blender, we'll avoid redundancy.
## Combining USD Preview and MDL Material
And we're going to, pick include USD preview surface and uncheck include Unreal materials. We need the USD preview surface as a backup for MDL. But we're only going one way to Blender, so we don't need to include the Unreal materials. And that's pretty much it.
We'll also click the Use Texture Source as Export Resolution to make sure we have the original texture source. And click OK. And this is going to take a while, as it's going to write out each individual asset to disk, and then when this is done we'll come back and run our Blender Importer.
So that can take actually quite some period of time, but we only have to do that once and then we're off to the races.
Okay, so let's take a quick look at how this file is structured and what it's written out. It's written out a top level file, Polar Facility Example Map Sunset. It's created a props subdirectory. Double click that, and you'll see that inside here, for each prop or each object in here, it's created as a separate folder.
If we double click in here, we can see that there's the materials, there's the MDL files, the preview surface textures, and the actual USD file.
## Blender Import
Next we're going to open up Blender. We'll press N to open up the side panels and click on the AutoShot add on.
## Asset Browser View
It's useful here to drag up the UI and change this to a asset browser view.
And we'll change that to the current file. And that way we'll be able to see the assets as they're created by the add on.
## System Console
You can also enable the system console with window toggle system console. And that way you can see the progress of what's going on. Alright, so then we go back over to our add on.
## Top Level USD File
I'm going to pick the top level USD file. Click the folder, and we'll browse to where we wrote out our U US D export and polar sci-fi, and we'll pick out our top Polar, Example Map sunset file, click accept. We'll go to the scene, click A for all X to delete, and that way we clear out our scene.
## Make Assets
Then we can click make assets and it'll start making assets. And I'm gonna bring up our console window so we can see that. So it's bringing in the USD import, and then it's going to start creating all the individual assets. And so you can see the assets being rendered and populated in the asset list below.
It's automatically packed and saved the asset file to a new asset blend file. So we're actually going to just copy this file name because that'll be useful later.
## Make Scene
And we're going to go back over to here and hit CTRL N for new.
Once again we can hit A for all, X to delete. And open up our panel again. And we're going to click on AutoShot open this out again. And we'll just paste that file name of the asset blend file.
Once again, let's click on our top level uSD scene file.
Now we'll click make scene. And we'll bring up our console. And what that's doing is it's now parsing through that entire USD file, finding the 3D locations of each individual asset in that and then repopulating the blend file using the recently created Blender assets.
And so that way we will actually have a cleanly generated recreation of the original Unreal scene, but now in Blender, built with Blender assets. And this will take a little bit of time to run.
Alright, so there it's created our file. And we can move around in it. Let's shift to material preview.
There's our scene. Now we can go inside.
And here is our internal scene. Fantastic.
# xGrids Scans to Jetset
Source: https://docs.lightcraft.pro/tutorials/blender-workflows/export-xgrids-scans-jetset
Use this workflow to bring in xGrids scans to Blender, add scene locators, and export a self-contained USDZ file containing compressed splats and scene locators to Jetset.
## Video Tutorial
# Transcription
In this tutorial, we are going to go through exporting a scan created with the X grids K one scanner into the Gaussian splat ply format and bringing that into Jetset. The scanner that can do some really remarkable work. This is a scan done by Roberto Schaeffer, ASC with a walkthrough in this part of the city.
The results are actually really remarkable. So we're going to bring this all the way into Jetset. We're reviewing it right now in the Lixel CyberColor viewer app. But let's go through the process.
## Exporting .ply from Lixel Cyber Color Studio
The first thing we're going to need to do is we're going to need to export.
We're going to go to the. Plaza Del Duomo and I'm going to make a new export directory. Call this export ply two.
And we're going to export the official Gaussian Splat PLY format. We're going to export all the PLY formats, including the downsampled versions of the Gaussian splat that we can actually run in real time in Jetset. And the attached mesh, which is the OBJ mesh. And we're going to show you how to use that to place Scene locators. We'll click okay. And close this.
## Importing OBJ and PLY to Blender
So we're going to go to Blender. We have already installed our 3D Gaussian Splat add-on as seen in other tutorials. We're just going to clear the scene with hitting A to select all and X to delete, and we're going to go to file and import, and we're going to import our OBJ file first.
We'll go into export ply two, going to go into our mesh files, and I'm going to pick our OBJ file and import that. And as by default, it comes in rotated 90 degrees and we're actually going to select our rotation, and zero out our rotation so that it is correctly vertical and we have no transformations in the X, Y, or Z.
So we'll just lock those, make sure they don't get moved around. Next, we're going to import our Gaussian Splatting, and this time we'll go to Point Cloud iteration 100. And we're actually going to pick our point cloud two. The full quality one is point cloud do ply Point Cloud one is the first level of down sample, and we usually want the second level down sample.
So we're going to import that as a Gaussian splat. And we're going to see that Gaussian splat is also rotated by default. We're going to remove that rotation and then we can see that our OBJ geometry and our Gaussian splat are now correctly aligned and we can hit Z and go to our mirror material preview so we can see the colorized version of this.
And we can toggle on our Gaussian splatting so we can see that they are in correct alignment. Okay, we're going to save this.
## Adding Scene Locators
Okay, now we can start to add scene locators into the scene. we're clearly going to want to add a scene locator near the front. This is going to be the main view of the duomo. I'm going to hit shift and right click to set our 3D cursor. I'm going to hit shift A into an empty with arrows and let's hit period to zoom in on that.
And I'm going to hit Z to wire frame so we can see things a little bit more clearly. Okay. And so this has placed a scene locator right at our front area, and we can actually switch into workbench and see things a little bit more clearly. So that's a reasonable spot. I'm going to actually move our, going to move our scene locator back a little bit. I'm going to hit shift and right click here and while our empty is still selected, I'm going to hit shift s and I'm going to move our selection to the cursor.
There we go. So now we're on flat ground. Going to hit. Seven to go to vertical rz so we can rotate a little bit. I'm going to shift our x axis to face directly at the entrance of the Duomo. I'm going to change our transforms to local and hit G and x move it up so it's more or less centered. Okay, so now we have a reasonable spot for our first scene locator, and we're going to hit F two to rename it, or we're going to name it. Scene loc. We'll call this Duomo
Front. All right?
## Duplicating and Snapping Scene Locators
That's one locator. And we're probably going to want to have a few different locators on this. Let's set our 3D cursor again. Shift right click. We'll set this as at the entrance to the plaza and I'm going to hit shift D to duplicate our scene locator.
Doesn't matter where you place it. We're going to hit shift S and move our selection to the cursor and that way we've snapped another scene locator there. We can f two to rename this and we can call this scene LOC Plaza.
All right. And this, you can just repeat these to for however many different shooting locations you want. I'm going to put one more over here to the right of the plaza. I'm going to shift, right click, place it here, shift D to duplicate our existing scene locator, and then shift s put our selection to the cursor.
So then period to zoom in and it looks like it actually put it below, below the surface. So let's hit shift. Right click again. Shift S, selection of cursor. The Blender 3D cursor is really useful. RZ 180 8 to move the X xa. Actually that's RZ 90. There we go. And that moves the x axis over here.
Let's go. We have our scene locator selected. Go to viewport visibility and put AEs and in front, and that way we can actually see the X axis so it's not buried under there. And seven to go vertical. Okay. That's, that should be reasonable. That's a reasonable spot to look at the side of the duomo.
Okay. And F two to rename it. And we're going to go scene Lo Plaza side oh one. Okay. So that should do the trick for scene locators.
## Adding Splat Locator
Now we also need to locate the Gaussian splat. We'll switch on the display of the Gaussian splat.
And by default, we have no position, no transformations on it. And this is what we want in this case because we want our positions to reference back to our original scan geometry. But we do need to tell Jetset where to load it. And the way we do that is with a splat locator.
To make this a little bit easier to see, we're going to hit Z and we're going to go to wire frame. So then we see our point cloud overlaid on top of our scan geometry. And I'm going to hit shift s and I'm going to move our cursor to the world origin.
Here's our world origin. And the scan happened to show up over there. And what we're going to do is we're just going to put a splatloc at our world origin with no transforms. We actually don't wanna move this splat in this case. We just want to tell Jetset where to load it, which is at the origin.
But we have to specify that for the automatic loading to work, which we're going to use later. So we're just going to hit shift A and add an empty with arrows. F two to rename it, name it a splatloc and then point cloud two. Okay.
## Splatloc Naming and PLY File
Now, where did Point Cloud two come from? Now remember that is the name that LCC studio automatically generated when we exported. So when we export it to Export Ply two and we go down into our. Point cloud iteration 100, we can see that it generated the original point, cloud point, cloud one, and point cloud two. And as we recall, point cloud two was the second most downs sampled one.
And a little bit later on we're going to embed our Gaussian splat into our USDZ file for automatic loading and we just need to tell Jetset that we're going to be loading point cloud two and hooking it to this splat locator. So splatloc under score point cloud two.
We will let it do that. And once again, we're going to have no position and orientation on this. No transforms and so I'm going to lock those. Then I'm going to select the Gaussian Splat. Control. Click on the splatloc and then control p to parent. I'm going to pick object without inverse.
And again, all this is doing is maintaining consistency between in Blender where we expect things to show up with our scene locators and in Jetset.
To reiterate we are loading the the Gaussian splat at its default location, but we're just keeping Blender up to date on that and Jetset up to date on that in case we had to transform it. Okay. So once we have those in place, we have this, we have our three scene locators and the different parts of our scene.
## Moving Geometry to Collection
So before we export, we wanna make sure that our geometry is not exported along with our point cloud, because our geometry doesn't have a texture and it'll show up as a, an ugly purple color in Jetset.
So we're going to go ahead and right click and make a new collection, and let's call this geo.
I'm going to select our Plaza duomo hit M and move that to geo and then we're going to disable our geo collection. Hit save. That way we can make sure that it's not being exported.
## Exporting USDZ
Now we can go to file and export. We're going to export a universal scene description. In this case we're going to export it into a specific spot. We're going to export it A-U-S-D-Z, right into the point cloud directory, and we're going to actually name it in this case, A-U-S-D-Z because Blender can actually directly write A-U-S-D-Z file. So we don't need to bother with the intermediate step of writing out a USD file and then turning it into A-U-S-D-Z.
You only really need to use that if you have to reprocess textures or something like that from Unreal. And we don't need to do that here. We're going to export USDZ and then we're going to open up Autoshot.
## Compressing and Embedding Splats Into USDZ
We're going to go into our models directory and I'm going to set our USDZ model folder and we're going to set this to just where we located our previous model Plaza Duomo and export PY two. And our point cloud. And then I'm going to pick that folder and as you can see, it can see our Plaza del VMO one US dz, and it can also see our point cloud two.
Now the key thing we're going to do here is we're going to actually embed. We're going to hold down control and click point cloud two and we're going to click embed images. And that is actually going to do two things. One, it has converted that point Cloud 0.2 PL, LY to a point cloud two dot splat with, and you'll see that it's shrunk by a factor of two it.
And that is converting it from the floating point format into a much more efficient splat format. And it has now created a new USDZ file with the suffix m for embedded, and it has placed that splat into the USDZ file. What this does is it lets us open it all at one shot.
Earlier iterations of our point cloud process required us to separately first load, the USDZ file, then the point cloud. And now this process lets us actually just load everything at once because it's embedded, which is usually far better to do this.
And we have Jetset up and running, so now we can just click push file to Jetset and it'll go through and push the file over.
And then we can go to Jetset and open it up.
## Opening Embedded USDZ in Jetset
Okay, so here we are in Jetset. We're going to go to our settings and then model. And then open our model.
And once we've loaded our model, we can click our scene locator, go to our duomo front, so there we can see the front of the duomo and pan around.
And see our, and we can move back further to our plaza entrance so we can see our nice Duomo overhead.
And then we can move over to our side plaza so we can actually see the view of the Domo from the side. All right. So that way we can actually move around in our newly generated splat.
# Exporting Animated Characters
Source: https://docs.lightcraft.pro/tutorials/blender-workflows/exporting-animated-characters
Use this workflow to solidify skinned characters in Blender for easier export to Jetset.
## Video Tutorial
# Transcription
Animated characters from motion capture or other sources are frequently rigged to have their limbs deform according to an internal rig or skeleton. This looks really good for post production, but trying to export this to USD can cause problems for previewing the animation inside Jetset.
## Default Animated USD Export Problems
This is because by default, Blender will export an animated skinned mesh as a vertex animation cache. And that means it will write out the position of every single vertex in every animated mesh for every single frame. This can result in huge files, even for very short animations. They're hard to load and render, it slows things down, and it's also overkill for the kinds of lightweight preview animation that we usually want in a Jetset scene.
## Skin 2 Solid Blender Add-on
So we built a Blender add on to assist in this called Skin to Solid. It's on the Lightcraft. pro downloads page. So go ahead and download it and install with the usual add on process.
We'll go to Edit, Preferences, Add ons, and click Install. We'll do Skin to Solid, and install the most recent one. And then let's go check, type in skin, and just make sure that it's enabled. And then we can come over here and we can see that there's now a skin2solid tab in the sidebar.
## Importing Move.One FBX
Now we'll clear the scene. We'll import an example FBX file written by the move. one app. By going to file, import FBX.
And picking the FBX file.
Alright, now hover the mouse over the timeline and hit home to scale the timeline out to the extents of the frames of the recorded take. And this has 1800 plus frames, so we'll set the end of the timeline to 1860 to get all the frames.
If we open the armature, we find that the figure is composed of several pieces of geometry. But, for example, if we open up the fingers. We'll zoom in. The fingers are a single piece of geometry, but they're moving independently. And that's because the fingers are defined by vertex groups of that piece of geometry, and that's all adjusted to individual bones.
## Skin to Solid Conversion
Now this is the standard method for animating figures inside Blender, but it causes the problems we mentioned above when exporting to Jetset. So we'll use the Skin to Solid app to make a rigidbody copy of the original. This will have the geometry separated into individual rigid pieces of geometry, with the animation baked into the motion of the individual pieces.
In the Skin to Solid panel, we'll pick the collection that the object is in, and we'll pick the rig, the armature that the object uses, and then we can click Prepare Objects. Okay, and so then we can see that it's created a whole second collection.
And then we'll set our start frame and end frame to match the animation. And then we can click Bake Animation.
Okay, so then when we scroll back and forth, we can see that our two collections have matched animation. And we can actually now disable the original collection. So now this collection is all rigid body objects, which we'll find if we click on an individual finger. We can see over here that now we've actually split every piece of geometry up into a separate individual piece of geometry, with the animation baked into that geometry.
And we can now export our usual animated USDZ for Jetset.
## Converting Mixamo Animation
.The same technique will work with animation sourced from Mixamo. We'll start with a new scene, clear it out. And we'll do file import FBX. And this time we'll pick our source object. We'll switch and turn on our materials, and we can see that we have an animated character, in this case we're running in a circle.
And we can see that, as before, this is a skinned character with smooth deformations on the joints. And we can see that for example the body, has separate vertex groups that cover all the different components. And so we're going to need to split all that up. We'll then open up our Skin to Solid panel, and pick our objects collection, and we'll pick our rig, and we'll prepare our objects.
And then we'll pick our frame beginning and frame end, in this case 179, and we'll bake our animation. And we can see the animation now matches. So we can disable the original collection and we can see our new rigid body animated character. Now we'll come in and you can see some of the issues with this approach.
Since the minimum weight removed any pieces of geometry that are weighted more than a certain level. That means that the geometry will get clipped at the edges of knees and elbows and shoulders and other areas that have a lot of deformation.
And this is okay for its purpose as an exported character preview in Jetset. When shooting an actual shot the recorded camera tracking data, would actually come back into Blender via Autoshot, then it would be rendered with the original skin character visible, instead of the rigid body character. That's just for a preview in Jetset.
Alright, so we can now export an animated USDZ with our usual process, and load the animated scene into Jetset.
# Gaussian Splats from Blender
Source: https://docs.lightcraft.pro/tutorials/blender-workflows/gaussian-splat-from-blender
Creating Gaussian splats from your Blender models lets you transfer complex lighting and scene geometry to Jetset.
## Video Tutorial
# Transcription
Okay, in this tutorial, we are going to go through our Blender Gaussian Splat creation process. And this will let us take a fully rendered Blender scene, and actually crunch it down into a Gaussian Splat that can then be rendered in real time on Jetset, retaining all the original lighting and reflections and information of the Blender scene, which is quite a remarkable process.
And in Blender, the Gaussian Splat doesn't actually display quite as well, but in Jetset it will display quite well.
Let's get started.
## Installing Autoshot Blender Add-on
First of all, in our Blender scene in this case it's a photogrammetry scene so it can render pretty quickly. But the same use pattern will work for pretty much any Blender scene. And we're going to first download and install our normal Autoshot Blender add on.
So we will go to our downloads file, and we'll go down to our Blender add on, and we're going to click download, and it'll download that to our downloads file, okay, there we go, and then we're going to go to Blender, and we're going to go to our edit, preferences, and then add ons, and go to our little install from disk, and go pick out our, type in Autoshot, There you go, there's Autoshot Blender, and there's the most recent 1. 16 version we want. Okay, so that says it's installed, we can check for it. There we go. And we can see that we have the current version installed. All right, great. That's what we need.
## Creating Capture Geometry
And, in order to generate a Gaussian splat, we actually have to first generate a series of images of your existing 3D scene.
So we can see here in our Autoshot tab here, you can hit N to make this appear. There's the tab. And under our Splat Training, you may have to open up this, we can see that there's a series of steps that we can go through to create our camera paths. And we actually first create geometry to represent where all the cameras are going to be.
In this case, I'm going to click on make a an icosphere pattern mesh. I'll pick the default radius and the subdivisions. And it'll make an icosphere. And we can see that it's just dropped it into the scene depending on where I had the last object selected.
We actually want to un parent it. So I'm going to hit Alt P. And I'm going to clear the parent and keep the transformation. And then we can just hit G and move it around the scene to the locations where we want. And what we're looking for is some centralized locations so that we can avoid intersecting geometry while still getting pretty complete coverage.
And, in fact, what we're also going to want to do is, you'll see this scene has Multiple areas in it. We're going to be generating the cameras based on all the faces of the mesh. In this case if we generated all the cameras, There's no mesh faces anywhere over here, and we're actually going to want some of those so let's go ahead and we're going to hit tab for edit mode, hit a to copy all the faces, and then Shift D to duplicate, and then we're going to move our duplicated mesh over to here, and we'll just move it around with the usual grab, grab commands usual Blender grab commands. Then if I hit tab again we go back out of edit mode.
And so you can see that I've turned our single camera icosphere into two camera icospheres that can be distributed around the range that you want to cover. And we can do three, but you get the idea. And this way we can actually have complete camera coverage in all these different areas.
## Creating Camera Path
Okay, so now we've got our icospheres, our kind of our geometry reference to position where our cameras are going to be. Note that we have unparented this, so we're not, parenting underneath something else in this scene. We're going to go ahead and click the camera track from the face normals.
We can pick our default focal length, and we'll use both directions, so it'll have a camera facing inward and outwards from each of these surface normals. And it's going to generate a splat camera. And as we can see down here, there's the animation of the splat camera. So it's, as we run through the animated sequence we can see where that camera is going to show up anywhere, here, we can turn off the icosphere, so we can see that a little bit better.
But the camera is now taking a, an image of the scene from every one of those icosphere polygons. Great, so that's gonna give us 300 and some images, which will give us a very good reference for this scene. So once we've generated that, that camera we then need to tell it where we're going to be rendering the images.
## Setting Output Directory
And this is an important step. So we're going to go down to our output panel. And I'm going to open up our folder and we have our inside our model file. I just created a separate splat folder to hold some of the splat information. We're going to right click, make a new folder and call it Images. There we go.
And we can actually name this output folder something besides images we'll just use that name for now. Just remember that as we're going to be dragging and dropping that folder into Postshot later.
And then we're going to call the file We can just call it chamber and then an underscore. By putting chamber underscore, it will number the files after chamber with the frame number, and that's going to be important later on. And then we're putting it into the images subfolder. So I'm going to click accept. Okay.
## Rendering Frames
And then the rendering frame rate doesn't matter. We're just going to render 320 frames of this. So we can just go over and click our render and render animation.
Oops. I have it set to cycles. It's going to take a long time. Go ahead back and set that to EEVEE. We're running Blender 4. 2. So this is Eevee Next. So it will render very quick. All right, so there we go So it's going to go through and render 320 of these animation images of the scene. So we'll let it go ahead and do that.
, so we've rendered all 300 and some images.
## Generating Camera Path
Okay, now once we have rendered our images, we're going to go back up to our splat training, and now we need to export our cameras. txt and images for COLMAP. We're going to be using the built in COLMAP solver inside Autoshot to actually generate our sparse point cloud.
So in order to do that we just click on the splat camera to select the camera path. And then we're going to click export cameras dot text for COLMAP.
Now, if you look at our renders directory we can see there are two new files, a cameras. txt and an images. txt file down here. And those are going to be the input files that we're going to use to load into COLMAP. So in order to load into COLMAP, we're going to switch back over to Autoshot.
## Creating Sparse Point Cloud in Autoshot
And we're just going to go under Splat Training, and we're going to click Generate Points 3D text from images. txt So we'll click on this, and we're going to paste our path to our Saved Images folder. And there is our images. txt . Going to highlight images. txt.
COLMAP is going to go through,, crunch down the data for all of the images.
So once it has completed that , we can look in our directory again, we can refresh and we'll see that in addition to our previous images and cameras.txt file, we now have a points 3D text and that is the sparse point cloud solve. So now we have all the pieces of information that we're going to need to bring this all into Postshot.
## Generating Gaussian Splat in Postshot
So let's go ahead and go open up Postshot and we're going to open up our file, explore, and we're just going to drag in our entire images folder right into Postshot and we'll uncheck start training. We're going to click import and there is our model. So we'll zoom back out a little bit, and we can see that our camera spheres are correctly aligned with our sparse point solve.
So we now have a have an initial solve and we can tell Postshot to just start training. And we're gonna let it crunch for a little while.
Okay, and we've trained it for a while and we can train it for quite a bit longer, but this will be enough to show that we basically have the correct splat building. So we're just gonna click pause training.
And we're just going to go over here and we're going to export our radiance fields.
And we're going to enter in our our path once again. And we can just call this ChamberSplat. ply. I'm going to save that.
## Importing Gaussian Splat into Blender
Now we can go back into our Blender scene. Let's go ahead and hide our camera iconosphere so we have a clear view of what's going on. We're going to go up to our 3D Gaussian splatting add on that we've covered in an earlier tutorial.
And we're just going to click Import Gaussian Splatting. And we're going to go to our our usual directory in Images. I'm going to open up the Chamber Splat that we just saved. Click Import Gaussian Splat. Okay. And you're going to see, first of all, that it looks like it's misaligned. And as we showed in our previous Gaussian Splat tutorial, by default the 3D Gaussian Splatting importer imports this with a negative 90 degree offset in the X. So we're just going to click 0 to that. Okay. And that's looking much more correct.
## Creating Splat Locator
As before, we're going to actually make a splat locator. Let's go ahead and set our cursor with Shift S to cursor to world origin. Gonna hit Shift A, and we're going to create an empty with arrows.
And we're going to rename that to splat loc, and we'll call that chamber chamber. Splat. And for the splat locators, the suffix doesn't matter so much. That's just a reference to tell us which splat we're going to load. I'm going to click on the Gaussian Splatting, I'm gonna hit M and move that into splat exports we have in the same collection.
And then I'm going to control, hold down control. Click that, and now hit Ctrl P to parent, and I'm going to parent it to the object without an inverse. So it has no additional transforms on it. And so now there is our splat locator, and it is locating our Gaussian splat. In this case, the Gaussian splat is already at the correct scale, because we preserved the scale with the images and cameras. txt exports through the COLMAP and Postshot process.
## Verifying Splat Alignment
So now when we zoom into the scene, and we can actually see that our Gaussian splat is cleanly overlaying our original image. Again, we only trained the splat for a couple of minutes, so it's still fuzzy, but we can turn off and on the Gaussian splat and see that in general things are lining up.
## Exporting USD
And now we can actually move to exporting this scene. And in this case, we're going to turn off our original geometry collection, and we only want to export our scene locators and our splat export. We can go ahead and go to file, and then export.
And we're going to export a universal scene description. And we're going to go into a a USD folder to store our USD file. We're going to call it Great Chamber Splat, in this case. 03, export USD. And we're going to go to Autoshot and refresh.
There is our newly created Great Chamber Splat 03.
## Compressing and Embedding Splats in USDZ
We're just going to click our normal and make USDZ. Now we're going to actually embed the splat file in there. And to do that, let's put the splat file in the same USD model folder. So we're going to go back to here, and we're going to go into our images folder, and we're going to just
we're going to copy our chamber splat, and bring it over to our USDZ folder, so we have everything in one place. Come back over here, and we're going to refresh. And now Autoshot has been updated so that we can now read the PLY files. So to embed that, we can actually click the USDZ file that we just created, and hold down control and click the chamber.
splat, and click embed images. We've also checked convert dot PLY to splat. This converts the PLY image to a splat format, which is a highly compressed version of the Gaussian splats that take up much less space.
You can see this in the chamber splat dot PLY was originally 110 megabytes and then when compressed into a splat file that is only 14 megabytes, which is a nearly 10x reduction. Then it created a new great chamber splat 03m USDZ with those two files combined. And you can see that it's much smaller than the original splat.
## Transfer Combined USDZ to Jetset
Now what we want to do is push this to Jetset. And we're going to be pushing the renamed underscore M file. The M is the one with the embedded image in it.
## Opening Combined USDZ in Jetset
Now we can move over to Jetset. In Jetset we can go to our gear icon. And now we can click on model. And we'll clear our model. We can go to model, open, and we'll scroll down to our great chamber splat 03, which we just pushed over. There we go, and we're going to load it up. And as you can see it now loads up our great chamber splat and our USDZ file combined.
So we have all of our scene locators from our USDZ file, while retaining all the imagery from the Gaussian splat. And this Gaussian splat we only cooked for about five minutes, so it's still a little fuzzy.
But you can see that it's retaining the color and information from the overall room very well.
## Summary
Okay, so that shows how we can actually compress a full 3D scene in Blender into a Gaussian splat and bring that in along with scene locators in a integrated USDZ file and preview it in Jetset.
# Gaussian Splat setup
Source: https://docs.lightcraft.pro/tutorials/blender-workflows/gaussian-splat-setup
Importing and scaling Gaussian splats into Blender. Adding splat locators and scene locators, then exporting to Jetset.
## Video Tutorial
# Transcription
We are going to work through our new Gaussian splat authoring workflow to construct composite models that can include both USDZ 3D information and Gaussian splat information.
And this will let us do a lot of neat things. Gaussian splats by themselves have some fantastic properties, but they don't inherently contain scale or orientation information. So if you want to use them in the real world, you need to provide them with scale and reference, and that's what we're going to do in this.
## Installing Gaussian Splat Add-on
First thing we're going to do is install a 3D Gaussian Splat Blender add on. This is forked from the original Reshot AI Gaussian Splatter Blending add on with a release package to make it easier to install. I'll put the link below.
So you just go to releases, click latest, and click on the Blender add on. And just go ahead and download that. And once it's downloaded, then we're going to go over to Blender, and we're going to go to edit, and preferences. And we'll go to add ons, click on this little arrow here and go install from disk.
And then under our downloads we can just type in star zip, star period zip. And we can just get our blender add on. Double click that and then we can see we have 3D Gaussian splatting installed. Once we have done that, then we get a new panel. We hit N. We have a new panel over here called 3D Gaussian splatting.
## Importing Gaussian Splat
And what we're going to do is we're going to import a Gaussian splat. And for our test today, we're going to use one from our colleague Josh Che over in Beijing who did a great job of putting together these really nice looking Gaussian splats as a test case. So we're going to go to our splats, and we're going to load in our Coffee02. ply. And we're going to actually make sure we remember that name, Coffee02, and double click on that.
And it's gonna take a second, and then the Gaussian splat's gonna load in, and it's gonna look really weird. And that is because to display it in Blender the plug in basically displays an approximation of the splat. So viewed in solid form, it's going to be really confusing.
## Viewing Splat
So we can click Z and go to our material preview and then it'll look a little bit better. Again, this is not the final version of the splat, but this is a reasonably convenient way to view it. So we're going to go in here, and we're going to realize a bunch of things. One, here's our default cube, hanging out in the middle of the the coffee shop.
And we're going to realize that cube is two meters tall, so our scale is going to be a little bit off. But we can roughly see where things are. Now, trying to work with this in this mode is pretty tricky, so we're just going to hit Z, go to wireframe mode. And now we're in a point cloud, and that's going to be a lot easier to work with.
We've got a few things here. We're going to go to our view. We can see, first of all, that things are kind of funny. They've got a funny scale and a slightly funny rotation. We typically are going to want to set the Gaussian splat to match a given reference in 3D.
## Adding Splat Locator
So in order to do that, I'm just going to turn off our Gaussian splat display for a second. And what we're going to do is we're going to hit Shift A and going to add an empty. We're going to go to its object and set its viewport display and show its name and axes and hit period to zoom in on that.
And we are going to rename this, and we're going to use what's called a splat locator. We already have scene locators, now we have splat locators. So we're going to type in splatloc, S P L A T L O C, and then capital C, offee. O2. And this is important because this Coffee02 name is how Jetset is going to know which file to load.
So before we loaded in the Gaussian splat named Coffee02, and so we're going to name our splatloc splatloc underscore Coffee02. And Jetset will see this when we import the USD file into Jetset. Jetset will recognize the SPLATLOC underscore prefix, and then when we load in the Coffee02 file, it knows where to go.
## Parenting Gaussian Splat to splatloc
Right now, our SPLATLOC is at the origin. And we're going to re enable our Gaussian SPLAT. And again, right now it's randomly located. We're going to click on our Gaussian splat, and we're going to control click on our splatloc, and move our cursor over here and hit control P and parent it to the object.
So if you look in our hierarchy, what we've done is we've just parented the Gaussian splat under the splatloc. So that way, we're not changing the scale of the Gaussian splat, but we can change the scale and position of the splatloc, and that's how we're going to control the position, orientation, and scale of the Gaussian splat within our Blender scene.
And we're going to show you how that works next.
## Transferring Rotation from Splat to Splatloc
We need to do one more thing to set our orientation correctly. The Gaussian splats, by default, came into Blender with a minus 90 degree rotation on the x axis, and that's so that they show upright in Blender by default. But in order for our splat locators to work correctly, we need to remove that minus 90 from the Gaussian splat, and then put it into the splat loc.
And that way Jetset will understand what to expect. So I'm going to switch that to zero, and I'm going to go up to our splat look, and I'm going to highlight that, and hit minus 90. There we go. Now things are correctly oriented, so that all the rotations are now out of our Gaussian splat, and are all contained inside our our splat locator where we want them.
## Creating 3D Reference Geo
Okay, so say we've got our Gaussian splat parented to our splatloc. Now we need to know what we're going to move it to.
Let's hit Z and then go to a wire, a material preview again. And we're going to zoom in here and let's take a look at our scene. So what we have here is a nice coffee shop. And so we want to scale reference. So one of the good scale references is a counter because those are a fairly standard height.
And at least in the U S that is a 42 inch tall counter for a normal restaurant bar and that works out to 1. 07 meters. So what we're going to do is we're going to change our cube to be 1.07 meters tall and use that as a reference. So let's go ahead and turn off our Gaussian splat. Then we can see our reference cube.
And let's hit one. So we can look at this from a front orthographic view. And we can see by default each one of these cross segments in the grid is one meter. So our cube is actually two meters wide and two meters tall and two meters deep.
That's not what we want yet. So let's scale it. We're going to highlight our cube and hit S 0. 5 and what we have just done is we've made our two meter Cube into a one meter cube. S is the keyboard shortcut for scale and then we just typed in 0. 5 and hit return and now we have a one meter cube.
We actually want a 1. 07 meter cube. So now we can, with a cube still highlighted, we can actually hit S again and then type 1. 07. Now we have a 1. 07 meter cube. And finally, we're going to hit G and move it up along the axis. To isolate it to vertical motion, I held down my middle mouse button and slid up and that way that isolates.
And then we can align the bottom of the cube to our actual ground origin. All right, that's fine. But now say we want to align our splat to, something far off the origin. So we're just going to grab and middle mouse button, click and drag, and put it over here somewhere out here on the x axis.
## Rough Splat Location
That's fine. This is for just a reference. Okay now we, let's turn on our splat again. And our splat is nowhere near there. And we're going to hit Z and go to wireframe so we can actually see what we're doing. And in this case we're going to select our splatloc. We're going to hit G for the grab shortcut.
Middle bounce button click and Drag in the, in that direction, that isolates into one direction, and we can see our cube a little bit. It's gonna be a little bit noisy in there and we've dragged it roughly over to our cube. So let's let's click our cube hit period to highlight it.
So we're, we're kind of in the right, in the right place, but but our splat, we're gonna scroll back here. We can clearly see that our splat is the wrong scale the wrong position, wrong height, wrong everything.
Alright, what are we gonna do about that? We know our cube is actually in the right position.
Our cube is, it's on Z equals zero, so it's on the floor. We want to move the splat upwards. Let's go ahead and grab our splatloc. As soon as we grab that, of course, we can't see our cube anymore. Let's go ahead and change our viewport display.
## Setting Wireframe Display
We're gonna go up to here, and for the viewport shading, for the wirecolor, we're gonna pick object. And that way, when we pick our splat locator, we can still see our cube wireframe. And we're going to hit GZ, that's going to isolate the motion of our splat locator to the vertical axis.
And let's see when we've aligned to the floor. I'm going to hit hit one on the numeric keypad and let's scroll back. And actually we see we've got a couple problems here. So before we even do that, we can see we've got the splat is a little bit misaligned rotary wise.
## Fixing Splat Orientation
It's not really flat. So let's do a couple of things. Our splat locator is still highlighted. We can hit R and since we're in an orthographic view, it's going to restrict it to just the the head on view. So now we can get that fairly straight in that direction and shift over to the side number pad one.
And once again, let's hit R and rotate that roughly. So it's at least flat. There we go, and go back to our three. Okay. So now at least it's fairly flat, and let's zoom in so we can see our floor. And with splatloc still highlighted, we're going to hit GZ. And that way we can bring up the floor of our splat to align with our cube.
## Fixing Splat Scale
Alright, so now we if we go, if we pull out here, we can see that our cube is fairly decently aligned with our floor. But we've got a problem, which is that our cube is the correct height, but our splat counter frame is not. Now we could go back and forth and adjust a bunch of times and fuss our way into position, but there's a much better way to adjust this with Blender's 3D cursor.
And it's a really cool trick, so I want to show this to you. We're going to turn off our Gaussian splatting for a second. And I'm going to zoom in on our cube, which is on the floor. I'm gonna hold down shift and right click to set our 3D cursor right on the corner of that.
You can make it snap to the vertex, but we don't need to be quite that precise for this. And what that has done is set our 3D cursor in Blender to be here. And then we can re enable our Gaussian splatting. And we can, to make this work, we're going to change our action center, our transform pivot point from the median point, which is a default to our 3D cursor.
And now if we highlight the splat locator and we hit S for scale, we see something pretty cool, we're scaling just the splatloc with respect to the cube ground, but it's taking the splat with it.
## Avoiding Splat File Changes
And the reason we do this is that we don't actually want to scale the splat.
Because we're going to have to sideload the the Gaussian splat later in Jetset. So we don't want to scale the actual splat file. We don't wanna modify it.
We just want to modify the splat locator so that it causes the splat to load into Jetset correctly scaled. So let's hit three as a side view or one as a orthographic view. Three is gonna work better and we're gonna zoom in a little bit, and with splatloc still highlighted, we're gonna hit s.
And we're going to shrink our shrink our splat until we just barely aligned. And I accidentally selected the splat. Go ahead and select the splatloc. And now you can see we're starting, it's starting to work. All right.
## Verifying Splat Scale
So we have, we can come down here and we can see that our floor the wireframe points that correspond to the Gaussian splat center points are aligned with the floor.
And we come up here to our countertop, And the Gaussian splat centers are roughly aligned with the top of our cube. So that's in the correct size. And we can go ahead and switch to material preview. And we can see that, yep, our cube is intersecting the top of the Gaussian splats.
And it's coming in at the floor level where it should be. Now our Gaussian splat is scaled correctly. Okay. Okay that's good. Hit save, and we'll just name that Coffee Shop Interior 02. That's fine.
## Adding Scene Locators
We've got that, but now we need to add our scene locators, because we need to be able to move around our scene in the way we're used to moving around in Jetset.
So we're going to want to have a scene locator say by this front door. Now if we just hit shift and right click, And try to put it there, it's going to get hit on the first Gaussian Splat cloud that's between it and the door.
So that's not going to work. So what we actually want to do is we're going to hide our Gaussian Splat, once again shift right click on our cube. And that sets our 3D cursor. So then what we do is we can re enable our splats and hold down, go to our wireframe, hit shift a, and we're going to add in another empty, we're going to rename it.
And this time we're going to name it as a scene loc, S C E N E, scene loc, and we're going to do say front door inner. All right. And so now we're going to show the name and show our, and our axes. And now we can actually switch to top view, hit G, and move that over to line up with our front door. We can hit R.
Oh, and if we hit R, look, we're going to try to rotate it, but it's rotating around the wrong axis. It's rotating around back around the, where the 3D cursor is. We're going to shift our view back to individual origins, our transform pivot point to individual origins. Now when we hit R then we can rotate correctly.
Now we can align our scene locator closely to the floor. Again, this is a little bit of an approximation because the Gaussian splats are a bit more statistical. And while we're looking at this, we can get it to line up closer. Let's go ahead and adjust along the local transform.
And that way, when I hit G and I can move along these axes. So I can come up to here, hit R Z, and align this with the edges of the door precisely. And then we can see that we are pretty well aligned in 3D space. There we go. And it is the scene loc front door inner. Okay, that's good. Let's add another scene locator on the top of the bar.
Again, gonna hide our Gaussian splat. Shift Right Click to set our 3D cursor, Shift A to add. Rename that to SceneLoc, and that is BarTop, and change our, show our name and our axis so we can see what we're looking at, and let's show our Gaussian splats, and now we can actually say if we wanted to move that to a corner of the bar, see that inside corner of the bar, hit G, middle mouse button over to, to isolate the axis, G, middle mouse button, isolate that axis, and period to zoom in, and center.
And we can see that we are, zoom out a little bit, sometimes it's harder to see it. All right, we're decently, it's a little on that corner of the bar. And then of course we can hit R Z and align it so that the X axis aligns up with the bar. Again, this is, it'll all depend on how you want to set it up.
But this is just an example of how we're going to align things. All right. And let's also align things so that we have say a view of this room coming in. So once again, hide the splat. Shift Right Click on, on the on the cube, Shift A, Empty, Add a Plane Axis, Scene, Loc, and we'll do Sliding Door, okay?
And going to turn on our Splats, and turn on our Name and our Axes, and then we can hit G, and Middle Mouse Click and Drag to isolate the Axes, and pretty quickly isolate it to the Door. Period to get close. Alright, and then we're going to move our x axis around. We're going to rotate our x axis with R Z 180. There we go, hit return.
And then we'll fine tune it a little bit more. We'll do G and isolate the axes. In this case, it looks like the floor is a little bit lower here, so we're going to do G Z. Bring it down just a little bit. Because of course the floor is going to vary from point, from different points in your scan.
Even if we try to level it out a little bit. But now we're actually reasonably well aligned with the edge of the door. All right, and then we can hit RZ and rotate that in. Okay that's a good start.
## Hiding Reference Geometry
Now it's time to export this. And we're going to want to hide the cube, so that doesn't show in our final result. So we'll go hide it in both the viewport and the render view. But it turns out we actually do need to have a piece of geometry somewhere in the scene, so that the Jetset loader knows what to do here.
I'm actually going to come over here and put a tiny little Shift A and a little cube here. Scale it down. And, there we go. We just need to have a little bit of geometry in the scene, but we don't want it to interfere with the visuals on the rest of it.
We'll just rename that Cube; we'll just call this Geo. And we'll call this cube the Counter. Height. There we go.
## Exporting to USD
Okay we're just going to go File, and we're going to go Export. And we're going to go Universal Scene Description, as before. We'll just name this Coffee Shop Interior 03, in this case. And we're exporting only the visible pieces.
And we'll just export our USD. And we're going to go to our Autoshot, and refresh. There's our new Coffee Shop Interior. I'm going to click make USDZ and it'll make a USDZ.
## Pushing Multiple File Formats
We can see down here something important, which is we see our USDZ files as we've always seen them down here. Now we can also see our PLY files.
And that is because in this updated version of Autoshot we can actually push both US DZ files as well as uh, .ply and other splat file formats, as well as image files, and it'll tell it to go to the right place. So let's push our coffee shop interior, and that'll go very quickly.
And we're also going to push the coffee shop O2 dot ply. We'll push that to Jetset and it'll take a few seconds as it's a larger file.
Okay so now we've pushed that over to Jetset, so let's move over to Jetset.
## Loading USDZ File
All right, and here we are in the Jetset scene, and we can just go File, and we'll do Open, and we're going to pick our Coffee Shop 03, so all the way loaded. All right, and so right now we don't see anything. We have the scene locators, there's the board top, there's the front door inner, there's the sliding door but we don't see anything.
And that's because we haven't yet loaded the PLY file, and we have to sideload that.
## Loading .PLY File
So we're just going to open our models file again. Click open. And over here we're going to scroll to the Coffee02. ply. And so remember that our USD file has a splatloc underscore Coffee02.
And so when we load Coffee02, it recognizes that and it knows where to put it. So it's going to take a second to load, and now it's displaying our Gaussian splat file.
## Switching Scene Locators
So if we set it out at the origin we don't see anything. If we set it to our scene locator of the front door inner then we can actually see our front door.
That was from that we set our scene locators earlier. And if we move it to our sliding door, Then we can see our sliding door, and it's the same way the scene locator is set correctly to locate that. Now, this is really cool, because now we have a full 3D scene that we can handle transparency.
## Reflection Handling
We're looking through pieces, panes of glass, and we can handle, and it handles reflections. So look at that. The reflections are shifting as we change our angle of view, and as we move forward to the floor, and it captures the material, the feel of that is shiny leather, and these are things that photogrammetry just does not capture.
The reflections from the light, and all these reflections are accurate. These are from the original photography. We can capture the look of plants in the scene, and this is a full 3D scene, so we can get up and move around, and this is all being rendered inside the phone in real time.
And we'll do another video shortly that has the rest of the workflow where we re render this inside the phone at a higher quality and then we composite it inside the editorial timeline with camera original footage matched by timecode.
So that's going to be a really exciting tutorial because it's a whole new way of working in virtual production. We don't need to have all the large scale equipment to get a good looking render out of this. So this is really exciting.
All right. Happy rendering.
# Import Megascans to Blender
Source: https://docs.lightcraft.pro/tutorials/blender-workflows/import-megascans-blender
Megascans are a set of high quality free and photorealistic scanned assets available on Fab.com. Use this free Blender add-on to batch import them and create clean Blender assets for use in your projects.
## Video Tutorial
## Download the importer
[Megascans Importer Releases](https://github.com/LightcraftTechnology/Batch-Import-BlenderAssets/releases/tag/v0.0.4)
# Transcription
We are going to go through a quick script today that does a neat job of doing batch import of Megascans FAB assets into Blender. And normally those are a little bit tricky. You can download in FBX, but it's hard to get it in. So we made a script that automates it. So this way you can go into your Blender Asset Manager and drag and, just drag and drop things in.
So if we have make a plane, we can actually drag and drop a material onto the plane and have it turn into our material preview, have it either turn into an overlay or an underlying material. This is from the Unfinished Building Megascans and it looks great. \\
So this is just the mid level. There's two higher levels of resolution you can go to but let's let's go through it.
## Installing Batch Asset Import Add-On
We're going to need to download a Blender add on from the Lightcraft Git repository. I'll put the link down below.
Okay, so here we are at the batch import Blender assets repository and we just want one of the releases. This is the most recent. Just click latest and we're going to click batch asset importer. We want the zip file. All right and just saving your downloads for now.
We'll open up Blender and we're going to go to edit, preferences add ons, and click this little downward facing arrow, and to install from disk, go to your downloads let's see, it's batch, there you go, batch asset importer, and that's the one we just downloaded here, so install from disk, alright, so then we can look at this over here, and Here is our batch import add on and it says you must save the file before you can import assets.
That's great. Let's go get some assets to import.
## Downloading Assets from Fab.com
Okay, so we're going to go to fab. com, the new Epic marketplace for all these all the items.
Okay. So we started off in fab and first we're going to go to environments and we'll type in Quixel. There we go. And that gives us a list of different Quixel environments. We'll be using the unfinished building for this one today. Okay.
We can see that included formats include Unreal and FBX. Click download. All right. And we're going to pick the unfinished building mid. You can go to a higher level, but it works the same. So let's go ahead and download that one. I'm going to put it in my put in my Unreal projects and I have a fab folder for this.
Okay. All right. So it downloaded that. So then let's let's go to that directory.
And here's our unfinished building mid. I'm actually going to delete the old folder. There we go. And I'm going to click, right click on this, and I'm going to extract all to unfinished building mid. The default file is fine.
All right. And what I'm going to do is copy this. This is the directory where the unfinished building mid is.
## Setting Blend File to Defaut Asset Folder
We're going to go back to Blender now. I'm going to make a new file. And we're going to hit A and X just to remove all the pieces from here. And type N and open up our batch import.
It says you must save the file before you can import assets. Let's save the file. And I'm going to save it in a very specific place, under Documents. And this is Windows, Documents, Blender, Assets. And I'm going to name it I'm going to save over this old file, unfinished building mid. There we go. And we're doing this for a very specific reason.
And that particular file folder is the default location for Blender to look for asset files. And that means if you put your asset file in there, then wherever you start up Blender, in whatever other directory, it can still read those.
## Selecting Megascans Folder and Import Options
Let's go to our folder path. And click our folder path.
And we're going to hit paste. To, for our building mid. There we go. Accept that. And I usually click use displacement. And I usually set my model asset type to objects instead of collection assets. You can, if you really want to use collection assets, you can do that as well. Alright, and then we're going to click import assets.
## Blender Asset Viewer
All right, so that's fairly quick, but if you wanna see the assets being formed, we can go here and hit shift F1 twice and we can go see the current file. And this is the Blender asset viewer. So this is the file, this is the assets that are in the current file, and it's created both material and 3d assets. The 3d assets are, you can tell that they're 3d assets as they're on a particular shape and the material assets are automatically rendered on a material sphere. So let's, uh, let's go ahead and hit save. And we can actually, uh, also external data. Let's pack our resources.
It packed 328 files. There we go. And we have just saved that back into the Blender. Assets directory. So what's really nice about that now is that now we can hit a completely new file. And we can say, delete this object. If I wanted to just start dragging and dropping in assets, it's really simple to do that.
So for example, hit my material preview and I'll move this up, shift F1 a couple of times to get our asset viewer. And we can see that we have unfinished building automatically shows up. And so we can either pick in our materials, so I can drag and drop materials onto this and you can just pick whatever materials you want which is really nice.
They make great materials. And we can also go to our models and you can actually drag and drop in models into the scene. And again, this is the mid level resolution. You can actually go to two higher level resolutions if you want.
They're considerably larger files. So we just did that for today. But you can actually drag and drop and all the material shaders are connected up. And now things just work. Alright, so that's pretty much it.
# In-Phone Re-Rendering
Source: https://docs.lightcraft.pro/tutorials/blender-workflows/in-phone-rerender
Jetset can re-render a clean background of recorded shots. This can be used to render Gaussian splats at high quality for post compositing.
## Video Tutorial
# Transcription
For this video, we're going to be going through our new in phone re rendering workflow. And this is a really interesting new way of doing virtual production that lets you recomposite shots in an editor without having to go through the traditional visual effects process of extracting EXR files and re rendering individual frames, etc.
## Self-Contained Re-Rendering
And for this, we actually will use the tracking data that's already stored in Jetset on the iPhone when you record. And especially when working with a Gaussian splat, we may have to dial down the quality settings to make that work well enough in real time.
But then we can go back and using the same tracking data on the take we just recorded, we can actually re render and generate an extremely high quality image. You can see this image in Fusion.
And we can see that this image is then correctly tracked with the camera original motion. And we can just re render a MP4 clip with it and then drop it into your editor. In this case we're using Fusion, but you could just as easily use Premiere or Final Cut or Avid.
And then the take is the same length as the camera original footage, so you just drop one on top of the other and do a composite. And pull a key, and then you actually have a fully composited shot. And so this can actually be used for either a very simple visual effects process where you're just going to rekey your camera original footage over the rendered and tracked footage, or you can use this as a proxy workflow.
If you're working on a larger project, you might want to actually just do a quick proxy render of your shot cine files, and use these for editing, since they will have the same number of frames and motion as the camera original files.
So this has a lot of potential for a lot of different use cases, and it's very easy to use because everything happens inside the phone. Let's get started.
## Capturing Tracked Takes
For this example we're going to be shooting with Jetset Pro, so the green screen footage will be recorded in the iPhone at 30p, and Pat is both modeling and operating the camera, so that's a little bit of parallel processing.
## Capturing the Source Take
I'll do a separate tutorial for re rendering with Jetset Cine as there are many more moving parts. We have the same coffee shop scene loaded as before, and we'll use the sliding door 2 scene locator. So we can go ahead and roll our take. And to move around a little bit in our coffee shop, see a little bit of the background.
And this is being rendered at fast mode for the real time processing work. And then we can go back and rerender at a higher quality for the Gaussian splats. Okay. So we can move around a little bit, see the tracking, et cetera. Let's see a bit of our coffee shop. Okay. That's a good take. Let's cut that.
## Re-Rendering Settings
All right. So go to our review panel. Okay, so there we have our recently shot camera original, and on the right we have our real time composite. You can see Pat's shoulder in the real time composite.
And now we have a one additional button, and that is the filmstrip button on the lower left of the live comp.
And when we click that, this is going to trigger our re-rendering workflow. And here we can see the settings that we're going to set for the background render. And right now, since we just shot this take, our current settings all match the current take. We don't need to do anything here.
## Re-Matching Previous Takes
Now, if we had shot a bunch of takes over the course of the day, and then we were going back and re rendering after we had moved scene locators and changed splats, et cetera, then we'd have the opportunity where we could hit match and force all those things to, to line up again before we re render.
## Rendering Clean Tracked Background Clip
They already match in this case, so we can just move forward. And we can see that we're re rendering at 1080p, we're going to be re rendering at 30 frames per second, which is correct, as we're shooting in this case in Jetset Pro. And we can go ahead and click render. And it's going to pull up a background render button, and it's going to work through the take that we just shot, and it's going to re render all of that and this time using a higher quality setting of the Gaussian splats, but with the exact same camera tracking motion.
And this is going to generate a clean tracked background plate. In this case we're using a Gaussian splat for the background render. But this will actually work with any 3D model. Okay, so it is now tracked and we can see that it has generated a new clean plate and now that clean plate has the suffix back.
Now we can move to the next part of this where we'll transfer the takes over with Autoshot and recomposite the shot in Resolve.
## Transferring Takes
Now we're going to go through the process of recompositing those files in Resolve. And in this case, Patrick already synchronized the takes over to Autoshot on his machine and sent me a take zip file which is this zip file. So when I unzip it, it creates a small a project folder subset with just the files that we need.
In this case, we'll go into the project folder. And we can find the actual videos in this folder.
Here you can see the four videos that will come over. And we can see our normal zlcam, that's our cam original. We can see our normal zlcomp, that was our live composite, and our zldepth. And now we have a new file, which is our zlback. That is, again, the background render file.
## Loading Takes Into Resolve
The ones we want are actually just zl back and zl cam. I'm going to grab those, drop those into Resolve. And then we can double click on this, and we can see our, here's our cam original file. And we can see that it's 1920 by 1080 at 30 frames per second. So that's a file typical of recorded inside a Jetset.
And there's our background re render file. So we page through it. There's a nice re render of the Gaussian splat all re- rendered completely inside the phone. And it has the same setup. All right, so let's go through this. So we're going to start with our zlcam file.
I'm going to right click and I'm going to make a new timeline. And let's just double check to make sure that our timeline frame matches the the actual clip, 30 frame per second, 1920, we had 10 80 hd and we're going to create that . We can double click on that one, and then we can go and see our file and in the timeline. We're going to be doing our compositing in Fusion so we can just highlight this and then go click on the Fusion tab.
## Replacing Duplicate Frames in Fusion
In this particular case we were shooting with a Gaussian splat loaded on an older iPhone 13 Pro, so we had a few duplicated frames in the cam original file. And we're going to want to go ahead and fix those first.
Ideally, you avoid duplicating frames on your capture, but just in case it happens, here's how to fix it.
We can actually type in control space, and type in frame, and we're going to want the frame replacer node. We'll add that, hold down shift, drag it into the timeline. We're going to highlight it, hit 2, so we can display the output in the second viewer here in Fusion.
And as we go through it I already know a couple of the frames that were problems. We can zoom in a little bit and see where it happened. Frame 618, I believe. So there's 617, 618, and then 619 is the same as 618. So that's going to cause a visual glitch in the composite.
So we want to repair that. We can use the frame replacer, and the way it works is that you click replace this frame. Now you have to be a little bit careful because this is a keyframed behavior, so you have to set your keyframes to turn this both on and off.
It took me a little while to figure this one out. We're on the central frame that is the problem. This is the duplicated frame. So 18 has motion. 19 has no motion. Before we replace the frame, we're actually going to click our keyframes on that frame. I'm going to go 2 back and set a keyframe, and 1 back, set a keyframe, and then go forward to 620.
And set a key frame and 6 21. So we've now set a the key frame on the center problem frame, and we've set key frames on the two frames before and after. And the reason we're going to do that is we're going to go back to six 19 and we're going to set this to optical flow because that does the best job.
We're going to click replace this frame, and we're going to go to the previous frame. And forward to to 620, as well, and replace that. And we're replacing three frames in a row. And that will smooth it out the best. You'll notice that we've keyframed it off when we go back to 617.
Alright, we have a key, we have turned off the keyframing. And we go forward to 618. And now we've turned on replace that frame. We replace the frame on 619, we replace it on 620, and then keyframe that off on 621. And then it behaves the way you'd expect it to. If you don't keyframe the behavior to be off, you're going to get some strange looking results.
And the other one we had a problem with was 620, 628. There we go. So 626, 627, 628. The duplicate. So we're, once again, we're going to go back 2, and add a keyframe. Go forward one, replace this. Two, add a keyframe, and 629, replace this, and turn off, and replace, and add the keyframe there. So we should have the central the central one which is 628, and on either side we replace the the frame on 629, 628, 627, and then on 626 we keyframe it to be off, Three, three frames on, and then once again off. And then it behaves the way you'd expect it to behave.
Alright. And so you just do that for the different frames that you have a problem with.
## Keying with Fusion's Delta Keyer
Okay, so once we've repaired our keyframes, we're going to want to key this. So I'm going to point you toward the tutorial I used for reference. If you want to go deeper into this by VFX Study, I'll put the link below. He did a great job with this, so I'm going to be following a lot of his process.
So we're going to hit control and shift, and we're going to type Delta. We're going to be using the Delta Keyer here. That's the main keyer we use in Fusion. Hold down shift drop it into the flow. We're going to be branching this frame replaced video out to multiple places so we're going to hold down alt and click here and that'll make a junction node here. Then what we're going to want to do is highlight our Delta Keyer, hit 2, so we see the result of it, and we're also going to hit 1, so we have the Delta Keyer on both of our viewers.
And why are we doing this? We want to change this viewer to show the alpha, so we can show the reaction, the behavior of the Delta Keyer while we're doing this. So then we can highlight our Delta Keyer, and come over here, grab our eyedropper and as we move around we can see what it's doing with the alpha channel on the Delta Keyer.
And depending on where we pick, then we can pick a, some reasonable area for it to pull the key from. And this is 4. 2. 0 compressed 8 bit h. 264 footage, so we're going to run into some limitations, but we'll go as far as we can.
## Making the Core Matte
Okay, so we've pulled the key and If we zoom in here, we're going to notice that we have some transparency in his pants, etc. And we could either try to really crank in that key but we're going to use a neat technique of using both a soft and a hard key, a hard core matte. So instead of just crunching in on this key, what we're going to do is actually make another Delta Keyer here and I'll just copy this and paste it. And so this Delta Keyer now has the same color pick as the first one.
We're going to feed in our feed from our frame replaced footage. And we're going to hold down alt and drag onto here. And we want to bring this into the input. That's a neat trick from VFX Study.
Over here, our second Delta Keyer is highlighted. We're going to type in both 1 and 2. And what we have done is we have just routed this to the left and right viewers in Fusion. And it doesn't look any different yet because we just copied the node, but we're going to change the keying settings on this one to just do a core matte. So to do that, we're, we can crank up our gain, and then we can go into our matte.
And we can up our thresholds to, to get rid of some more schmutz there. We can zoom in here. We have transparency on his arm. We're going to drop our threshold all the way down. So then we've got this very solid matte. Now this would look really terrible if we just used that as the matte because we have all these chunky edges.
We're actually going to erode it. So we're going to erode it and dial it down a little bit. And that gets rid of both some of the other schmutz in the scene and also cuts our matte back. So that's great. But how do we get back into the original Delta Keyer?
We're going to highlight this Delta Keyer and type in two. And we're going to zoom in here, and so we can see here, we have transparency in, in, in lots of the areas of the matte because we didn't pull it in like we did the second keyer. We're actually going to take the output of Delta Keyer, hold down alt, drag it into Delta Keyer one. And we're going to bring that in as the solid matte. All right. So that has just filled in the solid matte of the core of it. So we haven't crunched in the edges hard yet. We're still going to be working on that, but we now have a solid matte for that inner portion of it.
## Tracking a Garbage Matte with Planar Tracker Node
Great. That's a good step forward. And one of the next things we have is, of course, we have schmutz in the scene, right? So as we move over, We have a, we have a ladder over here that we're going to want to get rid of. And in order to do that, we're actually going to add yet another node.
And we're going to track a garbage matte around him. And this is a decent place to start. Let's look at the shot where he's walking through the scene.
Kind of walking over there. Okay, so there's a good spot where we care most about the garbage matte. So we're actually going to hold down control and we're going to type in our PLAN for and get our Planar Tracker. I'm going to add that there. And if it goes in the wrong place, just hold down shift and drag it.
And once again, we're going to drag in our feed from our junction node. Hold down alt, drag it in here, and we're going to pick Background which is that contains the region to track. Okay, and we're going to highlight this press 2 so we can actually see Our Planar Tracker interface and it's already put the kind of the center of the tracking area on him That's fine We can actually move that around a little bit To get a little closer to center on him And then what we're going to do is we're going to click and drag around him and we're going to make a rough garbage matte all right, and You don't want it to be like too tight for reasons we're going to see.
I'm going to drag that out a little bit. Okay, so we're going to come over here and we're instead of our point tracker using the hybrid point area is much more robust. For the motion, we don't actually want it to have perspective. We just want it to do translation, rotation, and scale. And we're going to click set here to set our reference time for the tracker.
It's a frame 290. So now we're going to track forward and it's going to track our garbage matte nicely around our shot to the rest of it. There we go. And we'll go back to 290 and then we're going to track backwards. So now we're going to go through the rest of the shot. And as he walks around, there's our garbage matte tracking him nicely, we may have to do a couple of adjustments. You'll see that it's translating and adjusting to scale as he moves around closer and further away from the camera.
So this is a really convenient way to get a fast tracking garbage matte around the subject moving around.
All right, so I just tracked a thousand frames, very quickly. That's pretty cool. Okay, so now, we have a tracked matte, but we need to change the output to be a mask. Alright, so now we have a white mask how do we tell that to the Delta Keyer?
Once again, we're going to hold on to the output of the Planar Tracker, hold down alt, drop it into the Delta Keyer. In this case, we want it to send it as a garbage matte. Alright, let's highlight our Delta Keyer, hit 2 so we can see our output. Okay, it just disappeared haha, we just garbage matted out the person we wanted to keep.
We're actually going to go over to our matte or over to our mask, and we're going to come down to garbage matte and click invert. There we go, there he's, now he's back.
## Adjusting Keyer Edges
Okay, so now we have a garbage matted shot, where he's walking through the scene. But we need to tune this a little bit. There's a lot of things we can do on Delta Keyer, and for the more detailed stuff, I'll point you toward VFX Study, or one of the other people out there.
We're going to do some simple things here. I'm going to highlight our Delta Keyer, and come over to our matte, and I'm just going to get rid of some of our threshold. Here, I'm going to zoom in a little bit and get rid of some of the fringe here.
And note, we can actually drop our thresholds without causing transparency to happen in the core matte. And that's because we're using a core matte to drive the center, and that way we can adjust the edges of this separate from the center.
We can walk through the shot, and it looks like we've got rid of most of the various bits and schmutz the ladder, etc.
## Adding Re-rendered Background
So now we want a background. And to do that we're just going to go over to Highlight Media Pool, and here's our tracked background.
And I'm going to drag this down and it'll come in as a media in node. I'll highlight one so we can actually see the process. Now it looks bright white. That's because we still have this set as an alpha channel. So let's go ahead and change this to color. There we go. There's our There's our nice tracked background and we're going to adjust the color of the background just a bit.
So I'm going to hit control space and type in color. We're going to get our color corrector and we'll bring that over here. And press one to highlight the color corrector in the left hand node. And this one, I'm just going to do a couple of simple things. I'm going to up the contrast just a little bit to make it closer to the photographic image.
And I'm going to also dial down our saturation a little bit because it's very saturated. All right. Next we're going to want to try merging this. We can actually do a shortcut hover your mouse over the central area, hit M for merge, there's a merge node, and we're going to drag our output of the color corrector, hold alt over the merge node, we're going to put that in the background, and we're going to take the output of the Delta Keyer,
and move that over. Once again, Alt click to get a junction node, take the output of the junction node, hold down Alt over the merge node, and we're going to put that in the foreground. And we're going to route the output of this into the I'll put this into the media out.
Okay. So now the key thing here is let's take a look at our merge node, highlight it too. So here we can see our composite put together for the first time. All right. And so that's, it's tracking, it's matched. That's good. But of course he's, he doesn't look like he's there yet. The color is really far off.
So let's go ahead and fix that. And so once again, we're going to hit ctrl space and give ourselves another color corrector node. There's a bunch of ways to do this. I'm just going to show a simple way. We're going to put this in right here.
Let's do some organization. Bring this down a little bit. Bring these down.
## Matching Foreground to Background Color
The color corrector has a bunch of different capabilities and one of the abilities it can take in is a match reference. So we're actually going to take in the output from this and just go hold down alt, bring it in, go to the match reference.
We're going to highlight our merge node, hit two, then we're going to highlight our color corrector. So we're modifying our color corrector node, but we're going to see the output on the merge node and for the color corrector, what we can then do is under the menus, we can just go to our histogram and we have a bunch of different tools we can use for this.
This shows the histogram of the background and foreground feeds, basically of what's coming into the match reference, as well as what's coming into the input. So we can see how the histograms compare and they're quite far off.
If we just click the match tool, it's going to do some strange things. So it, one, it affected the entire image and to fix that, let's go up to our options and we're going to click pre divide and post multiplying. That way we're only affecting our foreground image.
And this is, this really clearly took it way too far. You, there's a couple things we can do here. We can actually dial down our match RGB until it's something closer to what we'd expect to see. And there's a ton of layers we can go into on this. I'm just going to do a basic thing here, to show how the pieces go together.
Let's take a closer look, and we still have a lot of edge in our key, and I'm going to go back to our Delta Keyer here, and I'm going to adjust our threshold a little bit more. And cut back into that. But there we go. So now we have our composite put together.
## Exporting Composited Take
And so once we have our comp put together, now we can actually go and write it out. So we have the output of the merge going into the media out node. So that if we go back to our editorial timeline, we scroll back and forth, we are seeing the the finished composite in the timeline.
And we can set our in and out points here. The very first frame was black on the cam original, so we're going to have a kind of a strange frame 1.
But we can click I to set our in frame, and O to set our out frame. Now, One thing we have to watch out for here is when we render out this clip, on the very last frame we see something that says, no frame available for media out one. So what's going on there is we go over to here, and media out one is this key here.
And so in this case it looks like there is no frame data for the very last frame. And that, that can actually cause a few problems. Usually you're using this clip for editing, so we can just trim one frame off the back end. That's fine. So we can actually come back one and hit O. And that sets our output to a frame range that's valid.
## Matching Clip Name
For the frequent case where you are rendering a proxy, we're just going to keep the name matched. And let's just double click on our name. Let's copy on that, and come over to our output. And go to the file name CTRL A, CTRL V to paste.
And let's change the file name on this from ZL back. We can just change it to recomp. There we go. And we can write it to the same directory folder we were at before.
We can just go over to our our user files click here to grab our directory, come back here, control A, control V to paste. Okay. And then we can add that to the render queue and render.
All right. So we can go back and let's take a look at our folder. And we have rendered in our recomp file back into our folder. We'll drag this back into Resolve, take a look at it.
You can use this in editorial work, and the frame numbers, et cetera, will match back and forth between your camera original. And this isn't so important for this shot done with Jetset Pro because the real time composite is already 30 frames a second and makes a good editing proxy.
But we'll do another tutorial using re rendering with Jetset Cine which this should be extremely useful for making rendered proxies with because the proxy will have the same frame count and frame rate as the 24 frame per second or 25 per second uh, original.
# Patching AI Mattes
Source: https://docs.lightcraft.pro/tutorials/blender-workflows/patching-ai-mattes
Use Blenders built-in image sequence editing to fix glitches on AI-generated matte sequences.
## Video Tutorial
# Transcription
In this tutorial we'll see some common problems with the gray scale AI matte sequences that Autoshot generates, and how to go about fixing them with Blender's built-in image sequence painting tools.
As you saw previously, for our green screen shots, we primarily use AI mattes as a core matte or for use with a transparent image plane.
For compositing, it's eroded a bit and plugged into the core matte section of the keyer .
However, AI mattes can flicker, which then shows up as a flickering key as we see here in frame 786, and going forward in our timeline.
## Trying Different AI Matte Tools
Different AI matting tools have different results on a given piece of footage. So your first step would be to try changing the AI Matte tool selection in Autoshot, change from Monet to PPNet, and rerunning the shot. You'll still probably have a couple of flickers or unwanted areas, however that need some hand retouching.
## AI Matte Greyscale PNG Files
So, fortunately, Autoshot generates the AI mattes as a series of gray scale PNG files that are easy to do small paint fixes with inside blender. So as you see there, it's just a series of PNG files.
## Blender Scene Setup
We'll open up another instance of Blender and we'll hit A to select everything, X to delete, and then we're going to do shift A to Add Images as Planes.
And then we're going to browse to the directory where those AI mattes are stored.
And we'll click A to select all of the files and we'll click Animate Image Sequence to bring that in as an animated image sequence. We're going to set our material type to emit and then click Import Images as Planes. Okay, great. So Z for material preview.
Give it a second. Okay. There's Josh. Thanks Josh. And then we can scroll along the timeline to see the image animating. Great. That's our gray scale image.
## Blender Texture Paint Tab
Next we're going to change to our texture paint image tab. And then we're going to make a horizontal split, and add a, a timeline to the bottom. That's shift F 12 and zoom out until we can move our timeline over to the area we cared about. That was about frame 780.
And then N to open up our side controls and click image.
## Setting Sequence Numbers
And we're going to set frames to the number of frames in the AI matte directory, in this case, 976. Start is one and Offset is 1000, as all of our image sequences start at 1001. All right, so then we can jump to Frame 786. Let's actually change to our timeline, so then we can jump to frame 786 to see same frame where the flickers start.
We'll bring this a little bit water so we can paint more easily. For filling in AI matte patches, it's easier to turn off pressure sensitivity, so I'm gonna turn that off here. Now we can work through the image sequence patching the holes.
## Painting Out Flickers
We can right click and pick our radius.
All right, there's our radius. Then we can paint that and then hit right arrow to move to the next frame, right arrow and work our way through painting, painting the fixes pretty quickly.
All right. You get the idea.
## Saving Edits Back to PNG Sequence
And now we're going to choose Image and Save Sequence to save out the updated PNG files.
## Refreshing Mattes in Blender
We're going to toggle back to the original Blender. When we move back and forth in the timeline, all the flickers are gone. So then you can exit and discard the temporary blend file you made to edit the sequence as the image information is now saved in the PNG sequence. All right. That's it.
# Render Farm Export
Source: https://docs.lightcraft.pro/tutorials/blender-workflows/render-farm-export
Autoshot can wrap up the files needed for a render farm export into a zip file ready for upload to SheepIt.
## Video Tutorial
# Transcription
Now that we have the 2D composite working, we want to render out a shot and rendering with cycles can take some time, so it's useful to be able to farm out the shots. And there are two major ways we'll look at doing this.
One is to set up Blender's, Flamenco render farm system on your own machines. This is pretty easy if all the machines are in a local network, but it's trickier if they're distributed. We'll go over setting up Flamenco on a distributed VPN based network in the Blender shared workflow section.
Here we're going to focus on using Autoshot's built-in shot packaging system to wrap up our shot into a self-contained zip file that can be sent to SheepIt or similar Blender based render farms.
## Default Render Settings
First of, we'll check that our Render Out file output is set up the way we want. And under our render properties, you can see that Autoshot defaults to 128 render samples, with the denoiser checked, and it does that for all of the generated render files. You can increase this if you want, but the render times go up quickly and SheepIt requires your per frame render times to be under 20 minutes, so it's good to be careful on samples.
Under the Output Properties, we have the resolution set to 50% of our incoming U H D images for an output of 1920 by 10 80 hd, which is fine for this case. Jetset by default records to 30 frames per second, so that's fine here.
## Setting Frame Range
And this is the frame start and stop point that we set in Autoshot. You can of course change this before submitting the file. In our case, we're gonna change this to from frame four 60, uh, to frame four 70, just for a small example case
Our output is set to half float multi-layer EXRs with D W A A compression. And this is the required compression for EXR files in SheepIt, and it's a very good compression system. It retains the dynamic range of EXR files while giving jpeg like compression ratios.
## Trimming Layer Count
And next, let's look at the View Layers tab.
Sheepit has a maximum of six layers or passes. So we'll need to remove a couple of our default enabled passes to fit. So we'll scroll down, we'll turn off Cryptomatte, if it's checked as we're not using it for this scene. And that adds three more layers. We'll also come up, and if the Z depth layer is checked, we're going to uncheck it, as we're not using it for this scene.
The Mist pass is very similar and it's more useful for compositing.
## Matching Scene Render Engines
One thing you want to make sure you do is to go up to your garbage matte scene if you've made one, and make sure you switch the engine to Cycles.
SheepIt requires that the render engines for all your scenes be identical. So if you're rendering the main scene in Cycles, you need to set your garbage matte scene to Cycles as well.
## Keep \_rndr File Suffix
You're going to want to rename your working file back to \_rndr, even if you've made a couple of updates and incremented the file name. So we'll do that. Save as, and we're going to resave it as render.
## Autoshot Render File Pick
So then we'll pull up Autoshot. Now Autoshot is going to look for the render file that it's going to process based on the shoot day and take selection that you have set in the UI. if you haven't processed a different shot, the previous settings are retained, but just check.
We can see up here we're on scene 101 take 017 on the 01/21 shoot day. So 1 21 shoot day. Scene 1 0 1, take 17. All right, so we're matched.
In this case the scene blend file and the scene locator, we don't use that for the farm file setup, so that doesn't matter. That information is already baked into the render file that we've been working with.
## Farm File Output
This process will write the farm output data into this D:\tmp\tempfiles area, our temporary files directory. So we're gonna pull that up so we can see the output happening.
And we're going to pick File, Export Render Farm Zip, and the default is fine and we're gonna see a lot of things happening.
When Autoshot makes the linked \_rndr file, it bases it on EXRs with D W A A image compression for the camera originals. For most visual effects work image sequences are heavily preferred over dealing with movie type files like MP4 or mov, as it's much simpler to debug and make work reliably with image sequences.
## Re-Extraction of Camera MP4 Files
However, SheepIt has a file upload limit of 750 megabytes before zip compression and the DWAA compressed EXRs, or almost any EXR file, is much too big for this. So Autoshot to solve this is going to re extract the user selected frame range, what we picked above from 460 to 470, directly from the camera original MP4 file, and write it into a new MP4 file that just has the frame beginning and end selected by the user.
It's then going to relink the camera original follow reader in the Blender Compositor to this newly created MP4 file, so the \_farm file composite will render the same way it did when you had the \_rndr file open.
And this is done with a direct ffmpeg conversion, so there isn't much data loss or recompression, and then we can easily fit 4K videos into the farm file 750 megabyte upload limit.
If we double click in here, can see the generated camera original MP4 file. The AI mats, uh, are brought in over at as image sequences as they are black and white or gray scale image sequences, and they compress very efficiently, so it's fine to leave those as an image sequence.
Same thing with the depth files. They're very small, so bringing them over as a sequence is fine.
## Packing Linked Blend Files
Autoshot will also work through the linked scene files that were referenced in the render file and link all of that together into a single blend file in the farm directory so it has all the information packed into it.
We can check our size here. We can get to properties and we can see that before we start zipping, our folder size is, 663 megabytes, so we're fine under the 750 megabyte limit.
And finally it zipped it up into a zip file, that can be uploaded to SheepIt. So let's go to SheepIt.
## Uploading to SheepIt
We're on the SheepIt page under Get Started and we'll scroll down and we, we've already logged in. We can add your project, we can click choose file, and we'll go to where we wrote the file out tempfiles. There's our farm file. Double click that and we'll say send this file.
It's going to process that.
You can see that it accepted the file and it's giving us a couple of choices. We want to use Blender 3.4 right now as that's the most recent Blender build. We can add GPU to the compute method and it has the correct start and end frames and then we can click add this Blend.
Okay, now it's starting. And this will show, the progress of the frames as those, uh, 11 frames come in. And that's it.
# Blender Viewport Compositor
Source: https://docs.lightcraft.pro/tutorials/blender-workflows/render-scene-viewport-composite
The Empty Comp Starter option for Blender in Autoshot automatically creates a sophisticated viewport composite. Watch this to see how to access and adjust it.
## Video Tutorial
# Transcription
In this tutorial, we're going to take a tour through our newly created render scene, as well as the basics of the viewport compositor.
## Why Viewport Compositing?
First of all, why viewport compositing? We think that being able to work simultaneously in 2D is a huge advance to making visual effects shots.
And Blender is, is really unique with this respect, with its newly integrated realtime view Port Compositor. For the first time there's a fast c compositing system under the same hood as the real render engine in, you know, this case blender EEVEE or Cycles.
So you can actually see how the whole shot's gonna come together and interact with both the 2D and the 3D at the same time to match things. So for our workflow, we'll first use the Viewport Compositor to help refine alignment of 3D objects with the live action footage and the timing of the CG and live action.
And later, we'll use the traditional 2D blender c compositor to do the fine tuning of the composite shot, including matching black and white points, detailed keying and, and the environmental matching work.
## Render Scene Tour
all right, so let's look at our render scene. So first of all, press Z and go down for material preview.
We're going to middle mouse button, click and drag to rotate the scene. Uh, and if you see a doubled scene like this just make sure to go over to here and set your composite to camera. Should be that by default. Uh, we'll scroll back out and here you can see, uh, the, here you can see the image plane that Autoshot created in the scene.
And go ahead and click on it and then use the period key in the outliner to highlight that. And you're gonna see that that image plane is set to render in the view port, but not render in the final render. So we can see it, but it won't be rendered in the final image. And that is because we generally use the image plane just to kind of figure out where our person is in the 3D scene.
But we use the c composing system to actually composite them into the 3D scene because we have much more control in that way.
This is fun. We can scroll, move forward and we can see our actor walking and then running through the scene. And, and it's not perfectly accurate because the, the depth of, of the actor is set from a combination of the AI mattes and our realtime depth image from LiDAR. Uh, so it's not perfect, but it gives you a decent approximation of where they are in the scene and everything's one to one scale matched, so they are roughly their correct size.
## Correcting Animated Material Texture Display
And sometimes if you see your texture acting a little strange here; every other frame will be missing or something like that.
So in
case you run into that behavior, here's how to fix it. So we're going to right click here, we're gonna do a horizontal split, move that a bit down. So we've got two timelines. Then we're gonna go over here and, hit Shift F3 until we have our material shading node group.
So here you can see how the material group is built for the image plane. Note this is not the, the C composing group. This is just the image plane. If you have Node Wrangler installed, you can hold down shift control and click on, your different image inputs to the Principled BSDF shader.
And then you can see them show up in the view port. You can see the, the little node line relining. So we can see we have our camera originals that don't have any alpha channel. And then we have our AI mattes. So as we go through it, you can see our AI mattes animating and working. And this is great because it lets us put together the system in the the material nodes without having to try to key things out.
There are limits to the AI mattes. They only see people instead of objects. So if there's an object you're not going to see it in the AI matte, but again, it's, it's great for an approximation.
## Viewport Composite Node Graph
All right, so now we'll look closer at the Viewport Compositor system. So go up here and type Numpad 0 to go back to the camera view. And we can actually hide for a little while the doubled view from the image plane and shift F three in the lower area until we get our compositing nodes.
Now you can see the three original main nodes, the render layers, the Zipline node group, which will be renamed Autoshot soon, and the composite node.
## O for Output Node Shortcut
The Viewport Compositor displays what's connected to the composite node. The shortcut for this node is to hit the letter O for output. To see this, highlight the renderer layers and hit O for output.
One layer of the actor disappears as well as the non-green screen area of the set. The image plane is still there, but since the AI matte only recognizes the person, we don't see the set. So go ahead and highlight the Zipline node group and hit O, and our full composite is back.
## Disabling the Image Plane
We can temporarily turn off the view port visibility of the image plane to avoid confusion here. Just click this and now it's just the Viewport composite. So if we click this, you know, he disappears, click this, and he reappears.
## Expanding the Autoshot Node Group
We want to dive into the Zipline Autoshot node group, but we want to use our magic O button shortcut, so the easiest ways to ungroup the Autoshot group. So we're gonna right click on the Autoshot group and choose Ungroup. We'll scroll back out with the scroll wheel.
All right, so the node tree will expand. Uh, the render layer group is in the middle, causing some confusion. So go ahead and, and highlight that here. Hit G for grab and move it over here into, its, its blank area. And now we can navigate through the comp.
The two main areas of the comp are the render layers, which is the output of the rendering system, and we already saw that, and the keyer and live action areas, which take the original footage and key it and bring it together.
## Viewing Camera Original Footage
We already saw the render layer output earlier, so let's look at the live action and keyer side. So go ahead and, and highlight the camEXR node and hit O.
Now we can see our original image at full resolution. It's big because it has not yet gone through a scale node to fit it to the rendering resolution, in this case, the viewport size. But we can do something important here, which is to pick our core green background color.
So scroll down and click the sourceGreenPick green node bar and click the eyedropper. And then go ahead and pick an area of the greenscreen next to the actor.
And while we're on the topic of color, we should understand how the render scene color management is set up. So let's go over here.
## Color Management
Let's click our render properties and scroll down to color management.
So note that the view transform is set to standard and the look is set to none. And this is important to keep the live action footage looking like you'd expect it to look. We're not adding any other filters or view transforms over the original live action footage here.
## Filmic in Compositing Node
Now note that this means that we will need to use a node in the C compositing tree instead to use things like Blender Filmic to get the 3D rendering to look photographic. You'll want to look at the settings of your original scene file, and then transfer those settings to the correct node in the compositor.
We'll get into that in a later tutorial. Now that we've seen the original footage and we picked a green color, we want to look at the keyers. There are two keyer blocks here. There is the live key and the post keyer. We need two keying pathways at this time because the blender key node, which is this has not yet been ported to the view port compositor, so it won't work in real time.
## Live Keyer Nodes
So inside the live keyer block, highlight the screen compensation live node and go ahead and press O. And you'll notice that the background green screen turns a single color green. And this node group and the matching node group in the post keyer block do some special processing to remove color variations from the background before sending them to the actual keyer nodes.
We'll go through the screen compensation nodes in more detail in the post C compositing tutorials. And in this case, in this shot, this can make some strange artifacts like his white shirt. So we don't need to use this green correction group in all cases. We can highlight the screen comp live group and hit m to mute it and the node grays out and you can see the effect goes away.
So then we can go to our live keyer, and hit the O key. And we can see he's being keyed out, but there's still quite a bit of green spill and fringe. So we're gonna move on to the color spill node hit O and that has removed most of our color spill. Great. Again, this is a rough keyer done for the realtime viewport keyer, and then we have another pathway that lets us do more fine-grained keying. We'll go through that in the post c compositing workflow.
So note that the output of the color spill node goes to the keyer live switch node. It's set to on right now. So the node pathway we've been working on is active, which is the live keyer. We'll flip this off when we go to the post-compositing keying.
## Scaling the Comp
So now highlight the Scale node next to this and press O. Now the composite is scaled to fit the render size of the view port, so we're fitting our render size. If we scroll back and forth in the timeline, we can now see him walking through a gray void. All right, so he's been keyed and, and extracted from the live action.
Go ahead and highlight the Match Focal Length next and hit O. If the original shot was zoomed in to simulate a given focal length, here's where the adjustment to the 2D comp would be applied. Our original shot was at the native phone resolution, so it doesn't have any effect.
## Combining Key with Background
Then we're gonna highlight alpha over and press O. Alright. This composites the scaled comp over our render layers output from the 3D rendering engine and the render engine is already at the correct scale because it's rendering the view port.
## Cryptomatte Node
Finally select the used Cryptomatte node and press O, and we can move our composite output node back here. There we go. Cryptomatte is not active yet in the viewport compositor, but it's good practice to leave the composite node at the end of the node tree for actual renders
## Fast Viewport Render
it's also really fast and quick to make a viewport render of the shot. Go ahead and click N in the, in the viewport. If you don't see it, you can scroll up and down here to go through different options. There's our Zipline panel. And you can click Fast Viewport video, and this will make a viewport render of the scene to a MP4 video in the / preview directory.
This gives you the fastest possible render of the scene. And it will be just using the viewport preview to render it. So depending on your scene complexity, it could be a frame a second, two frames a second or up to nearly real time.
These are 4K frames we're dealing with, but if you have 720p or HD resolution, then Viewport Render can go nearly real time.
## Resetting Output to EXR Settings
And then after it's done, go ahead and click Standard R ender Settings to reset the OUT directory variable back to the EXR settings. And those are set here.
Okay. That should do it for our little tour of our viewport compositor and our blend file.
Next we'll look at building a 3D garbage matte.
# Animated Character Export
Source: https://docs.lightcraft.pro/tutorials/houdini/animated-character-export
Exporting complex animated characters with a simple technique.
## Video Tutorial
## Dragon Model
[https://eye-candy.xyz/](https://eye-candy.xyz/)
# Transcription
Hey guys, just talking about the workflow, uh, for the kind of re meshing and getting this into like a, uh, bone based animation here. So what I figured out is if you, I mean, this is how I did it before, I kind of ran through this with a screenshot, but, uh, to get the low res, uh, in Houdini, you just do the original Alembic time, shift it to the frame, you wanna frame hold.
## Picking Animation Frames
Um, if you have like a mouth opening or something like that, that's the better frame to do it. Otherwise you get like, you'll like. It'll kind of re mesh that mouth together. So sometimes I notice if you do like that frame and then you convert it, uh, you just do a convert node and that will make it, uh, so that this poly reduce works.
## Adding Poly Reduce
And then you put a poly reduce and then you're good. And then basically a point deform with the original. So this is the original point deform, and that's basically it.
## Conversion to Bone Based Animation
How to get the joints all over this, or the bones rather. So I reimport this as a Alembic just so it's faster. Uh, if you leave all this stuff, it's kind of long.
So you leave that on skinning converter node, which is what, uh, was recommended in that email. Now the trick here is you don't see anything actually happen. You can't see it play. You wanna preview it, you can plug it into a bone deform and then see how it's actually gonna look. So this is what it looks like with a bone deform.
## Exporting to Blender
Now to get this into blender, I mean you probably could export this as used easy, but you want to get in blender, uh, and have it work with like locators or just your scene. Um, wrap. F-R-F-B-X doesn't give you the animation. You have to use RFBX character output. And the key here is you have to basically check this box.
This is like the hidden box that I couldn't find. And if you don't do it, it it comes in the blender and it basically explodes. So yeah, remove scaling from joints will make it correctly interpreted and you can also turn off, uh, convert to specify unit system. 'cause sometimes it will come in, in a really large scale in Blender as well.
So those are the two check boxes. Um. And you can also manually convert the access if you need to, but I don't think that's necessary. It's really just this one that if you don't hit it, it's not gonna go well. So back in Blender, uh, how it comes in. If you import it, I'm just gonna
disable this for a second and say,
let's see, go to post mode, select all the bones, and I'm just
gonna delete this. So this is how it comes in. Normally it's like you have the mesh, but you have like all these huge bones kind of not very nice to look at. Uh, so what you can do is if you go to pose mode and just hit a and select all the bones, you can select the custom object.
So it just gave like a little tiny ICO sphere, so it's easier to look at for the bones. And then you just, uh, right click and say Copy is selected. You can see the bones. But basically blender's just not good at visualizing it. So that's what it kind of looks like. Now, if you go back into object mode, we have just object with our bones on it.
## Exporting to USD
This is properly working. And then what you can do is, yeah, just select, I think I had to select, uh, hierarchy and then export that as USDC. So I did, uh, file export USD. And then animation selection only. Uh, that's pretty much it. So I haven't figured out the object locator thing yet. I have one in my scene object lo uh, with the proxy.
I, I think I didn't parent the object to it 'cause I was messing around with this a few different times, so probably that's why it wasn't working. But, um, yeah, that was the workflow for joints. Uh, it's working pretty well as well. So this is, uh, 5,000. Actually it's less than that.
Only, yeah. A thousand versus actually, it's pretty low res, so, um, yeah.
# Aximmetry Setup
Source: https://docs.lightcraft.pro/tutorials/jetset-cine/aximmetry-setup
Aximmetry provides a high quality, real time software keyer that integrates with Unreal for a complete compositing solution. This temporary tutorial will be updated later but will get people started.
## Updated C-Tracking name
Aximmetry just renamed AGTP to 'C-Tracking', or a 'Comprehensive Tracking Protocol,' so in newer versions of Aximmetry (version 5.5+) please select C-Tracking instead of AGTP!
## Video Tutorial
# Transcription
All right, I'm going to start from scratch using for the Aximmetry setup the easiest way I can, I'm going to use the Decklink since anyone can use it. This is the installed one and its own little box with the USB plugged into the computer or into the laptop. And let's begin.
## Opening Unreal Editor for Aximmetry
All right, so I made a, I used a Courtyard modular courtyard free map, and I'm going to now open up the special Unreal from Aximmetry . Now I'm going to open up the modular Courtyard. And I'm going to hit Open and it's going to create a copy there. It is Modular Courtyard, and it's copying everything over, including the USD Zs and the Blend files that I made.
Notice that it added the 5.4 on there.
## Adding Camera
All right, here's our new model, and all I'm going to do is add camera, the second one, the second selection,
and there it goes. And this is the X axis that it landed on, which is very far away from the actual place. So that's why we need the.
The scene lock. Alright, so that's what I'm going to do there. I'm going to save this.
## Opening Aximmetry Composer
Now let's open up Aximmetry. So I'm going to open up Composer.
I already have the USDZ loaded in the phone.
All right, so I have the Decklink there with the camera feed. All right. And then here's the A GTP set up. Let me show you how that was done. So here we go.
So I got the 1 5 4, which is my computer, and that's how I have it set.
Here.
Right there. So saved to the computer. Now let's go back and let's, so let's drag these in from scratch. So the video, the Decklink is a feed. We're going to take the camera tracking and then drag that in there. And this is our output. Which is the monitor. Go to create a new blank document. All right, so there's nothing here.
We're going to drag the Unreal camera link right there,
step one.
And the next step is we're going to go find our folder. With the Aximmetry one, the new one created. So it's 5.4 and we're going to drag this in here.
And then since I didn't cook it, we're going to go live sync. And now I have to, now it's asking me to hit plain Unreal so I could connect it. So there we go. Let's do that. Boom. Nothing's, it doesn't look right. That's okay. Now we hit link. There we go. I like to sometimes turn on the low latency, which should work.
So that's only happened, shows up when I select it here. Alright, so we're going to go the two outputs so we can see what they're doing. Preview, and then out. Now you see that right here, we're going to get our feed. So there we go.
I hope that you can see my mouse in the recording. Yep. There it is. Okay.
All right. Now here's the steps. So let's go. This one's A. This one messed me up while so out has to go to rendered B, mask to B mask. Then we go. Match these up by name.
We're starting to see some progress b texture.
Okay. So we should have connection. There we go. Panning the camera back and forth. Looks good. All right. Next steps. Cameras. We can, I think we can ignore that one for now. Alright. Studio mass mode. Very important. Turn that off. There we go. Now we have the camera feed, so we got latency off. So let's start from studio and move on to input.
So this has our mapped inputs, Decklink one, and then right there, whatever the edit the tracking. So camera delay, we're going to set that to two. And for this one, the,
the AJA was negative two.
So this one seems to be the same. Alright, so tracking, delay negative two, camera delay two. That seemed to work great and it looks looking great here. Everything matches fantastic.
## Setting the Keyer
Okay, now we're going to go to Keyer. And this is the easiest way I found out how to key go over to color picker and let's make that a little bigger trigger.
Lemme go back to advanced P now. It should look great.
So I think there's a little delay. So let's go back to input tracking delays one. I think the Decklink is much quicker.
All right, let's go back to two.
All right let's key clean up our green screen a little bit.
There we go. Nice and clean. And
that pretty much concludes. How to do this. Make sure save it. I like to save it in the same folder real projects, modular courtyard, and I'll just call this modular courtyard as well.
Save. There you go. Modular courtyard.
Got my GPU running pretty good.
All right.
# Cinematic Compositing
Source: https://docs.lightcraft.pro/tutorials/jetset-cine/cinematic-compositing
Cinematic Composting is a new feature in Jetset that enables direct preview of the live cine camera video tracked, keyed, rendered, and composited completely inside Jetset.
## Video Tutorial
# Transcription
Today we are going to go through our updated calibration process and our new cinematic compositing feature, and this will let us actually composite the cine camera feed in real time at 24, 25 frames per second matched and composited in the iPhone.
This lets you actually see a one-to-one preview of what you are shooting with your cine camera composited and tracked live in the iPhone, so you have complete virtual production process in Jetset.
So let's get started.
## Connections
As you see, we have already connected our iPhone to our Cine camera. We have a live HDMI feed coming out of the cine camera into our Accsoon SeeMo then going into Jetset.
We also have a cooling fan on this, which is very important. And we have our Tentacle Sync and that is going to provide matching time code to both our cine camera and to our Jetset device.
## Timecode Verification
So you can see on the camera we have time code is at 12:06:44. And if we tap on our iPhone panel we can see that it shows the name of our Tentacle Sync that we're connected to, which is Johana.
We can see here that our code is coming in at a 24 frames per second frame rate. And that matches the recording rate at our cine camera in this case. So we want to make sure that your time code frame rate exactly matches the recording frame rate of your cine camera. Okay. So we can tap and close our external connections device.
## Creating Jetset Project
So the first thing we're going to want to do in Jetset, if you haven't done this before, you're going to make a new project. We're just going to click on the gear icon and we're going to go to Project, and we can actually create a new project.
And if you already have a project, don't worry about it. If you have not created a project before, just go ahead and click on the new project and click edit and then we can change the name of our project. We'll call this CineCal.
All right. And we're going to edit and change our prefix. In. The prefix is just a six letter prefix that is appended on the beginning of all the files that is generated by Jetset. There we go. All right, so now we've created our project. I click save. Okay, so now we can see in the bottom of our Jetset view that the CineCal project is currently loaded.
That means all of our calibrations will go into the Cine Cal project.
## Creating New Cine Calibration
So let's go ahead and start our, a new calibration. So we're going to go to recording, and we're going to just go to cine calibration and click new. And it shows up our usual calibration process.
And we can see in the upper left hand corner we have our Jetset video. And the lower left hand corner is the cine video with a Ready sign on it. That means that we actually have live video coming into that, and we can click start.
## Setting Cine Frame Delay
One of the things we'll see in the upper right hand corner is now we have an adjustable delay for the incoming Cine video, and this is so that we can correctly align the time offsets between the cine video and our and our iPhone video.
And as we scroll back, as we pan back and forth, you can see that both the images are pretty well aligned in time. Usually the delay is about two frames. That'll keep us pretty close in time synchronization, which will make it easier to do the calibration.
Okay? We can click okay on the two frame delay. You can adjust it back and forth to, to get your delays correctly aligned, depending on your equipment.
## Finding Feature Matches
All right, next we're going to pick up a camera and we're going to go find a spot that has a lot of feature matches. You don't Tentacle Sync do your calibrations on a green screen because in real time what you're seeing is all the little red squares are doing natural feature matches on the screen. And so a green screen has almost no natural features. So we are actually going to move over to a spot that has lots of natural features in it. And in this case, this something like the back part of a stage works really well.
because this is where all the random equipment is. And in this case, we have a set of storyboards on the on the wall that'll actually work Great. And what natural features are based on. Is a series of corners and the information around those corners. And as you can see here our natural features are detecting pretty well on the corners of all the images on the note board. Okay.
## Feature Distribution
So you can look through here, we can see we have quite a few feature matches, and importantly, they're all, they're showing up through the entire range of the scene. You want to have your feature matches both in the center and on the edges . We're going to be using this to actually do a full lens calibration including lens distortion.
Most of the lens distortion actually happens at the corners of the image. And so you want to make sure you have features at the corners of your image as well as the center. And this is another reason why we generally don't want to do our lens calibrations on green screen because again, there, there's no data in the corners but almost any random back area with a lot of detail, carpets, all these things can work really well.
## Capturing Test Frames
Let's start out we're going to click a test frame. And what that does is it detects all the corresponding features between the the iPhone camera and the cine camera. And it shows how many features. So that's quite a few features. That's plenty.
And it's showing a good detection. It'll draw lines between the top and the bottom of the matching features in a particular area. And that looks great. That's 159 features. That's plenty. So we can click keep frame. And what we're doing here is we're picking an area where we can be a similar distance from the board that we'd actually be using to shoot in an actor, a reasonably similar distance, because that way our focus distances are fairly similar.
So we can actually capture it another frame , and we can range around. Once we've captured a frame, we're going to move maybe a foot to the right and do another frame capture and, and then, hold the camera, still do another frame capture. After each one of the captures we can detect how many good frames we're seeing and make sure we have enough matches between both the iPhone camera and the cine camera --the up down matches as well as the matches between the first cine frame and the last and the most recent cine frame. Those are the matches on the right. So what we're going to want for this is to move in almost a semicircle around the objects that we're capturing and we're doing a little version of photogrammetry here. We're getting as many different points of view of the same set of objects as we can.
## Don't Use a Stationary Tripod For Calibration
You want to move in a semicircle around the object and not stand in place. For lens calibration, what you want is parallax between many shots of the same object as opposed to putting the lens on a tripod and panning it.
That will actually break the mathematics that we use for lens calibration. So what we're doing here is moving around and getting lots of different calibration points from multiple different angles of basically the same objects in the scene, and each time we're holding the camera stationary, triggering a a test frame, making sure we have plenty of captures, and then keep frame.
## Saving Calibration
Once we have 10 or 12 captures then we can actually click save, and we can enter in the calibration name. You're going to want to name this something descriptive. Usually with both a bit of the date and also the lens information. In this case, we're calibrating a 24 millimeter lens. So we have the calibration, the date, and the lens focal length.
There we go. So we can click okay. And then once we have that, now we can click exit.
## Red 2D Calibration Reticule
Now what we see here is that Jetset internally has done some some 2D processing, and it's done a very first order pixel map of the area that the cine lens covers, that 24 millimeter lens covers is shown as a red reticule inside the Jetset view. And again, this is an approximation.
It's not exact yet , but this is a good sanity check because the default iPhone lens is somewhere around 18 or 19 millimeters wide. Whereas we are calibrating a 24 millimeter lens on the cine camera. And you can see that the reticule is actually taking up less space than the iPhone camera.
So this is a good sort of sanity check to make sure things are about what you'd expect. Now the reticule is red because we have not yet solved this in 3D. We don't yet have a full lens calibration. This is just an approximation right now. So we're actually going to switch over to Autoshot now.
## Solving 3D Lens Calibration
Okay, so we've moved over to
Autoshot and we are in the calibration tab in Autoshot. It's automatically detected Jetset on the local network and we can see that our project name is Cine Cal. And we see that the current calibration that is loaded is Jetset Calibration 05-06-2025 at the 24 millimeter lens.
## Sensor Width
So that is the calibration we just completed. Now we're going to want to give it one more piece of information before you do the calibration, and that is the sensor width of the camera. This case, it is a Black Magic pocket cinema camera 6K. So we know that our sensor width is 23.1 millimeters. And when we solve for our focal length, it'll, it give us a more accurate focal length.
## Solving the Calibration
Okay. And we can click go ahead and click calibrate. What it's going to do is going to link to Jetset on the network and it's going to pull over those calibration images that we just captured, and it's going to do the math and find all those natural feature points that showed up as red squares during our calibration process, and it's going to match them.
It's going to match them both between the different cine images that we captured and also the different iPhone images we captured. And by doing this combination, we're going to be able to drive both the offset from our iPhone camera lens to our C camera lens and more specifically to the entry pupil of our cine camera lens, which is the part we need to know in 3D graphics as well as the actual focal length and a first level of radial distortion in the cine camera lens. So it's completed calibrating. It has already pushed the file to Jetset.
## Verifying Focal Length Solve
And you can see that our focal length that we solved for is 24.50 millimeters which is fairly close to the 24 millimeters that's printed on the side of the lens.
And this is a nice way to verify that your sensor width is reasonably correct : if you're solved focal length more or less matches. It's not going to be exact, because what's printed on the side is an approximation; no one wants to buy a 24.50 millimeter lens.
They want to buy a 24. But this is the actual true focal length that lens is calibrating to. Now we can move back to Jetset.
## Jetset Solved Calibration
Now back in Jetset, we can see that our aiming reticule has now turned yellow and that means that it is a correct 3D solve, and is also now dimming the area outside of that.
So this tells us that we have actually solved our lens calibration in 3D. Now this is great, but we still are seeing the reticule approximation. That is an approximation for what the cine camera would view overlaid onto the Jetset UI.
## Enabling Cinematic Compositing
But now we can take this one step further and we can click our our cinematic compositing button. And then we can immediately see the actual cine camera footage composited live in Jetset. And this is a great way to verify that your calibration is working correctly, because now you are seeing the actual cine footage. The cine feed sent live into Jetset, composited matched and tracked, And with a green screen with a green screen, key pulled on it so you can actually see your footage lined up in your 3D world. So this is a one-to-one match of what the cine camera is seeing to what is being composited in Jetset.
## Superdailies
So when we actually record, we will now record what we call a Superdaily, which is a 24 frames per second real time composite, time code matched to your camera original.
## Setting the Origin
So what we're going to do now once we have our calibration, we're going to reset our tracking and we're going to find the floor and let's let's make sure we get a good solid point cloud.
So let's pick the camera up and we'll move around the area just a little bit, and we're going to. Move around and capture our environment from a few different areas. And this is just to get the point cloud. And once we get a reliable point cloud in the system, so we have solid tracking. Now once we've done that then we can actually tap and define our origin.
## Scanning
Okay, so now we've got our origin set. Let's go ahead and go up to our main menu and let's go click scan. All right. Then we can click start our scan. All right. And it's going to map very quickly there, the scanning area and it, we don't need the whole area around it. We just really want the area that's going to be in the shot.
So we can just move a little bit left and right and get a quick capture of the surrounding environment. And there we go. Then we can come back and then we can click stop. And the goal of the Jetset scans is to have just enough scan coverage so that it it covers the shot area.
Okay. There we go. All right. And so then we can see in the scan that's still aligned pretty well. Our origins aligned pretty well, so that'll give us a good aligned scan for our shot. Okay. Now we can click hide, then we can click.
## Setting Keyer
Okay. All right. Now we can move on to keying. So we're just going to go into our main menu and then go to our key tab.
There we go. Okay, and we're already set on the green screen level, so let's go ahead and click the reset button for our key. And then we can click the keyer swatch the green swatch in the middle of the screen. And we can pick a reasonable color of green to operate the key at.
And as you'd click and drag on the screen, it'll update the key in real time. And so if we have, in this case, a something that's strength reflecting a lot of the green light, then it's going to be partially transparent. So we're going to want to boost the opacity. So we're going to bring up our black and white points; there we go.
So we actually harden up the mat. Okay, so now we can toggle off the mat display.
## Adding 3D Mattes
Now that we have set up our origins and done our scan we actually want to be able to set up our 3D garbage mattes. And this will let us pan off the existing 3D green screen and go seamlessly to our digital world.
And because we know where we are in the 3D environment, we can do this pretty easily. So we're going to go up to our main menu and we're going to go pick our key menu.
And now we have a green screen key working, but if we panned off to the left or the right off of our green screen, you immediately see that we've got just the sound stage going on there.
And we actually want to have a more seamless experience. So what we can do then is we're going to go over to our 3D mattes menu on the right, and we're going to click start.
And it's going to detect all the horizontal or surface or vertical plane of this. And it's also going to put in a proxy quad. So you can see where it would be dropping the quad if you hit the plus sign.
So we're going to start off with the back wall. And there we go. And we're just going to hit the plus sign to drop a quad on the back wall. There we go. And you can see it's created a little quad that quad is now stuck to the back wall. We're going to repeat the process for our sidewalls and our floor.
There we go. There's a sidewall and there's another sidewall. Come down to our floor and drop a quad in those locations. Great. Now we can click stop. because we don't want to add them anymore.
## Adjusting 3D Matte Corners
And now we're in a point where we can drag it around and edit it. So in this particular case, we can see that our quads are not going all the way to the end ends of the the green screen area.
So we can actually just drag those into place so that we have a more. More seamless match between the different quads. So again, and we just drag those silver balls around. And as you can see, they are actually staying stuck to the original vertical plane that they were originally on.
So we can just drag the corners to, to match so that we have a seamless match between our green screen, 3D mattes, and it takes a little bit. But this is we can save this for later.
And we just drag those in place for all the different sides of the wall of your green screen. There we go. And let's come down and get our other corner matched in. There's a corner.
And drag up our silver balls, plant up those different areas, have a little bit of overlap so we don't have gaps, and come down and bring it down to the floor.
There we go. And last we can bring in the floor quads into their correct spot. And this is nice because this lets you handle adding a green screen 3D mat. Even after your stage is full of stuff you can just drag the quads around the various objects. I'm just going to walk over here and drag those to a line.
And there's our last floor quad.
All right. Hash do nicely. Great. Now we can click. Okay. And now as we pan off this screen, it is seamlessly transitioning to a fully digital CG representation of the scene. So we can now move to large scale camera moves and we automatically map back and forth between the green screen and the 3D garbage mattes.
# iOS Timeline Processing
Source: https://docs.lightcraft.pro/tutorials/jetset-cine/ios-timeline-processing
Process and manage timeline data directly on iOS within the Jetset Cine workflow.
## Overview
This workflow covers syncing on-set footage, editing a timeline, exporting an OTIO file, importing shots back into Autoshot, running a Quick Render across all shots, and delivering Unreal Engine sequencer import scripts.
### Syncing and Editing
After syncing the Jetset files to the local drive, use Resolve to edit the camera original files. These are designated \*\_zl\_cam.mp4.
### Export OTIO From Resolve
* Optional: name individual shots with timeline markers as described in the previous 'Timeline Processing' video.
* Use File->Export->Timeline. Change the file format to OTIO and save the file.
### OTIO Import into Autoshot
* In Autoshot, choose Shots -> Import From OTIO File
* Navigate to the .otio file exported from Resolve and import
* Set Run Scope to Project to import the whole projects's worth of shots
* Apply handles if desired
* Check Autoname Shots unless you named the shots in Resolve previously
* Use the default Shot Prefix (`Shot_1`) or change it
* Check 'Use video file timecode over OTIO media reference timecode'
* Click *Import*
This will now populate the shots from OTIO into Autoshot.
### Shot Selection and Verification
* You can now cycle through the shots to verify their take arrangements and clip names.
* The Project Folder field is briefly cleared and reset during this phase.
### Configure Autoshot Unreal settings
### Quick Render
* From the Shots menu (or toolbar), select **Quick Render**. The dialog shows:
| Field | Value |
| ----------------------- | --------------------------------------------- |
| Key Color | Green (green square) |
| Scope | Project |
| Shots in scope | Shot\_001, Shot\_002, Shot\_003 (all checked) |
| Day | myproj\_2026-05-01 (all shots) |
| Take | myproj\_SC101\_TK001/TK002/TK003\_A001\_720P |
| USDZ | ChamberBaked\_02 |
| Hide Camera Media Plate | unchecked |
| Copy to Results Folder | unchecked |
| Destination | `D:\tmp\otiounreal` |
| Status | Ready to start |
| Elapsed | 00:00 |
* Click **Select Destination Folder** to set the target folder where the shot tree will be written to.
* Click **Run** to begin processing all three shots sequentially. The Console and progress bar show:
### Unreal Script
* Quick Render will copy Unreal scripts to the named shot directory, e.g. `\Shot_002\3D\UnrealFiles\ImportSequencerScriptShot_002.py`
* It will also write an aggregated paste file: `QuickRenderUnrealPaste.py`
### Output Folder Structure `[05:30–05:50]`
File Explorer is opened at `D:\tmp\otiounreal\`. An example render output structure is:
```
D:\tmp\otiounreal\
├── Shot_001\
│ ├── 3D\
│ │ ├── UnrealFiles\
│ │ └── UnrealRenders\
│ └── Editorial\
├── Shot_002\
├── Shot_003\
├── QuickRenderUnrealCleanup.py
├── QuickRenderUnrealPaste.py (1 KB)
└── QuickRenderUnrealPaste.txt (2 KB)
```
***
### Review Aggregated Shot Folders
The `otiounreal` tmp folder contains shot folders alongside the utility scripts. The File Explorer shows: e.g.
* `Shot_001` — File folder (5/4/2026 10:50 AM)
* `Shot_002` — File folder (5/4/2026 10:50 AM)
* `Shot_003` — File folder (5/4/2026 10:50 AM)
* `QuickRenderUnrealCleanup.py` — JetBrains PyCharm Community File (2 KB)
* `QuickRenderUnrealPaste.py` — JetBrains PyCharm Community File (1 KB)
* `QuickRenderUnrealPaste.txt` — Text Document (2 KB)
***
### Open and Review QuickRenderUnrealPaste.txt
* Open `QuickRenderUnrealPaste.txt` file is opened in Notepad.
* This file contains the auto-generated Unreal Engine Python console commands. Its structure is:
```
# Auto-generated by Autoshot Quick Render.
# The Unreal Editor Python console only accepts ONE line at a time;
# paste each line below individually.
# Optional cleanup (wipe /Game/autoshot for a fresh start):
# py "/QuickRenderUnrealCleanup.py" apply # dry run
# py "/QuickRenderUnrealCleanup.py" delete
```
* Paste this file into the Unreal CMD line to create all of the shots as Level Sequences.
# Shooting Blackmagic BRAW
Source: https://docs.lightcraft.pro/tutorials/jetset-cine/jetset-cine-blackmagic-braw
Jetset Cine software setup, calibration, and shooting with Blackmagic 6K BRAW. Automatic post processing and transfer of shot tracking data into Unreal for post rendering.
## Video Tutorial
# Transcription
We are going to go through a complete Jetset Cine round trip shooting with Blackmagic BRAW. We'll show connection of the Accsoon SeeMo, the connection setup of the Tentacle Sync time code to the main camera and to the Jetset app, the calibration process, and finally shooting and recording a take.
And then post-processing the take and then bringing over the take information including the BRAW data file, extracting proxies from the BRAW file, and then finally processing everything and getting frames. Let's start out.
## Accsoon See App Installation
So before we do anything else, we're going to install the Accsoon See app. Go to the App Store, search, there's the Accsoon See, great, and we can open it. Okay, we can, we can exit that for the, for the time being.
## Connecting Accsoon SeeMo to iPhone
We have our Accsoon SeeMo powered up and connected to the camera with a live HDMI feed. So then if we connect the Accsoon SeeMo to the iPhone. We're going to get this pop up. It says it would like to communicate.
Yes, go ahead and allow it. So that's going to bring up the Accsoon See app. We have currently connected a SeeMo 4K. So I'm going to slide that over. Let's check to make sure our firmware is up to date. Firmware, we'll click there; current firmware update. And it says that our current firmware is the latest.
That's great. Now we can go back to here. Now we can click the monitor button. And this lets us see the live feed from the camera. Right now the camera is pointed at a blank wall. But the live feed is working. So now we know that we have live signal coming into the Accsoon, and that we have the current updated firmware. We're all set for there. We can exit the See app just by swiping up.
## Tentacle Sync App Setup
And next we're going to want to connect our Tentacle. We have it connected to the timecode input on the camera. And we've verified earlier that the camera is reading timecode.
We can click on our Tentacle Sync app here and verify that it's hooked up. It's detected our local device called Sergio. We can click on it here and we can see that it's currently generating time code.
So now we can exit that, that device.
## Project Folder Setup
Now we'll start up our Jetset app, and we want to be in the Cine version.
And as usual, we're going to create our own project, main menu, create a project, new project, scroll down, highlight that, edit and we're going to call this
calibration test
and we'll edit our prefix
there we go.
Then we're going to switch over to Autoshot, make a folder with the same name as the project and I hit refresh and there we go. Our project folders on Jetset and on the local drive are now synchronized.
## Tentacle to Jetset Connection
Next we need to connect to the Tentacle device in Jetset so we can receive timecode both inside the app and to the camera.
So we're going to click our main menu, we're going to click settings, and we're going to click settings here. We can enable Tentacle Bluetooth timecode, as we see here. Okay, done.
And then bring this up, and you should see this, so we have our source, our Tentacle source, and here we can verify that our timecode matches this on the camera. It's all coming from the same source, and that our Tentacle name, Sergio, is correct.
And if that doesn't work, just go ahead and restart the app.
## Cine Calibration Setup
So we're, now we can start our calibration and the trick with the lens calibration is you, you don't actually need it to be on a stage. You don't want to calibrate on a green screen stage because there aren't that many feature points.
And what you're looking for is actually quite a few feature points that can match the natural features.
When we start up, we do need to set our origin. And we're going to pick any point. It doesn't matter.
## Cine Calibration Capture
Once we've set our origin, we can click Recording, I'm going to click Cine Calibration. I'm going to click New Calibration.
Okay, so here we have our calibration UI. And we're going to click Start.
And we can see, simultaneously, we have feed coming from the main camera and from the Cine camera. And in this case, the main camera on the Jetset device is about a 19 mm equivalent focal length. And we have about a 24 mm focal length on the Cine camera. So the Jetset camera is a little bit wider.
The red squares is where Jetset Cine is detecting the natural feature points in the scene. We're going to hold the camera still. While we're looking at this, we're going to click test frame and it's going to find all the matches between those two images.
So I have found quite a few different matches so that we're going to keep that frame. And then we're going to move over just slightly and we're going to keep the same basic object in, in the line of view as we work our way around the same carpet. So we're going to hold ourselves still again.
We're around the same object, around the carpet. 668 points, that's good. I'm going to keep, keep that frame. I'm going to move over another foot. And we're trying to keep these at roughly the same focus distance. I'm going to click a test frame. It's going to detect 597 points. Keep that frame. I'm going to move over yet again.
And on the right, it's keeping track of the number of points that match the original image. Keep that frame. So we're going to click 620 frames. I'm only seeing like 12 points that match the original image. But that should be okay. And we're going to come back here.
Test a frame. There's a lot of point matches. Keep a frame. Okay, so sometimes if it didn't work you just click ignore frame and we're just going to keep working our way around. Test frame, I'm going to keep that frame and move over into a roughly an 180 degree semicircle. Test frame, keep frame.
Alright, so now we're going to click save And we're going to enter in our our calibration in this case we're going to just going to say I'm going to call it the Blackmagic Pocket Cinema, And we were at a 24 millimeter frame
## Exiting Calibration Capture and Red Reticule
Now that we have saved, we're going to click exit and it'll be a black screen for a second.
And once it comes back to Jetset, you're going to actually see an overlay of the aiming reticule. This capture is actually taken from a 70 millimeter lens, so it's a much smaller reticule, and the aiming reticule that shows the equivalent area of the the cine camera is red because we haven't processed it yet.
Right now, we have just captured the 2D images for our calibration, but in our next step, we're going to calibrate, and once it's calibrated, then the aiming reticule will turn yellow instead of red.
## Autoshot Calibration Panel
Okay, so that's now saved. We've done our captures. Let's go to our calibration.
Now we're back to Autoshot and this time we're going to go to the calibration panel and we can see that we can detect our current Jetset device.
## Cine Camera Sensor Width
Very important, is we're going to enter the sensor width. We enter this in here and it's carried downstream into our metadata. In the Blackmagic Pocket Cinema 6K G2 in the 6K BRAW mode, this is 23. 1.
So now we can click calibrate, and it's going to go through, crunch the numbers,
## GUI Check of Calibration
Now if we want to do a GUI check to see how it, how it looks, we can click GUI check, and this opens up and it shows the pattern of where the images were shot. So we can see the cluster of camera positions as I moved around the, the carpet.
And we can actually even see the pattern of the carpet so that looks like a reasonable solve. And we can exit. And we don't want to save it and we really do want to quit.
## Numerical Check of Calibration
And you can see here that our convergence was about three tenths of a pixel, so that looks pretty good.
## Automatic Calibration Push to Jetset
And here we can see calibration pushed to Jetset. So let's go back to Jetset.
## Jetset Cine Setup
Okay. So there is our aiming reticule. Here we're going to set our usual things. We're going to go to our set tab. We're going to do a quick scan. of our environment, which are going to, it'll be really helpful for any post production tracking refinement work, especially our green screen with our markers on it. That's going to be useful.
All right. And we're going to click that to hide it. And we're going to click our green screen, green screen key. All right.
We can set our garbage mats and clear our garbage mats, set them. Add one there, add one there, and rekey. Okay. And we're going to set our mat to green. Okay.
## Framing Digital Slate
So here we have, we're in Autoshot. We can see we're connected to our Jetset device. Jetset is running on that phone. I'm going to click open webpage. There is our digital slate. And it's working.
Now to start to roll a take, we're actually going to come over and frame up on our digital slate. And we're going to make sure that both our cine camera and our Jetset camera can see the slate.
## Rolling Cine Camera then Jetset
To roll on, on the Blackmagic can roll on Jetset. And we're going to see the timing and the synchronization markers go. Now we can shoot.
So now we can move,
adjust. So we can see the framing of our, of our digital set and come up,
come down to our subject. Okay. And we'll come back up, cut. I'm going to set our background to our default. There we go. So we have, we're not seeing the background through that. Okay.
Frame up Cine and the Jetset take at the same time. Roll the Cine, roll Jetset.
Okay. And come back over and do a take and we can come down
and then back up.
All right. Cut the BRAW, cut the Cine. and cut Jetset.
All right, that should do it, and we can go pull some takes.
## Transferring Takes
Okay now that we've recorded the takes we're going to come back to Autoshot, we've got our project folders picked, and we can see our Jetset device, the network, and we're going to synchronize the day's worth of shooting. I'm going to click sync. And it's going to start pulling takes over.
## Setting Cine Source Directory
We need to set our Cine Source file. So let's go ahead and click browse and we can actually just make a another folder right next to here. We'll put a new folder next to the other folders. We'll just call this a BRAW.
Alright, now we need to go get our BRAW files. And they are connected on a USB drive. That's what we shot to directly. And here's our BRAW files. We can grab these and copy and bring them back into our open and hit paste into our Cine source file. So it's going to copy the BRAW files over there.
## Creating Cine Proxies
While we're at it, let's copy this path. And we're going to need to generate proxies. So in order to do that, we can use the Blackmagic proxy generator. This comes along free with Resolve, with an installation of Resolve. And we can go click add to our watch folders.
I'm going to come up here and hit paste. And there's our BRAW folder. So we're going to select that as the watch folder, and then we can click start and it's going to go look at and process our two raw files.
Click stop when it's done and we can exit that.
## Setting Cine Proxy Folder
Now we want to set our Cine Proxy and right now it's disabled because you typically only want that if you are using raw files. So we're going to click scan. and it says down here you will need proxy files for this.
Okay so now that that's that Cine Proxy is enabled let's click browse. And we're going to click proxy because that's put it just below that automatically. Okay.
## Scanning Cine Takes
Now that we've picked our Cine Proxy folder, we can click scan, and it's going to go start scanning through the video files for the different ID markers. It's working through its working through the videos, and you'll note that it's looking at the proxy video files, the mov files, and not the BRAW files.
Okay, so no video found matching current take. Okay it's probably because we don't have a BRAW file for that, so I'm just going to pick ones that we do have a BRAW file for. So here's TakeAAC, going to click scan. There we go.
Okay, so it detected the correct matching file, and it measured the difference between the start times of the two files and found the correct Cine Offset, which is 2.361 seconds, which it automatically entered in here, and that sets our correct timing match between the Jetset and the Cine take.
So let's also try a different take just so you can see what that's going to look like. Pick, we'll pick 177 and it has already chosen that. So as we switch back and forth, you can see our Cine offset changing. It's gone through and detected the different takes.
Okay, so that's how, that's how it can match.
## Manual Take Selection
And if you need to do this, you can actually do a manual pick. You can also just click manual pick, it'll take us to here, and we already know that it's 047, but just in case it didn't detect it, you can click that here, and , it tells us that it's using the proxy video and the Cine video.
## Viewing Cine Matches
One very useful thing is once you have finished a scan, you can quickly check which files were automatically correlated by just going to Cine and then you show Cine Source matches. This will open up a window that shows the matches between each BRAW file and each Jetset file.
So this is scene 101, take 59, and it correlates to this BRAW file. This is especially useful when you start having to deal with a large number of shots coming from editorial, where they may have initially edited with the BRAW footage, or the proxy footage with this name, but you want to associate it with the Jetset takes.
## Creating Level Sequence in Unreal
And finally we can tell it we want to go into Unreal.
So we're going to set an in frame of 200 and out frame of 400. So we have about 200 frames to work with, and let's regenerate our frames, and let's save and run.
Okay, it's created a sequence, we're going to move to Unreal.
## Enabling Unreal Plugins
Let's make sure we have our plugins for Unreal, we've got our plugins we want image plate . Yep, we have that. We want Composure. Yep, we want Python, both the EditorScript plugin and the Sequencer scripting. We want Movie Render Queue, and we want both the Movie Render Queue Additional Render Passes and the Normal Movie Render Queue. And okay, so that should do it.
## Pasting Command Sequence
Okay, and let's hit paste.
There we go. We can double click on the Sequencer. And see here's our video footage being played. And here's the scan data dropped into Unreal.
## Hiding Scan
If we don't want to see the scan right now,\ we can unspawn it for the time being.
Okay, so this is great. We can actually see our image playing through from the beginning of the sequence. Extend this out a little bit. There we go.
And if we want to see how it's going to line up, we can actually do our usual trick of going to our custom color difference keyer. And we're going to make a material instance and the usual bring this over here, move it in here, exactly the same as we did same as we did for the intro Unreal tutorial.
So I won't go over that and this in that much detail, but it's going to set our key color. Okay, great. Because we just basically want to see if things are lining up correctly and we can go back to our sequencer. And, oh yes, highlight our image plate and type in material and let's bring over our new custom color difference keyer there. Great. Next we want to, we want to go to our sequencer and lock our camera image. And this way, we can see how our plate is going to match up with our, our image.
I'm going to come down.
There's our little,
OK, that looks good.
## Adjusting Image Plate Distance
And as before, when we, we ran into the problem of her disappearing, we know that we can actually change our transform. So come to the right, show you what's going on Unlock our hit frame after frame. And what's going on in this part of the shot is that the image plane is intersecting the 3d surface of the road.
So we want her actually closer to camera. So we can go to the transform of our image plate and just click and drag over all the keyframes, delete those, come down to our location, increase our X location a little bit. There she is. Okay, and then let's look back at our image plate. Okay, and let's see how that's, how that's, how that's working. Get in our timeline. There she is.
Okay, so it looks like it's reasonably locked.
Then we can actually just hide the image plate. Come down here, set our details, and remove the search. And we're going to do actor hidden in game.
## Rendering EXR Sequence
And then when we go to our movie render queue, we can actually just go ahead, look at our settings, make sure they're correct. That all looks fine. Accept, and we can render.
# Jetset Cine Rigging
Source: https://docs.lightcraft.pro/tutorials/jetset-cine/jetset-cine-rigging
See how to quickly rig your iPhone and an Accsoon SeeMo to your cine camera to create a complete virtual production system.
## Video Tutorial
# Transcription
Okay, we are going to go through assembling and rigging all the pieces for uh, to connect to Jetset Cine uh, and an iPhone onto an actual Cine camera. And so we'll go through the different rigging pieces in this case for a Blackmagic, a pocket Cinema camera. And we'll start up.
## Cine Camera Cage
Okay, so let's start off with assembling the small rig cage onto the, onto the Blackmagic device. So we can just point our camera straight up, there
and it's a bit of a tight fit, okay. So then, once the cage is, is on, we have three screw, three screw points that we can Uh, to connect it with. There are two screw points at the bottom, at the base. Screw
those tight. And I'll get them mostly tight, but not completely tight. And we're going to flip it over. And there's a third screw mount on the top.
Okay. Once all three of those are reasonably tight, we can go ahead and tighten them completely. Oop. Don't go too hard, bebecause it's aluminum.
Okay. So our, our cage is on I'm gonna put this over to the side.
## iPhone Cage
Next we're going to assemble the phone cage to the rail blocks and the phone cage is set up so that has, has a direction so that the phone can fit into it correctly. So we'll mount the phone into this. It only has really one direction it's supposed to go into.
Then we can close it. And then, since We want the phone to be mounted, so that the lens side is down is down low close to the close to the Cine lens, that means we're going to put the rail blocks on the bottom of this and here's a set of our of the rail blocks This is a 15 millimeter rail block. All right, and uh, so then
There's a set of quarter 20 holes on the top or on the bottom of this that we're going to be mounting that to and if we just directly place these the holes will line up. Now the problem is if we screw them on in all the way the screws are just slightly too long for the depth of the the threaded holes that they're going into and so this will wiggle. So in order to remove the wiggle, we're going to add in two quarter 20 washers Just generic quarter inch washers between the threaded hole and the rail mount.
So now we can tighten those.
Go ahead and tighten it correctly. With all of these, we want to keep the connections fairly tight so that it won't wiggle. Okay. So there we have a phone. And so we have the mounting point of the phone low and close to the rail block, as close to the rail block mounts as we can get. Okay, great. So I'm gonna put that aside.
## Rails on Camera Cage
Next we're going to bring the camera back, we can put the rail block directly on top of the camera. As you can see, the two screws will line up correctly with two mounts on the top of the camera. So we can move that there and directly tighten those
and tighten it correctly.
And this one, we don't need the washers because it's mounting rigidly. Okay. Next step, we're going to attach the two rails. One here, one here, and these are carbon rails, so they're very lightweight. And then once they're in place, we can tension, we can tighten the knob, this knob.
And these are spring loaded ratchets, so these can be adjusted later on, you can pull them out and adjust the, adjust the tension on it so we can, we can move those where they need to be. Okay, so now we've got our rail block mounts, and now we can actually, come back out a little bit.
## Mounting iPhone to Rails
Now we can actually go ahead and mount our Cine, our iPhone directly onto the rails.
We can adjust our spring tension clamp, slide them on, and we're usually going to have this located as close as we can to the front element of the lens. Typically the entry pupil of a lens is somewhere very, very close to the, to the front front element on a wide end, wide angle lens.
And so our lens calibration system can account for misalignments, but in order to keep the preview through the aiming reticule on the Jetset Cine display as close as possible to the camera then we want to keep these, keep these fairly close. So once that's in place, we can, we can tighten the clamps.
Now we have the phone mounted on the camera.
## Accsoon SeeMo
And the next thing we're going to want to mount on is an Accsoon SeeMo. In this case we we have the, the SeeMo 4k. This is an HDMI camera. And so the SeeMo 4k can take an HDMI input. And so there's a couple of very useful things. One is the SeeMo comes with a cold shoe mount. So we're just going to mount that right into a cold shoe mount on On the camera.
## MagSafe Cooler
Okay. So now we have our CMO rigged onto the camera. The next thing we're going to want to do is a very important thing, which is a cooler. So the cooler is key to all of this. So this is a Razer cooler, we also have the Black Shark cooler on the website, and the cooler is what keeps the phone cool.
The iPhone can work very, very well in production for many hours at a time, but you have to keep it cool, and the cooler is the way to do that. And the cooler has an integrated uSB C 5V connector on it. So we're just going to MagSafe it. So the cooler just connects onto the the back of the iPhone via the MagSafe connectors.
Once that's connected, we can connect a power connector from the USB C port on the on the phone or on the mag, on the cooler back to a 5 volt output port on the Accsoon. And now we have, now we have cooling. Actually, we can flip it this way. Clean up the cables a little bit.
## Video Cable to iPhone
Alright next we're going to need to connect the actual video output from the Accsoon to the iPhone.
And this is done with a special, a special connector cable. Connects from video out over to the iPhone. And we'll flip that around a little bit to clear the battery slot.
## NP-F Battery
We're going to mount a battery. This is a standard NPF battery to the Accsoon.
## HDMI to SeeMo
And finally, we're going to connect an HDMI cable from the Axon, from the SeeMo.
To the input of the Blackmagic. So the Blackmagic camera has an HT MI port cover here. Now actually we will open up a couple of the different ports 'because we're going to use them. We'll open up the HDMI, we're gonna open up the microphone an audio port jack 'because we're going to use that for time code.
And we're going to open up the USB cable. So we're gonna, we're going to open up all three of those.
Okay, and so we're going to connect the, an HDMI cable, a short HDMI cable from, from the Axiom to the HDMI port of the Blackmagic. Okay, so, that is, that handles the connections for the CMO.
## Tentacle Sync
But there's an, an additional piece we're going to need to do, that's very important, and that is the Tentacle Sync.
The Tentacle Sync is a nifty device that we can both transmit time code into the Blackmagic so that we can synchronize all of our time code together, and it can connect to, to Jetset so that our tracking data will have the same embedded time code or almost the same embedded time code as the Cine camera.
Jetset records internally at 30 frames per second. And the Cine camera is usually set to 24 or 25. But other than that, it's actually a quite close match. And this will help us enormously for both the live render preview and for automatic take matching and post production. So this is a highly recommended little device.
So we'll connect the mounting point here.
We'll clamp,
make sure we clear the on off switch.
Okay. And then finally
we can connect our, the. timecode connector from here down into our timecode port on the Blackmagic device. There's a little microphone jack that goes in there. Okay. And that sets up our timecode.
## External Hard Drive
Lastly, we can, we're going to use an external hard drive to store our B RAW takes and connect USB C
over to our USB C port Internally.
## Set Up Timecode
Okay. So there we have a rigged camera. So let's go and do some setup to make sure things are working correctly. I'm going to turn on the Blackmagic camera.
Okay. I'm going to turn on the SEMO
and I'm going to turn on the Tentacle.
The Tentacle will start out blinking red which means that it is acquiring time code. We actually want it to send its own time code.
And we'll go into this in more detail, but I'm going to use the Tentacle app And we can see the current time code. We're gonna double, we're gonna tap on the time code that lets us either set it to a specific time, 10 o'clock, for example. We can see the Cine camera reflect the time code coming in. We're actually gonna change that back to time of date.
Phone time. All right. So now we can see that it's 1353 here and 1353 on the camera.
So, great. So now we know that our, our Cine camera is receiving the time code from the Tentacle correctly and then we can move on.
# Named Shots in Autoshot
Source: https://docs.lightcraft.pro/tutorials/jetset-cine/named-shots-workflow
This covers Autoshot’s new named shot workflow, so that multiple shots can be extracted from one take without overwriting files.
## Video Tutorial
# Transcription
Okay in this tutorial we're going to go over a new feature in Autoshot which is named shots. This is found under the file menu, under shots, add shot, delete shot.
## Shot Overwrite Problems
Originally, when you pick out your day and your take set your clip in and out, hit save and run. And then we can, click open and to see where it's generated the files. Over here you can see it has made a Cinecam EXR, it's made an EXR sequence and likewise under the AI mattes, it's made an AI matte sequence.
But, now, what happens if you have more than one shot that's coming from the same clip, and you punch in different numbers here and and tell it to run. The problem is it's going to overwrite those original files that were in this CineCam EXR and AI mattes directory. You don't want that to happen.
## Adding a Shot
So we actually now have a named shot technique. We're going to go to file and shots, we're going to call add shot, and we're going to call this SHOT001 .
And we'll hit enter. You can see it's added a shot here, right? So now we can go back to the default for the take, or we can actually go to SHOT001.
So we can say, okay, this one, we want 1200 through 1250. And then we can actually just save and run and it'll go through.
And it's going to crunch out the files for that, and then this part of Autoshot is unchanged. It's gonna do the frame extractions from our cam original, and go find the 50 frames from that. There we go. Runs Blender.
Alright, and there it's created our Blender file with this particular set of the sequence.
## Directory Suffix Name
Hit save, exit that, and let's take a look at what it did under the hood. So we're going to open up, once again, next to our take, we're going to click open up and we'll open a window right into the sequence.
And we can see that it made a new set of directories, CineCAM EXR, underscore SHOT001.
And if we look in here, we can see we've got 50 of these files. And once again, over in AI Mattes. It created a folder for that. And this way, we can actually, if we wanted to make yet another shot, and we're gonna go say, 1400 through 1450.
We'll go File, Shots, Add Shot, and we'll call this SHOT002. And let's bring this 1400 through 14. 1450. All right. And save and Run. It's gonna run the process and the sequence once again, pull out 50 frames.
All right. Here's a different part of the clip. We'll hit save. And so now you can see clip in frame for 1400 through 1450, that's SHOT002. We go back to SHOT001 we're at 1200 to 1250. And then we go back to the default take where we originally had our original numbers.
But in general, when you have multiple shots from a given take, and if you have started wanting to name your shots, so you can keep track of them, this makes for a really good technique for it. And again we can open up our file and we can see CineCam EXR underscore SHOT002, uh, actually, And there's our cine camera files. So we're using the same the same Autoshot, file generation system, etc. It's just now we can keep track of multiple shots per take.
## Application Files Created with Shot Suffix
And also note that in addition to creating a separate CineCAM EXR SHOT002, when we go to our Blender files, this is where our Blender files are stored, you can see that we now have created a separate blender Composite file with the same prefix, so we know which which take it's from.
But now it has a suffix of SHOT001 or SHOT002, and this holds true for all the different kinds of files that we would create in the system. So that simple suffix can actually help keep all your files organized so you don't have any files overwriting each other.
## Deleting Shots
Of course the other side of being able to create a shot is we can actually delete that shot. So if we have shot 002 selected we can go back and go file, shots, delete shot, then we're left with just the default and SHOT001.
And when we delete a shot it's only removing the reference to the shot in in Autoshot. It does not actually delete the underlying files.
So if we click open to look at our folder, we can see we still have the SHOT002 files here the XR files . So you can, manually delete those if you want. We just are actually removing the reference inside Autoshot to that shot. We don't delete any files beyond the reference.
So that's helpful when you're dealing with a larger timeline.
# Setting Cine Offset Manually
Source: https://docs.lightcraft.pro/tutorials/jetset-cine/setting-cine-offset-manually
If the automatic timing marker detection fails, it’s straightforward to calculate the Cine offset value by aligning both clips in a timeline.
## Video Tutorial
# Transcription
This tutorial will go over how to manually set the Cine offset if for some reason the automatic marker detection didn't work.
## Create Timeline at Cine Frame Rate
What we're going to do first is we're going to just go into Resolve and I've already loaded the Cine file and the Jetset Camera original take.
What we're first going to do is going to right click on the Cine file, and that one's at 23. 976 frames per second. So we're going to right click and create a new timeline using the selected clip, and we're going to double check to make sure that our format is actually the correct frame rate of the Cine clip, 23. 976 frames per second. The timeline resolution doesn't matter so much, so just click create. And we're going to go to that timeline. And there's our, our cine clip.
## Set Cine Marker
And we're going to scroll along until we see the very first frame that the very first fiducial marker is shown on.
There's the flash, and I'm hitting the right arrow key to go forward one frame at a time. There's the first marker. I'm going to select the clip, hit M to make a marker.
## Set Jetset Marker
And now I'm going to drag in the iPhone clip, and I'm going to do the same thing. I'm going to drag through until I see the very, very first frame that the fiducial marker ID shows up.
There we go. I'm going to select this clip, hit M for a marker.
## Aligning Clips
Now I'm going to drag one clip on top of the other until the two markers line up. Then I'm going to set the, the play head to the beginning of the first clip, and this tells me how far offset in time the two clips are.
## Checking Time Alignment
Once these are aligned, we can actually see how that they align through time.
So if I go to this clip and I set the opacity down to say 50%. You can see that they, one of them is a different frame rate, but they are actually both aligned in time, because they're both on the same timeline frame rate and the 30 frames a second Jetset clip is automatically adjusted to, to match the 23. 976 Cine clip. Okay.
## Calculating Frame Offset
So now we're going to put the play head at the beginning of that, and now we're going to measure the timing offset.
We can see here that we are one second and 20 frames in before we encounter the beginning of the clip. So we need to first find out how many frames total that is. Since we're 23. 98 frames, that is 23. 98 plus 20. All right. That tells us how many total frames we're at. And so that's 43. 98 frames. Now we're going to divide that by our overall frame rate, which is so 43. 98 divided by 23. 98. And we're going to calculate that real quickly. That works out to be 1. 834.
## Entering Offset in Autoshot
And then we can actually go back to Autoshot and enter that right here, 1.834. And then we can save and run. And that'll be very close to the same calculations that we got from the automatic marker detection.
# Matching Takes with Timecode
Source: https://docs.lightcraft.pro/tutorials/jetset-cine/timecode-take-matching
Autoshot’s new timecode-based matching massively speeds up the process of matching cine camera video files to Jetset takes, and makes the digital slate optional. This is a great addition for time-crunched projects.
## Video Tutorial
# Transcription
Okay, in this tutorial, we are going to go over our new timecode based take matching and sync method, and this is a new method for automatic take matching and sync based on matching timecode between the Jetset data and the Cine data.
## Tentacle Sync Device Required
And this requires the use of the Tentacle Sync E device so that the embedded timecode in the cine camera footage files matches the embedded timecode in the Jetset take.
## Replaces Digital Slate Markers
This replaces the use of the flashing optical markers in the digital slate, so it can be much faster than optically scanning through those files, looking for those markers, and also more reliable, because the digital slate is prone to having lights flash on it or reflections, etc., and the timecode is just a, a very reliable method. The digital slate is still a good idea to use as a backup visual reference, but in productions where you can't use it, then this can replace it.
## Autoshot Directory Setup
Let's take a look at this. We've uh, opened up one of Alden's shots. I've already loaded this into Autoshot. We already have our project folder. I've set our CineSource directory to the directory with the BRAW files in it, and since this is a BRAW project, we also use the the proxy videos generated from the BRAW files. And we've pointed the CineProxy toward these files just the normal way we process BRAW files. We've already picked our day and our take that we're using for this.
## Scanning Using Timecode
And you can see here that we don't yet have our Cine footage matched. We don't have our Cine time offset yet. We're going to find that out from doing this process. First of all, let's click Scan. And now when we click Scan Autoshot will pop up this little box.
You can tell it to use the old method of visual markers instead of timecode. But we actually don't want to do this. We're going to match with timecode.
And you can also tell it to ignore the cache. So when we, we click scan it'll actually store all the file matches in a json file, and if we want to get rid of that file, we can ignore cache. But we're just going from scratch here, so we're just going to use it as is.
We're going to click scan, and it's going to look for timecode, and bam! Already we found a matching cine video, And in that, quickly, we found the CineTimeOffset. So that was, you know, a second versus a long time scanning through the files optically, which is what the old method was.
## Limitations of Timecode Matching
So it's very, very fast. Now, the timecode based matching is really fast, but it has limitations. And the limitation is that timecode is frequently not completely frame accurate. It can be off a frame or two in one direction or the other depending on your camera model and how the timecode was embedded.
So, it's not precise enough by itself to be able to do the level of really accurate time matching that we need to for tracking data. Tracking data has to be at least frame matched, and sometimes sub frame matched.
## Optical Flow Refinement
So the great thing here is that we've added a secondary optimization which is optical flow.
And the optical flow will actually get us all the way to the final alignment. The optical flow requires an initial estimate to work correctly. It also requires a decent amount of camera motion in the image, so if you have a stationary image, the optical flow isn't going to work. But then you're not that worried about synchronizing the motion anyway. So in any case, this combination can work great. So we have our Cine Time Offset, our initial offset from the timecode.
We're going to click refine offset and it's going to say wait for the graph . We'll wait a couple seconds for the graph and There we go Use suggested offset of 5. 285. That's fine.
5. 308 was the initial Cine time offset between the start of the Cine clip versus the start of the Jetset clip. Remember we typically roll the Cine camera and then Jetset separately.
So in this case, it was a 5, 5. 3 second offset and this was the timecode based offset, and then we're going to refine it to 5.285 seconds using the optical flow offset. So we're going to click yes, and if you want to see the little meter we can now see a cross correlation of optical flow shifts so we can actually see how it's working out.
And I'm going to exit that.
## Picking Clip In/Clip Out Frames
Okay, so that's our offset. In Resolve we took a look at this clip and found an in point that we want at about 2250 frames and an out point at 2350. 23 2390. So we're just going to look at this portion of the clip for the shot.
So now we can go back to Autoshot and I entered in those numbers. And since this is processing BRAW, we don't need to set our gamma and color. It'll do this automatically for us.
## Setting Scene Blend File and Scene Locator
I set our scene blend file. Since we have a scene blend file, we'll just append that to the scene in our newly generated Blender file. And since we picked that Blender file, it's going to open up the the blend file in the background and give us our possible scene locators to use.
In this case, since the take had the origin, we're just going to use our origin in the file, and we can click save and run.
It'll extract the EXR files from the BRAW and generate a Blender file.
## Rendering Test Shot in Blender
Alright, there's our generated file. And it's going to compile the shaders in Blender for a second so we can see the actual image sequence.
There is, there is our shot. I'm going to scroll back. The beginning of our shot. And so we can actually hit play and see how it tracks through the shot.
And since the the playback performance is a little bit slow, cause these are 6K files, you can generally see the tracking is aligning, aligning well.
We can check by hitting N, going to Autoshot. And telling it we want to do a fast viewport video render, so that'll just go through and do a fast render of the viewport and compress it into an mp4 so we can check the tracking behavior.
## Viewing Fast Viewport Clip
If we want to see that clip, we just go to Autoshot, click open, and it opens it directly to our sequence directory, go to preview, and there's our preview render. We can double click that and kind of see him going walking through the scene. There we go.
Alright, and that looks like the synchronization is looking correct. We'll switch this back to 3D viewport.
Okay, so that shows us our tracking is correctly aligned in the scene in time. Now you can see it's a new way of automatically synchronizing the Jetset takes with Cine takes using embedded timecode from the Tentacle sync.
And this is far faster, and I think will be very, very useful for productions that don't always have time for the digital slate. And they have a very large number of takes to process through, since it's so much faster to match takes with timecode instead of going through the process with the optical targets.
Alright, so I hope that helps out.
# Timeline Processing
Source: https://docs.lightcraft.pro/tutorials/jetset-cine/timeline-processing
Autoshot can now batch process shots using an OpenTimelineIO file exported from most major nonlinear editors. Export named shots into VFX industry-standard folder and file naming structures, and batch process dozens of slap comps in an hour.
## Video Tutorial
# Transcription
In this tutorial, we are going to go through Autoshot's new timeline processing functions. We can now name shots in your non-linear editor via timeline markers, and we can then export an OpenTimelineIO in this case from Resolve. But Avid now has this as well.
Now we can actually batch process large numbers of shots at a time, and we can include automatic slap comp creation, automatic set linking, library overrides, we'll go through defining a shot structure, and be able to process dozens of shots in a short period of time.
We are going to be working with the shots that we did with the ETC Pathways project.
Make sure to check that out. I'll put that in a link below. It's a great project showing the future of blending traditional production with AI assisted background work. So let's get started.
## Defining Shot Names Via Timeline Markers
So as we come into our timeline, you can see there's the camera original file name there. We're going to set the playhead at the beginning of the clip, and click add marker. Now we're going to enter the shot name into the name or into the notes category of the marker.
In this case, it's going to be the PTH prefix, followed by the sequence number, followed by the shot number. So we use the first three tokens in the notes to name the shot. The ones after that are ignored. In this case we're gonna add a couple things that are useful for information, but those will not be used for the shot naming.
Okay. And then once we have done that, we also need to add in the clip name into the actual marker name. So in this case we are copying the cam, original file name BO oh four et cetera CO 18 into the name of the marker. And this is what will let us correlate the file name of the actual camera original to that particular shot name.
Here we're going to go through and export an OpenTimelineIO file. We'll highlight the timeline we want to export, and then come down and click OpenTimelineIO OTIO. Great. And then we can actually export an OTIO file and then we can read that file into Autoshot.
## Clearing Shots
And in Autoshot. In this particular case, we are going to be parsing multiple directories since frequently a shot will have multiple days of takes. So we'll show you how to go through and do that. We want you to start off with a clean slate. So we are going to go up to shots and clear and we'll just go project wide. Okay. It's going to clear up all of our existing shots.
Okay, great. So I'll clear that out.
## Scanning Cine Source Directories
Next, we're going to want to scan all the different cine source directories by day. We have two days in this shoot, so we're going to look at four 18 and we're going to browse and pick. Our Pathways, and here's our first day of shooting four 18, and there's the media directory. And again it can search under these directories. Like these are all the sub directories for the camera originals that were on this.
But we're going to pick the top level directory so it can scan underneath that. And we're going to click scan and we're going to go by day, because we just want to scan that day's takes and ignore our cache just to make sure we're going from a clean slate click scan and you can see how quickly it's going through.
In this case, we're doing time code based matching between the Jetset takes and the camera originals. We had a Tentacle Sync connected to both the Jetset iPhone and to the Cine camera we were using on the shoot. So the time code matches and it makes for very rapid matching between the Jetset takes and the Cine camera takes.
Okay, so we found all the matches there. Let's go to day four 19 again. We're going to go up and browse and let's go Pathways four 19. Pick our media folder. Again, we're going to scan and again, pick the day and let's ignore our cache. There we go, and it's going to go through and scan all those files and look for timecode, correlations, and matches.
Now it's going through and matching them to all the corresponding Jetset takes. And again, this is a lot quicker than having it doing the optical scan of the flashing marker codes. They can both work; it's just that the time code based matching is really fast.
## Importing OpenTimelineIO File
Okay, once we have that, we're going to import our OpenTimelineIO. We're going to going to go to shots and import from OpenTimelineIO. We're going to browse and pick our OTIO file this case in timelines. We're going to pick this Pathways OpenTimelineIO. And a couple things we want to be careful of.
We want to actually go for project wide in our import and we want to add a 10 frame header, a 10 frame handle to each side of the imported shot. This is very commonly done in visual effects to give each shot a few extra frames on the beginning and the end. So we're going to add the handles and we're going to click import.
Okay, so now we are going through, it's loading 86 shots worth of information and now it's going through and it's creating these shots. So it goes through and finds the takes information for each shot that we need to know and creates the shot in in Autoshot and updates the user interface and see here now we can actually see that if we go to a take that has an associated shot.
I think scene one, take five has an associated shot. Yeah, there we go. We can see that the shot appears and some of them have multiple shots. On this particular case, we have Pathways 0 0 1 0 20, so sequence one and shot 20. And as we go through some of the others, not every take has a shot. They didn't use every take, but the ones that do have that again, scene one, take five. There we go. There's our shot name. Okay.
## Quick Render
So now that we have the shots, we don't actually want to go through those one by one. So we have a new feature that lets us process these shots in batch and also make slap comps and copy the shots into a visual effects friendly folder structure.
And that is called quick render. So we're going to bring this up and in quick render it's going to default to a view where it just shows the individual shot from that take.
## Viewing Project Shots
And let's actually take, go take a look at the project view so we can see all the shots in the project.
So we can see a whole ton of shots here. And this is the list of shots that were in, in that project from beginning to end. And if you're trying to import these into Shotgun, it turns out that we also create a Shotgun folder from this. If you go down to our timelines then we have actually created a Shotgun csv file that you can load into Shotgun.
We're not going to be doing this at this point, but I'll just for reference. So here's our entire list of the Jetset shots in that, that we're going to process and it, for each one it shows you, for, say, Pathways oh 1 0 20. It shows you which day it was shot on and the take, and it shows you which USDZ file was used.
## Picking First Shot in Sequence
And in this case, the library. Library, no front wall. And that is important information because we want to configure one of the early shots in this sequence, the way we want the rest of them to be processed. And shot O 10.
I, I actually know from experience we can't see very much in the shot. We'll actually start with a 1 0 20 and let's actually just go with the take. All right. In 1 0 20 we are going to want to pick the day, the 18th, and the scene one take oh five, which is actually the one we already picked.
But in case you have to look around for which one it is, this is how you find it. Okay, so scene one, take oh five. So let's close that for now.
## Configuring First Shot
And we want to configure all these pieces, so that when we do the batch processing, it's correct. So we're going to pick the scene, blend file, and in this case uh, for the library because as we can see, our library, no front wall. We actually want the library Blender file we're going to use.
In this case, the library pathway is final, so we're going to pick that . Then, it will automatically match the scene locator that we used in the the shot, the scene loc center to the scene locator in the Pathways file. okay.
## Linking to Scene Blend File
And we're going to want to link scenes in this case.
And we're doing this for very specific reason, because that library final file is actually quite large. And if we just copy the contents into each individual shot file, that'd be a huge waste of space and we wouldn't have a way of updating the top level library file. We're actually going to link the scene.
And that means, number one, that our individual Blender shot files will be quite small because they're linking most of the data. And two, that when you update the library's Pathways final Blender file, all those changes will automatically ripple through into your individual shots. And this is a real key piece of production.
You want to link your top level scenes into your shot files so you can easily update these things.
## Configuring Frame Pull Color Space
We can, we'll leave this as extracted EXRs in the original format. For the gamma and color LUT, this was shot in Apple Log.
So we're going to set our gamma and color LUT to Apple, and we're going to want to bring these as always to ACEScg EXRs.
We're not going to be using an AI roto model on these. So we won't worry about that.
## Configuring Camera Plane
We do actually in these, want to set our depth. By default our camera image plane is set using the onset LiDAR that can result in a camera image plane that can move back and forth quite a bit. And since these shots frequently are intersecting with the live action scenery, it's easier to just set the depth to just to a meter distance for all of them. And then we can adjust from there. Okay, so once we've got the basic setup for this initial shot, we can actually go up to shots, quick render, and let's try processing it. So we'll just pick our take.
## Configuring Shot Directory Structure
We are going to tell it to copy to the the results folder in this case. I'll show you where that's at. I've created a new folder called Shots. We'll pick that folder and we can also configure how we are going to create our shot folder structure here in settings. And here you can see how it's going to be naming this and what the hierarchy of files this is going to be.
And this can change from production to production. This particular case, you can see our shot name and we can see that we're going to put in our Blender file into the 3D subdirectory under Blender files, and then our render output will go under the Blender render.
Likewise, our camera shot is going to go into editorial plate. The proxy will be an editorial proxy. The view port render will go into Blender, render slap. And so we'll be making slap comps there, and then we're going to be generating a set of
sRGB proxies for use in Blender, and this is where those will go. So again, you can configure these for your own project. This is fine for what we're doing now.
## Rendering First Shot
All right, let's go ahead and let's try rendering this. We're just going to click run and it's going to go through the process and extract the files from the original Apple log files and generate both EXR file sequence and also generate our proxy JPEG sRGB sequence for use in Blender, and it'll go through that reasonably quickly.
And here it's going to actually start rendering. You can see it's going through rendering all the Blender frames. Once it's created, the Blender file, it is actually automatically batch rendering the slap comp using the image on the internal image plane, and a built-in keyer we have in Blender.
## Checking Blender Render File
Okay, so it's created a Blender render file. And if you just start selecting one file as we did, it'll pop up in the Blender file automatically. And in this case we can actually see that we're mostly occluded here.
## Viewing Slap Comp
You can actually very quickly check to see how the files is coming out.
If you look in our, here's our shots directory, and then we're going to look at Pathways oh 1 0 20, and we can look at our Blender render, and there's our slap comp as we saw before. And then we can double click our slap comp and take a quick look and I can see our 3D bookshelf is intersecting and also we can see that our key isn't quite right. So the bookshelf we'll handle it in a slightly later area when we do library overrides. Let's fix the key because we're going to be setting this keyer color for the next 20 shots.
So we want to get that about right. So we'll exit from here and I'm going to exit from here right now as well.
## Viewing Source Proxy
Okay, so let's go down to our created proxy.
We can see that the directory has all of the sRGB JPEGs that we created for this, which is what Blender will read as an image sequence for its internal rendering. And we can also see that we added those together into an MP four file. So we can double click that.
This is just the live action version, the proxy live action version of the shot, so you can actually see what's going to be in the shot. Okay. As we're moving. Great. So we set the key and it was too light, so it's a little bit darker here and here. So we're going to do a color pick of the key here, it's considerably darker.
It's going to look down here to see if there's any much darker spots in the scene. And down here is actually a little bit darker. Okay. So I'm going to move that over here.
## Picking Keyer Color
I'm going to briefly move my Autoshot window back out of the way. And so we're actually going to pick our color here. So let's go ahead and under our keyer settings, let's do pick from screen and we're going to come over here and we're going to pick this darker area.
There we go. And our keyer color going to click. Okay. And let's go through and process that. I'm going to just run the same thing again. I'll exit out of our. This piece of it, and it's just going to do the same thing we did before, and it's going to go through and just render the entire slap comp but now with our updated keyer settings.
## Checking Slap Comp
Okay. And again, instead of looking at the Blender file, let us look at our slap comp under 3D. And we're getting, still getting a little bit of dirty key, but better than it was.
Okay, that's better. And we can do some more per shot cleanups if we want to do that. But let's go ahead and start the batch processing.
## Selecting All Shots by USDZ
We're going to come back to here and instead of going from our take level, we're going to actually go, we're going to pick same USDZ. And because we have set up our configuration to have our library, we want to have all the same USDZ files that use the library no front wall usdz from the same library Pathways project. So we actually want to process all those at the same time.
And so by picking same USDZ, now we're loading all those from Pathway Series one shot 10 all the way down to series six shot 10. And again, we're going to use the same pieces here.
## Batch Processing Shots
So we're just going to click run and we can see we're going to start processing a lot of shots.
And it's going to start off with the very first shot and it's going to do all the extractions. It's going to basically do all the pieces we did for the shot that we just processed, except now it's going to process all those shots and generate all those Blender files and all those slap comps.
And then we can go through those and do any fixes that we need to do on a per shot basis.
Okay, so we can see that it's finished processing a few shots. So let's go take a look at the first four shots in the pipeline.
I'm going to come over here and here's our shots. Here's Pathways. Oh one. And let's look at our slap comp. All and it looks like the camera is in a is behind the CG here, so we're going to probably have to look at that. Let's take a look at our Path two. Again, we're going to look at the slap comp and this is just a great way we can go very quickly to see if things that we're going to need to fix in these.
There's our shot. We're going to need to fix how the intersection of the live action of CG works because we have CG bleeding through the live action, and we'll do some key fixing. All right? That's fine. But the basic shot looks like it's working. And let's go back through Pathways is 30 and once again, we're just going to look at our slap com.
And sometimes you get lucky and all your slap comps just work. Other times we're going to have to go through and correct a bunch of these things. So here are the, okay. You're going to have to fix the gear and fix a couple other things. All right. That's fine.
## Correcting Slap Comps
Let's go back to the first one and let's just start going through and showing how to fix stuff.
We'll just close this. And it has created our 3D files and our Blender files, it has created a Blender shot file, which is named after the the same shot that we set up back up in the non-linear editor.
Pathways sequence one shot 10. So let's go ahead and open that up.
And we're working with Blender 4.4.
Okay. So we can play through our timeline and clearly the camera, we'll just, swing our swing on around our camera.
Let's go highlight our image plane. Our image plane is clearly in, in the middle of the bookshelf. All right, that's fine.
## Setting Library Overrides
So we're going to want to remove a couple of objects in the way, and we're going to start seeing how we do that.
And here we can see how our scene is set up. It has linked to all of the collections that were in our original library file. And these show up as linked sections over here. You can see it by the little link file, the little link icon over here.
These are collection instances; we may actually move to straight link collections, so this may look slightly different. But the key item here is you can see the link icon and that tells you there is a linked collection. So let's take a look at the one we're going to have to hide. If we just highlight the bookshelf and hit a hide the entire, then all the bookshelves go away.
That's not what we want. . Let's unhide those. And what we actually want to do is only only hide a section of that bookshelf. How do we do that? Since it's a linked collection, we're going to actually click on this and our bookshelves and right click on that, and we're going to do a library override and create that selected library override.
And now all of a sudden we can actually pick individual objects in that bookshelf so we can pick, and actually it turns out the books are going to be something else we have to do this with. So we're going to right click on the books and we'll do a library override and we'll make selected.
## Hiding Individual Objects
And then now we can select individual items and books and we can hide them, right?
Let's go through and here's our camera frustrum, so we can get a good idea of the things that are in the way. Hide this. And I think we still have to hide part of the hard general structure. Okay? Right click. Library override. Make selected. And then we can hide that wall and the rest of the wall is still there.
Now our camera has a clear field of view. Let's hide this little fellow as well. Okay. And now we can actually look directly at our camera . This area over here was actually in the original frame. It's supposed to be looking at an object from inside a bookshelf, but we just want to use the real bookshelf that was in the actual shot.
So there we go. So we can actually see our shot come together. All right, let's go ahead and and save our file.
## Re-Rendering Slap Comp
Then we can actually go hit N and we're going to go to Autoshot. And let's actually do a fast Viewport render. So before we do that, let's take a look at our Blender render file.
By default the output is set to a directory just below the current Blender file directory called Blender Render. And it will generate a Blender render file as an EXR. And we don't actually want to make EXRs for our fast slap comp, that's excessive. Fortunately, when we click Fast Viewport video, we can see this shows up and it has changed this to Blender Render now into the slap comp directory.
And it is making a slap comp and directly writing it to the MP four file format, which is exactly what we want for a slap comp. So it's going through the process of writing, rendering, all that. Alright, so that's done and let's go and look at our finished slap comp. So we're going to go back up to Pathways and in 3D we're going to look at Blender Render.
And I am going to look at our slap comp and let's check it out again. And before all we saw was a library wall. Now we see it's the correct shot and it's still named correctly. It's in the correct location with the correct naming. Okay, so let's let's hit save. Let's exit that one. So there is one slap comp done with our shot file updated. Let's go to the next one. And let's open up Blender files on sequence one shot 20. Again, we're going to pick Blender four four.
And again, it looks like we're looking through the wall for the most part. And then at the very end you can see our flute player. Okay, so we have a couple of things we're going to want to do again, we're just going to move out to the file and note that we're shooting through the library folder again.
In this case let's let's go ahead and find our library books, this books and right click and do our library override. And we know we're going to need to do it to the bookshelves as well. And right click and library override make selected. Okay, so now we can go click here, click H to hide, click r, bookshelves H to hide, and click our general structure, right click library override.
## Library Override Auto Update
And the nice thing about the doing these library overrides is when you update your central file and you update the links, these things will still remain hidden. You only have to do this once for every shot file. And that's the really efficient thing about the library override system is you can make local changes to your shot files on a per shot basis and still update the top level file and have those changes ripple through.
So let's go ahead and hide that piece. Go back to our image plane and let's see how this, and go back and play the shot. Okay, so now we no longer have any 3D intersections.
## Setting Slap Comp Keyer
We're still going to want to clean up the key a little bit. To do that, let's go over to our shading. And again, remember this is a temporary key.
We just want this for the slap comps. The actual production key will typically be handled elsewhere. So let's go and highlight our image plane and we're going to hit period to zoom in on this so we can actually see it clearly. Okay. Here is our flute player. And let us fix the key. I'm going to use a couple of shortcuts that are part of Node Wrangler.
And if you need to enable node wrangler, it's over in preferences add-ons, type in node node Wrangler. Let's turn that on. There we go. It's a really use, useful set of shortcuts as we're going to see. And so we've selected our, it's not our camera. We've actually selected our image plane, and so we can see the shader that is on the image plane.
And this is, has a real time keyer in it. We don't have quite the right color selected, so let's hit control shift click on our image sequence that is coming in. And that is a sRGB image sequence that Autoshot automatically created earlier in the proxies directory.
And now that connects this node directly to our output. So we actually have the the color of the green. So now we can actually click on our key color, pick our eyedropper, and let's see how that worked. I'm going to hit control shift left mouse button on there, and go back and reconnect our shader to this and let's play through it.
And let's see if our key is working working better. Now that looks a lot better there and it's looking pretty clean. All right. It is easily clean enough for a slap comp. All right. And that should do it. Okay. Then we can hit save .
## Rendering Slap Comp
We'll go to our layout view.
Make sure that we are, have the view port locked to the camera by hitting the num pad zero key. Or if you do not have a numpad, you're just going to go to view and cameras and active camera and. View. Okay. View cameras and set this as the active camera. All right. And then we once again are going to hit n to pop open our Autoshot panel, and we're going to hit fast viewport video again.
You can see as it goes from the default. If you would hit an image render, with a traditional rendering, it would go to correctly to the Blender render as an EXR file. We are doing something specific. We actually want a slap comp. We're just going to hit fast viewport video and it's going to come up and automatically set our output down here to our slap comp directory and to our slap com file name in the MP four file format.
So it's going through very quickly and rendering our slap comp for us. And yeah, we'll have some key key problems, but the focus on the slap comps is to get it all together so you can actually see how all the pieces are going to come together and if there's anything else that needs to get fixed.
Great. So then we can check let's say save exit out of our Blender file and we can take a look at our newly generated slap comp.
All right. And there is our slap comp with a clean key and a clean integration between the live action foreground plate and the CGI background library. And we'll just to, for completion, we'll go through Pathways 0 1 0 30.
## Correcting Additional Shots
Once again, we're going to go through and look at our slap comp first to see if anything is required due to this.
There we go. In this case, our key needs some fixing and there's something in the background. So let's go ahead and again, we're going to go back up to our 3D our Blender files and double click on this.
Go to Blender four, four.
All right, so we've got this and we've got a object in the scene. I'm not quite sure where that came from. Okay, so let's go ahead and correct that. And we're going to do instance bookshelves and library override. And, the process by now. Click on the vending object. We can show where it is in the hierarchy with a period ecosphere.
And we're just going to hide that. And let's see. Look, if there's anything else going wrong in here, going to look through the shot. Okay. So we need to fix the key. So let's go back over to shading and highlight our image plane. Zoom to that control shift. Click on this to get the pure green screen.
Set our key.
Do a darker spot on the key. Check the key, and I'm just hitting space to, to play through the sequence. We can hit number pad zero, so we can actually look at it through the actual camera view. There we go.
All right. That is a good shot. Okay, so now once again, we are going to hit N and we are going to do our fast view port video.
So the basic idea here is that we can generate these slap comps very quickly. And these slap comps are actually the shot file for the actual sequence. Over time, these Blender files will be refined to incorporate updates to the 3D background files and additional pieces such as re-lit foreground files.
But we have generated the actual production shot file for each individual shot.
And so that way you can go through quite a large number of shots very quickly with this piece of automation.
# 2D Backgrounds
Source: https://docs.lightcraft.pro/tutorials/jetset/2d-backgrounds
Adding 360 and still image backgrounds behind 3D USD scenes
## Video Tutorial
# Transcription
In addition to loading 3D USDZ scenes exported from Blender and Unreal, Jetset can also directly import and adjust 2D images right into the 3D scene. This lets you do a rapid 2. 5D matte painting style workflow right in the app, which is fantastic for quick concept iteration.
## Loading 2D Images
First, we'll need to load some 2D images onto the phone.
We'll use the iCloud web file transfer described earlier and navigate to our project's models subdirectory. Remember that the Jetset projects are all under the iCloud Jetset folder. So we can go up, kind of see where we were at. iCloud drive, Jetset Cine\_tests in this case, and then models.
Once we're here, we can drag and drop a few 2D images into the directory. Since we're just loading one background image, it can be a high Resolution 4K image. Alright. So those have been uploaded, and now we'll move to Jetset.
In Jetset, we'll go to the 2D tab. Here we can add or remove images. We'll set our ghost slider up all the way to the top to see a CG image. We'll hide this image, and I'm going to click import, and I'm going to navigate to where we had saved our images.
We'll go up a bit just into the models directory.
## Moving and Scaling Images
And there's a subway image. Once the subway's image is loaded in, then we can manipulate it inside Jetset. Currently you'll see a button on the top that says Pinch to Scale and that means that you can either move it around with one figure, or if you use a Pinch Zoom motion, it'll actually scale the image plane up and down.
## Translating Image Planes in Depth
The image is actually being placed into the 3D scene as a plane in that 3D scene. So we can actually click on the Pinch to Scale button, and it'll toggle to a Pinch to Move Forward button. Then you can still move it around with one finger, or if you use the Pinch to Zoom, you'll actually move it forward and backward in 3D space, and it'll show you exactly how far away in depth it's being moved from the tracking origin. In this case, we're moving it to about 5 meters backwards, about 15 feet. We're gonna move it over. Over a bit, and then there's our subway.
## Finishing Edit
Then click OK when you're done to go back to the composite view, and this makes for a really fast, efficient way to match in a 2D image into your existing 3D scene.
# Animation Controls
Source: https://docs.lightcraft.pro/tutorials/jetset/animation-controls
Controlling animated USD scenes in Jetset or remotely
## Video Tutorial
# Transcription
Jetset now has an integrated animation timeline interface with controls both on the app side and in the remote control browser page. And these controls let you easily scrub through an imported animation, find a good starting point, and then trigger the animation on command with the timing decisions automatically recorded and rippled back through Autoshot.
## Jetset Timeline UI
We'll start by loading in an animated USDZ model, and the updated Jetset UI now has a single control screen with a ghost slider that lets you slide between the camera original view and, and the composite view.
And on top there's now an animation timeline slider and a play pause button. So we hit play. Then we can see the animation starting to work. And it'll, it'll go through and naturally cycle around. So then we can scrub through the animation to find a good starting point. So we'll hit pause. And now we can roll a take.
And hit play to start the playback of the animation at that point.
And when we cut, the animation pauses.
## Remote UI
This is fine for simple things, but in many cases we want to have a remote operator control or trigger the animation on command. For these cases, we'll open up our remote control web browser. The easiest way to do this is to run Autoshot, go to our takes panel see the automatically detected Jetset model and click open web page. This will bring up our browser remote control interface.
## Timeline Settings
And by default, it'll go to the slate interface, as usual, but we can also go to the settings interface.
And we can see here we have a couple of different controls for the animation. We can tell it to restart animation when we hit record. And we can tell it to stop animation when we stop recording. That's how we currently have it set.
## Timeline UI
Well, now let's switch to the video panel. As before, we have our video controls, and we have a timeline control down here with animations for play and pause, as well as a slider. We can take our slider and scroll through the animation.
## Reset Point
And what this does is we've set our reset point at a point that's about six tenths of the way through the entire timeline.
This is the point that the animation will jump to when reset. And next to here we have the animation reset button. So a reset occurs when the reset animation is pressed. We can click it here and it's not going to do anything because it was already at the reset point.
Or if the restart animation on record checkbox is enabled, then the user rolls a Jetset take.
## Example Take
So let's work through this. So we've set the animation to a point just before where the robot gets up. It's going to get up here, so we're going to set the animation here.
And now we're going to roll the Jetset take.
Note that the animation is still paused, and at a cue on the director, we can hit play, and the robot will get up. Alright, and then we can hit cut. Now the animation is paused, we can click animation reset, and it'll move back to the reset point. And again, all of the animation timing decisions that we make here are saved in the take metadata, and we can automatically reconstruct them in the Autoshot post production files. So that covers it for animation remote control.
# Introduction to Jetset
Source: https://docs.lightcraft.pro/tutorials/jetset/introduction-jetset
Introduction to the Jetset iOS app and overall operations
## Video Tutorial
# Transcription
**Speaker:** We're going to go through a first pass of starting up Jetset and walking through the complete workflow so you can see exactly how all these different pieces work. Just started up the app, and we're at what's called the model pick page.
## Model Pick Page
**Speaker:** And here you can just pick out one of the default models to start with. In this case, we'll just pick the Charterhouse. We have lots of different models. And you'll just have to be a little bit careful with some of them depending on the size of your phone memory. So you pick the Charterhouse, and by default, what's going to happen is that it's going to load in the Charterhouse, and it's going to have already picked an origin in your scene, so you can immediately look around and see that you're in a 3D world that's tracking correctly in 3D space.
And if you stick your hand in front of the camera then you're actually going to see an AI matte version of your hand that you're going to see a little bit of choppy edges on that. And that is because we are extracting by default the human images out of there with AI. So this will work even if you don't have a green screen.
We do have a green screen set up in here, so a little bit later we're going to go through setting up the keyers and the mattes and all those sorts of things. This is just to give people a quick default experience to work with.
## Setting Live Action Origin
**Speaker:** One of the real key things in virtual production is matching your CGI world to your live action world and picking out where your live action origin's going to be.
And when we started up Jetset, it just did that by default. Since picking the origin is so important, we're actually going to jump to that first. So we are going to go to our main menu, which is up in the upper left hand corner, and we're just going to pick our origin panel and by tapping on it. And what this immediately does is it lets us see our real world area.
There's a number of controls on this, but the simplest thing we're going to do is just start tracking from scratch again. So to do that, we're just going to click the reset button and we'll just say, start a new map. And as we move the camera around what the iPhone is doing. Is it's finding all the natural features in the scene, and what you can see is those little red, green and blue kind of firefly looking sorts of things all over points on the floor.
And anywhere you see a corner in the scene. Those are the feature points that the phone is picking up in the scene. That's how Jetset tracks is by detecting those. Now, you also see it is detected the horizontal ground planes. And when we set our origin, we're always going to set our origin somewhere on a horizontal plane.
And so when we start it up, it's going to detect the different horizontal ground planes in the scene. And so now you can actually just point it somewhere at the ground plane and and tap to set your origin. So in this case we just put the origin kind of in the middle of the room. And if you want to see how that's affecting your CGI world, you can actually slide our ghost slider up on the left, up and down a little bit.
And that controls the mix between the CGI and the live action world. So you can see how you're lining up.
## Adjusting Origin with Dragging
**Speaker:** Now, once you tap it, you may want to adjust it. And so the simplest way of adjusting it is to just tap and drag on the three balls there that represent the axes of the origin.
You can see the X is the red axis. And so tapping it moves it, moves that laterally. The blue is the other horizontal A axis. So that, that moves things around back and forth. And then finally the purple ball on the right lets you adjust the orientation of this.
So as you move the purple ball to an angle and then, and we slide our ghost slider back up we're going to see that the whole room has now moved at an angle. Actually let's move that. There you go. So we can see that we're actually rotating the CGI room around our local origin. All right, so that's a good way to do fine grain control over this.
## Origin Axis Controls
**Speaker:** Now, you may actually want to move much more quickly in this scene. And to do that, we can actually use the controls up in the upper right hand corner. Now, we'll click the fore aft control And then we can actually drag back and forth, and that's going to move us into and out of the scene.
This is a quick way of navigating through a potential scene that you're working with. You can actually just change the increments of motion. If we wanted to move slower, we'd pick the one centimeter increment. And then we drag back and forth quite slowly, almost imperceptibly, or we pick the 10 meter increment, at which point we're moving quick.
So this is a fast way to zip around through the scene. And we're just dragging back and forth horizontally on the actual iPhone screen to do this. Now, We can do the same thing with the rotary axis. So if we can then click our rotary axis, we can click and drag and we can actually move our scene around around the origin.
And as you can see where we picked our origin, that's where we're rotating around. And, again, we can change our increments between slow and fast rotation just depending on how we're doing this. And we keep track of the offsets. You can see that the overall offset from the original origin, 5.
5 degrees, 4. 7 degrees, is shown on that particular axis. And last of all, if we needed to move to the vertical axis, we could click the Z up and down. And we can once again move slowly or quickly up and down in the scene. If we switch to one meter, we're moving quite quickly. And if we're switching to a shorter amount, then we move more slowly.
Now you have to be careful with the Z offset. You in general want to keep the Z offset at zero. Otherwise you'll see problems with your floor sliding, because Jetset is assuming that the zero location of the Z offset is aligned with the local floor. Because that's where we detected our floor ground planes and that's where we initially set our floor origin.
So let's go ahead and we can exit out of the Z adjustment and we'll go back and we're just going to reset our floor tracking again. Then we'll start a new map and again we're finding our ground planes and we'll just pick an origin for the floor.
## Horizontal Optical Origin Marker
**Speaker:** Now one other way we can do this is we can use our optical markers to set our origin.
And we have a both a horizontal and an optical Horizontal and a vertical optical marker and these are available in PDF form on our website and in two sizes. One of them is a small size, which is eight and a half by 11 are basically letter size and that's easy to print out for anybody locally on their printer.
But that turns out to be a little bit small to work well in a production setting. So we have a second set of these origin markers that are also available as a PDF on our downloads area. And those are designed as a standard 16 by 24 dimension size to be printed on canva. com. We just printed those on a standard eighth inch thick backing board.
So it's a nice rigid board. Let's go to our our floor or our horizontal floor origin and you can see that when we frame up in this, it is automatically detecting that, and it's making a little frame around that. Now, when we detect it, the align to marker button is now high lit.
So if we tap it, we have just set our origin to match the origin of our marker. And, the neat thing about that is that is adjusting both the position and the orientation. For example, if we changed our marker position yeah, there we go. If we change our marker position to another orientation and we click align to marker, it's going to change our origin to that new position. Now, we usually want to have our origin aligned with some feature in the stage, in this case, the alignment of the green screen.
So we'll just go ahead and click align to marker again and reset that.
## Vertical Optical Origin Marker
**Speaker:** This works well when you can, when you want to be able to face down and set your origin with the camera. Now, in production situations, very frequently you'll want to reset your origin, but you can't actually tilt the camera down that far.
And in these cases, we have the vertical origin marker. And once again, when you come into detection view of that marker, it's going to make a little frame around that, it's going to highlight the Align to Marker button.
So if we come up to that and close enough to that we can then detect the outline of that. Is it detecting? There we go. Now we can click align to marker. And what it does is it has moved the marker down to the base of wherever that vertical marker is standing. It didn't change the floor plane.
That was already set earlier when we set our initial tracking. But, it dropped our origin right below where the giant arrow points down. And so this is really useful in a lot of cases in in production sets, where you want to quickly move the origin, but the camera kind of has to stay put.
So in this case, you could have somebody run in front of the camera with a stand, put down the stand hit the line to marker and boom, your origin is wherever it needs to be. So that's a nice quick, rapid way of doing origin changes. Now the vertical origin marker does not adjust the orientation.
So even if we change our vertical marker, it's just shooting a ray straight down and our. original orientation that aligns with the stage floor is not changed. And that's because it's too hard to keep the vertical marker aligned in a rotary fashion. Once you have your directions on your stage, you generally don't want to change that alignment.
All right, so let's go ahead and reset our tracking at our horizontal origin at our horizontal stage marker. There we go. Align to marker. And then we can click OK.
## Scene Locators
**Speaker:** So we've set our origin. Now one key aspect of Jetset is called scene locators. And scene locators is how, once we've picked our origin, how we snap a particular location in the 3D scene to that origin, and that combination is really the key in virtual production.
Where is your live action origin, and where in your 3D scene are you going to connect to that? Right now, our scene locator menu is on the right hand side, with the upward facing arrow. Now, If you click on that, we can scroll through all the different scene locators that are in the scene.
These are just different locations that when we created the 3D scene for this we created a set of specially named 3D nulls that have a sceneloc prefix and then a name --in this case sceneloc chandelier, sceneloc fireplace. And when we load that into Jetset, it --looks for those names and it makes us a list. So when we pick fireplace, it's going to lock our origin right to the fireplace. So if we scroll down our ghost slider while we're looking down at the corner of the fireplace.
Let's scroll down a little bit. You can see that we actually locked the corner of the fireplace is locked exactly to our origin. Now, if we change our scene locator to say the main the main door. Boom! Our origin, our live action origin has not changed, but now we've actually just snapped the main door of the room to that origin.
This is really the key of how we align things in virtual production. We have an alignment point in our 3D scene, which we call a scene locator, and we have our live action origin, and Jetset is designed to let you snap those together really quickly and really easily. Okay, now we can switch back to the origin for the rest of the shots.
## Dashboard Indicators
**Speaker:** We can take a quick look at our tracking quality with our dashboard on the lower right.
And our dashboard on the lower right has all the different pieces to measure the current state of the iPhone and whether it's tracking well, whether the rendering is overtaxing it, if the thermals are okay and what the battery is. And so by tapping it, you can see this pop up panel that shows a real time metering of this.
And you don't generally need to see this that much. It's more of a debugging sort of thing. So we just tap the panel again. What's useful is you can quickly, while you're shooting, keep an eye on those four key elements. The leftmost one, which is the little compass needle, that is the tracking quality. Next over is how hard we're running the GPU. That's the the round tachometer. The next one over is the degree, the temperature of the phone. And the right most one is of course the battery level. And what you'll see typically is if you have the room mapped, if you've walked around to a decent degree and you've.
Found all the tracking points around the room. While you're walking around the room, it's going to shift yellow because it's adding more features to the to the scene. And then once you've gone through and added most of the features, then you're mostly in a green area of tracking. You generally want to keep things in a green and occasionally yellow area when you're working with Jetset. Green and yellow will still record okay. When you hit red, then you may start running into problems with tracking.
## Scanning 3D SCene
**Speaker:** Once we've set our origin, the next thing we're usually going to want to do is to do a scan of our 3D scene. And this is invaluable for post production, because it lets you know where everything was with respect to your tracking data.
And your tracking data is all oriented with respect to the origin that we just set. So let's go ahead and go to our main menu. And we're going to click Scan. Now, Scan is only available on Jetset Pro and Jetset Cine. And it will only work with devices that have LIDAR in it, which is the iPhone Pro, and the iPad Pro.
It's pretty simple. If you just click Start. Under the scan icon, it's actually going to go around and you're going to see it scanning very quickly a mesh that maps onto the 3D position and orientation in the scene. And it's using the LiDAR detector along with its optical information to basically build a mesh that, that kind of covers over all the different pieces of the scene.
Now, ou don't generally need to map your entire area. What you're usually looking for post production, is just the area within the next, four, three or four meters around where you're shooting. And that's that's usually enough for the tracking in post production to be able to recreate the 3D positions of the camera.
In general the place where the actors are going to be and the green screen behind them that's generally going to do the trick. Once we've got a decent scan, we can just click Stop. And by default, there we go, the scan will process. The, by default the scan will show kind of an overlay, a mesh overlay.
Now if you don't want to see that, you can toggle the the display button. Right now it says solid and that will move it to a solid display. We usually want to hide it, so we're going to go ahead and click hide.
And and that lets us hide the scan. If we wanted to delete the scan, we won't do that now. You just click clear. So let's go ahead and click okay. So now we have our scan.
## Keyer
**Speaker:** The next thing we're going to want to do is set our keyer.
So once again, we're going to go to the main menu and then we're going to click key. And here we have on the left a choice of all the different matting systems that Jetset has. And we already saw our default AI matting system. So for example if you put your hand in front of it right now, that's an AI generated mat.
And so you see the green edges around it. And again, the AI mat is what you want when you're just operating without a green screen. If you're just doing a test shot inside your living room, or if you're doing an exterior on location where there's no green screen, then you can use the AI mats.
Now, We'll do something simple first, which is we'll look at a CG matte. So if you click CG, now is just only going to display the actual CG image in there. And even if you stick your hand in front of this then, as I'm sticking my hand in front of it you won't see any part of the live action scene.
And this is good for pre visualization where you're dealing with a pure CG scene, and you don't want any of the live action elements. The next one we'll look at is green. We'll click on the green screen. There we go. And what this is going to do is it's going to run a keyer over on the scene.
And what's, now you can see that the hand's edges are no longer colored green because we're running a green screen keyer and the edges are much cleaner. And Jetset has a pretty simple but it works pretty well green screen keyer. We focus on making it more of a kind of a one touch keyer for most situations.
## Setting 3D Mattes
**Speaker:** Now, this is fine, but if we tilt it up, you're going to see things like the stage stuff and things that you generally don't want to see in the scene. And to fix that, we actually have, built into Jetset, a 3D garbage matting system which we call screen planes.
If we want to actually define the points of the green screen that we are going to run at the keyer, and then have everything outside of that be blocked off by CGI, we can do that. To set your screen planes, you're just going to click the start button on the screen planes. And it's going to automatically detect either a horizontal or a vertical plane.
So we can start off with, say, a horizontal, a vertical plane in the back in the back wall of this. And we'll come forward until it's detecting, and by default it's going to try to find a perimeter of the green screen. So we can hit a plus sign when it's detecting the perimeter of the green screen.
There we go. And what it has done is it's created a a simple polygon on that green screen with, but with some edge controls.
And you'll notice that it, the polygon it created is a little goofy looking, go ahead and we're going to go ahead and click clear, cause that one was a little bit, little goofy looking.
There we go. And we're going to try aiming on that to get a little easier of a polygon plane. You can move in a little bit until it detects it reasonably well. There we go. Now just click plus again. All right, there we go.
So now we have our trapezoidal looking polygon plane. We're going to correct that more a little bit later. But we have a set of points defined on the rear wall. We're going to do the same thing on the floor. We're going to back up a little bit. Look at the floor. Once again, it's going to create a crazy looking polygon on the floor and we'll click plus.
There we go. So now we've got our silver spheres. So now we can click stop. And now we have our garbage matting system working. And the way our garbage matting system works is that when your hand is over the keyer, the keyed area, then the keyer is working. And so you'll have a nice clean edge.
Now, when it goes over into the non keyed area, we automatically switch into AI mats. So you can see, as we go from the AI mats over there the edge is choppy. And as we go over here, they, it becomes a much cleaner key because now we're keying on a green screen area.
## Editing 3D Mattes
**Speaker:** So what we're going to want to do is actually just move our little Spheres around until they cover the green screen area that we want them to cover.
So we can actually just grab hold of them and click and drag the them around. We'll drag the vertical one first and we'll drag it down to the fit the corner of the green screen. And actually we can extend a little bit past and we move over to the other corner and click and drag that down.
And the neat thing is while we're dragging is we are staying restricted to that original vertical plane where the green screen wall is. So we're not accidentally randomly putting them in 3D space. We know they're supposed to be on a wall, and they're going to stay on a wall. Even though if we're going to edit them around.
And we can move up here and grab a couple of our other silver balls. And we can click and drag them into place, seeing how the matting is going to work.
And likewise, we can come down on the bottom, and we can drag the bottom in and to overlap the back, so that we don't have any any patchy spots in between them. And it's okay if they overlap a bit that's fine, we can bring one of the edges past the other edge, then the matting system will knit the two together correctly.
All right. And then once we have that, as we move around in the scene, we can put our hand in the scene. Oh, let's move the the top down just a little bit. So we get rid of those two areas.
## Set Keyer Color
**Speaker:** All right, now once we've got our garbage mat set, we can actually set the keyer color that we're going to want to work with. So for example, if we click the reset button to the upper right then it's going to reset that to a default color.
So once we've reset our keyer, we can then pick our color swatch, or we're just going to tap our green screen swatch, and then it's going to sample the scene as we'd click and drag over. different parts of the green screen area, and it's going to pick out an average green screen color. So now we have a pretty clear, clean key, but we can adjust the edges with the in and out black and white points there.
So we can tune our black and white points to tighten up our green screen edges. Okay, so we're getting a reasonably keen, clean key in real time.
Now if we wanted to do a blue screen key, we would click blue. We don't have a blue screen here right now, so we don't need to do that.
## Overlay Keyer
**Speaker:** And the overlay is worth talking about because Overlay doesn't use keying, it only uses depth imagery to decide which part of the scene is going to be in front of other portions of the scene.
And overlay is usually most useful when you have a CG object dropped into a live action scene, at which point it'll occlude everything correctly. It's going to look a little bit weird because what's happening is our CGI floor is fighting with our green screen floor. So we get some really strange Z depth fighting.
We'll go back to green for now. All right, now we can click OK.
## Ghost Slider
**Speaker:** To see how this this is going to work back and forth, again, we can use our ghost slider on the left. Okay, yeah, there it went. We can use our ghost slider on the left to see how our CGI scene is lining up with our live action scene. And so you can tune it up and down just to make sure things are lined up as we expect. Okay.
## Project Storage
**Speaker:** We have our origin, we have our scan, we set our key. We're about to record a take, but before we do that, we should actually pick out where we're going to store our take data. And this is actually an important question depending on the kind of production you're on. We can go tap our settings button, and we can tap project.
And, our project pick button lets us pick different spots where things can go. We can pick between a couple of different areas of storing. So you can see our initial path by default. We set it to iCloud and then Jetset makes a folder and then we store our projects there. Now you can choose to store this in different locations.
For example, if you're on a project where you do not want to store anything on iCloud, you can pick a couple of different options. And one of them is inside the Jetset app folder. And one of them is just a user selected folder somewhere on your iOS device. We can try picking the Jetset app folder.
Now you can see the path has changed. It is storing it on my Jetset and on, under the project name and takes. It's storing the project in the same memory space that the Jetset app is stored on. Now, this has the advantage that it will not be stored to iCloud, but you have to be careful because if you uninstall your Jetset application, it's going to uninstall the takes as well.
And the other option, if we click on our path again, is we can pick a user selected folder. So if we can pick our user selected folder, it's going to pop up and you can pick a spot on your phone that you want to store stuff in.
But actually, we can just default to the iCloud. And that's a good choice for a lot of things just because it provides some backup to your iCloud device. If you don't have iCloud, then you can just store it in the app storage or on a local folder.
## Creating a New Project and Prefix
**Speaker:** If you want to create a new project, you can actually hit the plus button. And it'll, and scroll down and it'll have created a new project. Now we probably want to name it something besides new projects. So we're going to tap the edit button and then tap the name.
And then we can name it whatever we want. We can call this test, test is fine. Now, the other part of it is the prefix. And every file generated by Jetset is prefaced by a six digit prefix that you can pick to fit your project name. And this is a huge help when you start to have large numbers of projects and files. You can always tell where a file belongs by just looking at the prefix.
So in this case, we're going to edit that, and we are just going to delete that. And we just Put in a few characters like test, and that, that can be our prefix for that. Yeah. Test 12. There we go. Now we can click done and now we can click save.
## Zoom Controls
**Speaker:** We have the ability to zoom in and zoom out on Jetset. So right now by default Jetset is showing the native focal length of the actual device, in this case, an iPhone 16 pro max. And so it's showing the super 35 millimeter equivalent of what that focal length would be. So by default the iPhone has a pretty wide field of view. So that's about a 15 millimeter super 35 equivalent.
We can change our virtual sensor width. We'll work on that in a separate tutorial. Because it's a bit more advanced. But we can also just in here zoom in and out by tapping the right and left arrow. And we can basically zoom in on the image. Now we're simulating the zoom. We're not actually zooming in the optics of the device, and so that's why we're cropping in on the image and the image is going to get a bit softer as we come in. But this is a nice way of being able to test shoot something and to see what your visibility will be at different focal lengths.
## Focus and Exposure Lock
**Speaker:** And we have controls to the left and the right of our lens focal length, which is the auto focus lock and the auto exposure lock.
And right now we have our auto exposure locked because we were working with the green screen and we want to have our auto exposure locked whenever we have a green screen matte. And in fact, if you pick the green screen matte in the keyer, it'll automatically lock the auto exposure.
Currently we have autofocus enabled in the phone. That's usually where you want it. If you need to lock focus, you can tap the autofocus button and it'll turn gold and put a little lock sign on it. There we go. And our focus is locked. We can leave it in auto focus for now.
## Recording a Take
**Speaker:** Okay we're ready to record a take, so now we can just hit record, and you'll see below Jetset will start recording the number of seconds in the take. We can move around the scene And find a you know, a nice take as we're, we're finding a nice shot as we're going through. And we can hit stop.
## Reviewing Takes
**Speaker:** And now you may want to review the take, and you can click the review tab in the upper right hand corner, and this will take a second or two to load this. Every time you roll and cut , Jetset is going to record both the camera original, which we see on the left , with the the name of the take and the number of the take with the suffix cam. And over on the right, you can see that it's recorded the real time comp with the suffix comp.
And we can see that our frame rate was 30 frames per second. Looks like we duplicated three frames on that; the phone may be heating up. We usually want to use a cooler if we can and if we want to hit play, we can actually hit play to replay the shot whether it's the live action or we can hit play on the live composite and see how the take was working.
So then if we go over and click the full screen button, we can actually full screen the image and look at our recent take and how we moved it and see if we like it or not.
## Sharing Clips
**Speaker:** All right, so that's good. We can hit pause and go back to our normal screen size. From here We can do a couple interesting things We can actually share a shot via Airdrop or other devices we'll just tap the share button and here this is the usual iPhone sharing system We can either Airdrop, mail it etc.
And Airdropping some of these takes is quite useful. The files are usually too big to email, but AirDrop, it can be really useful for sending a quick comp around. For transferring more takes in a systematic way, we actually have a tool that runs on the Mac and the PC called Autoshot.
That's the tool that you will use to transfer most of your captured takes.
## Re-Rendering
**Speaker:** On the left, you can see a little film frame icon. And what that would be is for re rendering if you wanted to re render your background.
Now, we have a separate tutorial on that, so we won't go into that detail here, but that's what that button is for. We can exit out of that. Okay, so once you've reviewed your shot we can go back to our main view, our main menu.
## Digital Slate Tab
**Speaker:** Jetset has a web browser built into it. And that web browser is what lets us do our digital slate as well as our remote control to Jetset. And so this is really key when you start working on a stage where the camera's in one place, but part of your team is over in Video Village, and you want to have remote operations.
First of all, the way you get to the slate is you enter the IP address of Jetset, and you can either find that out manually inside the main menu of Jetset you just click the the web server IP value under the settings, or in Autoshot, you can click open webpage if it has Jetset detected but either way, all it is the IP address of the Jetset device, and when you do that, You're going to see this basic display. This is the slate panel which gives you a digital slate that is connected to Jetset.
It shows you in real time the same time code that Jetset is currently using. Shows you the lens and a number of the different aspects of the current Jetset take.
## Editing Digital Slate Properties
**Speaker:** Now, this is color coded. And the items that are color coded gold are things that are editable locally on the digital slate. For example, if you're on a scene or a take and you wanted to change that, you could actually change the scene right here.
If you just tap, say, 5001 you can actually change that to 5000, 5032, or whatever scene you happen to be using. And that will update automatically in the Jetset UI. Now, we also see a couple other key pieces here. We can see the current UUID of the Jetset take. Now, each Jetset take, when we record it, has a 10 digit hexadecimal unique ID. And that UID is used to label every piece of data from that Jetset take from the camera recordings to the tracking data to everything.
And that means that we can always associate in post production, which data goes with which take, because everything has a unique name. Other fields here include the lens file. Now, if we were shooting with Jetset Cine, that would show you the currently loaded lens calibration file.
Over here on the right, we can see the cam and the comp. So right now, Jetset is recording the camera original at 720p and the composite at 540p. We can see our current project name; that is actually set inside Jetset in our project panel as we showed that earlier.
So we can't change that locally but we can actually change things like the director or the director of photography by, typing in the notes or just adding in a quick note on the current take. We can just tap something and add that into the keyboard. There we go.
## Record and Stop on Slate
**Speaker:** Now we can also remotely record and stop. So if you click the record button here, it's going to trigger Jetset to record. And you'll see a set of strange looking black and white markers. The take has now started rolling in Jetset. And we can go ahead and click stop to stop rolling the take, if you need to remote control that.
And those black and white markers are one of the ways that we use in Jetset Cine to both identify which Cine Camera take this Jetset recording was associated with and to synchronize with this so that we know exactly when the tracking data that's stored inside the iPhone should line up with the actual Cine Frame stored in the Cine Camera.
Now we're just testing with Jetset right now, so we won't use those immediately, but it that's why they're there.
## Settings Tab
**Speaker:** Now we can click on the settings tab, and this shows some of the basic background settings for Jetset that we can adjust remotely that weren't in the main panel. You'll mostly see these when we start doing external tracking data. We won't dive into that right now, but this is where they are.
## Video Tab
**Speaker:** And finally, we're going to go to the video tab. Now, the video tab, if you are on Jetset Pro or Jetset Cine, lets you click and play a remote feed of the current Jetset composite. This can be really useful to have in Video Village on set because that way you can see a live composite from Jetset as it's shooting without having to be next to the camera.
## Remote Controls
**Speaker:** Now, under here is a couple of key pieces. There's animation controls, which we'll go into in a different video, but over to the right, we actually have a set of remote controls and those remote controls are matched to the same UI that Jetset uses.
So right now we're on the main view, but if we click to the origin panel uh, under main, there we go. Now it's going to display and work this exact same way that it would if we were in the origin panel in Jetset.
## Remote Origin Alignment
**Speaker:** So for example, if we move our Jetset device down, and until we can actually detect the horizontal origin markers, They will detect, and we can click Align to Marker, and our origin will align to the marker. Now, there's another tool over here, which is called Drop, and that will simply drop the origin below wherever you happen to be standing.
If you click Drop, there we go, and we look down, our origin has dropped down exactly below where we were standing, and sometimes that's useful if you're having to do things on the fly. So let's go back and detect our origin marker panel, and we're going to click Align to Marker.
There we go. And so our origin is set where we want. Now, what happens if we, Now we can click OK to accept that. And again, this is, behaves exactly the way it does in the origin panel.
## Remote Scanning Control
**Speaker:** Now, we can go to the scanning panel in our remote control.
This lets us remotely operate the scanning system. This is phenomenally useful for a production where the camera team is over there and, getting ready to set up a shot, and Video Village needs to be able to quickly jump in and grab a scan without having to go up to the app.
We can click through and see the displays of the existing scan and we can clear the scan. Say if we have the, an incorrect scan on there, go ahead and clear it. And that'll remove the current scan and we can click start and it's going to start scanning.
So then we can immediately look around and detect all the different scans that are in operation. And if people in the camera team are just moving the camera around like they are during setup, you can be quickly grabbing a scan and everything's all set.
Now we can click stop.
There we go. And there's our scan. There we go, there's our scan locked in. All right, and then once again we can toggle our solid display so that we're not seeing the display. And now we can click OK as then we're done with our local scan.
## Remote Scene Locator
**Speaker:** While we're in the main tab, we can actually switch through different scene locators in the scene.
In our remote control, we can see the list of scene locators. Right now the origin scene locator is selected. So if we go ahead and click origin, this picks out all the different scene locators in the scene, and they work exactly like the scene locator pick in the main Jetset menu works. We were originally at origin, but if we pick say, the fireplace, then all of a sudden, our fireplace has now snapped to our origin.
Now we can go ahead and click our scene locator and pick back, go back to our origin. It'll work a little bit better for this scene. There we go. And so again, it works exactly the same as it does in the main menu panel.
**Eliot:** All right, there are many more details we can go in with other tutorials, but this should give you a quick overview of how Jetset works and how to do initial setup.
# Fixing a Misaligned Shot
Source: https://docs.lightcraft.pro/tutorials/tracking-compositing/fixing-misaligned-shot
Minor bumps and misalignments are part of any real production. With Jetset Cine’s integrated scan and tracking data, it’s easy to tweak the recorded camera keyframes to re-align shots rapidly. We also introduce Matt Merkovich’s Multi-Peel script for quickly finding all the trackable points in a scene.
## Video Tutorial
# Transcription
In this tutorial we're going to go over a Jetset Cine shot that had a lens calibration and a scan, but the iPhone got moved at some point, so things aren't quite lining up.
It's really easy to fix as the usually the change is more angular instead of position. If the phone got bumped by a centimeter or two, it's not that big of a deal, but if the angular setting is off, then it really throws things off. But fortunately, that's really easy to fix.
And we're also going to use the multi peel Synthia script from Matt Merkovich of SynthEyes Tracking fame that really speeds up the process of finding good markers.
## Importing Shot Script
We already generated the output script in Autoshot, just as we did before in the previous SynthEyes tutorial.
Just going to go to script, and run the script, and pick out our Sizzle script and open that. There we go. And we'll switch to our camera view.
I'm going to dial down our gamma a little bit. So we can immediately see. That there's our mesh, and there's the edge of our of our cardboard boxes over here, so clearly that's misaligned. Now fortunately we can make a pretty easy fix here.
## Viewing Camera Motion Graphs
So we can just go over to setting our cameras and graphs. And we're going to open up our cameras and objects. And we're going to go ahead and open up camera01. And we're just going to hide the solved velocity path. Actually, let's first save our file.
Let's uncheck our solved velocity because we don't need to see that for now. We're going to open up our solved path. This is where Autoshot will at first write in our Jetset real time tracking data. So for example, if we go to our tilt and we highlight our tilt angle this is all the key frames from our tilt.
And if we went over to our X, Y, or Z, we could actually see where all those different key frames are.
## Correcting Camera Angles
So for now we're going to fix first a tilt angle, and we're going to come over here and we're just going to drag and select all of our key frames. And we're going to just pick on a key frame and drag adjust up down just very slightly until we can actually see that.
Now our bottom our scan is now lining up correctly with this stack of cardboard in the back. And it's not lining up with this set of pads here. I think that's just because the pads got moved and I'm picking things that in the background that probably were not moved from shot to shot.
So that's how I'm making that decision. So I'm going to go over and our tilt is fixed. I'm going to go switch our pan angle. Once again, I'm going to down here. I can either drag select from here or upward into the left drag, select these, and then we can adjust our pan and till there we go.
Now we're lining up pretty well with with here, the edge of our edge of our cardboard stack. You can see. This is aligning up over here with the edge transition here. And things are generally looking about like they should look. That'll help our alignment.
## Installing Multi-Peel Synthia Script
So now we're going to go and find our tracking points, just as we did before, except this time we're going to use a nice automated process. First of all, we're going to go to our webpage, our download web page. And we're going to click SynthEyes Multi-Peel. And this will bring up the Multi-Peel script. It is a Synthia script by Matt Merkovich. And it's quite interesting to see this. We're going to click download.
And go ahead and download this. And we're going to open it up to get an idea of how it's set up. So if we open up the the script, we can see that it's actually written in plain English. SynthEyes has a couple of different scripting systems. The default Autoshot script is written in SizzleScript, which is a, a more regular scripting language for handling mathematical work.
And this is Synthia, which is more of a natural language interface. It has a voice interface, And we don't really need that but this text interface is actually quite useful. And so you can see we're going to make the current camera's small blip density 0. 8, 0. 04, and we're going to tell it to do blips all frames, and we're going to peel all buttons. We're going to change the blip size to 12 and 24. Again, we're going to do blips all frames and peel all. And so it's going to go through four separate sets of blip sizes and increase each one on each size, and then peel all the trackers.
And this actually is going to automate a task that would otherwise be pretty tedious for us to do. So we'll go back to SynthEyes, and the next question we have is, how do you install that? And so we're going to go to script, and I'm going to open our user script folder. And this will open up the script, and we'll just drag and drop our multi peel script over here. Now, it actually, I already have one there, so I'm going to skip this for the time being. Normally you would just do that.
And once we've got it loaded, we'll go to file and pick find new scripts and that will reload the scripting directory into SynthEyes. And now we can actually use the script.
## Using Alternative AI Rotomattes
So before we do this, we're going to go over to camera. And we're going to see, look at the Rotomask. In this case, we used, instead of InSpyReNet, we used PPMatte, which is available by default on both Windows and Mac.
And it has some advantages in this case over InSpyReNet. InSpyReNet is much more precise, but it acts on the foreground objects. Usually it picks the person, but sometimes, in this case, it picked the exercise mats. PPMatte and ModNet are both designed to do human segmentation, so they only recognize the people.
So they are automatically rotomasked out and sometimes it misses background people, so we may have to delete a couple of frames from that, but we can work that out.
## Running Multi-Peel Script
And we're just going to go to script, and we'll go down to a multi peel MattMerk style and run that.
And that is going to drive SynthEyes through a series of calculations. I can move the, it's going to pop up a little Synthia window. We can move that over for the time being. And so it's running through the frame calculations and it's going to first start with the smallest feature sizes.
It's going to go through all the frames and 'compute the blips' in SynthEyes speak and look for all those trackers at the smallest scale. And it's going to go through this and repeat the process for the small, medium, large, and extra large trackers that we described in that file.
Okay, and so you can see that it has detected very tiny features. These little green diamonds. Those are the smallest feature points that it detected. And it's going to go through and detect the the smaller ones. And now it has slightly larger green diamonds. Now I'm moving on to the medium sized features.
All right, there's another set of larger diamonds. And now we're going to the largest features. I'll go through and sweep through these 200 frames and detect those features. All right. Those are the extra large features. All right. So now as we scroll through, we can see that it is has detected all these features in the shot.
## Picking Initial Survey Frame
And, we're going to go look through, and, sometimes the tracking shifts around a little bit in the middle of the shot, and that's fine. When we're doing this retracking work, the nice thing is we only really need to find one frame ideally fairly early on where our tracking lines up correctly and where there are plenty of features that are located on top of the mesh.
And remember, we are going, we're going to use the track drop onto mesh feature to set our survey data for this. So I'm going to move over here, find a spot where we've got a few different markers. There's a few on both the floor and on this background stack of cardboard and that should be pretty reliable.
## Picking Survey Markers
So we're going to click between the mesh, drag around, find all of our feature points. I'm going to pick them here. I'm not going to pick them on here because I think this set of mats got shifted between when it was scanned and when the shot happened, so I don't think that's going to be reliable and I'm going to hold down my shift button, and I'm going to click and drag around these here, and I'm also going to hold down my shift button and drag around a few over here.
Over here on the wall, I'm looking for just ones that should be at reliable distances that are scattered somewhat around the frame so we have a reasonable combination of tracking points across the frame. So I'm going to do track and drop onto mesh. And maybe I'll grab a couple more of these.
I'm going to hold down shift and grab a couple of the, more of these tracking points that I think are going to be nice and reliable spots and track and Drop on the mesh.
## Solving and Refinement
Okay. Now we can go to our solver and Go to here. I'm going to click go and our solver is going to crunch away and it's going to Start doing its iterations There we go.
All right, so it did an initial solve, and we have a 2. 6 pixel error, which is a fair amount of error, but we're going to reduce this. So the most important thing is, you can see that we've retained our mesh alignment. You can see that our mesh is still pretty well aligned with the floor. And that we can see where the mesh transitions up to match with the wall, wall pieces.
And it looks like this is actually reasonably well aligned. So great, so now we're going to do our usual Shift C, get rid of our bad frames, and our high error trackers, which there are 551 of them. And since we did that complete sweep between our small, medium, and large trackers, we can get rid of quite a few trackers, switch to refine, and our solve will still work.
All right now we're good, and we're down to 1. 2 pixels of error, and again, I'm checking to make sure that our that our mesh is still reasonably well lined up. Again, our floor contact, our floor intersections look like they are largely correct and yeah, that looks like it's pretty good. I'm going to hit shift C and remove 18 more high error trackers, and click go to refine that again.
There we go. And we're actually just going to keep removing these kind of higher trackers. And we're just going to eliminate the worst offenders first. And, just make sure as we go through this that we're not, we haven't deleted too many trackers, and that we're still retaining our alignment with our 3D mesh.
As you can see, our camera is still staying in the correct general location. So let's let's Hit shift C again, and now we've run out of our high error trackers at 30 pixels of error. We're going to start lowering this down to 20 pixels of error. And until we start to find some high error trackers going to fix that hit go to refine it.
And we're just going to keep removing the worst worst defenders and refining our solve. And going through it and dialing down our error. And we're down under a pixel. And go. All right, and things are tightening up.
## Solving for Lens Distortion
And we can actually, in here, we can actually start calculating distortion.
And once again, we're going to pick our standard radial fourth distortion enable our C2 and UV, V2, U2 and V2. And calculate that. And that's dropped our distortion. Now it's going to start iterating. All right, we're down to 8 tenths of a pixel. And maybe one more, one more run of our bad trackers.
Okay. And we can actually keep going through this and working through it. But you get the idea. The key important thing here is that as we scroll through, our mesh is staying reasonably aligned with our 3D scene. We can see that our mesh is, that our floor is tracking correctly.
Along with the mesh.
There we go. And we can hit play and walk through that. So we actually have it looks like a reasonably tracked shot with about three quarters of a pixel of error. There we go. Our active trackers are not on the people. They are around them, but not on them in general.
All right. That's a good example of fixing a shot where you calibrated and you were operating on set and then something got knocked out of whack and now you need to fix it in post. But fortunately we can fix it in post in in a short period of time if we have that scan and the Jetset data.
# Tracking Refinement in GeoTracker for Blender
Source: https://docs.lightcraft.pro/tutorials/tracking-compositing/geotracker-blender
Jetset’s on-set capture of 3D scan geo, camera motion, and cine lens calibrations makes a perfect match with KeenTools’ GeoTracker for an incredibly easy way to refine on-set tracking data while remaining inside Blender. This tutorial is a game changer for anyone interested in moving their camera.
## Video Tutorial
# Transcription
This is a short tutorial that nonetheless helps change the tracking game. Jetset tracking is good, but it's not a sub pixel track, and in fact no real time tracking is really a sub pixel track. Alignment wise, it's too hard to get things to line up.
So we want to have a refinement pipeline that can take our onset tracks all the way into a high quality sub-pixel post production track when we have ground contact like this. In this case, we have a very visible ground and floor contact. We have an existing refinement pipeline with SynthEyes, which is very powerful, but it's very complex. And that's frequently overkill, when you just want your shot to stick.
## GeoTracker Tracks Different
Today we're going to look at GeoTracker. And GeoTracker is a special Blender add on from Keen Tools. It enables a geometry based tracking refinement workflow. And that's different than what you've traditionally seen in most tracking.
Most tracking is a feature based workflow. Track a bunch of features in the scene, and you tell the solver to solve, and it works out the solve. That's how Blender's Internal tracking system works, and it's how SynthEyes works, and how most trackers work.
And GeoTracker's different. GeoTracker is actually using the geometry of the scene, as well as the optical flow analysis of the video, and coupling those together. And it makes for a very interesting and powerful workflow that's also really easy.
## Jetset Output = GeoTracker Input
To make this work, GeoTracker needs two things. It needs to have a piece of geometry that matches a part of the scene geometry and it needs to have an initial alignment. And so one of the great things about using Jetset Pro and Jetset Cine that have LiDAR scanning built into it, is that they provide both.
And this makes for a very fast, very simple tracking workflow. So let's go ahead and actually walk through that.
## Creating New Shot in Autoshot
\ Let's start off as usual in Autoshot.
We're going to create a a new shot to start off with using our new shot naming workflow. Go to shot, add shot, and we can name this shot 01.
All right, and we're going to click add. Okay, and so now here you can see that we've added a shot to this, and what the shot naming workflow does again is it lets us extract multiple shots from a single take without overriding any names so we can just name your shots whatever you want to name them.
We've already done our Cine Calibration, we already have our Cine Footage Match as we've seen in other tutorials, we're going to pick Blender as our export tool.
And, for right now we can just use the empty comp starter, that'll just give us an empty Blender scene. And we're going to want to set in our clip in and out frame, so we can do 1200 and 1250. And this was shot on a Sony S Log3, S Gamut Cine.
And we're going to use our InSpyReNet roto model cause that gives us a really high quality roto model that we're going to use for masking in this case. And we're going to go ahead and save and run, and we'll let this go through and extract our frames.
And now InSpyReNet will start up and generate our AI mattes.
## Blender Scene Setup
Okay. Autoshot has just finished generating a Blender shot file and, since it's one of the empty comp starters, let's actually turn off the viewport compositor. And by default we bring in our footage on an image plate with the footage added as a material.
And that's fine for most cases. In this case, we want to see the image through GeoTracker's interface. We're just going to go ahead and highlight our image plane and go ahead and delete that.
And then frame up again.
## GeoTracker Setup
Now what we're going to do is make sure we have Keen Tools installed. So preferences, add ons, and we can type in Keen Tools. So I've already gone through the installation process. Keen Tools has a really good overview of this on the site.
So I'm on a floating license over my license server port. And so that is all working. And then when I come over and hit N I can come down to GeoTracker and I can create a new GeoTracker. So let's go ahead and do that. And it's going to highlight this as red. It's asking us for a source clip. So we're going to click this, and we'll just go up one directory level, and then we can see that our shot, our CineCam EXR files are in this directory with the suffix shot001. That's, there we go. And highlight all those, and load the clip. And there is our tracking footage.
## Setting Sequence Start Frame
At first you think something's going wrong. If we scroll back and forth, we see the footage does not match the camera tracking data. And the answer is found down here. KeenTools has automatically set our playback to start off with 1001.
And that is because our images are named starting with 1001. By default, it set our start frame in the Blender timeline to 1001. We actually don't want that. Our Blender timeline is starting at 1. So we're just going to hit the number 1. And now things align correctly.
## Setting Color Profile and Analyzing Clip
Then we're going to set our color interpretation to ACEScg since by default Autoshot generates all of its EXR files in ACEScg formatted file with AP1 primaries. So now we have the shot coming in, and the color's correct, and it looks like it should, and things are lining up.
Now, this is still with the original Jetset tracking data, so it's good, but it's not a subpixel lock but that's what we're going to be going through in this tutorial. So now, what we're going to want to do is we're going to want to analyze this. First of all GeoTracker needs to do a optical flow based analysis of the scene.
We're going to first pick out a place where we're going to put in our file. We can just go, once again, go up one directory, right click, and make a new folder. We'll just call it Geo. And we can just use the pre existing file name.
Just click an analysis file. Okay, and now we need to analyze it. So we're going to click analyze. Starting with frame 1, going to frame 51, and click OK. And now what it's doing here, is it's not tracking yet. It's actually going through and mapping the optical flow for each pixel in the scene , as it goes through the shot.
And this is one of the neat features of this, because we're depending less upon individual feature detection. As again, most trackers work with. And now we're just working with optical flow. Now it's going to analyze backwards through the shot again.
Again, calculating the optical flow forwards and backwards.
## Camera Parameters
Now we have our analysis. And now we can actually, we can open up our camera. And what we see here is we currently have a fixed lens and we have our focal length at 23. 94 millimeters and our sensor width at 24. 9. And these values are coming in from our Jetset calibration, so we don't need to change them. It has already populated the camera with an accurate default lens setting, which is important as we're going to see for setting our first keyframes.
And likewise, down under tracking GeoTracker can be set to, as its name says, track either geometry, or we can actually track camera. And it's the same math, it's just simply a choice of whether you have a stationary camera and are tracking moving geometry or stationary geometry in our case and are tracking a moving camera. We want to track the camera.
## Setting Alignment Keyframes
And we're going to go ahead and start pin mode. And this is the alignment process inside GeoTracker. The more standard GeoTracker workflow is to pull in a model and to set its location with pins and then to set a keyframe. Now, the great thing with the incoming Jetset data is that it's already pre aligned.
We already have the correct model, we already have the correct alignment. All we have to do is drop a keyframe. So I'm just going to open up This little bit in our timeline so you can see what I'm editing so it becomes more clear. We've got our camera selected, and here's the existing keyframes from the Jetset track.
So we have both keyframes from our object transform for the camera, and that's our position and orientation. And down here we have keyframes for the camera aspect of the camera, right? So we have the object of the camera and then the camera specific aspects of it. This is just how Blender stores camera information. So down here under the camera specific info, we've got the focal length, et cetera.
And KeenTools has already automatically picked our geometry, which is our mesh and our camera. Now we can actually come down here, and we're on frame one, and we're just going to delete all the forward looking keyframes from there. So here we've deleted our object transforms, and we kept our camera, our focal length, we're not changing that. But now, all of a sudden, if I move the camera through the scene, you'll see the geometry is no longer moving because we've removed all of our camera keyframes except our initial position.
## Tracking The Shot
All right, so we just set a GeoTracker keyframe here because things are aligned and we're going to tell it to track forward. So now GeoTracker is tracking forward with this initial pose estimate from Jetset, but now it's using the optical flow data along with the geometry data that we entered and told GeoTracker to use to track the shot.
And so it's going to do so pretty quickly. And there it is. Everything is moving correctly. This is great! We just tracked a shot inside Blender. The mesh did not move. We tracked the camera to move relative to the mesh. And so our camera is still in the same coordinate space.
## Using AI Generated Tracking Masks
In this case, we did not mask out the actor. And it tracked fine even with doing that. Which is really impressive, because usually on a feature based tracker you have to use roto mattes on the actor. Otherwise the actor generates a lot of features and throws off the tracker.
And, KeenTools didn't blink. Now, there are cases where you do want to mask out the actor. And fortunately, we generated AI mattes for this. So let's go ahead and learn how to do that. We're going to go back to frame 1. And we're going to go ahead and delete our keyframes again.
So no keyframes. So we, yep, so the geometry is no longer moving. We'll go back to frame 1. And we're going to go down to masks. And KeenTools has a number of different mask types it can use, surface masks, compositing masks that you can build in Blender. We actually just want the sequence mask, because we have an image sequence.
We're going to click our folder icon, go up one level, and under AI mattes, shot 001, we have a conveniently generated set of AI mattes. And I'm going to load our mask sequence. Now, wait a second, we don't see it. Why don't we see it? It's the same sort of thing. Autoshot automatically names the frame starting with 1001, a VFX standard.
And GeoTracker assumed that we wanted to have our start frame also at 1001. But no, in our timeline we have our start frame as 1. And, so I'm going to enter that. And then all of a sudden, here's our mask.
And as we move forward in the shot, you can see the mask is following and tracking the person. Now you'll notice the geometry is not tracking, so that's what we're going to do next. We'll go back to frame one, where our alignment is correct.
Now we can go back up, add a keyframe, and we can tell it to track forward. And it's going to go track forward, and add in all the keyframes that we want for the position and orientation of the camera.
And you can see that GeoTracker is using the AI generated mask to ignore the motion estimation under the actor. It's only looking at the rigid body transform of the actual scene. All right, that was quick. Adding the mask slows down GeoTracker a little bit.
So if you don't need to use the masks, you don't need to, but in some cases it'll be very useful to have that. So once again, we have a nicely tracked shot and now you can just work forward in your scene, render out and have an accurately matched image, or we can actually do a few other nifty tricks here. So I'm going to, going to click back to 3D. So we're out of the pin mode now, and we're viewing the normal Blender viewport.
## Texture Projection Setup
You can actually use GeoTracker to project textures onto geometry from multiple angles. And this can come up really useful if you're trying to make a clean plate for something, or if you just want the texture projected onto geometry. And we already have geometry that closely matches the scene, which is our scan data from Jetset.
Let's see how to do that. Before we jump into anything else, we want to make sure that our projected textures are displayed.
By default, Autoshot sets up the mesh to be displayed as a wire frame. So you can see how it overlays the footage. So if we go over to our object panel, and we have our mesh selected, and we can come down and look at our viewport display, and we see that is displayed as a wire.
Now, in order for it to display textures, we need to tell it to display as textured. Okay, so then all of a sudden we see our we see our tracking mesh displayed with a kind of a gray scaled texture. Now, let's take a quick look at the textures over there. If we look at our material, and I'll pull this down, and this is the scan that's coming from Jetset.
## Removing Scan Materials
And so the way that works is that as you s you map around the room, it's actually automatically adding a bunch of different pieces of geometry and materials. So what you end up with is a ton of different materials with a gray shader on it, just for one piece of geometry.
And if we leave these on here, we're not going to be able to see the texture . So we can just come up here and and I, the minus button was hiding, so I just clicked up here and then down here. Now I can see the minus button. Start up at the top and then go ahead and click the minus button a bunch of times. And that gets rid of all the the extraneous materials that are, that were applied to this. Okay. So now we've removed the unneeded materials.
## Creating UV Maps
So back to GeoTracker under our texture tab, it's telling us that it is missing UVs because the scan data from Jetset's LiDAR doesn't generate UVs by default. So we are going to tell it to create a Smart UV. So it created a smart uv. And what that is doing is if we click under UV editing you can actually see that it's made a UV map for the scan.
And the scan's going to be a little bit rough, so our results will be a little bit rough but it'll be enough to see what's going on.
## Choosing Projection Frames
And If we now go under texture, now we're tracking through the scene. And we can pick out the frames that we want to sample texture from. And it's going to project from the camera view.
Remember we've solved the camera point of view in GeoTracker. It's going to project the image that it has for that frame onto the geometry. So over here, we can have a pretty good view of the geometry. So we're going to, we can add a frame 51. And if we wanted to come back over to the side, we can add, frame 31 and come back over here and frame 13 and maybe back to frame one.
So we'll add all those frames as sampling points, and then we can actually tell it to create the texture. And it's going to give us a warning that it has overlapping UVs, and that's just a factor of having data from these real time scanners.
Sometimes the UVs are overlapping. So we can just click OK, we'll project that, it's going to work through, and bake. All right, so it has gone through and projected and baked onto that material, so let's Come out to the side. So we can see that it has projected onto our 3D geometry from multiple viewpoints.
And of course the camera was only on one side so the other side is in shadow, so that's not projected. But you can see that the projections are matching pretty well to the underlying geometry. Things are lining up. And you can see how the different overlapping projections have different, slightly different incident lighting.
But they're all lining up. So this is a nice way of making if you need something like a textured piece of geometry then you can achieve that pretty easily.
## Rendering Wireframe Animation
Now if we want to check the quality of the shot and to see how well it's sticking, then we're going to want to render out a preview. And it's frequently convenient to see the preview with the wireframe overlaid on top of the geometry.
So if we want to generate a rendered preview then we actually come down and click on the render wireframe and we can once again go up to our geo folder and type in shot one
The controls for the display of the wireframe while rendering are actually in the render output panel. So this is important to note. If we want to, for example, increase our line width to a larger point, that control is only over here. If we want to change our color to something else other than bright green, this is the area where we would actually adjust those controls. We just go ahead and click on Render Wireframe.
And it's gonna start chugging through the frames to render it.
## Viewing PNG Sequence
Okay, it has rendered out a series of PNG images, and we can actually go next to our take. We'll just click open and That will show us our directory of all the subfolders in that take. Here's our geo subfolder So we can see those are the PNG images and if we want to be able to actually see those in a real time player I'm just using mrv2, but you can use your player of choice Open a sequence And paste this in.
And there is our sequence. We'll tell it to open. There we go. So now we can look back at the playback, and we can see that the plate is sticking really nicely to the wire frame which is exactly what we want.
## Workflow Wins
There's a lot of wins here. The first win, of course, is that we didn't need an external tracking app. We got a really tightly locked shot right inside Blender. And we also didn't need to move the mesh. One thing you have to watch out for when tracking is that the camera tracker can throw out position, orientation, and scale information, and you have to go guess and recreate that.
And we didn't do that since our underlying wireframe mesh didn't change position, and we tracked the camera with reference to that mesh, then we maintained the position and orientation of the camera that we had in Jetset all the way through post production, and only changing that a tiny incremental amount to get a sub pixel shot versus the on set shot. And so if you have 50 shots in the same space, then everything lines up.
The other win is that we are less dependent on tracking individual features. Since they are doing the optical flow over the entire block at once, we didn't have to worry about a feature going behind someone's leg or going behind a pole or something like that, which is normally an enormous amount of work to fix if you're depending upon individual tracked features in a more traditional tracking application.
## Limitations
Now there's some limitations as well. GeoTracker assumes undistorted footage. And in our case, Our footage didn't have that much distortion in it, and so it tracked fine. This will work fine for quite a few shots. And if you really want to use the traditional undistort, render oversize, then redistort pipeline and compositing, you can certainly do that.
We have that working in SynthEyes. But for a lot of shots where you just want your track to stick, and you're not dealing with a wide angle lens with a lot of very visible barrel distortion, this will just work.
## Bridging the Gap
So this is an unusual bit of serendipity In that the on set tracking data generated by Jetset, the camera position, orientation, cine calibration, and the scan data, turns out to be exactly the input data that GeoTracker needs to generate a finished shot.
So this is great. Jetset builds part of the bridge, GeoTracker built the other part of the bridge, you put them together, and all of a sudden you can track a lot of shots of complex moving 3D camera work without actually having to go into one of the traditional tracking apps.
## Nuke Workflow
In addition, KeenTools has GeoTracker built into other hosts. We've been working with GeoTracker for Blender, but there's also a GeoTracker for Nuke. And the exact same data that we're bringing in, which is the camera original, the initial camera track camera optics the lens calibration, and the scan data that we brought into Blender is exactly the same data that we bring into Nuke via the Autoshot Nuke exports. So we're going to go through and do the same tutorial in Nuke using the Jetset input into GeoTracker.
## A Complete High Speed Tracking Pipeline
So this is really exciting, so now we can bring this high speed workflow, taking on set tracking data all the way into post production without throwing away data or starting over. If you've been wanting to do 3D tracked camera shots, but have been scared of the time and difficulty of doing it in a more traditional manner, this is how you want to start.
# Tracking Refinement in SynthEyes
Source: https://docs.lightcraft.pro/tutorials/tracking-compositing/tracking-refinement-syntheyes
Jetset’s tracking and scanning data capture can be used along with AI roto mattes to dramatically increase the speed of post production tracking refinement, including overscan rendering in Blender and matched distortion nodes in Nuke.
## Video Tutorial
# Transcription
In this tutorial we'll go through our tracking refinement process with SynthEyes. We'll also show how to integrate InSpyReNet, a state of the art AI rotoscope matting generator, and we're going to use that to avoid automatically putting tracking markers on the people in the shot, because that'll mess up the solver.
We'll take this all the way through Nuke and Blender, including an overscan workflow and bring it back. So let's start off.
## Installing InSpyReNet
First of all, we're going to download InSpyReNet and I'll put the link below.
And we'll put that into our downloads. This is a, quite a large AI model to do automatic rotoscopes, but the quality level is very, very high. So this could also be used for exterior work, where you're trying to rotoscope out a person in the middle of an environment. So we'll let that download for a second.
Okay, so that has finished downloading, and we're going to click here to open it.
We're going to want to unzip this into a particular directory. So I'm going to right click, and we're going to extract all. And we're going to browse.
And we're going to go to our program files.
iCraft Technology. We're going to extract into here. Go ahead and click extract.
Provide permission.
All right, and let it extract. It's going to be quite a few gigabytes as it's containing the AI model weights.
## Verifying Installation
And so just to double check to make sure that it's in the right spot, we're going to go over to C:\ Program Files. And, Lightcraft Technology, and you can see that in the Lightcraft Technology folder, it should be at the same folder hierarchy level as Autoshot and Autoshot Tools. And this way, InSpyReNet will be recognized correctly when we start up Autoshot.
So now we can start up Autoshot. And if we go to our AI Roto model, We can see that now it includes InSpyReNet.
## Take Color Space
We're going to process this take from the barn shot, the barn shoot. And we only want a couple hundred frames of this. So I'm going to go from start in frame at 1150 and out frame at 1320. This was originally shot ProRes RAW and then transcoded to ProRes 4444 XQ with the Sony S Log3, S Gamut Cine color spaces. So just to make sure we pick Sony and we're going to use our S Log3, S Gamut 3, or S Gamut Cine color space.
And it's going to be converting that over to ACEScg. And we're going to use our extracted EXRs and we'll just do our first test in Blender so you can see how it's going to work and let's click save and run.
## Generating AI Roto Mattes
We have already extracted our EXRs and in addition what's going to happen is InSpyReNet is going to start calculating the AI Roto images for this. And that's going to take actually a little bit of time. Here it's going to first start loading up the model weights to run the AI processing.
And that can take a few minutes actually to load up the model weights.
Okay, and here you can see it's going to start processing the frames. We can see it's at 0 percent right now. And as it processes the frames, you can see the increment of the frame number out of the total number of frames and the percentage completion. So we'll let that go and complete our 171 frames.
And it's working at about a frame and a half per second.
InSpyReNet is the slowest of the AI matte generators that we're using, but it's also the highest quality. It only works at this point on Windows. On the Mac Autoshot platform, there are two other AI Roto models, ModNet and PPNet.
And those, those are both faster, but the segmentation is not as, quite as high of a quality. So we wanted to show you what InSpyReNet is capable of doing.
## Blender Render File
Here is our generated Blender render file. So as we scroll through, we can see we have a moving camera and we have the moving image sequence. And because we generated the AI rotomattes with InSpyReNet, we are automatically using those to cut out the actor from the background.
All right. So this is good. This as we know, has the Jetset real time track in it, and we're going to replace the Jetset real time track with the refined track a little bit later on. All right, so we've got this. Let's go ahead and click Save and save it.
## Generating Autoshot Scripts
And now we're going to go back over to Autoshot. And we're going to go from Blender and we're going to move to Others. And we're going to go ahead and keep the same values. Click Save and Run. Okay, so it went through and generated all the different scripts that we are going to need.
And the first one we're going to want is the SynthEyes script. So we'll go ahead and click on this, and that will just open the directory that the SynthEyes file was created in. So here's our SynthEyes file, and we're actually just going to want to click up here and grab the path control C to copy and go over to SynthEyes.
## Running SynthEyes Script
And now we can click a script and run script and go ahead and come up here and hit paste.
And that way we know we're in the correct directory. Then we can pick that Sizzle script and open it. All right. So it imported it and we can just click our camera view. And here is our imported shot. Has all the different, has the individual frames and has the the Jetset based camera tracking data, as well as the Jetset generated 3D scan of the scene.
And we're going to use all of these to make our shot. Okay. First thing we can do is we can click Rotomask because this is the automatically generated AI roto matte that InSpyReNet generated and that Autoshot automatically put into the scene. The file and that will prevent trackers from being placed on the person.
## Set Color Space
First thing, let's click P for the image preprocessor. And let's set our color. We're going to use a ACES to 709 as we're using ACES CG files as a standard. Click okay. Before we go any further, let's go ahead and click file and save as, and we're just going to go up a directory and back to SynthEyes.
The script automatically saves a SynthEyes file. We just need to click on ,it and click save. That's fine. And that way it will automatically save to the correct location.
## Syntheyes Feature Settings
What we want to do next is we want to actually get as many good points as we can. And we could just run the autotracker, but that'll just run it with the default settings that we set in the script.
We want to actually go through a few different settings, and I'm borrowing this technique from yet another of Matt Merkovich's SynthEyes Tutorials, and if you have the thought that I'm simply going through all of his tutorials and borrowing the techniques, that would be correct! Ha, ha, ha.
So let's go to Features, and we're going to click Advanced, and we're going to do a set of small, medium, and large. So I'm going to start off with 12 and 24. And this is actually the small setting, because we're on a 4k plate the pixel counts slightly larger.
And I'm going to click Auto Re-Blips. So you can actually see which things are being detected when we do this. So here we have all these will be detected on on this round. And we're going to click blips all frames.
All right. And. Then we're going to click Peel All, and so that got a set of trackers for the small features.
We're going to go up to a medium set of features, and we'll just do 16 and 32. And once again, we're going to do Blips All Frames.
And then Peel All.
And what this process is doing is that Blips All Frames finds all the the features with these sizes that are good features to track through the range of motion, and then Peel All grabs the best of those and keeps those as trackers.
And we're going to finish off with the large ones, which is going to be 20 and 40.
And once again, we're going to do Blips All Frames.
And then we can do Peel All.
Okay, so now what we have, and I'll we can exit our feature control panel, and I'll turn on, on our display, we can turn off our meshes for a second, and we'll turn off our image display, so you can actually see what's going on. And actually, we'll move into our tracker tab.
So you can actually see, we have a set of small, medium, and large features all being detected. Now these are all still 2D features, and we're going to need to get 3D data from them in the next step.
## Creating Survey Data
So we're going to display our, turn it back on our image and our mesh, because the next step we want to see where we have a scan mesh.
We want to have several fairly stable trackers to work from. So I'm going to pick out a wide area of the shot that has trackers spread out over the areas that we want. And every once in a while, when you see a tracker that's doing strange things, just go ahead and delete it at this phase.
Because it's going to cause, it can cause problems later on. It's like this tracker here that's, that's following his foot. We can just go ahead and delete that.
We'll get rid of more of them later on, but right now what we're looking for is a place to set our scale. And so we want several trackers that are acting stably over a range.
So this, this looks all right. So we can actually just grab a few of our, a few of our trackers.
We can actually just lasso select a few of these. And we can click track and drop onto mesh.
And what that has done is converted those into survey tracker marks. And that locks the 2D position of that onto the 3D location of the mesh. It doesn't completely lock it, it's kind of a soft lock that SynthEyes uses, but it'll be really useful for the next phase.
We can only drop onto the mesh in places where we have a scanned mesh. So that's why I picked this area of the floor to drop onto mesh We don't actually have any mesh on the the vertical background so we can't do it there.
## Initial 3D Solve
Once we've dropped the markers onto the mesh, We established those as survey data, we can go over to our solver And we can click go and it crunches and it has solved our camera in 3d space.
## Enabling Tracker Radar
One thing we're going to want to turn on to give us an assist in this next step is our radar.
So you just right click and do view and then show tracker radar. The tracker radar is these red rings that show the error for a given tracker on a particular frame. So some of these you have just huge error, right?
## Refining the Solve
So what we're going to do real quick is just go Shift C.
And we're going to just remove our bad frames and hit Disable instead of Clear. And we can just click Fix. And we'll click, change this from Automatic to Refine. And then we can click Go. And what this is doing is it's refining the solve. It's, it's keeping our 3D orientation, but now we can focus on getting rid of the trackers that are causing lots of error.
So if we, we click here to deselect all of our trackers, we can see as we go through some of the trackers have a pretty large error, and the easiest way to, to catch these is to hit shift C, and just keep getting rid of the high order, high error trackers, and hit re, and re solving. And sometimes as you're going through, one of them will be particularly egregious.
You just, you can just manually delete that. We're down below a pixel of error. Let's hit shift C, and here we can actually start coming in and lowering our threshold for our high error trackers. We're going to go down to, say, 18, fix those, and click go, and again, we're still in refine, so we're just systematically kind of walking down our error by getting rid of all the, the, the worst trackers.
And refine, and we have a lot of trackers, so we can actually get rid of quite a few trackers that we're not, we're not breaking our solve. All right. So our solve is mostly green and we can kind of go through and look at, at some of the different, different pieces of it. I'm just going to hit this and get rid of a few more of the errors.
And we're, you know, getting down to a reasonable level of, of error here. So shots, mostly green, and we still have plenty of trackers. Let's just look at one of the areas where we still have some yellow here. See if there's any trackers that are just really leaping out at us. I'll hit shift C again and get rid of a few more of these, the higher trackers and we're down to, okay.
So the shots basically green at this point. So we have a three D solve and trackers that are, that are working out pretty well.
## Setting Sensor Width
This is a good place to set our sensor width. With the current workflow the sensor width is set during lens calibration in Autoshot. And so this should already be set. This is an older take, so it doesn't have the sensor width integrated into that.
So either way, if it, if it's wrong, you can fix it here. And so you can click this to open shot, and the sensor size of a Sony FX3, the width of it is actually 35. 6. And as we click here, you can see that it's automatically adjusted the sensor height to match the image size.
And we'll remember this 35. 6 millimeter dimension for later, when we start transferring things to Blender. So we can click OK.
## Verifying Focal Length
Now we can see that our focal length is about 34 millimeters. If we look back at the shot in Autoshot, the Cine calibration is for a 35 millimeter lens. And so that's, that's a good way to verify that you're actually reasonably close to what the actual lens was.
If you have put the correct sensor back in and solve, then you'll get a focal length that's reasonably close to what you had marked on the barrel.
## Solving Lens Distortion
Okay, so we want to actually then start to solve for lens distortion. This was shot with a slightly wide angle lens, a 24mm lens, so it has some distortion. So we can go ahead and click Calculate Distortion, and click Go to refine. And it hasn't changed that much here. It's a fairly subtle amount of distortion in this.
SynthEyes now uses a much more modern lens distortion system. So instead of classic, we can switch to standard radial fourth order. And this exposes several parameters here. We don't want to just turn them all at once. The solve in SynthEyes is iterative. So we want to turn on a parameter and calculate it and see if it's dropping the dropping the solve values.
We'll turn on U2 and V2 and calculate those. All right, so it's dropped off the solve a little, the error a little bit and calculate C4. And that's that's actually increased it a little bit. So we may not need to do that. So just go back to manual, set that to zero and recalculate. Okay, so we are in the green.
Okay, so that is probably sufficient for our lens distortion calculations.
## Adding Chisels
So each one of these points is now solved in 3D space. To make it easier to see how things are really aligning, we're going to add some geometry through the scene.
Since we are aligning the top edge of the platform here with our CG later on, then we want to have our markers where we're going to be aligning the CG to. So we can pick some of our markers and pick the ones that are aligned here.
It's a little hard to see, but they're closely following the, the top scan of the surface. Now we're going to go to Script and we're going to go Mesh and Duplicate Mesh Onto Trackers. We're going to set our chisel size to five and leave this part empty as no name will produce chisels and we'll click okay. All right. So that generated eight meshes onto there.
## Hiding Scan Mesh and Rendering Preview
So there are our chisels and we're going to use those to be able to kind of see how things are aligning. One of the last things we're going to do is once we have our ground mesh,
We can go ahead and go into 3D and just click hide that. That way we won't, we won't see it. It's a good time to save.
Let us do a render so we can make sure that our 3D points are sticking correctly. To do that, we're going to do a render preview. So we're going to go to shot and then save sequence.
And we can just set our file name. Once again, I'm going to hit control V to save to put us into the correct SynthEyes folder. And then this default file name is fine, click save, and we're going to include the meshes. Let's change our compression settings. And, we can go up to 20, 000. All right, let's click start.
There is our shot going through. So we can see our chisels rendered on top of our CG object. In general those look like they're staying put. Okay, that's good. So I'm going to exit out of here. All right, that's a good time to save.
## Setting Lens Distortion Workflow
Before we export to Blender, we need to run a lens workflow and this will generate our overscan, and we'll talk about what exactly that means. So let's go ahead and click our lens workflow.
The way we're going to work is to render an image in Blender with overscan, and then we're going to re add the distortion to it back into our Nuke composite.
So, we're going to use the number two lens workflow. We're going to use a 5 percent margin so we can have a standard margin here. And we're going to set our maximum rounding error as 1. And that will prevent too large of a border from being generated. So then we can click OK.
What it's going to do is crunch through and undistort the image, and pad the outside of the image, there we go, to handle the undistortion level. So, we only have a small amount of distortion in the image, and that's why we used a 5 percent margin.
But you can see the image has now been undistorted, and it's stretching slightly outward in the corners. That's because we had a slight amount of barrel distortion, and then when we undistorted it, it pulls it slightly outward. And this black border on this is the 5 percent margin that we entered.
Now, an important thing is, let's go look at we'll hover over here, and we can see that we have our original sensor size, 35. 6, but now below that we have an effective size of 37. 8, and we're going to want to remember that number because that is the sensor width that we're going to send to Blender. That is how we're going to handle overscan in this workflow. All right, so that, now that's working. Hit save.
## SynthEyes Blender Export
So now let's go to our Blender export. We're going to go to file and export, and we're going to go to Blender Python, and we can just accept the default name. Let's go ahead and highlight here, hit paste. And we can just leave this in, in SynthEyes for now, in the SynthEyes directory. That's fine. We're going to click save.
There's a few setup things we need to do here in the exporter. First thing we want to do is we want to export the entire shot, and we're going to go with starting frame of 1. And lots of different ways to do this, but this is how we'll do it for now.
We're going to use the Blender version 4. 0 plus, because we're on Blender 4. 1. We're going to create a new scene, and we're going to rescale the scene to .01 because the data in SynthEyes, as you saw was in centimeters, and in Blender it's in meters, so we need to rescale the scene to 01.
And we'll set our tracker size to 5. We're also here going to want to open up in Blender automatically. And to do that, we need to find our Blender executable. Fortunately, we can go just back to SynthEyes and go back to Blender. Pick our executable here. Grab our path. Let's cancel out of there.
Come back to come back to SynthEyes and come in here and highlight this and control V to paste it. And that will take it right to the Blender executable. Now we can click OK. And it's going to work on it for a second and generate a whole new Blender scene.
All right, here is our tracked Blender scene. So here is the SynthEyes generated scene with our 3D markers and our chisels. And I've left the chisels in here cause we're going to use those for reference. All right, so now we can just go ahead and click save and you can just save it into the blend into the Blender directory as a tracked camera, this is a temporary file. So it doesn't matter what we name it. So I'm going to save it.
## Disabling Collections
Now we're going to go back to our other Blender file. Here is our original Blender render file. And before we bring in the SynthEyes information, let's turn off some things to avoid confusing ourselves.
This initial collection has the Jetset tracked camera in it, as well as the image plane. We're not going to need that. So we're going to turn off that collection.
This is the scan collection. And so if I highlight the scan that has the, that has the the geometry from the LiDAR set. We're actually going to bring in a, our new geometry along with this from SynthEyes, so we don't want this either.
So I'm going to turn off the scan collection. I'm going to highlight scene collection because when we append the SynthEyes scene, we want it to come in under the scene collection. All right. So let's go ahead and save that.
## Appending Refined Camera to Render Scene
Go over to file. And we're going to append our tracked camera file that we just created and we're going to pick a collection and we're going to pick the SynthEyes collection. We're going to append that. And here they are! They appear inside our scene in the correct orientation and almost the correct position. This is the SynthEyes tracked scene that is now in our main scene.
## Rough Alignment of Tracked Camera and Points to Scene
So before we do anything else, let us align this mesh and our trackers with our CG environment. And you can see that on set we were a little bit misaligned here, so we're going to fix that. And we're going to highlight our SynthEyes world null and hit G X and Blender's usual move command and G X and G X.
And what we're doing is we're moving it so that the edge of our physical scan, which correlates which with our actual live action world is going to line up with the edge of our CG world. And that way things are going to all line up correctly. And we can click GZ to bring it down a little bit until our the edge of our markers here are just in contact with the top of the CG world.
And we're going to want to align these closely. So this is pretty close, but you'll notice back here that our markers are now poking down through there. And we want to make this very precise. The mesh has, has done all we need it for. So we can highlight that come over to, to the outliner, hit period to highlight it and hide it both from the viewport and from the render.
## Fine Alignment of Tracked Camera and Points to Scene
And now let's do something kind of fun. This side is, is correctly oriented so we can highlight that. And we're going to hit Shift S and move our cursor to our selected. And that moves the Blender 3D cursor here. Now we can actually tell things to manipulate around that 3D cursor.
So, I'm going to highlight our SynthEyes world, and we can do R Y. And this is rotating our entire scene. I'm only adjusting the SynthEyes world top level null, but I'm adjusting it around the point that's correct. So we're not, like, fiddling around a lot. And I'm actually going to move over here, and I'm going to do RY again, so I have more finer control.
We just want those to exactly pop into the top surface of that. These points were calculated from the, from the actual optical solve of the top edge of our live action, you know, walkway. And now we've made those align perfectly with our CG walkway. So things are lined up now.
## Rendering Sequence
Now we are going to want to control click on the SynthEyes camera to select it. And then we can hit numpad zero to look through it. Okay, there's our shot. And now we're going to click on the, the camera and the camera settings. And we can look in the sensor.
And we can see our sensor size. And our width is 37. 38 millimeters. And that's going to look really familiar because if we're in SynthEyes, we hover over our shot. This is the effective size of 37. 38 millimeters by 21. 06.
And remember, this is the overscan size. This is, this is the key thing that is going to let our our redistorted footage come in and match.
So we have already told Blender we're going to render this view with overscan. Now we need to add a few more pixels to, to the overscan so we're going to go over to our output and we're going to change that to 105 percent so we'll have a few more pixels to work with.
## Hiding Foreground Guardrail
And also before I do this, I'm going to hide these two pieces of it. Normally we would render this in a separate layer but this tutorial is just for, for showing alignment. So I'm going to highlight our guardrail and highlight over here and hide it from the viewpoint and the render And maybe this one as well.
Period to highlight, and there we go.
## Rendering Blender Image Sequence
Okay. Hit 0 to go back to our view, and go ahead and render our animation.
And we're just rendering a quick EV animation here. This is not the, the actual final model, but it's a good reference to verify our lineup. And also I'm, I'm rendering the chisels into the final image. Normally you would not do this. I'm just going to do this to to verify the correct alignment in Nuke as we bring all the images together.
## Renumbering Blender Image Sequence
Okay. So we've finished rendering. There's one other thing we need to do before we move on. Next to our take, we're going to click open. And we're going to look in our render EXR. That's where our renders have gone.
And we have a nice EXR sequence, except they start with 001 and all of our other files for example, our EXR files start with 1001, 1002 because that's the standard for visual effects work.
So we need to renumber those. So we'll go back to our render EXR. And fortunately we have a tool in Autoshot to do that. Let's go over to File, Renumber Image Sequence, and Autoshot has actually already brought us over to the correct place. So we're going to go to render EXR and pick our first file, click open and it recognizes that as a sequence.
And we're going to have our new starting number as 1001, going to run it. And then we can look at what our, Numbers have turned into, and there we are, 1001, 1002, etc. Great, now we can move on.
## SynthEyes Nuke Export
Okay, so we're going to move back to SynthEyes. We already wrote out an export to Blender, now we need to write a matching export into Nuke. We're going to go to File, and Export, and we're going to go to Nuke, We're going to use current because that's the most modern version of this.
Again, the default location for this in the SynthEyes directory is fine. I'm going to click save and our settings for Nuke is a little bit different. We are going to use the same five percent overscan here. It's called Maya overscan, but that's the same one for all the devices.
In this case we'll do match frames because that works better to drop it into Nuke. We're going to come down here and we're going to tell it, to export it to a clipboard, and that will load all the generated Nuke nodes into the clipboard copy. And we're going to click OK.
And that has now loaded all the the nodes into our buffer.
## Pasting SynthEyes Nodes into Nuke
we're going to open up Nuke and make sure our node graph is here and we can hit control V to paste it. All right. So a lot of nodes came in here. Let's move the title up over here.
And what these are is a set of undistortion and redistortion nodes. So if I double click here for our cam viewer this shows our, cam viewer. camera original footage, and all of these are our chisels that are then routed into with our a 3D scene, connected to a 3D camera and it rendered and then overlaid on top of our footage.
When we bring in the background video, we're going to verify that those all line up. So let's go ahead and grab a hold of all these, and let's move it over here.
And we have a problem here is, is that right now the default frame starts here from 1. And, you know, see, nothing's happening. And that's because our actual frames go from 1001 upwards to about 1171 in this case. That'll get fixed when we do our Autoshot export.
So, let's go ahead and click save. And we'll, you can save it as whatever you want. There we go.
## Autoshot Nuke Export
Let's go back to Autoshot, and now our Nuke script is here. So let's go ahead and click on that. It's going to open up a directory and. We'll highlight that, click control C, and we'll go back to Nuke, and we'll hit Alt X, and that lets us run a script.
You can also do this with the file and then run script, same thing, and let's go ahead and hit paste, and that brings us to a Nuke directory, and there is our Nuke script that Autoshot generated. We can open this, and we can see where it created some nodes. Let's grab our nodes, and bring them all.
Fairly close together so we can see what's going on. And this is a set of nodes that Autoshot generates, and it has a couple of other pieces that are very useful.
## Corrected Start/End Frames
And as we imported it, you'll notice something quite useful, which is it has reset our frame timing and our frame timing now matches from 1001 to 1171 which is the correct number of frames in our shot. There we go. That's good.
## Original and Overscan Resolutions
Now that we have our shot working let's go through a couple of the resolution sizes. Let's double click on our camera one viewer, this is the undistorted viewer, and highlight our merge node and hit one.
And what this is, is we can follow the pieces coming through. These are all the geometry of our chisels. They are feeding with our tracked camera, going into a scene, and being rendered. And if we highlight on our render, we can see this is our undistorted size of our footage. You know, if we, if we go back over here and hit one on our camera original, we see we are at 4264 by 2408.
And we come down to our I'll double click our undistorted viewer, and hit, highlight merge, and hit one. This is our undistorted size of our camera original footage. And go back, double click on our redistorted viewer, highlight the merge, hit 1. Okay, so now we are looking at our camera original footage, but with the chisels composited onto it.
And the set of nodes that does the redistortion are this set of vertical nodes here. Okay, so now we've got chisels.
## Loading Blender Renders
So let's actually bring in our actual blender rendered footage. So we're going to hit R for a reader node, and this is our render EXR file. And we're going to click on this sequence.
There's our rendered images. Looks a little bit funny now, but we're going to fix that. And if we just highlight this and hit one to link it to the viewer, we don't see anything. And that is because by default, Blender drops it into a specially named layer. So we're going to highlight this, hit tab, and pick out a shuffle node.
And then in our shuffle channel, we can set our input layer to the V layer combined. There we go. Now we can see our image. It's a little dark, so let's hit tab and add a grade node. That's probably a better way to do this, but we'll do this for now. I'm going to up our gain and our multiply by quite a bit so we can make this brighter.
Again, this is not a finished compositing tutorial. We just want to show how things are going to line up correctly.
## Reformatting Blender Renders
Okay, so now we're going to highlight a grade node, hit 1, and here we see this is a resolution coming out of Blender 2116 by 1134, but that's quite a bit smaller than what we need to redistort it.
So we're going to highlight this, and we're going to hit tab, and we're going to add a reformat node, and we're going to pick on our reformat, and we're going to go down to SynthEyes, gives us a couple of choices. Let's double check which one we need.
Highlight this. 4478 by 2528. So I'm going to go back to reformat, double click that and pick 4478 by 2528. There we go. Okay. Now that's the correct one. Now we can see we have our full frame of rendered footage, and it has reformatted to 4478 by 2528.
## Verifying Alignment
We'll move these up here, and I'm going to highlight this node, and I'm going to add a merge node onto that, and we want to actually merge the chisels, A, over our reformatted node, B. And here we can see these are lining up correctly.
Hit one to make sure merge is going to the viewer, and And highlight merge nodes, merge node, and if we hit D to disable and enable it, we can see that our chisels rendered in Nuke are now dropping perfectly on top of the 3D chisels we rendered in Blender.
And again, normally you wouldn't do this in a production situation, but this is just to show our lineup is precise.
Now we actually want to see how the composite's going to, going to look. Over here we took our chisels, and we rendered, and we brought it through the distortion, and rendered them on top of our image.
## Compositing Camera Original
And we're going to do something very similar. We are going to actually redistort our footage instead of the chisels, and then render the image on top of that. Let's unhook our resize, and from here, and hook it to our reformat node. And so now we are busily resizing our footage.
And if we highlight our merge, we can see that our resize is on top of that. But we actually, we don't want that. We want the keyed footage on top of our CG background. So to do that, we're going to let's go ahead and control C and control V. Let's grab our cam original footage and highlight this and hit tab.
We want a Keylight keyer and bring this down and let's, let's highlight Keylight and hit one. And we're going to enable this, and this is not a King tutorial, so this is going to be a pretty rough key. So we'll just pick a spot here, and get a very fairly ugly looking key. But enough for our purposes.
And we're going to want bring our viewer down, and we have our merge node again. This time around, we want to merge this over that, so we're going to unhook our two nodes, and there is our B, and our A is here, and highlight this and hit one. There we go, and there is our composite and we can look at it, see how it's going together.
There we go, that looks like it is matched. Let's save here for a second. That's fine.
## Rendering Flipbook
And we're going to want to see how this actually looks when we render it together. Make sure it's actually doing what we expect it to be doing. So we can make a flipbook very quick, very quickly.
And let's go ahead, highlight our merge node, and do render. And we're going to do a flipbook on the selected. And make sure our frame range is correct, which is fine; let's do okay. And it'll calculate through and render our frames out.
Alright, so here is our flipbook. And I'm going to hit space and run it full frame. There we go. And what we want to see here is just to make sure that our shot is tracking correctly. And that things are not floating. There we go, and in general it looks like it is matching up.
This would be the starting point where we actually go in and do a little bit better job of keying and compositing.
But this, this shows that the things are lining up correctly, and that our distortion is being correctly reapplied to the shot. Okay, good. So we have a, a flip book check is working.
## Writing EXRs
So now we actually want to write the nodes out. So, move our cam viewer here. And over here from our Autoshot generated, generated script, we have a write node.
And we'll bring that over here and wire that into the merge node. And double click on that. And this is already set up to write EXRs into the correct directory. So change this to EXR and set our compression to DWAA. And there we go. And now we can actually highlight our write node, save and render our selected write node.
That's the correct frame range. Let's click. Okay. And it's going to go through and render our frames.
Okay, and then we can see that we have generated our EXR series into the correct subdirectory. And we can go and bring them into Resolve or another package for finishing.
## Review
It's important to understand what we're doing here. We started with our Jetset onset data, we created our initial Blender render file. We then exported a script into SynthEyes with our onset information on it including our scan and our track data.
We generated a new solve and pinned it to the original onset tracking data, information, and scan information so that when we solved it, we maintained our coordinate system. We maintained our scale in our frame, so it just drops back into the 3D frame.
We exported from SynthEyes back to Blender with an overscan with a wider sensor back and a larger pixel count to downsample from.
We also exported a node group from SynthEyes to Nuke that had the redistortion nodes, and we combined the camera originals with the overscanned Blender background rendered files in Nuke.
Inside Nuke, we loaded in our render file, and we reapplied our distortion, and then recomposited it with the original live action plate. And here we have a clean match. So things are lined up and working at a sub pixel level. And now we can actually take this to a farther level of composite.
But this is the, this is sort of the the base piece you need if you have a shot with ground contact, is to have correctly matched distortion brought all the way through the pipeline and then brought back into the 3D model.
# Autoshot Unreal Round Trip
Source: https://docs.lightcraft.pro/tutorials/unreal-workflows/autoshot-unreal-round-trip
A complete round trip workflow from Unreal to Jetset and back
## Video Tutorial
# Transcription
We are going to do an Unreal Jetset Autoshot roundtrip. We're gonna make a project with this free Fantasy Castle. We're gonna download it from the Unreal Marketplace, add some scene locators, export it to Jetset, shoot a green screen take, and bring the tracked shots in back into Unreal.
## Creating Jetset Project
As usual, before we do anything else, let's go to our main menu, our project folder, and let's create a new project. And I'll scroll down and I'm going to call that, change the name, call that fantasy.
And call that Fantasy Castle. And we can go over here and again click the edit button and just change our prefix. To, fantasy Castle. Done. Let's delete this one. There we go. All right, and we'll set it as our current project. Okay, so we can see on the bottom we have Fantasy Castle selected as our current project.
## Installing Autoshot Tools & Autoshot
And let's go back to our desktop and we need to install the set of tools for Unreal, Jetset and Autoshot. So let's first go to the lightcraft. pro slash downloads page. I'll have the link below and we're going to download both Autoshot and we're going to download the Autoshot tools
Okay, and once those are downloaded we can just go up to here and go ahead and install our Autoshot tools; it doesn't matter which order you install them in just install both of them and Autoshot tools is a larger download that doesn't change as often. It contains the large DLLs and AI model weights that we use for our AI processing.
There we go. So install that. If you get a blue, a warning just go ahead and click okay on that.
Okay. And then let's install Autoshot itself.
## Create Matching Autoshot Project Folder
Let's go ahead and start up Autoshot. . So the first thing we're going to want to do is make a new project folder. So we're going to click Browse, and we're going to go to our D drive. And I can make, we can put it anywhere. I'm going to right click, I'm going to make a new folder. And I'm going to name it Fantasy Castle. And I'm going to name it the same name that I named our project in Jetset, just for convenience.
And later on, Jetset and Autoshot will check back and forth to make sure that those projects match, so we don't end up accidentally synchronizing footage into the wrong directory.
## Project Folder Organization
So after we've created this directory, we can click open, and that'll open a file explorer that will go directly to that.
And it's automatically created three subfolders, assets, footage, and sequences. And each one of those is used for pretty much exactly what you'd expect to use it for. When we write out our USD and USDZ assets, we're going to put them in assets.
When we're synchronizing takes from Jetset over to our desktop, Autoshot will automatically put that into the footage folder. And when we generate processed sequences from our takes, that's going to automatically go in the sequences folder. So those are all automatically set up.
All right, so we're going to, let's just click on here, going to go into assets and let's copy our file path for the time being. That'll come in useful later.
## nVidia Omniverse Setup
And next we'll go to Omniverse. And we're going to be using the NVIDIA Omniverse USD exporters. They are really, really good. Unreal has a built in USD exporter, but it frequently misses things, whereas the Omniverse exporters will catch everything. What you want to install is the Omniverse launcher.
We'll put these links below. So we can go click here and go through the process of adding the various bits of information and we can click submit. Great.
## Omniverse Launcher Download
We actually just want the launcher right now. We're going to click the launcher for Windows. It's going to download, pretty small download, and we're going to click Omniverse launcher win, and that's going to set up the Omniverse launcher.
Okay. And here we have the NVIDIA Omniverse launcher. And you'll probably need to create a NVIDIA account and do all the login, et cetera for that, and then this is what the launcher will look like. And this controls the installation of a number of really useful tools.
## Installing USDView
This is the front page of Omniverse and we can go to the library to see what tools we already have installed. I've already installed USD Composer before. We actually are going to need two other tools, so to get those tools we'll go to the exchange, go to click to apps, and we want to have I'm going to scroll down.
We want USD view and we can click install. I don't know why there are two of those, but go ahead and install this.
## Installing Unreal Connectors
And then while that's installing, we can go over to our connectors and we can type in Unreal. And to narrow our search, and we're going to want the Unreal 5. 3 connector, because we're using Unreal 5.3 to click install. We'll look at how some of those are used a little later.
## Unreal Plugin Setup
Let's go back to Unreal for now. And here's our Unreal scene and let's do some plug in setup. We're going to go to plugins and let's go first check to see that, you know, Omniverse, there we go.
So NVIDIA Omniverse is installed. We don't need the groom plugin because we're not dealing with air right now. We are also going to want the We're going to want the USD plugin, USD importer. That's actually both an importer and exporter, exporter. We're going to want the image plate. Make sure you check these if they're not checked.
And we're going to want the Composure plugin, that one. And we're going to want the Python, Python. And we will want both the Python editor. script plugin as well as the sequencer scripting and we're going to want to type in movie render queue and we're going to want both the movie render queue and the movie render queue additional render passes.
This will let us render out things like EXRs with Cryptomatte later on. Okay, and if you need to restart now that now is a good time.
## Installing Castle Scene
We should talk about a little bit where this came from. This is the Infinity Blade Castle. Here's where we got it from. This is a free asset on the Unreal store. And we'll provide a link below.
If you click open the launcher, it'll open in the Epic Games launcher. There we go. And then you can add it to your project. And I've already done that, but just to show you how it works, we can go down to our project and add the project, and I already added this so I won't, I won't click yes, but this is where you would add it to the project.
And then inside the Unreal project, you would find it in inside your content directory, and you'd go to, so you can go into the environment pack, and then Infinity Blade, and then the maps, and we're using the terrain demo.
## Adding Scene Locators
I'm going to hold down my right mouse button and slide the scroll wheel forward so I can move around quickly. But we can see that this is actually quite a large, large scene, right?
So this, there's a full castle here. And so in Jetset, we're going to want to be able to move around to several places in this environment very quickly. And to do that, we're going to add scene locators. And a scene locator is just an empty, it's just a named null that we place in, in a particular location.
So let's, let's put a couple of them in here. So I'm just navigating with WASD as usual in Unreal. So I'm going to first come over here and I'm going to, let's right click here and we're going to make a new folder. I'm going to call them, I'm going to call it scene locators. This name isn't that important because it's just us storing them in a folder.
The actual name of the scene locator is important and we'll go into that in just a second. So, then we're going to add an empty actor. I'm going to go click here, and I'm going to click basic, and actor, and there's our actor showing up in the scene.
## sceneloc\_\* Prefix
Let's hit F2 and rename him real quickly, and we're going to name him a special name, and that name starts with the prefix sceneloc, S C E N E, L O C, then our own name for whatever locator we want. And in this case, we're gonna call it Castle Approach.
## Aligning Scene Locator
And we want to make sure that he is right on the ground.
What the scene locator is going to do is it's going to pin this part of the 3D scene to wherever we pick as our live action origin in Jetset. So we want to be precise about where we put this. So, I've moved this up and down. We're going to do a couple things. We're going to make sure our snapping is set down to one, and we're going to set our our axes to display our object axes.
And that shows us which direction the object is pointing, and therefore the scene locator is pointing. We can see that by, if we click our rotate axis, and we move move here, we've shifted our red x axis this direction, and we'll hit W and switch back to position. We can see that we've actually rotated the object.
And we can see that because our object coordinate systems are being displayed. If we click to world, then it's going to display the world coordinate systems, which line up with this World Coordinate System indicator down here. And for here, when we're editing scene locators, we want it to be in object space, so we know which direction they're pointed.
We can move this over here. I can drag this so that our X axis, our red axis is pointing off to the right. Hit W again to bring back the translation handles. And I'm going to move this up and down until you can see that they just intersect with the ground.
## Organizing Scene Locators
There we go, there's our first, our first scene locator. And let's actually put it in the scene locator folder just for organization purposes. I just type in SCE into the search bar and that puts them right next to each other. So now I can drop one right into the other.
There we go. So that's one. And so let's let's fly around our, our environment. Let's, and find another, another one. Let's look over here. That's kind of a neat environment. Let's put one on the bridge. That looks like a good place to put one on. So I'm going to speed up a little bit and we'll come down here.
## Additional Scene Locators
Okay, this is a good place to put one. Pick a little wide spot on the bridge. I'll remove our filter so we can see it. I'm going to add a a basic actor here. There's our scene locator. Gonna move him over to the center of the road. You can see carefully that it's dark when it's below the level of the surface. And just as we bring above, above it, it comes into the light. I'm gonna hit F to frame it so we can see that more closely and we can move that around. We want it to be exactly on the level part of the surface. And once again, we're gonna hit F two and we're gonna name that S-C-E-N-E-L-O-C, and we'll call it bridge.
All right? Again, we're gonna type in SCE and the, and the search panel. And just drag and drop that in here. Save the scene. All right, let's find another one. Let's find another location. Zoom! All right, let's come up to one of these turrets. This looks like a good spot.
Okay. So, I'm gonna There we go. I'm gonna come down a little bit. And I'm going to add an actor. And I'm going to remove our filter. Add an actor. There we go. And You can see he kind of came in a little bit below the world. We're gonna bring him over here and bring him up. I'm going to look for the point, yep, there, there, now it's emerging, right through the stone.
And I'm going to hit F to frame up, and I'm going to move him over here a bit, and move him this way. Now, the red axis is the X axis, and by default in Jetset when we drop an origin, the red axis is going to point off to the right. If we do that in here and we snap our scene locator, we're going to be looking right at the wall instead of at the pretty view.
So let's click our, our rotation. And we're going to rotate that around 180 degrees. And hit W again to show our to show our our translation axes. And hit F to frame. I'm going to move, and I'm going to move that over just a little bit more.
## Object Axis Hotkey
Okay. And once again we are in our object axes. You can cycle back and forth with Ctrl Tilde. If you need that shortcut. All right. And so let's take a look at where we're at. Let's take a look at the view we'll have here. I'm going to slow, slow this down a little bit when I'm moving around in Unreal.
It looks kind of like a neat view and we're going to double check that we're exactly on the surface. There we go. Okay. That looks good. He's highlighted in the outliner, hit F2, S C E N E L O C, and we're going to put this overlook. All right. And once again, type in SCE in the search bar, drag and drop that into scene locators. Okay. That's a, that's good for our scene locators right now.
## Export Layer
The next thing we're going to want to do is actually go get a layer window. So under window, we can bring up layers. There we go. There's our layers window. And, what we're going to do with the layers is we're going to select all the pieces that we want to export, and, only that.
Typically, an Unreal scene can be really large. And, I'll speed up so you can see this. So, an Unreal scene can be very large. And, we don't, we don't want to export all of that. Right? That there's a lot of terrain in there and a lot of these things aren't really going to help us that much when we're, when we're trying to do our our tracking, which the things we see are more local.
So, but we can actually export a pretty decent amount of the castle.
## Selecting Export Items
So let's switch to our top view, which is alt J and hit F to frame and scroll out a little bit. And here we can see our top view of our castle and we can actually just click and drag and pick the whole castle.
Now in this case We've picked all of the underlying terrain if we try to export that it's going to take It's going to be quite a large model, and we don't actually really need the terrain So I'm going to hold it down control I'm going to click on the terrain and that gets rid of that I'm going to hold down control and click this as well
## Adding to Layer
And and I think that should actually that should do it for now. So those are all selected and we're going to do a right click, and we're going to add the selected actors to a new layer. We can name that USD export. It doesn't need to be that name, but just a convenient name for it.
Then we can go back here and alt G back to our perspective. We'll hit escape to unselect everything. Once you've created a layer, you can actually select all the items on that layer by just double clicking it.
## USD Export Directory
Now, before we export, let's go find out where we're going. Let's go back to Autoshot and we're going to go to our project folder and we can actually copy that.
And let's go back to Unreal. Again, we're going to double click our USD export. We're going to go to file export selected and we're going to pick USD. And we're going to come up here and just enter in our, our fantasy castle path, and that brings us right to our project folder and we can double click on assets and I'm going to right click and I'm going to make a new folder.
I'm going to call that USD and inside that folder, we can actually save this castle one. That's fine. And I click save. Okay.
## USD Export Settings Speed Comparison
Now this is the Omniverse exporter pop up, and there's lots of different controls on here. Now, one of the interesting things is we have a lot of control over how we export it, and we have simpler ways, and we have more complex ways that we can do this.
So one of the things that I want to show you is that geometry is fast and textures are quite a bit slower, but after you export them once it goes pretty fast. So at first I'm going to do an export where I just unclick MDL and unclick include materials. We're going to export as a Y up axis, because the phone uses Y up.
And I'm going to click OK. And it's going to start exporting. And boom, that was it. So it just exported. That's pretty fast.
So next I'm going to go ahead and I'm going to Export Selected. And this time I'm going to do the same thing. I'm going to go to USD I'm going to put it under Castle O2 this time.
And this time around, we don't need the MDL cause we're going to the phone and I'm going to include the USD preview surface. We don't need the Unreal materials. We still want the export as YUP. And this is probably going to be the way you, you usually export. You're going to use your, your materials so you can see what it looks like.
You're going to use your YUP axis and we won't, we don't really have any landscape grass, but we don't need that. I'm going to click OK, and this is going to take considerably longer, and what it's doing is it's actually exporting the physically based textures for each individual object in that castle.
And so this can actually take several minutes to honestly, to on a big file, it can take quite some period of time. But the useful thing is, you only have to do this once.
## Inspecting Export with USDView
Now one of the things we can do here is we can go down to our Omniverse launcher, and we're going to go over to our library, which is where all of our installed tools are. I'm going to click on USD view, and we're going to launch it.
And USD view is a really good tool to check to see how your your assets have exported. And we're just going to go here again. We're going to hit paste to go to our fantasy castle, go to assets USD, and we're going to open up our castle three USD. Take a second to load.
And we can open this up a little bit and hit F to frame. Okay, so there is our there is our Fantasy Castle.
USDView is an excellent tool to check your core export to see if things have come through. It's the default verification tool that Pixar built for exactly this purpose. And if it comes into USD view, you can be certain that it exported correctly.
Okay, so clearly things are there, and we can search for our prim.
And we can see that we actually forgot to export one of our scene locators. That's a good way to check.
## Fixing Export
Okay, so let's go back to Unreal. Let's go back to our outliner. There, oh, yep, see? We had our Selection selected, but we missed our bridge.
So we're going to select our bridge actor, go to layers, right? Click and add selected actor to selected layers. . Now, when we double click our export, there we go. Look at our outliner. All of our scene locators are selected along with the other material.
So we can go back and file export selected, USD. We'll do Castle 04
and again, no MDL, USD preview surface, no unreal materials, Y up, no landscape grass. Okay. Export that. Okay. Back to USD view.
File, open, paste in our path, assets, USD. Let's look at Castle 04. There we go. And again, we can just type in scene loc. Hey, and all three scene locators are on for the ride. Okay, so that's, that's good. Now we can move on.
## Converting from USD to USDZ
Now we're going to bring up Autoshot next, and we're going to go into the Models tab. In this tab, we have a real time view of whether our Jetset client is connected, which it currently is. We can see that the current project name is, is Fantasy Castle. And if we push files over to that, it's going to be storing them in this folder, which is currently in the iCloud Jetset Fantasy Castle models subfolder.
That's, that's where when we generate models, it's going to go. And we have the USD file already set to our. USD Fantasy Castle Assets USD folder. We're going to set our USDZ model. Gonna hit come here. And this may be already created for you, if not, you can right click and folder and type in USDZ. Again, this is just a convenient place to put the assets when they're processed.
## Downsampling Textures
We're gonna pick our Castle 4. And This is a texture downsampler, so if we use the original size textures, it will probably be too big for the phone. The phone can handle quite a bit of geometry, but we have to downsample the textures pretty heavily to fit. However, we can downsample it to just 512 pixels for each texture, and that should, that should help quite a bit.
So I'm going to click Make USDZ, and over here you can see it working on it. It's doing all the resizing of the different textures in that model down to 512 pixels.
Okay.
## Verifying Export and Pushing to Jetset
And down here we have an exit value of zero. And so you can see that it's finished successfully and it created the Castle Four U S D Z file. And we can click over here to push that to Jetset. Once again, we are pushing it to the project name Fantasy Castle that we created, and it's going to go into this folder, which is in Jetset Fantasy Castle models.
And so we can push USDZ to Jetset. And it's going to send that file over and we can see it sending it over via the wifi version and it's making it across.
## Wired Network Transfer
Now this is a fairly small model. And so that, that came over successfully. If you start processing larger models and especially if you're running an iPhone 15 pro that has a USB connector, you can use a USB to gigabit ethernet connector and directly connect your phone to a network.
So in this case, I have this on my iPhone 12, so it's only a Lightning port. I'm going to connect that, and I'm going to refresh this to make sure that I'm, there we go. That I, that I'm seeing the Jetset device on the new IP address because now it's directly wired.
So then I can click push USDZ to Jetset and you can see that it's going quite a bit quicker. And that's just limited to a 100 megabit line. So if you have a, an iPhone 15 Pro with that USB C port, really, you're going to want to make more use of this if you start working with large files. And also very useful for downloading files.
Okay, so that, that brought it over. And, now we can load it up.
## Model Setup in Jetset
So back in Jetset, as we noted before, we're already in the Fantasy Castle project. We're going to click our main menu, go to Model, and going to go to Open. And we can see that our Castle 4 model is already here for us. We're going to click that, it's going to open it up.
## Set Live Action Tracking Origin
And at first it doesn't look like anything because we're starting off a bit in the middle of nowhere on the model. Let's first set our live action tracking origin. So, we're going to click go to origin. And we're going to go reset. Going to start a new map. And we move back and forth. We can detect parts of the ground plane and we can pick a good spot for our origin.
I'm going to pick this over here and I'm just going to align it with something convenient in the, in the scene. There we go. And we can see already where things are going to line up. Ghost slider up. If I move over to clicking our scene locator If I pick our castle approach, then we can actually see that we are at the castle approach.
## Greenscreen Setup
I'm going to set our green screen. So I'm going to go to set, and I'm going to set our green screen color, and I'm going to set our garbage matting. So I'm going to click start, and it's going to detect a, the plane, vertical plane of the garbage matte.
I'm going to click plus to set a matte there. Now scroll down on the ground, come back a little bit so I can detect the green. I'm going to click plus. Then I'm going to click stop. Then I'll reselect our green color. That has set our garbage mattes. So now we can pan around the scene and we can both set our key and it'll automatically matte out the other parts of the scene. So even if you have a small green screen like I do here then it can work to show the, the wider world.
Okay, so now our green screen is working, can put my hand in front of that, then move off to the side It's gonna go to the garbage math, but that's okay.
## Recording Takes
Okay, so we've loaded our model. We set our tracking origin. We set our green screen matte. We've added a subject for the green screen. All right, so now we can shoot.
It's important to note that all my dashboard markers are green. Over on the right, my tracking is green. My GPU usage is green and my temperature is green. I'm running a cooler on this. That's really important.
Especially when you're running a slightly heavier scene, it's really important to be running a cooler. Otherwise, you're going to start seeing red pulses on your GPU monitor, which is the Round device that looks like a little bit like a speedometer and then you're gonna get dropped frames So right now we've got green on everything.
So let's let's roll the take There we go Framing up coming down
Back
And we'll roll another take starting high and coming low.
## Reviewing Takes
And now that we've recorded, let's go check our review. And that way we can see how our recent takes went. And we can see here that we had just a couple of frame drops in this one. Couple of frame, one frame drop, zero frame drops. Okay, good. So we have, we have a pretty small number of frame drops going on, going on here.
Okay, good. Okay, so that, that means that our, our tracking is going to be reasonably good, and the phone didn't overheat while we were shooting, and we should get some good results.
## Project Folder Name Check
Okay we're going to go to our Takes tab and as we noted before we can see here that on our current connection on our Jetset Client, we have the project name as Fantasy Castle, and as we noted, we already set our project folder to Fantasy Castle.
Now I want to show you something if we, if we set the project folder to something that doesn't match the project name on, on Jetset Client. I'm going to hit shift and click browse. And that will list a set of other projects that we, that we've used.
So, here we picked a different project. If I click open, it has the correct assets, footage, and, and sequences directory. It's an Autoshot project directory. However, since this name here does not match the project name, we're getting the error here that says does not have same folder name as current Jetset project.
And it will still sync. If I, we hit sync, it's still going to pull the takes. You can override it. But we have this warning here just to help you avoid accidentally synchronizing takes from the wrong project.
## Previous Projects List
So, All right, we're going to hold down shift. And go back to Fantasy Castle. And the list of previous projects is also under the file menu. We can look under the recent project folders. But the shift click on the browse is also a hotkey, or a shortcut. Okay, so, we have these, these synchronized.
## Picking Sync Date
Now we can pick our synchronization date. This is the date that we're shooting. You can also pick all. That's going to sync all the takes under that project. But you usually don't want that, you typically want to synchronize just a day's worth of takes.
So we're gonna click today, and going to click sync. All right, and that was quick, because we already had some of those synchronized.
## Manually Setting Jetset IP Address
It's important to note that typically, Autoshot can automatically detect the Jetset clients on the network and those will just appear.
However, if yours is not automatically appearing in the client list then we, you can actually manually enter it with the IP address of Jetset. And the way you do that is we'll go, we'll go to Jetset briefly.
And in Jetset, we can click the main menu, and show the IP address with this toggle at the bottom. And it'll show you the current IP address of this Jetset client. I'll toggle back
And you can manually enter that into this IP address line here and click connect and it will link up to your Jetset connection and it'll show it here.
## Viewing Takes in Resolve
Okay, so we've synchronized our takes, and we want to pick which take we're going to use and we want to pick which frames from that take that we want to work with. And for that, we're going to jump into Resolve and we can go and do a file and we'll do, go to the project manager and we can just create a new project
and we'll call that trip three.
Okay. And what we're going to do here is to look at our takes as we remember our project file was under the D drive. And we're just going to go to We're going to Fantasy Castle. We're going to look in our footage folder, and in the takes. And here we can see the take that were today's takes that were synchronized over here.
We'll look in videos. And here we see a whole list of videos. And we can actually just grab all these and drag them into our media page here.
## Matching Project Frame Rate
And it's asking us if we want to change the project frame rate. The default project was probably 24 frames a second. These are iPhone clips, so they are all 30 frames per second.
So we're going to change the project to 30 frames a second. And it's important to do that if you're just dealing with the straight Jetset clips that are 30 frames per second.
If you're shooting Jetset Cine, then of course your project frame rate is whatever you're shooting your Cine footage at, typically 23. 98 or 24 or 25 frames per second. In this case, we're just dealing with Jetset clips, and so those are 30 frames per second.
## Sorting Files with Smart Bins
For each Jetset take, it stores the real time composite, it stores the camera original, and it stores a depth pass for that.
And right now we just want to look at the live composites that we recorded, as we can edit with those. So we'll go down to a smart bin here, going to right click and add a smart bin, and we can just name this live comps. And the important part is here, where we're in our media pool properties, we're going to filter that our filename contains underscore comp, because all of our composites have an underscore comp suffix.
So we're going to click create, and here we can see our four composites. Live action takes. And here we can, we can quickly review which one that we like. And we can sort them. They start from take 54, to take 55, take 56, take 57. And we can look at take 54, and take 55. Kind of look at the, the motion, how we want, want to do this.
## Setting In and Out Points
I think I like the last one, which is take 57. Yeah, it's kind of a nice move. Looks up at the, looks up at the castle. Okay, so we'll work with take 57. Gonna right click and make a new timeline with that clip. And we'll create that. And let's go to that timeline in editorial. And and actually let's let's remove that clip for now.
Because we actually wanted to pick our in and out points for that. Let's pick our smart bin. And pick our take 57. And here we want to pick our in and out points of the frame. And there we go. So that's kind of a nice shot. So I'm going to come in right where the camera starts to come down.
There we go. I'm going to hit I for my in point and come down and there's the end of the shot. Hit O for your out point. And we're going to drag our clip down to here. Okay.
## Finding In and Out Frames
Now to process this clip correctly in Autoshot, we want to know what our in point, our in frame and our out frame exactly were.
And there's a couple different ways of doing this, but for our purposes, we're going to click on the Fusion tab. And we can see in the Fusion tab over here, that our in point, our in frame was 57 and our out frame was 212. And so what we can do is immediately we're going to go back to Autoshot.
## Take Selection in Autoshot
We are going to pick the, the current, the date we were shooting on in terms of our take selection. And we're going to pick the take that we wanted and that was take 57. And we'll note that each take has both a scene, a take number, and a roll number. Those are traditional kind of human friendly names.
It also has this 10 digit hex ID over here. That hex ID is unique on every single take. And so you can shoot for years and you will never have the same hex ID. And this is really ideal because it lets you identify takes automatically very, very quickly, even if you have hundreds or thousands of takes.
## Selected Take Metadata
And so here we can see different pieces of information from that take. So here we can see that the USDZ file that we had loaded was Castle\_04. The scene locator was the sceneloc castle approach, as we know. The equivalent focal length on this was the iOS Native Focal Length on that phone which turns out to be about 18. 5 millimeter as a super 35 equivalent with a 68 degree field of view. This was not a Cine take. This was just a straight Jetset take. So there is no Cine calibration file and there's no Cine footage match.
## Setting Run Values and In/Out Frames
And down here under our run values we can pick different programs.
And in this case, we're, we're going to use Unreal. Our scene locator that we will use will default to the sceneloc castle approach, and we are going to use extracted pngs. EXR files are typically for when you are working with raw or log footage, where you have to manage an extended dynamic range of a modern camera, you know, 15 or 16 stops.
For ours, we are working with 8 bit compressed footage from the iPhone, so we just, we just use PNGs, and that keeps the color science simpler.
And over here, we have our clip in frame and out frame, and as we saw from Resolve, we want 57 and 212. So we're going to enter 57 and 212.
We do not need an AI roto model here, and we'll just use our camera original and default. For our gamma and color, since it is a normal Jetset clip we set it by default to a standard rec 709 color space. And then we can save and run.
And it will go through and it'll extract our frames. And it gives us its command set that we can paste into the Unreal Editor console command box.
## Take Info Window
And before that, one other additional little bit is next to our take, we can click our take ID and we can see some thumbnails of our take, as well as how many duplicated and dropped frames we have. So we can see that our tracking was was normal. We had a good mapping. So basically this looks like a reasonable take and we should get some good values out of it.
## Creating Unreal Level Sequence
Okay. So let's move to Unreal. Here we are in Unreal, and we're going to click on our command console here, and hit control v to paste, and hit enter, and it's going to run the automated script to create a level sequence. You can see where it has created that inside the Autoshot folder. Here is our, the name of the subfolder with the elements in it. And here is our level sequence. Again, all named after the take. So we can double click our level sequence here.
And here's our level sequence. So, you can see here that here is our preview of the motion of the clip. And up here you can see the you can see the tracked camera placed correctly in the scene, next to the the sceneloc castle approach. So everything is going to match up correctly inside the Unreal scene to what we had it here in our, our live action scene.
## Level Sequence Organization
Okay, so let's take a quick look at this level sequence. So we have the tracked camera, and we have the camera animation data and down here we have the image plate, and that is where we are bringing our live action image into Unreal with, and we also have the scene locator of the castle approach, and importantly the way that we are parenting the camera to the castle.
That scene locator is with this. This is an attach track, and the script has added the attach track to the camera and set it to sceneloc castle approach. And that is how we are correctly and automatically parenting that into the scene.
## Changing Scene Locator Parenting
And if you wanted to parent this to a different scene locator, if we wanted to put this, for example, on the bridge we could add a track, and we could add the actor to sequencer, and we want sceneloc, bridge. All right, and so here's our sceneloc bridge. We'll hit the up arrow key to bring it back to frame one.
And then we can go to our attach track right click that, edit that, and we'll change that to the sceneloc bridge. And then all of a sudden, that has now moved. So let's go see where it went. Let's go find our Cine camera actor, hit F to frame. And there it is. We have just moved our camera back over to the bridge.
And we can actually process it there if we wanted to. Well, okay, that's, that's fine, but let's, let's go back to our original one. So we'll go back to our attach track again. Let's go back to frame one. So we don't have any timing issues. Right click on the attach track and we'll do edit and go back to scene loc castle approach.
All right. That's back where we are. Select the camera, F to frame. There we are.
## Green Screen Material Keyer
Okay. Well, now we have a green screen. We want to remove the green screen so we can see the, the element in the green screen. And by default Autoshot has added in the content browser, a nice material keyer.
This is based on a lesson by the Pixel Professor on YouTube. I'll add the link below, but this is a custom color difference keyer that is included in Unreal's Composure stack, and we have it applied as a material. This is located in the content Autoshot folder.
So we can right click on that, and we're going to make a material instance, and we're going to drag and drop that into this folder, move that here, double click on the folder, and then we're going to double click on that material instance.
And what we're doing is, each material instance is driven by a separate camera texture, so we want to keep the material instance in with the rest of the elements from that particular sequence. So we're going to enable the color plate and the key color, as well as our blue and red weights. And you can do more with this.
This is just the defaults. We're going to make sure that it's set to a, a plane, and we're going to drag and drop the camera texture onto the color plate. And we're going to click on the color picker on the key color and pick our eyedropper and go pick over here on our plate and click. Okay. All right.
And so then we can see we've got a decent key going so far. We may adjust our. blue weight these a little bit but we don't need to do much there. We adjust our alpha offset just a little bit and we'll, we'll check against more when we have our actual little tiger in the frame. So we'll click save and we can exit this
and we're going to come back out to our sequencer and pick our image plate. And then come over to the details and we can type MAT into the, into the search bar and click on our content browser and drag and drop our color difference keyer back onto the material. And all of a sudden it's keyed.
## Fixing Image Plane Distance
And so now in our sequencer, we can start to look how that shot's going to work. And there's our little, there's our little tiger. And we can come up to here and click our camera cuts icon and see that he's locked in. There we go, and so we've got a fairly decent track going on there.
Now, something's funny is going on, so let's check what that is. Let's unlock our camera cut. And let's rotate around here, let's see what's going on. And what's going on is that the distance of this image plate from the camera is determined by the real time data we have from the LiDAR, from the iPhone LiDAR. And so it, it basically looks at a, an approximation of the scene, and sets the, the focus distance to that distance derived by the LiDAR.
And sometimes we want to be able to override that in, in cases like this. So, we can do that pretty easily. We're gonna come down to our image plate. We're going to drag all around our transform nodes, and our transform keyframes, and we're going to click delete. And it disappears, so then we're going to open up our transform.
And we're going to open up our location. And we're going to drag our X location out. Until he's just a little bit in front of the plane. Let's see how, where he starts to cut off here. So I'm going to click our camera cuts. There he is, there's our little, there's our little fellow. There we go, that looks like he's sitting sitting nicely on the virtual plane and we're going to do a little bit of garbage masking out of him later on in Resolve, but we want to just want to make sure that he's he basically works through the shot Yeah, it looks pretty good. I like it.
And we can just go back to the first frame here now that we have we have a nice position and we'll just click add keyframe for X to lock it in place and we can save this.
## Hiding the Image Plate in Renders
Now before we start rendering, since we're going to be compositing in Fusion to give us more control over this and to be able to garbage matte out, the, the stands and the floor, etc., then we want to actually not render the little tiger in here or the live action plate.
So to do that, we set this usually by default we can go over to our image plate and go into our, let's remove the search, go into our rendering, and we're going to click actor hidden in game. And that will, that will hide the actor when we start rendering.
## Rendering Background EXR Frames
So let's go down to our movie render queue. And here it has automatically generated a job with the name of the current take, we can look at our settings and the configuration. It's creating an EXR sequence. And we can look at that, all the individual settings. We're going to compress that with DWAA compression.
We can look at the different rendering settings. And the camera settings. And the frame color. And that will give us an idea of how this is all going to come out. And there's, there's our output. We can see where it's going to be rendering this. And the file name format into the slash render EXR directory, the correct resolution and the frame rate. Okay, so we can accept and just click render and it's going to spool up and warm up the engine and then start rendering frames. Okay, and it's going to render its way through and render our 155 frames.
all right, now we can move into Resolve.
## Making Sequences Bin in Resolve
Okay, we are going back into Resolve. We're going to do a little bit of compositing here. We're going to need to go get those image sequences. In order to do that, let's first create a new bin for those. And we'll call that sequences. And what we're going to do here is we're just going to use this to store the image sequences that we extracted in Autoshot of the PNG files and also of the rendered EXR sequences.
And the reason we do this is that it's much more reliable to work with image sequences for a visual effects shot inside Fusion.
## Dragging in Sequences
Now actually first we're going to drop back to Autoshot. And here's our selected take. We're at take 57. There we go, and we're going to click the open button next to the take dropdown. And that'll open up, as we know, the list of subdirectories for our processed sequence. And we're actually going to want two of these.
So we'll go over to Resolve and toggle up our Windows Viewer. And now we can see that we can actually just click on our camera PNG. And click control A to highlight all of them and drag and drop them over into our sequence. It is automatically detected as a numerical sequence of image files.
Go back up, over to our render DXR, control A to select all, drag and drop.
Alright, so now we have our two sequences. And we can actually now drop these into the timeline. Right on top of our original live action image. And we'll do that with our Green screen take as well.
## Creating Fusion Clip
So now we actually want to create a composite with these two. So we're gonna click on these two, right click and create a new Fusion clip and now we can highlight that new Fusion clip and go into our Fusion panel. And here we have a very simple composite with our background and our foreground. And if we highlight our background and click one we, it'll put into viewer one and highlight this and hit view two and it'll put this in viewer two and we can see how our live action and CG clips are going to link up.
## Ultra Keyer Setup
The first thing we're going to want to do is let's actually do a key on this fellow. So we're going to hit shift spacebar, and we're going to type in the ultra keyer. And that's a good simple keyer for this. The ultra keyer has an input here.
So we're gonna drag our output of our green screen plate into the input of the Ultra Keyer. And with the Ultra Keyer selected, we're gonna come over here and grab our eyedropper, and pick a spot nice and close to the tiger, and let's, and let's highlight this and hit 2. And you can see, alright, so we have the beginnings of a key there.
## Planar Tracker as Garbage Matte
And we're going to be doing some more work with that. But the next obvious thing we're going to need to do is we need to garbage matte this. So, let's once again come down here. I'm going to hit shift space bar, and I'm going to get a tracker. We want a PLA planar tracker in this case because it has a nice built in mask.
We can highlight over the different inputs of the tracker. All right, so we want to put this into the background. Put that here and let's go over and we're going to track from the end of the frame because in the beginning he's out of frame. So we're going to use the garbage man on the end of frame. All right, so we're going to click set to set a reference time and I'm going to move this little guy down to his center and we're going to let's just use a hybrid point area and in our case we can actually just pick translation, rotation, and scale to kind of keep it simple. And we'll drag a little matte around this guy.
I'm going to try to keep some shadows in there. And close it off. And now we can track backwards. There we go. And we actually want him to, instead of being the output of the background, we actually want him to be the output as a mask. So we highlight this, click 2, and there we see that he's being output as a mask.
## Connecting Mask to Ultra Keyer
Then we can actually feed that mask into the Ultra Keyer. So we're going to take our output from here, and we're going to go to our garbage matte. Let me click over this in the Ultra Keyer. Alright, and so let's take a look at the Ultra Keyer. And come down here. Highlight that. And let's take a look at our matte.
And we want to the garbage matte. We can invert that. And we can tell it to post multiply the image. Alright, so now we have a nice garbage mat around our little tiger. All right, so as we scroll back and forth, the mat's tracking along with him.
## Blurring Matte Transition
It's a little bit harder to see through compression, but we can have kind of a harsh edge on the outline.
So let's go ahead and move this over and shift space and type in B L U and add a blur. And I'm going to hold down shift, drop that right in here. There we go. And for our blur, we're going to just give it a bit of a blur here and that helps to feather in everything. And once we have that, and we can actually then bring our merge node over here.
and unhook, unhook the old merge node and hook our output to the input of our merge node and type in 2, and then we can see our merged composite. So there's our little tiger in his little castle world. And we can see the the shot comes down nicely. flows over it. There he is.
## Rendering in Place for Timeline Playback
So now we want to go back up to the editorial timeline and to make sure he plays back correctly at speed, we're just going to hit right click and we're going to render him in place. And this is a, just a good way to render a temporary file, so he'll play back in an editorial speed at a good, at a good rate.
And we can just go into our resolve cache and just it doesn't matter where we're going to put this for the night right now. So I'm just going to put it into the cache clip directory. And select that folder. So it's going to render out that clip internally into a Fusion kind of cache. And then we can play it back on the timeline and he'll play back at the correct speed.
## Summary
We started off with our castle in Unreal. We added a few scene locators. We exported a proxy file, converted it into USDZ, brought that into Jetset, picked our scene locators and our origin, shot a couple shots, pulled the files back into Autoshot, picked out our shot in Resolve, set our in and out points, and rendered our frames.
# Export USD Unreal 5.4+
Source: https://docs.lightcraft.pro/tutorials/unreal-workflows/export-usd-unreal-plugin
Unreal 5.4 features a much-improved USD exporter which is now the standard export technique as the Omniverse Unreal connector is deprecated.
## Video Tutorial
# Transcription
Alright, in this tutorial we are going to go through exporting from Unreal to USD using Unreal's default USD importer. In some cases this can work differently than how the Omniverse export, exporter can work, but it has some advantages as well, including being able to simplify the meshes upon export, and that can actually work to make a very large scene like this much more tractable inside Jetset.
## Adding Scene Locators
So let's get started. So this is a Temples of Cambodia scene. It's a really nice scene, has lots of detail in different places. As usual in our Unreal workflow, we've added a scene locator for the front table.
And we've added one for a walkway. And just to reiterate how those work that's just a an empty actor object. So We'll just come back here to one of these to one of the rooms here. Watch out for the skull. And I'm just going to drag and drop an empty actor. Basic actor there.
And frame up on him. And then what we immediately see if we hit W. We want to make sure that it's intersecting with the floor. So as we go up and down, we can see where it intersects with the floor. I'll move it closer to us with the, by just dragging on the various axis arrows.
And let's, is he on the floor? Nope, not yet. So let's drop him down a little bit until we can just just see where it exactly intersects with the floor. And that way we'll have correct ground contact when we bring it into virtual production. So there we go. He's on the floor.
## Object Orientation
We already have our object orientation axes in object space, that's how we can actually view their accurate orientation.
I'm going to click E and change his orientation over to 90 degrees here. And again, switch back to W. This is exactly how we did it earlier in the in the original Unreal tutorial. So the X axis is the red axis pointing off to the right. So when we drop into our scene locator in Jetset, we'll actually be facing up toward this walkway.
And again, we can add as many scene locators as we want. Just showing you how it works. So let's go rename this one. And hit F2, and we're going to call it scene loc we're gonna call this passageway. There we go. So now we're going to go to the export process.
## Unreal USD Import Plugin
Before we export, in this case we're going to be using the default Unreal Importer.
Uh, So it actually also works as the exporter, it's confusingly named. So we're going to go to edit plugins and going to type in USD. And we're going to make sure that our USD importer is checked. And in this case our NVIDIA Omniverse plugin is unchecked because we're just going to be using the stock USD importer.
Now, of course, it imports and exports, so it's a little, again, it's a little bit strangely named. But we're going to make sure that is enabled. And then once we have that enabled, now we can go and export the entire scene. We'll just go File, Export All, and we're going to pick our directory. And as before, we just picked a USD export directory.
I put it in Saved. And we can pick our USD file. Again, this will default to a binary description of USD. And so we'll just call it the Full Interior, and I'll just name it number 2. That's fine. Hit Save.
## Unreal USD Export Settings
And it's going to bring up a slightly different USD export dialog. Once again, we're going to make sure that we have the Y axis up, because we're going to be exporting over to the iPhone, which uses Y axis up.
We're also going to enable Bake Materials. This is, by default, it's unbaked. We're going to click Baked. And, by default, there are nearly 12 materials that are selected. We don't need most of them. We're just only going to export the Diffuse, Roughness, Emissive, Opacity, and Normal here. You can actually go ahead and delete, delete that, that layer with a trash can.
By default, it's going to export to a texture size. of 128 by 128. So that'll be pretty heavily downsampled, but that's actually probably what we want for exporting this larger scene into Jetset, otherwise it's, it doesn't have a chance of fitting.
## Geometry Export LOD Control
The most important thing is down here. We can control the level of detail, the LOD, of the mesh export. And the stock level of detail is LOD 0, and that is the in game mesh level. And that's actually quite dense, and that's probably a little bit too dense for exporting a whole scene to Jetset.
And we're actually just gonna pick LOD equals 1. So that actually lets us drop our polygon count substantially while not really changing the outside shape of the mesh very much at all. Okay, so that should do it. And then we can just click Export.
And the first time you export, it's actually going to take quite a bit of time because it has to go through all the USD files and bake out the maps. Subsequent times are actually much faster. So it already exported.
## Generating USDZ in Autoshot
Now we're going to switch over to Autoshot. And I'm going to click refresh.
And so we have our newly exported TempleFull interior 02. Since we already baked down to 128 by 128, we don't actually need to change our texture size. We can just keep the texture size as original, and we don't want to change our scale units because we have that set correct in Unreal.
And we can just click make USDZ. And it will go through and crunch through and make the USDZ assets.
Now, once it's done that, we can see that it has exit value zero, it's finished successfully, and we have a TempleFullInterior02, which is our new temple. So then, we can actually go ahead and push that file to Jetset.
## Opening Scene in Jetset
Once we've pushed that to Jetset, we can actually go ahead and open Jetset, and then we can go to our our main menu, and we can go to our model and click open. And then pick our Temple Full Interior 02 that we've just just pushed over. We're gonna load that up. And it's gonna take actually a little bit of time to unpack that and load because that was an awful lot of models and textures. So we'll just give that a minute to load.
Okay, so that took a couple minutes to load, but it's the entire Unreal level. You can always cut it down if you want.
## Choosing Scene Locators
We can actually switch down to our scene locators. And here's our front table. Scene locator at the base of the temple table. If we move back a little bit, there's our walkway.
So we can look up and around. There's our temple. And looking back at our walkway. And then finally if we move to our passageway you can see that this is where, here is where we actually set down our scene locator in that particular area. And if we look over to here, there is a scary looking set of stakes there.
Alright so in that case we've actually imported an entire Unreal scene level into Jetset, and it's running fine. No GPU hits. This is an iPhone 16 Pro Max, so it's a modern iPhone. And it's actually getting a little bit warm. As you can see by the thermal indicator, because I don't have a cooler on this right now.
But that should get you going. And that way you can see how using the default Unreal exporter can actually work really well for lots of cases, including when you want to simplify the geometry of a scene.
# Exporting Animated Scenes
Source: https://docs.lightcraft.pro/tutorials/unreal-workflows/exporting-animated-scenes
Use this method to export animated scenes from Unreal to Jetset.
## Video Tutorial
# Transcription
In this tutorial we'll go through exporting an animated scene to Jetset. We're using NVIDIA's Omniverse system and Unreal Connector to translate part of this Unreal scene to USD. And there's a technique with Omniverse that works particularly well for this kind of animated scene. So let's make sure we have the needed Omniverse tools installed.
## Omniverse Setup
We'll go to Omniverse Launcher. And then we first want to make sure that we have a local Nucleus server here. And you know, we can see here it's up and running. You can see I have, local Nucleus service here. And here's our local host. And there's the projects. I've already made a CABBi subfolder and an elevator folder that we're going to store our files in.
Back to Unreal, go to our Omniverse menu, add server, and we're going to accept the default local host server, and this connects us to the local Nucleus server that we just saw.
## Select Export Layer Objects
We have an animated level sequence of an elevator here. So, let's go to our level sequences, here's our elevator moving up and down. And over in our scene, you can see that we have our elevator and we have a scene locator parented to that in the scene.
We already rendered out the far background as a 360 degree pano in a previous tutorial. So we just want to export the elevator, the scene locator, and a couple of the surrounding buildings so that we can see the scene moving as the elevator descends. So I already put these in the CABBi USD layer, so we can double click that. And that will select all of the items in the layer. You can see which, which buildings that we're selecting, just the main one behind it and a couple around it.
## Adding the Level Sequence Animation
Now to include the animation in the elevator level sequence, we're going to scroll down. And there's our elevator sequence. We're going to hold down ctrl and click it. And now the elevator level sequence containing the animation is highlighted along with the rest of the objects we're going to export.
## Exporting to Omniverse
Now we can right click and choose export to Omniverse. And we don't need MDL for this. We'll use our USD preview surface. We won't need the Unreal materials in the phone. And we're going to export as the y up axis, as that's what the phone expects. We don't need landscape grass. And click okay. And we're going to pick our, we're in our content going to go in content.
And here is our Omniverse localhost. Just as we specified it. Here's our projects cabbie and then elevator. So, I'm gonna name this elevator descend.
All right. I'm gonna save. And it's gonna start exporting all the various objects in the scene. And the first time you export it, it might take a little while because it's writing out all of the materials.
## Downloading USD Zip from Nucleus
Okay, so that just exported. That will typically be faster on following exports. And we're going to jump to the Nucleus server we had previously. And it has created, as we see, we're in our elevator panel, and it created a elevator\_descend\_01 USD file. And we actually want to export not just that file, but also the materials and props.
So we can just click on the elevator folder and click download. And it's going to zip up and download pretty quick here. And then we can actually just write that download to our USD exports directory. And go ahead and save that. And replace that. All right.
Here we are in our USD exports directory, and I'm just going to right click on elevator, and we're going to extract it all. That's fine.
And again, this is our very first export, so we're writing out all the materials and the submodels. If we start writing out additional USD files that are just different animations, then we could just export the top level file and we wouldn't have to re export the materials.
All right. So now we can look into our elevator and we see our elevator to send a one USD file as well as our materials and props.
## Converting USD to USDZ in Autoshot
So now we'll go to Autoshot and I'll open up our models tab. We'll set our model temp folder to the same elevator file, click select, and there's only one USD file in there. And now we can click that and click make USDZ. So it's going to go through that and with our default 128 pixel texture size and compress all the textures to make it fit into a phone.
All right so it created that and we'll now go find our freshly created elevator\_descend usdz and we'll push that usdz to Jetset. We have Jetset running so it's going to just push that file fairly quickly.
## Loading Scene and 360 Pano in Jetset
In Jetset we can open up the model, and, uh, elevator\_descend, there we go. And we'll click on our Scene Locator of Elevator. Okay, we can see our animated model. There's our elevator going by. And there's a building coming up. All right, but now we want to add in our 360 backgrounds. We'll click 2D, do Import, and we're gonna look for our project images. And we'll pick our 360 Piano 4.
And it recognizes that it's a sphere, that it's a 360 piano, so we'll click sphere. All right, and so you can actually move that around. Uh, there's our building that's supposed to be behind us over there, so we'll put that around here. All right, so now we have our foreground images moving, and, 360 background behind us, so we can see our dystopian future.
## Synchronizing Animation Start
And we're going to want the animation to start at the beginning of the, uh, take. So we'll go to settings. And we'll click, uh, settings. And then we're going to start animation playing at the start of record. Just click that to make sure it's right. And we'll click done.
## Rolling a Take
Okay, so now we can, uh, roll take.
It's going to start with the animation here. And we can pan down and look over and there's our building coming up.
All right, we can click stop. In our next tutorial, we'll bring in the tracked shot and relink it to the Unreal Animated scene.
# Importing Animated Takes
Source: https://docs.lightcraft.pro/tutorials/unreal-workflows/importing-animated-takes
Import camera animation from Jetset and link to the matching animated level sequence cleanly and quickly.
## Video Tutorial
# Transcription
In this tutorial, we'll go through importing the animated take we just shot, and matching the camera tracking data back to the original Unreal animated sequence with the elevator in it.
Since the animated camera was parented onto the moving elevator when we shot the take, we need to do the same process back in Unreal.
## Autoshot Settings
We'll start out in Autoshot. We already synced the take back to the project folder. And so here's the take we're going to work with. And we can see that it's using the elevator descend USDZ with a sceneloc elevator.
And our take focal length was 26. We have our program set to Unreal. And we typed in sceneloc elevator here. We'll generally use EXRs for the camera media. Unreal needs to use an image sequence to work correctly in Sequencer, and EXRs are much more compact than PNG files.
PNG files are faster to extract, so they're sometimes good when doing debugging, or when working with a program that doesn't read EXRs. I'll avoid using a denoiser, as the camera original footage isn't very important for this shot and it takes longer. I'll click Save and Run. Alright, and it gives us a string to paste into the Unreal Editor command box.
## Creating the Unreal Sequence
So we'll go over to Unreal and hit CTRL V to paste that. We'll double click on our created sequence. We'll select our camera and hit F to frame. Alright, and we see that our camera is in the middle of nowhere on the ground somewhere.
And this is because it needs to be parented to the elevator. This level sequence only contains the camera animation. It doesn't yet contain the elevator animation.
## Adding the Elevator Subsequence
So to get that, we're going to add a subsequence track. And we'll go into our subsequence track, click plus, and we'll type elevator.
To search for that, we'll click our elevator track. And just before you do that, make sure to hit up arrow to bring you back to the first frame so that you don't get your animation offset.
## Adding Scene Locator to Elevator Sequence
We need to go into the elevator track. In case you haven't already done that, you need to drag and drop your sceneloc elevator into this subsequence so that we have a reference to link to in the next step.
## Adding Camera Attach Track
We'll go back up. Now we need to tell our camera that it needs to parent itself to that animated scene locator in the subsequence. So we'll click this, and we'll do an attach track, and we're going to go into the elevator subsequence, and then pick sceneloc elevator. Once again, we can highlight our camera, hit F to frame it.
And There's our camera, and our screen.
## Fixing Animated Texture Display
Right now we don't see the animated textures on the image plane. So we can go back to our content browser, and just double click on our camera texture. And hit ctrl s. And go back and back to our sequencer. And now our textures work. It's just kind of a quirk that's there in Unreal 5. 3.
## Fixing Texture Brightness
And that's really bright. So we can dial down the brightness of the EXR files. So we can go to our image plate. Highlight that. And go over here and type in mat for material. And that brings up our material. And we'll double click the image plate material. And we can see our our node set up here and we're going to right click and we're going to add a multiply node here.
And we're going to set this to instead of 1. 0, we're going to set it to 0. 01, which is quite a bit less bright. And then we can drag in our RGB plate and drag this out to our emissive color. Alright, and then we can click apply. And come back, and sure. Alright, and so that has that has dropped our brightness down considerably.
## Disabling Image Plate
In this particular case, we're not interested in the live action. We're just using this for a previs. So we can just go to our image plate, highlight that, go over to our details and Click actor hidden in game and that will remove it and we can hide that there for now.
Unclick our camera cut and we can see that our image plate is no longer visible, but we still have our camera animation. So if I click our camera track, you can see, all right, now things are moving. You can see our, our shot as we plunge down into the city.
And the buildings go by and our camera tracking is now merged in with our original elevator motion.
## Command Line Encoder Shortcut
Let's render the sequence out. And as usual, to get a video, we'll need to set the command line encoder properties first.
And you've seen this before, where we had to go find the installation location of our FFmpeg executable. But we just added a shortcut. So we're going to go back to Autoshot. We're going to click tools. We're going to say copy command line encoder path to clipboard. And come back over here to edit. And then project settings.
And then type in command line. And then we can just double click this and hit paste. For the video codec, we do libx264. For the audio codec, we use aac. And for the output file extension, we'll enter in mp4.
## Rendering with Movie Render Queue
And with that done we can go to our movie render queue and we'll delete the old one.
And this is our, our current one. Click render. And that'll spool up our renders.
All right. We'll let that render out.
## Comparing With Jetset Live Comp
Okay, so we've finished rendering. We're going to want to test this against our original. To test this, we'll overlay the original live composite clip and the re rendered clip in Unreal. So, back in Autoshot, go ahead and click the Open button next to the clip. And this opens up in Windows Explorer, the list of subdirectories under that processed shot so we can easily find the pieces we need.
So let's go up to Resolve And we'll go back to our media panel, and we'll make our test bin. Open up our third test bin, and we'll go back over to our folder, and we'll go into our preview. And we'll pull over our unreal preview. Go up over to our source data, and get our videos and get our live comp. Pull that back over. And we'll go ahead and make a new timeline out of this. Let's make sure we have the correct format. Okay. 30 frames a second. Great. We'll open up our timeline and we will drop our live comp on top of this. Line it up. Alright, and then we're going to crop our bottom. There we go.
And as we scroll through,
we can see that we are frame on with our foreground 3D images, which is great. , I eyeballed in the background 2D image a little bit off, but that's alright, that's good enough for our reference use.
And we can see that our animation is, is correctly synced..
# Rendering 360 Backgrounds
Source: https://docs.lightcraft.pro/tutorials/unreal-workflows/rendering-360-backgrounds
Configure the Unreal Movie Render Queue to render high quality 360 panoramic images for background use in Jetset.
## Video Tutorial
# Transcription
Unreal can make some fantastic backgrounds with very heavy imagery. But to get this to fit on the phone, we can do some simple tricks like just exporting the 3D objects that are near the shooting locations and scene locators, and rendering a 360 degree panorama for the objects and environment that are further away.
And thanks to Winbush for his tutorial on this, which was really helpful. I'm using the same process he describes and just modifying it to fit this application. I'll put the link below.
In this Unreal scene example from Joshua Kerr, we have some 3D buildings and an elevator that we'll use for the foreground information, and that's the 3D model we'll export into Jetset, but we want to render the rest of the scene with the complex buildings, et cetera, as a 360 degree background.
## USD Export Layer
We already placed all the foreground 3D objects in a layer; I'll double click that. So we can see we have the main tower and the elevator selected. The elevator is animated in this scene in the elevator level sequence, as we can see. There we go.
So, as the sequence goes, the elevator goes down. It has a scene locator in it, and the elevator is animated through this sequence, so we'll just pick out a midpoint to take the 360 degree pano from.
## Placing 360 Camera
So, go down about here. Halfway down. Okay, and next what we're going to do is We already created a 360 degree pano camera, which is just a normal Cine Camera actor.. We'll just locate that roughly the same position as the elevator. So we can go to our elevator, right click on our location, copy that, come over to our 360 pano, right click and paste. There you go, and that's an easy way of copying location.
## Setting Foreground Objects to Hidden in Game
We want to make sure that the 3D foreground objects that we're going to export don't render into our panorama.
And to do this, we're going to double click our Cabbi\_USD elevator layer. So we select all the objects we're going to use. And we're going to go down here and click Actor Hidden in Game. And that will remove those actors from view when we render them. Except there's one extra one there. Let's add that to our layer.
There we go. And double click that again. And actor hidden in game. Great.
## Creating 360 Level Sequence
Now we're going to make a new level sequence to render this. We'll just click here, add a level sequence. We'll call this 360 Pano 4. We're going to want to put in our panoramic camera into that. So we'll drag and drop that into the scene. And lock our camera cuts file. And this is the viewpoint from the camera.
And we're going to actually shrink down our rendering section considerably. Because we only need a single frame. So I'll drop that down quite a bit. We'll just render like two frames to make sure they all work. Okay.
## Movie Render Queue Panoramic Setup
Now we're going to want to click on the Movie Render Queue icon. And we're using the Movie Render Queue Advanced.
So, let's remove that. Here's our 360pano4 configuration. We'll click on the unsaved configuration. And we're going to remove the deferred rendering. And we're going to add panoramic rendering. And we're going to click allocate history per pane. This was recommended by Winbush and that seems to work well.
And we'll go to our output. I'm going to change our output directory to a directory I've already picked to load the 360 images and we're going to change our output resolution to 4096 by 2048 and that gives us a two to one aspect ratio image, which is the standard for panoramic rendering.
And when Jetset opens a two to one panoramic image under our 2D tab, it recognizes that as a two to one image, and we'll bring that in as a 360 degree pano, which is just what we want. So we're gonna click accept, and we're gonna click render. And it'll take quite some time to render this several minutes, but we'll let that go for for now and then come back.
## Reverting Scene and MRQ Settings
Okay, so we've finished rendering. And we're gonna go back and make sure we can exit this, and uncheck our camera cuts so we're back in the scene. Now we can double click our layer again, and uncheck the actor hidden in game. And that brings our objects back, because when we render, we actually want to see those objects.
Also in our movie render queue, let's go ahead and re add the deferred rendering. Otherwise, when you try to render our normal sequences it will be blank. Great.
## Transferring 360 Images to Jetset
And now we can move on to getting our new panorama onto the phone. We have logged into iCloud Drive on the browser, and we'll navigate through the folder tree to where we want to bring our files. Jetset files in iCloud are located under the Jetset directory and then the project name. And finally we'll go into the images file. And we're going to drag and drop our image.
360pano4. Alright, and it's uploading. We'll double click that to see what our image looks like. Alright, there's a 360 of our futuristic city, and we'll import that when we load in the animated elevator in the next tutorial.