Immersive Intelligence: Applied AI for AR/VR Systems

The XR landscape: AR, VR, MR and spatial computing, and what actually distinguishes them. Hardware survey: Meta Quest 3, Apple Vision Pro, mobile AR, tethered PCVR. The industry map, the roles in it, and where the Spotlight startups sit on it.

Unity editor, scene graph, GameObjects, components and prefabs; transforms and hierarchies in practice; project structure and version control for Unity projects.

Diagnostic quiz; environment check; build and deploy a first scene to your own Android device.

A deployed APK plus an initialised project repository.

Startup Spotlight 1, mobile and consumer AR: a shipped AR product, its stack, its market and its hiring.

C# essentials: classes, inheritance, interfaces, coroutines and events. The MonoBehaviour lifecycle and why the distinction between Awake, Start, Update and FixedUpdate matters. Unity Input System. Physics: colliders, rigidbodies, raycasting, layer masks.

An interactive object-manipulation scene, followed by an open debugging clinic.

Grab, throw and reset interaction with correct physics behaviour.

ARCore and ARKit architecture and how AR Foundation abstracts across both. Session lifecycle, device tracking, trackable managers.

Plane detection, hit testing, anchors and anchor persistence. Placing, scaling and orienting content in a detected environment. Building and signing for Android and iOS.

AR placement application, built along with the session.

Image tracking and marker-based AR; face tracking. AR user experience: onboarding, scale cues, drift, failure states.

Light estimation, environment probes, reflection and shadow. Occlusion and the depth API: making virtual content sit behind real objects.

Marker-based product visualiser with correct occlusion. Project team formation and problem-statement selection, in teams of three or four.

A product visualiser that survives poor lighting and rapid camera movement.

XR Interaction Toolkit: rig architecture, controllers, interactors and interactables. Simulator sickness: vestibular mismatch, vignetting, frame-rate discipline.

Grab, socket, ray and poke interactions. Locomotion: teleportation, continuous movement, snap and smooth turning. Working in the XR Device Simulator and Meta XR Simulator; building for Quest.

VR interaction sandbox in the simulator, with a Quest build prepared for Immersion 1.

Three working interaction types in a single scene.

Startup Spotlight 2, enterprise VR training and simulation: what a deployed VR product has to hold, and who pays for it.

Industry: spatial UI, world-space canvases, diegetic interfaces, readable typography at distance; spatial audio and haptics; accessibility. Faculty: rendering and performance theory, the frame budget, draw calls, batching, holding 72 and 90 Hz, and diagnosing whether you are CPU- or GPU-bound.

Profiler laboratory: take a failing scene from 45 fps to a stable 72 Hz budget.

Why XR needs machine learning: perception, content generation, interaction. The machine-learning system view: data, model, inference, deployment. The training loop, explained on the board.

PyTorch: tensors, autograd, the training loop in code. Convolutional networks and transfer learning. Exporting to ONNX and running inside Unity through Sentis.

Train a classifier, export it to ONNX, and run it inside a live AR application through Unity Sentis.

An AR application that classifies what its camera is looking at.

The pinhole camera model; intrinsic and extrinsic parameters. Camera calibration and lens distortion correction. The 3D mathematics of the foundations pack, revisited where it is used.

Feature detection and matching: ORB, SIFT, and the speed/robustness trade-off. Homography estimation and perspective-n-point pose recovery. An introduction to visual-inertial odometry: what ARCore and ARKit are actually doing frame to frame.

OpenCV: calibrate your own phone camera, recover marker pose, and drive a Unity object with it.

Calibration report for your own device, with reprojection error.

MediaPipe pipelines: hand landmarks, face mesh, full-body pose. From landmark streams to classified intent.

Gesture recognition: temporal models over landmark sequences. Streaming a live perception pipeline into Unity within an acceptable latency budget.

Gesture-controlled AR interface, end to end.

Startup Spotlight 3, computer vision and edge AI in production: inspection, drones, robotics or logistics, and the market the employable core serves.

The YOLO family and real-time detection trade-offs. Evaluating a detector: precision, recall, IoU and mAP. Building a custom dataset: collection, labelling, augmentation strategy. Training, evaluation and export for on-device inference.

Train a detector on your own object class, convert it, anchor AR content to detections, and benchmark frame rate on device.

Model compression for the edge: post-training quantisation and quantisation-aware training, pruning, knowledge distillation. Accuracy against latency, memory and the thermal envelope, and how to measure each honestly.

Deployment pipelines in production: ONNX Runtime, LiteRT and Unity Sentis; GPU and NPU delegates; asynchronous inference; battery and memory budgets; versioning and continuous integration for models.

Take your Week 10 detector from FP32 to INT8, benchmark it on your phone, and prepare the Quest build for Immersion 1.

Benchmark report: three model variants on your phone, with a deployment recommendation. The headset column is added at Immersion 1.

Large language models for application builders: prompt architecture, structured output, function and tool calling, retrieval-augmented generation for domain-grounded assistants. Failure modes and how to evaluate them.

Production integration: speech transcription and synthesis, the latency budget for natural conversation, streaming, cost and rate limits, and safety guardrails.

An agent that answers questions about the scene it is standing in.

Visual-inertial odometry in depth. Mapping, loop closure, drift accumulation and relocalisation.

Scene reconstruction and meshing from depth. Semantic segmentation of environments: labelling floors, walls and objects. Persistent, shared and cloud spatial anchors.

Build a semantically aware AR application whose content respects real floors and walls.

A room-aware AR application with a persisted anchor that survives an app restart.

Startup Spotlight 4, 3D capture, digital twins or generative 3D: a production capture-to-asset pipeline and the business behind it.

The asset bottleneck: why 3D content, not code, is the real cost in XR production. The photogrammetry pipeline. Neural radiance fields and 3D Gaussian splatting: the underlying idea and the practical workflow. Importing and rendering splats inside Unity.

Process the Immersion 1 capture: photos to splat to Unity, end to end.

Your Immersion 1 capture rendered as a splat inside a Unity scene, with a note on capture quality.

Text-to-3D and image-to-3D generation: current capability and honest limitations. AI texture and material synthesis; generating PBR map sets. Procedural and rule-based generation.

The production asset pipeline: LODs, decimation, baking and atlasing; optimising captured and generated assets for standalone headsets; licensing and provenance of generated content.

Build a complete environment from captured and generated assets, profiled to a 72 Hz budget.

Ethics and governance for immersive AI: consent, likeness rights, synthetic media, biometric capture, and obligations under the Digital Personal Data Protection Act for XR data.

Avatar systems; viseme-driven lip sync; procedural gesture and gaze. Agent state, memory and personality persistence across a session.

Conversational AI avatar operating inside a VR scene, with a phone-deployable AR variant.

Networking models for XR: authority, state synchronisation, latency compensation. Colocation and shared spatial anchors.

Netcode for GameObjects and Photon Fusion in practice. Backend and distribution: cloud services and authentication; telemetry and analytics; WebXR as a zero-install channel; submitting to the Meta Horizon Store and Google Play; build automation.

Two-player shared AR scene with synchronised anchors, phone to phone.

A shared scene in which two devices see the same object in the same physical place.

Startup Spotlight 5, healthcare, education or conversational-AI XR: how a product is tested with real users, and what evidence customers ask for.

Industry: design thinking for spatial products, problem framing, storyboards, comfort and accessibility, onboarding, failure states. Faculty: evaluation methodology, designing a user study, task-based testing, standard instruments (SUS, SSQ), A/B tests and the basic statistics needed to report them.

Run a five-user test of your Milestone 7 build, analyse the results and write them up.

A user-testing report with findings and prioritised fixes.

Track-specific domain session, delivered by a practising engineer or founder from that vertical: three parallel classes, one per track.

Track-specific technical deep dive. Industrial: OPC-UA and MQTT streams, time-series into Unity, digital-twin synchronisation. Heritage: large-scale site capture, level-of-detail streaming and outdoor localisation. Healthcare and education: motion-data skill scoring, clinical and educational validation, learning analytics.

Sprint 1: technical spike, integration plan, defect triage. Mentor review 1 (architecture).

Progress clinic: each team presents a two-minute status against its design document; live triage of the blockers that recur across teams.

Optimisation masterclass: profiling with the Unity Profiler, RenderDoc and OVR Metrics Tool; thermal throttling and battery constraints on standalone headsets.

Sprint 2: core feature build, model training run, first integration.

Optimisation clinic: bring your build; profiling live on the cohort's own scenes and models.

Faculty: writing the technical report, structure, the results chapter, honest reporting of negative results. Industry: demo video production and the two-minute technical pitch.

Sprint 3: integration, performance tuning, hardware test plan. Mentor review 2 (build).

Startup Spotlight 6, followed by a joint hiring panel of all six partner startups: open roles, internships, and what each one tests for.

Results validation: benchmark methodology, model metrics, statistics for the project report, and how a jury reads a results table.

Sprint 4: testing, benchmarking, and a cross-team bug bash. Mentor review 3 (scope lock).

A portfolio for computer-vision and XR roles: Git presentation, demo video, one-page technical summary, résumé; how hiring managers screen in five minutes; the technical interview and the take-home task.

Pitching the project and preparing for the viva: anticipating the jury's questions, defending trade-offs, presenting benchmarks.

Mock technical interviews in rotation; pitch rehearsal; report compilation and demo video production.

Portfolio page and demo video published.

Final dry-run clinic: feedback on the demo, the deck and the results chapter.

Demo Day briefing: jury rubric walkthrough, hardware logistics, timing and the viva format.

Final integration and rehearsal ahead of the immersion.