PKT 0000 — vrOS

Your Mac's display, in a headset.

vrOS captures your Mac's screen, hardware-encodes it, and renders it in lens-corrected stereo on an iPhone — over a USB cable, no Wi-Fi, no cloud.

View source Read the wire protocol →

Pipeline

Capture to display, five hops.

macOS Sender App

ScreenCaptureKit → encode

StreamController
VideoEncoder (actor)
USBClient (TCP)
SCStream

iOS Receiver App

decode → Metal + VR distort

ReceiverViewModel
USBListener
VideoDecoder
MetalRenderer

0001 Capture
ScreenCaptureKit grabs the primary display at 1920×1080 @ 30 fps.
0002 Encode
VideoToolbox hardware-encodes each frame to H.264 Annex B, with SPS/PPS sent as a config packet.
0003 Transmit
Frames are packetized and sent over TCP via USB tethering — iproxy 2345 2345.
0004 Decode
iOS converts Annex B to AVCC and feeds it to VTDecompressionSession.
0005 Render
Metal applies barrel distortion and chromatic aberration correction per eye, drawn side-by-side.

Internals

Under the hood.

All-hardware pipeline

ScreenCaptureKit → VideoToolbox → Metal GPU render. No software fallbacks, minimal CPU load.

USB

USB transport

No Wi-Fi required — communication runs over USB tethering via TCP, forwarded with iproxy.

LNS

VR lens correction

Barrel distortion shader compensates for headset lenses, with tunable K1/K2 coefficients.

Chromatic aberration correction

Per-channel offsets cancel the red/blue fringing lenses introduce at the edge of frame.

SBS

Stereo rendering

Side-by-side stereo via Metal instanced draw calls — each eye renders with its own distortion pass.

PKT

Custom wire protocol

30-byte binary header — magic, type, sequence, timestamp, checksum — plus a variable payload.

SW6

Swift 6 concurrency

Actors and AsyncStreams throughout. The encoder is an actor; the network layer runs on async streams.

HUD

Latency HUD

On-screen overlay for connection state, resolution, framerate, frame counts, and estimated latency.

Protocol

The wire format.

Custom binary protocol over TCP, port 2345. Every packet opens with a 30-byte header.

Header

Offset	Size	Field	Notes
0	4	0x55534250	Magic — "USBP"
4	2	Version	Protocol version
6	2	Packet type	config / keyFrame / videoFrame / heartbeat / keyFrameRequest
8	2	Flags	isKeyFrame, isEndOfFrame, isConfig, isHeartbeat
10	4	Sequence	Monotonic counter
14	8	Timestamp	Nanoseconds
22	4	Payload length	Variable
26	4	Checksum

Packet types

Type	Value	Payload
Config	0x0001	SPS + PPS (Annex B start codes)
ConfigAck	0x0002	—
KeyFrame	0x0010	H.264 key frame (Annex B)
VideoFrame	0x0011	H.264 frame (Annex B)
Heartbeat	0x0100	—
KeyFrameRequest	0x0101	—

NALU format: macOS sends Annex B (start-code prefixed). iOS converts to AVCC (4-byte length-prefixed) before handing off to VideoToolbox.

Setup

From clone to headset.

0006 Generate projects
chmod +x setup_xcode_projects.sh && ./setup_xcode_projects.sh
0007 Sign
Open each .xcodeproj in Xcode and set your development team under Signing & Capabilities.
0008 Build & install
Run both targets from Xcode — the macOS sender and the iOS receiver.
0009 Connect
iproxy 2345 2345, then launch the iOS app first — it listens — followed by the macOS app.

Status

What's shipped, what's next.

macOS display capture at 1920×1080 @ 30fps
H.264 hardware encode → decode pipeline
USB TCP transport via iproxy
Metal rendering with SBS stereo split
Barrel distortion + chromatic aberration correction
Audio capture
USB hot-plug detection
Key frame request / error recovery
Display selection UI
Unit tests

Requirements

What you'll need.

macOS 13.0+ (Ventura) — for the sender
iOS 16.0+ — for the receiver
Xcode 15.0+ with Swift 6 language mode
A USB cable, for tethering and iproxy forwarding
A Cardboard-compatible VR headset for the iPhone

ScreenCaptureKit · VideoToolbox · Metal · Network.framework · SwiftUI · Swift 6 · CoreVideo · CoreMedia