Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ The native side should own anything that depends on macOS frameworks, `xcrun sim
Wraps `xcrun simctl` for discovery, lifecycle management, app launching, URL opening, appearance toggles, and simulator log capture.
- `packages/server/native/DFPrivateSimulatorDisplayBridge.*`
Owns headless private display frames plus HID-based touch and keyboard injection.
- `packages/server/native/DFDTUHIDClient.*`
Sends touches, buttons, and keys to the guest `dtuhidd` digitizer service on CoreSimulator 1155.4 (Xcode 27) and later.
- `packages/server/native/XCWAccessibilityBridge.*`
Owns private CoreSimulator accessibility snapshots through `AccessibilityPlatformTranslation`.
- `packages/server/native/XCWPrivateSimulatorSession.*`
Expand Down Expand Up @@ -82,6 +84,8 @@ CoreSimulator service contexts resolve the active developer directory from `DEVE
Accessibility recovery may use simulator launchctl UIKit application state plus hit-tested translations to recover candidate foreground pids; the returned tree must still be rooted at tokenized `AXPTranslator` application objects, because `translationApplicationObjectForPid:` can omit the bridge delegate token after private display lifecycle changes. Full-tree snapshots merge those recovered roots with the private frontmost application translation. Shallow snapshots with `maxDepth <= 2` use the tokenized frontmost application translation directly when it is available, and only run the expensive recovery sweep if frontmost lookup fails, so agent-oriented describe loops avoid launchctl and hit-test recovery overhead. Interactive-only snapshots also prune non-actionable native AX leaves during Objective-C serialization before the Rust-side compacting pass; keep this native pruning conservative so selector taps still retain actionable rows plus their ancestors. When multiple candidate application roots are discovered, serialize all of them in preferred order: non-extension app roots first, then largest translated roots, with `.appex`/PlugIns processes de-prioritized so SpringBoard and Safari app roots stay primary while widgets and WebContent roots remain debuggable. Widget renderer extension roots may report local frames; normalize those roots and children against matching SpringBoard widget placeholder frames before returning the snapshot.
Physical chrome button support uses DeviceKit `chrome.json` input geometry for browser hit targets. Volume, action, mute, Apple Watch digital crown, Watch side button, and Watch left-side button dispatch through `IndigoHIDMessageForHIDArbitrary` with consumer/telephony/vendor HID usage pairs from the device chrome metadata; home, lock, and app-switcher remain on the existing SimulatorKit button paths. Apple Watch Digital Crown rotation dispatches through `IndigoHIDMessageForDigitalCrownEvent` when SimulatorKit exposes it, with `IndigoHIDMessageForScrollEvent(..., target=0x34)` as the fallback. Browser mouse/trackpad wheel input over the device screen sends the normalized screen point with the scroll delta, moves the SimulatorKit pointer target there, then dispatches native scroll packets through `IndigoHIDMessageForScrollEvent(..., target=0x2)` with a digitizer-target fallback instead of synthesizing touch drags. tvOS simulators do not support direct screen touch; browser/API tap maps to Enter, swipe maps to arrow keys, and the native bridge rejects tvOS touch packets before they reach guest `SimulatorHID`. watchOS/tvOS skip dynamic pointer/mouse service warm-up because those guest runtimes abort on unsupported virtual services. Apple TV and Apple Watch simulators are fixed-orientation devices, so client and server rotation paths must not expose or dispatch device rotation for those families.
On macOS/Xcode 27-era CoreSimulator profiles, `mainScreenWidth`, `mainScreenHeight`, and `mainScreenScale` may be absent from `profile.plist`; DeviceKit chrome rendering must read `capabilities.plist` `ScreenDimensionsCapability` or the primary `displays` entry before falling back to the framebuffer mask PDF. If none of those sources produce usable display geometry, the chrome profile must fail instead of returning a tiny synthetic bezel that hides the stream.
From CoreSimulator 1155.4 (Xcode 27) the guest routes HID through `dtuhidd` and drops legacy Indigo input: SimulatorKit's HID client accepts the messages and the guest ignores them, so success from `sendWithMessage:` proves nothing. When the loaded CoreSimulator bundle (not the selected Xcode) is 1155.4 or newer, the display bridge sends touches, edge and two-finger touches, Home, the App Switcher, hardware buttons, and keyboard usages through `DFDTUHIDClient` instead: XPC dictionaries (`messageType`, `isBarrier`, `featureIdentifier`, `payload`) on `com.apple.coredevice.feature.remote.hid.digitizer`, reached by `-[SimDevice lookup:error:]`, `xpc_endpoint_create_mach_port_4sim`, and `xpc_connection_enable_sim2host_4sim`. The client connects lazily and confirms a live daemon with a barrier request before use, because `dtuhidd` can abort during boot while its service lookup still succeeds. Buttons go only as HID usages there; legacy button codes such as Apple Pay and the software-keyboard toggle have no `dtuhidd` equivalent. Scroll and Digital Crown still use the Indigo path.

Two-point multi-touch dispatch prefers the current SimulatorKit/Indigo packet constructor and falls back to SimDeck's manual Indigo packet adapter. On Xcode 26 SimulatorKit, the constructor expects pixel-space points and stable two-finger movement requires sending `LeftMouseDown` for both `began` and `moved`, then `LeftMouseUp` for `ended`/`cancelled`; using `LeftMouseDragged` for multi-touch moves only advances one contact in UIKit. Do not coalesce multi-touch move packets in the WebSocket or WebRTC control paths, because gesture recognizers need the intermediate two-contact samples.
WebKit inspection uses the simulator `webinspectord` Unix socket named `com.apple.webinspectord_sim.socket` and WebKit's binary-plist Remote Inspector selectors. It lists only WebKit content that the runtime exposes as inspectable. For app-owned `WKWebView` on iOS 16.4 and newer, the app must set `isInspectable = true`.

Expand Down
1 change: 1 addition & 0 deletions packages/server/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ fn main() {
let files = [
camera.join("SimDeckCameraService.m"),
cli.join("DFPrivateSimulatorDisplayBridge.m"),
cli.join("DFDTUHIDClient.m"),
cli.join("XCWH264Encoder.m"),
cli.join("XCWProcessRunner.m"),
cli.join("XCWPrivateSimulatorBooter.m"),
Expand Down
49 changes: 49 additions & 0 deletions packages/server/native/DFDTUHIDClient.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#import <Foundation/Foundation.h>

NS_ASSUME_NONNULL_BEGIN

typedef NS_ENUM(NSInteger, DFDTUHIDTouchPhase) {
DFDTUHIDTouchPhaseDown,
DFDTUHIDTouchPhaseUp,
};

/// Input for simulators whose guest routes HID through `dtuhidd` (CoreSimulator 1155.4, Xcode 27, and later).
/// Those guests accept legacy Indigo messages from SimulatorKit's HID client and then drop them, so touches, buttons
/// and keys must go to `dtuhidd`'s digitizer service as plain XPC dictionaries instead.
@interface DFDTUHIDClient : NSObject

/// Whether the CoreSimulator loaded in this process routes guest input through `dtuhidd`.
+ (BOOL)isRequiredByLoadedCoreSimulator;

/// Connects to the device's digitizer service and waits until `dtuhidd` answers, retrying while it starts.
- (nullable instancetype)initWithDevice:(id)device error:(NSError **)error;

/// One contact at a normalized top-left point in 0...1. The first down starts the contact, later downs move it.
- (BOOL)sendTouchPhase:(DFDTUHIDTouchPhase)phase
x:(double)x
y:(double)y
edge:(uint32_t)edge
error:(NSError **)error;

/// Two contacts at normalized points, tracked separately from the single contact.
- (BOOL)sendTwoFingerTouchPhase:(DFDTUHIDTouchPhase)phase
x1:(double)x1
y1:(double)y1
x2:(double)x2
y2:(double)y2
error:(NSError **)error;

/// A hardware button identified by its HID usage page and usage (Home is consumer 0x0c / 0x40).
- (BOOL)sendButtonUsagePage:(uint32_t)usagePage usage:(uint32_t)usage down:(BOOL)down error:(NSError **)error;

/// A USB HID keyboard usage.
- (BOOL)sendKeyboardUsage:(uint32_t)usage down:(BOOL)down error:(NSError **)error;

/// Whether the connection is still usable; `dtuhidd` invalidates it when the simulator shuts down.
@property (nonatomic, readonly, getter=isValid) BOOL valid;

- (void)disconnect;

@end

NS_ASSUME_NONNULL_END
Loading
Loading