Building a Zero-Middleman Peer-to-Peer Remote Desktop using WebRTC and Native OS APIs

Abstract

Most traditional remote desktop solutions operate on a client-server relay paradigm: every desktop video frame and every mouse movement is piped to a centralized cloud datacenter, decoded or buffered, and forwarded to the remote technician. While architecturally simple for the vendor, this topology introduces severe drawbacks in real-time latency, cloud infrastructure costs, and data sovereignty compliance under regulations like GDPR and NIS2.

This article details the technical architecture and engineering decisions behind Conexa Remote, a native desktop remote access application designed to establish direct, zero-middleman peer-to-peer (P2P) connections using WebRTC and native operating system capture APIs (DXGI Desktop Duplication on Windows and ScreenCaptureKit on macOS). We cover low-latency GPU frame acquisition, dual SCTP DataChannel multiplexing for 60 FPS input injection, deterministic ICE NAT traversal, and a stateless signaling model that enforces floating concurrent session limits without intrusive telemetry.

1. The Relay Trap: Why Remote Desktop Architecture Needed a Reset

For over two decades, the dominant commercial model for remote desktop tools combined proprietary protocols (or wrappers around VNC/RDP) with centralized relay networks.

flowchart LR
    subgraph Legacy Architecture ["Legacy Cloud Relay (High Latency / Intermediary Buffer)"]
        A1["Host Machine (Windows/Mac)"] -->|"Video Payload (Encrypted to Cloud)"| R1["Vendor Cloud Relay Cluster (US/Third-Party)"]
        R1 -->|"Relayed Video"| B1["Technician Machine"]
        B1 -->|"Input Commands"| R1
        R1 -->|"Relayed Input"| A1
    end

    subgraph P2P Architecture ["Conexa Remote P2P (Direct DTLS-SRTP Stream)"]
        A2["Host Machine (Windows/Mac)"] <-.->|"Signaling Only (SDP / ICE)"| S2["Signaling Node (EU Romania)"]
        B2["Technician Machine"] <-.->|"Signaling Only (SDP / ICE)"| S2
        A2 <===>|"Direct DTLS-SRTP Media & DataChannels (AES-256 GCM)"| B2
    end

In the legacy topology:

1. Latency Overhead: A connection between two offices in Frankfurt routes through an intermediary relay server in London or Virginia, adding unnecessary network hops and buffer bloat.
2. Privacy & Compliance Exposure: The intermediary relay handles the unencrypted or re-encrypted transport layer. Under EU GDPR and strict Data Processing Agreements (DPAs), routing corporate desktop video through foreign third-party infrastructure represents a serious compliance liability.
3. CPU and Memory Bloat: Modern enterprise remote utilities frequently ship as 60MB+ bundles wrapping web runtimes (Electron/CEF), consuming 300MB to 1GB of RAM on idle systems.

By adopting WebRTC as a standardized, secure transport layer combined with native platform binaries (pure C++ on Windows and Swift/Objective-C on macOS), we eliminated the intermediary video relay entirely for direct connection topologies.

2. System Architecture: The Three-Tier Topology

The Conexa Remote architecture separates session negotiation, NAT hole-punching, and payload transport into three distinct layers:

sequenceDiagram
    autonumber
    participant Host as Remote Host (Controlled Endpoint)
    participant Sig as Stateless Signaling Server (Go / EU)
    participant STUN as STUN / TURN Relay (Bucharest, EU)
    participant Tech as Technician Client (Controller)

    Host->>Sig: Authenticate & Register 9-Digit Channel ID
    Tech->>Sig: Request Connection to 9-Digit ID (Validate Floating Slot)
    Sig-->>Tech: Session Token Approved (Pool: 2/2 Active)

    Tech->>STUN: Gather ICE Candidates (Host, Server-Reflexive)
    Host->>STUN: Gather ICE Candidates (Host, Server-Reflexive)

    Tech->>Sig: Send SDP Offer + ICE Candidates
    Sig->>Host: Dispatch SDP Offer + ICE Candidates
    Host->>Sig: Send SDP Answer + Selected ICE Pair
    Sig->>Tech: Dispatch SDP Answer

    Note over Host,Tech: Signaling Server Disengages — Direct DTLS-SRTP Handshake
    Host<===>Tech: Direct Peer-to-Peer Tunnel (AES-256-GCM)
    Host-->>Tech: 60 FPS DXGI / ScreenCaptureKit Stream
    Tech-->>Host: SCTP Unordered Mouse Telemetry + Ordered Keystrokes

Key Architectural Tenets:

• Stateless Signaling: The signaling backend (written in Go) acts solely as a message broker during the initial SDP offer/answer exchange and ICE candidate pairing. Once the WebRTC PeerConnection transitions to connected, the signaling server does not touch, buffer, or inspect any packet.
• Zero Video Interception: Video never traverses the signaling cluster. Even in asymmetric network topologies where TURN relaying is unavoidable, the TURN server forwards opaque, encrypted UDP datagrams without possessing the DTLS session decryption keys.

3. High-Performance Native Screen Capture Pipelines

WebRTC is traditionally associated with browser-based video calls. In a native remote desktop application, feeding generic webcam APIs will result in severe lag, washed-out color reproduction, and high CPU utilization. We engineered separate, dedicated capture engines for Windows and macOS.

3.1 Windows Capture: DXGI Desktop Duplication API

On Windows 10 and 11, we bypass legacy GDI BitBlt and user-mode graphics hooks entirely in favor of the DirectX Graphics Infrastructure (DXGI) Desktop Duplication API (IDXGIOutputDuplication).

// Windows C++: DXGI GPU Frame Acquisition Pipeline (Simplified Architecture)
#include <d3d11.h>
#include <dxgi1_2.h>

class DxgiCaptureEngine {
private:
    ID3D11Device*           m_pD3DDevice        = nullptr;
    ID3D11DeviceContext*    m_pImmediateContext = nullptr;
    IDXGIOutputDuplication* m_pDeskDupl         = nullptr;
    DXGI_OUTPUT_DESC        m_OutputDesc;

public:
    bool AcquireFrame(ID3D11Texture2D** ppDesktopTexture, DXGI_OUTDUPL_FRAME_INFO* pFrameInfo) {
        IDXGIResource* pDesktopResource = nullptr;

        // Timeout set to 16ms (~60 FPS sampling boundary)
        HRESULT hr = m_pDeskDupl->AcquireNextFrame(16, pFrameInfo, &pDesktopResource);
        if (hr == DXGI_ERROR_WAIT_TIMEOUT) {
            return false; // Screen contents did not change; skip frame encode
        }
        if (FAILED(hr)) {
            HandleDeviceLossOrReinitialization(hr);
            return false;
        }

        // Query the Direct3D 11 texture surface directly from GPU VRAM
        hr = pDesktopResource->QueryInterface(__uuidof(ID3D11Texture2D), 
                                              reinterpret_cast<void**>(ppDesktopTexture));
        pDesktopResource->Release();
        return SUCCEEDED(hr);
    }

    void ReleaseFrame() {
        if (m_pDeskDupl) {
            m_pDeskDupl->ReleaseFrame();
        }
    }
};

Why DXGI is superior:

1. Direct GPU Memory Access: Frames remain in video RAM. When hardware encoding (NVENC / Intel QuickSync / AMD AMF) is active, frame buffers can be passed directly to the hardware encoder without round-tripping through system CPU memory (Zero-Copy Pipeline).
2. Dirty Rectangles: DXGI_OUTDUPL_FRAME_INFO reports dirty rects and move rects. When only a cursor moves or a single terminal window scrolls, only modified sub-regions require processing.
3. Sub-Millisecond Acquisition: Frame acquisition typically takes under 0.8ms on discrete GPUs.

3.2 macOS Capture: Apple ScreenCaptureKit

On macOS 12+ (Monterey through Sequoia), we utilize Apple's modern ScreenCaptureKit framework via native Swift/Objective-C bindings.

// macOS Swift: ScreenCaptureKit Frame Stream Implementation
import ScreenCaptureKit

class MacCaptureEngine: NSObject, SCStreamOutput, SCStreamDelegate {
    private var stream: SCStream?

    func initializeStream(display: SCDisplay) async throws {
        let contentFilter = SCContentFilter(display: display, excludingWindows: [])
        let config = SCStreamConfiguration()

        // Match native Retina refresh rate and color space
        config.width = display.width * 2
        config.height = display.height * 2
        config.pixelFormat = kCVPixelFormatType_32BGRA
        config.showsCursor = true
        config.queueDepth = 3
        config.colorSpaceName = CGColorSpace.sRGB

        stream = SCStream(filter: contentFilter, configuration: config, delegate: self)
        try stream?.addStreamOutput(self, type: .screen, sampleHandlerQueue: DispatchQueue(label: "com.conexa.videocapture"))
        try await stream?.startCapture()
    }

    func stream(_ stream: SCStream, didOutputSampleBuffer sampleBuffer: CMSampleBuffer, of type: SCStreamOutputType) {
        guard type == .screen, let imageBuffer = sampleBuffer.imageBuffer else { return }

        // Pass CVPixelBuffer directly to WebRTC VideoTrackSource
        WebRTCBridge.shared.pushNativeFrame(imageBuffer, timestamp: sampleBuffer.presentationTimeStamp)
    }
}

ScreenCaptureKit operates directly within the WindowServer compositing engine, allowing hardware-accelerated capture of HDR displays, spaces, and multi-monitor layouts without triggering high CPU spikes.

4. Real-Time Transport: WebRTC Media & Dual SCTP DataChannels

A common trap in remote desktop design is multiplexing user input (mouse, keyboard, clipboard) over the same channel as video, or relying on separate, non-synchronized TCP sockets.

In Conexa Remote, we utilize a Dual SCTP DataChannel strategy alongside the DTLS-SRTP media stream:

// WebRTC DataChannel Initialization in C++
webrtc::DataChannelInit cursorConfig;
cursorConfig.ordered = false;
cursorConfig.maxRetransmits = 0; // Unordered, partial-reliability for cursor positions

webrtc::DataChannelInit controlConfig;
controlConfig.ordered = true; // Fully reliable, ordered delivery for keyboard & clipboard

auto cursorChannel  = peerConnection->CreateDataChannel("input_cursor", &cursorConfig);
auto controlChannel = peerConnection->CreateDataChannel("input_control", &controlConfig);

5. ICE NAT Traversal & European Data Sovereignty

Establishing a direct peer-to-peer connection over modern consumer and enterprise networks requires traversing multiple NAT tiers (Full Cone, Restricted Cone, Symmetric NAT, and CGNAT).

Conexa Remote executes a strict ICE gathering prioritization:

[Gather Host Candidates] (LAN direct — 0ms RTT)
        │
        ▼
[STUN Reflexive Candidates] (Direct WAN traversal — EU STUN Server)
        │
        ▼ (STUN Success Rate: ~82% in real-world IT fleets)
   Direct P2P
        │
        ▼ (Symmetric NAT on both endpoints or UDP hard-block)
[EU-Hosted TURN Relay Candidate] (DTLS-SRTP opaque datagram relay)

Deterministic Low-Latency Optimization:

1. Aggressive ICE Candidate Pruning: Standard WebRTC implementations often wait 2 to 4 seconds gathering all possible network interfaces before returning candidates. Conexa Remote prunes redundant virtual adapters (e.g., hypervisor host-only bridges, VPN stubs) and gathers candidates in parallel, completing ICE pairing in under 350ms.
2. Infrastructure Residency: All STUN/TURN fallback relays are hosted in Romania (EU). For European enterprises, network traffic stays on high-speed European internet exchange backbones (typically <25ms RTT between Western Europe and Bucharest).

6. Commercial Logic Built into the Protocol: The Floating Concurrent Pool

Most enterprise remote software enforces licensing by embedding aggressive telemetry agents that phone home constantly, tracking named user identities, hardware IDs, and local system configurations.

Conexa Remote takes an architectural approach to licensing: The Floating Concurrent Session Pool.

stateDiagram-v2
    [*] --> Idle: Technician logs in with company email
    Idle --> SlotAllocated: Open connection to Remote Host
    state SlotAllocated {
        [*] --> ActiveSession1: Consumes 1 Slot
        ActiveSession1 --> ActiveSession2: Multitasking: 1 Tech opens 2nd Client (2/2 Active)
    }
    ActiveSession1 --> SlotFreed: Close Window (Host Disconnect)
    ActiveSession2 --> SlotFreed: Close Window (Host Disconnect)
    SlotFreed --> Idle: Slot instantly restored to company pool in 0ms

How Concurrency is Enforced Without Spying:

1. No Named-User Lock: An organization can create 10, 50, or 100 technician profiles and deploy the agent to thousands of endpoints.
2. Session Slot Tokens: When a technician requests a connection, the signaling backend checks the company's active slot counter:
   • Personal Tier: Free (0 EUR) — capped at 1 active session.
   • Professional Tier: 29 EUR/month — pool of 2 active concurrent sessions.
   • Scale: +17 EUR/month per additional concurrent slot.
3. Multitasking Mechanics: A single technician working on dual monitors can open 2 parallel remote sessions to assist two clients simultaneously. This consumes both slots (2/2). Alternatively, two separate technicians can each run 1 session.
4. Instant Slot Recovery (0ms): When the WebRTC PeerConnection transitions to closed or the socket disconnects, the signaling server decrements the active session counter in real-time. The slot is immediately available to any team member.

There are no per-machine license keys to activate, no machine de-authorization hurdles, and no periodic license audit crawls.

7. macOS TCC Security Framework: The Unattended Support Challenge

One of the most complex engineering hurdles in modern desktop support is handling Apple's TCC (Transparency, Consent, and Control) privacy framework on macOS 12+.

To control a remote Mac without an end-user physically present, two permissions are mandatory:

• kTCCServiceScreenCapture (Screen Recording)
• kTCCServiceAccessibility (Input Injection via CoreGraphics Event Taps)

The Security Boundary:

Apple deliberately prevents non-root and root user space processes from modifying the local TCC database (/Library/Application Support/com.apple.TCC/TCC.db). Any utility claiming to "silently enable Mac remote access without user consent or MDM" is either using a transient vulnerability or violating Apple's security model.

Conexa Remote's Dual Strategy:

1. Enterprise Fleets (MDM Deployment): For managed Macs (Jamf, Mosyle, Kandji), we provide a signed configuration profile payload (.mobileconfig). This pre-authorizes the application bundle ID and cryptographic signing requirements silently:

   <dict>
       <key>Services</key>
       <dict>
           <key>ScreenCapture</key>
           <array>
               <dict>
                   <key>Allowed</key><true/>
                   <key>CodeRequirement</key>
                   <string>anchor apple generic and identifier "com.conexa.remote"</string>
               </dict>
           </array>
       </dict>
   </dict>

1. Ad-Hoc / SMB Environments (Non-MDM): The application detects missing entitlements on launch and opens the exact deep link directly into System Settings (x-apple.systempreferences:com.apple.preference.security?Privacy_ScreenCapture), guiding the user through a single guided verification step.

8. Performance Benchmarks

We measured the performance of Conexa Remote against legacy remote desktop solutions over a symmetric 1 Gbps fiber connection across Western and Eastern Europe:

9. Conclusion & Takeaways

By stepping away from legacy client-server relay topologies and leveraging standardized WebRTC protocols with native OS APIs:

• We achieved glass-to-glass latencies down to 12ms on local networks and under 45ms across trans-European links.
• We eliminated the need for intermediary video relay clusters, drastically reducing operational server overhead and guaranteeing EU GDPR compliance by design.
• We replaced predatory per-seat licensing models with a fair, transparent floating concurrent session pool that aligns directly with how technical support teams actually work.

The Conexa Remote clients for Windows 10/11 and macOS 12+ are available for download at conexaremote.com. The Personal tier is permanently free (1 concurrent session, no credit card required).

Discussion & Questions

If you've built real-time WebRTC tools or desktop capture pipelines:

1. What approaches have you taken to handle macOS pre-boot FileVault login screens for unattended access?
2. How do you balance congestion control backoff against cursor responsiveness during network packet loss spikes?

We'd love to hear your insights and architectural war stories in the comments below!