Mobile collaboration layers

Mobile co-apping

Mobile co-apping lets an agent observe a visitor use your app in real time. By default, the SDK captures the content of your app’s own views. You can also switch it to capture everything on the screen, including content outside your app. Co-apping then continues while your app is in the background, for example when the visitor switches to another app.

For the configuration properties that govern mobile co-apping, refer to Configuring the mobile co-apping collaboration layer.

In-app co-apping

In-app co-apping is the default capture mode. The SDK takes a screenshot whenever the content of your app changes and transmits it to the other conversation participants. It captures only your app’s own views, so co-apping pauses when the visitor sends your app to the background.

This capture mode needs no additional setup.

Out-of-app co-apping

Out-of-app co-apping captures everything on the screen, not just your app’s own views. Co-apping keeps running when the visitor switches to another app. This lets agents guide visitors through device settings, a third-party app, or any other screen. Capture stops when the visitor closes your app or ends the session.

To capture content outside your app, the SDK uses the Apple ReplayKit framework and a Broadcast Upload Extension that you add to your app. For the extension and the App Group it needs, refer to Capturing beyond your app.

Before you offer out-of-app co-apping, check the isReplayKitAvailable property. It reports whether the device and operating system support ReplayKit capture but doesn’t tell you whether the Broadcast Upload Extension is configured.

You select the capture mode with the setCaptureMode(_:completion:) method of the UnbluMobileCoBrowsingModuleApi, passing one of the CoBrowsingCaptureMode values:

screenshot captures your app’s own views. This is the default.
replayKit captures everything on the screen.

You can set the capture mode before or during a session. If you switch to replayKit while co-apping is active, the visitor is prompted to allow your app to capture the screen.

When the agent marks an area of the visitor’s screen, the marker isn’t shown outside your app by default. To display it, call the setAgentMarkerVisibleOutsideApp(_:) method with true. The SDK then shows the marker in a Picture-in-Picture (PiP) window when the visitor switches to another app. This requires iOS 15 or later and the "Audio, AirPlay, and Picture in Picture" background mode in your app. It only takes effect in replayKit mode.

Listing 1. Enabling out-of-app co-apping

// `coBrowsingModule` is your UnbluMobileCoBrowsingModuleApi instance.
// See the Modules article for how to create and register it.

// Set the capture mode to capture everything on the screen instead of just your app's own views.
// The trailing closure runs when the switch completes.
coBrowsingModule.setCaptureMode(.replayKit) { success, error in
    if success {
        // The capture mode is now ReplayKit.
    } else {
        // Switching failed; `error` describes why.
    }
}

// Show the agent's marker in a PiP window while the visitor is outside your app.
coBrowsingModule.setAgentMarkerVisibleOutsideApp(true)

marking areas outside your app is an experimental feature. Test it extensively before using it in your app.

Private views and areas

Private views and areas apply to mobile co-apping in both capture modes. In out-of-app co-apping, they only take effect while your app is in the foreground. Once the visitor switches to another app, Unblu can no longer hide content on the screen.

As part of the UnbluMobileCoBrowsingModule, Unblu provides a simple way to specify views that shouldn’t be displayed on the remote side. After adding a view’s tag, the view is automatically overlaid by an image before the screen is transmitted to the remote side.

The UnbluMobileCoBrowsingModuleApi provides the function addPrivateView which takes a tag as its input parameter. You can define a different tag for each view.

You can make a view visible to the remote side again by calling the removePrivateView function, which is also exposed on the UnbluMobileCoBrowsingModuleApi.

The SDK also provides a way to specify areas that shouldn’t appear on the remote side. This is especially useful if you use a webView or a canvas-based framework to render your UI. Often, there aren’t any tags on the view that could be used to overlay the element during co-apping.

To hide an area, the UnbluMobileCoBrowsingModuleApi provides the function addPrivateArea. The function takes parameters that specify the area’s ID, the x and y coordinates of the upper left-hand corner, and its width and height. Each private area has its own ID that you can call the method with again to update the area’s values.

After an area is added, it’s automatically overlaid by an image before the screen is transmitted to the remote side. You can make an area visible to the remote side again by calling the function removePrivateArea, which is also exposed on the UnbluMobileCoBrowsingModuleApi.

Animations and private views and areas

If you use animations to transition from a view that contains a private view or area, the overlay may not conceal the content during the entire transition.

If you know your app uses animations in views that contain private views or areas, you should take measures to ensure that the content isn’t exposed during the animation. For example, if you know the animation is always horizontal, you might make the overlay larger, so that it covers the area the concealed content traverses during the animation.

Screen sharing publishes everything on the visitor’s screen to the conversation as a screen sharing layer. The shared screen appears in the agent’s screen sharing view, just as it does for screen sharing on the web.

On mobile, screen sharing works in one direction only: a visitor can share their screen with the other conversation participants, but the SDK can’t display a screen that another participant shares.

Screen sharing on mobile uses LiveKit, which you must set up as described in Setting up screen sharing with LiveKit. The screen is captured through the same Broadcast Upload Extension as out-of-app co-apping. Refer to Capturing beyond your app for setting up the extension and App Group involved.

You start and stop screen sharing with the conversation’s startScreenSharing(onComplete:) and stopScreenSharing(onComplete:) methods. Starting screen sharing requires the UnbluCallModule to be registered. It shows the iOS system broadcast picker, which the visitor uses to begin sharing.

To check whether screen sharing is active, read the isScreenSharingActive property.

Each method reports its outcome through a Result. On failure, the UnbluConversationError is one of:

failedToStartScreenSharing(_:): Screen sharing couldn’t be started. The associated string describes the reason.
failedToStopScreenSharing(_:): Screen sharing couldn’t be stopped. The associated string describes the reason.

Listing 2. Starting screen sharing and handling errors

// `conversation` is the UnbluConversation you want to share the screen from.
// See "Working with the Unblu iOS mobile SDK" for information on how to obtain it.
conversation.startScreenSharing { result in
    switch result {
    case .success:
        // Screen sharing is now active.
        // Reflect it in your app's UI, for example by showing a "stop sharing" button.
        showStopScreenSharingButton()
    case .failure(.failedToStartScreenSharing(let reason)):
        // `reason` describes why screen sharing couldn't start.
        showMessage("Screen sharing isn't available: \(reason)")
    case .failure:
        // Any other error.
        showMessage("Something went wrong. Please try again.")
    }
}

Before the SDK can start screen sharing, an administrator must enable mobile screen sharing on the Collaboration Server with the com.unblu.mobiledevice.mobileScreenSharingEnabled configuration property. Otherwise, startScreenSharing fails with failedToStartScreenSharing.

For the full set of screen sharing configuration properties, refer to Configuring the screen sharing collaboration layer.

Capturing beyond your app

Out-of-app co-apping and screen sharing both capture beyond your app through the Apple ReplayKit framework (in-app co-apping doesn’t need it). ReplayKit performs the capture in a separate Broadcast Upload Extension, which passes the captured frames to your app through a shared App Group. You must add and configure this extension before either feature works.

A single Broadcast Upload Extension serves both features. To set it up:

Add a Broadcast Upload Extension target to your app in Xcode.
Make the extension’s sample handler subclass UnbluScreenSharingHandler and set it as the extension’s principal class. The subclass needs no code of its own. The SDK captures and forwards the frames.
Enable the App Groups capability on both your app target and the extension target, using the same App Group identifier for each.
Add the App Group identifier to the extension’s Info.plist under the key RTCAppGroupIdentifier. If you don’t, the SDK uses the default App Group identifier group.com.unblu.cobrowsing.shared.
Add the extension’s bundle identifier to your app’s Info.plist under the key RTCScreenSharingExtension. Without this key, screen sharing isn’t offered and out-of-app co-apping stays in the screenshot capture mode.

Once you add the key, a request to start screen sharing appears each time co-apping starts. If the user denies the request, the SDK launches a normal mobile co-apping session instead.

Listing 3. Broadcast Upload Extension sample handler

import ReplayKit
import UnbluCoreSDK

/*
 Set as the Broadcast Upload Extension's principal class.
 Subclass `UnbluScreenSharingHandler` instead of the generated `RPBroadcastSampleHandler`.
 You don't override anything; the SDK captures the screen and forwards the frames.
 */
class SampleHandler: UnbluScreenSharingHandler {
}

Privacy and masking

When capture extends beyond your app, Unblu can’t control what the device shows. The private views and areas you define only hide content while your app is in the foreground. Once the visitor switches to another app, its content is captured as it appears, and screen sharing captures everything on the screen.

For content that must never be captured, rely on iOS’s own protection rather than Unblu’s masking. An app can detect an active capture through the UIScreen.isCaptured property and hide or blank its sensitive views while capture is in progress.

Document collaboration

Document collaboration lets an agent present a document to the visitor during a conversation. The visitor can view the document and, depending on the conversation’s native document collaboration configuration, annotate and sign it. Document collaboration is provided by the UnbluDocumentCoBrowsingModule, which renders documents natively using the Apryse iOS SDK.

Registering the module

Create an instance of the module with UnbluDocumentCoBrowsingModuleProvider.create and register it on your UnbluClientConfiguration before you create the client:

Listing 4. Registering the UnbluDocumentCoBrowsingModule

import UnbluCoreSDK
import UnbluDocumentCoBrowsingModule

let documentCoBrowsingModule = UnbluDocumentCoBrowsingModuleProvider.create()
try configuration.register(module: documentCoBrowsingModule)

For information on creating the UnbluClientConfiguration, refer to Unblu client configuration.

Presenting the document

When document collaboration starts, the module provides a UIViewController that displays the session. The presentationMode property controls how to show the view controller:

.default: the SDK presents the view controller for you.
.custom(_:): you present or embed the view controller yourself. The case takes a closure that the SDK calls when the document UI should appear. Call the closure’s completion parameter with true once you’ve presented the view controller, or false if presentation failed.

Listing 5. Setting the presentation mode

documentCoBrowsingModule.presentationMode = .default

// Or, to present the document UI yourself:
documentCoBrowsingModule.presentationMode = .custom { completion in
    // Present or embed the view controller received in the delegate callback
    completion(true)
}

To respond to the collaboration lifecycle, assign a delegate that adopts the UnbluDocumentCoBrowsingModuleDelegate protocol. The delegate is notified in both presentation modes. In .custom mode, unbluDocumentCoBrowsingModuleDidStartCoBrowsing is also where you receive the UIViewController to present:

Listing 6. Handling document collaboration events

documentCoBrowsingModule.delegate = self

func unbluDocumentCoBrowsingModuleDidStartCoBrowsing(_ api: UnbluDocumentCoBrowsingModuleApi, _ uiController: UIViewController) {
    // In .custom mode, present or embed uiController in your app.
    // In .default mode, the SDK has already presented it.
}

func unbluDocumentCoBrowsingModuleDidStopCoBrowsing(_ api: UnbluDocumentCoBrowsingModuleApi) {
    // Perform any clean-up
}

Controlling the document collaboration session

The UnbluDocumentCoBrowsingModuleApi provides functions to control the document collaboration session:

showDocumentCoBrowsing(success:) presents the document collaboration UI on the device. Use it to show the UI again if it was dismissed while the session is still active.
stopDocumentCoBrowsing(stopLayer:success:) closes the document collaboration UI.
- If stopLayer is true, the call also stops the collaboration layer. This ends the document collaboration session for all participants.
- If stopLayer is false, the collaboration layer remains active and the session continues.
isDocumentCoBrowsingActive(success:) reports whether there’s an active document collaboration session in the conversation.

Each function takes a success closure that’s called with a Boolean result.