Person segmentation not being recorded

[developer-appdam]

Hi all,

First of all, thanks for this amazing library! I'm trying to record an ARSCNView that is configured with personSegmentation:

if ARFaceTrackingConfiguration.supportsFrameSemantics(.personSegmentation) { config.frameSemantics.insert(.personSegmentation) }

but it doesn't seem to record actually the segmented output that I see in the liveview.
What can be the cause?

[digitallysavvy]

The reason you don't see the segmentation is because currently Apple doesn't expose an API for getting the rendered image from an ARSCNView. The way ARVideoKit works around this limitation is using an offscreen SCNRenderer, (leveraging the ARSCNView to access the scene graph) and the ARKit callback that provides the raw image from the camera. to the composite the rendered 3D assets and with the raw image.

When using an offscreen renderer in this way, its not accounting for any image manipulation that is done for segmentation. To achieve the segmentation there are a couple different ways, either by modifying ARVideoKit or using its output. Modifying the library to handle the segmentation would be the most efficient way, but would require you to a bit more effort to make it flexible enough to fqualify for a PR to the library (vs. coding just what you need). So I'll leave that up to you.

If you want to modify the image using the output from ARVideoKit, then you need use the raw image (from ARKit), the CVPixelBuffer (output from ARVideoKitRenderARDelegate), along with CoreML (to detect the person) and OpenCV (to manipulate the image to create the mask, that will be used to merge the rawImage with the rendered image). Once you've modified the frame using OpenCV, you'll either create an image using the the frame or create an asset writer to push the frame into a video file.

    open func frame(didRender buffer: CVPixelBuffer, with time: CMTime, using rawBuffer: CVPixelBuffer) {
       // use coreML and OpenCV to Manipulate the Image
       // write the output from OpenCV to a video file.
    }

[royherma]

@digitallysavvy Any updates regarding this functionality? Seems like a high demand functionality for this library so hoping there is some good news 🙌

[digitallysavvy]

I'm not actively working on this feature, my comment was meant to provide enough context to allow any developer looking for this feature in their app to help the community and extend the library.