Ios swift – PHPickerViewControllerDelegate — special handling for limited case? Why is didFinishPicking not working seamlessly?

I am using the new PHPickerViewController & Delegate on iOS15+.

I want my code to get get a callback from the picker (regardless of full photo access or limited) that the user has selected photo x, y, and z to attach to this chat. I’d like this callback regardless of the level of access given to my app to the photo library.

Here is how I show the photo picker:

            var configuration = PHPickerConfiguration(photoLibrary: .shared())
            // Set the filter type to images only for now
            configuration.filter = PHPickerFilter.any(of: [.images])
            // Set the mode to avoid transcoding, if possible, if your app supports arbitrary image/video encodings.
            configuration.preferredAssetRepresentationMode = .compatible
            // Set the selection behavior to respect the user’s selection order.
            configuration.selection = .ordered
            // Set the selection limit to enable multiselection.
            configuration.selectionLimit = 8
            // Set the preselected asset identifiers with the identifiers that the app tracks.
            configuration.preselectedAssetIdentifiers = self.selectedAssetIdentifiers
            // Show it
            let picker = PHPickerViewController(configuration: configuration)
            picker.delegate = self
            (UIApplication.shared.delegate as? AppDelegate)?.window?.rootViewController?.present(picker, animated: true, completion: nil)

And here is my delegate:

func picker(_ picker: PHPickerViewController, didFinishPicking results: [PHPickerResult]) {
        dismiss(animated: true)
        let existingSelection = self.selection
        var newSelection = [String: PHPickerResult]()
        for result in results {
            let identifier = result.assetIdentifier!
            newSelection[identifier] = existingSelection[identifier] ?? result
        }
        
        // Track the selection in case the user deselects it later.
        _selection = newSelection
        selectedAssetIdentifiers = results.map(.assetIdentifier!)
        selectedAssetIdentifierIterator = selectedAssetIdentifiers.makeIterator()
        
        // Convert and load into the attachment manager
        let fetchResult = PHAsset.fetchAssets(withLocalIdentifiers: results.compactMap(.assetIdentifier), options: nil)
        // Loop through the objects in the fetch result using enumerateObjects
        fetchResult.enumerateObjects { (asset, index, stop) in
            // Do something with 'asset' and 'index'
            let options = PHImageRequestOptions()
            options.deliveryMode = .highQualityFormat
            options.resizeMode = .fast
            options.isNetworkAccessAllowed = true
            // Ideally we handle this better in the UI without freezing.
            options.isSynchronous = true
            
            let image = asset.image(targetSize: PHImageManagerMaximumSize, contentMode: PHImageContentMode.default, options: options)
            let handled = self.attachmentManager.handleInput(of: image)
            if !handled {
                // throw error
            }
        }
    }

In the limited photo mode, it seems like my didFinishPicking delegate method is never called—or at least never called in situations where the user has selected a photo from their library that they did not give me permission with the first time round.

Are there other methods I need to implement to support this privacy focused mode?

Here is the user flow generally, use case is a chat app where users can pick photos to attach to a message. The user selects ‘use only selected photos’, which then opens the picker to select which photos to give the app access to

After that photo picker modal, nothing happens in privacy mode. No delegate methods are called, nothing. So my app now has access to certain photos—however, I’d like/need the user to further pick which photos, of the allowed photos, to attach to this specific chat.

If the user had selected full access, they would get a similar picker, but the delegate methods would fire and tell me which photos the user picked.

This is where my confusion is. I can’t find a definitive source for handling this case for what I expected to be a straight forward application of "User picks photos, I get photo in a callback so I can upload it to my backend"