Most efficient way to remove these objects – Ios swift

I’m trying to loop through a set of Users, and remove them from another set LostFollowers (if they exist). Users are identifiable by id.

struct LostFollower {
  let user: User
  let dateLost: Date
}

let users1: Set<LostFollower> = // …
let users2: Set<User> = // …

users2.forEach { user in
  // Need to remove anyone in the set users1, whose `user` property is equal to user
}

How can I do this? Will I need to use filter or is there a better way to do it?

Note: Set operations won’t work because users1 and users2 are of different types.