Patent-wise, Apple’s #1 project in 2020 and 2021 has clearly centered around their future mixed reality headset. Today, the U.S. Patent and Trademark Office published an issued patent for Apple relating to 3D modeling of fingers for use in a mixed reality headset (not glasses) for object and gesture recognition.
Generation of a 3D model of a fingertip for visual touch detection
Apple’s patent describes in part the generation of a 3D model of fingers that could be used to touch virtual objects and gesture in the air for the headset to understand commands.
Apple notes in patent point #31 that “a user may turn a finger in front of a camera in order to interact with a physical environment or log into an account” and so on.
Apple’s patent FIG. 4 shows a flowchart of an example technique for using a 3D model of a fingertip for gesture recognition; FIG. 5 shows an example system diagram illustrating a system configuration for generating a 3D model of a fingertip.
Following Fig. 4, hand model can include 3D models of a hand corresponding to various hand poses. The flowchart continues at No. 410, and electronic device 100 (iPhone) updates the 3D model of the hand based on the first and second images. For example, a depth image can be obtained from a perspective such that a tactile point between the touching object and the surface is occluded by the touching object (for example, the fingertip touching a surface can be occluded by the hand from the user’s point of view).
The model of the touching object may be generated or refined based on the combined data from the touch images and/or the depth data. The additional finger information can be used to update or refine a 3D model of a hand that includes the finger.
In one or more embodiments, ICP (iterative closet point) algorithms, edge alignment based algorithms, or any mutual information based algorithm could be used to fit the hand model or the finger model with an image of the hand or finger to enhance or enable hand tracking or touch sensing.
Apple’s patent covers both finger gesture recognition and visual touch sensing.
For details, see Apple granted patent 10,970,922.