face-recognition

How it is built

Architecture overview

A face-unlock system decides, in a fraction of a second and without the user doing anything, whether the face in front of a camera belongs to the enrolled owner, and only then releases access to the device. It is a pipeline of five small stages (sense, detect and align, embed, compare, decide) wrapped in a security envelope that matters as much as the neural network. The network turns a face into a short vector, a template, so that two images of the same person land close together and two different people land far apart; recognition is then a distance test against the stored templates. The one idea that makes it work: the biometric is a probabilistic check, so every stage that could be fooled or bypassed is pushed into hardware that the operating system cannot read or inject into, and the check only releases a key, never a flag. The same pipeline runs in two modes: 1:1 verification (is this the owner? phone unlock, payment approval) and 1:N identification (who is this? door access, photo grouping). Face ID on iPhone is the documented reference for the first mode; open pipelines such as InsightFace and OpenCV's face models show the second.

Parts:

• Sensor and depth — infrared camera plus a projected dot pattern, giving a depth map and an infrared image — structured-light-depth (no page yet)
• Detection and alignment — find the face, locate a few landmarks, warp to a canonical crop — Object detection, facial-landmark-alignment (no page yet), Convolutional networks
• Embedding network — crop in, unit-length vector out — face-embeddings (no page yet), Convolutional networks, Loss functions, Contrastive learning, Weight quantization
• Presentation-attack defence — reject photos, screens, masks and a sleeping owner — liveness-detection (no page yet)
• Matcher and template store — compare against enrolled templates; for 1:N, search a gallery — HNSW, Calibration, Evaluation metrics
• Secure execution and key release — run the matcher where the OS cannot see it, release a key on success, count failures — trusted-execution-environments (no page yet), hardware-backed-keystore (no page yet), AEAD (AES-GCM, ChaCha20-Poly1305), Message authentication codes, Replay protection window, Secure boot chain, Secure memory zeroization

 wake (raise, tap)         ┌────────────────────── secure processor (TEE) ────────────────────────┐
        │                  │                                                                      │
 ┌──────▼──────┐  signed   │  ┌─────────┐   ┌────────────┐   ┌──────────┐   ┌───────────────┐    │
 │ IR camera + │  frames   │  │ detect, │──►│ embedding  │──►│ matcher  │──►│ threshold +   │──┐ │
 │ dot pattern │──────────►│  │ align   │   │ network    │   │ vs. enrol│   │ liveness gate │  │ │
 │ (depth + IR)│  (MAC)    │  └─────────┘   └────────────┘   │ templates│   └───────────────┘  │ │
 └─────────────┘           │       ▲             ▲           └────▲─────┘          │ match     │ │
                           │  attention check    └─ model weights │ encrypted      ▼           │ │
                           │                                     │ templates   failure counter  │ │
                           └─────────────────────────────────────┼─────────────────┬────────────┘ │
                                                                 │                 │ release key  │
                                       enrolment (once) ─────────┘                 ▼              ▼
                                                                          keystore: unlock / sign / pay
                                                                          (no match: passcode fallback)

The design has a runtime side (latency, power, error per attempt) and an offline side (training, threshold choice, evaluation); they share one contract: the exact preprocessing and the embedding version.

How the parts fit together

Sensor and depth

A plain colour camera sees a photograph as a face. The unlock design therefore adds a second, hard-to-fake signal: Apple describes Face ID as projecting and reading thousands of infrared dots to form a depth map of the face plus a 2D infrared image (Apple Platform Security guide). structured-light-depth (no page yet) is that idea: a known pattern deformed by the surface gives depth by triangulation, in the dark and under any visible light. Time-of-flight and stereo cameras are alternatives with different power and accuracy; the contract downstream is the same, a depth map aligned with an infrared image. Apple also randomises the order of the 2D and depth captures and projects a device-specific random pattern, so a recorded sequence from another device does not fit (same source). Depth separates a face from a picture of one; it does not replace the liveness check below.

Detection and alignment

Recognition accuracy depends on seeing the same crop every time, so a detector finds the face and a landmark step places a handful of points (eyes, nose, mouth corners). Object detection covers the box-regression half; single-stage face detectors add landmark outputs to the same network, as RetinaFace does with five landmarks, and the MTCNN cascade predicts box and landmarks in three coarse-to-fine stages. facial-landmark-alignment (no page yet) is the next step: fit a similarity transform (rotation, scale, shift) that moves the landmarks to a fixed template and resample the face to a small fixed crop. A bad landmark gives a rotated or off-centre crop and the embedding silently gets worse, so alignment quality is monitored. The detector and landmark networks are Convolutional networks; they run first and most often, so they are the smallest model in the system and the one tuned hardest for latency.

Embedding network

The crop goes into a convolutional network whose last layer is a vector of a few hundred numbers, L2-normalised so that cosine similarity equals the dot product (face-embeddings, no page yet). What matters is the training objective (Loss functions):

• Triplet loss (FaceNet) pulls an anchor and a positive together and pushes a negative away by a margin; FaceNet reported 128 bytes per face and 99.63 % on the LFW benchmark, and introduced online triplet mining to pick informative triplets. It is a form of contrastive learning.
• Margin-softmax losses (ArcFace and relatives) train a classifier over identities but add an angular margin between classes, which gives the same tight clusters without triplet mining; they are the common choice for open recognisers.

The deployment contract is a single embedding version: the template stored at enrolment and the vector computed at unlock must come from the same network and the same preprocessing. FaceNet's paper discusses "harmonic embeddings" because upgrading the network otherwise invalidates every stored template; a phone avoids the problem by keeping the enrolment images or re-enrolling, a server by re-embedding the gallery. On a phone the network is quantized to 8-bit integers and run on a neural accelerator, because the budget is tens of milliseconds and a little energy, not accuracy at any cost.

Presentation-attack defence

Liveness (liveness-detection, no page yet) is a separate classifier, not a property of the embedding. Face ID combines several signals: depth that a flat photo lacks, infrared texture, and an attention check that the eyes are open and directed at the device, so that a sleeping or forcibly held owner does not unlock it (Apple Platform Security guide, which states that neural networks decide attention, matching and anti-spoofing). Android's compatibility definition measures this explicitly as the spoof acceptance rate (SAR) per attack type, apart from the false acceptance rate (source.android.com, "Measure biometric unlock security"): a model can be excellent against random strangers and weak against a printed photo. Keep these two numbers separate in every evaluation.

Matcher and template store

At unlock, the matcher computes the similarity between the new embedding and the enrolled templates and compares it with a threshold. The enrolled data is not one vector: Apple states it is a mathematical representation captured across a variety of poses, and that the device adapts it as the face changes, asking for the passcode before a larger update such as shaving a full beard (Apple Support, "About Face ID advanced technology"). Updating a template only from confirmed matches keeps drift slow and makes poisoning (an attacker slowly shifting the template) harder.

For 1:N identification the gallery can hold millions of vectors, so exact comparison is replaced by approximate search with HNSW, followed by a verification step on the top candidates. This is the one place the graph's index block does real work; a phone unlock is 1:1 and needs no index. A caveat that is arithmetic, not a measurement: if one comparison has a false-match rate of p, a search over N people produces a false match with probability about N·p when p is small, so a gallery search needs a threshold far stricter than a 1:1 check to keep the same overall error.

The threshold itself is a product decision made with Evaluation metrics: plot false non-match rate against false match rate on held-out pairs, pick the operating point from the false-match target, and read the false non-match rate (the owner locked out) from the same curve. Calibration converts raw similarity into a score whose meaning does not change when the network is retrained. Targets differ by product: Apple states that the chance a random person unlocks a device is less than 1 in 1,000,000 with a single enrolled appearance, while Android's Class 3 (strong) biometric requires a false acceptance rate of 1/50k and a false rejection rate of 10 % or less, plus a spoof acceptance rate of 7 % or less for the common attack species (source.android.com).

Secure execution and key release

This is the envelope. The design rules visible in both the Apple and Android documents are the same:

• Separation. The sensor captures and sends; the matcher and the template live in a secure processor or trusted execution environment (trusted-execution-environments, no page yet). In Apple's design the sensor frames are digitally signed and sent to the Secure Enclave, a portion of the neural engine protected inside it does the matching, and the enrolled data is encrypted with a key only the Secure Enclave holds. A kernel compromise must not be able to read raw biometric data or inject a synthetic image; Android's CDD names exactly those two properties as "architectural security".
• Authenticated frames. The channel from sensor to matcher is protected with a MAC and a replay window, so a captured frame cannot be fed in again. Templates at rest use AEAD.
• A match releases a key, not a boolean. On success the secure processor lets a hardware-backed key be used (hardware-backed-keystore, no page yet): to decrypt data, to sign a login challenge as in Passkeys, to authorise a payment. An application that merely asks "was it Alice?" can be fooled by patching the answer; one that needs the key cannot. Android's classes make this explicit: a Class 2 biometric cannot be tied to Keystore keys, a Class 3 can.
• Rate limit and fallback. Failures are counted by hardware that survives a reboot: Face ID allows only five unsuccessful match attempts before the passcode is required (Apple Support). The secure attempt counter has no block page yet (secure-attempt-counter, no page yet).
• Trusted software. The matcher and the model weights are only as trustworthy as the boot path, so the system is anchored in a secure boot chain; key material and templates in memory are handled with secure memory zeroization.

Data and control flow

One unlock attempt (verification):

1. A wake event (raise, tap, notification) starts the sensor; the camera looks for a face (order of tens of milliseconds, all hardware).
2. The detector finds a face; the attention check confirms open eyes and gaze. If either fails, stop here and save the energy.
3. The infrared projector and camera capture a randomised sequence of depth and infrared frames; the sensor signs them and passes them to the secure processor.
4. In the secure processor: check frame signatures and freshness, align the crop, run the embedding network and the liveness classifier.
5. Compare the embedding with the enrolled templates; decision = similarity above threshold AND liveness passes.
6. On success, release the unlock key and, if the match was confident, optionally fold the new embedding into the template. On failure, increment the failure counter; after the limit require the passcode. Latency is the sensor exposure plus the networks; energy-hungry steps are gated behind cheap ones.

Enrolment:

1. The user turns their head through guided poses, so the capture covers many angles and the quality check can reject blur and glare.
2. For each accepted frame the same pipeline computes an embedding in the secure processor.
3. The embeddings are stored encrypted under a key only that processor holds; no raw image has to leave it.

One identification query (1:N, server or edge):

1. Detect and align the probe face; embed it with the gallery's embedding version.
2. Search the HNSW index for the nearest k vectors.
3. Re-score the top candidates with a stricter threshold and, if required, a second model; return identity or "unknown".
4. Log the decision for audit; a human reviews low-margin matches in sensitive uses.

Scaling

In the order you would pull the levers:

1. Latency and power on the device. Smaller backbone, int8 quantization, running the detector at a low rate and the embedder only when the face and attention checks pass, hardware blocks for the sensor timing.
2. False-accept rate. Better data and a margin loss, quality gating (do not match a blurred frame at all), a stricter threshold paired with a cheap retry, a second sensor signal (depth) rather than a bigger network.
3. Population coverage. NIST's face recognition evaluation reports false match and false non-match differences across demographic groups (NISTIR 8280, December 2019, listed on pages.nist.gov/frvt); test the operating point per group, because one global threshold can mean different error rates for different people. This is a data and evaluation problem before it is a model problem.
4. Gallery size (1:N). HNSW with a second-stage verifier, sharding by region or enrolment group, a coarse metadata filter to shrink the search, then the stricter threshold from the arithmetic above.

Failure modes

Real implementations

The alternative: a server-side recogniser (cameras upload frames, a service searches a gallery) trades the hardware envelope for scale; its 1:N error budget is far tighter than a 1:1 unlock because of the arithmetic above.

Sources

• Apple, Apple Platform Security guide, Biometric security (Face ID security): sensor to Secure Enclave separation, signed and randomised captures, attention check, neural-network matching. page
• Apple Support, About Face ID advanced technology: false-match probability, five attempts, template adaptation, key held only by the Secure Enclave. page
• Android Open Source Project, Measure biometric unlock security: architectural security, SAR/FAR/IAR, biometric classes. page
• F. Schroff, D. Kalenichenko, J. Philbin, FaceNet: A Unified Embedding for Face Recognition and Clustering, 2015. paper
• J. Deng et al., ArcFace: Additive Angular Margin Loss for Deep Face Recognition (TPAMI version), 2018. paper
• J. Deng et al., RetinaFace: Single-stage Dense Face Localisation in the Wild, 2019. paper
• K. Zhang et al., Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks, 2016. paper
• Y. Zhong et al., SFace: Sigmoid-Constrained Hypersphere Loss for Robust Face Recognition, 2022. paper
• NIST, Face Recognition Technology Evaluation (FRTE), 1:1 Verification and Demographic Effects (NISTIR 8280, 2019). FRTE 1:1, demographics