← Back
Home › Systems › AI applications

visual-search

Answers a question asked with a picture by retrieving similar items from a huge collection of stored vectors and then ranking a few hundred with a heavier model.

Google Lens and Circle to Search, Apple Visual Intelligence and Visual Look Up, Pinterest Lens, Bing Visual Search, Amazon Lens

Open the visual-search pagevisual-search Scatter-gather querydistributed: Send a query only to shards that can match, merge their small partial results, and bound the wait for stragglers.›Image resampling filtersgraphics: Resize an image by convolving with a kernel (box, triangle, cubic, Lanczos) whose width scales with the shrink factor, done as two separable 1-D passes with precomputed weights.›Contrastive learningmachine-learning: Train an encoder so two views of the same thing score higher than every other pair in the batch; InfoNCE with a temperature, no labels needed.›Embeddingsmachine-learning: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›Entity linkingmachine-learning: Map each mention to a knowledge-base id, or to NIL, by generating candidates from an alias table, scoring them on prior and context, and thresholding.›Image segmentationmachine-learning: Pixel/instance segmentation networks and promptable segmentation (Mask R-CNN, SAM-style)Learning to rankmachine-learning: Train a scorer on query-grouped data so that the order it induces maximizes NDCG; LambdaMART (boosted trees with NDCG-weighted pair gradients) is the production default.›Matryoshka embeddingsmachine-learning: Training embeddings whose prefixes are usable at lower dimensions for cheaper searchObject detectionmachine-learning: A network turns an image into scored, classified boxes in one pass; anchor-based, anchor-free and set-prediction heads differ in how they assign labels and remove duplicates.›Optical character recognition(text recognition)machine-learning: Find text lines in an image, read each crop with a CTC or attention model, then correct with a lexicon and score confidence for review routing.›Transformer blockml-training: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.›Vision transformerml-training: ViT: patch embeddings, position encodings, encoder blocks; vision-language connectorsWeight quantizationml-training: Store weights as 8/4-bit integers with per-group scales; decode is memory-bound, so 4-bit weights give about 3-4x tokens/s and fit 4x larger models.›Cross-encoder rerankingsearch-sort: Re-score the top-k candidates of a cheap first stage with a model that reads query and candidate together; more accurate than separate embeddings, but one model call per pair.›Nearest-neighbour search withfilterssearch-sort: Combine approximate nearest-neighbour search with attribute predicates by pre-filtering, post-filtering or filter-aware graph traversal, chosen by how many items pass.›HNSWsearch-sort: Hierarchical Navigable Small World: a layered proximity graph searched greedily from a sparse top layer; the default in-memory approximate nearest-neighbour index.›IVF-PQsearch-sort: Inverted file plus product quantization: cluster vectors into lists, store each as a few bytes of codes, scan only the nearest lists via table lookups.›Perceptual hashingsearch-sort: Compact fingerprints (64 to 256 bits) of an image that stay close under resizing and recompression, compared by Hamming distance; pHash, dHash and Meta's PDQ.›Query expansionsearch-sort: Rewriting queries with synonyms, pseudo-relevance feedback or LLM-generated variants (multi-query, HyDE)K way mergesearch-sort: Merge k sorted streams with a heap or loser tree (merging iterators)Deadline propagationsystems: Carry an absolute deadline and cancellation along a call chainHedged requestssystems: Sending a backup request after a latency percentile and taking the first answer to cut tail latencyConvolution image filtersgraphics: Separable 2D convolution kernels for blur, sharpen (unsharp mask) and edges with border handlingNegative samplingmachine-learning: Train by contrasting each observed positive with a few sampled negatives instead of normalising over every candidate; the choice of sampler decides bias and quality.›AdamWml-training: Adam's per-parameter adaptive steps with weight decay applied directly to the weights; the default optimizer for transformers and LLM pretraining.›Reverse-mode automaticdifferentiationml-training: Record the forward computation on a tape, then sweep it backwards once to get the gradient of a scalar loss for all inputs.›Named entity recognitionmachine-learning: Find the character spans of entities in text and give each a type, by labelling every token and decoding a valid label sequence.›Inverted indexstorage: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.›Loss functionsmachine-learning: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.›Evaluation metricsmachine-learning: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.›Gradient-boosted decision trees(GBDT)machine-learning: Add small regression trees one at a time, each fitted to the loss gradient; histogram splits over 256 bins make it the strongest tabular default.›+ 17 moreConvolutional codes and theViterbi decodersignal-processing: Streaming error-correcting code: each output bit is a parity of the last K input bits; the Viterbi decoder finds the most likely path through a 2^(K-1)-state trellis.›Bit packing and PFORcompression: Store blocks of integers with exactly b bits each and patch rare outliers separately (PFOR); the fastest integer codec family, billions of values per second.›Delta encodingcompression: Store differences between consecutive values instead of the values, turning sorted IDs, timestamps and counters into small numbers that pack into few bits.›Varint (LEB128)compression: Variable-length integers in 7-bit groups with a continuation bit; small numbers take 1 byte, the wire format of Protobuf, WebAssembly and DWARF.›BM25 rankingsearch-sort: The standard lexical relevance score: IDF-weighted term frequency with saturation (k1) and length normalization (b); default in Lucene, Elasticsearch and SQLite FTS5.›Sorted-list intersectionsearch-sort: Intersect sorted id lists by merging, galloping from the short list into the long one, and skipping whole compressed blocks, shortest list first.›Bootstrap confidence intervalsstatistics: Resampling-based confidence intervals for arbitrary statisticsQuantile sketchesdata-structures: Mergeable quantile sketches: t-digest, DDSketch, HDR histogramsRegularizationmachine-learning: Techniques that trade training fit for generalization: weight decay, dropout, early stopping, label smoothing and data mixing, with the exact update rules and defaults.›+ 1 moreUnicode text tokenization forsearchsearch-sort: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.›

How it is built

Architecture overview

A visual search system answers a question asked with a picture: "what is this shoe, where can I buy it, what plant is that, what does this sign say". The user points a camera at something or circles a region of the screen. The system finds the objects in the picture, turns the chosen one into a vector with an embedding network, retrieves the nearest items from an index of products, places, images or entities, ranks the candidates with more expensive models, and shows them, often together with a text answer written by a multimodal model. Text inside the image is read separately and feeds the same query. Part of this runs on the phone (capture, region choice, sometimes the embedding) and the rest in the data centre, where the index holds up to billions of items.

The design splits retrieval from ranking. Retrieval must be cheap per item and approximate: compare one query vector against a billion stored vectors in a few tens of milliseconds. Ranking can afford a heavy model, but only for a few hundred candidates. Published systems follow this shape: Bing describes a cascaded learning-to-rank framework over deep visual features, serving an index of tens of billions of images with responses in under 200 ms (Bing, 2018); Pinterest, eBay and Alibaba have published comparable systems.

Parts:

 camera / screenshot
        │ resize, crop
        ▼
 object detector / segmenter ──► region(s) ──► embedding net ──► query vector (full or short prefix)
        │                                             │
        └──► OCR (text lines) ──► query text          ▼
                    │                       scatter ─► shard 1: ANN (graph or IVF-PQ) + filter ─┐
                    ▼                              ├─► shard 2: ...                              ├─► gather: merge top-k
              entity linking, query expansion      └─► shard N: ...                              ┘
                                                              │ few hundred candidates
                                                              ▼
                                         learning-to-rank (features) ─► cross-encoder rerank ─► results / answer

There are two products of one design: the query path (latency matters, one request at a time) and the index build path (throughput matters, every new item is detected, embedded and inserted). They must use the same embedding model, or vectors are not comparable.

How the parts fit together

Capture and region choice

A photo has several things in it, and the user means one. The pipeline first finds candidate regions. Object detection turns an image into scored, classified boxes in one pass; its network is built from Convolutional networks, post-processed with Non-maximum suppression so one object yields one box, and trained with Data augmentation (crops, colour jitter, flips) so that phone photos in poor light still work. For "circle to search" the region comes from the user's gesture; to refine it into an object outline the system uses image-segmentation (no page yet), now commonly a promptable model in which a click or a stroke is the prompt (the Segment Anything model of Kirillov et al. 2023 is the reference), whose image encoder is a vision-transformer (no page yet).

Before the network sees the crop, the image is resized to the model's input size. Image resampling filters matters here: shrinking by a large factor without a proper low-pass kernel aliases fine texture and changes the embedding, and the same filter must be used at index time and query time. Using the bilinear filter in training and a different one in serving is a classic silent accuracy loss.

For a catalogue index, the same detector runs offline on each listing image, so a "red dress" photo of a model in a room is indexed by the dress crop rather than the whole scene.

Embedding model

The embedding network maps a crop to a vector, typically a few hundred dimensions, so that two photos of the same product (different light, angle, background) are close and unrelated items are far. It is trained with Contrastive learning: two views of the same item are a positive pair, other items in the batch are negatives. The loss pulls positives together and pushes negatives apart; the Embeddings produced are normalised so cosine similarity is a dot product. Hard negatives (similar-looking but different products) keep the model from learning only categories. Where labelled pairs are scarce, they come from user behaviour (clicks, saves) or from augmentations of one image, and CLIP (Radford et al. 2021) shows that image-text pairs can train an image encoder and a text encoder into one shared space, which lets a text query and an image query hit the same index.

The backbone is either a convolutional network or a vision-transformer (no page yet): patch embeddings, position encodings and the encoder blocks of Transformer block. Transformers dominate large-scale training; small convolutional nets still win on phones.

Index size and memory drive the embedding design. matryoshka-embeddings (no page yet) train the vector so that its first 64, 128, 256 ... dimensions are each a usable embedding (Kusupati et al. 2022); the first stage can search with a short prefix and the second stage rescore with the full vector, trading nothing at serving time for a smaller, faster first pass. Weight quantization shrinks the network itself, 8-bit or 4-bit weights with per-group scales, so the model fits and runs on the phone's accelerator; on-device embedding also means the image never leaves the phone if only the vector is sent.

Text reading

Much of what people photograph is text: menus, signs, packaging, documents. Optical character recognition (text recognition) finds text lines in the image and reads each crop with a CTC or attention model; Connectionist temporal classification (CTC) loss lets the recogniser train without labelling which pixel column belongs to which character, and Document image preprocessing (deskew, perspective correction, even lighting) lifts accuracy on phone photos. A lexicon and confidence scores correct and filter the output. The text becomes extra query terms and a filter ("the brand name printed on the bottle"), and translation, if offered, takes the recognised lines.

Vector index

The index stores one vector per item and answers "the k nearest to this query". Exact search compares against everything: a billion 512-dimension float32 vectors is about 2 TB (512 × 4 bytes × 10⁹) and a scan per query. Two families of approximate index cut this:

  • Graph index: HNSW keeps a layered proximity graph and finds neighbours with a greedy walk from a sparse top layer; recall is high at low latency, but the graph and the full vectors must sit in RAM, and updates are inexpensive.
  • Compressed inverted index: IVF-PQ clusters the vectors with k-means into lists, probes the few closest lists, and stores each vector as a short product-quantization code. At 64 bytes per code, a billion vectors take about 64 GB (64 × 10⁹), small enough for a few machines, at some loss of recall that a rescore with the exact vectors recovers. Faiss, whose GPU design is described by Johnson et al. (2017), is the reference implementation.

Which one to use depends on size and churn: graphs for tens of millions of vectors per shard with tight latency; IVF-PQ for the larger shards where memory dominates.

Real queries carry constraints: category, price, in stock, region, age-appropriate. Nearest-neighbour search with filters combines the neighbour search with attribute predicates, either by checking a filter bitmap while walking the graph, or by applying a bitmap over the candidate lists; compressed bitmaps keep the allow-list small. A filter that is too selective makes the graph walk wander, so the engine switches to a brute-force scan over the few matching items when the match set is small.

Perceptual hashing sits beside the vector index for a different job: compact fingerprints that stay close under resizing and recompression catch exact and near-duplicate copies cheaply, which is used to drop duplicates from results and to flag reposts of the same image before the heavy path.

Distributed retrieval

One machine cannot hold the index. The vectors are split into shards, and Scatter-gather query sends the query to all (or the relevant subset of) shards, each of which returns its local top k, and a coordinator merges them with Top-k selection. The cost is the slowest shard: end-to-end latency is the maximum over shards plus the merge, so tail latency grows with the number of shards. The remedies are the standard ones: sending a duplicate request to a replica after a short delay, per-shard timeouts with partial results marked as such, and giving the whole request a deadline that every stage respects. Sharding by category lets the coordinator skip shards when the query implies one.

Ranking

Nearest neighbours in embedding space are candidates, not answers. The candidate set (hundreds) is re-scored by Learning to rank: a model trained on query-grouped data with a ranking loss, using features that retrieval did not have, such as visual similarity at several scales, text match with the OCR output, price, availability, popularity, seller quality and the user's context. Gradient-boosted trees are the usual workhorse here because they handle mixed numeric features and are fast; quality is measured with NDCG on held-out sessions (Evaluation metrics), not with accuracy.

For the top few dozen, a Cross-encoder reranking model looks at the query crop and the candidate together instead of comparing two precomputed vectors; it is much more accurate and much more expensive, which is why it only touches the final handful. This three-step ladder (cheap vector, feature ranking, joint model) matches the cascade Bing describes.

Entity and text side

Not every answer is a product. A photo of a landmark, a plant, a logo or a recognised name must resolve to a knowledge-base entity. Entity linking maps a mention (an OCR string, a classifier label) to an entity id, or to "none", by generating candidates from an inverted index and scoring them with context; the surrounding NER step finds the mention spans. query-expansion (no page yet) rewrites a short, ambiguous text query with synonyms or generated variants when the text and the image are combined: "red" plus a picture of a bag.

For open questions ("what is this, how do I fix it") a multimodal model receives the crop, the OCR text and the top retrieved items as grounding, which keeps its answer tied to real items.

Data and control flow

One query:

  1. The phone captures the image or the screen region and resizes it; a detector or the user's gesture selects a region.
  2. The region is embedded (on device or on a server); in parallel OCR reads text.
  3. The coordinator sends the query vector and filters to the shards (scatter).
  4. Each shard runs its ANN search, applies filters, returns its top k.
  5. The coordinator merges, drops duplicates, runs the feature ranker, then the cross-encoder on the top part.
  6. Entities are linked, the answer model is called if needed, and the result list is rendered.

One item added to the index:

  1. A new listing image arrives; the detector crops objects.
  2. Each crop is embedded with the current model version and compressed to a code.
  3. The code, id and attributes go into the right shard; the graph or list is updated.
  4. When the embedding model changes, the whole index is rebuilt under a new version and switched atomically, because old and new vectors cannot be compared.

Scaling

  1. Memory bites first: vectors dominate. Levers: shorter prefix dimensions, PQ codes, 8-bit or binary quantization with rescoring, disk-resident exact vectors for the final rescore.
  2. Tail latency grows with shard count. Levers: fewer larger shards where memory allows, replicas and backup requests, deadlines, category routing.
  3. Embedding throughput at index build: billions of crops through a network. Levers: batch on GPUs, smaller backbones for the long tail, embed only on change.
  4. Update rate: a marketplace changes constantly. Lever: a small mutable shard for fresh items searched beside the large immutable ones, merged in the background.
  5. Ranking cost grows with candidates times model size. Levers: strict budgets per stage, distilling the cross-encoder into the feature ranker.

Failure modes

FailureSymptomCauseFix
Wrong objectresults for the background, not the itemdetector picks the largest region, not the intended oneshow the box and let the user adjust, use the gesture as a prompt
Domain gapworks on catalogue photos, fails on phone snapstraining images are clean studio shotsaugment with phone-like noise and crops, train on click pairs
Version mismatchnonsense neighbours after a model updatequery embedded with a new model against an old indexversion tag on vectors, rebuild and switch atomically
Resize mismatchsilent recall dropdifferent resampling filter at index and queryone preprocessing spec, golden-file test
Low recall under filtersnothing returned for a narrow filtergraph walk blocked by the filterpre-filter to a scan when selectivity is high
Duplicate-heavy resultsten copies of one imageno duplicate suppressionperceptual hash grouping before ranking
Slow shardp99 latency spikesone overloaded replicahedged requests, partial results, rebalance
OCR misreadbrand typed wrong, wrong entityblur, glare, unusual fontsconfidence threshold, spell correction against a lexicon, show candidates
Rank driftclicks fall after a retrainranker trained on logs biased by old rankingrandomised exploration slice, counterfactual evaluation
Sensitive contentpeople or private documents retrievedno policy on faces and documentsfilters at index time and query time, on-device processing where possible

Real implementations

namewhat it isrepolicencenotes
Google Lens, Circle to Searchcamera and screen-region searchproprietaryproprietarythe reference product
Apple Visual Look Upon-device recognition in Photos and Cameraproprietaryproprietarypartly on device
Pinterest Lens, Bing Visual Searchpublished production systemsproprietaryproprietarysee Sources
Faissvector similarity search: flat, IVF, PQ, HNSW, GPUfacebookresearch/faissMITindex layer
hnswlibheader-only C++ HNSW with Python bindingsnmslib/hnswlibApache-2.0graph index
Milvusdistributed vector database with filteringmilvus-io/milvusApache-2.0sharding and filters
OpenCLIPopen image-text embedding models and training codemlfoundations/open_clipsee repoembedding model

Sources

  • Web-Scale Responsive Visual Search at Bing, 2018: tens of billions of images, under 200 ms, cascaded learning to rank. paper
  • Y. Jing et al., Visual Search at Pinterest, 2015. paper
  • Visual Search at eBay, 2017. paper
  • Visual Search at Alibaba, 2021. paper
  • J. Johnson, M. Douze, H. Jégou, Billion-scale similarity search with GPUs, 2017. paper
  • Y. Malkov, D. Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs, 2016. paper
  • A. Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP), 2021. paper
  • A. Kirillov et al., Segment Anything, 2023. paper
  • A. Kusupati et al., Matryoshka Representation Learning, 2022. paper

Similar systems

Share the most building blocks with this one.

embedding-service 23 %rag 18 %recommender 15 %