retail-merchandising

How it is built

Architecture overview

A merchandising system decides, for every item and store, what is sold, where it sits, how much arrives and what it costs, and it makes those decisions months before the shopper arrives. The decisions are chained: a forecast of demand drives the assortment (which items each group of stores carries), the assortment drives space (how many shelf facings each item gets), space and forecast drive allocation (units sent to each store from a limited stock) and, once the season turns, markdown (how to clear what did not sell). Each step is an optimisation over a different set of numbers, but all read the same item and price master, and all write plans that planners edit by hand. The one idea that makes it work: the forecast is a distribution at every level of a hierarchy (item, store, cluster, chain), kept consistent across levels, and each optimiser consumes that distribution, not a single number. Oracle Retail, Blue Yonder and RELEX are the commercial suites; open pieces exist for the forecasting, optimisation and data-movement layers, not for the whole.

Parts:

• Item and price master — one record per product and price across all source systems, with history — Entity resolution, Change data capture, Bitemporal data (no page yet)
• Demand forecast — per item, store and week, with uncertainty, consistent across the hierarchy — Time-series forecasting, Gradient-boosted decision trees, Deep forecasting models (no page yet), Quantile regression (no page yet), Hierarchical reconciliation (no page yet), Linear and logistic regression
• Clusters and assortment — which stores are alike, which items each cluster carries — k-means clustering, choice-model-estimation (no page yet)
• Space and planogram — shelf facings per item, fixtures laid out — Rectangle bin packing (no page yet), Simulated annealing (no page yet)
• Allocation and replenishment — units to each store from limited stock — Simplex method, Min-cost flow (no page yet), Inventory policies and safety stock (no page yet), Largest-remainder rounding (no page yet)
• Markdown optimisation — price cuts over time to clear stock — markdown-price-optimization (no page yet)
• Planning workspace — planners edit plans at any level and see totals — olap-cube-write-back (no page yet), Columnar storage
• Batch orchestration — the nightly and weekly chain of jobs — DAG workflow execution

 ERP, POS, supplier feeds, e-commerce ─► entity resolution ─► item & price master (bitemporal)
        │ change stream (CDC)                                         │
        ▼                                                             ▼
 sales history ─► features ─► forecast models ─► quantiles ─► reconcile up/down the hierarchy
 (columnar)       price, promo,  (GBDT, deep, ts)                         │
                  weather, events                                         ▼
         clusters (k-means) ─► assortment ─► space (bin packing + SA) ─► planogram
                                    │                                     │ facings, capacity
                                    ▼                                     ▼
                         allocation (LP / flow, rounding)  ◄──────────  stock on hand, safety stock
                                    │
                    markdown optimiser ◄── choice / elasticity model ──┘
                                    │
          planning workspace (cube, write-back) ◄──── DAG scheduler runs all of it

How the parts fit together

Item and price master

The master answers "what is this item, what does it cost and sell for, and what was true last quarter". It is assembled from sources that disagree: supplier catalogues, the ERP, point-of-sale systems, an online shop. Entity resolution decides that three supplier rows and two shop SKUs are one product (block on brand and size, score pairs, cluster the matches). The master then feeds every other system, so changes leave it as a stream through Change data capture rather than as nightly full copies, which keeps forecasts, planograms and the shop in step without every writer calling every reader.

Merchandising needs two clocks on every fact. A price change entered today for next Monday has a valid time (Monday) and a record time (today); a correction made later changes what the system knew, not what was true. Bitemporal data (no page yet) stores both, so a forecast can be re-run "as it would have looked then" and a markdown plan can be audited. Contract with the rest: every consumer asks for a fact "as of valid date X, as known on date Y", never "the current row".

Demand forecast

Demand is forecast per item, store and week, usually with price, promotion, holidays, weather and local events as inputs. Most items sell a few units a week, so the series are sparse and noisy; a few sell thousands. Time-series forecasting supplies the baselines and the backtesting discipline that every model must beat. For the bulk of items a Gradient-boosted decision tree model on lagged sales, price and calendar features is the strong default: one global model across all series learns from the many short ones. The M5 retail forecasting competition (Walmart sales data) is the public evidence for this pattern; its organisers report that the top entries were dominated by LightGBM-type models. Deep forecasting models (no page yet), such as DeepAR or the Temporal Fusion Transformer, are the alternative when there are many related series and rich covariates and the team can afford training and serving them.

A point forecast is the wrong object for inventory and markdown. The cost of too little stock (a lost sale) and too much (a cut price) are different, so the optimiser needs a chosen quantile, or the whole distribution. Quantile regression (no page yet) trains a model for a given quantile directly by the pinball loss, which GBDT libraries support. Promotion lift and price effects are often fitted separately with Linear and logistic regression, because planners want an interpretable coefficient ("a 10 % cut lifts this category by x %") that they can argue with.

Then the numbers must add up. Item forecasts summed to category will not equal a separately made category forecast, and a planner will edit the category total. Hierarchical reconciliation (no page yet) adjusts the forecasts at all levels so that they are coherent; the standard references are Hyndman and Athanasopoulos (the chapter on hierarchical forecasting) and Wickramasuriya et al. on trace-minimising reconciliation. Contract with downstream: forecast quantiles per (item, store, week), coherent across levels, with the model version and the backtest error recorded.

Clusters and assortment

Stores differ, but planning every store alone is impossible and noisy. k-means clustering groups stores by what they sell (sales mix, size, demographics of the catchment) into a few dozen clusters; plans are made per cluster and refined per store. Scale the features and choose k by stability or business sense; planners will not use clusters they do not recognise.

Assortment planning chooses which items each cluster carries: more items raise the chance a shopper finds a match, but split demand and tie up space and stock. The core difficulty is substitution: if the item is missing, some demand moves to a similar item, some is lost. choice-model-estimation (no page yet) fits a discrete-choice model (multinomial logit and its extensions) from sales and availability data to estimate how demand redistributes, which is the model class behind the assortment-optimisation literature (Talluri and van Ryzin 2004 for the revenue-management case). Contract with space: per cluster, a ranked assortment with expected volume per item at a reference number of facings.

Space and planogram

A planogram assigns items to fixtures, shelves and positions. Two layers exist. The aggregate layer decides facings per item subject to shelf length and category space; the Hübner and Kuhn review covers the quantitative models (space-elastic demand: more facings, more sales, with diminishing returns). The layout layer packs actual product dimensions onto shelves. Rectangle bin packing (no page yet) is the geometric core: items are rectangles, shelves are bins of fixed width and variable height, and constraints add adjacency (keep one brand together), eye-level preference and presentation stock. The problem is NP-hard in general, so practical systems use heuristics refined by Simulated annealing (no page yet): start from a greedy layout, try swaps and moves, and accept some worse layouts early so as to escape local optima. Why heuristics: the objective (profit with space-elastic demand) is non-linear and merchants want answers in minutes for thousands of fixtures; exact integer programs suit the smaller aggregate layer. Contract with allocation: per store, a capacity per item (facings times units per facing).

Allocation and replenishment

Allocation takes a limited stock, say a delivery of one seasonal item, and splits it across stores and warehouses. A linear program or a flow does this: sources with stock, stores with demand (the forecast quantile) and capacity (from the planogram), edges with a transport cost. The Simplex method solves the general form, Min-cost flow (no page yet) is the faster specialised case when the structure is a pure network. Objective: maximise expected sales, or minimise shortfall plus transport.

Two things come around the optimiser. Inventory policies and safety stock (no page yet) set the target: a reorder point or order-up-to level equal to expected demand over the lead time plus a buffer sized from the forecast uncertainty and the service level wanted; this is where the forecast quantile enters. And a solver returns fractions (7.4 units to store A), while shipping is whole units. Largest-remainder rounding (no page yet) rounds the fractions so that the total shipped equals the total allocated, giving the extra units to the stores with the largest remainders. Contract: allocation is run in rounds, keeping a reserve at the warehouse so that later forecasts can redirect stock.

Markdown optimisation

As a season ends, the retailer picks a price path that sells the remaining stock by the deadline at the highest total revenue. markdown-price-optimization (no page yet) combines three inputs: remaining stock, the time left, and a price-response model (how sales respond to a cut, including how a cut now steals from sales later). The classical formulation (Smith and Achabal 1998) chooses clearance prices over time for a chain with stock and a deadline. The response model comes from the elasticity regression and the choice model above; the optimiser enumerates allowed price steps (a ladder with end-digit and margin rules) and a dynamic program or LP picks the path. Constraints are business rules: a limit on cuts per week, no price below cost, cross-store price consistency. Contract: a proposed price per item and cluster and week, with predicted sell-through, to be accepted or edited by the planner.

Planning workspace

Planners work in a grid (item by location by week by measure) and edit at any level: "category total up 5 % in January". olap-cube-write-back (no page yet) spreads such an edit down to the children in proportion to their current values (or locks), recomputes totals up the hierarchy, and stores the change as a versioned scenario. The queries underneath are wide scans of a few columns over billions of rows, which is the case for Columnar storage: per-column compression and vectorised scans. Contract: edits are scenario-scoped and never overwrite the approved plan until accepted.

Batch orchestration

The nightly chain (load, resolve, forecast, reconcile, cluster, allocate, publish) is a dependency graph with partial reruns. DAG workflow execution runs each step when its inputs are ready, retries failed tasks, and reruns only what depends on changed data. Contract: every step reads versioned inputs and writes versioned outputs, so a rerun of the forecast does not silently change an already approved allocation.

Data and control flow

One planning cycle (weekly):

1. Change streams from POS and ERP land in the columnar store; entity resolution updates links; the master gets new facts with valid and record times.
2. The DAG triggers feature building and model scoring: GBDT or deep models produce quantile forecasts per item, store and week.
3. Reconciliation makes the hierarchy coherent; accuracy against last week's actuals is logged.
4. Planners review exceptions in the workspace (items whose forecast changed most), edit totals, and the cube spreads their edits.
5. Allocation runs per delivery with the approved forecast, the planogram capacities and the stock; rounding yields whole-unit shipments sent to the warehouse and store systems.
6. Late in a season, the markdown optimiser proposes price cuts; accepted prices flow back into the master as future-dated changes.

One seasonal round (months ahead): cluster stores and fit choice models; choose assortment per cluster; lay out space (facings, then planograms by simulated annealing); forecast season volumes, set buy quantities with a safety-stock target, and agree the plan with buyers.

Scaling

In the order the limits bite:

1. Forecast breadth: millions of item-store series each week. Levers: one global model per family; forecast at cluster level and spread down for sparse items; rescore only series whose inputs changed.
2. Hierarchy consistency: reconciliation over a large hierarchy grows with the number of series. Lever: reconcile per category branch, use a structured form (top-down shares for sparse leaves).
3. Space optimisation time: thousands of fixtures with heuristics. Levers: optimise per fixture family, warm start from last season's planogram, parallelise per store group.
4. Workspace latency: cube queries over many dimensions. Levers: columnar storage, pre-aggregation of upper levels, restricting edits to a scenario.
5. Data movement and reruns: nightly windows shrink as the chain grows. Levers: change data capture instead of full loads, incremental DAG runs, and tasks that touch only affected partitions.

Failure modes

Real implementations

The alternative: spreadsheet-based planning with fixed rules: sales of last year times a growth factor, allocation by store share, markdowns on a calendar. It is transparent, needs no models and works for a small chain; it breaks when assortments change quickly, when promotions dominate sales, or when stock is scarce enough that where it goes matters.

Sources

• R. Hyndman, G. Athanasopoulos, Forecasting: Principles and Practice (3rd ed.), chapter Forecasting hierarchical or grouped time series. otexts.com
• S. Wickramasuriya, G. Athanasopoulos, R. Hyndman, Optimal Forecast Reconciliation for Hierarchical and Grouped Time Series Through Trace Minimization, Journal of the American Statistical Association, 2019. DOI 10.1080/01621459.2018.1448825
• S. Makridakis, E. Spiliotis, V. Assimakopoulos, M5 accuracy competition: Results, findings, and conclusions, International Journal of Forecasting, 2022. DOI 10.1016/j.ijforecast.2021.11.013
• D. Salinas, V. Flunkert, J. Gasthaus, DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks. arXiv:1704.04110
• B. Lim et al., Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting. arXiv:1912.09363
• J. Hübner, H. Kuhn, Retail category management: State-of-the-art review of quantitative research and software applications in assortment and shelf space management, Omega, 2012. DOI 10.1016/j.omega.2011.05.008
• K. Talluri, G. van Ryzin, Revenue Management Under a General Discrete Choice Model of Consumer Behavior, Management Science, 2004. DOI 10.1287/mnsc.1030.0147
• S. Smith, D. Achabal, Clearance Pricing and Inventory Policies for Retail Chains, Management Science, 1998. DOI 10.1287/mnsc.44.3.285
• M. Fowler, Bitemporal History.