infra-inventory

How it is built

Architecture overview

An infrastructure inventory is the database that says what exists and how it is connected: which sites hold which racks, which devices sit in them, which cables join which ports, which IP prefixes are assigned to which networks, which virtual machines run on which clusters, and who owns each of them. Other systems read it to decide what to do: a provisioning tool asks which switch ports are free, an automation job asks which hosts belong to a service, a monitoring system asks which devices it should be watching, an auditor asks who changed an address assignment last month. NetBox calls this role a "source of truth" for network automation, and positions itself as IP address management (IPAM) combined with datacenter infrastructure management (DCIM) behind APIs and extension points (source: NetBox documentation).

The inventory is a small database by data volume and a demanding one by meaning. The hard parts are:

• a typed data model that matches how infrastructure really works (a prefix contains prefixes, a device has interfaces, a cable connects two interfaces);
• address space management that never hands out the same address twice;
• trust: the inventory is only useful if it matches reality, so it needs a way to compare what is recorded with what is found, and a clear history of who changed what;
• integration: other tools must be able to read, change and be told about changes.

Parts:

• Data model and validation — typed objects and relationships, with schema checks on every write — JSON Schema validation, Coverage-guided fuzzing
• Address space management — prefix hierarchy, free-space search, allocation without duplicates — ip-address-management (no page yet), built on longest-prefix-match (no page yet), radix-trie (no page yet), Roaring bitmaps, Leases, conditional-writes (no page yet)
• Discovery and reconciliation — finds what is actually on the network and compares it with the record — network-discovery (no page yet), declarative-diff-plan (no page yet), built on Topological sort and deterministic-serialization (no page yet)
• Relationship queries — follows cables and containment to answer "what is connected to this" and "what breaks if this fails" — BFS and DFS
• Search — finds any object by name, tag, address or free text — Inverted index, built on Delta encoding, Bit packing and PFOR, Varint (LEB128), Sorted-list intersection, BM25 ranking
• Concurrent edits — keeps two simultaneous writers from overwriting each other — Optimistic concurrency control, Deterministic simulation testing
• Change history — an attributable, append-only record of every create, update and delete — Tamper-evident audit log, built on Segmented log and Digital signatures (Ed25519)
• Access control — who may read or change which objects — Authorization models, Mutual TLS
• Change notification — tells other systems when an object changes — Webhook delivery, built on Idempotency keys, Dead-letter queue, Circuit breaker and Durable job queue, Graceful draining
• Operations — keeps the service running, fast and recoverable — hdr-histogram (no page yet), Process supervision, Primary-backup replication, Point-in-time recovery

   people (UI)         automation (API, scripts)        discovery agents
        │                      │                              │
        └────────────┬─────────┴──────────────┬───────────────┘
                     ▼ authz                   ▼ proposed state
          ┌──────────────────────┐     ┌───────────────────────┐
          │ API: validate (schema)│◄───│ reconcile: diff found │
          │ version check on write│     │ vs recorded → plan    │
          └──────────┬───────────┘     └───────────────────────┘
                     ▼
   ┌────────────────────────────────────────────┐   ┌──────────────┐
   │ relational store: sites, racks, devices,   │──►│ search index │
   │ interfaces, cables, prefixes, addresses    │   └──────────────┘
   └──────┬─────────────────────┬───────────────┘
          ▼ same transaction    ▼ after commit
   change log (before/after)   job queue ─► webhooks / event rules ─► consumers

How the parts fit together

Data model and validation

The model is relational. NetBox lists the object types it curates: hierarchical regions, sites and locations; racks, devices and device components; cables and wireless links; power; circuits and providers; virtual machines and clusters; IP prefixes, ranges and addresses; VRFs and route targets; VLANs; tenancy; contacts (source: NetBox documentation). Its stack is a Django application on PostgreSQL with Redis, served by Gunicorn or uWSGI (source: NetBox installation docs). A general-purpose configuration database can store anything; a network inventory gets its value from a model that already knows what an interface or a VRF is.

Every write goes through validation. JSON Schema validation is the block for the payload checks at the API edge (types, required fields, enums, formats) and for the schemas of user-defined custom fields. NetBox offers custom fields and custom model validation as extension points (source: NetBox documentation). The rule worth stating: validation lives in the server, not in the client. Forms, scripts and imports all arrive at the same checks; otherwise the inventory is only as clean as the least careful tool that writes to it.

Everything that parses outside input is a target for Coverage-guided fuzzing: the JSON and CSV importers, the custom-field schemas and the address and prefix parsers. The fuzzer feeds mutated payloads to the validation entry point and keeps the ones that reach new code paths, so a malformed prefix or an oversized nested document is found in testing and not by a bulk import in production.

Address space management

IPAM is the part where mistakes have a direct cost: a duplicate address is an outage. NetBox's model is a hierarchy: an aggregate is the root of an addressing hierarchy (typically a large public or private block allocated to the organisation); prefixes nest inside it; an IP range is a run of addresses inside a prefix; an IP address sits automatically beneath its parent prefix. Prefixes carry a role and a status, and VRFs separate overlapping private address spaces. Utilisation is computed automatically: for a container prefix it is the space consumed by child prefixes, for other prefixes the space consumed by child addresses and ranges (source: NetBox IPAM documentation).

Four techniques make this work:

• Finding the parent. To place an address under the right prefix, take the most specific prefix that contains it: this is longest-prefix-match (no page yet), the same lookup a router does. A radix-trie (no page yet) over the prefix bits answers it in a number of steps bounded by the address length, instead of scanning all prefixes.
• Finding free space. "Give me a free /28 inside this /24" is a search over the gaps between allocated children. Representing used addresses as a compressed bitmap (Roaring bitmaps) turns "find the first run of N free addresses" into fast set operations, even for large IPv4 blocks.
• Reserving without duplicates. Two people (or two automation jobs) asking for the next free address at the same moment must get different answers. The allocation is a conditional write (conditional-writes, no page yet): insert the address only if it is still free, and retry on conflict. For a multi-step allocation (find, hold, confirm) a Lease puts a time limit on the hold so a crashed requester does not block the address forever.
• Hierarchy maintenance. When a new prefix is inserted between an existing parent and its children, the children must move under it. This is a tree operation done in the same transaction as the insert.

Discovery and reconciliation

An inventory that people maintain by hand drifts from reality within weeks. network-discovery (no page yet) closes the loop: agents scan address ranges, read neighbour tables and interface lists from devices (SNMP, LLDP, APIs) and report what they find. The inventory then answers "what is on the network that I do not know about, and what do I record that no longer answers?".

The comparison between recorded and found state is a plan, in the same sense as an infrastructure-as-code tool: declarative-diff-plan (no page yet) takes two sets of objects, computes what must be created, changed and deleted to make one match the other, and orders the actions by their dependencies (Topological sort: a prefix before its addresses, a device before its interfaces, interfaces before the cable that joins them). To compare reliably, both sides are brought to a canonical form (deterministic-serialization, no page yet) so equal objects compare equal byte for byte. The plan is shown to a human or applied under a rule, never applied silently by default: discovery can be wrong (a device briefly unreachable is not a decommissioned device).

Infrastructure-as-code tools such as OpenTofu, which manage infrastructure declaratively, follow the same shape: desired state in, a computed set of changes out. The inventory is the intent; discovery reports the fact; the plan is the difference.

Relationship queries

Infrastructure is a graph: cables join interfaces, interfaces belong to devices, devices sit in racks, racks in locations, circuits connect sites. Questions are graph traversals: "trace this cable path to the far end", "which devices are downstream of this switch", "which services lose redundancy if this circuit goes down". BFS and DFS are the traversals that answer them: breadth-first for "everything within N hops" and shortest paths, depth-first for "follow this path to its end" and cycle checks. Keep traversals bounded (maximum depth, visited set) because a cabling loop in the data must not hang the request.

Search

Operators search by half-remembered names ("the firewall in the Frankfurt rack with 'edge' in its name"), by tag, address or serial number. An Inverted index maps each term to the sorted list of objects containing it. Queries with several terms intersect the lists (Sorted-list intersection); lists are stored compactly with Delta encoding, Bit packing and PFOR and Varint (LEB128); results are ordered by relevance with BM25 ranking when text matters. For this size of data a single node's index is enough. The key rule: the index is derived data. It is updated after the transaction commits and can always be rebuilt from the relational store.

Concurrent edits

Several people and tools edit at once. Locking a record while someone fills a form is a bad fit (the form stays open for minutes). Optimistic concurrency control lets each writer read a version, edit freely, and write back only if the version is unchanged; if someone else saved in between, the write is rejected with a conflict and the editor reviews the difference. API clients use the same mechanism: read, change, write with the version; on conflict, re-read and retry.

The version check and the conditional address insert are exactly the code that only fails under unlucky interleavings. Deterministic simulation testing runs several simulated writers, the transaction layer and the clock in one thread under a seeded schedule, asks for the same free address from many clients and checks that no address is ever handed out twice; any failure replays from its seed.

Change history

The inventory is evidence in incident reviews: when did this address move, who deleted that cable, what did the record look like before? NetBox saves a serialised copy of each object before and after every create, update and delete, with the time and the user, in the same database; each request gets a unique id so that all changes from one bulk edit can be grouped; a user can attach a short message (up to 200 characters) to give the reason or a ticket reference; and records are exposed through a read-only API endpoint (source: NetBox change-logging documentation).

For a strong audit story the log can be hardened with the Tamper-evident audit log block: entries chained by hashes and periodically signed (Digital signatures (Ed25519)), stored in a Segmented log that is never rewritten, so a deleted or edited record is detectable even by someone with database access. A plain change table is editable by anyone who can edit the database.

Access control

Different people should see and change different things: the network team edits devices, the application team edits their VMs, a vendor may only read circuits. Authorization models cover the choices: role-based (role → permissions), attribute-based (rules over object attributes such as site or tenant) or relationship-based. In an inventory, the useful scope is usually a constraint on object attributes ("may edit devices in tenant X"), applied on every query and every write, including bulk operations and the search index. NetBox's REST API uses token authentication that maps API clients to user accounts and their permissions (source: NetBox API documentation).

Discovery agents and automation jobs are machines, not people. Mutual TLS gives each of them a verified certificate identity on the connection to the API, which authorization then maps to the narrow permissions that agent needs (for example, write discovered interfaces but never delete devices).

Change notification

Other systems need to react: a monitoring system should start watching a device that became active; a DNS tool should update records when an address changes. NetBox's event rules can send a webhook, or run a custom script, on the create, update or delete of an object type, with optional JSON conditions on the object, and its documentation gives exactly the example of configuring a monitoring system when a device's status changes to active (source: NetBox event-rules documentation). Webhook delivery is the block behind it: the event is written to a queue in the same transaction as the change (Durable job queue), a worker sends it with an Idempotency key so receivers can discard repeats, failed deliveries are retried with backoff, and messages that keep failing go to a Dead-letter queue. A Circuit breaker per destination stops the system from hammering a receiver that is down.

Webhook workers hold deliveries in flight. On a deploy or restart, Graceful draining makes a worker stop taking new jobs from the queue, finish or release the deliveries it has started before a deadline, and exit, so a restart does not leave half-sent events or duplicate bursts.

Operations

Four blocks keep the inventory trustworthy while it runs. Process supervision starts the API workers, the queue workers that send webhooks, the search indexer and the discovery receiver, restarts any that exit, and stops them with a polite signal before a forced kill. Primary-backup replication keeps a standby copy of the relational store: the primary orders all writes and ships them on, so a lost host does not lose the record of what exists; the same standby can serve the heavy list queries that automation tools issue. Point-in-time recovery covers the other kind of disaster, a bad bulk import or a wrong discovery plan that deleted devices: base backups plus the archived change stream let the operator rebuild the database as it was just before the mistake, and the before/after change log then shows what to re-apply. hdr-histogram (no page yet) records the latency of the API calls that matter most (next-free-address allocation, list queries, graph traces, webhook delivery lag) as percentiles, so a slowdown shows up in the tail before users complain.

Data and control flow

A change through the API:

1. A client sends a request with a token. Authentication maps it to a user; authorization checks the permission for this object type and these attributes.
2. The payload is validated against the schema and the model rules (required fields, uniqueness, address inside its prefix).
3. The write is conditional on the object's version. A conflict returns an error with the current state.
4. In one database transaction: the object is written, the before/after copy goes to the change log with the request id, and an event is queued for matching event rules.
5. After commit: the search index is updated and the queued events become webhook deliveries or script runs.

Reserving an address:

1. The client asks for "next free address in prefix P".
2. The server finds the first free address from the bitmap of used addresses and tries to insert it with a conditional write; on conflict it picks the next free one.
3. The result returns with the new object; optionally a short lease marks it as reserved until the owner confirms.

Discovery run:

1. A scanner reports found devices, interfaces and addresses.
2. Reconciliation canonicalises both sides, computes the diff, orders it by dependencies and stores it as a proposed plan with the evidence.
3. A person (or a rule for safe classes of change) approves; approved actions are applied through the normal API, so validation, history and webhooks all apply as for any other change.

Scaling

The inventory is a modest OLTP workload, and the first scaling concern is usually correctness and trust rather than throughput.

1. Reads dominate: automation tools poll or list thousands of objects. Levers: paginated list endpoints, filters pushed into queries, read replicas of the relational store, caching of immutable lookups. Prefer event-driven consumers (webhooks) over polling.
2. Writes are bursty: imports of whole sites, discovery sweeps, bulk edits. Levers: bulk endpoints in a single transaction, batch size limits, a queue between discovery and the API.
3. Graph queries are expensive on large cabling graphs. Levers: depth limits, caching of computed paths (invalidated by the change log), precomputed adjacency for hot views.
4. Search needs re-indexing after bulk changes; do it from the change log, in the background.
5. Webhooks can fall behind during a bulk import. Levers: per-destination queues and concurrency, a dead-letter queue, circuit breakers.
6. Multi-region and multi-team use usually means one authoritative inventory per domain with APIs between them, not multi-master replication of one inventory; address-space conflicts are far worse than a stale read.

Failure modes

Real implementations

Choosing: a CMDB aimed at IT services (software, business services, ITSM workflows) differs from a network source of truth. NetBox's documentation says it deliberately uses a curated data model for network engineers rather than the general-purpose model of a CMDB (source: NetBox documentation). Pick that style when the questions are about addresses, ports, cables and racks; pick a broader CMDB when the main need is to relate business services, owners and change tickets across all kinds of assets.

Sources

• NetBox documentation, introduction: IPAM and DCIM, object types, customisation (custom fields, custom validation, export templates, event rules, plugins), REST and GraphQL APIs, Apache 2 licence.
• NetBox documentation: IPAM (aggregates, prefixes, ranges, addresses, utilisation, VRFs), change logging, event rules, API integration, installation.
• Repository pages for NetBox and OpenTofu (licence and description as shown there).