social-media-management

How it is built

Architecture overview

A social media management platform lets a team plan, publish, monitor and report on many social accounts from one place. It connects each account through that network's own API, stores posts in a calendar, publishes them at the scheduled time, collects the replies, comments and mentions into one shared inbox, listens for brand keywords across public posts, and rolls everything up into analytics. The platform owns almost none of the data; the networks do, and each one has its own authentication, media rules, quotas and webhooks. The one idea that makes it work: put each network behind an adapter with a common contract, and treat every call to a network as unreliable, rate-limited and possibly duplicated. Publishing is a durable job with an idempotency key; inbound content arrives by webhook or polling and is normalised into one event shape; and the expensive part, understanding text at volume, is a pipeline of cheap steps before any model.

Parts:

• Account connection — authorise each profile, keep tokens fresh — OAuth 2 and OpenID Connect
• Composer and media — build a post, prepare images and video per network — Image resampling filters, Video transcoding ladder, URL normalization, open-and-click-tracking (no page yet)
• Approval — optional review before a post may go out — dual-control-approval (no page yet)
• Scheduler and publisher — fire at the right time, call the network, retry safely — Durable job queue, Hierarchical timing wheel, Exponential backoff with jitter, Distributed rate limiting
• Inbound events — webhooks and polling into one inbox — Webhook delivery
• Understanding — language, topic, sentiment, entities and duplicates in inbound text — Text classification, Language identification, Named entity recognition, Near-duplicate detection with MinHash LSH
• Listening — stored keyword queries matched against the public stream — boolean-expression-indexing (no page yet), Aho-Corasick multi-pattern matching, Inverted index
• Analytics — time-series of reach and engagement, spike alerts, trends — Rollup aggregation, Time-series anomaly detection, time-decay-scoring (no page yet)
• Operations — keeps scheduled publishing correct, recoverable and tested — Primary-backup replication, Point-in-time recovery, Process supervision, Graceful draining, Mutual TLS, hdr-histogram (no page yet), Deterministic simulation testing, Coverage-guided fuzzing

 team UI ── compose / approve / schedule ──► post store ──► timer (wheel) ──► publish queue
                                                  ▲                               │ lease, rate-limit per account
                                          media pipeline                          ▼
                                  (resize, transcode, per-network spec)     network adapter ──► network API
                                                                                  ▲  │ 429 / 5xx: backoff
   network ──webhook / poll──► ingest ──► normalise ──► dedupe ──► classify ──► shared inbox
                                                           │
   public stream / search ──► query matcher (keywords) ────┴──► listening feed ──► alerts
                                                           │
                              metrics pulls ──► rollups (hour/day) ──► dashboards, reports

How the parts fit together

Account connection

Each profile (a page, a business account, a channel) is connected by OAuth 2 and OpenID Connect: authorisation code flow, scopes asked per feature (publish, read comments, read insights), tokens stored encrypted and refreshed before expiry. The platform is in the middle of a three-way relation: user, platform, network. A user may connect an account a colleague also connected, a token may be revoked at any time by the user in the network's own settings, and a network may change what a scope allows. So the token record has a state (healthy, expiring, needs reconnect), every adapter call can return "token invalid", and that outcome flips the account to "needs reconnect", pauses its scheduled posts and notifies the team. Silent failure here is the most common support ticket of the whole product.

Composer and media

The composer holds one post with per-network variants (text, media, link). Media is prepared once per target spec. Images are resized and re-encoded to the network's accepted sizes and aspect ratios (Image resampling filters gives the quality choices: shrink with an area-aware or Lanczos kernel, not nearest-neighbour). Videos are transcoded to the container, codec, length and size each network accepts, which is a small Video transcoding ladder with a different top rung per network. The contract with the publisher is a media handle (stored object, checksum, derived renditions), not a URL that can change under it. Links are cleaned with URL normalization, then optionally rewritten to a short tracked link; open-and-click-tracking (no page yet) is the redirect that counts a click and forwards, with the same caveat as in email: bots and link previews also fetch links, so a click count needs bot filtering.

Approval

Larger teams want a rule that a post by an intern needs sign-off before it goes out. This is dual-control-approval (no page yet) in a lightweight form: a post carries a state (draft, in review, approved, scheduled), an approver cannot be its author, and any edit after approval returns it to review. The scheduler only picks up posts in the scheduled state, so the rule cannot be bypassed by timing.

Scheduler and publisher

This is the heart of the product, and it is a Durable job queue: a persisted task per post and per network, leased by a worker for a visibility timeout, acknowledged on success, retried on failure. Hierarchical timing wheel is the in-memory index of "which jobs become due in the next minutes", with the database as the truth; a calendar of scheduled posts is far larger than what is due now.

Publishing is a multi-step call on most networks. Instagram's content publishing flow, for example, is documented as creating a media container first and publishing it afterwards (Meta, Instagram Platform content publishing), which means a crash between the two steps leaves a created but unpublished container. The job must therefore be written as resumable steps: store the container id after step one, and on retry continue from it instead of starting again.

Duplicates are the failure to design out. A network call can succeed on the network's side while the response is lost, and the retry then creates a second post. Where the API offers an idempotency mechanism, use it: Mastodon's status endpoint accepts an Idempotency-Key header to prevent duplicate submissions, and the key is stored for up to an hour (Mastodon docs). Where it does not, the adapter writes a "publishing" marker before the call and, on a retry after an unknown outcome, first looks the post up on the account's timeline by a token embedded in the text or metadata before posting again, or hands the case to a human.

Limits are per app, per user and per endpoint, and they differ by network and plan. The X API documents per-endpoint rate limits with a time window and the x-rate-limit response headers, and answers 429 when exceeded (X API rate limits). The adapter reads such headers into a per-account state, and Distributed rate limiting keeps the count consistent when many workers publish for the same account; a worker that is told to wait releases the job back with a delay computed by Exponential backoff with jitter. Priority matters: a post that is due now outranks a metrics pull, so give the two their own budgets.

Inbound events

Comments, replies, direct messages and mentions come in by two routes. Where the network offers webhooks, it calls the platform; the receiving side of Webhook delivery applies: verify the signature (Meta documents signed webhook payloads; see its webhooks guide), answer 2xx quickly, put the payload on a queue and process later, and be idempotent because the network will redeliver. Where it does not, or when webhooks are not reliable enough, the platform polls with a stored cursor per account, and a periodic reconciliation pull closes gaps left by lost webhooks. The contract with the next stage is one normalised event (network, account, thread id, author, text, media, timestamp, raw payload reference), so everything downstream is network-agnostic.

Understanding

A shared inbox is only useful if it is sorted. The pipeline is cheap first:

• Language identification routes a message to the right team and models and avoids classifying Portuguese with an English model.
• Text classification assigns labels: question, complaint, praise, spam, urgent. Start with rules and a linear model on a few thousand labelled messages; use a language model for the labels where wording is ambiguous.
• Named entity recognition finds products, people and places so a message can be linked to a campaign or a product line.
• Near-duplicate detection with MinHash LSH clusters copy-pasted spam and the same complaint posted by many people, so the team sees one cluster with a count instead of fifty rows.

Each step returns a label with a confidence; low-confidence items are shown unlabelled instead of mislabelled, and corrections by the team become training data.

Listening

Social listening is the reverse of search: instead of one query against many documents, there are many stored queries (every customer's brand terms and competitor names, with AND/OR/NOT, phrases and exclusions) and one stream of incoming posts. A plain loop over all queries per post is too slow at stream rates. boolean-expression-indexing (no page yet) indexes the queries so each post only evaluates those whose required terms it contains; Aho-Corasick multi-pattern matching finds all the fixed phrases of all queries in one pass over the text; and an Inverted index over stored posts answers the opposite direction, a new query run over history. The contract: a match event (query id, post id, matched spans) written to the listening feed, with spam and near-duplicates filtered before it is counted. Public stream access depends on what each network allows through its API terms, so the source of the stream is a business and legal question before it is a technical one.

Analytics

Reach, impressions, engagement and follower counts are pulled from each network on a schedule and stored as counters per post and account. Rollup aggregation folds raw samples into hourly and daily partial aggregates, so a quarter-long report reads a few hundred rows per account. Time-series anomaly detection turns the same series into alerts ("mentions three times the usual for this hour"). Ranking "what is trending" or "best posts of the week" needs a recency weight: time-decay-scoring (no page yet) discounts older engagement so last week's hit does not outrank today's. Note that network metrics are revised after the fact (late-counted views, deleted spam), so rollups should be re-pulled for a trailing window instead of treated as final.

Operations

The post store (drafts, approvals, schedule, publish state) is what the team trusts to publish at 9:00, so it is replicated with Primary-backup replication: a failover must not lose a scheduled post or forget that one has already gone out.

Point-in-time recovery serves the team that deletes a whole campaign or has a bulk reschedule go wrong: the post store is restored to the minute before, and the publish state is reconciled against what the networks really show, so nothing is published twice.

The scheduler, the publish workers, the network adapters, the ingest workers and the media workers run under Process supervision; a publisher that dies after the network accepted a post but before the record is written is restarted and finds the post by its idempotency key instead of posting again.

A deploy uses Graceful draining: a publish worker finishes the network call it has started and records the network's post id before exiting, and the timer hands its due posts over, so a release at 8:59 does not delay or duplicate the 9:00 posts.

The scheduler, the adapters that hold account tokens, the media pipeline and the inbox service call each other over Mutual TLS, since only the adapters should ever see network tokens, and only services with a certificate may ask them to publish.

A scheduled post is judged by how close to its time it goes out, so the delay between scheduled time and network acceptance, and the time from a comment on a network to the shared inbox, are recorded as an hdr-histogram (no page yet) per network with percentiles; the tail shows the network that is rate-limiting us.

The scheduler and the publish retries are tested with Deterministic simulation testing: a simulated clock and network time out, return 429 and accept a post while the reply is lost, run daylight-saving changes and crash workers mid-publish, and the check is that every scheduled post is published exactly once. Failures replay from their seed.

The webhook payload decoders, the URL normaliser and the keyword-expression parser read text from networks and users, so they run under Coverage-guided fuzzing with mutated payloads and queries, keeping every input that reaches new code.

Data and control flow

Publishing one scheduled post:

1. The composer saves the post with per-network variants; media jobs prepare renditions; the state moves draft → (in review) → scheduled.
2. A timer marks the job due; a worker leases it and checks the account's token state and rate budget. If the budget is empty, it releases the job with a delay.
3. The adapter calls the network (upload media, create container, publish), storing each intermediate id; on a 429 or 5xx the job is retried with jittered backoff.
4. On success the network's post id is stored and the first metrics pull is scheduled; on a permanent error (content rejected, token invalid) the post moves to failed with the reason shown to the user. Time goes to media upload and the network's own processing, not to the platform's code.

One inbound comment:

1. The network's webhook arrives; the signature is verified, the raw payload is stored and a 2xx is returned.
2. A worker normalises it into an event, drops duplicates by network event id, and links it to the post and thread.
3. Language, labels, entities and a duplicate cluster are attached; spam is hidden by rule.
4. The event lands in the inbox with an assignee rule (by label, by account); an alert fires if the label is urgent. Typically the whole path should be seconds, and the pull that reconciles missed webhooks runs behind it.

Scaling

In the order the limits bite:

1. Network quotas. The first wall is the provider's rate limit per app and per user, not the platform's servers. Levers: webhooks instead of polling, conditional and incremental pulls, shared per-account budgets, queueing by priority.
2. Media processing. Video transcoding dominates compute and storage. Levers: transcode once per target spec, cache renditions by checksum, skip work when the original already fits the spec.
3. Timer and job volume. A burst of posts scheduled for "9:00 Monday" hits one minute. Levers: spread executions by a few seconds of jitter inside the minute, shard the job table by account, lease in batches.
4. Text understanding at stream volume. Levers: language filter first, rules before models, small models on CPU, batch inference for the heavy ones, sample low-value streams.
5. Listening matching. Levers: index the queries, not the posts; shard the query set by term; drop common words from the required terms.
6. Analytics storage. Levers: rollups at several granularities, retention per tier, partitioned time series.

Failure modes

Real implementations

No open-source project was found for this page that covers publishing, a shared inbox, listening and analytics together; the open projects below cover the publishing and scheduling core. Commercial tools such as Hootsuite and Buffer are not listed because none of their internals was verified.

Sources

• Network API documentation read for this page: Mastodon: posting a status and Idempotency-Key, X API rate limits, Instagram Platform content publishing, Meta Graph API webhooks.
• OAuth 2.0: RFC 6749 rfc.
• URLs: RFC 3986 rfc.