Skip to content

ADR: Backends, side by side

ADR: Backends, side by side

Status
accepted
Date
2026-09-08

One card at the top of the Admin tab, on both sites, with this container and its peer on the same rows: the store, the cold start, the seed and what it cost, and how long the things a visitor does take on each, with the request charge beside every number the document store can put one on. Parent: ADR: A second store on Cosmos DB, and what it costs. The peer read is retired on 19 October: both sites are one process, and the card says so in a sentence (ADR: One web app for every .NET process).

In plain words

One card on the Admin tab puts the two backends side by side (the SQL database and the document database, Azure Cosmos DB) and shows how long each thing a visitor does takes on each. Each site reads its partner's numbers through its own API, so either tab shows both.

What that is worth: a developer can compare the two stores row by row on live numbers, and the organization can show the comparison on its own site instead of in a pasted document.

Context

Steve wanted the comparison built into the product rather than pasted into a document, and he wanted the same Admin section on both sites. One component, one image, deployed twice: whichever tab he opens, he sees both backends.

Decision

The peer is read through this container's API, not from the browser. /api/admin/peer fetches the peer's /api/admin/metrics server side and returns it. That keeps CORS closed on both origins and keeps the peer's address out of the client bundle: one configuration value per container, Peer:Url, holds the peer's public origin, which is a URL and not a credential, and the browser only ever learns the peer's host name.

/// <summary>
/// The other container, read through this one's API rather than from the
/// browser (ADR: Backends, side by side). Two reasons. Keeping the read server
/// side keeps CORS closed on both origins, and it keeps the peer's address out
/// of the client bundle: one configuration value per container holds the
/// peer's public FQDN, which is a URL and not a credential.
///
/// <para>The peer is a thing that can be down. No peer configured, no answer,
/// a slow answer and a wrong answer are four different sentences on the card,
/// and none of them is an exception: the rest of the Admin tab renders whatever
/// this returns. Two and a half seconds, not a hang, because the card refreshes
/// every thirty and a peer that takes longer than that is a peer that is
/// down for the purposes of a comparison.</para>
/// </summary>
public sealed class PeerReader(string? configuredUrl, Outbound client, bool bothStoresHere = false)
{
    /// <summary>
    /// What the card says when this process serves both sites (ADR: One web app for every .NET
    /// process): the other site is this process, and the comparison card already sets its two
    /// stores side by side. The retirement is an addendum on ADR: Backends, side by side.
    /// </summary>
    public const string OneProcess =
        "no peer to read: since 19 October both sites are this one process, so the comparison card above sets the two stores side by side";

    public static readonly TimeSpan Patience = TimeSpan.FromSeconds(2.5);

    /// <summary>The peer's origin, or null when none is configured or the deploy left its placeholder behind.</summary>
    public Uri? Url { get; } =
        string.IsNullOrWhiteSpace(configuredUrl) || configuredUrl.StartsWith("__", StringComparison.Ordinal)
            ? null
            : Uri.TryCreate(configuredUrl.TrimEnd('/'), UriKind.Absolute, out var url) ? url : null;

    /// <param name="cancellation">Cancelled when the poll that asked has gone; the peer is then no longer waited on.</param>
    public async Task<PeerView> ReadAsync(CancellationToken cancellation)
    {
        if (Url is null)
        {
            return new PeerView(false, false, bothStoresHere ? OneProcess : "no peer is configured on this container", null, DateTimeOffset.UtcNow, null, null);
        }

        string host = Url.Host;
        try
        {
            // The caller's cancellation and this read's own patience, so whichever comes
            // first stops the ask: a poll the Admin tab abandoned no longer holds a connection
            // open for two and a half seconds nobody is waiting out.
            using var timeout = CancellationTokenSource.CreateLinkedTokenSource(cancellation);
            timeout.CancelAfter(Patience);
            using var response = await client.GetAsync(new Uri(Url, "/api/admin/metrics"), timeout.Token);
            if (!response.IsSuccessStatusCode)
            {
                return new PeerView(true, false, $"the peer answered {(int)response.StatusCode}", host, DateTimeOffset.UtcNow, null, null);
            }
            var metrics = await response.Content.ReadFromJsonAsync<JsonElement>(cancellationToken: timeout.Token);
            string? version = await VersionAsync(timeout.Token);
            return new PeerView(true, true, null, host, DateTimeOffset.UtcNow, metrics, version);
        }
        catch (OperationCanceledException)
        {
            return new PeerView(true, false, $"the peer did not answer within {Patience.TotalSeconds:0.#} seconds", host, DateTimeOffset.UtcNow, null, null);
        }
        catch (Exception ex) when (ex is HttpRequestException or JsonException or InvalidOperationException)
        {
            // The type, never the message: a message here carries a host name
            // and a port, and this is a public page.
            return new PeerView(true, false, $"the peer could not be read ({ex.GetType().Name})", host, DateTimeOffset.UtcNow, null, null);
        }
    }

    /// <summary>
    /// The peer's build, from its public /api/version, inside the same patience as its metrics.
    /// The metrics answer carries no version, and two sites on different builds explain most
    /// differences between them, so the card asks. A version it cannot read is null, never a
    /// failure: the comparison stands without it.
    /// </summary>
    private async Task<string?> VersionAsync(CancellationToken token)
    {
        try
        {
            using var response = await client.GetAsync(new Uri(Url!, "/api/version"), token);
            if (!response.IsSuccessStatusCode)
            {
                return null;
            }
            var body = await response.Content.ReadFromJsonAsync<JsonElement>(cancellationToken: token);
            return body.TryGetProperty("version", out var version) && version.ValueKind == JsonValueKind.String
                ? version.GetString()
                : null;
        }
        catch (Exception ex) when (ex is HttpRequestException or JsonException or InvalidOperationException or OperationCanceledException)
        {
            return null;
        }
    }
}

/// <summary>What the comparison card learns about the peer: whether there is one, whether it answered, and if so its metrics as it reported them.</summary>
/// <param name="Configured">True when this container has a peer to compare with.</param>
/// <param name="Reachable">True when the peer answered with its metrics.</param>
/// <param name="Reason">Why the peer could not be read, or null when it was.</param>
/// <param name="Host">The peer's host name, or null when none is configured.</param>
/// <param name="FetchedAt">When the read was attempted, UTC.</param>
/// <param name="Metrics">The peer's metrics as it reported them, or null when it could not be read.</param>
/// <param name="Version">The peer's build from its /api/version, or null when it could not be read.</param>
public sealed record PeerView(bool Configured, bool Reachable, string? Reason, string? Host, DateTimeOffset FetchedAt, JsonElement? Metrics, string? Version)
{
    /// <summary>
    /// What the comparison card titles the peer's column, said here so the card prints it rather
    /// than composing it (ADR: Backends, side by side, the addendum on the peer's sentence).
    /// </summary>
    public string Title => !Configured ? "No peer is configured on this container"
        : !Reachable ? "The other site is not answering"
        : $"The other site ({StoreOf(Metrics) ?? "store unknown"})";

    private static string? StoreOf(JsonElement? metrics) =>
        metrics is { ValueKind: JsonValueKind.Object } m && m.TryGetProperty("store", out var store) && store.ValueKind == JsonValueKind.Object
            && store.TryGetProperty("store", out var name) && name.ValueKind == JsonValueKind.String
            ? name.GetString()
            : null;
}

Live from api/TheYard.Api/Shared/Instance/Peer.cs, region peer, read from this build at 8003063.

The peer is a thing that can be down. ADR: The relational store established that the site comes up when the database does not, and the same rule holds here: no peer configured, no answer, a slow answer and a wrong answer are four different sentences on the card, and none of them is an exception. Two and a half seconds of patience, not a hang, because the card refreshes every thirty seconds and a peer that takes longer than that is down for the purposes of a comparison. The rest of the Admin tab renders whatever the peer says.

The tests were written in the order the record asks for, the failures first:

[Fact]
public async Task With_no_peer_configured_the_endpoint_says_so_and_the_rest_of_admin_is_untouched()
{
    await using var api = Api(null);
    var client = api.CreateClient();

    var peer = await client.GetFromJsonAsync<JsonElement>("/api/admin/peer");
    Assert.False(peer.GetProperty("configured").GetBoolean());
    Assert.False(peer.GetProperty("reachable").GetBoolean());
    Assert.Contains("no peer", peer.GetProperty("reason").GetString());

    var health = await client.GetAsync("/api/health");
    Assert.Equal(HttpStatusCode.OK, health.StatusCode);
}

[Fact]
public async Task A_placeholder_the_deploy_left_behind_is_no_peer()
{
    await using var api = Api("__PEER_URL__");
    var peer = await api.CreateClient().GetFromJsonAsync<JsonElement>("/api/admin/peer");
    Assert.False(peer.GetProperty("configured").GetBoolean());
}

[Fact]
public async Task An_unreachable_peer_is_a_sentence_within_the_patience_not_a_hang()
{
    // Port 9 is discard; nothing listens there and the connection is refused.
    await using var api = Api("http://127.0.0.1:9");
    var client = api.CreateClient();

    var clock = Stopwatch.StartNew();
    var peer = await client.GetFromJsonAsync<JsonElement>("/api/admin/peer");
    clock.Stop();

    Assert.True(peer.GetProperty("configured").GetBoolean());
    Assert.False(peer.GetProperty("reachable").GetBoolean());
    Assert.False(string.IsNullOrWhiteSpace(peer.GetProperty("reason").GetString()));
    // A refused connection answers in milliseconds on an idle machine. The
    // margin is for the suite, not the code: with two stores booting in
    // every application at once, a timer's callback and a refused socket
    // both wait their turn on the thread pool, and this took 4.6 seconds
    // once (ADR: One container, both stores). The claim is "not a hang",
    // which ten seconds still holds against a peer that never answers.
    Assert.True(clock.Elapsed < PeerReader.Patience + TimeSpan.FromSeconds(8), $"took {clock.Elapsed}");
    // The type of the failure, never its message: a message names a host and a port.
    Assert.DoesNotContain("127.0.0.1:9", peer.GetProperty("reason").GetString());
}

[Fact]
public async Task A_peer_that_answers_slowly_is_reported_as_slow_and_nothing_waits_for_it()
{
    // Twenty seconds, against a patience of two and a half. The assertion
    // below is generous on purpose: what it proves is that nothing waits
    // for the peer's full answer, and a tight number would measure the
    // machine, which under the whole suite in parallel once took four
    // seconds to fire a timer (ADR: The SQL Server backend, the addendum on
    // the CI failure that could not explain itself).
    using var slow = new CannedPeer(delay: TimeSpan.FromSeconds(20));
    await using var api = Api(slow.Origin);
    // The client first, so the clock measures the peer read and not the
    // host coming up: creating the client is what boots Program.cs, and
    // that migrates, seeds and indexes a hundred thousand vehicles.
    var client = api.CreateClient();

    var clock = Stopwatch.StartNew();
    var peer = await client.GetFromJsonAsync<JsonElement>("/api/admin/peer");
    clock.Stop();

    Assert.False(peer.GetProperty("reachable").GetBoolean());
    Assert.Contains("did not answer", peer.GetProperty("reason").GetString());
    Assert.True(clock.Elapsed < TimeSpan.FromSeconds(15), $"took {clock.Elapsed}");
}

Live from api/TheYard.Tests/PeerTests.cs, region peer-down, read from this build at 8003063.

and the happy path last, against a canned peer on the loopback:

[Fact]
public async Task A_peer_that_answers_is_relayed_whole_with_its_host_and_nothing_else_about_its_address()
{
    using var canned = new CannedPeer(delay: TimeSpan.Zero);
    await using var api = Api(canned.Origin);

    var peer = await api.CreateClient().GetFromJsonAsync<JsonElement>("/api/admin/peer");

    Assert.True(peer.GetProperty("reachable").GetBoolean());
    Assert.Equal("127.0.0.1", peer.GetProperty("host").GetString());
    Assert.Equal("Azure Cosmos DB", peer.GetProperty("metrics").GetProperty("store").GetProperty("store").GetString());
    // Two reads: the metrics, then the build, so the card can name both sites' versions.
    Assert.Equal("1.0.3.999", peer.GetProperty("version").GetString());
    Assert.Equal(2, canned.Hits);
}

Live from api/TheYard.Tests/PeerTests.cs, region peer-up, read from this build at 8003063.

The rows are routes, not paths. A bid on one vehicle and a bid on another are the same operation, and a card that listed a hundred thousand paths would say nothing. Timing is grouped by route template, with the identifiers taken out, and the store log's operations are grouped the same way so the document column can say what a route costs in request units as well as milliseconds:

/// <summary>
/// Timing by route rather than by path, for the rows the comparison card puts
/// side by side: a bid on one vehicle and a bid on another are the same
/// operation, and a card that listed a hundred thousand paths would say
/// nothing (ADR: Backends, side by side).
/// </summary>
public static partial class Routes
{
    [GeneratedRegex("^/api/vehicles/[^/]+")]
    private static partial Regex VehicleId();

    [GeneratedRegex("^/api/docs/diagrams/[^/]+")]
    private static partial Regex DiagramName();

    [GeneratedRegex("^/api/docs/[^/]+")]
    private static partial Regex DocSlug();

    [GeneratedRegex("^/api/images/.+")]
    private static partial Regex ImageFile();

    public static string TemplateOf(string path)
    {
        string template = VehicleId().Replace(path, "/api/vehicles/{id}");
        template = DiagramName().Replace(template, "/api/docs/diagrams/{name}");
        if (!template.StartsWith("/api/docs/diagrams/", StringComparison.Ordinal))
        {
            template = DocSlug().Replace(template, "/api/docs/{slug}");
        }
        return ImageFile().Replace(template, "/api/images/{file}");
    }

    public static string Of(string method, string path) => $"{method} {TemplateOf(path)}";

    /// <summary>Per-route timings, busiest first.</summary>
    public static IReadOnlyList<RouteTiming> ByRoute(IReadOnlyList<RequestEntry> requests) =>
        requests
            .GroupBy(entry => Of(entry.Method, entry.Path), StringComparer.Ordinal)
            .Select(group =>
            {
                long[] durations = group.Select(entry => entry.DurationMs).ToArray();
                return new RouteTiming(group.Key, durations.Length, Percentiles.Of(durations, 50), Percentiles.Of(durations, 95), durations.Max());
            })
            .OrderByDescending(timing => timing.Count)
            .ThenBy(timing => timing.Route, StringComparer.Ordinal)
            .ToArray();

    /// <summary>
    /// What each route costs the document store, from the operations the store
    /// log attributes to it: how many operations a request runs and the request
    /// units they add up to, as the median over the requests seen. Requests are
    /// told apart by their trace identifier, so twenty sign-ins are twenty
    /// samples and not one sum. Grouping by the path string instead would
    /// report twenty sign-ins as one request costing the sum of all twenty.
    /// </summary>
    public static IReadOnlyList<RouteCharge> ChargesByRoute(IReadOnlyList<StoreOperation> operations) =>
        operations
            .Where(operation => operation.Request is not null)
            .GroupBy(operation => (operation.RequestId ?? operation.Request!, operation.Request!))
            .Select(perRequest =>
            {
                var parts = perRequest.Key.Item2.Split(' ', 2);
                string route = parts.Length == 2 ? Of(parts[0], parts[1]) : perRequest.Key.Item2;
                return (Route: route, Operations: perRequest.Count(), Charge: perRequest.Sum(operation => operation.RequestCharge), CrossPartition: perRequest.Count(o => o.Partition.StartsWith("cross", StringComparison.Ordinal)));
            })
            .GroupBy(sample => sample.Route, StringComparer.Ordinal)
            .Select(group =>
            {
                double[] charges = group.Select(sample => sample.Charge).OrderBy(charge => charge).ToArray();
                return new RouteCharge(
                    group.Key,
                    group.Count(),
                    Math.Round(group.Select(sample => (double)sample.Operations).Average(), 1),
                    Math.Round(charges[(int)Math.Ceiling(charges.Length * 0.5) - 1], 2),
                    Math.Round(charges[^1], 2),
                    group.Sum(sample => sample.CrossPartition));
            })
            .OrderByDescending(charge => charge.Requests)
            .ThenBy(charge => charge.Route, StringComparer.Ordinal)
            .ToArray();
}

/// <summary>One route's request timings for the comparison card.</summary>
/// <param name="Route">The method and route template, such as GET /api/vehicles/{id}.</param>
/// <param name="Count">How many requests to the route are in the window.</param>
/// <param name="P50Ms">The median duration, in milliseconds.</param>
/// <param name="P95Ms">The 95th percentile duration, in milliseconds.</param>
/// <param name="MaxMs">The slowest duration, in milliseconds.</param>
public sealed record RouteTiming(string Route, int Count, long P50Ms, long P95Ms, long MaxMs);

/// <summary>What one route costs the document store per request, for the comparison card.</summary>
/// <param name="Route">The method and route template, such as GET /api/vehicles/{id}.</param>
/// <param name="Requests">How many requests to the route were seen.</param>
/// <param name="OperationsPerRequest">The average number of store operations per request, to one decimal place.</param>
/// <param name="RuP50">The median request units per request.</param>
/// <param name="RuMax">The most request units any one request spent.</param>
/// <param name="CrossPartition">How many of the route's operations crossed partitions, across all its requests.</param>
public sealed record RouteCharge(string Route, int Requests, double OperationsPerRequest, double RuP50, double RuMax, int CrossPartition);

Live from api/TheYard.Api/Shared/Instance/Peer.cs, region routes, read from this build at 8003063.

The request-unit figure per route is exact per request: every operation the store log records carries the trace identifier ASP.NET Core gives the request that caused it, so twenty sign-ins are twenty samples of 2 RU and not one sum of 40. The first version of this card grouped by method and path instead and reported twenty sign-ins as a single 34 RU request, which the first measurement session caught (ADR: Measuring both stores).

Cold start is measured on each container, at its own start. How long the store took to answer, how long the catalogue and the bids took to load, and when the container was ready to serve, all counted from the process's own start time and exposed in the metrics, so the two columns compare two real cold starts rather than one stopwatch and one memory:

/// <summary>
/// How long each part of coming up took, and when the container was ready to
/// serve, measured from the process's own start. The comparison card shows
/// these for both containers on the same rows (ADR: Backends, side by side).
/// </summary>
public sealed class StartupTimings
{
    /// <summary>When this process started, UTC, which every figure here is measured from.</summary>
    private readonly DateTime _processStart = System.Diagnostics.Process.GetCurrentProcess().StartTime.ToUniversalTime();

    /// <summary>How long each named step took, in milliseconds.</summary>
    private readonly Dictionary<string, long> _steps = new(StringComparer.Ordinal);

    /// <summary>Milliseconds from process start to the point the host finished warming, or null until then.</summary>
    public long? ReadyMs { get; private set; }

    /// <summary>
    /// Runs one step of coming up, records how long it took under <paramref name="step"/>, and
    /// passes its result on. The step is handed the caller's cancellation rather than closing over
    /// one, so a step that forgets to stop is a signature this timer refuses and not a token a
    /// lambda happened not to capture.
    /// </summary>
    public async Task<T> Time<T>(string step, Func<CancellationToken, Task<T>> work, CancellationToken cancellation)
    {
        var clock = System.Diagnostics.Stopwatch.StartNew();
        var result = await work(cancellation);
        _steps[step] = clock.ElapsedMilliseconds;
        return result;
    }

    /// <summary>
    /// Runs one step of coming up that returns nothing and records how long it took under <paramref
    /// name="step"/>, handing the step the caller's cancellation.
    /// </summary>
    public async Task Time(string step, Func<CancellationToken, Task> work, CancellationToken cancellation)
    {
        var clock = System.Diagnostics.Stopwatch.StartNew();
        await work(cancellation);
        _steps[step] = clock.ElapsedMilliseconds;
    }

    /// <summary>Marks the host warm: <see cref="ReadyMs"/> becomes the time since the process started.</summary>
    public void Ready() => ReadyMs = (long)(DateTime.UtcNow - _processStart).TotalMilliseconds;

    /// <summary>How long a step took in milliseconds, or null when it has not run.</summary>
    public long? Ms(string step) => _steps.TryGetValue(step, out long ms) ? ms : null;
}

Live from api/TheYard.Api/Shared/Observability/AdminObservability.cs, region startup-timings, read from this build at 8003063.

The card. Two columns, this container first, the peer second, on rows for the store, the cold start, the store check, the seed with its charge, the catalogue and bid loads, the requests window, the seven routes a visitor actually takes, and the store operations window:

/** The rows the comparison card puts side by side: the paths a visitor actually takes (ADR: Backends, side by side). */
const COMPARED_ROUTES: { route: string; label: string }[] = [
  { route: 'GET /api/vehicles', label: 'Listing page' },
  { route: 'GET /api/vehicles/{id}', label: 'Vehicle page' },
  { route: 'GET /api/facets', label: 'Filter values' },
  { route: 'POST /api/vehicles/{id}/bids', label: 'Bid write' },
  { route: 'POST /api/auth/login', label: 'Sign in' },
  { route: 'POST /api/auth/register', label: 'Register' },
  { route: 'POST /api/market/tick', label: 'Room tick' },
];

/**
 * One column of the comparison: a store, wherever it runs, on the rows the card
 * draws. A timing is null when the store's window is empty, which the column
 * says in words rather than as a zero.
 */
type StoreColumn = {
  title: string;
  store: string;
  startup: Startup;
  requests: {
    window: number;
    p50_ms: number | null;
    p95_ms: number | null;
    by_route: RouteTiming[];
  };
  charges: RouteCharge[];
  storeOps: string;
};

function storeOpsLine(store: StoreSummary | null, sql: SqlSummary | null): string {
  if (store !== null) {
    return `${percentileWords(store.p50_ms, store.p95_ms, 'no operations yet')}, ${store.ru_total} RU over ${store.window}, ${store.cross_partition} cross-partition`;
  }
  if (sql !== null) {
    return `${percentileWords(sql.p50_ms, sql.p95_ms, 'no statements yet')} over ${sql.window} statements`;
  }
  return '';
}

/** A whole container's metrics as one column, which is what a peer answers with. */
function columnOf(title: string, m: Metrics): StoreColumn {
  const cosmos = m.store.store === 'Azure Cosmos DB';
  return {
    title,
    store: m.store.store,
    startup: m.startup,
    requests: m.requests,
    charges: cosmos ? m.store_by_route : [],
    storeOps: storeOpsLine(cosmos ? m.store : null, cosmos ? null : m.sql),
  };
}

/** One store of a container that runs more than one (ADR: One container, both stores). */
function columnOfBackend(backend: BackendMetrics, current: boolean): StoreColumn {
  return {
    title: current ? `${backend.store}, serving this visit` : backend.store,
    store: backend.store,
    startup: backend.startup,
    requests: backend.requests,
    charges: backend.store_by_route,
    storeOps: storeOpsLine(backend.store_metrics, backend.sql),
  };
}

/**
 * The columns the card compares. A container running both stores compares
 * them with each other, in one process, the current one first; a container
 * running one compares itself with its peer, whatever /api/admin/peer relayed,
 * and when it relayed nothing the column says why and the rest of the card
 * stands (ADR: Backends, side by side).
 */
function columns(
  mine: Metrics,
  peer: Peer | null
): { columns: StoreColumn[]; note: string | null } {
  const backends = mine.backends ?? [];
  if (backends.length > 1) {
    const current = backends.find((b) => b.store === mine.store.store) ?? backends[0];
    const ordered = [current, ...backends.filter((b) => b !== current)];
    return {
      columns: ordered.map((b) => columnOfBackend(b, b === current)),
      note: 'Both stores run in this container, so the two columns share a process, a region and a request ring; only the store differs.',
    };
  }
  const theirs = peer !== null && peer.reachable ? peer.metrics : null;
  // The column's title is the API's sentence (ADR: Backends, side by side); only a read that
  // failed outright is said here, because then there is no answer to print.
  const peerTitle = peer === null ? 'The peer could not be read' : peer.title;
  const peerNote = peer !== null && peer.configured && !peer.reachable ? peer.reason : null;
  const empty: StoreColumn = {
    title: peerTitle,
    store: '',
    startup: {
      ...mine.startup,
      ready_ms: null,
      schema_ms: 0,
      seed_ms: 0,
      seed_ru: null,
      catalogue_ms: null,
      bids_ms: null,
    },
    requests: { window: 0, p50_ms: null, p95_ms: null, by_route: [] },
    charges: [],
    storeOps: '',
  };
  return {
    columns: [
      columnOf(`This site (${mine.store.store})`, mine),
      theirs === null ? empty : columnOf(peerTitle, theirs),
    ],
    note: peerNote,
  };
}

/** A line of the comparison: what is measured, and its reading in each store's column. */
type ComparedRow = { label: string; cells: (column: StoreColumn) => string };

function Comparison({ mine, peer }: { mine: Metrics; peer: Peer | null }) {
  const { columns: cols, note } = columns(mine, peer);
  const blank = (column: StoreColumn) => column.store === '';

  const routeCell = (column: StoreColumn, route: string): string => {
    if (blank(column)) return '';
    const timing = column.requests.by_route.find((r) => r.route === route);
    if (timing === undefined) return 'not seen yet';
    const charge = column.charges.find((r) => r.route === route);
    const time = `${percentileWords(timing.p50_ms, timing.p95_ms)} (${timing.count})`;
    return charge === undefined ? time : `${time} · ${charge.ru_p50} RU`;
  };

  const rows: ComparedRow[] = [
    { label: 'Store', cells: (c) => c.store },
    {
      label: 'Cold start, process start to ready',
      cells: (c) => (blank(c) ? '' : ms(c.startup.ready_ms)),
    },
    {
      label: 'Store check (schema or containers)',
      cells: (c) => (blank(c) ? '' : ms(c.startup.schema_ms)),
    },
    {
      label: 'Seed, first boot only',
      cells: (c) => (blank(c) ? '' : msAndRu(c.startup.seed_ms, c.startup.seed_ru)),
    },
    { label: 'Catalogue load', cells: (c) => (blank(c) ? '' : ms(c.startup.catalogue_ms)) },
    { label: 'Bids load', cells: (c) => (blank(c) ? '' : ms(c.startup.bids_ms)) },
    {
      label: 'Requests, all paths',
      cells: (c) =>
        blank(c)
          ? ''
          : `${percentileWords(c.requests.p50_ms, c.requests.p95_ms)} (${c.requests.window})`,
    },
    ...COMPARED_ROUTES.map((entry) => ({
      label: entry.label,
      cells: (c: StoreColumn) => routeCell(c, entry.route),
    })),
    { label: 'Store operations, this window', cells: (c) => c.storeOps },
  ];

  return (
    <DataTable
      label="Backends side by side"
      rows={rows}
      rowKey={(row) => row.label}
      columns={[
        { name: 'Measured', cell: (row) => row.label },
        ...cols.map((column): Column<ComparedRow> => ({
          name: column.title,
          mono: true,
          cell: (row) => row.cells(column),
        })),
      ]}
      note={note === null ? undefined : { content: note, testId: 'peer-note' }}
    />
  );
}
/** This site's build beside the other site's, the first thing to rule out when two columns differ. */
function versionsLine(own: string | null, peer: Peer | null): string {
  const mine = own === null ? 'this site: version not read' : `This site runs v${own}`;
  if (peer === null || !peer.configured) return `${mine}; no other site to compare.`;
  if (!peer.reachable || peer.version === undefined || peer.version === null) {
    return `${mine}; the other site's version could not be read.`;
  }
  return `${mine}; the other site runs v${peer.version}${own === peer.version ? ', the same build' : ''}.`;
}

/** The process under both columns: when it started, and how much it compiled for itself. */
function runtimeLine(m: Metrics): string {
  const started = `This process started ${clockLabel(m.startup.started_at)}`;
  const prepared = m.startup.prepare_ms === null ? '' : `, prepared in ${ms(m.startup.prepare_ms)}`;
  if (m.runtime === undefined) return `${started}${prepared}.`;
  const r = m.runtime;
  return `${started}${prepared}; the runtime compiled ${r.jit_methods.toLocaleString('en-US')} methods itself in ${ms(r.jit_ms)}, ${r.gc} garbage collection${r.gc_concurrent ? ', concurrent' : ''}, ${r.processors} processor${r.processors === 1 ? '' : 's'}.`;
}

Live from src/components/admin/BackendsCard/BackendsCard.tsx, region comparison, read from this build at 8003063.

What it looks like

Both cards on 1.0.0.92, taken a minute apart after six paired rounds of the measurement script had given every row on both sides something to show. The live site first, reading its peer:

Backends, side by side, on the Azure SQL site: this site's column first, the Cosmos DB peer second, with the request charge beside the peer's sign-in, register and bid rows

And the same card on the Cosmos DB container, where the columns swap and the request charge sits in the first column:

Backends, side by side, on the Cosmos DB site: this site's column first with its request charges, the Azure SQL peer second

Read across any row and the two containers agree with each other about the other one, which is the property the peer endpoint exists to give: each site reads the other's metrics rather than remembering them. The bid write row is the one to look at twice. Server side the document store spends 11 ms on a bid and the relational one 86, and the measurement record shows what is left of that difference by the time it reaches a visitor in Missouri (ADR: Measuring both stores).

Consequences

  • Both sites carry the same card, and opening either shows both backends.
  • The comparison is live, not a screenshot: every number is the last few hundred requests each container has seen, and the cold start is the one it actually had.
  • /api/admin/peer and /api/admin/store are excluded from the timing rings like the other observability reads, or the card would fill with itself.
  • A peer address is one more environment variable on each container group, set in the deploy, never in the repository.
  • The one thing this card cannot do is compare on equal traffic. The live SQL site has visitors and the Cosmos one has whoever is comparing; the measurement record runs the same requests against both in one session for that reason (ADR: Measuring both stores).

Addendum, 2026-09-08: the two columns moved into one process

The evening's ask was a toggle at the top of the page, and the answer to it (ADR: One container, both stores) changed what this card compares. A container that runs both stores now puts them on these same rows with each other, the one serving the visit first, and says in a note that the two columns share a process, a region and a request ring: only the store differs. The request ring records which store served each request, so one ring is split two ways rather than two rings being read from two places.

Nothing above is withdrawn. The peer endpoint, its patience and its four sentences are what a container running one store still uses, and the second container still runs the same image with the other default, so the two-tab comparison this record was written for still works. What the in-process comparison adds is the one thing the two-container one could not have: a measurement with no network in it.

Addendum, 2026-09-09: parity, every number for both stores

The morning after both stores went into every container, Steve asked for the Admin tab to show every figure for Cosmos DB that it shows for SQL Server. The Admin tab was inventoried card by card on both sites before anything changed, from the page's source and from each site's own answers to the endpoints behind it, and each number marked as belonging to one store, to both, or to the container:

  • Backends, side by side: both, by construction; the columns are the backends, and the store-operations row already put the relational window and the document window on the same row.
  • Same performance, proven: both, by construction.
  • The partition key, live: the document store's alone, and it stays so; the ask is that the relational store never has a statistic the document store lacks, not the other way round.
  • Application health: both, one check per store with its duration.
  • Azure's view of the container, Traffic, last hour, Recent errors: the container's, not a store's.
  • Timing: the relational store's alone. The card said "SQL: p50, p95, slowest" from the SQL ring, and on the site whose default store is the document one that line was about the other store while the store serving the visit got nothing. The window sentence named the request ring and the SQL ring and not the store ring. This was the gap.
  • The SQL this application ran and What the document store ran: both, on both sites. The document store's card was already drawn whenever the container had a document store, whichever store served the visit; its rule compared the log's label to a name, and the SQL card's rule read the backends list, so the two rules were the same rule written two ways.
  • The page's opening paragraph described the SQL statements and the log and did not mention the operations at all.

What changed. The Timing card has a document store line under the SQL line, with the same three numbers and the two only that store can put a figure on: the request units the window cost, and how many of its operations fanned out across partitions rather than reading one. Its window sentence names all three rings. On a container with no document store, a developer's machine or CI, the line says so in words rather than going missing, which is the standard the comparison card already held. The words are arithmetic in src/lib/metrics.ts, read from the backend that has a document store and not from the top-level block, whose label is the store serving the visit while its numbers are always the document ring's; a peer on an older build with no backends list falls back on the label. The document store card's rule reads the same backends list the SQL card's does, with the label as the fallback. The opening paragraph names both logs and states the standard.

/**
 * The document store's window on this container, or null when it runs none.
 * The per-backend list is the authority: the top-level `store` block is
 * labelled with the store serving the visit, which on a container running
 * both is the relational one half the time, while its numbers are always
 * the document store ring's. A peer on an older build has no list, and for
 * it the label is all there is.
 */
export function documentStore(metrics: TimingMetrics): StoreWindow | null {
  if (metrics.backends !== undefined) {
    const backend = metrics.backends.find((candidate) => candidate.store_metrics !== null);
    return backend?.store_metrics ?? null;
  }
  return metrics.store.store === 'Azure Cosmos DB' ? metrics.store : null;
}

/** The sentence that says what the card's numbers are measured over: every ring the container keeps. */
export function timingWindow(metrics: TimingMetrics): string {
  const store = documentStore(metrics);
  const rings = [
    `${metrics.requests.window} requests`,
    `${metrics.sql.window} SQL statements`,
    store === null
      ? 'no document store on this container'
      : `${store.window} document store operations`,
  ];
  return `Measured in this process, over the last ${rings[0]}, ${rings[1]} and ${rings[2]}, with the endpoints this page reads left out so it does not fill with the act of being read.`;
}

/** The relational store's line: the same three numbers it has always shown, or that it has none yet. */
export function sqlLine(metrics: TimingMetrics): string {
  const sql = metrics.sql;
  if (sql.p50_ms === null || sql.p95_ms === null) {
    return 'SQL: no statements in the ring yet, so there is nothing to time.';
  }
  return `SQL: ${percentileWords(sql.p50_ms, sql.p95_ms)}, slowest ${timingWords(sql.max_ms, 'none')}.`;
}

/**
 * The document store's twin of the SQL line, plus the two numbers only it
 * can put a figure on: what the window cost in request units, and how many
 * of its operations fanned out across partitions instead of reading one.
 */
export function documentStoreLine(metrics: TimingMetrics): string {
  const store = documentStore(metrics);
  if (store === null) {
    return 'Document store: none on this container, so there is no operation to time.';
  }
  const fanned =
    store.window === 0
      ? 'no operations yet'
      : `${store.cross_partition} of ${store.window} operations fanned out across partitions`;
  const timing =
    store.p50_ms === null || store.p95_ms === null
      ? 'nothing to time'
      : `${percentileWords(store.p50_ms, store.p95_ms)}, slowest ${timingWords(store.max_ms, 'none')}`;
  return `Document store: ${timing}, ${store.ru_total} RU over the window, ${fanned}.`;
}

Live from src/lib/metrics.ts, region document-store-window, read from this build at 8003063.

What holds it: src/lib/metrics.test.ts for the words, including the container with no document store and the quiet window; AdminObservabilityTests for the shape the card draws from, every store a container runs answering a window of its own with the fields the lines need, both kinds at once on a two-store container; and one assertion in tests/e2e/admin.spec.ts on the two lines, which the gate runs on SQLite alone and again with both stores.

Before and after, from both sites. The Timing card on 1.0.0.100, at 08:15 CDT on 2026-09-09, on the live site and on the second site: one store's line, and on the second site that line is about the store not serving the visit:

The Timing card on the live site before: the window sentence names requests and statements, and the lines are Requests, SQL and Answers

The Timing card on the second site before: the same three lines, the SQL line about the other store

The same card on 1.0.0.101, at 08:47 CDT, both sites: the window sentence names all three rings, and the document store has its line under the SQL line with the request units and the fan-out count beside the percentiles:

The Timing card on the live site after: requests, SQL statements and document store operations in the window sentence, and a Document store line with p50, p95, slowest, RU over the window and operations fanned out across partitions

The Timing card on the second site after: the same two store lines

Addendum, 2026-10-19: the peer read retired

Steve, 07:47 to 07:49 CDT: "since I've demoed the shed, let merge all .net into one new app service, and delete the old", and "on the existing ADRs indicate the separate app services would be for increased load and separation down the road but we merged due to low traffic on the update and ram".

The separate web apps were the design for increased load and separation down the road: one process per site and per product, so one could roll, fail or be scaled without the others. They were merged on 19 October because traffic is low and the memory on the B2 plan is better spent (read live at 13:46 UTC that morning: the plan at 72 per cent, 2.52 of 3.5 GB; the four apps 977 MB together, the SQL site 404 MB, the Cosmos DB site 370, the rendering service 105 and The Shed 98; and the plan outside the apps 1,603 MB). The split is one bicep change and one edge rule away when load returns: a second entry in infra/appservice.bicep's sites list with its own names, and that domain's rule in netlify.toml pointed at it.

The peer card read the other web app's metrics through this one's API. On 19 October the other site became this same process, so there is no other container to read, and a read of itself would show the same numbers twice under two names, which is not a comparison. The template no longer sets Peer__Url, and with none set the card says in a sentence that both sites are this one process and that the comparison card sets the two stores side by side, which it has done since the addendum of 8 September. The reader, its patience and its four failure sentences stay in the code for the day a site gets a web app of its own again.

Where it sits

The comparison is a host and front end job: Peer.cs in the Api reads the other site's /api/admin/metrics server side and groups timings by route template, AdminObservability.cs records each container's own cold start, and BackendsCard with src/lib/metrics.ts draws the two columns in the browser. Single responsibility, one reason to change per class, shows in Peer.cs owning the peer and nothing else, including the four ways a peer can fail to answer, so the rest of the Admin tab never has to know a second site exists. It cost a Peer:Url setting on each site, a two and a half second wait before a slow peer counts as down, and a comparison that is never on equal traffic because the two sites have different visitors. Since both stores now run in one process (the 2026-09-08 addendum), a team starting fresh could rely on the in-process columns alone and keep the peer read only for sites that run one store.

Files

Addendum, 2026-10-20 (1.0.4.0): the peer's sentence lives on the API

The comparison card no longer composes the peer column's title. GET /api/admin/peer answers it as title, the sentence the card prints ("No peer is configured on this container", "The other site is not answering", or the other site and its store), and the card says only what it must when the read itself failed. The card's own description no longer describes the retired peer compare: one web app runs both stores, so the two columns share a process. PeerTests holds the sentences.

Store

The Yard, home

Welcome to The Yard

Steven Stout’s showcase: a used-vehicle auction on .NET and React, built and run by one engineer, live on Azure, with its decisions written down.

Answering in

unread

0255075100

server time at p95, the usual five minutes of the last day.

Azure this month

unread

the bill, by resource, read from Azure. The Admin tab shows every line.

Since the last roll

unread

 

rolled blue and green, with no downtime.

People who have visited

unread

30 days agotoday

Scanners, my own tools' and my own reads left out.

  •    
  •    
  •    

Tech stack

Read from the build, never typed by hand

What TheYard runs on

What we use, where it runs, the version as built, and the file that proves it.

  • .NET 1014 projects
  • React 19.3TypeScript 7.0
  • Azure6 Bicep files
  • 9GitHub Actions workflows
LayerWhat we useWhere it runsVersionProof
Backendthe API
Backendthe API.NET and ASP.NET CoreApp Service, the API container10.0TheYard.Api.csproj
Entity Framework Core (SQL Server)App Service, talking to Azure SQL10.0.12TheYard.Infrastructure.csproj
Identity with JWT bearer authApp Service, the API container10.0.12TheYard.Api.csproj
OpenAPI with Scalar API docsApp Service, /api docs2.17.13TheYard.Api.csproj
Frontendthe page
Frontendthe pageReactYour browser19.3package.json
TypeScriptYour browser7.0package.json
ViteBuild step, in the container image8.3package.json
Server-side rendering on NodeApp Service, the renderer container24render/Dockerfile
Datatwo stores, one port
Datatwo stores, one portAzure SQL DatabaseAzure, managed SQLEF Core 10.0.12TheYard.Infrastructure.csproj
Azure Cosmos DBAzure, NoSQLSDK 3.63.2TheYard.Infrastructure.Cosmos.csproj
Azureinfrastructure as code
Azureinfrastructure as codeApp Service and Container Registry (Linux container)Azure App Service and ACRlive resourceappservice.bicep
Front Door (written, switched off)Off: the subscription refuses Front Door, so the Netlify edge stands in front todayoffmain.bicep
Application Insights with OpenTelemetryAzure Monitor1.6.0TheYard.Api.csproj
Communication Services emailAzure Communication Services1.1.0TheYard.Api.csproj
Azure Functions, Flex Consumption (clock, snapshot, tracker merger, Sheet mirror, ATS sweep, daily card)func-jobhunt-ss, Flex Consumption.NET 10 isolatedJobHunt.Functions.csproj
Bicep for every resource it deploysDeployed from the repository6 Bicep filesinfra/
Edge
EdgeNetlify edge in front of the siteNetlify edge networklive resourcenetlify.toml
Quality gatesevery push
Quality gatesevery pushxUnitGitHub Actions runners2.9.3TheYard.Tests.csproj
NetArchTest architecture rulesGitHub Actions runners1.4.5TheYard.Tests.csproj
VitestGitHub Actions runners5.0package.json
Playwright with axe accessibility checksGitHub Actions runners, against the built site1.63package.json
oxlint and PrettierGitHub Actions runners1.86 and 3.9package.json
Delivery and AIhow it ships
Delivery and AIhow it shipsGitHub Actions (azure-read, blog-post, ci, deploy-function, deploy-render, deploy, edge-branch, gate, nightly)GitHub9 workflowsworkflows/
Built with Claude Code under the test gateDeveloper machine, then the gateevery pushgate.yml
Claude Code on GitHub Actions runs the job hunt's nightly lanesGitHub Actions runners, on the Azure share4 AM and every 4 hnightly.yml

Generated, not typed. A build step reads the project files, package.json, the Dockerfiles, the Bicep and the workflows; a test fails CI when this table drifts from the repository.

How it is built

How it is run

Who and why