GPU Infrastructure

What makes installed compute usable?

A GPU cluster can have enough servers and still lack usable capacity. A rack needs power at its location, a path that removes its heat, and support systems that remain available. Site totals hide those local requirements.

Higher density couples these requirements. Placing several powerful racks on one electrical branch can overload that branch while another branch remains underused. A cooling fault can remove a group of healthy GPUs from service. More hardware does not repair either mismatch by itself.

For the architecture inside one GPU and the kernels it executes, see GPU Architecture and CUDA Programming. The useful unit is therefore not only the installed GPU. Follow the electrical path, the heat path and the failure domain. Then ask how much available compute produces useful work. This note gives conceptual models for those questions; it is not an installation, chemical-treatment or electrical-protection procedure.

Density moves the constraint beyond the server

Power density is power per rack or per area. Concentrating the same total load into fewer racks reduces the compute footprint, but increases the load that each local connection must support. The building must still reject the heat and supply the power.

A rough density comparison is 10–40 kW per air-cooled rack versus 100–150+ kW per liquid-cooled rack. Treat those as context, not universal boundaries. Air cooling has no single density threshold independent of server design, room conditions and heat-removal architecture.

A scoped modern example is more useful than a supposed industry maximum. NVIDIA’s DGX GB rack guide reports approximately 120 kW for its described rack and a 72-GPU NVLink domain. Its GB200 SuperPOD reference describes an 8-rack scalable unit with 1.2 MW thermal design power. Those values have different accounting boundaries; do not infer that eight rack values alone exhaust the unit’s design load. DGX GB rack hardware ↗; GB200 SuperPOD architecture ↗.

The change is not simply “liquid is better.” Liquid moves heat efficiently near hot components, but adds fluid distribution, maintenance and isolation requirements. A deployment exchanges one set of constraints for another.

Three maps of the same cluster

A fast communication group need not coincide with one electrical or cooling failure domain. A training job spanning several domains can lose useful work when any required group becomes unavailable. Conversely, placing every worker together can improve one communication path while concentrating operational risk.

Record shared ancestors, not only rack-level labels. Two apparently separate branches can depend on the same upstream panel or pump. The toy below models independent branches beneath one site cap; it deliberately omits such deeper structure.

Distributed Training and Parallelism owns data, tensor, pipeline and expert placement. Infrastructure supplies the physical constraints against which those layouts run. Do not turn physical proximity alone into a guarantee of collective performance.

Follow the heat, not just the water

In direct-to-chip cooling, a cold plate transfers heat from a component into coolant. The technical cooling loop circulates that coolant through the IT equipment. A coolant distribution unit (CDU) manages the loop and can transfer heat to a separate facility loop through a heat exchanger.

In the liquid-to-liquid design shown here, heat crosses the exchanger while the two fluids remain separate. The facility loop then carries heat to the site’s heat-rejection equipment. Moving heat between loops does not remove the need to reject it outside the compute system.

This is one architecture, not the definition of every cooling deployment. Liquid-to-air systems and immersion systems have different boundaries. The documented DGX compute trays also retain air cooling for components outside the CPU/GPU cold plates. Removing most chip heat with liquid does not eliminate the room’s air-cooled load.

Read the scene as a progressive responsibility map. The lines indicate heat-transfer roles, not actual piping, valves or fluid routing. The assigned 75% liquid fraction is illustrative, not a vendor heat-capture specification.

Bulk temperature rise follows an energy balance

Suppose the liquid loop carries 90 kW with assigned density 1 kg/L and specific heat 4 kJ/(kg·K). At 120 L/min, mass flow is 2 kg/s and bulk temperature rise is 11.25 K. At 60 L/min, the same heat load produces 22.5 K. The load did not double; less coolant carries it each second.

The equation is a steady energy balance using constant properties and a defined measurement boundary. Temperature difference in kelvin has the same numerical size as the corresponding difference in degrees Celsius. Absolute inlet temperature remains a separate variable.

Specify which heat enters this loop. Using total rack power as Q̇ overstates liquid load when some components reject heat to air. Mixing L/min with kg/s, or joules with kilojoules, can create a large error even when the formula looks correct.

Flow is not a component-temperature guarantee

Bulk coolant temperature rise is not GPU junction temperature. A component also depends on inlet temperature, thermal interfaces, cold-plate resistance, local distribution and its changing load. The simple energy balance does not model those relationships.

Flow balancing matters because each branch needs sufficient cooling for its assigned load. In a coupled network, resistance or valve changes can redistribute flow. Excess flow in one branch can reduce another branch’s flow, but that outcome depends on the pump and network controls; it is not universal.

Clogging or contamination can change pressure drop and heat transfer. Biology, corrosion and particulates also depend on coolant composition, materials and maintenance. Use the supplier’s requirements and qualified facility procedures rather than inferring treatment from a conceptual diagram.

A large total flow is not proof that every cabinet receives its required flow. Observe the local paths and component behavior. Likewise, a small bulk ΔT is not proof that every hot component is safe.

Preserve options when the building predates the cluster

An existing facility can offer available floor space without the distribution capacity that dense racks need. Retrofitting must consider the paths to the racks, not only the site’s nominal megawatts.

Optionality means retaining useful choices: reassign capacity, isolate equipment and support maintenance without an unnecessary redesign. It does not mean every project should oversize every subsystem. Extra capacity has its own cost, space and maintenance consequences.

Raised floors and overhead distribution are design options, not a universal ranking. The appropriate layout depends on the building, access, deployment schedule and approved safety requirements. Preserve the ability to adapt without presenting one tour’s arrangement as a standard prescription.

Detect, isolate and recover are different operations

Detection does not itself isolate a fault. Isolation does not guarantee that the remaining system can carry the load. Recovery does not become safe merely because a dashboard returns to green. Each operation needs a defined owner and verified behavior.

Valves, filters, fittings and connectors belong in the reliability model because each can affect availability. Redundancy should be assessed against shared dependencies and maintenance states. Two devices are not independent if one upstream failure disables both.

For training, connect the physical failure domain to checkpoint and restart consequences. A short interruption can lose more useful work than its wall-clock duration when recovery includes replaying work. That cost belongs in run planning, not only the facility uptime chart.

The Jane Street tour gives a concrete recovery example at 12:19–12:44. Buffer tanks hold cooled fluid to bridge a chiller restart and damp temperature changes as workloads change. Stored thermal capacity buys time; it does not replace sustained heat rejection. The usable interval depends on fluid mass, permitted temperature rise, heat load and the remaining circulation path. The tour gives no general ride-through duration.

At 11:14–12:03, the speakers describe topology-aware monitoring that reads breaker information and can stop workloads. That ties an action to the constrained path rather than only a site total. It is a site-specific observation, not permission to oversubscribe an unverified installation. Jane Street tour: cooling buffers ↗.

Every electrical path has its own load constraint

A site total and a local path limit answer different questions. The total checks aggregate demand. A path constraint checks the sum of loads that actually depend on that path. Both must hold.

With two 120 kW branches and a 240 kW site cap, two 90 kW racks fit in aggregate. Putting both on A gives 180 kW on a 120 kW branch. Putting one on each branch gives 90 kW per branch. Lowering the site cap below 180 kW instead makes both placements infeasible.

The smallest branch capacity is not a bound on total usable site power. Independent parallel branches can carry more together than either carries alone. A minimum is appropriate for serial constraints on the same path, or another explicitly defined bottleneck—not an arbitrary total-site formula.

This illustration checks assigned steady loads only. It does not model electrical routing, transient demand, reserve policies, protection coordination or dynamic failover. Real ratings and safety margins come from the engineered installation.

Normal operation and failure operation are different budgets

A placement that fits during normal operation may not fit during a failure. If a surviving path must carry transferred load, its failure-state constraint differs from its normal one. Maintenance can create another relevant state.

Describe what redundancy covers: a power supply, a rack path, a pump group or an upstream facility dependency. The label alone does not establish the supported load after a component fails.

Separate reservations from actual demand. Keeping capacity available for an approved transfer can make it unavailable for another workload even while the current meter shows spare power. Oversubscription is therefore a policy with assumptions, not discovered capacity.

The scene’s green feasible result is limited to its three inequalities. It intentionally makes no claim about fault tolerance. A future scenario must declare which components fail and how assignments change before a failure budget can be computed.

Connect facility telemetry to workload decisions

Facility observability becomes useful when signals connect to decisions. Monitor electrical paths, cooling equipment and workload state together so an abnormal condition can be localized. A temperature rise without the corresponding flow and workload context can remain ambiguous.

Protection and engineered limits remain independent of the workload scheduler.

Cluster software can coordinate approved pausing, load reduction or recovery and correlate events across racks. It should not create its own protection thresholds from this note. The electrical and cooling systems need their own fail-safe behavior even if the scheduler or telemetry service fails.

A useful qualitative mnemonic combines physical margin, monitoring quality and control quality. Read that as a qualitative reminder, not an additive law with common units. Better monitoring can reveal a constraint earlier; it cannot increase a breaker rating or manufacture heat-rejection capacity.

Define the response owner, latency, stale-signal behavior and verification step. A requested action is not evidence that the physical condition changed. This distinction also applies to software-only training controls.

Count useful GPU-hours, not only installed GPUs

Installed GPUs provide a potential time budget. Availability determines how much of that budget can be used. Productive use determines how much available time performs the work you intended.

For 72 GPUs over 24 hours, the installed budget is 1728 GPU-hours. With 99% availability and 80% productive use of available time, useful capacity is 1368.576 GPU-hours. Downtime accounts for 17.28; available but unused time accounts for 342.144. The three categories sum to the installed budget.

This is an accounting illustration, not model FLOPs utilization (MFU). A job can occupy GPUs while wasting work, or run productive work below a hardware throughput ceiling. Define “productive” for the decision instead of substituting an occupancy metric.

Downtime, idle time and low computational efficiency require different investigations. More scheduling cannot recover time when equipment is unavailable. More installed capacity can increase unused hours if another dependency prevents useful work.

Opportunity cost needs a counterfactual

Operating cost and opportunity cost are different quantities. Operating cost includes the applicable hardware allocation, power, facilities, networking, cooling and operations. Opportunity cost asks what feasible alternative you displaced.

When comparing decisions, combine accounting cost with the value of the best displaced feasible alternative only under a common boundary and without counting the same loss twice. This is a chosen decision model, not a universal physical or financial identity.

A scarce cluster may have an alternative workload with substantial value. An avoidable outage or slow expansion can then matter beyond the electricity bill. That value is not universally equal to revenue, and an unused GPU-hour does not automatically imply a valuable displaced job.

Choose the counterfactual explicitly. Compare the best feasible alternative under the same constraints, not unlimited demand on an imaginary unconstrained cluster. Keep uncertainty in expected value visible. Do not add missed revenue twice under both downtime loss and opportunity cost.

Serving economics distinguishes serving costs and resource boundaries. The infrastructure reader supplies the physical availability boundary, not a universal financial valuation.

A smaller compute area can require more support infrastructure

Dense racks can compress the compute area while expanding support requirements. Heat exchangers, pumping, heat rejection, electrical distribution, monitoring, maintenance access and redundancy still occupy space and resources.

Compare the whole boundary. A smaller server footprint is not proof of lower facility cost, lower water use, or lower operational risk. Cooling architecture, climate, hardware and utilization affect those outcomes; this note has no measured sustainability comparison.

The central design question is where complexity moves. Cold plates reduce one airflow problem but introduce a fluid network. Higher rack load reduces rack count but stresses local distribution. More observability helps operate the system but adds monitoring and response dependencies.

A coherent deployment considers these interactions before buying the next batch of GPUs. The same principle appears in training-system design: removing one bottleneck can expose another.

A compact investigation checklist

Use these questions to structure an investigation, not as a commissioning checklist. Vendor instructions, facility engineering and approved safety procedures govern operation. For research planning, record the assumptions that translate a physical budget into a useful training budget.

Continue with Pretraining: Experiments and Run Design for checkpoints and run decisions, Parallelism for model placement, and Roofline Model for a different boundary: kernel compute and memory throughput.

Sources and connected topics

The historical starting point is a Jane Street training data-center tour ↗. Contextual density ranges are not specifications for a new deployment. The fresh scoped hardware examples use NVIDIA’s documentation: DGX GB rack hardware ↗ and GB200 SuperPOD architecture ↗.

All diagrams here are original conceptual illustrations, not measurements or vendor piping drawings. The models deliberately expose their units and omitted dependencies. No example certifies electrical safety, fluid compatibility, fault tolerance or a component temperature.

Disaggregated Prefill–Decode Serving explains the workload-pool boundary. Separating workload pools does not remove their physical infrastructure dependencies.