Workflow#
PyPSA-USA is orchestrated by Snakemake: every
artifact — a shape file, a clustered network, a solved model — is produced by a rule,
and rules chain into a directed acyclic graph (DAG) that Snakemake resolves from
whatever target you request back to raw data. This page walks through that graph
stage by stage: what each rule does, what it produces, and why the pipeline is ordered
the way it is. For the wildcards that parameterize the file names
({interconnect}, {simpl}, {clusters}, {ll}, {opts}, {sector}), see
Wildcards.
Rule graph for a power-sector run of the default (Western Interconnection) configuration. Each node is a Snakemake rule; arrows point from producer to consumer.#
The shape of the pipeline: cluster early#
The load-bearing architectural fact about PyPSA-USA is that spatial aggregation
happens before the data-heavy build stages. The base network is reduced from
thousands of nodal buses to the {simpl} cluster resolution first, and then
renewable profiles, demand, generators, and costs are built directly at that
resolution. Building per-bus data at ~50-200 cluster buses instead of ~3,000 nodal
buses cuts the memory and runtime of the heavy rules several-fold, and it means every
downstream artifact is keyed to a stable set of cluster buses.
The practical consequences:
{simpl}(set inscenario: simpl:) is the resolution at which data is built;{clusters}is the final transmission resolution the model is solved at.Most intermediate files carry
_s{simpl}in their name; anything keyed to buses downstream ofcluster_resourcesrefers to cluster buses, not substations.The bus-to-cluster mappings for every aggregation stage are saved as busmaps (
resources/busmaps/), so precomputed substation- or node-keyed datasets can be remapped into the clustered space (this is exactly whataggregate_egsdoes for geothermal supply curves).
Stage by stage#
1. Shapes and base network#
Rule |
Script |
Key outputs |
|---|---|---|
|
|
|
|
|
|
|
|
|
build_shapes assembles the geographic backbone: interconnection boundaries, states,
counties, EIA balancing authorities, REeDS zones, and offshore wind areas.
build_base_network reads the chosen transmission dataset (the synthetic
TAMU/Breakthrough Energy nodal network, or a ReEDS zonal topology) and produces the
first PyPSA network: buses, lines, links, and transformers, each bus annotated with
its geographic memberships (state, county, balancing area, REeDS zone — see
Network Schema). build_bus_regions computes the Voronoi-style onshore
and offshore service regions around each bus that later steps use to allocate
resources and demand.
2. Aggregation and pre-clustering#
Rule |
Script |
Key outputs |
|---|---|---|
|
|
|
|
|
|
aggregate_to_substations is a pure topology reduction: nodal buses collapse onto
their substations, preserving electrical connectivity. cluster_resources then runs
kmeans (or modularity) clustering to the {simpl} resolution, honoring the
model_topology: topological_boundaries setting so clusters never straddle county,
state, or REeDS-zone lines. Its outputs — the clustered network, the
matching cluster regions, and the busmap — define the spatial frame everything below
is built in. (In configuration, this stage’s options live under
clustering: simplify_network:, a historical name kept for compatibility.)
3. Data at cluster resolution#
Rule |
Script |
Key outputs |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Renewable capacity-factor profiles come either from the pre-computed GODEEEP
dataset (default) or are computed with Atlite from ERA5 weather cutouts; either
way they are produced per cluster region (see Generators & Storage Units). Demand is
disaggregated from balancing-authority or state level to cluster buses using
population- and load-based allocation factors and attached by add_demand
(Electricity Demand). add_electricity then attaches everything to the network:
existing thermal and hydro fleets from PUDL/EIA data, renewable profiles, fuel
prices, and technology costs. Its output is a pickled network because it carries
non-netCDF metadata used downstream.
4. Final clustering and scenario preparation#
Rule |
Script |
Key outputs |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
cluster_network reduces the network to its final {clusters} resolution (kmeans or
modularity), with optional suffixes controlling how existing transmission is treated
(see Spatial Configuration). add_extra_components adds extendable storage
(battery durations, pumped hydro) and other investment options.
prepare_network applies the {ll} transmission-limit scenario and the {opts}
tokens — temporal resolution (3h, segmentation), emissions limits or prices, and
per-carrier cost/potential adjustments (Custom Constraints). Power-only runs
pass through add_sectors unchanged ({sector} = E); sector-coupled runs attach
the natural-gas, buildings, transport, and industry representations here
(Sector Coupling).
5. Solve and postprocess#
Rule |
Script |
Key outputs |
|---|---|---|
|
|
|
|
|
|
|
|
|
solve_network builds the optimization problem with Linopy,
registers PyPSA-USA’s custom constraints (policy standards, reserve margins, capacity
targets — see Custom Constraints), and hands it to the configured solver. The
solved network, per-carrier statistics, and standard figures land under results/,
organized by run name and interconnect.
Example outputs from a solved California test system: optimized capacity by carrier
(left) and a week of dispatch against load (right). Regenerate with
snakemake docs_figures.#
Data retrieval#
Upstream of everything sit the retrieve_* rules, which download and cache raw data
on first use:
Rule |
What it fetches |
|---|---|
|
Core data bundle (transmission networks, shapes, seed data) |
|
PUDL generator, plant, and fuel data (EIA 860/923 derived) |
|
CAISO fuel-price data |
|
NREL Electrification Futures Study demand scenarios |
|
EER forecasted demand profiles (when |
|
Historical grid emissions data |
|
NREL reV-based land-access exclusion layers |
|
One compressed GODEEEP capacity-factor file, from the local mirror or Zenodo record declared in |
|
NREL End-Use Load Profiles (sector runs) |
|
Sector-coupling data bundle (sector runs) |
|
Seismic-risk exclusion mask (EGS runs) |
Historical EIA demand additionally requires a free EIA API key (see Installation).
Running parts of the pipeline#
Two targets are worth knowing beyond all:
data_modelbuilds everything up to the assembled, unsolved network — the full data pipeline with no solver required.dagregenerates the rule-graph image on this page (workflow/repo_data/dag.jpg).
Any intermediate file can also be requested directly, and --until <rule> /
-R <rule> stop early or force re-execution — see Usage for worked
commands.