Skip to content

Network

The river network a simulation routes over: identity, topology, and Muskingum parameters read from a network file, plus the concurrent routing partition, the signs of the routing coefficients derived from them, the sub-reaches and subcycles that use_subreaches and use_subcycles route to make every coefficient non-negative, and the stabilized network that writes those sub-reaches as rows. A Router builds one from its Configs and reuses it, so a network table is parsed and partitioned once no matter how many simulations run over it.

Network

A wrapper around the network table, the dataframe that describes the number, topology, and DFS order of the rivers and their Muskingum parameters.

When instantiating this class, the network file is eagerly read and the connectivity is set. This may make instantiation have a small but non-negligible time cost.

size property

The number of rivers, one per row of the network table.

river_ids property

(n,) id of each river, in the order of the network table.

next_river_ids property

(n,) id of the river each river drains into, -1 at a basin outlet.

k property

(n,) Muskingum k of each river, its travel time in seconds.

x property

(n,) Muskingum x of each river, its weighting of inflow against outflow, from 0 to 0.5.

upstream_counts property

(n,) number of rivers upstream of each river, which in DFS order are the rows immediately before it.

synthetic property

(n,) True for each sub-reach stabilize added, or None on a network that was never stabilized.

parent_river_ids property

(n,) id of the original river each row was split from on a stabilized network, or None.

original_river_ids property

Id of each river that is not a synthetic sub-reach, in order: the rows of every routed discharge array.

__init__(network_file)

Parameters:

Name Type Description Default
network_file PathInput | DataFrame

the network file parquet, or the network table itself as a DataFrame, which is copied

required

from_configs(configs) classmethod

Build a Network from the network_file of a Configs object, its k and x routed exactly as given.

check_parameters()

Check that k and x are in expected, valid ranges. np.isfinite checks for NaN and inf

routing_blocks(threads=1)

The blocks of rivers each job routes, packed for threads jobs by streams.assign_blocks. They are derived once per thread count and cached, so repeated simulations over this network never rebuild them.

Each block is a contiguous range of rows read from the upstreamCount column, so nothing is reordered: the forcing, the state, and the routed discharge stay in network file order from end to end. The best blocks depend on the thread count, so they are rebuilt for each one. On one thread, the main stem is the whole region.

Parameters:

Name Type Description Default
threads int

number of jobs the sub-watershed blocks are packed into

1

Returns:

Name Type Description
tuple tuple[JobBlocks, ...]

the (block_starts, block_stops) of each job, and last the main stem's, which routes after them.

validity_window()

The inclusive range of routing time steps over which every Muskingum coefficient of each river is non-negative, (2*k*x, 2*k*(1-x)), the steps at which its routing is valid. Outside it the scheme is still stable and conserves mass, but its discharge can go negative.

invalid_mask(dt, layout=None)

Per-river masks of the two ways a river's coefficients go negative at dt, one step for every river or one per river, for the whole river or, with layout, for any of its sub-reaches at its own step, dt over its subcycles.

Returns:

Name Type Description
tuple tuple[ndarray, ndarray]

(too_long, too_short). too_long is 2*k*x > step, which makes c1 negative and is fixable by sub-reaches. too_short is 2*k*(1-x) < step, which makes c3 negative and is fixable by subcycles.

sub_reaches(dt)

The equal sub-reaches of each river at dt, one step for every river or one per river: the fewest sub-reaches, laid out as conditioning lays out equal ones, whose coefficients are all non-negative at dt, which is what use_subreaches routes for rivers too long for their step. An equal sub-reach of travel time k/N has a non-negative c1 when k/N <= dt + (1-2x)*k, and a sub-reach whose x is 0 has one at any step, so the count is ceil(k / (dt + (1-2x)*k)): 2 for every river too long at x = 0.2, and never more than ceil(1/(1-2x)). No split into fewer sub-reaches of any travel times is valid at dt.

The shortest sub-reach has a non-negative c3 when dt <= 2*k_N*(1 - x_N). A river too short for dt cannot meet it, since its sub-reaches are shorter still, and for x <= 3/8 every river too long for dt does. A river whose sub-reaches do not is kept as one reach and reported as unresolvable, and conditioning splits it into unequal sub-reaches that take subcycles of their own instead, where it uses subcycles too.

Returns:

Name Type Description
tuple tuple[IntArray, ndarray]

(sub_reaches, resolvable). sub_reaches is int64 and at least 1 for every river. resolvable is True where every coefficient of the sub-reaches is non-negative at dt.

subcycles(dt)

The subcycles of each river at dt, one step for every river or one per river: the fewest equal steps a river is routed in within each step of dt, overriding dt with the shorter routing step dt / subcycles of its own. Configs(dt_routing='auto') identifies each river's own valid step with them, dt_runoff / subcycles, and use_subcycles gives them to rivers too short for a dt_routing in seconds. The count is ceil(dt/(2*k*(1-x))): the smallest that brings the river's own step under the upper bound 2*k*(1-x). A river that is not too short takes 1.

Its own step can also fall under the lower bound 2*k*x, when the window is narrower than one whole count of subcycles; such a river keeps one step and is reported as unresolvable. For x <= 1/3 the window always holds one.

Returns:

Name Type Description
tuple tuple[IntArray, ndarray]

(subcycles, resolvable). subcycles is int64 and at least 1 for every river. resolvable is True where every coefficient of the river is non-negative at its own step.

conditioning(dt, *, use_subreaches=True, use_subcycles=True)

How each river is routed at dt, one step for every river or one per river, as the use_subreaches and use_subcycles of a Configs route it: a river too long for dt is split into its equal sub_reaches with use_subreaches, and a river too short for it is routed in its subcycles, equal steps of its own, with use_subcycles. Without them a river is routed whole at dt. With both, a river neither makes valid, which only a river with x > 1/3 can be, is split into the fewest sub-reaches of unequal travel times that each take subcycles of their own and keep the river's diffusion, those of splitting.fewest_sub_reaches, and is kept whole if no split into at most the 256 sub-reaches the kernels route one river as is valid. stabilize writes this layout as rows.

Returns:

Name Type Description
Layout Layout

the sub-reaches of each river, and the travel time, x, and subcycles of dt of each

check_routable(sub_reaches)

Raise ValueError unless the routing kernels can route this network as sub_reaches sub-reaches per river, as from conditioning: no more rivers, sub-reaches, sub-reaches of one river, or inflow rivers of one river than river_route/kernels/c/checks.c allows. Routing checks it before any runoff is read, so a network the kernels would refuse is refused while it is prepared, with the river that breaks the limit named. A river with an x near 1/2 is split into the most sub-reaches; see the routing algorithm docs for holding x lower.

Parameters:

Name Type Description Default
sub_reaches IntArray

the sub-reaches of each river, 1 for a river that is not split

required

check_validity(dt, action='raise', layout=None)

Report rivers with a negative Muskingum coefficient at dt and take the configured action. With layout a river is checked as its sub-reaches, each at its own step.

Every coefficient is non-negative when 2*k*x <= dt <= 2*k*(1-x). Outside that window the scheme is still stable and still conserves mass, since c1+c2+c3 == 1 holds for negative coefficients too, but its discharge can dip below zero ahead of a rise (c1 < 0) or alternate from step to step and go negative (c3 < 0). The kernels write negative discharge as routed.

Parameters:

Name Type Description Default
dt float

the routing time step to check against

required
action Literal['log', 'raise', 'ignore']

raise raises ValueError, since the routing of those rivers is not valid, log logs it at the error level and routes on, ignore does nothing

'raise'
layout Layout | None

optional sub-reaches of each river and the subcycles of each, as from conditioning

None

check_stabilizable()

Raise ValueError for rivers with an x above MAX_STABILIZED_X, which routing with use_subreaches or use_subcycles, and stabilize, refuse. A river needs up to ceil(1/(1-2x)) sub-reaches, 50 at x = 0.49, and many more above it, and at x = 1/2 its coefficients are all non-negative only at a step equal to its travel time, so that it can be impossible to stabilize.

stabilize(dt, *, weights=None)

Stabilize this network in place: every river that conditioning splits at dt is replaced by its sub-reaches in series, whose coefficients are all non-negative at dt or in the subcycles of it each takes, which routing with use_subcycles gives each row. The original river keeps its id and becomes the outlet sub-reach, and the added sub-reaches are injected directly upstream of it with synthetic True and ids counting down from -1,000,000. Deleting the synthetic rows gives back the original row order. A network is stabilized once: to stabilize it at another dt, build a new Network from the original network file.

The sub-reaches are those of conditioning, which is what use_subreaches and use_subcycles route inside the kernel, so they keep each river's mean travel time k and its travel time variance k**2 * (1 - 2x). A river they cannot make non-negative is kept whole. Rivers left with a negative coefficient are not reported here: invalid_mask finds them, and routing reports them as invalid_coefficients says.

Parameters:

Name Type Description Default
dt float

the routing time step at which, or in subcycles of which, every coefficient of every sub-reach must be non-negative

required
weights list | None

optional explicit sub-reaches, one sequence of relative lengths per river in network table order. Each river's k is apportioned in proportion to its weights, which is what matches sub-reaches to real segment geometry, and its sub-reaches all take x = 1/2 - (1/2 - x)/sum(w**2) of their shares w of k, which keeps its variance, or 0 where that is negative. A river with a single weight is not split. Unlike the sub-reaches laid out for dt, which keep whole a river they cannot make non-negative, sub-reaches given here are always built as asked, even for a river with a sub-reach that has a negative coefficient.

None

Returns:

Name Type Description
Self Self

this network, stabilized, with a synthetic flag on every row

Raises:

Type Description
ValueError

if this network is already stabilized or dt is not positive

write_stabilized(dt, path=None, *, weights=None)

Stabilize this network for dt and write it as a network table with the synthetic column.

Parameters:

Name Type Description Default
dt float

the routing time step at which, or in subcycles of which, every coefficient of every sub-reach must be non-negative

required
path PathInput | None

where to write the parquet. Defaults to <network file stem>_stabilized<dt>.parquet next to the network file this Network was read from.

None
weights list | None

passed to stabilize

None

Returns:

Name Type Description
Path Path

the file written

to_parquet(path)

Write the network table, with every column in its dtype of NETWORK_DTYPES, to a parquet file.

Parameters:

Name Type Description Default
path PathInput

the parquet file to write

required

Returns:

Name Type Description
Path Path

the file written

Layout

Bases: NamedTuple

How each river is routed at a step: as one or more sub-reaches in series, each river's upstream to downstream, each with its own travel time, x, and subcycles of the step. A river routed whole is one sub-reach of its own k and x.

Splitting

The search for the fewest sub-reaches of unequal travel times, each in subcycles of its own, that Network.conditioning splits a river into where equal sub-reaches and subcycles cannot make it valid alone.

river_route.network.splitting

The fewest sub-reaches a river splits into when equal sub-reaches cannot make every coefficient of it non-negative at a step. The sub-reaches may differ in travel time, each takes subcycles of the step of its own, and every one keeps the river's diffusion: a sub-reach of travel time k_i takes x_i = 1/2 - (1/2 - x) k / k_i, which keeps the river's travel time variance k**2 (1 - 2x) across them whatever their travel times. The window of steps at which a sub-reach is valid, 2 k_i x_i <= step <= 2 k_i (1 - x_i), is then |step - k_i| <= (1 - 2x) k, as wide for every sub-reach of the river.

In m subcycles of a step dt, a sub-reach may take any travel time within (1 - 2x) k of dt/m and no shorter than (1 - 2x) k, where its x is 0, so the travel times a sub-reach may take are a union of windows about dt, dt/2, dt/3, and so on. A river splits into n sub-reaches when its k is a sum of n of them. The totals of 1, 2, 3, ... sub-reaches are grown as sorted intervals, and n sub-reaches make k when a total of the first n - n//2 of them and a total of the last n//2 do, so the totals are grown only to half the fewest n. The window of each sub-reach is then read back from them.

fewest_sub_reaches(k, x, dt, most)

The travel times, upstream to downstream, of the fewest sub-reaches of a river, at most most, whose coefficients are all non-negative at dt, and the subcycles of dt each takes: the fewest whose step is in its window, as Network.subcycles gives a river. Empty arrays where no split of at most most sub-reaches is valid, as at x = 1/2, whose windows close to single steps.

split_rivers(k, x, dt, most)

The fewest_sub_reaches of each river.

Returns:

Name Type Description
tuple tuple[IntArray, Float64Array, IntArray]

(sub_reaches, travel_times, subcycles). sub_reaches is the count of each river, 0 where none is valid, and travel_times and subcycles have a value for each sub-reach, river after river.

Streams

The functions a Network partitions its rivers with, the analysis of how evenly a partition divides a network, and subsetting a network file, and its grid weights, to one river and every river upstream of it.

river_route.network.streams

Functions of a network table in DFS order: the sub-watershed blocks threads route and how evenly they divide a network, and the subset of a network file, and of its grid weights, to one river and every river upstream of it. The blocks and the subset only read the contiguous run of rows that a river's riverIndex and upstreamCount give.

assign_depth_first_search_order(df)

Assigns a Depth First Search (DFS) order and upstream count to a DataFrame representing a river network: the rows reordered so that every river follows the rivers upstream of it, and the riverIndex and upstreamCount columns that describe that order. Rivers draining into the same river, and basin outlets, keep their relative order, so a table already in DFS order keeps its order. The DataFrame given is not changed.

Parameters:

Name Type Description Default
df DataFrame

the network table, with a riverId column and a nextRiverId column that is -1 at an outlet

required

Returns:

Type Description
DataFrame

pd.DataFrame: df copy with new riverIndex and upstreamCount columns ascending sorted by the new riverIndex

Raises:

Type Description
ValueError

column is missing, riverId repeats, nextRiverId is not a riverId, river drains to itself

assign_blocks(upstream_counts, downstream_indices, threads)

Divide a region in DFS order into sub-watershed blocks, pack the blocks into threads jobs that route concurrently, and leave every other river to the main stem, the last job, which routes after them.

A loose cap leaves a short main stem but packs unevenly, a tight one packs evenly but leaves more rivers to the main stem, and the trade turns over at a different cap per network. So each cap of _BLOCK_CAP_FRACTIONS is tried, and the one whose busiest job and main stem together route the fewest rivers is kept. Blocks are kept only when they beat routing every river in one job, which they never do on one thread.

Parameters:

Name Type Description Default
upstream_counts IntArray

(n,) number of rivers upstream of each river, the upstreamCount column

required
downstream_indices Int32Array

(n,) index of each river's downstream river, -1 at a basin outlet

required
threads int

number of jobs to pack the blocks into

required

Returns:

Name Type Description
tuple tuple[JobBlocks, ...]

the (block_starts, block_stops) of each job, its blocks in index order, each the rows from a start up to a stop. The last is the main stem, whose blocks are the runs of rows between the sub-watershed blocks. A block's outlet is its last row, the one river of the block that drains out of it.

analyze_partitioning(network, threads=4)

Static analysis of dividing a network into blocks for multithreaded routing (see assign_blocks).

Reports the block count, how evenly they pack into threads jobs, how much of the network falls into the sequential main stem, and the speedup that implies. The speedup is an upper bound from work alone: it assumes perfect scaling and ignores memory bandwidth, which this kernel is largely bound by. Also prints a human-readable report.

Parameters:

Name Type Description Default
network Network

the network to divide

required
threads int

thread count the blocks are sized for

4

Returns:

Name Type Description
dict dict

the summary statistics

subset_network_to_river(target_river, network_file, out_network_file, weights_file=None, out_weights_file=None)

Subset a network file, and optionally its grid weights, to the target river and every river upstream of it.

In DFS order a river's upstream watershed is the run of rows ending at it, so the subset is the rows whose riverIndex lies from the target's riverIndex minus its upstreamCount to its riverIndex. The target river becomes the outlet (nextRiverId set to -1) of the subset. Every river keeps its riverIndex and upstreamCount, which describe the subset as well as the whole network. With weight table paths the weight table is also filtered to the rivers of the subset.

Parameters:

Name Type Description Default
target_river int

riverId of the river to subset to (becomes the outlet)

required
network_file PathInput

path to the full network file parquet

required
out_network_file PathInput

path to write the subset network file parquet

required
weights_file PathInput | None

path to the full grid weights netCDF file (optional)

None
out_weights_file PathInput | None

path to write the subset grid weights netCDF file, given exactly when weights_file is

None

Raises:

Type Description
ValueError

if the target river is not in the network file, or only one of the weights files is given