cross_probe_qc

class cross_probe_qc.BuildXTable

Bases: object

Takes a dictionary of instances of qaqc.ApplyFlags(). The keys to the dictionary are probe codes.

static assemble_cross_table(probe_flags, probe_list=None, ppt_col='adj_precip')

Create a cross comparison table (like creating a pivot table on probe) where each column is a different probe.

Takes a dictionary of instances of qaqc.ApplyFlags(). The keys to the dictionary are probe codes. Data is then pulled from each site.

Parameters:

• probe_flags – dict. Dictionary of instances of qaqc.ApplyFlags()

• probe_list – list or str. Each probe in list is a column in the output table. If str == ‘all’ or val == None, list is generated from probe_flags keys.

• ppt_col – str. Column name of precipitation data.

Returns:

pd.DataFrame of precipitation by probe. Each column is precip from a different probe.

static assemble_wy_acc(df_tot)

Calculate the water year accumulation (cumulative sum) for eac column.

This will function on any dataframe, but its column-wise operations make it most suited for pivot table style DataFrames. The water year is defined as 10/1/xxxx 00:00 - 9/30/xxxx 23:55.

Parameters:

df_tot – pandas DataFrame containing precipitation data

Returns:

pandas DataFrame containing accumulated totals for each water year by column

static calc_wy_acc(data_series)

Reset the water year accumulation (cumulative sum) for adjusted values. To be preformed occasionally if this column of data is needed.

Note

This was originally applied by assemble_wy_acc() using df_tot.apply(calc_wy_acc)`. However, this method was 40% slower. As of 11/17/25 this method is no longer used.

Parameters:

data_series – pandas data series containing precip data

Returns:

pandas data series of cumulative some of precip by water year

class cross_probe_qc.Probe2ProbeXQc(df_index)

Bases: object

Cross compare 2 probes against each other to identify clogs and anomalous accumulations.

Each site has its own relationship to the base site, defined in qa_param.yaml. This class compares 2 sites at a time. New pairs can be imported into a new instance of the class using import_QaRules_data()

clog_pair_flagging_wrap(base, match, min_accum=34, lowest_normal_ratio=0.025, rolling_window='8D', window_precision=0.01, precision_val=0.254, window='1h', n_std=1.5)

Compare a base probe to a matched/paired probe to identify periods where the base probe was clogged and flag all clogs with U for undercatch, C for delayed accumulation, or no flag during rain free periods.

Flagging clogs requires a multistep approach for each probe pair (base and match). This method wraps together multiple steps into a single function:

1. identify periods where the matched probe indicates a clog in the base probe

2. calculate the running average of precip + n*std_deviation for both probes

3. flag undercatch during the clog

4. flag delayed accumulation (Cumulative total) within 3 timesteps of a clog ending

Parameters:

• base – str. The name of the probe being assessed for clogs.

• pair – str. The name of the probe to be compared to the base probe.

• min_accum – see set_clog_event()

• lowest_normal_ratio – see set_clog_event()

• rolling_window – see set_clog_event().

• window_precision – see set_clog_event()

• precision_val – see qaqc.QaRules.calc_rolling_val()

• window – see qaqc.QaRules.calc_rolling_val()

Returns:

three pandas Series of boolean values; clog_events, U_flags, C_flags

flag_clog_delayed_accum(clog, precip_run_avg, pair=())

Assess a pair of probes for periods where the base probe is measuring delayed accumulation from an earlier timstep.

A clog can prevent accumulated precip from being measured (undercatch). A sudden unclog, a slow leaking clog, or a slow or sudden melt can all lead to precip accumulation after the preciptation fell, including during periods of low or no precipitation. The amount accumulated represents the cumulative total precipitation since the clog/undercatch began. The amount of precip is often correct (and large), but is recorded at the wrong time.

Parameters:

• clog – pandas Series of booleans that are True when the pair identifies a clog.

• precip_run_avg – pandas Series of running avg + n*std_deviation.

• pair – tuple of str. The base probe name and the paired probe name.

Returns:

pandas Series of booleans that are True when the pair identifies delayed accumulation.

flag_clog_undercatch(clog, precip_run_avg, pair=())

Assess a pair of probes for periods where the base probe is measuring less precip than the paired probe.

Find periods where the paired probe indicates that the base probe is clogged and is accumulating substantially less precip than the paired probe.

Parameters:

• clog – pandas Series of booleans that are True when the pair identifies a clog.

• precip_run_avg – pandas Series of running avg + n*std_deviation.

• pair – tuple of str. The base probe name and the paired probe name.

Returns:

classmethod import_QaRules_data(df_2probe_tot, df_2probe_acc, df_2probe_ratio)

Populate an instance of Probe2ProbeXQc() with precip, YTD accumulated precip for each water year, and a ratio of the paired YTD accumulation.

Parameters:

• df_2probe_tot – pandas DataFrame containing precip totals since last time step for a pair of probes.

• df_2probe_acc – pandas DataFrame containing YTD accumulated precip for each water year for a pair of probes.

• df_2probe_ratio – pandas Series containing the ratio of YTD accumulated precip for a pair of probes.

Returns:

instance of class Probe2ProbeXQc()

set_clog_event(pair=(), min_accum=34, lowest_normal_ratio=0.025, rolling_window='8D', window_precision=0.01, max_ratio=0.4)

Asses a pair of probes to identify clog events.

Identify clogs by periods where the ratio between a pair of probes drops below the running average of the ratio. This effectively identifies periods where the ratio has a dropping trend, signaling that the base probe is accumulating substantially less than the paired probe, which is interpreted as a clog event.

Parameters:

• pair – tuple of str. The base probe name and the paired probe name.

• min_accum – The minimum water year accumulation required before clog analysis can begin.

• lowest_normal_ratio – The lowest allowable ratio between the base probe and probe. Set as clog below this value.

• rolling_window – str. The size of the rolling window applied to ratio for clog analysis. Must be a valid format for pandas.to_timedelta().

• window_precision – float. The minimum amount of change when the ratio between probes is dropping (negative trend).

class cross_probe_qc.XProbesQc(df_index, base_probe)

Bases: object

Cross probe quality checks to find anomalous collections. This class works with a pivot table by probe.

Takes a base probe, and compares every other probe to that probe. Once all have been compared, site specific weighting is applied to the boolean DataFrames to determine if an event occured and then the flag with the highest weighted number is applied.

Uses Probe2ProbeXQc() to identify clogs and flags through comparison of individual pairs.

calc_accum_ratio(ACC)

Calculate the ratio of YTD accumulated precip for all probes compared to the base_probe.

Base probe is set for the instance.

Parameters:

ACC – pandas DataFrame where each column is the YTD water year accumulated precip for a probe to be compared to the base_probe.

Returns:

pandas DataFrame where each column is the ratio of YTD water year accumulated precip.

static calc_ratio(base, compare)

Calculate the raio between a base value and a comparison value.

Parameters:

• base – pandas Series of values being investigated.

• compare – pandas Series of values being compared to base values.

Returns:

pandas Series containing raio between base and compare.

flag_x_clogs(clogstot, Utot, Ctot)

Assign flags based on weighted totals from all probes assessed.

Parameters:

• clogstot – pd.Series of integers of weighted clog scores totalled across all probes.

• Utot – pd.Series of integers of weighted U-flag scores totalled across all probes.

• Ctot – pd.Series of integers of weighted C-flag scores totalled across all probes.

get_Probe2ProbeXQc_inst(tot, acc, probe='')

Initialize and return an instance of Probe2ProbeXQc.

Create a paired comparison between the base probe and another probe.

Parameters:

• tot – Pandas DataFrame containing the total accumulated precipitation since last measurement (default 5 min).

• acc – Pandas DataFrame containing the accumulated precipitation for this Water Year.

• probe – str. The name of the probe to be compared to the base probe.

Returns:

Instance of Probe2ProbeXQc() comparing probe against base_probe.

get_weight_x_clog(weights)

Weight each cross comparison. Probes at the same site are weighted 0.; probes at a simlar elevation or along the same ridge are weighted 0.3; and other probes are weighted lower. The total clog value will be determined as the sum of all weights time the clog value.

values > 2/3 will be flagged as a clog. or at least 2 out of the probes

• uplo- vara and mack each rated 0.334, within site rated 0.6. CENT 0.07 per probe, 0.21 total prim 0.2, and all others 0.17.

• vara- uplo and cent 0.17 per probe, 0.51 total, prim/mack 0.2, and all others 0.17.

Parameters:

weights – dictionary of integers defining the weight to apply to each site. Keyed by site string.

Returns:

Single column of total weighted score for clog, U flag, and C flag

plot_clog_wind_thresholds(probe, xppt, xacc, min_ratio, min_accum=[50], days=[6, 8, 10], prec=[0.02, 0.03, 0.04])

Plot clogs identified using moving windows of different sizes and precision thresholds.

This is used to determine the ideal parameters for identifying clogs from the ratio of a particular site to the base probe.

Parameters:

• probe – str. Probe being compared to the base probe.

• xppt – pandas DataFrame containing total accumulated precip since last measurement with a column for each probe.

• xacc – pandas DataFrame containing total accumulated precip for the wateryear with a column for each probe.

• min_ratio – float. Lowest normal ratio. All values < min_ratio are flagged as a clog.

• min_accum – iterable (list or tuple) of float. Minimum water year accumulation before ratios are evaluated for clogs.

• days – iterable (list or tuple) of int. Size of the moving window in days.

• prec – iterable (list or tuple) of float. The minimum amount of change to ID a clog when the ratio between probes is dropping (negative trend).

plot_wy_start(acc, comparison_probe)

Plot the beginning of the water year to determine minimum accumulation before usable/stable ratios exist.

Parameters:

• acc

• comparison_probe

Returns:

plot_x_clogs(day, tdelta='10D')

Plot clog identification by multiple probes.

The ratio of each probe (accumulation compared to the base probe) is plotted for the specified days and periods each probe identifies as clogs are highlighted. This helps visualize the pattern of the ratios during the period and shows which combination of probes signaled a clog. Final clogs are the weighted composite of individual probe clogs.

Parameters:

• day – Pandas Datetime.

• tdelta – Str. Defines how much time is plotted after day. Must be a valid format for pandas.to_timedelta(). Defaults to ‘10D’.

Returns:

matplotlib axes.

set_accum_ratio(ACC)

Set the instance parameter self.ratio by calculating the ratio of YTD accumulated precip for all probes compared to the base_probe.

Parameters:

ACC – pandas DataFrame where each column is the YTD water year accumulated precip for a probe to be compared to the base_probe.

set_x_clogs(tot, acc, params)

For each probe, perform a cross comparison against all other probes to identify clogs.

This function creates a pandas DataFrame where each column contains boolean values for a probe paired with a base parameter.

:param tot:Pandas DataFrame containing the total accumulated precipitation since last measurement (default 5 min). :param acc: Pandas DataFrame containing the accumulated precipitation for this Water Year. :param params: dict. Keyed by site with kwargs for Probe2ProbeXQc.clog_pair_flagging_wrap().