Checking Diurnal Flux at VARA

VARA rain gauge had a period of extreme fluctuation for at least 2 years. The magnitude of the fluctuations requires that the quality checks be carefully reviewed to make sure they are still functioning properly in this extreme case.

Site Overview

This site had a sensor that failed over several years, eventually leading to totally bogus data. However, in the lead up, there were extreme fluctuations that can hopefully be managed with the diurnal_flux protocol that has been put together. After sensor replacement, the data is the cleanest on the Andrews.

# Jupyter magic to make plots display interactive
# must install ipympl (Ipython-matplotlib) and nodejs
from ipywidgets.embed import embed_minimal_html
from ipywidgets import Layout
import matplotlib.pyplot as plt

%matplotlib ipympl

# expand all plots to comfortable viewing size
plt.rcParams['figure.figsize'] = [8, 5]
Layout(width='400px', height='300px')

import pandas as pd
from numpy import nan, arange
import scipy

import sys
sys.path.append("../")
from post_gce_qc import qaqc, data_transfer, cross_probe_qc, main

import os
x = os.environ.get('READTHEDOCS')
print(x)

None

# laod data
prov = data_transfer.LoadProvisionalData(file_n='../config_new.yaml')
prov.load_ppt_data(strtyr=2019, endyr=2022)

df = prov.pivot_on_probe(prov.df, 'VAR', '02')

# apply rules
#-------------------------
param = qaqc._load_yaml('../qa_param.yaml')['VAR_02']
qc = qaqc.QaRules(df, qa_params=param)

df.ACC.plot(grid=True, legend=True)

<Axes: xlabel='Date'>

df.ACC.groupby(pd.Grouper(freq='YE-SEP')).max()

2019-09-30     3339.620117
2020-09-30     8075.040527
2021-09-30    90421.734375
2022-09-30    90421.734375
2023-09-30      842.203979
Freq: YE-SEP, Name: ACC, dtype: float[pyarrow]

Right off the bat, we go from 3m of precip, to 8m, to 90m. Let’s look at our impressive 8m year. Anything over 4m is probably bogus, but let’s dig in and check it out. Let’s switch over to raw tank numbers.

plt.figure()
df.INST.plot(grid=True, legend=True)

<Axes: xlabel='Date'>

WY 2021 the sensor appears pegged near max levels, so I think we can write off that WY. I believe that year we actually capped the rain gauge to stop it from accumulating. What is the current flagging for that year?

strt, end = pd.to_datetime('10/1/2020'), pd.to_datetime('9/30/21')

pd.options.display.min_rows = 30
df[strt:end]

	INST	INST_Flag	TOT	TOT_Flag	ACC	ACC_Flag
Date
2020-10-01 00:00:00	96.599998	Q	0.0	<NA>	8075.040527	<NA>
2020-10-01 00:05:00	96.599998	Q	0.0	<NA>	0.0	<NA>
2020-10-01 00:10:00	96.300003	Q	0.0	<NA>	0.0	<NA>
2020-10-01 00:15:00	96.599998	Q	0.0	<NA>	0.0	<NA>
2020-10-01 00:20:00	96.599998	Q	0.0	<NA>	0.0	<NA>
2020-10-01 00:25:00	96.599998	Q	0.0	<NA>	0.0	<NA>
2020-10-01 00:30:00	96.5	Q	0.0	<NA>	0.0	<NA>
2020-10-01 00:35:00	96.599998	Q	0.0	<NA>	0.0	<NA>
2020-10-01 00:40:00	96.300003	Q	0.0	<NA>	0.0	<NA>
2020-10-01 00:45:00	96.599998	Q	0.0	<NA>	0.0	<NA>
2020-10-01 00:50:00	96.599998	Q	0.0	<NA>	0.0	<NA>
2020-10-01 00:55:00	96.300003	Q	0.0	<NA>	0.0	<NA>
2020-10-01 01:00:00	96.599998	Q	0.0	<NA>	0.0	<NA>
2020-10-01 01:05:00	96.599998	Q	0.0	<NA>	0.0	<NA>
2020-10-01 01:10:00	96.300003	Q	0.0	<NA>	0.0	<NA>
...	...	...	...	...	...	...
2021-09-29 22:50:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 22:55:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 23:00:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 23:05:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 23:10:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 23:15:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 23:20:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 23:25:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 23:30:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 23:35:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 23:40:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 23:45:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 23:50:00	495.399994	M	0.0	M	90421.734375	M
2021-09-29 23:55:00	495.399994	M	0.0	M	90421.734375	M
2021-09-30 00:00:00	495.399994	M	0.0	M	90421.734375	M

104833 rows × 6 columns

df.loc[strt:end, 'TOT_Flag'].describe()

count     25703
unique        6
top           M
freq      16892
Name: TOT_Flag, dtype: object

Mostly Missing. That seems appropriate. Let’s look at what happens after we apply those M flags.

qc.flag_propagate_EM_from_tank()

flag = qaqc.ApplyFlags(df.index, param['precision'])
flag.import_provisional_data(df)
flag.apply_QaRules_flags(qc.qa_events, qc.qa_flags)

flag.apply_0_val()
flag.apply_NAN_val()

plt.figure()
cross_probe_qc.BuildXTable.calc_wy_acc(flag.data.adj_precip).plot(grid=True, legend=True)

<Axes: xlabel='Date'>

plt.figure()
df.INST.plot(grid=True, legend=True)

<Axes: xlabel='Date'>

I don’t think that’s salvageable, and a lot of it isn’t flagged missing in GCE. It’s not oscillating daily, but is oscilating multiple times daily, and the values are at or near max. I’ll add a manual flag to take care of it.

flag.apply_manual_flags(param['manual_flags'])

flag.apply_0_val()
flag.apply_NAN_val()

plt.figure()
cross_probe_qc.BuildXTable.calc_wy_acc(flag.data.adj_precip).plot(grid=True, legend=True)

<Axes: xlabel='Date'>

One problem down! 8m of rainfall is a big improvement on 90m. Now let’s see abou those 5 m of rain in July and Aug of 2020.

Flag Precip During Tank Flux

Apply post_gce_qc.qaqc.QaRules.flag_precip_during_tank_flux() and see what gets flagged and zoom into those days or weeks for testing.

qc.flag_precip_during_tank_flux(fluxprecision=5, accprecision=5, fluxwindow=289, accwindow='1D', extend_ahead=3)

plt.figure()
df['INST'].plot(legend=True, label='tank height')
df.loc[qc.qa_events.diurnal_flux, 'INST'].plot(legend=True, grid=True, label='flux precip', linestyle='', marker='.')

<Axes: xlabel='Date'>

flag.apply_QaRules_flags(qc.qa_events, qc.qa_flags)

flag.apply_0_val()
flag.apply_NAN_val()

flag.prorate_precip_during_tank_flux()

from scipy.ndimage import median_filter
trend = median_filter(df['INST'], size=289, mode='nearest')

# the plotting function is expecting a very specific format for a trend.
trend = pd.DataFrame(data=trend, index=df.index, columns=['mean'])

day = pd.to_datetime('4/17/19')
axppt, axtank = flag.plot_flagged_day(day, 'VAR02', tdelta='3D', auto_qa_event=qc.qa_events, running_tank=trend)

flag.data.tank_height[day]

25.059999465942383

end = day + pd.to_timedelta('3D')

flag.data.loc[day:end, ['precip', 'adj_precip']].cumsum()

	precip	adj_precip
Date
2019-04-17 00:00:00	0.0	0.0
2019-04-17 00:05:00	0.0	0.0
2019-04-17 00:10:00	0.0	0.0
2019-04-17 00:15:00	0.0	0.0
2019-04-17 00:20:00	0.0	0.0
2019-04-17 00:25:00	0.0	0.0
2019-04-17 00:30:00	0.0	0.0
2019-04-17 00:35:00	0.0	0.0
2019-04-17 00:40:00	0.0	0.0
2019-04-17 00:45:00	0.0	0.0
2019-04-17 00:50:00	0.0	0.0
2019-04-17 00:55:00	0.0	0.0
2019-04-17 01:00:00	0.0	0.0
2019-04-17 01:05:00	0.0	0.0
2019-04-17 01:10:00	0.0	0.0
...	...	...
2019-04-19 22:50:00	8.800001	3.5082
2019-04-19 22:55:00	9.050001	3.7582
2019-04-19 23:00:00	9.050001	3.7582
2019-04-19 23:05:00	9.050001	3.7582
2019-04-19 23:10:00	9.050001	3.7582
2019-04-19 23:15:00	9.170001	3.8782
2019-04-19 23:20:00	9.300001	4.0082
2019-04-19 23:25:00	9.300001	4.0082
2019-04-19 23:30:00	9.300001	4.0082
2019-04-19 23:35:00	9.300001	4.0082
2019-04-19 23:40:00	9.300001	4.0082
2019-04-19 23:45:00	9.550001	4.2582
2019-04-19 23:50:00	9.550001	4.2582
2019-04-19 23:55:00	9.550001	4.2582
2019-04-20 00:00:00	9.670001	4.3782

865 rows × 2 columns

acc = flag.data.loc[day:end, ['precip', 'adj_precip']].cumsum()
tank_acc = acc + flag.data.tank_height[day]

axtank.plot(tank_acc, label=['raw precip ACC', 'prorated precip ACC'])
plt.legend()

<matplotlib.legend.Legend at 0x386207370>

def plot_prorate(flag, day, site, tdelta, qa_event, trend):
    axppt, axtank = flag.plot_flagged_day(day, site, tdelta=tdelta, auto_qa_event=qa_event, running_tank=trend)

    end = day + pd.to_timedelta(tdelta)
    tank_start =  flag.data.tank_height[day]

    acc = flag.data.loc[day:end, ['precip', 'adj_precip']].cumsum()
    tank_acc = acc + tank_start

    axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')
    axtank.plot(tank_acc.adj_precip, label='prorated precip ACC', color='mediumspringgreen', linestyle='-.')

    plt.legend()

    min_tank = flag.data.loc[day:end, 'tank_height'].min()

    axtank.set_ylim([min_tank*0.98, tank_acc.precip.iloc[-1]*1.02])

plot_prorate(flag, day, 'VAR02', '3D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/3862428168.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

This is an interesting case where the prorating and flagging are both working imperfectly. The middle day, all precip is flagged and the difference from midnight to midnight is so small it’s rounded to 0. So the accumulation is prorated to zero.

The first day, however, starts with some actual precip and flows right into a fluctuation. The first part of the fluctuation is unflagged, so the prorating evenly distributes precip between the real increase and the first 3 precipitation records of the fluctuation.

The last day is the opposite, ending with real precip. This means that the day didn’t overaccumulate, so none of the precip is flagged and it is not prorated.

The current structure applies all rules before manual flags, so that manual flags can change the results of any rule. They can change the flag and change whether the data is set to 0 or NAN, either reversing or adding a zero. However, zeroing out values after they have been prorated will make the daily total incorrect. So the only effective manual flag is just to change the flags. The first day is already flagged E with event code INTPRO (internal processing error). So the last day could be manually flagged Q. But that’s about it… Or we could flag all days like this as M and fill with NAN. Or flag all days like this as Q.

day = pd.to_datetime('4/21/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/3862428168.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

The only problem here is ``simple_pre.m`` is underaccumulating during actual rain. Everything else looks pretty good. Even though it isn’t flagging every bit of rain during the fluctuations, it is all prorated to 0 rain.

day = pd.to_datetime('4/28/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/3862428168.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('5/6/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/3862428168.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Two of those days need to be flagged, and aren’t. Let’s see what’s going on in a bit more detail.

day = pd.to_datetime('5/9/19')
plot_prorate(flag, day, 'VAR02', '4D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/3862428168.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

end = day + pd.to_timedelta('1D')

dly_over = qc.find_over_accum(df, 0.2794, nprecision=5)

dly_over[day:end]

Date
2019-05-09 00:00:00    False
2019-05-09 00:05:00    False
2019-05-09 00:10:00    False
2019-05-09 00:15:00    False
2019-05-09 00:20:00    False
2019-05-09 00:25:00    False
2019-05-09 00:30:00    False
2019-05-09 00:35:00    False
2019-05-09 00:40:00    False
2019-05-09 00:45:00    False
2019-05-09 00:50:00    False
2019-05-09 00:55:00    False
2019-05-09 01:00:00    False
2019-05-09 01:05:00    False
2019-05-09 01:10:00    False
                       ...
2019-05-09 22:50:00    False
2019-05-09 22:55:00    False
2019-05-09 23:00:00    False
2019-05-09 23:05:00    False
2019-05-09 23:10:00    False
2019-05-09 23:15:00    False
2019-05-09 23:20:00    False
2019-05-09 23:25:00    False
2019-05-09 23:30:00    False
2019-05-09 23:35:00    False
2019-05-09 23:40:00    False
2019-05-09 23:45:00    False
2019-05-09 23:50:00    False
2019-05-09 23:55:00    False
2019-05-10 00:00:00     True
Length: 289, dtype: bool[pyarrow]

day_grp = pd.Grouper(freq='1D')

dly_ppt = qc.calc_dly_precip(df, day_grp)

dly_ppt[day:end]

Date
2019-05-09 00:00:00     3.49
2019-05-09 00:05:00     3.49
2019-05-09 00:10:00     3.49
2019-05-09 00:15:00     3.49
2019-05-09 00:20:00     3.49
2019-05-09 00:25:00     3.49
2019-05-09 00:30:00     3.49
2019-05-09 00:35:00     3.49
2019-05-09 00:40:00     3.49
2019-05-09 00:45:00     3.49
2019-05-09 00:50:00     3.49
2019-05-09 00:55:00     3.49
2019-05-09 01:00:00     3.49
2019-05-09 01:05:00     3.49
2019-05-09 01:10:00     3.49
                       ...
2019-05-09 22:50:00     3.49
2019-05-09 22:55:00     3.49
2019-05-09 23:00:00     3.49
2019-05-09 23:05:00     3.49
2019-05-09 23:10:00     3.49
2019-05-09 23:15:00     3.49
2019-05-09 23:20:00     3.49
2019-05-09 23:25:00     3.49
2019-05-09 23:30:00     3.49
2019-05-09 23:35:00     3.49
2019-05-09 23:40:00     3.49
2019-05-09 23:45:00     3.49
2019-05-09 23:50:00     3.49
2019-05-09 23:55:00     3.49
2019-05-10 00:00:00    13.32
Name: TOT, Length: 289, dtype: float[pyarrow]

dly_tank = qc.calc_dly_tank_change(df, day_grp, 0.2794)

dly_tank[day:end]

Date
2019-05-09 00:00:00   -0.8382
2019-05-09 00:05:00   -0.8382
2019-05-09 00:10:00   -0.8382
2019-05-09 00:15:00   -0.8382
2019-05-09 00:20:00   -0.8382
2019-05-09 00:25:00   -0.8382
2019-05-09 00:30:00   -0.8382
2019-05-09 00:35:00   -0.8382
2019-05-09 00:40:00   -0.8382
2019-05-09 00:45:00   -0.8382
2019-05-09 00:50:00   -0.8382
2019-05-09 00:55:00   -0.8382
2019-05-09 01:00:00   -0.8382
2019-05-09 01:05:00   -0.8382
2019-05-09 01:10:00   -0.8382
                        ...
2019-05-09 22:50:00   -0.8382
2019-05-09 22:55:00   -0.8382
2019-05-09 23:00:00   -0.8382
2019-05-09 23:05:00   -0.8382
2019-05-09 23:10:00   -0.8382
2019-05-09 23:15:00   -0.8382
2019-05-09 23:20:00   -0.8382
2019-05-09 23:25:00   -0.8382
2019-05-09 23:30:00   -0.8382
2019-05-09 23:35:00   -0.8382
2019-05-09 23:40:00   -0.8382
2019-05-09 23:45:00   -0.8382
2019-05-09 23:50:00   -0.8382
2019-05-09 23:55:00   -0.8382
2019-05-10 00:00:00    0.5588
Name: INST, Length: 289, dtype: double[pyarrow]

OK, so the reason 5/9/19 isn’t flagged is because it has a negative tank change. Negative tank changes of 2 x precision are zeroed out. But this is 3 time precision…I’ll make that a flexible input.

%reload_ext autoreload
%autoreload explicit
%aimport post_gce_qc.qaqc

from importlib import reload
import post_gce_qc.qaqc
reload(post_gce_qc.qaqc)

<module 'post_gce_qc.qaqc' from '/Users/gregcohn/Library/CloudStorage/GoogleDrive-cohng@oregonstate.edu/Other computers/My Computer/workspace/IM_programs/fsdb_ppt/lab/../post_gce_qc/qaqc.py'>

# reload updated methods
qc = qaqc.QaRules(df, qa_params=param)

# reflag fluctuations
qc.flag_precip_during_tank_flux(fluxprecision=5, accprecision=5, zero_precision=3.1, fluxwindow=289, accwindow='1D', extend_ahead=3)

# apply flags
flag.apply_QaRules_flags(qc.qa_events, qc.qa_flags)
flag.apply_0_val()
flag.apply_NAN_val()

# prorate
flag.prorate_precip_during_tank_flux()

day = pd.to_datetime('5/9/19')
plot_prorate(flag, day, 'VAR02', '4D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/3862428168.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Fixed it! Made a wider swath of values considered zero tank change.

day = pd.to_datetime('5/13/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/3862428168.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Let’s check out that one day mid-rainfall.

day = pd.to_datetime('5/18/19')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/3862428168.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

end = day + pd.to_timedelta('1D')

dly_tank = qc.calc_dly_tank_change(df, day_grp, 0.2794)

dly_tank[day:end].iloc[0]

2.5145999999999997

dly_ppt = qc.calc_dly_precip(df, day_grp)

dly_ppt[day:end].iloc[0]

3.2300000190734863

Well that’s a double problem: 1) the day didn’t overaccumulate; 2)simple_pre.m only had 2 moments of precip designated outside the flux. So if we removed all of the fluctuating precip by tightening the accprecision, we still wouldn’t have precip when we’re supposed to and it would dump all of the precip into just two timesteps…

Manual Q flags to the rescue!…?

day = pd.to_datetime('5/20/19')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/3862428168.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

end = day + pd.to_timedelta('1D')

dly_tank = qc.calc_dly_tank_change(df, day_grp, 0.2794)

dly_tank[day:end].iloc[0]

5.587999999999999

dly_ppt = qc.calc_dly_precip(df, day_grp)

dly_ppt[day:end].iloc[0]

5.769999980926514

This is a pretty similar situation to the last: total accumulation is about right, but the timing of the accumulation is wrong. It’s only wrong by 4 x 5 minute totals, but without those, the daily total would be off. Since the daily total is so precise here, I’m not going to flag anything.

day = pd.to_datetime('5/23/19')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2023004291.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

It didn’t flag all of the precip, but it prorated everything to 0, so It worked out. I added E flags to all prorated values.

day = pd.to_datetime('5/27/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2023004291.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

You could argue a little of false capture at the beginning, but the prorated values track the troughs of the tank values pretty well.

day = pd.to_datetime('6/3/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

There is just a little bit of precip on during the flux on the drain day. Everything else looks good. I’ll throw in a manual flag on the drain day.

day = pd.to_datetime('6/8/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/Users/gregcohn/Library/CloudStorage/GoogleDrive-cohng@oregonstate.edu/Other computers/My Computer/workspace/IM_programs/fsdb_ppt/lab/../post_gce_qc/qaqc.py:343: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). Consider using `matplotlib.pyplot.close()`.
  plt.figure()
/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Looks good.

day = pd.to_datetime('6/15/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Looks pretty good. There is a little bit of unflagged precip on the last day, but it looks like it’s still prorated to 0 precip for the day.

day = pd.to_datetime('6/22/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

6/26/19 has the clasic pattern: daily accumulation is correct but some of it is misplaced during the fluctuation. This one is even worse, because after the fluctuation, the precip doesn’t come down to where it started, so some of the fluctuation was real precip.

day = pd.to_datetime('6/29/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

7/1/19 is same as the last graph: correct total with some wrong timing.

day = pd.to_datetime('7/6/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Oooo, missing a lot here. Let’s figure out why.

day += pd.to_timedelta('2D')
end = day + pd.to_timedelta('1D')

dly_tank = qc.calc_dly_tank_change(df, day_grp, 0.2794)

dly_tank[day:end].iloc[0]

0.2794

dly_ppt = qc.calc_dly_precip(df, day_grp)

dly_ppt[day:end].iloc[0]

1.0299999713897705

We can try reducing the threshold for overaccumulation.

# reload updated methods
qc = qaqc.QaRules(df, qa_params=param)

# reflag fluctuations
qc.flag_precip_during_tank_flux(fluxprecision=4, accprecision=3, zero_precision=3.1, fluxwindow=289, accwindow='1D', extend_ahead=3)

# apply flags
flag.apply_QaRules_flags(qc.qa_events, qc.qa_flags)
flag.apply_0_val()
flag.apply_NAN_val()

# prorate
flag.prorate_precip_during_tank_flux()

day = pd.to_datetime('7/6/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

That just leaves one day. I’m affraid to reduce the thresholds much further. I’ve gone back and done some spot checks and this is dangerously close to overflagging. I’ll take care of that day with manual flags.

day = pd.to_datetime('7/13/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

I’ll manually flag and zero out 7/15, but on the rain day, the total is correct, so even though some is accumulated during the fluctuation, I’m leaving it as is, because the total looks correct.

day = pd.to_datetime('7/20/19')
plot_prorate(flag, day, 'VAR02', '10D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('7/28/19')
end = day + pd.to_timedelta('1D')

dly_tank[day:end].iloc[0]

-0.0

dly_ppt[day:end].iloc[0]

0.5

Not enough precip to trigger a flag. I’ll do it manually.

day = pd.to_datetime('7/29/19')
plot_prorate(flag, day, 'VAR02', '20D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('8/3/19')
end = day + pd.to_timedelta('1D')

dly_tank[day:end].iloc[0]

-0.0

dly_ppt[day:end].iloc[0]

0.2199999988079071

day = pd.to_datetime('8/5/19')
end = day + pd.to_timedelta('1D')

dly_tank[day:end].iloc[0]

0.0

dly_ppt[day:end].iloc[0]

0.7900000214576721

Both days are just under the threshold for overaccumulation. I’ll manually flag them.

day = pd.to_datetime('8/17/19')
plot_prorate(flag, day, 'VAR02', '12D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

It’s under accumulating, but not due to fluctuation flagging. There are two days (8/19 and 8/24) where the precip is obviously during a fluctuation AND the tank is flat. Both were added to manual flags.

day = pd.to_datetime('8/29/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Wow, that’s some big and repeating precip. Luckily our repeat precip function will catch a lot of it, but it’s great that this catches it too. And what isn’t caught (because the fluctuation ended), is zeroed out by the prorating. So all around, a win. There are two days with a trace amount of precip all during fluctuations. The amounts are too small to be caught, so I manually flagged them.

day = pd.to_datetime('9/24/19')
plot_prorate(flag, day, 'VAR02', '3D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

end = day + pd.to_timedelta('1D')

dly_tank[day:end].iloc[0]

-1.3969999999999998

It’s too negative to get a flag. It’s treating it like a drain. I could adjust the zero change threshold, but I don’t want positive 1.369 to be counted as 0. So I’ll just manually flag it.

day = pd.to_datetime('9/25/19')
end = day + pd.to_timedelta('1D')

dly_tank[day:end].iloc[0]

-48.894999999999996

The second instance isn’t flagged for the same reason. I’ll manually flag it.

day = pd.to_datetime('9/27/19')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Two instances of the total being right, but the precip is placed inside a fluctuation. But there is 1 day to zoom in on and take a closer look.

day = pd.to_datetime('10/1/19')
plot_prorate(flag, day, 'VAR02', '2D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

end = day + pd.to_timedelta('1D')

dly_tank[day:end].iloc[0]

0.0

dly_ppt[day:end].iloc[0]

0.8999999761581421

Not a big enough difference to flag, but I’ll through on a manual flag for good measure.

day = pd.to_datetime('10/5/19')
plot_prorate(flag, day, 'VAR02', '3D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Too small to trigger a flag. I’ll throw on a manual flag.

day = pd.to_datetime('10/11/19')
plot_prorate(flag, day, 'VAR02', '3D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('1/24/20')
plot_prorate(flag, day, 'VAR02', '3D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('1/30/20')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('1/30/20')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

That’s a clear case of a double accumulation after an unclog. However I didn’t think I’d applied that rule…? The prorated number looks like it matches the tank. I’ll leave it at that for the moment.

day = pd.to_datetime('3/11/20')
plot_prorate(flag, day, 'VAR02', '3D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Looks good.

day = pd.to_datetime('3/17/20')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('3/26/20')
plot_prorate(flag, day, 'VAR02', '3D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

There’s at leas one questionable precip value in there, but the prorated total looks really close to the tank, so I’ll call that good enough.day = pd.to_datetime(‘1/30/20’) plot_prorate(flag, day, ‘VAR02’, ‘7D’, qc.qa_events, trend)

day = pd.to_datetime('3/30/20')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('4/6/20')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

The over accumulation occurred before the fluctuations during real precip, so this checks out OK.

day = pd.to_datetime('4/16/20')
plot_prorate(flag, day, 'VAR02', '6D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

4/19/20 is yet another example of the total precip being in the right ball park, but the timing is during an accumulation. 4/20/20 on the other hand is a small amount of precip when it should be flat. I’ll manually flag it. The flagging rules continue to capture the largest and worst instances of the problem.

day = pd.to_datetime('4/23/20')
plot_prorate(flag, day, 'VAR02', '3D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('5/3/20')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

5/5/20 is another case of a good total with poor timing of accumulation. Also, 5/8 should haveno precip. I’ll manually flag.

day = pd.to_datetime('5/14/20')
plot_prorate(flag, day, 'VAR02', '4D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Looks great!

day = pd.to_datetime('5/23/20')
plot_prorate(flag, day, 'VAR02', '8D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

end =  day + pd.to_timedelta('1D')

dly_tank[day:end].iloc[0]

0.5588

dly_ppt[day:end].iloc[0]

1.0299999713897705

I could flag it, but I wouldn’t know where to put that mm of precip if I did. But the next day is fair game to get rid of.

day = pd.to_datetime('5/31/20')
plot_prorate(flag, day, 'VAR02', '6D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Overaccumulation right out of the gate, but not by enough of a margin to trigger a flag, and I wouldn’t want to manually flag that day, it would need careful prorating. But then there are two other days with about 0.5mm each that should be wiped with a manual flag.

day = pd.to_datetime('6/11/20')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

That prorated line looks damn near perfect!

day = pd.to_datetime('6/21/20')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

6/22/20 looks like it should have a flag, but it also looks like there is a small increase in the baseline level, so it’s hard to know where that precip should go. It’s only 0.5mm, I’m going to leave it.

day = pd.to_datetime('7/7/20')
plot_prorate(flag, day, 'VAR02', '2D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

I can’t figure out what’s going on there. There are records that I visited the site that day and rewired the radio with the sw12v controlling a relay instead of powering the radio directly. There is a not that there was < 0.1” of rain after 1900…but that does not explain the 675 mm jump in the tank. It seems appropriate that it is being flagged and removed, though I have no real explanation for it.

day = pd.to_datetime('7/14/20')
plot_prorate(flag, day, 'VAR02', '21D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

70 mm of precip reduced to 0! Flat tank in the summer. Looks excellent.

day = pd.to_datetime('8/5/20')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('8/12/20')
plot_prorate(flag, day, 'VAR02', '9D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('8/21/20')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

end = day + pd.to_timedelta('1D')

dly_tank[day:end].iloc[0]

27.101799999999997

dly_ppt[day:end].iloc[0]

152.19000244140625

OK, so there is spike right at midnight! That shows up as a tank change, and then it is prorating that change across the handful of unflagged values from the day. There is no automated way to correct for that spike at midnight. I’ll manually flag it.

day = pd.to_datetime('8/22/20')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

end = day + pd.to_timedelta('1D')

dly_tank[day:end].iloc[0]

-107.8484

dly_ppt[day:end].iloc[0]

181.22999572753906

Continued consequences of the spike around midnight on 8/21-8/22. This looks like a giant drain, so it isn’t flagging anything. I’ll simply extend the manual flag.

day = pd.to_datetime('8/23/20')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

We can zoom out a bit now that the pattern has gone back to the usual daily peaks with no shenanigans at midnight.

day = pd.to_datetime('8/24/20')
plot_prorate(flag, day, 'VAR02', '15D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Let’s zoom in and look at that in chunks, but I think it’s looking allright.

day = pd.to_datetime('8/24/20')
plot_prorate(flag, day, 'VAR02', '3D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('8/27/20')
plot_prorate(flag, day, 'VAR02', '3D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Looks good.

day = pd.to_datetime('9/8/20')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('9/17/20')
plot_prorate(flag, day, 'VAR02', '7D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Some under-accumulation during real precip, but not something we attempt to deal with here.

day = pd.to_datetime('9/27/20')
plot_prorate(flag, day, 'VAR02', '5D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Two low precip days to manually flag. And that brings us to WY 21, which is entirely flagged, because the sensor went absolutely nuts. The manual flags were applied above to exclude that craziness.

There are a few flags in more recent years after the LT420 was replaced with the new LPRefineMe sensor, which had some clogging issues, but has behaved much better. We’ll spot check those flags to make sure everything is still working right.

day = pd.to_datetime('11/05/21')
plot_prorate(flag, day, 'VAR02', '14D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

Flagged one big 3mm dump, but missed two little precip events during flux.

end = day + pd.to_timedelta('1D')

dly_tank[day:end].iloc[0]

0.5588

dly_ppt[day:end].iloc[0]

1.0199999809265137

These events below 2mm are problematic for VARA, but when I lowered the threshold further, it overflagged, especially on days where it fluctuated, but had real precip at the beginning or end of the day.

day = pd.to_datetime('12/29/21')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

That looks like a clog catch up: a cumulative amount of precip since the clog began was suddenly released. We have notes from this day explaining that the drain was also clogged, so that they couldn’t get the tank to drop after getting the high, cumulative value. This will be manually unflagged. The clog identifiying routines should catch it, but to make sure the manually flag doesn’t undo any clog flags as well, the flag will be replaced with a C flag for cumulative/catch-up precipitation.

https://bitbucket.org/hjandrews/met_benchmarks/issues/85/sa-float-stuck

12/29/21 Update on vara clogging issue: Shaking the tube seemed to produce a good accumulation reading of about 464mm. That was about what we had estimate based on precip at other sites since the last clog, and the readings stabilized at 462-464mm. BUT when we tried to drain, only a very tiny dribble came out. The readings went down to 460, which is ambiguous as to whether the standpipe actually lost volume or was within the sensor wobble. But the most important thing is that we could not get the tank drained. We dug down to the exit pipe so we know it was a non-draining issue not a reading issue. We know there was liquid water in the tank and that the heater has been working well. Bump the orifice control temp up to 12 C and ran it up to that temp just to rule out ice clogs. The most likely explanation seems to be the clog in the drain that we initial thought happen in November has happened. Ran out of daylight and tools to try and fix the clog. It is definite;y closer to the tank than a screwdriver can reach from outside. We left the drain open in the hope that the tank might slowly drain itself before overtopping, but if the standpipe clog is still there we won’t know if the tank has drained. At this point I think that trying to keep Vara even semi functional this winter is going to be beyond our resources. I think a trip to get tank drained and orifice covered may be the the better part of valor for this winter, all things considered.

day = pd.to_datetime('7/3/22')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

I think this should be caught by the empty tank rule because it looks like it goes close to 0, but catching it two ways isn’t a bad thing. Let’s take a look at that rule and make sure it also catches this.

Note: from notes the sensor was disconnected for several days and reconnected at this time.

qc.flag_empty_tank()

day = pd.to_datetime('7/3/22 1530')
qc.qa_events[day:].head(20)

	drain_event	neg_delta_tank	overflow	duplicate	tank_empty	clog	diurnal_flux
Date
2022-07-03 15:30:00	False	False	False	False	False	False	False
2022-07-03 15:35:00	False	False	False	False	False	False	False
2022-07-03 15:40:00	False	False	False	False	False	False	False
2022-07-03 15:45:00	False	False	False	False	False	False	False
2022-07-03 15:50:00	False	False	False	False	False	False	False
2022-07-03 15:55:00	False	False	False	False	False	False	False
2022-07-03 16:00:00	False	False	False	False	False	False	False
2022-07-03 16:05:00	False	False	False	False	False	False	False
2022-07-03 16:10:00	False	False	False	False	False	False	False
2022-07-03 16:15:00	False	False	False	False	False	False	False
2022-07-03 16:20:00	False	False	False	False	False	False	False
2022-07-03 16:25:00	False	False	False	False	False	False	False
2022-07-03 16:30:00	False	False	False	False	False	False	False
2022-07-03 16:35:00	False	False	False	False	False	False	True
2022-07-03 16:40:00	False	False	False	False	False	False	True
2022-07-03 16:45:00	False	False	False	False	False	False	True
2022-07-03 16:50:00	False	False	False	False	False	False	False
2022-07-03 16:55:00	False	False	False	False	False	False	False
2022-07-03 17:00:00	False	False	False	False	False	False	False
2022-07-03 17:05:00	False	False	False	False	False	False	False

qc.df_orig[day:].head(20)

	INST	INST_Flag	TOT	TOT_Flag	ACC	ACC_Flag
Date
2022-07-03 15:30:00	8.86	MQ	0.0	M	713.460022	M
2022-07-03 15:35:00	8.86	MQ	0.0	M	713.460022	M
2022-07-03 15:40:00	8.86	MQ	0.0	M	713.460022	M
2022-07-03 15:45:00	8.86	MQ	0.0	M	713.460022	M
2022-07-03 15:50:00	8.86	MQ	0.0	M	713.460022	M
2022-07-03 15:55:00	8.86	MQ	0.0	M	713.460022	M
2022-07-03 16:00:00	8.86	MQ	0.0	M	713.460022	M
2022-07-03 16:05:00	8.86	MQ	0.0	M	713.460022	M
2022-07-03 16:10:00	8.86	MQ	0.0	M	713.460022	M
2022-07-03 16:15:00	8.86	MQ	0.0	M	713.460022	M
2022-07-03 16:20:00	4.382	Q	0.0	M	713.460022	M
2022-07-03 16:25:00	4.381	Q	0.0	<NA>	713.460022	<NA>
2022-07-03 16:30:00	4.38	Q	0.0	<NA>	713.460022	<NA>
2022-07-03 16:35:00	27.18	Q	22.798	J	736.257996	J
2022-07-03 16:40:00	27.309999	Q	22.927999	W	759.185974	W
2022-07-03 16:45:00	27.68	Q	0.37	<NA>	759.55603	<NA>
2022-07-03 16:50:00	27.68	Q	0.0	<NA>	759.55603	<NA>
2022-07-03 16:55:00	27.68	Q	0.0	<NA>	759.55603	<NA>
2022-07-03 17:00:00	27.68	Q	0.0	<NA>	759.55603	<NA>
2022-07-03 17:05:00	27.68	Q	0.0	<NA>	759.55603	<NA>

Looks like this was never at or below 0, so it won’t be caught by that rule. Let’s at least make sure it gets flagged for the repeated value. Even though this is catching the problem with our fluctuation flag, that’s not a very straight forward way to catch a sensor removal, so I’ll add a manual flag for overkill. It also won’t be caught by the GCE F flag rule, so it’s skipping a lot of our fail-safes .

qc.flag_double_precip()

day = pd.to_datetime('7/3/22 1600')
qc.qa_events[day:].head(15)

	drain_event	neg_delta_tank	overflow	duplicate	tank_empty	clog	diurnal_flux
Date
2022-07-03 16:00:00	False	False	False	False	False	False	False
2022-07-03 16:05:00	False	False	False	False	False	False	False
2022-07-03 16:10:00	False	False	False	False	False	False	False
2022-07-03 16:15:00	False	False	False	False	False	False	False
2022-07-03 16:20:00	False	False	False	False	False	False	False
2022-07-03 16:25:00	False	False	False	False	False	False	False
2022-07-03 16:30:00	False	False	False	False	False	False	False
2022-07-03 16:35:00	False	False	False	False	False	False	True
2022-07-03 16:40:00	False	False	False	True	False	False	True
2022-07-03 16:45:00	False	False	False	False	False	False	True
2022-07-03 16:50:00	False	False	False	False	False	False	False
2022-07-03 16:55:00	False	False	False	False	False	False	False
2022-07-03 17:00:00	False	False	False	False	False	False	False
2022-07-03 17:05:00	False	False	False	False	False	False	False
2022-07-03 17:10:00	False	False	False	False	False	False	False

day = pd.to_datetime('7/13/22')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

day = pd.to_datetime('7/13/22 1520')
qc.df_orig[day:].head(20)

	INST	INST_Flag	TOT	TOT_Flag	ACC	ACC_Flag
Date
2022-07-13 15:20:00	28.73	Q	0.0	<NA>	760.585999	<NA>
2022-07-13 15:25:00	28.85	Q	0.14	W	760.726013	W
2022-07-13 15:30:00	28.860001	Q	0.01	<NA>	760.736023	<NA>
2022-07-13 15:35:00	4.56	Q	0.0	R	760.736023	R
2022-07-13 15:40:00	4.558	Q	0.0	<NA>	760.736023	<NA>
2022-07-13 15:45:00	4.557	Q	0.0	<NA>	760.736023	<NA>
2022-07-13 15:50:00	4.557	Q	0.0	<NA>	760.736023	<NA>
2022-07-13 15:55:00	4.556	Q	0.0	<NA>	760.736023	<NA>
2022-07-13 16:00:00	4.556	Q	0.0	<NA>	760.736023	<NA>
2022-07-13 16:05:00	28.23	Q	23.67	J	784.406006	J
2022-07-13 16:10:00	28.35	Q	0.12	<NA>	784.526001	<NA>
2022-07-13 16:15:00	49.400002	Q	21.049999	J	805.575989	J
2022-07-13 16:20:00	49.66	Q	0.26	<NA>	805.835999	<NA>
2022-07-13 16:25:00	49.650002	Q	0.0	<NA>	805.835999	<NA>
2022-07-13 16:30:00	49.650002	Q	0.0	<NA>	805.835999	<NA>
2022-07-13 16:35:00	49.650002	Q	0.0	<NA>	805.835999	<NA>
2022-07-13 16:40:00	49.900002	Q	0.24	<NA>	806.075989	<NA>
2022-07-13 16:45:00	49.91	Q	0.01	<NA>	806.085999	<NA>
2022-07-13 16:50:00	49.91	Q	0.0	<NA>	806.085999	<NA>
2022-07-13 16:55:00	49.91	Q	0.0	<NA>	806.085999	<NA>

The sensor was jammed and the standpipe connection removed, allowing the float to drop to the bottom of the sensor. Once the float was properly repositioned, water was carefully re-added. This will have an additional manual flag to make sure all precip is removed.

day = pd.to_datetime('9/3/22')
plot_prorate(flag, day, 'VAR02', '14D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

dly_ppt[day]

0.5609999895095825

Too small to trigger anything, but I’ll remove it with a manual flag all the same.

day = pd.to_datetime('9/23/22')
plot_prorate(flag, day, 'VAR02', '5D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

dly_ppt[day]

0.42000001668930054

Another mini-precip day. The total amount of precip if we had one of these every other day would only be 90 mm per year, but it does make the number of rain days look a lot larger than reality, and number of rain days can be important to some analyses.

Impact of Fluctuation Flagging

Let’s see how we did. There should be a notable reduction in totals.

cross_probe_qc.BuildXTable.calc_wy_acc(flag.data[['adj_precip', 'precip']]).plot(grid=True, legend=True)

<Axes: xlabel='Date'>

That looks like it was able to restore WY’s 2019 and 2020, and that 2022 was marred by clogs, but at least maybe the fall is salvagable.

Additional WY’s

All other evaluations have stopped at WY 2022. However ‘23 and ‘24 are now available. Let’s take a look at how this flag behaves. Those water years at VARA should have a some clogging issues, and some sensor/standpipe rebuilds, but over all have less fluctuation because the sensor was replaced with a new sensor.

# load data
prov = data_transfer.LoadProvisionalData(file_n='../config_new.yaml')
prov.load_ppt_data(strtyr=2023, endyr=2024)

df = prov.pivot_on_probe(prov.df, 'VAR', '02')

# apply rules
#-------------------------
param = qaqc._load_yaml('../qa_param.yaml')['VAR_02']
qc = qaqc.QaRules(df, qa_params=param)

plt.figure()
df.ACC.plot(grid=True)

<Axes: xlabel='Date'>

For starters, the totals look pretty good for both years. Right around the corrected values for the 2019 - 2022 period.

qc.flag_precip_during_tank_flux(fluxprecision=5, accprecision=5, fluxwindow=289, accwindow='1D', extend_ahead=3)

plt.figure()
df['INST'].plot(legend=True, label='tank height')
df.loc[qc.qa_events.diurnal_flux, 'INST'].plot(legend=True, grid=True, label='flux precip', linestyle='', marker='.')

<Axes: xlabel='Date'>

plt.figure()
df['INST'].plot(legend=True, label='tank height')
df.loc[qc.qa_events.diurnal_flux, 'INST'].plot(legend=True, grid=True, label='flux precip', linestyle='', marker='.')

<Axes: xlabel='Date'>

That is only 3 periods to check! That’s encouraging. I can find other periods of fluctuation, but between the measurement averaging the logger is doing and the new sensor, the magnitude of the fluctuations is too tiny to generate any precip. Let’s take a look at what’s flagged.

flag = qaqc.ApplyFlags(df.index, param['precision'])
flag.import_provisional_data(df)
flag.apply_QaRules_flags(qc.qa_events, qc.qa_flags)

flag.prorate_precip_during_tank_flux()

flag.apply_0_val()

trend = median_filter(df['INST'], size=289, mode='nearest')

trend = pd.DataFrame(data=trend, index=df.index, columns=['mean'])

day = pd.to_datetime('1/25/24')
plot_prorate(flag, day, 'VAR02', '5D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

That is a tank clog. The process is to shake the stand pipe to get the float to jump up. Here it clearly wasn’t able to be shaken down very far when it was drained. If it had shaken below the starting value, it would have ID’d it as a drain and not flagged it. But since the float got stuck on the drain as well, it thinks it’s a spiky fluctuation. I’ll manually unflag.

day = pd.to_datetime('9/11/24')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

I’m really not sure what’s happening here. On the plus side, it is behaving as expected and removing the negative drop until the tank comes back up to it’s previous value. However, it is very unclear what is going on in that dip. I can’t find any records or photos from that day. The dip is extremely low, near 0 and then appears to be followed by real rain for over 12 hours…Let’s see if HI15 got rain during the same period.

First let’s see how long the rain lasted for.

day = pd.to_datetime('9/11/24')
plot_prorate(flag, day, 'VAR02', '3D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

h5 = prov.pivot_on_probe(prov.df, 'H15', '02')

end = day + pd.to_timedelta('2D')

plt.figure()
(h5.loc[day:end, 'INST'] - h5.loc[day, 'INST']).plot(grid=True, label='h15 tank', legend=True)


cf = prov.pivot_on_probe(prov.df, 'CEN', '01')
(cf.loc[day:end, 'INST'] - cf.loc[day, 'INST']).plot(grid=True, legend=True, label='cen sa tank')

uf = prov.pivot_on_probe(prov.df, 'UPL', '01')
(uf.loc[day:end, 'INST'] - uf.loc[day, 'INST']).plot(grid=True, legend=True, label='UPL sa tank')

(df.loc[day:end, 'INST'] - df.loc[day, 'INST']).plot(grid=True, label='vara tank', legend=True)

<Axes: xlabel='Date'>

OK, I think that’s real rain. And not only is it real rain, but it started well below our drop. I’m not sure what happened there, but I’ll unflag the dip manually.

I can remove the flag manually, but I can’t remove the prorating… the prorating is applied to data['adj_precip'] and won’t be reverted…

qaqc._load_yaml('../qa_param.yaml')['VAR_02']['manual_flags'][-1]

{'start': '9/11/24 1025',
 'end': '9/11/24 1915',
 'replace_wNAN': False,
 'replace_w0': False,
 'flag': 'None',
 'reset_adj': True,
 'explanation': 'False positive for diurnal fluctuation during unknown tank drop'}

it_happened = qaqc._load_yaml('../qa_param.yaml')['VAR_02']['manual_flags'][-1]

'reset_adj' in it_happened

True

if it_happened['reset_adj']:
    print('tadah!')

tadah!

Ok, this has been added as an option in manual flagging, but has not been added to all manual flags. With the exception of pro-rating, all other flagging simply sets the column SetO or SetNa to True or False. Those columns are added to in each qaqc.QaRules.flag_* call, and in applyying flags, using an |= True, so that they accumulate True values with each method. Prorating is unique in that it actually produces new values for precip that are unique to each timestep.

This new option wipes any prorating by replacing the adj_precip with the original precip data (data['precip']).

# intended to reverse adjustments.
# At time of programming (Sep 2025), prorating is the only method that adjusts precip
# All other methods set a Set0 or SetNa flag.
if 'reset_adj_precip' in it_happened:
    if it_happened['reset_adj_precip']:
        self.data.loc[strt:end, 'adj_precip'] = self.data.loc[strt:end, 'precip']

day = pd.to_datetime('9/12/24')
plot_prorate(flag, day, 'VAR02', '1D', qc.qa_events, trend)

/var/folders/vs/y0_kk_gj2jxcb2z5xvlgv9g80000gq/T/ipykernel_86401/2046169542.py:10: UserWarning: This axis already has a converter set and is updating to a potentially incompatible converter
  axtank.plot(tank_acc.precip, label='raw precip ACC', color='m', linestyle='-.')

That looks OK. The updated prorating function places an E flag on every prorated value. But the fluctuation moments (purple squares) have an E flag and a Set0 flag, which looks like a strikethrough of the E flag. It filtered out the spiky values, and redistributed the total throughout the rest, flagging all the precip that was estimated by prorating as an E-estimate.

Summary

At VARA 2018-2024 there were very few false positives. The main source of false positives were either: a clog cumulative measurement followed by a an inability to drain, or precip that increased the tank level above the level of the fluctuation. The latter often had the correct total precipitation, but because of the flux, the distribution of the precipitation within the day was incorrect. This meant that precip during the flux couldn’t be removed without creating an incorrect total precipitation for the day, so these values were left unflagged.

There were more false negatives. Many small precipitation amounts were not caught and were added as manual flags. The small precip days totalling < 0.83 mm did not account for a significant amount of rain, but do create many more days of rain within the dataset. For this reason, when they were found, these days were flagged manually to remove the data and replace it with an estimate of 0.

Despite these false negatives, with just the auotomated flags, multiple meters of precip were removed from WY 20 and the data for WY’s 2019 and 2020 were restored. The prorating and automated flagging of diurnal flux appear to be working as expected in a predictable manner with a managable number of days, but manually checking the flags remains necessary, especially when sensors are known to have diurnal fluctuations consistently throughout a period.