Summary of drain quality checks¶
Why are we draining and why does it matter?¶
Key factors of how precipitation is captured and measured¶
We have many types of raingauges at H.J. Andrews. The three main types used are:
Storage gauges: these collect water in a storage tank, and the tank level is manually measured at certain intervals (MS004).
Recording gauges: these create a record of precip as it falls over short intervals. We use two main types (MS001 - 13):
Tipping buckets: these tally the number of full buckets (of known size) of precip as it falls.
Tank gauges: these collect precip in a storage tank and measure the depth of the tank.
One difference between these is that tipping buckets are directly counting the amount of rain in small increments, while tank gauges require precip to be derived by measuring the change in the tank level.
Another difference is that, while storage gauges can hold as much as a year of precip, recording tank gauges at H.J. Andrews can only hold a month or two of precip. This means that they must be drained regularly during the water year.
A final difference is that, while storage gauges simply have before and after measurements, recording gauges are recording all the time, so they can measure a tank that is in an intermediate state. Examples include: a tank that is only partially drained, attached tanks that are un-equalized and pushing water back and forth (shelter rain gauges at UPLO/CENT), and tanks where recharge is added (NOAHIV’s).
Melting frozen precip¶
A small additional note on melting forzen precip: Heated rain gauges quickly raise the temperature of precip above freezing, but unheated rain gauges use anti-freeze (non-toxic propylene glycol) to quickly lower the freezing point of the precip. Both are effective at melting, but anti-freeze is dilluted with accumulation and lost after each drain. Therfore, the unheated rain gauge must be recharged with anti-freeze with each drain.
How do drains affect quality¶
One can imagine that an overly simplistic calculation of precip from tank level would show all of the drains as negative precip. But any ability to derive a positive precip value when the tank value is dropping is ambiguous. Further, where recharge is added we can’t gaurantee that it happens in between the same timesteps as the drain. Let’s create a simple example:
12:00- tank records 425 mm
12:03- someone starts draining
12:05- tank records 30 mm
12:06- someone stops draining. Tank level 15 mm (unrecorded)
12:07- someone adds 20 mm of recharge
12:10- tank records 35 mm
A simple calculation would show 5 mm of precip at 12:10. Which would be wrong. However, if everything started 2.5 minutes earlier, there would be no perceived increase. In which case, any increase at 12:10 would be real. It is not uncommon for it to take 5-10 minutes to empty a tank. And in the case of our unheated gauges, it can be longer, since the fluid is manually pumped out rather than drained via gravity.
And it is not just recharge that can create this phenomenon. As mentioned before, unqueqlized tanks can push water back and forth. The UPLO and CENT shelter rain gauges, have dual tanks (diameters 10.4” and 6.25”) linked at the bottom. The tanks do not drain at the same speed which causes the levels to bounce up and down as they equalize. This could log a low tank value followed by a high tank value.
And at the same time, it can be raining while a tank is drained. And it is not possible to know how much precip was collected without being measured, i.e. fell in the tank and immediately drained out. Some effort is made to avoid draining during heavy rain, but it is not always possible.
So there are 3 problems that impact quality:
Calculated precip during a drop in tank level is ambiguous or impossible to measure
Note, NOAHIV gauges measure the tank in between records and use an algorythm to calculate accumulation. But it cannot be verified.
Tanks in intermediate states can create false perceptions of precip during or immediately after a drain
Precip directly after a drop in tank level may be real or the result of a recorded intermediate state
How big an impact does this have¶
As a second review of provisional data, which has already had checks preformed, we understandably expect additional errors to be relatively contained, and minimized by the work that has already been put into this data. For example, in the work below, many clog events were identified where the precip had already been removed by GCE, and substantial effort was made to leverage the flags already applied by GCE.
However, the amount of precip flagged by the additional QA rules developed below is not insignificant. Just using the test case of CS2MET, the following values were removed because it is very unlikely that they were actual precip.
[1]:
import pandas as pd
import matplotlib.pyplot as plt
# Jupyter magic to make plots display interactive
# must install ipympl (Ipython-matplotlib) and nodejs
from ipywidgets.embed import embed_minimal_html
%matplotlib widget
import sys
sys.path.append("../../")
from post_gce_qc import qaqc
from post_gce_qc import data_transfer
[ ]:
%cd ../
[3]:
# load data
prov = data_transfer.LoadProvisionalData(file_n='../config.yaml')
prov.load_ppt_data(strtyr=2018, endyr=2022)
df = prov.pivot_on_probe(prov.df, 'CS2', '02')
[4]:
# apply rules
#-------------------------
param = qaqc._load_yaml('../qa_param.yaml')['CS2_02']
qc = qaqc.QaRules(df, qa_params=param)
drn_param = param['auto_flag']['flag-drain-recharge']
# FIND DRAINS
qc.set_drain_event(tank_col='INST', event_window=drn_param['drain_event_window'],
neg_delta_threshold=-15)
# calc running average
run_avg, run_std = qc.calc_run_avg_rainfall(ppt_col='TOT', wind=drn_param['ppt_runavg_window'],
nstd=drn_param['drain_event_nstd'])
# flag problems
qc.flag_drains(run_avg, ppt_col='TOT')
qc.flag_recharge(run_std, ppt_col='TOT', max_recharge=drn_param['drain_event_max_recharge'])
[5]:
# apply flags to data
flag = qaqc.ApplyFlags(qc.df_orig.index)
flag.import_provisional_data(qc.df_orig)
flag.apply_QaRules_flags(qc.qa_events, qc.qa_flags)
flag.apply_0_val()
flag.apply_NAN_val()
running = {'mean':run_avg, 'std':run_std}
[6]:
wy = flag.data[['precip', 'adj_precip']].groupby(pd.Grouper(freq='YE-SEP'))
wy.cumsum().plot(grid=True, legend=True)
[6]:
<Axes: xlabel='Date'>
[7]:
wy.sum().diff(axis=1)['adj_precip']
[7]:
2018-09-30 -5.333984
2019-09-30 -61.213989
2020-09-30 -18.287964
2021-09-30 -3.302002
2022-09-30 -27.686035
Freq: YE-SEP, Name: adj_precip, dtype: float[pyarrow]
[8]:
wy.sum().pct_change(axis=1)['adj_precip'] * 100
[8]:
2018-09-30 -0.282598
2019-09-30 -3.032589
2020-09-30 -1.01223
2021-09-30 -0.165921
2022-09-30 -1.836258
Freq: YE-SEP, Name: adj_precip, dtype: float[pyarrow]
And a brief test of UPLO shelter also shows up to an inch of flase precip caught by these rules.
[9]:
upl = prov.pivot_on_probe(prov.df, 'UPL', '02')
[10]:
# apply rules
#-------------------------
param = qaqc._load_yaml('../qa_param.yaml')['UPL_02']
qu = qaqc.QaRules(upl, qa_params=param)
drn_param = param['auto_flag']['flag-drain-recharge']
# find drains
qu.set_drain_event(tank_col='INST', event_window=drn_param['drain_event_window'],
neg_delta_threshold=-15)
# calc running average
run_avg, run_std = qu.calc_run_avg_rainfall(ppt_col='TOT', wind=drn_param['ppt_runavg_window'],
nstd=drn_param['drain_event_nstd'])
# flag problems
qu.flag_drains(run_avg, ppt_col='TOT')
qu.flag_recharge(run_std, ppt_col='TOT', max_recharge=drn_param['drain_event_max_recharge'])
[18]:
# apply flags to data
flu = qaqc.ApplyFlags(qu.df_orig.index)
flu.import_provisional_data(qu.df_orig)
flu.apply_QaRules_flags(qu.qa_events, qu.qa_flags)
flu.apply_0_val()
flu.apply_NAN_val()
running = {'mean':run_avg, 'std':run_std}
[19]:
flu.data[['precip', 'adj_precip']].groupby(pd.Grouper(freq='YE-SEP')).sum().diff(axis=1)['adj_precip']
[19]:
2018-09-30 -24.369873
2019-09-30 -14.060059
2020-09-30 0.0
2021-09-30 0.0
2022-09-30 -10.469971
Freq: YE-SEP, Name: adj_precip, dtype: float[pyarrow]
Quality Checks/Rules¶
To deal with the issues defined above, the following rule sets were created. Final rules are programmed in postgce_qc.qaqc.QaRules. Parameters for each rule may vary between probes and are defined in qa_params.yaml. These rules are applied to provisional data post-GCE.
CS2MET was used as the test case because the NOAHIV gauges has a very sophisticated, commercially tested, on-board algorithm to calculate precipitation from tank levels, is unheated, requiring recharge after every drain, and only holds 15” of precip, so it is drained very frequently. It is also a good test case because the test data (ranging WY 2018-2022) have a strong GCE filter on the second half of the data, but not the first. This means we should catch a lot of issues in the firt few years, and then see a issues drop away once the GCE filter comes into affect.
Identify when a tank is drained (an event!)
set_drain_event: defined by a running tank average greater than tank level, and any moments where the tank has a negative change in depth.
Flag precip measured during a drain
flag_drains: When the tank level is dropping, all precip values are changed to nan unless the precip value is <= running average of precipitation, which is flagged Q.
Flag recharge after a drain
flag_recharge: For a small window after each drain, check that precipitation values are below defined levels.
Window after drain is defined as a tank level that has not dropped since the last measurement, but is still below the running average of tank level.
Criteria:
value > precip running avg +N std deviation of running avg
value > probe max recharge (75th percentile of precip during drain events)
No Flag: no criteria are met
Q: questionable whether value is rainfall or recharge. Only meets one of NA criteria, but does not meet both.
NA: impossible/highly-unlikely value if precip meets both criteria
Examples¶
Precip accumulation on left axis marked with blue ‘x’. Tank value on right axis marked with orange dots.
Archetypal problem¶
This is a classic example of recharge; it is below the provisional threshold check, but is more than 10x the max 15 min precipitation all day, and there is no rain for hours before or after.
[13]:
day = pd.to_datetime('10/9/18')
flag.plot_flagged_day(day, 'CS2-02', auto_qa_event=qc.qa_events, running_vals=running)
[13]:
(<Axes: xlabel='Date'>,
<Axes: title={'center': 'CS2-02 - 2018-10-09 00:00:00'}, xlabel='Date'>)
Rain or recharge¶
Here are some less clear cut examples where these rules try to distinguish precip from recharge. The rules are well tuned and have very few false positives or negatives that have been found.
In this example there was no precip when the tank was dropping, and the next two timesteps have very low precip that is consistent with the rest of the day. There is no reason to flag or change anything.
[14]:
day = pd.to_datetime('1/22/19')
flag.plot_flagged_day(day, 'CS2-02', auto_qa_event=qc.qa_events, running_vals=running)
[14]:
(<Axes: xlabel='Date'>,
<Axes: title={'center': 'CS2-02 - 2019-01-22 00:00:00'}, xlabel='Date'>)
This example is similar, but the first timestep after the tank drop has a slightly high value, so we can’t be confident that it wasn’t a result of recharge.
[15]:
day = pd.to_datetime('2/12/19')
flag.plot_flagged_day(day, 'CS2-02', auto_qa_event=qc.qa_events, running_vals=running)
[15]:
(<Axes: xlabel='Date'>,
<Axes: title={'center': 'CS2-02 - 2019-02-12 00:00:00'}, xlabel='Date'>)
Here there is a very small precip value, but it is while the tank is actively dropping, which spans 2 timesteps in this example. And while it is a small amount, it is the daily max and over half of the daily total. So there is no confidence that the precip value is real.
[16]:
day = pd.to_datetime('10/24/17')
flag.plot_flagged_day(day, 'CS2-02', auto_qa_event=qc.qa_events, running_vals=running)
[16]:
(<Axes: xlabel='Date'>,
<Axes: title={'center': 'CS2-02 - 2017-10-24 00:00:00'}, xlabel='Date'>)
By contrast, in this example, there is a similar amount of precip while the tank is dropping, but it is a relative low for the precip during an 8 hour period. So, while there is never complete confidence in precip while the tank is dropping, it could be real precip.
[17]:
day = pd.to_datetime('1/28/20')
flag.plot_flagged_day(day, 'CS2-02', auto_qa_event=qc.qa_events, running_vals=running)
[17]:
(<Axes: xlabel='Date'>,
<Axes: title={'center': 'CS2-02 - 2020-01-28 00:00:00'}, xlabel='Date'>)
Key take aways¶
In investigating how to assure quality during drains, there were some surprising things found by accident, and some key lessons for working on other potential quality issues.
Filtering by large values alone left inches of false precip in the data
Manual flags are still necessary to unflag times when the rule is over-agressive and deal with the bizarre signal when snow overtops the sensor
When GCE removes INST values, it is very difficult to reconstruct what is happening in the data.
Interpolation between tank values leads to erratic flagging and precip