PAT Resolution

Both H15_02 and GSMACK use Position Analog Transmitter (PAT) to measure tank depth. These are unique sensors with a different set of measurement characteristics than the other sensor types discussed. This section goes into detail breaking down the resloution for these sensor.

import pandas as pd
import matplotlib.pyplot as plt

# Jupyter magic to make plots display interactive
# must install ipympl (Ipython-matplotlib) and nodejs
from ipywidgets.embed import embed_minimal_html
%matplotlib widget

import sys
sys.path.append("../")
from post_gce_qc import qaqc, data_transfer, cross_probe_qc, main

Sensor Fundamentals

The PAT is fundamentally a potentiometer: A voltage is applied to a circuit with a resistor, a wiper makes contact to the resistor at a mid-point, and by measuring the resistance at the mid-point relative to the total resistance signals how far into the range the wiper is.

Logger Precision

This insturment is measured using what’s called a half bridge: the logger supplies and excitation voltage, measures the voltage across part of the ciruit, and divides that by the supplied voltage, providing a ratio as an output. The measured voltage corresponds to the position of the wiper (mid-point contact) on the potentiometer. The full range of the potentiometer is known, but the current position isn’t. By returning a ratio, we can determine how far along in the range we are.

\(\frac{Measured_{mV}}{Supplied_{mV}} = \frac{Position_{mm}}{Range_{mm}}\)

\(\frac{Measured_{mV}}{Supplied_{mV}} * Range_{mm} = Position_{mm}\)

So unlike our other sensors, the logger returns a ratio, which is unitless. With a half bridge and the supplied voltage of 2,500 mV, the logger’s resolution is \(667_{uV}\). So the question is how many bins is the ratio broken into?

\(\frac{Range}{Resolution} = Bins\)

\(\frac{2,500_{mV}}{0.667_{uV}} = 3,748.126 bins\)

Now, to convert back into mm:

\(\frac{1}{Resolution} = \frac{Bins}{Range}\)

\({Resolution} = \frac{Range}{Bins}\)

\(0.203_{\frac{mm}{bin}} = \frac{762_{mm}}{3,749.126 bins}\)

So the logger can break the input signal into chunks of 0.203 mm.

Like before, we need to convert that from tank depth to precip by multiplying by the ratio of \(\frac{TankArea_{cm^{2}}}{OrificeArea_{cm^{2}}}\). Luckily, at HI15 the orifice and the tank have the exact same diameter, but at Mack they are different.

from math import pi
(pi*(12.352/2)**2)/(pi*(13.1875/2)**2)

0.8773030126007954

print(f'Mack logger resolution {0.203 * 0.877303}')

Mack logger resolution 0.178092509

Sensor Precision

Unfortunately, this sensor does not provide any specifications for repeatability, precisoin, or other sensor noise under stable conditions. It only shows accuracy. Since accuracy is generally a fairly stable offset, it doesn’t help quantify the minimum tank movement that signifies precipitation added to the tank. However, since it’s all that’s provided, we’ll take a quick look here.

0.38% of 2.5 ft.

print(f'range in mm = {2.5*12*25.4}')

range in mm = 762.0

print(f'Error = {762 *0.0038} mm')

Error = 2.8956 mm

print(f'Error = {2.5*12 *0.0038} in')

Error = 0.114 in

So the error is quite large, nearly 3 mm and over 0.1 inches. Hard to say how that woulld impact the precip data, exactly.

Let’s look at what the data actually looks like.

Deriving Precision from Data Bounce

By looking at rain free periods, and then measuring the change in tank level, we can assess how much bounce is normally in this data set. Using the LT420/LPRefineMe sensors as a guide, the average observed signal bounce was larger than logger resolution for all but one sensor, implying that the sensor’s precision was adding to the signal noise. However, in all cases, the observed signal noise was below the specifications for sensor precision. Since the data only has one decimal point, when values were rounded to a single decimal point, the logger resolution and observed noise were the same.

Based on that, with the PAT’s logger resolution of 0.2, the expected noise in the data is below 0.25. Let’s see if that pans out.

H15

prov = data_transfer.LoadProvisionalData(strtyr=2019, endyr=2024, file_n='../config_new.yaml', fname_base='MS00413_PPT_L1_5min_')
prov.load_ppt_data()

h15 = prov.pivot_on_probe(prov.df, 'H15','02')
mck = prov.pivot_on_probe(prov.df, 'GSM','02')
cen = prov.pivot_on_probe(prov.df, 'CEN','02')

Check to make sure the data looks reasonable. Compare to CENT as a baseline.

plt.figure()
h15.ACC.plot(grid=True, legend=True, label='HI15')
mck.ACC.plot(grid=True, legend=True, label='Mack')
cen.ACC.plot(grid=True, legend=True, label='CEN')

<Axes: xlabel='Date'>

zeroppt = h15['TOT'] == 0

# `.diff()` of boolean yields True whenever consecutive rows have different boolean values
# `.cumsum()` then assigns a number at the beginning of each group.
ppt_turn_onoff = zeroppt.diff()
number_continuous_grps = ppt_turn_onoff.astype('boolean').cumsum()
# this returns the size of each group of consecutive values
how_many_in_group = zeroppt.groupby(number_continuous_grps).transform('size')
hour_wo_ppt = (how_many_in_group > 12) & zeroppt

tank_change =  h15['INST'].diff()
drains = tank_change < -10
no_rain_no_drain = tank_change[~drains & hour_wo_ppt]

no_rain_no_drain.describe()

count    542427.0
mean      0.00016
std      0.040787
min          -2.5
25%           0.0
50%           0.0
75%           0.0
max      2.709999
Name: INST, dtype: double[pyarrow]

plt.figure()
no_rain_no_drain.plot(grid=True, marker='.', linestyle='')

<Axes: xlabel='Date'>

plt.figure()
h15.INST.plot(grid=True)

<Axes: xlabel='Date'>

That’s some real stable data.The logger resolution of 0.2 seems very reasonable from the graphs, but there are so many 0 values that the standard deviation reads shockingly low. I think we can stick with logger resolution here.

MACK

zeroppt = mck['TOT'] == 0

# `.diff()` of boolean yields True whenever consecutive rows have different boolean values
# `.cumsum()` then assigns a number at the beginning of each group.
ppt_turn_onoff = zeroppt.diff()
number_continuous_grps = ppt_turn_onoff.astype('boolean').cumsum()
# this returns the size of each group of consecutive values
how_many_in_group = zeroppt.groupby(number_continuous_grps).transform('size')
hour_wo_ppt = (how_many_in_group > 12) & zeroppt

tank_change =  mck['INST'].diff()
drains = tank_change < -10
no_rain_no_drain = tank_change[~drains & hour_wo_ppt]

no_rain_no_drain.describe()

count    518783.0
mean     0.000157
std      0.038948
min     -3.110001
25%           0.0
50%           0.0
75%           0.0
max      1.199997
Name: INST, dtype: double[pyarrow]

plt.figure()
no_rain_no_drain.plot(grid=True, marker='.', linestyle='')

<Axes: xlabel='Date'>

plt.figure()
mck.INST.plot(grid=True)

<Axes: xlabel='Date'>

plt.figure()
mck.INST.plot(grid=True)

<Axes: xlabel='Date'>

A very similar story with Mack. While the figure above shows a lot of bounce, a lot of the flat periods don’t show this. The reading appears fairly stable.

Rounding

Based on the observed signal noise, and the known logger resolution, I am comfortable using a resolution value of 0.2 mm for both. The comparably stable signal from the PAT’s seems to lead to much fewer values that need to be rounded. Let’s do a quick test of what size moving window to use to try to scrape any rounded off values together. I suspect a small window will be just fine with these sensors.

plt.figure()
mck.TOT[(mck.TOT<0.2) & (mck.TOT>0)].hist()
plt.title('Mack')

Text(0.5, 1.0, 'Mack')

plt.figure()
h15.TOT[(h15.TOT<0.2) & (h15.TOT>0)].hist()
plt.title('HI15')

Text(0.5, 1.0, 'HI15')

hqa = qaqc.ApplyFlags(h15.index, 0.2)
hqa.import_provisional_data(h15)

hqa.round_precip_to_min_increment(scrape_remainder_window=6)

wy = hqa.data[['precip', 'adj_precip']].groupby(pd.Grouper(freq='YE-SEP')).sum()
wy.adj_precip - wy.precip

2019-09-30   -73.300049
2020-09-30    -67.73999
2021-09-30   -66.309937
2022-09-30   -71.719971
2023-09-30   -62.890015
2024-09-30   -76.309937
2025-09-30          0.0
Freq: YE-SEP, dtype: float[pyarrow]

(wy.adj_precip - wy.precip)/wy.precip

2019-09-30    -0.03855
2020-09-30   -0.041367
2021-09-30   -0.038559
2022-09-30   -0.039636
2023-09-30   -0.039927
2024-09-30   -0.039356
2025-09-30         NaN
Freq: YE-SEP, dtype: float[pyarrow]

Those seem like acceptable numbers in line with our other gauges. Let’s do the same for Mack.

mqa = qaqc.ApplyFlags(mck.index, 0.2)
mqa.import_provisional_data(mck)

mqa.round_precip_to_min_increment(scrape_remainder_window=6)

wy = mqa.data[['precip', 'adj_precip']].groupby(pd.Grouper(freq='YE-SEP')).sum()
wy.adj_precip - wy.precip

2019-09-30    -52.26001
2020-09-30   -42.939941
2021-09-30   -44.880127
2022-09-30   -82.289795
2023-09-30   -43.869873
2024-09-30   -58.519775
2025-09-30          0.0
Freq: YE-SEP, dtype: float[pyarrow]

(wy.adj_precip - wy.precip)/wy.precip

2019-09-30   -0.025783
2020-09-30   -0.021887
2021-09-30   -0.020404
2022-09-30   -0.033161
2023-09-30   -0.023023
2024-09-30   -0.022071
2025-09-30         NaN
Freq: YE-SEP, dtype: float[pyarrow]

drops = hqa.data.adj_precip.resample('1D').sum() < hqa.data.precip.resample('1D').sum()

pd.options.display.min_rows = 30

drops

2018-10-01 00:00:00    False
2018-10-02 00:00:00     True
2018-10-03 00:00:00    False
2018-10-04 00:00:00    False
2018-10-05 00:00:00     True
2018-10-06 00:00:00     True
2018-10-07 00:00:00    False
2018-10-08 00:00:00     True
2018-10-09 00:00:00     True
2018-10-10 00:00:00    False
2018-10-11 00:00:00    False
2018-10-12 00:00:00    False
2018-10-13 00:00:00    False
2018-10-14 00:00:00    False
2018-10-15 00:00:00    False
                       ...
2024-09-17 00:00:00    False
2024-09-18 00:00:00    False
2024-09-19 00:00:00    False
2024-09-20 00:00:00    False
2024-09-21 00:00:00    False
2024-09-22 00:00:00    False
2024-09-23 00:00:00    False
2024-09-24 00:00:00    False
2024-09-25 00:00:00     True
2024-09-26 00:00:00     True
2024-09-27 00:00:00    False
2024-09-28 00:00:00    False
2024-09-29 00:00:00    False
2024-09-30 00:00:00    False
2024-10-01 00:00:00    False
Length: 2193, dtype: bool[pyarrow]

day = pd.to_datetime('10/2/18')
end = day + pd.to_timedelta('1D')

ax1, ax2 = hqa.plot_flagged_day(day, 'H15_02')
hqa.data.loc[day:end, 'adj_precip'].plot(ax=ax1, marker='.', label='rounded precip', color='r', grid=True)

<Axes: xlabel='Date', ylabel='Precip (mm)'>

day = pd.to_datetime('10/5/18')
end = day + pd.to_timedelta('1D')

ax1, ax2 = hqa.plot_flagged_day(day, 'H15_02')
hqa.data.loc[day:end, 'adj_precip'].plot(ax=ax1, marker='.', label='rounded precip', color='r', grid=True)

<Axes: xlabel='Date', ylabel='Precip (mm)'>

day = pd.to_datetime('9/26/24')
end = day + pd.to_timedelta('1D')

ax1, ax2 = hqa.plot_flagged_day(day, 'H15_02')
hqa.data.loc[day:end, 'adj_precip'].plot(ax=ax1, marker='.', label='rounded precip', color='r', grid=True)

<Axes: xlabel='Date', ylabel='Precip (mm)'>

was_rain = hqa.data.precip.resample('1D').sum()
is_rain = hqa.data.adj_precip.resample('1D').sum()

drops = (is_rain < was_rain) & (is_rain == 0)
drops[drops==True]

2018-12-03 00:00:00    True
2019-01-01 00:00:00    True
2019-02-07 00:00:00    True
2019-03-02 00:00:00    True
2019-03-26 00:00:00    True
2019-05-08 00:00:00    True
2019-06-16 00:00:00    True
2019-07-29 00:00:00    True
2019-07-30 00:00:00    True
2019-08-17 00:00:00    True
2019-08-30 00:00:00    True
2019-09-02 00:00:00    True
2019-09-06 00:00:00    True
2019-09-21 00:00:00    True
2019-10-01 00:00:00    True
                       ...
2023-05-23 00:00:00    True
2023-06-29 00:00:00    True
2023-06-30 00:00:00    True
2023-09-05 00:00:00    True
2023-12-05 00:00:00    True
2024-05-11 00:00:00    True
2024-05-12 00:00:00    True
2024-05-26 00:00:00    True
2024-05-27 00:00:00    True
2024-06-11 00:00:00    True
2024-06-14 00:00:00    True
2024-06-28 00:00:00    True
2024-07-10 00:00:00    True
2024-07-28 00:00:00    True
2024-08-18 00:00:00    True
Length: 99, dtype: bool[pyarrow]

drops[drops==True].groupby(pd.Grouper(freq='YE-SEP')).sum()

2019-09-30    14
2020-09-30    23
2021-09-30    12
2022-09-30    25
2023-09-30    14
2024-09-30    11
Freq: YE-SEP, dtype: int64[pyarrow]

So 99 rain days were removed across 5 years of data at HI15. 10 to 25 fewer rain days per year with rounding, averaging about 20 days per year.

day = pd.to_datetime('5/23/23')
end = day + pd.to_timedelta('1D')

ax1, ax2 = hqa.plot_flagged_day(day, 'H15_02')
hqa.data.loc[day:end, 'adj_precip'].plot(ax=ax1, marker='.', label='rounded precip', color='r', grid=True)

<Axes: xlabel='Date', ylabel='Precip (mm)'>

day = pd.to_datetime('12/3/18')
end = day + pd.to_timedelta('1D')

ax1, ax2 = hqa.plot_flagged_day(day, 'H15_02')
hqa.data.loc[day:end, 'adj_precip'].plot(ax=ax1, marker='.', label='rounded precip', color='r', grid=True)

<Axes: xlabel='Date', ylabel='Precip (mm)'>

day = pd.to_datetime('1/1/19')
end = day + pd.to_timedelta('1D')

ax1, ax2 = hqa.plot_flagged_day(day, 'H15_02')
hqa.data.loc[day:end, 'adj_precip'].plot(ax=ax1, marker='.', label='rounded precip', color='r', grid=True)

<Axes: xlabel='Date', ylabel='Precip (mm)'>