Testing and Development of QA Methods

• Tank Drain QA Methods
  • Summary of drain quality checks
    • Why are we draining and why does it matter?
      • Key factors of how precipitation is captured and measured
      • Melting frozen precip
      • How do drains affect quality
    • How big an impact does this have
    • Quality Checks/Rules
      • Examples
    • Key take aways
  • Rain during drains
    • GCE flags
    • CS2MET Test case
      • Quick detour to create a timeseries for all NOAHIV columns
      • Tot >= precision
      • Tank below instrument
      • No accum during drain
      • Test Drain and Recharge Rules
• Diurnal Signal Noise QA
  • Summary of Diurnal Signal Noise QC
    • What is Normal Signal Noise?
      • Normal Diurnal Pattern
    • What is Extreme Diurnal Pattern?
    • How Big an Impact Does This Have?
      • Removing Precip from Fluctuations
      • Prorating During Fluctuations
    • Quality Checks/rules
      • Flag precip during tank flux
      • Prorate precip during tank flux
      • Known Limitations
  • False Precip from Diurnal Tank Signal
    • Normal Diurnal Pattern (<1mm)
    • Diurnal Pattern > 1mm
      • Negative Swing
      • Positive Swing
      • State Switch
    • Possible Filters
      • PyTimeTK Signal Decomposition
      • StatsModels Signal Decomposition
      • Filter Midnight to Midnight
      • Wavelet Transform
      • Trend Fitting
    • Integrate Trend with Midnight Filter
  • Testing Across the Data
    • Refactoring
      • Pro-rating data
    • UPLO
    • CENT
  • Checking Diurnal Flux at VARA
    • Site Overview
    • Flag Precip During Tank Flux
    • Impact of Fluctuation Flagging
    • Additional WY’s
    • Summary
• Resolution QA
  • Summary of Resolution Quality Checks
    • Understanding Precipitation Resolution
      • Understanding Digital Resolution
      • How Resolution Impacts Precipitation
      • Incorporating Averaging
      • Resolution of HJA Sensors
      • Looking at Signal Bounce
    • How Big an Impact Does This Have?
      • Daily Reduction in Precip
      • Water Year Reduction in Precip
      • Number of Days Per Year Without Precipitation
    • Quality Checks/Rules
      • Round Precip to Min Increment
  • Precipitation Resolution and Sensor Precision
    • What’s Recorded in HJA Data
    • What is Our Sensor Precision
      • Converting Tank Depth to Precip
      • Converting Sensor Precision to Precip Precision
    • What is Logger Precision
      • Converting Signal to mm
    • Does Precision Match the Noise in Our Data?
      • UPLO SA
      • CEN SH
      • Quick Numbers for Other Sites
      • Observed Noise Summary
    • Choosing an Increment for Precip Data
    • Rounding to Nearest Increment
      • Check Rounding: Floating Point Errors
      • Accumulating What Rounded Off
      • How Big is the Reduction in Precip
      • Looking at Days with Reduction
    • Summary
  • PAT Resolution
    • Sensor Fundamentals
    • Logger Precision
    • Sensor Precision
      • Deriving Precision from Data Bounce
    • Rounding
• Artificial Precipitation QA
  • Summary of Artificial Precipitation Checks
    • How Big an Impact Does This Have?
    • Quality Checks/Rules
      • Flag double delayed precip
      • Flag repeating values
      • Flagging Estimates of empty tanks
  • Finding Artificial Precip
    • False precip with flat tank
      • Compare to raw data
    • Filtering for a flat tank
      • Max == min
      • Max==min w/ a buffer
      • QaRules.flag_duplicate_precip
    • Comparing tank change to precip
      • Rolling sum of tank
      • QaRules.flag_overaccum_precip
      • Smooth tank first
• Clog QA Methods
  • Summary of clog quality checks
    • What is clogging and how can we tell?
      • How a gauge get’s blocked
      • Identifying a clog
      • Unclogging
      • How does this affect the data?
    • How big an impact does this have?
      • Pre-cleaning
      • Clogs
    • Quality Checks/Rules
      • Pre-cleaning
      • Cross Probe Quality Checks
  • Exploring Clogs
    • Let’s get some Data
      • Clear up 10/1 repeats
      • Track down missing values
    • GCE flagging during clogs
      • J flag
      • F flag
      • Do GCE flags ID clog periods
      • Is precip always 0 when M
      • Flags need to propagate
      • Remove duplicate values
      • 9/6/22 ‘EM’ & “0 tank” issue
    • Comparing paired sensors during clogs
      • Precip Minus PairedPrecip
      • Looking for optimal timestep
      • What metric is consistent for entire clog?
  • Capturing clogs from ACC ratio
    • Create clean ACC data
    • Find the clog
    • Fine Tune parameters
      • threshold for below normal ratio
      • running average
    • Check false positives
    • Early season lag
    • WY19 test
      • Weird exceptions
  • Flagging Clogs: Finding U or C
    • Get Some Data structured
    • CENT 02 (Shelter gauge)
      • Fix overflagging issue
      • Undercatch or Cumulative: WY 2019
  • Using multiple probes to catch clogs
    • First Look
      • Pros
      • Cons
      • Strategy
    • Finding Clogs Using CS2MET
      • Setting levels
    • Weighting Clogs From Other Sites
      • How heavily should within site gauges be weighted.
      • Testing U and C flag weights
    • Reverse Relationship to Identify Known CS2MET Clogs
      • Over flagging
      • Delayed Accumulation Could Signal a Clog in Reverse
  • Reverse Clogs- When the Pair has a Clog
    • Distinguishing Between Clog and Delayed Precip in a Pair
      • Finding the Shift in Ratio Slope
      • Multiple Running Averages
      • Ratio Slope Compared to Ratio Dropping
    • Applying to Test Clog Detection
      • Check False Positives
    • Restart by Smoothing the Differential
      • ratio.diff().rolling(window=’16D’)
      • ratio.rolling(window=’8D’).diff().rolling(window=’16D’)
      • Broader Clog Check Finds Early Season Issues
    • Need to Reduce Impact of Low ACC Spikes
      • Reset Min-Accum
    • Final Check for False Positives
      • 11/14/17
      • 10/29/18
      • 11/25/19
      • Nov 2020
    • Do True Positives Still Work?
• Tipping Bucket QA
  • Tipping Bucket Integration and Checks
    • Maximum Rainfall Rate
    • Clog Detection
      • Minimum Ratio
      • Window Precision
      • CEN_01 and Obvious Clogs
      • Shelter Tipping Buckets
• Flag Assignment QA
  • Summary of Flag Quality Checks
    • Accumulating Flags
    • Checking Flags
    • How Common is This in The Data
      • Multiple Flags
      • M Flags Match NA Data
      • Flag E Where Value Set to 0
      • Flagging During Clog Events
  • Assigning Final Flags
    • Check NA has M Flag
    • SetZero Must Have Flag
    • Finding Multiple Flags
    • GCE Flags During Clogs
    • Including Provisional Flagging
      • T Flag
      • E Following R
      • Separate Tank Flags
      • E Flags
      • Q Flag
      • J and R Flags
      • W Flags