Assessment Guidance: Data Acquisition, Monitoring and Management

Just as it is important to develop a project plan, it is essential to develop data quality objectives and a data management plan. Doing so will help project managers think through the data life cycle and determine the type, quantity, and quality of data needed to reach defensible decisions or make credible estimates, and plan for how to maintain and manage the data once it is collected.

The California Water Board’s Surface Water Ambient Monitoring Program (SWAMP) had developed guidance on developing a Quality Assurance Program Plan, based on the EPA’s systematic data quality objective planning process (Steps 1-7 below). The process follows a series of questions that build on each other, which once answered, provide the project manager with sufficient level of detail and comprehensiveness to design an effective monitoring project. The work done in the Assessment Preparation phase will help inform and guide your answers to the following questions.

Step 1. What is the project going to study?

Identify the issue the project will evaluate. See the Planning, Development of Goals, Objectives, and Expected Outcomes, and Assessment Preparation pages for detailed guidance.

Step 2. How are the data going to be used?

Identify specific study questions and define alternative outcomes. The goals for either decision or estimation problems should explain how the data will be used to answer questions and choose among the standard alternatives.

Identify the intended use of the data
Identify the decision(s) to be made from the study
Identify any alternative actions that would result
Identify priorities or activities the project will address
Be able to explain why you are conducting the research in the first place

Step 3. What kind(s) of data and information are needed?

Determine what kinds of measurements and observations will be needed to support making a decision or estimation.

What measurements and/or observations are already available?
What measurements and/or observations will need to be collected?
What assessment thresholds will be used?
What reporting limits (RLs) are needed?
What other information inputs could be included in the assessment (e.g. climate, pesticide use, geology, land use, etc.)?

Step 4. What are the boundaries and limits of the study?

Consider this step to be your reality check. Use it to keep the project realistic. Identify the spatial and temporal scale and restraints of the project. Project planners will also need to consider the physical and financial limits to the study.

Identify the geographical area of the study
Identify the time frame
Identify the sampling frequency
Identify the population parameter to be measured
Identify the practical constraints for the project
- Site access
- Staff availability and capabilities
- Partnerships and collaborations
- Fiscal constraints/budget
- Natural and unnatural variations in weather and land use

Step 5. How will decisions be made?

Begin to design the analytical approach and logic for drawing conclusions or making decisions from findings. For decision problems, the result will be “if/then” statements that link the results with conclusions or future actions. For estimation problems, the result will identify the estimator and estimation procedure.

Develop a decision rule using if/then statements that includes:
- The population parameter of interest
- The action or threshold levels, including the frequency and magnitude of occurrence.
- The scale of the decision
- The alternative actions
Develop the test of the hypothesis, if applicable

Step 6. How much error is acceptable?

Here you will specify the performance and/or acceptance criteria for use in the hypothesis test and/or decision making

For hypothesis testing: identify the probability limits for decision errors
For estimations or other analytic approaches: identify performance criteria (for new data being collected) or acceptance criteria (for existing data being considered for use) for each measurement/observation. This helps control how much error is acceptable and are often expressed in terms of data quality indicators:
- Sensitivity
- Accuracy
- Bias (Native and non-native)
- Precision
- Comparability
- Completeness
Determine limits and consequences of the data not meeting the assigned criteria
Identify levels of verification and validation needed for the project based on these consequences. When should data be rejected?

Step 7. Optimize the sampling design for obtaining the data

Choose a sampling design that meets the project requirements you outlined above and the budget. This will help you assure the data you are going to collect is of good quality and useful.

Step 8. How will the data be governed and managed?

Define how you will store and manage the data.
Where will the data be stored for the long term?
Who will be responsible for maintaining the data in the long term? Who will be the steward of the data?
Does the data set have complete metadata and an accompanying data dictionary? (hint: the answer should be a resounding yes!)
Can some or all of the data be made publicly available?
- If so, how will you make your data available?
  - Pro tip: For California agencies, upload your data to the California Open Data Portal!
- If not, how can others gain access to the data? What is the data request process?
- Learn more about Open Data at the Water Boards by checking out their Open Data Handbook (coming soon)!
At what point will the data be archived or purged?

By going through the above steps, you will have essentially developed your data life cycle. If not, go back through your data management plan and add the missing pieces.