Current Projects - Reproducible Accreditation

Objective

Develop and mature a process for systematically developing the recommendations for accrediting the capabilities of simulation products and services for their intended uses

Approach

  • Draw well-defined techniques from current software and simulation standards and best practices to derive reproducible and defensible recommendations for simulation use from the available accreditation evidence
  • Collect and maintain the traceability of accreditation evidence from citable sources including the packages from prior accreditations, V&V evidence in the form of reports, briefings, spreadsheets and electronic communications, and supplementary evidence from the technical literature
  • Apply claims-argument-evidence argumentation to construct accreditation recommendations from the citable evidence
  • Estimate the uncertainties in the accreditation evidence and use those to estimate the uncertainties in the accreditation recommendations
  • Document the accreditation process and results to the quality of a scientific report or publishable paper
  • Mature this process over the course of our work on quantitative V&V and risk-based accreditation

Results

  • Recently applied the mature process to the accreditation of two different simulations of complex aerosol flows and learned several lessons from these applications

Conclusions

Lessons learned:

  • Claims-argument-evidence argumentation and uncertainty quantification techniques can successfully develop strong accreditation recommendations that specify the uncertainties of simulation predictions and the uncertainties in the accreditation recommendations
  • Strong accreditation recommendations tend to depend only weakly upon the acceptability criteria due to criteria insufficiency and uncertainties
  • Accreditation recommendation quality can be meaningfully measured by the uncertainties in the recommendations
  • Improving process repeatability reduces the uncertainties introduced by the process and its participants (e.g., SMEs) so that accreditation recommendation uncertainty depends primarily upon the uncertainties in the accreditation evidence
  • Supplementary technical research can reduce accreditation evidence uncertainties but project schedule and resources usually limit the amount of supplementary research that can practically be performed
  • Science and risk communication techniques are essential to develop recommendation language that the simulation users and other consumers of the accreditation recommendations unambiguously understand and use to determine how they will apply the simulation and its results
  • All VV&A approaches, including this one, require tailoring to accommodate, often substantial, differences in the availability and quality of the accreditation evidence and the expectations of the user community
  • Strong accreditation recommendations derived rigorously from the accreditation evidence cannot be easily modified to respond to reviewer comments or program management preferences without changing the accreditation evidence, the arguments upon which the recommendations depend, or the statement of intended use