Validation Procedures

Every nQuery Advisor® sample size table, distribution function table, standard deviation table, and table-specific side table has been tested for computational accuracy across table-specific ranges of parameters, and has been tested for full functionality of editing options and menu items (side tables, plots, statements) for one or more table-specific parameter sets.

View Validation Reports.

Methods for testing are described below. Following these general discussions are individual validation reports for the sample size and distribution function tables. nQuery Advisor® sample size tables are indexed by goals (means, proportions, survival, agreement and regression), by number of groups (one, two, or more than two), by type of analysis (test, confidence interval, equivalence test), and by equal vs unequal n.

Global testing of menu items

All menu items which are not table specific were tested to verify that they functioned appropriately. Each item within each menu was tested with examples for three sample size tables. The tester checked off each item for each table; when any problem was found, a description of the problem was recorded. Retesting took place after any necessary corrections were made.

Computational accuracy testing for a range of parameter values

Every nQuery Advisor® table had one table-specific test problem tested for computational accuracy and for full functionality of editing, side table, and menu options, see next section. For every nQuery Advisor® table, sets of table-specific parameter values were selected to cover a wide range of values for significance level, power, effect size, etc and computations were made for each set of parameter values and checked against other sources. The range of parameter values tested for each table is listed in that table’s validation report. For the two- sample t-test for example, three sets of test problems were created : Set 1) the user entered effect size and power and nQuery Advisor® solved for n, Set 2) the user entered effect size and n and nQuery Advisor® solved for power, Set 3) the user entered power and n and nQuery Advisor® solved for effect size. Printed tables with each of these sets of test values were included in the testing notebook. These sets of solutions were reviewed by Janet Elashoff, who checked for consistency, face validity, and for computational accuracy against other sources. The sources used for checking accuracy for each table are listed in that table’s validation report. The results from other sources were recorded in the bottom section of the printed nQuery Advisor® results tables. In spite of the extensive testing done, only a small subset of the values users might want to enter can be explicitly tested; for some tables reliable alternative sources were available to check only a few of the test problems.

Functionality testing for editing options, side tables, plots, statements

For every sample size table, plus the distribution function, standard deviation, and side tables, the full functionality of all options was verified for one test problem. Results for this problem were checked against one or more of hand calculation, a published table, other sample size software, or a published paper, depending on what alternate sources were available. For this example, tests were made to ensure that (1) the same results were obtained regardless of the order in which values are entered or edited, and (2) nQuery Advisor® components specific to the table (plot, statement, side tables, unequal n version, and guide cards) functioned properly. Note that answers may differ slightly when sample sizes are small depending on whether power is entered and n computed or n is entered and power computed since the integer sample sizes computed are those necessary to obtain power greater than or equal to that requested.

In brief, the directions to the tester for sample size tables and non-specific side-tables were:

• Check for consistency of results for each possible order of entry or editing of row values. For small tables, values were entered in all possible orders. For larger tables, a “minimum” example was defined and all possible orders investigated only for the minimum example. For example, for the two-sample t-test, the “minimum” example required filling in the significance level, one or two-sided, and two of effect size, power, sample size. The “maximum” example is the same numerical example with all rows filled in. For disallowed entry orders, check that the appropriate error messages are seen.
• Obtain and print statements, print the table, obtain and print plots.
• Edit all possible values with auto recalculation off, and with auto recalculation on. Edit row names where applicable.
• Review all Guide cards.
• For unequal n’s tables, reproduce the equal n maximum example (using r = 1).

Table specific side tables (usually obtained from Calculate effect size) were checked at the same time as the main table. The following items were checked for each specific side table:

• Test all buttons on the side table.
• Test that the data are saved appropriately.
• Enter values in all possible orders.
• Edit all possible values.
• Test for the effect of deleting values.
• Test with incorrect values to determine if an appropriate error message is displayed.
• Review Guide cards (in Help system).

Testing was documented in two ways. Each item to be reviewed was checked off for each table. When any problems were found, a description of the problem was recorded with supporting printed table or screen print. In addition, the printed tables generated for specific testing steps were labeled as required and together with printed statements and plots were saved for review. All this documentation was reviewed by Janet Elashoff to ensure that the tables were internally consistent – i.e., the results for a given example did not differ depending on which row was computed, and that plots and statements were correct. Tables were re-tested after any necessary corrections were made.

Validation reports for each table

The validation reports for each table contain a list of parameter values tested and the sources they were tested against. Validation Reports for the table-specific side table and the unequal n’s version of the table appear following the main validation report for the table.