All patients who had enrolled for treatment during a stipulated five-year period were assigned random numbers and were examined for adequacy of records. (This entire roster included 743 subjects.) For the purposes of this study, subjects were considered satisfactory for inclusion in the final sample if their charts included the following records at both the beginning and the end of treatment.

a. Lateral cephalogram

b. Upper and lower study casts

c. Panoramic x-ray

d. Facial photographs (profile, frontal, and frontal smiling).

The plan was to enter the records for the entire roster in an unbiased manner and collect a subset of approximately 150 subjects with adequate records representing the major diagnostic categories of interest (Angle Class I non-extraction, Angle Class I non-extraction, Angle Class II extraction, and Angle Class II non-extraction.) These categories were chosen as our major focus of interest because these are the most common types of patients seeking orthodontic treatment. The examination in random order of the first 526 charts yielded 169 subjects who met these criteria. The next task was to prepare sets of records that would allow clinician-judges to make blinded and unbiased evaluations of records of different types:

As stated above, the primary intentions of the study were (1) to examine the concordance/discordance

1. among different skilled clinician-judges, and

2. between subjective (i.e., non-quantitative) and objective (i.e., measured)

for typical parameters used by orthodontists in planning treatment and in evaluating treatment outcome.

To this end four matched groups of twelve subjects each were constructed from among the subjects with complete records. Assignments to the groups were made blinded to any access to information on the course or outcome of treatment. Rather, they were based only on the demographic criteria of age, sex, type of malocclusion (Angle Class I or Class II), and assignment by the clinician to extraction or non-extraction treatment. Charts were first sorted by Sex and then (within Sex) into the four categories listed above: Class I extraction, Class I non-extraction, Class II extraction, and Class II non-extraction. Within Sex and Angle Class, subjects were then matched for Age between extraction and non-extraction treatment.

For example, a typical pair might consist of a 14.2 year old Class I non-extraction male and a 14.1 year old Class I extraction male. The pairs were then randomly assigned to four Groups, each of which contained 6 pairs. Thus the final sample consisted of 48 subjects, arranged in four Groups of 12 each. (Note: This assignment process was conducted under the general supervision of Dr. Edward L. Korn, Head, Clinical Trials Evaluation, Biometrics Research Branch, National Cancer Institute.)

1. Non-quantitative Assessments:

Each of the four groups was examined independently in multiple passes called “Tasks” by each of 7 clinical instructors/professors in the graduate teaching program of the Department of Orthodontics, UMDNJ. Each clinician was a University-educated specialist in orthodontics.

In specifying the several “Tasks,” an attempt was made to design the instructions to the judges in such a way as to generate the greatest amount of important information with the minimum expenditure of clinician time. Thus, in general the pattern, with each task, was to ask the judge to identify the four “best” and the four “poorest” cases in each group of twelve, and then to provide up to three brief open-ended “reasons” for each such judgement.

Eight of the ten posited non-quantitative judgments (called “Tasks”) have already been completed. These tasks were performed in the following order. (Note: The original number sequence is used here - this point to be discussed prn.)

Task 1 Evaluation of Post-Treatment Study Casts

Post-treatment study casts for twelve subjects who have received full bonded orthodontic therapy are arrayed on this table. Please examine each set of casts. Then indicate below the I.D. numbers of the four sets of casts representing the “most satisfactory” occlusion and the four representing the “lease satisfactory” occlusion. For each of the “most satisfactory” or “least satisfactory” occlusions, please indicate briefly no more than three reasons leading to your conclusion.

Task 2A Evaluation of Post-Treatment Photographs

Post-treatment photographs for twelve subjects who have received full bonded orthodontic therapy are arrayed on this table. Please examine each set of photographs. Then indicate below the I.D. numbers of the four sets of photographs representing the “most favorable” facial appearance and the four representing the “least favorable” facial appearance. For each of the “most favorable” or “lease favorable” sets of photographs, please indicate briefly no more than three reasons leading to your conclusion.

Task 2B Evaluation of Post-Treatment Photographs

Post-treatment photographs for twelve subjects who have received full bonded orthodontic therapy are arrayed on this table. Six of the patients had extractions and the other six patients did not have extractions. Please examine each set of photographs. Then, circle “X”, if you believe this patient was treated with extractions and “NX” if you believe this patient was treated with non-extraction.

Task 2C Evaluation of Pre-Treatment Photographs

Pre-treatment photographs for twelve subjects who have received full bonded orthodontic therapy are arrayed on this table. Six of the patients have been diagnosed as Class I and the other patients have been diagnosed as Class II. Please examine each set of photographs. Then, for each patient circle the Angle Class you believe this patients is most likely to belong to.

Task 3 Initial Clinical Impression

Please review the pre-treatment records for another group of patients and fill out the table below for each patient.

ID#

Angle Class

Difficulty

Estimated Treatment Time (Yrs)

X or NX

Surgical

CLI,II,orIII+division

and subdivision

(each case)

Where S= Simple

& D= Difficule

(indicate 4 most S cases + 4 most D cases)

(all cases-indicate estimated time with vertical slash)

1 2 3 4

likelihood

(all cases)

Likelihood

(√ for each case over one in 20)

Task 4

Pre- and post-treatment records for twelve subjects who have received full bonded orthodontic therapy are arrayed on this table. We suggest you spend 2-3 minutes examining each set of records. Please examine the records for all twelve subjects. Then indicate below the I.D. numbers of the four cases representing the “greatest improvement” and the four cases representing the “lease improvement”. For each of the “greatest” and “least” improved cases, please indicate briefly no more than three reasons leading to your decision.

Task 4B Evaluation of Treatment Outcomes: Second Pass

Post-treatment cephs, panos, and study casts but no facial photographs are arrayed on this table for twelve subjects who received full bonded orthodontic treatment. Please examine the available records for all twelve subjects. Then record your choice of the four “most satisfactory” and the four “least satisfactory” in the spaces provided below, indicating briefly no more than three reasons for each choice.

Task 4C Evaluation of Treatment Outcomes: Third Pass

Post-treatment cephs, panos, study casts, and facial photographs are arrayed on this table for twelve subjects who received full bonded orthodontic treatment. Please examine the available records for all twelve subjects. Then record your choice of the four “most satisfactory” and the four “least satisfactory” in the spaces provided below, indicating briefly no more than three reasons for each choice.

The two remaining Tasks, which are now in progress, are:

Task 5a Evaluation of root parallelism at the end of treatment (from panoramic x-ray images)

Task 5b Evaluation of initial root angulation (from pre-treatment panoramic x-ray images)

Sample Clinician Scoring Forms for all tasks except 4b and 4c are supplied as Appendix A. As each Task was completed, data from all scoring forms for all judges were tabulated in Excel files.

1. Quantitative Assessments

1.1 Lateral x-ray cephalograms

Pre and post-treatment lateral cephalograms were “traced” independently by each of three third year residents. The tracing process involved the location and marking of approximately 35 hard tissue and soft tissue anatomical landmarks and the recording of three anatomical superimpositional relationships which orthodontists and other users of cephalometric x-ray images employ for the evaluation of lateral cephalograms. All landmarks and techniques for superimposition of tracings were operationally defined –written definitions are available. Locations of all points on each tracing were encoded without bias using an electronic xy coordinate digitizer. The replicate assessments for each landmark were averaged and outliers were discarded automatically using specialized software of our own manufacture. Best estimates of landmark coordinates and data on the reliability of the judges for each estimation of each landmark on each image were retained in a specialized data base, together with demographic information for each subject and each image. Additional software of our own design was used to calculate standard and specialized angular, linear, superimpositional relationships for the pre and post treatment lateral cephalograms for each subject. This software routinely outputs desired values at T1, at T2, and for the changes between T1 and T2 fore each variable of interest. Output is conventionally formatted as input to SAS.

1.2 Panoramic x-ray images

Pre and post-treatment panoramic x-ray images of the teeth and jaws have evaluated using the same techniques and software described above for the lateral cephalograms. Here, because the specialized training of an orthodontic resident presented no advantages, each image was evaluated twice by a single fourth year dental student. Cusps and root apices were located for all teeth in each quadrant, but no superimpositions were performed. The replicate assessments were automatically evaluated for concordance and third tracings were obtained where disparities between the first two estimates were considered excessive. These data have also been loaded into the database and extracted as SAS compatible data sets. (They complement the “subjective” assessments of Tasks 5a and 5b.

1.3 Study Cast Quantification

Data from the pre and post-treatment study casts have not yet been quantified but work in this area has been commenced. Our plan is to use a commercially available system (OrthoCad) to generate 3D digital files from alginate impressions of the original casts. Desired dimensional measurements will then be made on the computer graphic images after suitable tests of method accuracy and precision. An additional advantage of using this system is that the 3D digital files it produces will amenable to interactive manipulation and display on our web site.

2. Written Records Analysis

In addition to the physical records, all progress treatment notes for each patient have been reduced verbatim to electronic (word-processed) form and an initial attempt has been made to encode to these data in rather detailed numerical categories. These numerical-encoded data are available in SAS and some very simple sorts have been done. We consider these data to be an important preliminary data set looking toward the development of a voice-and-screen-implemented database of progress treatment notes in orthodontics.

The outline above contains a brief problem specification and a brief description of all the physical records and data we have generated in connection with this study. It should also be noted that all photographs and x-ray images have been scanned and are available in electronic form. Also, because all landmark locations on the x-ray images are referenced to within-film fiducials, it will be possible, using software already designed but not completely fabricated, to overlay angular, linear, and landmark location information on screen-displayed jpeg images. Some work has been done in our department aiming at the evaluation of these data but we believe you should approach the analysis problem from your own perspective, using us as resource-persons help define the questions of interest. We are particularly interested in analyzing the concordances and discordance’s among the reasons, which clinicians give for their judgements, but we have no good ideas on how to approach this problem.