NPLEX has an extensive process of reviewing items on the examinations both before and after they have been administered. The following is an explanation of the process that occurs after the exam administration to ensure that results are based on valid items.
When all the answer sheets have been returned to NPLEX, they are scanned via an optical scanner. If the scanner cannot read any information on the answer sheet (personal information or answers) the scanning operator is alerted, the answer sheet is inspected, and corrections are made. Even though examinees’ names appear on the answer sheets, all scoring and analysis is done without reference to individuals or schools.
Statistics are calculated for individual exam items (item analysis) and for each examination as a whole. The examinations and individual items are reviewed in a post-test analysis (PTA). The purpose of the PTA is to review exam items that do not, on the item analysis, perform statistically as expected. Items are evaluated individually on the basis of three criteria: the percentage of examinees who chose the correct response, how well the item discriminated between examinees who had high scores and those who had low scores, and the spread of responses (i.e., the percentage of examinees who chose each response). Items that few examinees answered correctly, items that were poor “discriminators”, and items for which equal numbers of examinees chose three or four responses are flagged for review.
A list of the flagged items is sent to the Exam Chair for review. The Exam Chair uses standard reference sources to verify that there is a correct answer, that the keyed answer is correct, that none of the other response alternatives are correct, and that stem is clearly written. The Exam Chair submits his or her recommendations to the PTA Committee, who makes the final decision regarding the disposition of the item. If an item is deemed ambiguous, or if more than one answer is correct, credit may be given for more than one answer. Alternatively, the item may be deemed valid and appropriate, in which case no key changes are made. After a decision has been made about every item in question, changes are made to the scoring key and all examinations are re-scored. This process is done to ensure that the items on which the examinee’s results are based are appropriate and fair. Due to this extensive post-test analysis process, it takes approximately six weeks to complete the scoring process.
Because the NPLEX is a criterion-referenced examination, each examination has a passing score that is independent of the passing scores of previous or subsequent examinations. The Angoff method (a nationally accepted testing procedure) is used to establish this “cut score”. Naturopathic physicians rate the difficulty of each exam item by answering the question, “What percentage of minimally competent examinees should be able to answer this item correctly?” The assessments of all the raters are averaged to determine a cut score for an exam item. Then the cut scores for every item on the examination are averaged to determine the cut score for the examination. An examination that is judged to be difficult will have a lower cut score than an easy examination (i.e., for a difficult examination, the examinee will be required to correctly answer fewer questions in order to pass).
The cut score for each exam area is subjected to analysis based on a formula that takes into account eight different aspects of the exam statistics (exam characteristics, rater characteristics, and cohort characteristics) in relation to the cut score. Cut scores may be lowered if this analysis suggests that the original cut score was set too high. If a cut score is lowered, more people pass the examination.
The NPLEX Examinations are criterion-referenced. The criterion is the minimal passing score (the cut score) that the examinee must achieve in order to demonstrate that he or she is competent to enter the clinical phase of training (Part I Examination) or to practice safely as a naturopathic physician (Part II Examinations). Scores that exceed the value of the cut score indicate that the examinee has demonstrated competence in (mastered) the content domain, and scores below the criterion indicate that the examinee has failed to demonstrate minimal competence. This “mastery” model of scoring is most commonly used in situations where decisions are being made about eligibility for certification or licensure/registration. The NPLEX Examinations measure whether or not the examinee has demonstrated minimal competence in either the biomedical or clinical concepts of naturopathic medicine. If the candidate has mastered the concepts, he or she will receive a “P” (Pass).
For purposes of licensure/registration, the regulatory authority is only concerned with whether or not the examinee has passed the examination, not with the magnitude of his or her score. Consequently, for both the Part I – Biomedical Science and Part II – Core Clinical Science Examinations, NPLEX reports a “P” (Pass) or “F” (Fail) for each examinee (i.e., there are no numerical scores). To receive a passing mark for the Part I – Biomedical Science Examination, the examinee must pass each of two general areas (Stucture/Function and Disease/Dysfunction). To receive a passing score for the Part II – Core Clinical Science Examination, the examinee must pass all four general areas (Diagnosis, Materia Medica, Other Modalities, and Medical Interventions). Each examinee is given a visual scale that illustrates how he or she performed, relative to the cut score, in each general exam area. If the examinee failed the examination, the report will also include a visual scale that illustrates the strength of the examinee’s performance in each specific exam area relative to his or her performance in the other specific exam areas. If the examinee took either or both of the Part II – Clinical Elective Examinations, the report will also include a section that shows a “P” (Pass) or “F” (Fail) designation, along with the raw score, the cut score, and the converted score(s) he or she achieved.
Approximately seven (7) weeks after the exam administration, NPLEX gives the score reports to NABNE for distribution. Part I – Biomedical Science exam results are sent to the examinee only. Part II – Clinical Science exam results are sent to the examinee and to the single regulatory authority that he or she indicated his or her Part II – Clinical Science Exam Application.