On a criterion-referenced examination such as the NPLEX, the examinee’s knowledge is measured against predetermined criteria. In this case, the criteria are defined by competencies that describe what examinees are expected to know at a specific stage of education (i.e., after biomedical training or after graduation); these competencies are outlined in the NPLEX Study Guides. An examinee is assessed solely on the basis of having met the criteria, (i.e., examinees are not assessed in relation to the performance of other examinees).
The following is a description of the steps NPLEX takes to ensure that exam results reflect the examinee’s true knowledge in relation to the criteria.
Cut Scores (Passing Standards) are Established: On the Saturday after the examinations have been administered and before the answer sheets have been scanned, a group of naturopathic physicians meet to set the initial cut scores. Because every NPLEX examination is a unique combination of old and new items (exam questions), cut scores for the current examinations are independent of the cut scores of previous examinations. NPLEX uses the nationally accepted Angoff method to set these cut scores. In this criterion-referenced method, naturopathic physicians rate the difficulty of each exam item, on the basis of the criteria, by answering the question, “What percentage of minimally competent examinees should be able to answer this item correctly?” The assessments of all raters are averaged to determine a cut score for an exam item, then the cut scores for every item in each general exam area (GEA) are averaged to determine the cut score for that GEA (e.g., Structure/Function, Diagnosis, etc.). A GEA that is judged to have many difficult items will have a lower cut score than a GEA that has many easy items (i.e., for a GEA with difficult items, the examinee will be required to correctly answer a lower percentage of items in order to pass).
Answer Sheets are Scanned: When all the answer sheets have been returned to NPLEX, they are scanned using an optical scanner. If the scanner cannot read any information on the answer sheet, the scanning operator is alerted and the answer sheet is inspected visually to ensure that the scanner has correctly read the examinee’s intended entry. Even though examinees’ names appear on the answer sheets, all scoring and analysis is done without reference to individuals or schools.
Exam Statistics are Reviewed and Item Statistics are Analyzed in Post-Test Analysis (PTA): Exam statistics (i.e., for each examination as a whole) are reviewed to assess the reliability and appropriateness of each examination. The purpose of the item analysis is to review the content of exam items that do not statistically perform as expected. Statistics calculated for each exam item are reviewed by the staff psychometrician. An item is flagged for content review if few examinees answered the item correctly, if approximately equal numbers of examinees chose each of the four responses, if the item did not appear to discriminate well between high scorers and low scorers, and/or if a re-used item did not perform as it had in the past.
Flagged Items are Reviewed by an Exam Chair: A list of flagged items for each specific exam area (SEA, e.g., anatomy, botanical medicine, emergency medicine) is sent to an exam chair. Exam chairs use standard reference sources to verify that for each flagged item the keyed answer is completely correct, that the question is clearly stated, and that none of the other response alternatives are correct. Exam chairs submit their recommendations to the PTA Committee, which makes the final decision regarding the disposition of the item. If an item is deemed valid and appropriate, no key change is made. If an item is deemed ambiguous, or if more than one answer could reasonably be construed as correct, credit may be given for more than one answer. After a decision has been made about every item that has been reviewed, changes are made to the scoring key and all examinations are re-scored to obtain raw scores for each GEA.
Initial Passing Rates are Determined: The computer compares each examinee’s raw GEA scores to GEA cut scores. If all GEAs have been passed, the examinee receives a passing score for comprehensive mastery. The percentage of examinees who pass each examination (BSE, CCSE, and the four Elective Examinations) is reviewed.
Cut Scores Established by the Angoff Method are Reviewed: Using the statistics calculated in the previous steps, the cut scores determined by the Angoff method are reviewed to ensure that they are appropriate. In this review, NPLEX applies a quantitative 8-factor algorithm using three types of information: exam characteristics (e.g., how difficult this examination was compared with previous examinations, etc.), cohort characteristics (e.g., how the current cohort performed on items that had been used before, etc.), and cut score team characteristics (e.g., how stringent the current cut score team was compared to previous cut score teams, etc.). If appropriate, the cut score for one or more GEAs might be lowered.
The NPLEX Board Reviews the Exam Results: The NPLEX Board reviews the exam results of examinees who did not pass, looking at the pattern of performance on the GEAs. If, for example, a large number of examinees failed the same GEA but passed all the others, it could indicate that the cut score for that GEA was set too high, and depending on other factors, the cut score might be lowered. The new (lowered) cut scores (if any) are applied to examinee scores and new passing rates are generated and examined.
Unofficial Pass/Fail Results are Posted to the Examinee’s online Profile: Approximately six (6) weeks after the exam administration, unofficial exam results (Pass/Fail only) are posted to the examinee’s online Profile. Examinees are notified by email when these results can be viewed online. NABNE makes every attempt to ensure that these unofficial exam results posted online are correct; however, it is not possible for NABNE to guarantee that the information posted online is accurate.
Official Exam Results are Sent to Examinees: Official hard copies of NPLEX exam results are sent to examinees by regular mail within the same week. These official (hard) copies provide more detailed information than that provided in the examinee’s online Profile. The report of exam results includes a visual scale that illustrates how the examinee performed relative to the cut score in each general exam area. If the examination was failed, the report will also include a visual scale that illustrates the strength of the examinee’s performance in each specific exam area (SEA) relative to the result achieved in the other SEAs. If the examinee took any or all of the Part II – Clinical Elective Examinations (Acupuncture, Elective Pharmacology, Minor Surgery, and/or Parenteral Medicine), the report will include a section that shows a P (Pass) or F (Fail) designation, along with the converted score(s) achieved on any or all of these examinations.
If an examinee has not completed all requirements for graduation before taking the NPLEX Part II – Clinical Science Examinations, they must complete all requirements for graduation no later than March 31st if they are taking examinations in February and no later than September 30th if they are taking examinations examinations in August. If NABNE does not receive a letter from the ANMP verifying graduation by either deadline, the examinee’s results will be nullified and all fees will be forfeited.
Part II Exam Results are Sent to Regulatory Authorities: In the same week that official results are sent to examinees, reports of exam results for examinees who took the Part II – Clinical Science Examinations are also sent to the single regulatory authority the examinee designated on the Part II – Clinical Science Exam Application.
In situations where decisions are being made about eligibility for licensure or registration, a mastery model of scoring is most commonly used. NPLEX uses a mastery model to measure whether or not the examinee has demonstrated minimal competence in either the biomedical sciences or the clinical science concepts of naturopathic medicine. To receive a passing mark for the Part I – Biomedical Science Examination, the examinee must pass each of two general exam areas (GEAs): Structure/Function and Disease/Dysfunction. To receive a passing mark for the Part II – Core Clinical Science Examination, the examinee must pass all four GEAs: Diagnosis, Materia Medica, Other Modalities, and Medical Interventions. If the examinee has mastered the competencies and passed all GEAs, they receive a P (Pass) for comprehensive mastery. To receive a passing mark on any of the Part II – Elective Examinations (Acupuncture, Elective Pharmacology, Minor Surgery, or Parenteral Medicine), the examinee must demonstrate overall competence (i.e., there are no GEAs).
For purposes of licensure or registration, a regulatory authority is concerned only with whether or not an examination has been passed, not with the magnitude of the examinee’s score. Consequently, for both the Part I – Biomedical Science and Part II – Core Clinical Science Examinations, NPLEX reports a P (Pass) or an F (Fail) for each examinee (i.e., there are no numerical scores).