Item Writing Guide

Item Writing Guide


1. Purpose

Thank you for your participation! As a subject matter expert (SME) in your field, you will be helping to create items for an upcoming exam. This exam will help determine whether or not test takers have the necessary knowledge and skills to work in a specified role in your industry.

2. Security and Confidentiality

Because these items might be used on an upcoming exam, it is extremely important that you keep the item completely private and unavailable to view by others, including while working on electronic devices such as cell phones.

3. Overall Process

There are three main steps to creating items. Most experts participate in all of these activities, so please note that you might be asked to also review items.
  1. Write Items: A SME writes an item.
  2. Item Review: A different SME(s) reviews the item based on research-based item writing principles. The item either “passes” the review or is returned to the item writer for revision. Oftentimes an item is required to pass multiple reviews from different SMEs.
  3. Revision/Revise Items: The initial SME revises the item while considering the feedback from the reviewer. (Initial feedback is common, so please be aware that you might be receiving input from reviewers.)

4. Choosing a Topic

From the Authoring Portal, select “Write Items” under your exam name.

Each Certiverse exam is created based on a list of pre-determined topics, along with an estimated number of items that need to be written for each topic. Certiverse also includes descriptions, which provide detailed information about the topics.

To start writing items, you must first choose a topic. Make sure to read the description carefully and understand its goal. It is critical that each item aligns with its respective topic.

Select “Write” to choose your topic and begin writing an item.

5. Item Writing

Certiverse has two different ways for you to write your items:
  1. The authoring wizard, which walks you through the process of writing each part of the item.
  2. AI-assisted item writing, where an AI will suggest an item for your review and revision.

AI-Assisted Item Writing

We’ve already written most of a prompt for an AI engine. All you need to add is the specific topic of your item. Type exactly what you want the item to be about (e.g., “what SQL statement stores data into a database” or “what civil rights law

The AI engine will then suggest three items. We expect the AI to continue to get better, but our current research suggests that over 80% of items will need at least minor revision, so you should review the suggestions and pick one that is a good start. The AI is good (not perfect) at making up well-formed sentences that are related to your topic, but you should check the correctness of the answer(s) marked as correct (the “key(s)”) and that the distractors are truly incorrect. You may need to select the best key(s) and then edit the items.

You can reject an item and it will be replaced (up to 10 tries). You can also “Try Again” and revise your specific item topic.

When you accept an item, it will be loaded into the Authoring Wizard for your review and revision.

Tip: Make sure you select the answer(s) that is/are the best key. That way, those answers will be loaded into the “Correct Response(s)” field(s) of the authoring wizard.

Authoring Wizard 

The authoring wizard walks you through writing each part of an item. You’ll also use this interface to edit AI-generated content and to review your items. (The authoring wizard can also be switched the the standard item writing page – select “Standard” to switch views. It’s recommended that you use the wizard.)

The exam may contain one or a variety of the following item types:
  1. Multiple Choice - Single Correct Response - Two Options
  2. a question
  3. one correct answer
  4. one incorrect answer
  1. Multiple Choice - Single Correct Response - Three Options
  2. a question
  3. one correct answer
  4. two incorrect answers
  1. Multiple Choice - Single Correct Response - Four Options
  2. a question
  3. one correct answer
  4. three incorrect answers
  1. Multiple Choice - Single Correct Response - Five Options
  2. a question
  3. one correct answer 
  4. four incorrect answers
  1. Multiple Correct Responses - Dichotomous Scoring - Four Options, Two Correct, 2 Incorrect
  2. a question
  3. two correct answers
  4. two incorrect answers
  1. Multiple Correct Responses - Dichotomous Scoring - Four Options, Three Correct, 1 Incorrect
  2. a question
  3. three correct answers
  4. one incorrect answers
  1. Multiple Correct Responses - Dichotomous Scoring - Five Options, Two Correct, 3 Incorrect
  2. a question
  3. two correct answers
  4. three incorrect answers
  1. Multiple Correct Responses - Dichotomous Scoring - Six Options, Two Correct, 4 Incorrect
  2. a question
  3. two correct answers
  4. four incorrect answers
  1. Multiple Correct Responses - Dichotomous Scoring - Six Options, Three Correct, 3 Incorrect
  2. a question
  3. three correct answers
  4. three incorrect answers
  1. Multiple Correct Responses - Dichotomous Scoring - Seven Options, Two Correct, 5 Incorrect
  2. a question
  3. two correct answers
  4. five incorrect answers
  1. Multiple Correct Responses - Dichotomous Scoring - Seven Options, Three Correct, 4 Incorrect
  2. a question
  3. three correct answers
  4. four incorrect answers
  1. Multiple Correct Responses - Dichotomous Scoring - Eight Options, Two Correct, 6 Incorrect
  2. a question
  3. two correct answers
  4. six incorrect answers
  1. Multiple Correct Responses - Dichotomous Scoring - Eight Options, Three Correct, 5 Incorrect
  2. a question
  3. three correct answers
  4. five incorrect answers
  1. Multiple Correct Responses - Dichotomous Scoring - Eight Options, Four Correct, 4 Incorrect
  2. a question
  3. four correct answers
  4. four incorrect answers
Additionally, each item requires:
  1. a rationale for each correct and incorrect answer. The rationale helps the reviewer understand why the correct answer is truly correct and the incorrect answers are truly incorrect.
  2. your estimation of the item difficulty; by asking what percentage of test takers will answer it correctly. Your response is based on your own evaluation of the item.
  3. a cited reference, which includes support for the correct answer and rationale. These references help avoid opinion-based items and will be used by reviewers to verify the rationale.

Writing Principles for the Item

Writing principles are research-based guidelines authors should follow to help create high-quality items that are more likely to pass the review process. Please ensure that each item:
  1. focuses on significant, non-trivial information related to the topic.
  2. is clear, grammatically correct, and free from unnecessary words or details.
  3. possesses only one meaning (cannot be interpreted multiple ways).
  4. does not include “trickery” intended to misdirect from the correct answer.
  5. is inoffensive and free from bias (age, ethnicity, gender, disabilities, etc.).
  6. contains all the information needed to determine the correct answer and does not require the test taker to make assumptions.
  7. avoids negative phrasing (see Example A).
Items should focus on higher-order thinking, where test takers apply their knowledge to situations. Items focused on lower-order thinking, such as simple recall of facts, are less useful (see Example B).

Writing Principles for the Correct and Incorrect Answers

In addition to focusing on the question, it is also important to write high-quality answers. Please ensure that the correct and incorrect answers:
  1. have a similar format and are of similar length to ensure that the correct answer does not stand out (see Example C).
  2. are independent in content and do not “give away” the correct answer (see Example D).
  3. avoid specific determiners such as “never,” “always,” or “only” to make them false (see Example E).
  4. avoid complex multiple-choice items with answers such as: “all of the above,” “none of the above,” “A and D,” etc.

Good incorrect answers include common misconceptions and words that “sound official.” They should help distinguish between test takers who have and have not mastered the topic.

6. Examples

Note: Correct answer is in bold font.

EXAMPLE
INCORRECT
CORRECT
A

(Item avoids negative phrasing.)
With what condition is the fluid imbalance known as edema NOT commonly associated?

a) Viral infections.
b) Congestive heart failure.
c) Allergic reactions.
d) Protein deficiency.
With what condition is the fluid imbalance known as edema commonly associated?

a) Congestive heart failure.
b) Viral infections.
c) Multiple Sclerosis.
d) Asthma.
B

(Item focus on higher- order thinking.)
Who was the second President of the United States?

a) John Adams.
b) Thomas Jefferson.
c) James Madison.
d) James Monroe.
Historians characterize the 22nd Amendment to the U.S. Constitution as a reaction to the presidency of which person?

a) Franklin D. Roosevelt.
b) Herbert Hoover.
c) Calvin Coolidge.
d) Warren G. Harding.
C

(Correct and incorrect answers have similar format and length.)
Which of the following is the best indication of high morale in a supervisor’s unit?

a) The employees are willing to give first priority to attaining group objectives, subordinating any personal desires they may have.
b) The employees are rarely required to work overtime.
c) The supervisor enjoys staying late to plan the next day.
d) The unit gives expensive birthday presents to each other.
Which of the following is the best indication of high morale in a supervisor’s unit?

a) The employees willingly give first priority to group objectives.
b) The employees are rarely required to work overtime.
c) The supervisor enjoys staying late to plan for the next day.
d) The unit members give expensive birthday presents to each other.
D

(Answers are independent and do not “give away” the correct answer.)
When conducting library research in education, which of the following is the best source to use for identifying pertinent journal articles?

a) Current Index to Journals in Education.
b) A Guide to Sources of Education Information.
c) Resources in Education.
d) The International Encyclopedia of Education.
When conducting library research in education, which of the following is the best source to use for identifying pertinent articles?

a) Current Index to Journals in Education.
b) A Guide to Sources of Education Information.
c) Resources in Education.
d) The International Encyclopedia of Education.
E

Avoid specific determiners (e.g., never, always, only.)
What should you do to avoid infection after receiving a puncture wound to the hand?

a) Ensure that no foreign object has been left in the wound.
b) Always go to the immunization center to receive a tetanus shot.
c) Be treated with an antibiotic only if the wound is painful.
d) Never wipe the wound with alcohol unless it is still bleeding.
What should you do to avoid infection after receiving a puncture wound to the hand?

a) Ensure that no foreign object has been left in the wound.
b) Go to the immunization center to receive a tetanus shot.
c) Be treated with an antibiotic when the wound is painful.
d) Wipe the wound with alcohol unless it is bleeding.

7. Contact Us

If you have any questions or need additional assistance, please contact us by either emailing support@certiverse.com or by submitting a ticket from this article.

8. References

Burton, S. J., Sudweeks, R. R., Merrill, P. F., Wood, B. 1991. How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty. Brigham Young University Testing Services and The Department of Instructional Science. https://testing.byu.edu/handbooks/betteritems.pdf

Haladyna, T. M. & Downing S. M. (1989). A taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 1, 37-50. https://doi.org/10.1207/s15324818ame0201_3

Rush, B. R., Rankin, D. C., & White, B. J. (2016). The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Medical Education, 16, 1-10. Article 250. https://doi.org/10.1186/s12909-016-0773-3



Example items are adapted, with permission, from the referenced Burton, Sudweeks, Merrill and Wood guide (1991). Example C created by Certiverse 2020.


    • Related Articles

    • Item Review Guide

      1. Purpose Thank you for your participation! As a subject matter expert (SME) in your field, you will be reviewing an item for an upcoming exam. This exam will help determine whether or not test takers have the necessary knowledge and skills to work ...
    • Item Review Feedback Guide

      1. Purpose Thank you for responding to feedback regarding an item you wrote. Your item is currently in the review process. A reviewer has reviewed your item and provided feedback for you to consider. Please read the feedback and edit your item as ...
    • Job Task Analysis - Tasks - Guide

      1. Purpose Thank you for participating in this critical step of Job Task Analysis. You were chosen as a Subject Matter Expert for your deep knowledge of this job/role. You will be writing task statements to describe what people in this role do “on ...
    • Certiverse Subject Matter Expert Terms

      Terms last updated on September 12, 2024 pValue, Inc. dba Certiverse, a Delaware corporation (the “Company”) reserves the right, in its sole and absolute discretion, to modify, update and/or make changes to these Terms (as may be amended from time to ...