Recently, I published an article on item-sort tasks (Found Here). In light of this article, I thought I’d
provide a brief review of item-sort tasks’ logic, method, and statistics.

Item-Sort Task Logic

What are item-sort tasks used for? In the social
sciences, we often use a self-report scale to
measure a construct of interest, such as extraversion or workplace performance. When making a scale, authors often create too many items, called an over-representative item list, to ensure that all aspects of the construct of interest are gauged; however, the over-representative item list is often too long to use in a study. For this
reason, authors must reduce this over-representative item list to create a shorter scale which still measures all aspects of the
construct of interest.

In most cases, an exploratory factor analysis (EFA) is the best method to reduce this over-representative item list. To perform an EFA, a sample provides their
responses to all items from the over-representative item list (i.e. Strongly
Disagree to Strongly Agree, Extremely Unlikely to Extremely Likely), and the EFA determines which items measure the same construct. Items which do not
measure the common construct are removed. Unfortunately, certain factors may prohibit the use of EFA. For instance, researchers and practitioners may not have access to enough participants to perform an EFA.

When an EFA cannot be used, an item-sort task is a comparable replacement. When performing an item-sort task, a sample does not provide their responses to the items in an over-representative item list. Instead, a sample provides their opinion about each item on an over-representative item list. Several previous studies have demonstrated item-sort task results to be comparable to EFA
results. So, if an item-sort task is acceptable, how do we perform one?

Item-Sort Task Method

To perform an item-sort task, you need to clearly define the construct that you want to measure. Then, you need to think of other constructs that are both (a) similar to your construct and (b) should not be included in your measure. For
instance, if I wanted to measure conscientiousness, I probably would not want to include any aspect of intelligence in my measure. Although they both lead to life and work success, they are clearly different constructs. After identifying other
relevant but different constructs, you should clearly define these constructs. Once you have the constructs and their definitions, you are ready to perform an item-sort task.

To make your survey, you need to begin with a list of each construct and their
definitions. Your participants should read these constructs and definitions. Then, you should list all the items from the over-representative item list, and each should be followed by the list of constructs. Participants should be instructed to read each item and indicate the construct that they believe that the item
measures. For instance, if an item reads “I am smart” and the constructs of
interest are conscientiousness and intelligence, the participant would likely choose intelligence. Lastly, you may want to include a free-response blank after each item, so participants can indicate which items may be confusing or have poor wording.

Lastly, you’ll need a sample. The sample may be as small as five participants and there is no limit; however, most authors use sample sizes of 20 in their item-sort tasks. Most studies also use subject-matter experts, but this is not a
requirement for item-sort tasks. Also, with such a small sample, it is likely a good idea to include checks for insufficient motivation. After you have your sample, you should administer the survey

Once you have the survey completed and the sample collected, you are ready to analyze your results.

Item-Sort Task Statistics

After performing the item-sort task, you should have a number of responses for each item. To determine whether an item from the over-representative item list should be retained in the final measure, it is necessary to perform a statistical significance test on each item result. This test can determine whether the item was assigned to the posited construct more so than random chance,
representing an item which likely measures the posited construct.

The statistics used to analyze item-sort tasks vary greatly; however, only two methods are statistically supported. The first method is slightly complicated, and it was invented by Anderson and Gerbling (1991). The second method is much easier, and it was invented by myself (Matt Howard) with Robert Melloy in our
recent publication (2016). The math is provided in our article, but we also provide an easy-to-use table. For each item, you should simply compare the number of assignments to the posited construct to the table below:

For each item, if the number of assignments to the posited construct is equal to or greater than the value provided for the respective sample size, then the item is likely representative of the posited construct and can be retained. Items with a smaller number of assignments should be rejected, as it is likely not representative of the construct of interest. After performing the statistical test on each item, you should have a reduced over-representative item list which resembles an
initial measure of your construct of interest.

Conclusion

While the general theory, method, and statistics behind an item-sort task are presented above, some aspects may still be confusing. If you are still confused, check out my full publication on item-sort tasks (Link Here) and feel free to email me at mch248@psu.edu. Lastly, if you want to cite any of this, please cite my publication on item-sort tasks. All the information provided above is included in the article. The citation is:

Howard, M.C. & Melloy, R. C. (2016). Evaluating item-sort task methods: The presentation of a new statistical significance formula and methodological best practices. The Journal of Business and Psychology, 31(1), 173-186.