Evaluating the Validity and Reliability of the Gender Equitable Men's Scale Using a Longitudinal Cohort of Adolescent Girls and Young Women in South Africa

"Given the high vulnerability to HIV acquisition for the adolescent population, and the evidence linking gender norms to HIV acquisition and current and future programming designed to modify norms, it is important to understand the measurement properties of the GEMS instrument for this population."

Relationship power dynamics resulting from inequitable gender norms that restrict female agency or promote male risk taking may facilitate the conditions that increase risk for HIV acquisition. The Gender Equitable Men's Scale (GEMS), first developed to study gender norms among men in Brazil, has been implemented in many populations (including men and women) around the world to measure gender equity. In light of the fact that a population's developmental transitions may shape response patterns to the GEMS instrument, this study investigated GEMS' consistent measurement of the same underlying construct within the same population over time - that is, its longitudinal measurement properties - by applying it to a cohort of adolescent girls and young women (AGYW) in rural South Africa from 2011 to 2015.

The HIV Prevention Trials Network (HPTN) 068 was a randomised controlled trial (RCT) set in Mpumalanga Province, South Africa, to study the effect of increasing adolescent girls' school attendance on risk of HIV acquisition. The GEMS instrument was part of the annual survey administered to study participants, but it included only 13 of the 24 items in the original GEMS instrument. (A previous analysis of GEMS found that, in adapting the instrument to the South African context, restricting the entire instrument to 13 items was optimal when measuring gender-equitable beliefs among adult women.)

Through a process detailed in the article, the researchers used item response theory [e.g., Differential Item Functioning (DIF), which measures whether, on average, subgroups of respondents answer items differently (i.e., are more or less likely to endorse an item) according to the defining characteristic of the subgroup] and measurement invariance confirmatory factor analysis (CFA) models to assess the validity and reliability of the GEMS instrument among the 2,533 AGYW who were enrolled in the HPTN 068 cohort at baseline. Cronbach's alpha was used to measure the internal consistency of the 13-item set at each study visit (5 visits in total).

Most items showed substantial shifts in item difficulty over time (either becoming easier or harder over time):

• The following five items became harder to endorse (or agree with) over time: The man decides what type of sex to have (Item 1), A woman who has sex before she marries does not deserve respect (Item 5), Only when a woman has a child is she a real woman (Item 7), The husband should decide to buy the major household items (Item 11), and A man should have the final word about decisions in his home (Item 12).
• The following four items became easier to endorse (or agree with) over time: A real man produces a male child (Item 8), Men are always ready to have sex (Item 2), Women who carry condoms on them are easy (Item 6), and Men need sex more than women do (Item 3).
• The following four items remained relatively unchanged in terms of difficulty to endorse over time: A woman should not initiate sex (Item 4), A woman should obey her husband in all things (Item 13), Changing diapers, giving a bath, and feeding kids are the mother's responsibility (Item 9), and A woman's role is taking care of her home and family (Item 10).
• Item 1 (It is the man who decides what type of sex to have) and Item 10 (A woman's role is taking care of her home and family) remained the easiest to not agree with and the hardest to not agree with, respectively, at baseline and the last visit.

Seven of the 13 items showed evidence of differential functioning based on demographic characteristics or life experiences (p < 0.05) in the DIF analysis. For example, experiencing intimate partner violence (IPV) coincided with the most items showing evidence for DIF (n = 6). One item, A woman who has premarital sex deserves no respect, showed evidence for DIF across all four categories of demographic characteristics and life experiences. For each of these categories, the group that was older or had experienced the life event was more likely to not agree with the gender-inequitable statement. The item A real man produces a male child showed evidence for DIF for all categories except for age. For each of these categories, those who had not experienced the life event were more likely to agree with the gender-inequitable statement.

The analysis suggests that the GEMS instrument shows strong longitudinal invariance - that is, the same underlying construct is measured over time. Participants' standings on the latent gender norms construct can and do change over time (i.e., AGWY are becoming more gender equitable over time), but the latent construct itself is not changing over time. Although the GEMS instrument, as a whole, appears to be reliable over time, the results raise concern for the performance (or validity) of individual items. For instance, Item 5, A woman who has sex before she marries does not deserve respect, was flagged for DIF, meaning that it was easier for the group that was older or who had experienced the life event to disagree with item 5. Many of the items flagged at the first visit for DIF, or item bias, directly related to one's personal experiences.

Thus, the results highlight the importance of evaluating an instrument holistically, as well as the individual items. While the "analyses indicate that the GEMS instrument is a reliable assessment of gender equitable beliefs, results also provide evidence for item bias differentially impacting subgroups. As a direct implication, these results should caution researche[r]s against pulling individual items from an instrument to include in a separate survey and interpreting those individual items as a proxy for what the instrument as a whole is measuring. This practice could result in measurement error if items systematically perform differently based on lived experiences, or other demographic characteristics. With respect to GEMS specifically, the evidence for individual item bias tied to experiences during this transitional stage of the life-course may also warrant a separate set of items that are designed specifically for the adolescent experience..."

In conclusion: "Additional items specific to the adolescent experience are warranted for a more stable assessment of gender equitable beliefs in a population facing shifting norms as they mature."