Is U.Va.'s Campus Climate Survey statistically significant?

Survey designer, statistician weigh in

ns_rotundaconstructionjpappas

On Sept. 21, 2015, the University released the findings of a comprehensive survey intended to measure the effects of sexual assault and violence on Grounds.

John Pappas | Cavalier Daily

On Sept. 21, 2015, the University released the findings of a comprehensive survey intended to measure the effects of sexual assault and violence on Grounds.

The University was one of 27 members of the Association of American Universities to participate in the survey, which was designed and conducted by statistical services company Westat.

This survey, known to students as the Campus Climate Survey, was circulated to all undergraduate and graduate students by the Office of Institutional Assessment and Studies during the 2014-15 academic year.

The primary goals of the survey were “to estimate the prevalence of different forms of nonconsensual sexual contact, harassment, intimate partner violence and stalking, to better understand the attitudes and experiences of students with respect to sexual assault and sexual misconduct, and to gauge students’ awareness of available resources in response to such incidents,” according to a press release.

Comparing survey results

The survey revealed that nearly one in four University women has experienced sexual assault or misconduct by touching since enrolling at the University.

David Cantor, one of the survey’s authors, noted that the results of the University’s survey demonstrate results similar to the aggregate results seen amongst all AAU members that participated in the survey.

“[The results] were certainly in the range of a lot of other universities,” Cantor said. “Nothing particularly stood out to me.”

Although the results of the survey were similar to the averages results of the other 26 universities that participated, the University’s results diverged in several instances. Only 28 percent of students said they believed it was very or extremely likely that University officials would take action against an offender in a case of sexual assault, compared to the AAU average of 45 percent.


Graphic By: Anisha Kumar


University students are also significantly more likely to say sexual assault is a problem on Grounds than are students at other schools. Nearly 39 percent reported sexual assault is very much or extremely problematic, compared to an overall average 20 percent.

When compared to the aggregate AAU survey findings, University students reported more knowledge of resources, policies and procedures surrounding sexual assault, such as the University’s Counseling and Psychological Services and the University’s Title IX coordinator.

Confirming statistical significance

The results of the survey revealed that of the 20,743 undergraduate and graduate students that received the survey, 5,476 students responded while 15,267 students did not. This represents an overall response rate of 26 percent.

Cantor said the overall response rate is lower than Westat would have liked. He partially attributed the lower-than-expected response rate to an overly ambitious timeline that could not be followed.

“We started this process at the beginning of the school year with the intention of finishing within the same calendar year, but that time frame was extremely ambitious,” Cantor said. “We had to send it out at the end of the school year when students are busy with finals and have already completed a lot of surveys at this point.”

Although seemingly low, statisticians such as Karen Kafadar, chair of the statistics department, said the low response rate does not invalidate the findings of the survey.

“Response rates on these kinds of surveys are notoriously low,” Kafadar said. “I wasn’t surprised to see such a low response rate, as it’s very hard to get a high response rate on a survey. You get a better response rate if you actually talk to the person, which becomes a very expensive survey.”

Cantor said if anything, the results of the survey suggest an overestimation of the number of sexual assault incidences on Grounds, as those who have not been victims of sexual assault or violence tend to be less likely to respond to surveys on the subject.

“Our analysis so far suggests that a few of the estimates are a bit too high,” Cantor said. “If there is a problem, it’s that people that have not been victims tend not to respond.”

However, Cantor does not believe this factor to be statistically significant.

Kafadar added that Westat conducted analysis to determine whether or not the people who participated in the survey are systematically different than the people who did not, and then adjusted the results accordingly. In other words, Westat used statistical analysis to project what the results may have been had all 20,743 students who received the survey responded.

Overall, the results of this projection were not found to be significantly different than the raw data actually collected.

In general, graduate and female students were more likely to respond to the survey. However, Cantor said the reason that most students tended not to respond is not because of a demographic characteristic, but a human characteristic.

“Most people don’t respond to surveys simply because they don’t want to do a survey,” Cantor said. “That’s why having a low response rate doesn’t necessarily mean there’s a bias, because most of the non-response tends to be random.”

The University response rate was significantly higher than the AAU response rate of 19.3 percent amongst all universities that participated in the survey. Although there is no evidence, both Cantor and Kafadar said Rolling Stone’s now-discredited article, “A Rape on Campus,” may have brought attention to the issue of sexual assault on Grounds.

Acting on the survey results

Christina Morell, associate vice president for student affairs and University point-person for the survey, said that because this was the first survey of its kind conducted at the University, it was intended to provide administration and the broader community with a baseline of knowledge regarding sexual assault.

“Because it was our first survey of this nature and intended to provide us with baseline data, we had no preconceptions of the results,” Morell said in an email statement.

However, the survey revealed particular areas that need the most improvement, or that fell furthest below the aggregate average of other AAU universities that participated in the survey.

“The results show that there are opportunities for increased education and engagement across our entire community, as well as in certain populations such as female graduate students, students with disabilities and students who identify as LGBTQ,” Morell said.

The findings of the survey are being used to better inform the efforts of programs and initiatives that preceded the administration of the survey, Morell said. These include the implementation of the Policy on Sexual and Gender-based Harassment and Other Forms of Interpersonal Violence, improvements to the Just Report It system, the GreenDot bystander intervention strategy and the mandatory online training modules introduced in August.

Specifically, Morell noted that the results of the survey have proven helpful in assessing which groups in the University community are most vulnerable to sexual assault and violence.

“The campus climate survey data are being used to further inform specific education and program strategies, including opportunities to better support some of our most vulnerable populations,” Morell said.

Additional policy changes on the horizon include subsequent follow-up surveys and the launch of a student committee that will advise the Title IX coordinator.

“We plan to continue to collect data in the spring, not only from a follow-up survey but from student focus groups, to ensure our programs and measures are effective in ensuring a safe environment free of sexual and gender-based harassment and violence and one that reflects our values,” Morell said.

Read this article translated into Chinese here

related stories