Ad Hoc Committee - Examining the Efficacy of the GRE

Ad Hoc Committee -

Examining the Efficacy of the GRE

Response to Charge S-2203

August 22, 2022

Charge Number: S-2203

Title- Examining the Efficacy of the GRE

Description- Examine the effectiveness of the Graduate Record Examination (GRE) and its impact

on Rutgers University graduate programs. Investigate other peer aspirants.

Abstract

The committee's overriding concerns are the cost of taking the GRE exam as well as how it creates

barriers for underrepresented groups and often fails to predict graduate school completion.

Rutgers Graduate Admissions on each of the campuses are allowing for the GRE to be optional in

many of their program offerings. In response to this recent charge, an Ad Hoc Committee was

created to examine the current practices among leading national institutions, and to review the

formal literature review that examines the correlation between GRE scores and success in

graduate school performance and retention.

Background

An Ad Hoc Committee was convened to explore the current research literature examining the

Graduate Record Examination (GRE) as a data point in securing admissions to a graduate school

program for masters' programs and PhD programs, primarily in the STEM field. The LSAT and MCAT

examinations were excluded for the purposes of addressing this charge. Ashley Bernstein, a PhD

candidate in Chemistry and a former senator, submitted this Charge with the hope of Rutgers

examining its reliance on the GRE tests for entry into STEM PhD degree programs.

Holistic Review Process is a mission-aligned admissions or selection processes that takes into

consideration applicants’ experience, attributes, and academic metrics as well as the value an

applicant would contribute to learning, practice, and teaching. The Holistic Review process

involves the review of everything a candidate has submitted, understanding of the presence and

impact of unconscious biases, and deliberate awareness of inequities in access and opportunity.

Extracurricular activities, the rigor of college level courses, letters of recommendation,

demonstrated interest, and college interviews may be important factors in a Holistic Review

process. Holistic Review is utilized as an alternative review admissions process when either the

GRE is optional or when GRE scores fall below benchmarked mean ranges for a given discipline or

program. Holistic Review is currently in use at Rutgers University because of a reduced availability

of GRE scores for applicants. When faced with the challenges of reduced data sources, Holistic

Review has been viewed as a component solution to provide alternative data points to review the

quality of applicants.

The GRE is a standardized test that provides substantial normalized data on academic

performance. The GRE and other standardized examinations, like the LSAT and the MCAT, have

been used as an assessment of an applicant’s cognitive abilities and skill sets in preparation for

graduate study. Thus, admissions committees have historically relied heavily on the use of GRE

score to select students because students with higher scores were deemed more likely to succeed

(Kuncel & Hezlett, 2007; Kuncel, Hezlet, & Ones, 2001; Kuncel, Wee, Searfin, & Hezlett, 2010).

There are, however, problems with utilizing the GRE to select the highest caliber students in

admissions, specifically related to impeding enrollment of a diverse student body (Durka, 1999;

Pruitt, 1998; & Toyama, 1999) The GRE quantitative portion correlates positively with male

gender, Caucasian, and Asian American ethnicity. Specifically, women perform 80 points below

average, while African Americans perform 200 points below the mean (ETS, 2014); (2015a) & 2015

b). There are problems with utilizing the GRE to predict success. There is no correlation between

persistence rates and women who score well. Men scoring in the higher 25% in the GRE

quantitative portion were also more likely to leave school without a degree than those who scored

in the lowest 25% (Halford, 2019).

Rutgers’ utilization of GRE scores as an important factor in graduate admissions appears

problematic because of its biases against female and underrepresented minority (URM) students

and its questionable prediction of success. The committee does not support eliminating the use

of the GRE, nor is this a call to admit academically unqualified students in favor of a more diverse

student body in graduate programs. The committee does seek to facilitate development and use

of alternative pathways in graduate admission processes that involve innovative, augmented

processes that utilize proven success markers and program completion with a diminished reliance

on GRE data points.

Nationwide, Graduate Schools have been working to expand program access to a more diverse

group of applicants. However, the dependency on GRE quantitative scores remains steadfast. The

committee reviewed several journal publications (Wilson, M. Odem, M., Walters, T, Depass, A., &

Bean, A. (2019); Miller & Stassum, (2014); Cantwell, Canche, M.; & Sutton, F., (2010); Kuncel, N.R.,

Ones, D. S. A, (2001) & MacLachlan, A.J., (2017).

Discussion and Considerations

The committee utilized a two-prong approach to explore this charge. First, the committee

conducted a detailed review of admissions-related literature on the GRE within the past 10 years.

Next, key Graduate School staff representatives from each campus were invited to outline their

admissions-related processes and, when applicable, the Holistic Review processes instituted

during the 2020-21 pandemic year when GRE testing became optional.

Invited guests included Richard Welsh, Jennifer Soyka, & Julianne Apostolopoulos from the Office

of General Counsel, Vice Chancellor Courtney McAnuff, and Assistant Chancellor Marco Dinovelli,

Dean Taja-Nia Henderson, Newark, & Dean Henrik Pedersen, New Brunswick.

Both Dr. McAnuff and Dinovelli outlined a well-defined holistic review practice conducted at

Rutgers University Undergraduate admissions since the pandemic Spring 2020 semester. It should

be noted this was the inception of a SAT and GRE optional phase because of students’ inability to

complete testing during New Jersey state test facility closures. The selection of students was and

is still based on specific individualistic criteria when test scores when submitted were slightly

below the mean range. The Holistic review criteria included a student’s academic record, letters

of recommendation, personal qualities involving leadership experiences, enrollment in a pre-

college program, research experience, military experience, coupled with any socioeconomic or

environmental factors that the student experienced during their education and upbringing.

Additionally, information regarding the level of parents’ college achievement and household

income provided the admission committee with a gauge for students drive to succeed, learn and

inhabit a sense of grit. A rubric was developed and annually updated for this team to score as

much as possible to address some implicit bias. It is understood that substituting a GRE score for

other viable options within a holistic review will not eradicate bias entirely. It was suggested the

university train as many admissions counselors and volunteer staff in this methodology for the

holistic review process.

Dean Taja-Nia Henderson and Dean Henrik Pedersen have outlined similar approaches for specific

programs utilizing a holistic review process and have additionally offered webinar training sessions

to faculty to enlist their support and understanding of this process. While many programs have

agreed to this methodology, several PhD program directors are maintaining their steadfast

support of the GRE quantitative scores as degree completion indicators for the PHD. Dean

Henderson posited more frequent and timely data from OIRAP to uphold this claim. The

committee members argue the GRE scores simply do not indicate how successful a student will be

in their programs when 50 percent leave halfway through their degree pathways.

Literature suggests statistically significant increases in diversity within doctoral programs when

holistic review is used (Kent, J.D., & McCarthy, M.T., (2016); Okahana, H., Zhou, E. (2017);

Paceheco, W.I.,Porter, J., & Appleyard, C.B. (2015); Posselt, J.R., (2016); Scott, L.D., & Zerwic, J.

(2015); & Wilson, M.B., Depass, A, (2018). The committee discussed adding a prerecorded

interview submission from a standardized list of questions. For example, “Spark Hire” is currently

in use at specific Rutgers programs with some positive results in its ease of use in conducting

interviews remotely, eliminating interview scheduling challenges, and screening graduate

applicants in a time sensitive manner.

Dr. Eduardo Molina, VP of OIRAP attended a scheduled meeting with our committee on September

20, 2022. The discussion points included:

• Identifiable Success markers as defined by each department.

• Tracking data that is meaningful and useful

• Holistic Review along coupled with a series of student support resources that aligns with

student retention and degree completion

• Develop customizable dashboards that have department identified success markers

• Incorporate quantitative and qualitative data points

University of Michigan/Rackham Graduate School, Dean Michael Solomon attended one of our

scheduled meetings on October 3, 2022 at 9:30 am to explain the multitiered process in removing

GRE from its admissions process through several faculty open forums. The University of Michigan’s

Rackham graduate school had decided to stop using the GRE in any of its internal decisions about

fellowships, program review, program quality, etc. It left the choice about using the GRE with each

individual program. It also hosted numerous workshops in which it reviewed the evidence about

GRE efficacy and its potential impact on access to doctoral education. It is noted that 10% of the

U. of Michigan’s graduate programs voted against removing the GRE. In these years, many

programs on their own stopped using the GRE. The graduate school asked all programs to stop

using the GRE during the pandemic, and most did. Some of the graduate programs’ chairs reported

positive admissions outcomes in this period. It was in this context that the proposal was made to

the established Executive Board. The U. of Michigan instituted an Executive Board of elected

faculty members for a three-year term to conduct ongoing programmatic admission reviews. All

Rackham doctoral programs use a common application; they can add a supplemental page that

requests department specific information. Within the bounds of the law, all programs then

conduct their admissions processes independently; however, programs are accountable to their

admissions practices considering the goals of their programs and this is a discussion point in U. of

M. Rackham Program Review. The action of the Executive Board here was to agree to adopt a

common approach for doctoral programs to not use the GRE. Finally, U. of M. Rackham reviews

each graduate program on a five-year cycle. The review is accomplished by a team of Rackham

associate deans and staff. Instead of formally reviewing admissions practices we offer support

through holistic admissions workshops and consultation services.

Summary and Synthesis

The issues discussed in the biweekly sessions throughout the summer months were as follows:

-Predictive validity of GRE and its utility in graduate admissions. Removing barriers for

underrepresented students without jeopardizing the academic caliber of applicants.

-GRE preparatory courses incur a high cost to many impoverished students.

-Compare and contrast traditional quantitative measures, such as GRE and grade point average

(GPA), with Holistic Review.

-Holistic Review as a practice tends to minimize dependence on one specific data point, for

example the GRE score, and signal other areas of strengths in the- students’ application.

Recommendations

Be it resolved, the University Senate recommends

1. Graduate school deans shall advise graduate program directors on the strengths and

weaknesses of the GRE, as well as on the strengths of holistic review, before the end of the

Spring 2023 semester and on an annual basis thereafter.

2. Graduate admission committees need to decide on whether to require, eliminate, or

maintain the GRE as optional for the admissions process for the incoming class of 2024-

2025 AY.

3. Programs that decide to use the GRE must explicitly define how and why the GRE is used,

which candidates will be asked to take the GRE, and must make this information readily

available to all candidates. Moreover, this needs to be part of a holistic review process.

4. If a program chooses to use the GRE, the University should provide financial support,

such as free or low-cost GRE preparation materials, to address financial inequities and

reduce barriers.

5. In consultation with the Office of Institutional Research and Academic Planning (OIRAP),

each graduate program shall develop reporting systems on graduate student outcomes.

Respectfully submitted,

Ad Hoc Committee Chairs

Richard Dool, New Brunswick SC&I-Faculty

Adam Kustka, School of Arts and Sciences-Newark Faculty

Suja Patel, Newark-Staff

Lucille Foster, Newark RBS, Vice Chair University Senate

Ad Hoc Committee Members

Gloria Bachmann, Faculty, Robert Wood Johnson Medical School

Joseph T. Barbarese, Faculty, Camden, English Department

Ashley Bernstein, Graduate Student, former SGS Senator, New Brunswick

Robert Boikess, Executive Committee Member, Faculty-School of Arts and Sciences

Alison Clarke, RBHS-New Brunswick Staff

Jon Oliver, Immediate Past Chair-University Senate, Staff New Brunswick

Adrienne Simonds, Chair-University Senate, Faculty-School of Health Professions

Jose Torres, PTL Camden Faculty

Michael Van Stine, Graduate Student Camden

References

Antonio, A. L. (2002). Faculty of color reconsidered: Reassessing contributions to scholarship. Journal of Higher

Education, 73, 582–602.

Atwood, K. L., Manago, A. M., & Rogers, R. F. (2011). GRE requirements and student perceptions of fictitious

clinical psychology graduate programs Psychological Reports, 108(2), 375–378. doi: 10.2466/07.28.PR0.108

.2.375-378.

Becher, T, & Trowler, P. R. (Eds.). (2001). Academic tribes and territories: Intellectual enquiry and the culture

of disciplines (2nd ed.). Buckingham: Society for Research into Higher Education & Open University Press.

Cantwell, B., Canche, M., Milem, J., & Sutton, F. (2010). Do the data support the discourse? Assessing holistic

review as an admissions process to promote diversity at a US medical school. In annual meeting of the

Association for the Study of Higher Education, Indianapolis.

Clark, B. R. (1989). The academic life: Small worlds, different worlds. Educational Researcher, 18(5), 4–8. doi:

10.2307/1176126.

Cleary, T. A. (1965). Test bias: Prediction of grades of Blacks and white students in integrated colleges. Journal

of Educational measurement, 5(2), 115-124.

Durka, J.T. (1999). Beyond inclusion: Multicultural perspectives in the service of transformation. The GRE,

FAME Report Series, Vol 3, Princeton, NJ, ETS.

Durning, S. J., Dong, T., Hemmer, P. A., Gilliland, W. R., Cruess, D. F., Boulet,J. R., & Pangaro, L. N. (2015). Are

commonly used premedical school or medical school measures associated with board certification? Military

Medicine, 180(suppl 4), 18–23. doi: 10.7205/MILMED-D-14-00569.

Educational Testing Services (ETS). (2011). Guidelines for the Use of GRE Scores. Princeton, NJ. Retrieved June

25, 2018, from www.ets.org/gre/

institutions/scores.

ETS. (2014). A snapshot of the individuals who took the GRE revised general test. Princeton, NJ. Retrieved June

25, 2018, from www.ets.org/s/gre/

pdf/snapshot_test_taker_data_2014.pdf.

ETS. (2015a). Graduate Record Examination: Guide to the use of scores,

2015–2016. Princeton, NJ. Retrieved June 25, 2018, from www.ets.org/s/gre/pdf/gre_guide.pdf.

ETS. (2015b). Identify the best applicants for your next incoming class: Avoid these five common mistakes

when using GRE scores. Retrieved June 25,2018, from www.ets.org/gre/bestpractices.

Gibbs, K. D., Basson, J., Xierali, I. M., & Broniatowski, D. A. (2016). Decoupling of the minority PhD talent pool

and assistant professor hiring in medical

school basic science departments in the US. Elife, 5, e21393. doi:10.7554/eLife.21393

Gibbs, K. D., McGready, J., Bennett, J. C., & Griffin, K. (2014). Biomedical science Ph.D. career interest patterns

by race/ethnicity and gender. PLoS

ONE, 9(12), e114736. doi: 10.1371/journal.pone.0114736.

Grossbach, A., & Kuncel, N. R. (2011). The predictive validity of nursing admission measures for performance

on the National Council Licensure Examination: A meta-analysis. Journal of Professional Nursing, 27(2), 4–128.

doi: 10.1016/j.profnurs.2010.09.010.

Heggeness, M. L., Evans, L., Pohlhaus, J. R., & Mills, S. L. (2016). Measuring diversity of the National Institutes

of Health-funded workforce. Academic Medicine, 91(8), 1164–1172. doi: 10.1097/ACM.0000000000001209.

Jacobs, J. A. (2013). In defense of disciplines: Interdisciplinarity and specialization in the research university.

Chicago: University of Chicago Press.

Kanter, R. M. (1977). Men and women of the corporation. New York: Basic Books.

Kent, J. D., & McCarthy, M. T. (2016). Holistic review in graduate admissions: A report from the council of

graduate schools. Washington, DC: Council of Graduate Schools.

Kuncel & Hezlett, (2007) Standardized test predict graduate students’ success. Science, 315, 1080-1081.

Kuncel, N. R., Ones, D. S., & Hezlett, S. A. (2001). A comprehensive meta-analysis of the predictive validity of

the Graduate Record Examinations: Implications for graduate student selection and performance. Psycho-

logical Bulletin, 127(1), 162–181.

Kuncel, N.R., Wee, S., L., & Hezlett, S.A, (2010). The validity of the graduate record examination for masters

and doctoral programs: A meta-analysis investigation. Educational and Psychological Measurement, 70, 340 -

352.

Kyllonen, P. (2011). The case for noncognitive constructs and other back-ground variables in graduate

education (ETS GRE Board research report ETS GREB-00-11). Retrieved June 25, 2018, from

www.ets.org/research/policy_research_reports/publications/report/2011/ined.

Lazarsfeld, P., & Merton, R. (1954). Friendship as a social process: A substantive and methodological analysis.

In Berger, M., Abel, T., & Page, C. H. (Eds.), Freedom and control in modern society (pp. 18–66). New York.

MacLachlan, A. J. (2017, February). Preservation of educational inequality in doctoral education: Tacit

knowledge, implicit bias and university faculty (UC Berkeley CSHE, 1.17, January 2017). Berkeley: Center for

Studies in Higher Education, University of California–Berkeley. Retrieved June 25, 2018, from

https://escholarship.org/uc/item/5zv6c3nj.

McPherson, M., Smith-Lovin, L., & Cook, J. M. (2001). Birds of a feather: Homophily in social networks. Annual

Review of Sociology, 27(1), 415–444. doi: 10.1146/annurev.soc.27.1.415.

Mervis, J. (2016). Scientific workforce. NSF makes a new bid to boost diversity. Science, 351(6277), 1017. doi:

10.1126/science.351.6277.1017

Milkman, K. L., Akinola, M., & Chugh, D. (2015). What happened before? A field experiment exploring how pay

and representation differentially shape bias on the pathway into organizations. Journal of Applied

Psychology,100(6), 1678–1712. doi: 10.1037/apl0000022

Miller, C., & Stassun, K. (2014). A test that fails. Nature, 510, 303–304.

Moneta-Koehler, L., Brown, A. M., Petrie, K. A., Evans, B. J., & Chalkley, R.(2017). The limitations of the GRE in

predicting success in biomedical graduate school. PLoS ONE, 12(1), e0166742. doi: 10.1371/journal. 0166742

National Science Foundation. (2017). Women, minorities, and persons with disabilities in science and

engineering: 2017 (Special report NSF 17-1310). Arlington, VA.

Okahana, H., & Zhou, E. (2017). Graduate enrollment and degrees: 2006 to2016. Washington, DC: Council of

Graduate Schools.

Pacheco, W. I., Noel, R. J., Porter, J. T., & Appleyard, C. B. (2015). Beyond the GRE: Using a composite score to

predict the success of Puerto Rican students in a biomedical PhD program. CBE—Life Sciences Education,14(2),

ar13. doi: 10.1187/cbe.14-11-0216

Park, H. Y., Berkowitz, O., Symes, K., & Dasgupta, S. (2018). The art and science of selecting graduate students

in the biomedical sciences: Performance in doctoral study of the foundational sciences. PLoS ONE, 13(4),

e0193901. doi: 10.1371/journal.pone.0193901

Posselt, J. R. (2016). Inside graduate admissions: Merit, diversity, and faculty gatekeeping. Cambridge, MA:

Harvard University Press.

Roush, J. K., Rush, B. R., White, B. J., & Wilkerson, M. J. (2014). Correlation o fpre-veterinary admissions

criteria, intra-professional curriculum measures, AVMA-COE professional competency scores, and the NAVLE.

Journal of Veterinary Medical Education, 41(1), 19–26. doi: 10.3138/jvme.0613-087R1

Scott, L. D., & Zerwic, J. (2015). Holistic review in admissions: A strategy to diversify the nursing workforce.

Nursing Outlook, 63(4), 488–495. doi: https://doi.org/10.1016/j.outlook.2015.01.001.

Stichweh, R. (1992). The sociology of scientific disciplines: On the genesis and stability of the disciplinary

structure of modern science. Science in Context, 5, 3–15. doi: 10.1017/S0269889700001071.

Valantine, H. A., & Collins, F. S. (2015). The National Institutes of Health addresses the science of diversity.

Proceedings of the National Academy of Sciences USA, 112(40), 12240–12242. doi:

10.1073/pnas.1515612112

Wilson, M. A., DePass, A., & Bean, A. J. (2018). Institutional interventions that remove barriers to recruit and

retain diverse biomedical PhD students. CBE—Life Sciences Education, 17(2), ar27. doi: 10.1187/cbe.17-09-

0210.