Google looks to reduce pushback bias in developers’ software code review

Google it trying to make its software development code review process more equitable after finding that women, Black+, Latinx+, and Asian+ developers face pushback on code changes more frequently than White, male engineers. It also found that older developers faced higher odds of pushback than younger developers.

Google revealed details about code review pushback in its study “The Pushback Effects of Race, Ethnicity, Gender, and Age in Code Review”, published in computer industry journal Communications of the ACM. 

The study looked at the day-to-day experiences of traditionally underrepresented engineers in tech.

SEE: Software skills will get you far, but you don’t have to be a coder to make it big in tech

The study found that “excess pushback” costs Google more than 1,000 extra engineer hours every day, or around 4% of the estimated time engineers spend on responding to reviewer comments. The cost was borne by non-White and non-male engineers, it found. 

“Code review is fundamentally a decision-making process, where reviewers must decide if and when a code change is acceptable; thus, code review is susceptible to human biases,” noted Google researchers Emerson Murphy-Hill, Ciera Jaspan, Carolyn Egelman, and Lan Cheng. 

They found that women at Google faced 21% higher odds of pushback than men during code review. Also, Black+ developers faced 54% higher odds than White+ developers; Latinx+ developers faced 15% higher odds than White+ developers; Asian+ developers faced 42% higher odds than White+ developers; and older developers faced higher odds of pushback than younger developers. 

Before the study, the authors actually wrongly thought Asian developers would face less pushback because of stereotypes, but the study showed otherwise. “We hypothesize that those who identify as Asian will face more positive evaluations than those who identify as White, because Asians are stereotypically viewed as having higher role congruity in engineering fields,” they noted.     

For context, the researchers explained that at Google code changes must be reviewed by at least one other engineer. Most reviewers are on the same team as the author. Authors can choose their reviewers or have one allocated from the code review tool, which Google calls Critique.

“The code review tool provides authors and reviewers with opportunities to learn about each other, including their full names and photos (more in the supplementary material),” they explained. 

To address these issues in code review, Google has been exploring the effectiveness of anonymous code reviews, which it hopes reduces the gaps in pushback faced by developers from different demographic groups. 

It tested the idea last year by asking 300 developers to do their code reviews without the author’s name at the top of the report. It did this using a browser extension that removed the author’s name. One potential problem with anonymous code reviews is when the reviewer needs to contact the author for complex discussions. 

SEE: Upgrade your job: 5 ways to get that career boost

All Google code resides in one large repository. When an engineer wants to make a change to some code, they create a “changelist”, which is similar to pull requests on GitHub that need to be vetted and approved.    

The results from the extension experiment showed that review times and review quality appeared consistent with and without anonymous review. They also found that, for certain types of review, it was more difficult for reviewers to guess the code’s author.

“Through continued experimentation with anonymous code review, we’re hoping to reduce gaps in pushback faced by developers from different demographic groups. And through this work, we want to inspire other companies to take a hard look at their own code reviews and to consider adopting anonymous author code review as part of their process as well,” said Murphy-Hill.