[ENCORE] Changing Hiring Practices to Build a More Diverse Technology Organization

A Case Study from Avvo

Photo of a forest, February 2024 - by Kevin Goldsmith

Note from Kevin: August 10th, 2025

Hello all! I’m taking a break during July and August. I’ll be back in September with new newsletters. While I’m gone, I’m re-sharing some articles that you may have missed.

In October, I will once again be speaking at the Lead Dev New York event. Registration and information for that is here. You can use the discount code “KEVIN15” for 15% off on your registration.

And now for the encore article!

---

Hallo zusammen! (Hello everyone in German)

Again, I’m humbled by the reaction to this newsletter and the podcast and delighted to see the number of readers and listeners grow each week. If you find value in these articles, please share them with your friends!

Last week’s podcast was “Taking a thoughtful approach to the job search process,” describing how to be more deliberate and prepared when looking for a job or interviewing. You can download the episode from the link above or listen to it on Spotify, iTunes, YouTube, Amazon, or wherever you get your podcasts.

I got my first copy of the Hardcover edition this week, and it looks terrific! Something that will inspire comments when it sits on the shelf behind you during your video calls. We’re getting very close to the March 4th release, and I’m seeing the book appear in more and more international stores.

Why wait? Pre-order today! 

I’ve given up trying to keep up with all the bookstore links, but https://itdependsbook.net will always have the most up-to-date links to everything.

As always, a reminder for newsletter readers and podcast listeners… Ask me questions! I’m always happy to answer. You can ask about the chapters I’m sharing or anything related to technology leadership or the industry. Send an e-mail to contact@itdependsbook.net, and please let me know if it is okay to share your name when I answer your question.

About this week’s chapter

We’re hiring a new role for my team at DistroKid, and we’ve adopted a new candidate tracking system that I’m excited about. This system hides features of the candidates that might introduce bias in the hiring process. It’s a way to help improve the diversity of the candidate pool. I’ve worked with recruiting to do something like this manually in the past, but I’m glad that tooling is starting to catch up to this need.

I won’t mention the tool specifically because we’ve just started using it, and until we’ve lived with it for a while, I’m uncomfortable recommending it. That said, if you want to know, ping me by e-mail or on social media, and I’m happy to share.

The chapter I’m sharing this week is a case study of how we systematically worked to improve diversity and inclusion at Avvo while I was the CTO there. I’m amazingly proud of what we achieved in a relatively short amount of time. Since then, I’ve incorporated the lessons learned at each company I work at and realized improvements in diversity at each.

I’ve also been more conscious about choosing companies that already value diversity, so I haven’t had to be as overt as I was at Avvo. When I joined DistroKid, the tech team and broader company were already more diverse than any tech or media company I’ve worked at, at all levels. That has only increased over the last year. As I say in another chapter of the book, the more diverse you are, the more diverse you become.

I’ve been disappointed that while diversity was a significant priority for the tech industry for several years in the 2010s, it seems to have become less so since Covid and the Covid downturn. We, current and future technology leaders, must not let our organizations slide into the past.

---

Changing Hiring Practices to Build a More Diverse Technology Organization

A Case Study from Avvo

Originally published on June 12, 2020

Introduction

I was the Chief Technology Officer at Avvo from 2016 until 2018 when the company was acquired. Building a more diverse and inclusive technology team was one of the proudest achievements of my tenure there.

Given current events, I hear from friends at tech companies that they are re-evaluating their lack of diversity. They want to improve. While companies have tried different tactics for years, we haven’t made much progress as an industry. It is daunting if you are trying to address diversity for the first time. It is easy to try some things, make no progress, and then give up.

This chapter isn’t a prescription for improving diversity at any technology company. It is a case study of how we grew diversity at Avvo. The different strategies we employed may be helpful for your efforts. They may inspire you to try new things appropriate for your company and culture.

Starting Point

Avvo is an online legal marketplace connecting consumers and attorneys in the US. The company overall was reasonably diverse from both a gender and a race perspective. When I joined in 2016, the technology team was a different story.

The technology organization was not very diverse at the start of my tenure, as displayed in the charts below. Additionally, there was no one from any under-represented groups with management responsibilities in the Technology team.

Figure 29 - Avvo Technology Team Racial Diversity in 2016

Figure 30 - Avvo Technology Team Age Diversity in 2016

Figure 31 - Avvo Technology Team Gender Diversity in 2016

The team’s status quo was not the result of overt discrimination, but it did demonstrate a lack of prioritization of diversity, inclusion, and reducing unconscious bias.

My vision for the company was to make it an exemplar of diversity and inclusion within the region. I wanted to not only make Avvo’s technology team diverse to prove that it was possible, but I also wanted to show that a diverse group is more than capable. I additionally believed that we could use our diversity as part of our employer branding, helping us stand out in a crowded talent market.

Making Improving Diversity a Priority for Technology

I set the tone I intended to take as part of my interview process for the Chief Technology Officer role. I didn’t want any potential for disputes between the rest of the senior leadership and myself.

Once I had accepted the job, I had a multiple-month notice period with my former company. I used the time to talk to the Director of Engineering and head of recruiting to prioritize diversity in hiring, set expectations, and start building a better pipeline.

Building a Coalition and Setting Expectations

If you are working to build a more diverse team, you must get recruiting as your partners in on the effort. While it may seem natural, it is not. Recruiting (especially external recruiters) are incentivized or measured on their ability to close roles as quickly as possible. Most recruiters I have worked with are delighted to help find diverse candidates, but it is critical to let them know you understand that it means it will take the whole process longer. It is also essential that their managers understand this.

If you have hiring managers reporting to you, you need to get them on board and excited about this effort as well. One measure of a manager is their ability to hire well. Being serious about diversity will likely mean they will take longer to fill roles for their team.

You may need to assure them you will consider this if they miss their KPIs (Key Performance Indicators). You cannot set up or hold conflicting goals for your managers. You will likely fail at both if you cannot prioritize building diverse teams over hitting KPIs.

One way to handle the increased hiring time is to start sourcing for a role much earlier than scheduled. Spend the time before the position is scheduled to open only looking at under-represented candidates. Create an agreement with your manager that you can hire ahead of schedule if you find a suitable candidate early. If you reach the planned opening date for the role and haven’t located anyone, you can open the pool to your normal channels without losing time.

Your manager will also need to understand the potential impact of a diversity hiring focus on your KPIs.

With your manager’s support, create a reasonable KPI or OKR as your goal to track your efforts. The OKR should give you a target and some cover if there are challenges to some of the tradeoffs you may need to make because you are hiring more slowly. Share your goal with your team. Update them on your progress regularly. Transparency is essential but also helps to enlist the rest of your team to do their part to help achieve the goal.

At Avvo, my diversity goals were part of my OKRs. I shared our progress with a monthly update to the organization. This repeated communication demonstrated the level of importance I placed on this goal and showed our improvement in real time.

The good news is that as you progress towards your goal, your time-to-hire will come down while maintaining or increasing your diversity. It does get easier. You will soon stop making tradeoffs as you continue to improve.

Fixing the Interview Process

When I joined Avvo, the hiring process involved multiple pre-interviews, a do-at-home coding challenge, a review meeting, an in-person interview loop, and a follow-up meeting. The process may have taken a single candidate over a month to get from the recruiter’s first e-mail to an offer. Even without considering diversity, it was not an efficient process for a company that wanted to grow.

The Avvo process kept inappropriate candidates out by design. It did not find appropriate candidates, however. In machine learning parlance, the false rejection rate was too high to avoid raising the false acceptance rate. If you are trying to screen out, you miss a lot of good people. Often, biases are part of the filters. Those biases reinforced the lack of diversity.

The current process's state gave me a license to make some drastic changes. I had the support of the recruiting team and many of the managers. They all had stories of losing good candidates because of the length of the process.

Some employees and managers defended the status quo hiring process as required to maintain the quality of the development team. Luckily, there were few of them. After listening to their position and having a good discussion, I decided to move forward, knowing that I had the support of most of the group.

Removing the Coding Challenge

Many companies believe the at-home coding challenge is critical to establishing a candidate's bona fides before investing too much time in them. Many sources discuss the problems with coding challenges from a diversity perspective. The arguments against the practice that resonated with me were the following:

·     Candidates often take much longer than companies expect them to complete. The Avvo guidance was that the candidate should spend no more than two hours on the challenge. One candidate estimated that he spent 12 hours on it. He wanted to make sure that he did well.

·     A coding challenge is unpaid labor with no value for the results. The demand sends a message that the company places no worth on the candidates’ time. I do know that some companies pay candidates a nominal fee for completing the challenge. I encourage this, but eliminating the challenge also solves this problem.

·     Not all candidates have sufficient time to work on coding challenges. If a candidate is on a job search, they may get challenges from multiple companies, each with an expectation of hours of work. A candidate may already work at a stressful job or multiple jobs. They may be a caretaker to children or other relatives or have a long commute so that they can live in an affordable home. The candidate may have other issues that require their time. The challenge can be an unreasonable burden for those people.

A coding challenge requires that the candidate have the equipment and connectivity to complete the project. This requirement may not seem like an unreasonable expectation for a software developer, but not every developer may have the economic or personal safety to make this happen.

The combination of all of the above creates a bias towards people who do not have significant constraints on their time and finances. Those biases reinforce and perpetuate the lack of diversity in the industry.

As part of the hiring process, it is still critical to understand a candidate’s technical maturity. We replaced the coding challenge with multiple in-person challenges during the interview loop described later in this chapter. To not overwhelm our interviewers, we focused on ensuring that the recruiter and hiring manager screens established a high culture and technical bar. This expectation on the screening required training the hiring managers on questions to ask for candidates at different levels. As we got comfortable with this process, the new screening process achieved equivalent success in identifying candidates while increasing diversity in the candidate pool and significantly decreasing the time-to-hire.

The Interview Loop

There are many articles on reducing bias in the interview loop. We combined the ideas that we felt resonated the best with our culture and some of the best ideas I had experienced from prior companies.

Interviewer Training

We required that anyone participating in the interview loop participate in two training sessions:

An overall interviewer training that myself and the head of recruiting for my team presented. This training included much of the standard content of a traditional interview training effort. We added implicit bias training led by someone from Ada Developers Academy (a Seattle developer training program focused on women and gender-diverse people). We had previously hired interns from Ada, and their mentors were required to take this course. It was so valuable that we made a deal with Ada to occasionally present that class at Avvo to help with our interviewer training. We also required that anyone participating in interviews became qualified on the interview questions.

A group of developers for each of the different disciplines got together to create two or three technical problems to solve, reflective of the work they did every day. Each question had a set of representative answers in a 3×3 grid. One axis represented the candidate’s experience level. The other axis was unacceptable, acceptable, and excellent solutions.

Figure 32 - The answer grid for interview questions at Avvo

We asked the same questions to any candidate regardless of the role level or their experience level. Over time, we added to the set of representative answers as we heard new answers from candidates.

To train on the question, ideally, the interviewer would attempt to answer the problem themselves. Then, they would observe a trained interviewer working with a candidate on the challenge twice. Then, a trained interviewer would watch them working with a candidate on the question themselves.

Only once the person completed all those steps could the person interview candidates. Any trained interviewer could interview for any team hiring for a role in that discipline. There was no job level required for an interviewer.

The Interviewer Panel

We ensured that at least one person from an underrepresented group was on any interviewing panel. This requirement was challenging at first. Some of the interviewers ended up having many interviews to do.

We felt it was important for two reasons. If we were interviewing someone from an underrepresented group, they would see someone like themselves and be able to ask questions (if they wanted) about their experience. If we were interviewing someone not from an underrepresented group, we would potentially catch any red flags indicating they might have challenges in a diverse environment.

This practice turned out to be very valuable.

The Interviews

As the interview training required shadowing, almost every interview would have two people from the panel. One person worked with the candidate, and one observed the discussion. This pairing meant that the hiring manager got two interpretations of what happened in the meeting and how the candidate responded to different questions. Multiple perspectives were beneficial in reducing the effect of any interviewers’ biases.

We offered to let candidates bring their tools if they chose so that they could be comfortable rather than presenting the candidates with an abstract problem and having them solve it on a whiteboard or a company laptop. The interviewer and the candidate solved the technical challenges collaboratively. This approach mirrored the way that many teams at Avvo worked in a pair-programming style. The goal was to recreate, as much as possible, the actual working environment of our teams.

Gaining Credibility as an Inclusive Technology Employer

Improving our interview process alone wouldn’t make a significant difference if Avvo was not visible in the market as a company serious about diversity and inclusion.

The team already had multiple people volunteering at the Ada Developers Academy but had never hired an intern. We hired our first two interns from Ada in their next cohort.

We followed that by joining the new Washington Technology Industry Association Apprenti program (a registered technical apprentice program focusing on underrepresented groups and veterans). We hired two apprentices from the first cohort of that program as well.

We always hired interns and apprentices in pairs to ensure they would have someone else going through a similar experience that they could use for mutual support in addition to their Avvo mentors.

Working with these programs let us partner with people deeply enmeshed in the community. We wanted to learn and listen. Working with Ada and Apprenti also introduced us to their volunteers, often experienced developers from other companies who felt strongly about increasing representation in technology.

Along with our partnership with training programs, we ensured we attended the local meetups for underrepresented groups. One of our recruiters, engineering managers, developers, or I would attend these meetups to listen and understand the challenges these groups in the industry faced. We networked as well, but only tentatively at first. We wanted to establish credibility and not just come to a single meeting and disappear.

The sustained efforts to become a part of the community taught us a lot and raised awareness of what we were trying to do at Avvo. Candidates would apply and mention the people they had already met at the company. Avvo was already hosting meetups for different technologies. Those meetings eventually became more diverse as candidates from underrepresented groups visited to see our offices.

Evolving the Culture

The focus on increasing diversity did not have universal support. When I joined, the technology team culture was not very inclusive, and some were resistant to making any changes.

I continued to educate and discuss, but I also pushed firmly forward.

Our agile coaches and I taught new, more inclusive facilitation techniques for meetings and discussions. One of the senior managers started a mentoring program for everyone in the organization designed to support all, but especially those who might feel impostor syndrome.

Eventually, those not interested in the new culture decided to find other opportunities. In the end, this was a relatively small percentage of the team. Most were interested in being part of the new culture.

The Results

In under two years, we increased the percentage of women in the technology team from 17% to 27%. We increased the share of Black, Hispanic, and multi-racial people from 2% to 11%. Our age diversity also improved significantly. We did this while increasing the size of the team by nearly 50%. We also significantly improved our employee net promoter scores during this time.

Figure 33 - Avvo Technology Team Racial Diversity in 2018

Figure 34 - Avvo Technology Team Age Diversity in 2018

Figure 35 - Avvo Technology Team Gender Diversity in 2018

If one thing stands out to me about what we accomplished, it is a question asked by one of our developers right after our acquisition. The acquiring company’s executive team hosted a town hall to answer questions from their new employees. An Avvo developer asked, “We are very proud of the diversity of our technology team at Avvo. What efforts have you put in place to make sure that your team is diverse?”

Acknowledgments

There are several people mentioned in this article by title. I want to acknowledge them here as this was always a team effort and would not have succeeded without their involvement and leadership. Specifically: LaQuita Hester, a fantastic recruiter dedicated to improving diversity in the technology industry; Hunter Davis, Director of Engineering and the creator and guide for our mentoring program; Justin Weiss, Director of Engineering; and Leslie Zavisca, Engineering Manager.

Many, many others were instrumental in ways large and small. Every developer, tester, data scientist, and manager helped.

I also want to acknowledge my manager and executive team peers who had built an excellent company and supported me as I brought my team up to a level of diversity closer to what they had created. The executive team consisted of Mark Britton, Eric Dahlberg, Monica Williams, Bhavani Murugiah, Sachin Bhatia, Kelly McGill, and Jason Moss.

---

Thanks again for reading! If you find it helpful, please share it with your friends.