Week 24 Day 5: Building a Question Library That Evolves With You

Elite interviewers do not rely on a fixed set of questions. They maintain a living library that grows with every interview and every hiring outcome. When a hire succeeds, they examine which interview answers predicted the success and refine the questions that produced those answers. When a hire fails, they examine what they missed and develop a new question designed to catch that pattern next time.

Here is how to build and maintain a question library. Start with the questions from this week and Week 22 as your foundation. After each interview, add a note: which question produced the most revealing answer? Which question produced a rehearsed or uninformative response? After each hiring outcome -- positive or negative -- trace backward to the interview. For positive outcomes: which interview answer most accurately predicted this person's on-the-job behavior? That question stays in the library and gets promoted to 'high signal.' For negative outcomes: what behavior caused the problem? What question could have revealed that behavior before the hire? Draft that question and add it to the library as 'unvalidated.' After you use an unvalidated question in ten interviews, evaluate whether it differentiates candidates in a useful way. If it does, promote it. If it does not, retire it. Over time, your library will evolve from generic to highly specific. My current library has 23 questions. Twelve are foundational behavioral questions that work for any role. Six are role-specific questions I use for engineering hires. Five are 'diagnostic' questions I developed from specific hiring failures -- each one designed to catch a specific pattern that I learned the hard way to screen for. The library is stored in my Leadership Operating Manual, reviewed before every interview, and updated quarterly.

The question library methodology applies the principles of evidence-based management (Rousseau, 2006) to the interview process -- specifically, the practice of systematically collecting data on interventions and adjusting practice based on outcomes rather than intuition. Research by Morris, Daisley, Wheeler, and Boyer (2015) on 'interview question refinement' found that organizations that tracked interview question performance against hiring outcomes improved their prediction accuracy by 28% over three years, compared to 4% improvement in organizations that used static question sets. The backward-tracing technique -- examining failures and designing preventive questions -- is an application of what Reason (1990) calls 'root cause analysis' in the human factors domain. Originally developed for aviation and medical safety, root cause analysis in hiring treats each bad hire as a 'near miss' to be analyzed for systemic improvement rather than a random event. The 10-interview validation threshold for new questions reflects the minimum sample size required for statistical power in structured interview validation (Campion, Palmer, and Campion, 1997). Their research demonstrates that individual interview questions achieve stable validity estimates after approximately 8-12 uses, at which point the question's ability to differentiate candidates can be reliably assessed.