The interview guide for domain experts in AI
This article is a cutout of my forthcoming book that you can sign up for here: https://www.danrose.ai/book
When interviewing domain experts for artificial intelligence solutions, it's essential to avoid discussing a specific solution but instead focus on the business outcome and the problem at hand. When you interview experts, they sometimes settle on a particular solution too early, even without knowing it. As the solution architect, you might also do the same and miss out on better alternatives. I often catch myself doing that as finding the perfect solution is the most satisfying part of the discovery phase. To focus on the problem and business outcome, I use the following guide as inspiration for questions.
Question: Tell me about the last time you did X (E.g. forecasted sales or did shift planning at the ice cream store)
The question works better than "How do you do forecasting?". Asking this way will provide you with a polished best-case answer. The subject matter expert will tell you how everything is supposed to be done. We all want to present our best version of ourselves, and we can be a little afraid of admitting that we jump hoops when we are busy or things are a little messy. But we are all busy, and everyday work is messy. Teresa Torres has a great example in her book "Continues Discovery Habits.": When you ask people how they buy jeans, they will tell you that they go by brand and quality. When you ask them how they bought jeans the last time, they will tell you that there was a nice discount.
When building AI, you are looking to identify all the mess and procedure bypassing. That is where you will face challenges, and can you decrease these with AI; you can provide much value.
Question: How will you use the information provided by the AI? (E.g. Information about how many ice creams are sold on a given day)
That question focuses on the business need and outcome and not just the wish for the information or the technical solution. The value in any AI can be found in what action we decide on based on the information provided by the model. Uncovering the intended actions reveals the potential value of the AI solution. It also exposes the reasoning (And sometimes the lack of) behind the need for the AI solution.
Question: How would the solution help your new colleague?
Experienced employees can have a hard time seeing the idea of assistance (from AI or not). They can always find a solution to challenges. They don't need help. But when their inexperienced colleagues become the subject, they have an easier time seeing the value and can explain how a solution will help them.
Question: Why can't you solve this problem in any other way than AI?
That will often result in the subject telling you how they think AI will solve the problem. It uncovers potential misunderstandings about what AI can and cannot do.
It also uncovers how well thought through the idea is. Is AI just solutions chosen due to the hype, or have alternatives seriously been considered? Don't be afraid to challenge the idea of using AI. Any good decision can stand that test and is it not a good decision, you will know at some point no matter what. Better sooner than later.
Question: Why will this solution fail?
Have you ever heard people say: "I knew that would fail"? If that is true, even occasionally, then asking this question can save you trouble. You might also know the feeling that you ignored the signs of challenges when you were too excited about a solution. I certainly do.
When asking this question, I often get the answer: "We will fail because we will try to solve everything and not get it done." That is a usual challenge and making the subjects say this brings some realism to the project.
Question: Show me how you do X?
Make the person show you how they do their work. Observing a subject's actions will uncover intangible knowledge. What has become type 1 and routine for the subject will confuse you, and you can point that out and ask what is going on.
Question: What will be hard about (X, Y, Z)?
I often ask questions such as "What will be hard about getting a high accuracy?" or "What will be hard about onboarding users to the solution?". Questions like that uncovers will uncover data features that might not be as trustworthy as you thought. Answers like "We changed the way we log data for X recently" are typical here.
For tips, sign up for the book here: https://www.danrose.ai/book
