Addy Osmani posted a compelling essay on critical thinking when using AI agents, titled “Critical thinking when agents do the work.” While it’s mostly about code, it gets very relevant to solutions engineers about halfway through:
“If an agent has synthesized fifty customer calls for you, or a competitive analysis, check a couple of the source documents to make sure the summary is accurate. Check a number that your recommendation depends on to make sure it’s being calculated correctly.”
When I think about how we as solutions engineers use AI to summarize our notes from client calls, that underlines how important it is to validate the output, because it determines how you plan the demo and scope the solution. Any error could cost the deal.
Models get better with each release, and blatant hallucinations and errors become less frequent. As humans, we take that as a false sense of security and gradually stop checking the model’s work, which doesn’t end well.
Osmani recommends prompting for transcript timestamps or quotes that feed the main points, to confirm the output matches what was actually said.
Then check the data your custom demo plans or scope depend on. Ask what the model inferred. Have the model list what it concluded and what those conclusions were based on. Find which objections it left out. Then fine-tune your prompts or your project and skill instructions to keep it from happening again.
Once you click send, it’s your work. “You may offload the work and much of the auditing, but not the accountability.”
My takeaway for pre-sales technical professionals: AI can save time, but it shifts your work from writing the recap to vouching for it. Don’t confuse speed with value. Being right drives credibility, and this is no exception.
Frequently asked questions
- How should I check an AI-generated client-call summary before using the summary to plan a demo?
- Solutions engineers should compare an AI-generated client-call summary with source documents and request transcript timestamps or quotes supporting the main points. Solutions engineers should also verify the data that demo plans or solution scope depend on, ask the model to identify inferences and supporting evidence, and check for omitted objections.
- What should I ask AI to include so I can verify a call recap?
- Ask the AI model to provide transcript timestamps or quotes supporting the main points in the call recap. Ask the AI model to list conclusions, the evidence supporting those conclusions, and any inferences the model made.
- Why do I still need to review AI call notes as models improve?
- The post warns that fewer obvious AI errors can create a false sense of security and lead solutions engineers to stop checking model output. Errors in AI-generated call notes can affect demo planning and solution scoping, potentially costing a deal.
- Who is accountable when a solutions engineer sends an AI-generated recap?
- The solutions engineer who sends an AI-generated recap remains accountable for the recap's accuracy. AI can perform summarization and much of the auditing, but AI does not remove the solutions engineer's responsibility to validate the work.
- What should I change when AI leaves customer objections out of a call summary?
- Solutions engineers should identify the customer objections missing from an AI-generated call summary and examine the model's conclusions and supporting evidence. Solutions engineers should then refine prompts or project and skill instructions to help prevent repeated omissions.