If you have been building agents in Microsoft Copilot Studio, you already know that building is only half the battle. The other half, and arguably the more important half, is making sure your agent actually works the way you expect it to before it reaches real users. That is where the Evaluation feature in Copilot Studio comes in. It lets you run structured tests against your agent to verify its behavior. But the moment you open the Evaluation tab and click “New evaluation,” you are presented with a choice under Data type: Single Response or Conversation (preview).
In this post, we will see different types of data type and when you should use each one, and also, we will walk through a real scenario where we will discover that picking the wrong data type can give you a passing result on a workflow that is completely broken.
Most agents handle two very different kinds of interactions. Sometimes a user just asks a quick question like “How many paid holidays do we get?” – the agent looks it up, responds, and that is it. Other times, a user says “I need to take some time off,” and the agent has to guide them step by step, collecting information along the way. These two types of interactions need fundamentally different testing approaches, and that is exactly why Copilot Studio gives you two data types.
Single Response evaluation is like a pop quiz. You hand the agent a question, it gives you an answer, and then it completely forgets everything that just happened. The next question starts with a totally blank slate. This is perfect for testing Knowledge base retrieval, things like policy questions, FAQs, or factual lookups where each question stands on its own. You can include up to 100 test cases in a single evaluation run.
Conversation (preview) evaluation is like a job interview. The agent is tested over a continuous, multi-turn interaction where every response depends on what was said before. The agent must hold context, remember earlier inputs, and follow a logical flow from start to finish. This is essential for testing multi-step workflows where the agent collects information over several turns. You can include up to 50 conversations, with a maximum of 8 question-and-answer pairs per conversation.
| Feature | Single Response | Conversation (Preview) |
| Memory between questions | No, each question is isolated | Yes, full context is maintained |
| Best for testing | Knowledge base Q&A, factual accuracy | Multi-step workflows |
| Max test cases | 100 | 50 conversations (8 turns each) |
Now, let us see how this plays out with a real agent.
We built a simple HR assistant in Copilot Studio called Contoso HR Agent. We uploaded an Employee Handbook as a Knowledge source covering company policies like the 401(k) match, paid holidays, and pet policy. We also created a leave request workflow where the agent dynamically collects the leave type, start date, and end date from the user through natural conversation.
Now the real question: how do we properly evaluate this on a scale?
We started with Single Response evaluation and created a CSV file with five test cases:
| Question | Expected Response |
| What is Contoso’s 401k match? | Contoso matches 50% of your contributions up to 6% of your salary. |
| How many paid holidays do we get? | Contoso employees receive 12 paid holidays per year. |
| Can I bring my dog to the office? | Contoso offices are currently pet-free, with exceptions for registered service animals. |
| I want to take Sick leave. | (Left blank) |
| From next Friday to next Monday. | Your leave is booked |
The first three tests are the Knowledge base. The last two are fragments of the leave request workflow. But in Single Response mode, each question is thrown at the agent in complete isolation.
The Knowledge base questions worked great. But look at what happened with “I want to take Sick leave”:
The agent recognized the intent and asked for the dates. Because we left the expected response blank and used “Compare Meaning” as the test method, the evaluator simply checked whether the response was reasonable. It was, so the test passed.
Now look at “From next Friday to next Monday”:
The agent had absolutely no memory of the previous question about sick leave. It received “From next Friday to next Monday” as a completely cold, out-of-context message. It guessed this might be about a leave request and asked what type of leave the user wanted. But our expected response was “Your leave is booked.”
The “Compare meaning” evaluator compared both responses and gave it a similarity score of just 25 out of 100. Since our pass threshold was set to 70, the test rightfully failed.
And that one failure tells the whole story. The agent completely failed to book the leave because it had no context. It did not know the user wanted sick leave; it did not know those dates were meant for a leave request. It just saw “From next Friday to next Monday” and had to start from scratch.
Now let us test the same workflow with Conversation (preview) evaluation. We used the Conversation CSV template and structured the test cases like this:
| Conversation Number | question | response |
| 1 | What is Contoso’s 401k match? | Contoso matches 50% of your contributions up to 6% of your salary. |
| 1 | How many paid holidays do we get? | Contoso employees receive 12 paid holidays per year. |
| 2 | I want to take Sick leave. | What is the first day you will be absent? |
| 2 | From next Friday to next Monday. | Got it. Your sick leave has been submitted. |
Conversation 1 contains two Knowledge base questions back-to-back. Conversation 2 is the leave request workflow; both turns share the same conversation number, so the evaluation engine maintains context between them.
This time, when the agent receives “From next Friday to next Monday,” it remembers from the previous turn that the user wants sick leave. It extracts both dates from a single sentence, realizes all slots are filled, and completes the booking. This is something that Single Response evaluation could never verify.
Conclusion
Copilot Studio’s Evaluation feature is a powerful way to test your agent before it reaches real users, but choosing the right data type is critical. Single Response works best when you need to verify factual accuracy, things like Knowledge base lookups and one-off questions where each interaction stands on its own. Conversation is what you need when your agent handles multi-step workflows that depend on context and memory across multiple turns.
As we saw in our scenario, using Single Response to test a workflow gave us a failed test case because the agent had no memory of what the user said earlier. The same workflow tested with the Conversation data type passed successfully because the evaluation engine maintained context between turns, exactly like a real user interaction.
FAQs
1. What is the difference between Single Response and Conversation evaluation?
Single Response evaluates each question independently, while Conversation evaluation maintains context across multiple turns in the same interaction.
2. Why does context matter when evaluating an agent?
Many agent workflows depend on information provided earlier in the conversation. Without context, the agent may not understand follow-up responses or complete the workflow correctly.
3. Can I use Single Response evaluation to test a multi-step workflow?
Not effectively. Since each test case is treated as an independent interaction, Single Response cannot verify whether the agent correctly carries information from one step to the next.
4. How should I test a workflow that requires multiple user inputs?
Use Conversation evaluation and place the related questions and responses under the same conversation number. This allows the evaluation to maintain context across the different turns.
5. How do I choose the right evaluation data type in Copilot Studio?
Use Single Response for standalone questions, Knowledge base answers, and factual lookups. Use Conversation for workflows where the agent needs to remember information and respond based on previous turns.