The Future of Open Ends: New Research Reveals Big Potential

IntelliSurvey has long been a leader in conducting sophisticated research quickly. Using AI to gain new insights even more quickly is, of course, a natural fit for us. That’s one reason that we recently introduced our “AI prompt” feature, which allows our IS Pro survey software users to code open-ended (OE) responses in real-time by querying an LLM. This same feature then allows researchers to generate custom prompts to instantly create follow-up questions that probe respondents for more information. Our system makes this incredibly easy to implement, which you can see here in our open-ended coding demo video.
We recently conducted some research-on-research with this feature to assess how effective AI follow-up techniques are in improving data quality and depth. This post dives into those results.
Designing the Experiment
For this experiment, we asked respondents about their exercise habits and their satisfaction with their gym, if they had one. We split the sample into four groups. The control group was asked only closed-ended (CE) questions, with no use of AI to prompt more deeply. For the other three groups, we used IntelliSurvey’s AI prompt feature to follow up to varying degrees.
The following is a summary of some of the learnings of this experiment.
Finding #1: Discrepancies in Self-Reporting
Example 1: “How Much?”
When respondents were asked, “How often do you exercise?” as an open-ended question, they reported exercising more frequently than when asked the same question in a closed-ended format. Note that when using our AI prompt feature, we instantly coded the open-ended responses into the correct frequency groups, so we were able to branch to the appropriate next question from the open-end.

Why did people report more exercising in an open-end? We aren’t sure. But this looks like a social desirability bias. Just like they might do when talking to an acquaintance or a qualitative interviewer, the conversational tone of an open-ended question might lead respondents to overstate their exercise patterns. As opposed to the closed-ended format, in which the presence of closed-end choices like “1 to 3 times per month” or “About once a week” makes it socially acceptable to pick one of those options.
It remains unclear which responses most accurately reflect reality.
Finding #2: Differing Results
Example 2: “Why?”
TL;DR: AI-coded open-ended questions differ materially from closed-ended questions, and may be a valuable complement.
Until recently, open-ended questions were often understood to be slower and more expensive than closed-ended questions. With AI, speed and cost are less pressing constraints - for example, IS Pro includes AI OE coding as a standard feature.
In our experiment, open-ended responses revealed a result that was unobserved in traditional question formats. Many open-ended responses indicated dissatisfaction with rules and policies, which had not been in the closed-ended question set. “Crowded/busy” was also not in the closed-ended set and did not appear in any “other specify” open responses.

The AI coded open-ends delivered insights that could not have been uncovered with the close-ended format. In a real project, we might consider a 2-step approach: 1) start with open-ends in soft-launch; and 2) move to a close-ended set derived from spontaneous answers for a more quantitative read.
Which story is “true”, though? Perhaps both are. The provision of an option set in the CE may fundamentally alter the respondent’s mindset, influencing more tactical responses. The absence of framing offers top-of-mind results, which seem authentic and resonant.
There are other cases in which the option set may offer an imperfect response framework, and incorporating OEs early (or for a fraction of respondents) could limit the risks and offer an interesting alternative perspective.
Finding #3: AI Follow-up Prompts Offer Deeper Insights
Example 3: “Please explain more.”
Within the context of “Why”, AI offers a substantially new opportunity that has not been possible with open-ends before AI. Now with IntelliSurvey’s AI prompt feature, researchers can automatically create targeted and customized follow-up questions for respondents. Here are some examples:

Surprisingly, with judicious prompting, the AI was able to generate a broadly sensible follow-up question nearly 90% of the time. Our team found this result astonishing. With strong follow-up questions, respondents offered significantly more context and details about the reasons for their response. Surprisingly, even when the follow-up question was poorly targeted, respondents sometimes added details – and remained engaged.
This ability to dynamically probe more deeply into answers on the fly may be a game-changer for quantitative research projects. We are now encouraging our clients to consider adding AI-assisted follow-up questions for many types of essay open-ends in their surveys.
Conclusions
AI is not perfect. It provides some confusing and potentially misleading responses to some areas of investigation.
AI is particularly well-suited to analyzing open-ended responses rapidly and providing exhaustiveness checks for option sets.
When paired with “follow-up”, today’s open-ends can be much better than they were in the past. Follow-ups can get actionable specifics and add color that was previously only possible through slow and expensive qualitative interviews.
We’ll keep experimenting with AI in quantitative survey research and share our findings to help move the industry forward. Follow us on LinkedIn to see our latest Research-on-Research results, and reach out to your IntelliSurvey contact to explore how these AI tools can add value to your next project.