Skip to content

Blog

Reporting Variables: Spews, Lags, and Choke Points

Reporting Variables: Spews, Lags, and Choke Points

Of special interest to: complex survey wranglers

Consider the following example:

AGE. How old are you? [ADD VARIABLE, QAGECAT(18-24, 25-34, 35-44, 45-54, 55+)]

__ Years

Most survey platforms support the creation of variables like QAGECAT, punched as per Boolean logic as respondents complete surveys. These are Reporting Variables, or RVs. RVs are a basic tech, but they’re not always simple or easy.

Onerous to Change RVs Mid-Field

Consider the following age categorization, QAGECAT, based on OE numeric age question, QAGE.

CodeDescriptionLogic
118-24QAGE>=18 and QAGE<=24
225-34QAGE>=25 and QAGE<=34
335-44QAGE>=35 and QAGE<=44
445-54QAGE>=45 and QAGE<=54
555+QAGE>=55

A respondent, ABD123, who answers “70” for QAGE, punches “5” for QAGECAT. During field, a researcher might add a “55-64” age bucket, and a “65+”. The code frame for QAGECAT now looks thusly:

CodeDescriptionLogic
118-24QAGE>=18 and QAGE<=24
225-34QAGE>=25 and QAGE<=34
335-44QAGE>=35 and QAGE<=44
445-54QAGE>=45 and QAGE<=54
555-64QAGE>=55 and QAGE<=64
665+QAGE>=65

With the change, ABD123’s “5” no longer has the same meaning. Platforms have different ways of addressing changes to the codeframe, but it can be just tricky enough that it becomes labor-intensive and open to error.

Onerous to Handle RVs Added During Field

Sometimes, RVs are added during field. If these variables are to be punched during the respondent interview, they must be populated for those who have already completed the survey. This isn’t always trivial. Challenges include…

  • Re-creation chuff. Some users create an assignment in Excel or SPSS. This opens up possibilities for the logic used in re-creation to differ from the survey’s.
  • In-progress respondents? There are often respondents in the midst of a survey. If in-progress respondents have proceeded past the RVs adjudication point, they may not reside in the file for coding (many reflexively download only completes), and will not be populated. Emptiness ensues.
  • Upload logjams. Variables may be added to tracking surveys that have tens or hundreds of thousands of completes. Appending RV data at such volumes may be difficult.

Complexification of Logic when Outside of the Survey Environment

There are some platforms that support the addition of RVs via a direct channel. For some contexts, the proper definitions get be thorny.

Consider a survey with 2 tracks: alcohol and soft drinks. Respondents are assigned to one or the other. On the alcohol track, the survey asks how many drinks the respondent consumed last week. The survey, as an RV, QDRINKFRQ, in the alcohol section, is defined thusly:

CodeDescriptionLogic
1Non-drinkerQDRINKWK<1
2OccasionalQDRINKWK [1-3]
3ModerateQDRINKWK [4-6]
4HeavyQDRINKWK [7+]

A programmer / analyst creates a reporting portal variable using this exact logic, only to find that the reports look profoundly wrong. Why? The reporting portal variable doesn’t “know” that the variable is only to be populated for alcohol track respondents, and a blank may be interpreted by the code above as a zero. An analyst could write the logic for Non-drinkers in a more complex way (as below), but there is increased complexity and risk:

[QTRACK(‘ALC’) and QDRINKWK<1]

RVs defined within a survey are only punched when visiting associated content, simplifying logic and reducing error likelihood. Thus, many researchers prefer to have RVs defined in the survey itself.

Inability to Suss Out and Assess Complex HVs

Some complex RVs aren’t easy to define or review. Consider a variable that seeks to qualify respondents for a more detailed deep dive based on a combination of drinking levels and other criteria. Heavier drinkers qualify, but as the drinking category lightens, conditional qualifying criteria become necessary.

(QDRINKFRQ=1) or (QDRINKFRQ=2 and QAGECAT=2,3,4,5) or (QDRINKFRQ=3 and QAGECAT=2,3,4 and QINCOME=4,5)

More complex surveys often have RVs with much more logic. When this occurs, it can be quite challenging for a researcher to see whether the criteria are properly defined or desirable. Bouncing forward and back in survey test environments where categorizations are announced can ease testing and assessment of complex HV logic. This can greatly lower the overall QA efforts, and reduce error likelihood.

The Spew, and Latency

Mindful of these challenges, researchers often want to make sure that they have a full compendium of variables necessary for analytics. A kitchen sink of dozens, or hundreds, of reporting variable into surveys. This is especially the case for tracking surveys.

Modern survey platforms, and the hardware they run on, process data speedily. But more processing takes more time. Consider a subcategory survey with 5 brand-level RVs for each of 10 brands. Perhaps, each RV has ~20 logic expressions, for a total of → 10 brands x 5 variables / brand x 20 expressions / variable → 1,000 expressions to evaluate. Most platforms should execute this with aplomb.

Now, imagine that this subcategory study became a multi-category survey, with 500 brands instead of 10. And, another brand-level variable is added; it’s a beast, requiring the evaluation of ~100 expressions. Now, the page that once evaluated 1,000 expressions now chews through 500 x (100 + 100) → 100,000 expressions. That’s a lot, even with enhanced hardware and software.

With these burdens, respondents may be impacted. Some page latencies can extend from milliseconds (<50 ms, as is typical) into multiple seconds. With delays come increased abandon rates and practical limits on concurrent respondents. Thus, survey platform documentation (e.g., Forsta and Qualtrics) may caution programmers about adding too many RVs in any one place in the survey.

A Solution: Reporting Only Variables

When analysts crave concurrently accessible RVs, they must be crafted in advance, and calc-ed as respondents through surveys.

Nope.

IS Pro enables RVs to be defined as reporting-only variables (ROVs). ROVs are like regular RVs, with a key difference: they are never hard-punched. They are calculated on the fly whenever accessed within the reporting portal. With optimization, the analyst doesn’t experience added latencies, even though values are always being calculated on the fly. Database wizardry speeds the concurrent calculation of the ROVs, so reports and downloads with ROVs show with comparable to punched variables.

ROVs…

  • Are not calculated during survey runtimes, so respondent latencies don’t increase
  • Render during testing tools to ease researcher review
  • Never require manual calculation, uploading, or re-coding
  • Are forbidden from driving any survey runtime actions (e.g., skip patterns)

What this enables:

  • Lower latencies across our entire infrastructure, concentrated on larger surveys
  • Significantly greater concurrent respondents on larger surveys
  • Eased creation of added variables during and after field
  • Zero complaints from clients about increased reporting latencies

We think that the ROV toolkit is pretty nifty. If you’d like to geek out more about it, please let us know.