Skip to content

Blog

Returns on Rigor: Stimulus Assignment Balancing by Key Demos

Returns on Rigor: Stimulus Assignment Balancing by Key Demos

For the ResOps Nerd: Tooling to Easily Balance Stimulus Assignment

This is a follow-up to an earlier post about the iterative process about how IntelliSurvey focused on making improvements to our tooling to achieve greater efficiency.

The Old Approach

When we were younger and full of hubris, we imagined researchers might want to balance stimulus by many demographic variables – perhaps, age, gender, region, income, etc. We implemented an algorithm that enabled the needfulness of different characteristics to be smashed together with demo-level configurable weights, so that gender could be 5X as important as income if so desired. Nifty!

But this framework mostly just sat on the shelf. Most researchers principally cared about balancing stimulus across key demos, which this algorithm didn’t do as perfectly as desired. While it approached even-ness, in-field differences could only be discerned by puzzling out the intermediary state. The reader can imagine some cases where the assignment wasn’t 50-50, but 45-55, and the ensuing consternation as individuals tried to discern why the 157th respondent had been assigned stimulus B.

Because the “baked in” feature wasn’t hitting the spot, some of our users built relatively convoluted structures manually to address balancing. The structures would have instructions along these lines:

  1. Set suitable quotas for stimulus for each demographic slice (e.g., 8 for Concept A, M, 25-34, W)
  2. At the decision point, check to see which stimulus items are open for that respondent
  3. Select the most needed master stimulus item (i.e., A, B, or C)

This worked! But some badness festered.

  1. Because the selection was based on the master level (A, B, C) before quotas were hit, in-field stimulus demographic balancing would occasionally skew. Some users crafted machinery to address interim balancing, further increasing complexity.
  2. Numerous stimuli (e.g., 100+ financial institutions) sometimes bog down in heavy computing.
  3. The machinery itself was complex and bespoke. As the world moved towards DIY and DIT workflows, this machinery was plainly not accessible to a generalist. They’d need a proper survey programmer to take some crafting time. The survey programmer’s work would then need QA review and to be closely watched in the field.

The New Algorithm

Realizing that we’d missed the mark, we brainstormed with clients. We modeled real-world loads with different algorithms to assess fill efficiency and performance under heavy load. After much experimentation, we settled on a new algorithm that works thusly:

  1. Set suitable quotas for each master-level stimulus item (e.g., A, B, or C)
  2. At the decision point, select the stimulus item most needed for respondents with specific characteristics (e.g., among M[25-34][W]).
  3. Tiebreak with the most needed item at the master level (i.e., A, B, or C)

This algorithm can get out of balance at the master level. Consider the following situation, balancing stimulus [A,B] by ([M|F] & [<45|45+]). On the left is hypothetical respondent-level data; on the right, its summary.

Respondent data

Resp IDStimulusGenderAge
1AM<45
2AM45+
3BF<45

Counts

Resp IDA countsB counts
M <4510
M 45+10
F <4501
F 45+00
Total21

Imagine that the next arrival is F<45; this respondent will be assigned “A” as it is the most needed stimulus for F<45. However, the master level is already 2A | 1B. So balance will momentarily get worse, at 3A | 1B. Not good.

However, while it may not stay perfectly balanced at the master level, it doesn’t drift far. And the interim state is easily auditable (via crosstabs of stimulus assignment X demos). Most in our orbit have indicated that this is OK. So, we have something that seemed usable.

But we tested it, and found something unpleasant that the uber ResOps Nerd might have already anticipated. Consider step 2 of the new algorithm, which depends on the counts of the assigned stimulus for the selected demo. Calculating these can be pretty intense as data sets scale. Will silicon melt down when a survey already has 5,000 completes, and 1,000 respondents hit the stimulus assignment area at exactly 7:00:01 AM UTC on Friday the 13th? Unlikely? Perhaps. But across longer time scales, unlikely things happen.

Performance Under Heavy Load

To support performance under heavy load, IntelliSurvey platforms cache stats needed during survey runtime. So, on that Friday the 13th at 7:00:01 AM UTC, the system would not really be assembling stats for each respondent. Instead, it would just read from the stats pre-assembled by the heavily optimized and indexed database. Easy peasy, no?

But caching is never concurrent. Unfortunately, under selected conditions, flooding of a particular demographic cell (say, 100+ older men in CA like me) who hit the decision point at exactly the same time might result in skewed assignment. While this might be rare, if it happened on a study, we might find ourselves in the same boat – clients users distrusting the new tooling, and back to building bespoke contraptions.

So, to reduce the probability of a flooding skew, we lowered selected caching refresh times across our entire infrastructure. Without going into proprietary particulars, this is both hideously onerous and dangerous, and not a big deal at all. As a result, while stimulus assignment asymmetry remains possible, it’s now very unlikely.

And, with this in mind, we now enable our users (whether scripters or generalists using our GUI) to write stimulus selection instructions in their questionnaires and programs exactly like this:

     **\[BALANCE STIMULUS ASSIGNMENT BY AGE, GENDER, AND REGION\]**

and feel confident that the balancing will be done properly, with stimulus sets of almost any size, without any fuss, at almost any load (Super Bowl advertisement invites excluded).

The result? Survey programs for which rigor is much more adjacent. Less need to train scripters. Less need for QA. Faster delivery of surveys. A better world, one bit of tooling at a time.