As referenced in a few of my previous posts, I work with consumer opinions for a living. Lately, many colleagues wanted to investigate people’s motivation to participate in programs and what factors “caused” behaviors. As you can imagine, it’s going to get messy!
We study consumer motivation to answer questions like, “what brought this product to your attention, why did you purchase this product over another, did you use a coupon with this purchase,” and so on. Our clients want to know which communication channels work best for the customers, if their discounts/rebates change behavior, and if their marketing efforts could be altered to reach an even broader demographic. Survey research answers these questions and more, but only if you maintain a valid design.
Research questions, like those above, come from research goals. As part of my job, I help project managers craft research questions to inform those research goals. Let’s look at a few research goals and if common the applied methods adequately fulfilled these goals.
Imagine a fitness club offers an online tool for its customers that allows users to track their weight, get health tips, and make fitness commitments. The client suspects that making health commitments leads to more frequent use of the gym (which might lead to higher satisfaction with the gym). In order to motivate people to use the online tool, the gym gave users discounted membership fees if they used the tool at least five days out of the month.
Goal #1 – What marketing channels do my customers use?
Research question: How did you hear about our program? options – online, TV, radio, newspaper, friend/relative, staff member, or other
With this questions, we can determine which form of advertisement reaches your customers. Further, we can detect demographic trends. One would assume younger populations would be using internet channels more frequently than elderly populations. The client may take that information and make online advertisements to target younger crowds, and radio/newspaper advertisements to target older crowds. If you’ve read a copy of Reader’s Digest lately, you know it targets a 50+ crowd based soley on the advertisements you’ll find.
Goal #2 – Did the experience with the program increase satisfaction?
Research question: Did your opinion of the gym change by using this program? (Likert scale decrease to increase)
If 50% of respondents reported that their satisfaction with the gym increased as a result of using the program… that’s good news right? Well, imagine other reports show that 80% of all customers state that they are very satisfied with gym already. How could their satisfactions increase so dramatically if most people report high satisfaction?
This design attempts to drawn causation, but directionality mucks it up! For the uninitiated, directionality refers to factors influencing each other but researchers fail to determine which causes which (more specifically – multicollinearity). How can one survey question assess how the program influences you? You have two fairly simple options – use a control group and only ask general satisfaction, or measure before and after the program is used. Asking the question alone will not show causation!
Goal #3 – Did the money spent on offering incentives (discounts) lead to behavioral change?
Research question: How influential was the membership discount in your decision to use the tool? (Likert scale, very influential to not at all influential)
Unfortunately, standard surveys fail to adequately measure something as sensitive as behavioral attribution in a single question. Even with a question battery designed to assess attribution, it may still fall short. As some of my colleagues have stated, “some folks think we can measure anything with a magic survey.” Indeed, surveys cannot conquer all of our problems! For example, if you want to measure someone’s weight, you use a scale. You don’t ask them for their opinion on their weight!
Moreover, people tend to take ownership of their behaviors, especially when it’s important to their self-image (see my post on justification). The more effort something took, the more likely I’m going to believe I engaged in the behavior because it was the right thing to do. If the behavior in question took less effort, I’m more likely to attribute my behavior to external factors, like a reward or discount. Imagine a new refrigerator was $200 off, but it still cost you $800 dollars. Did you buy it because it was discounted, or because you felt the item would meet your needs. If you felt that $800 was a large investment on your part, you are more likely to take personal ownership of that decision regardless of the what really occurred. Our justification system serves to reduce cognitive dissonance, not lead us to logical conclusions (and spending a lot of money does cause dissonance)!
Simply put… people’s attributions of their behavior are very inaccurate. It’s tough to draw meaningful business conclusions when you’re seeking invalid data! In order to consider other methods, we need to briefly discuss how memory influences decision making.
In a related post (see my “building a better survey” post for more info), I referenced validity in design as being a major hurdle for survey designers. In my work, survey designers try to assess motivations explicitly through a survey, but can well-meaning respondents actual provide accurate information about their motivation? To really dig into this discussion, we should recall dual-process memory briefly. We posses two memory processes that function concurrently, each one influencing our decisions.
When you hear “explicit memory” or “episodic memory” think of recalling events like you’re watching a movie. When we detect stimuli and analyze it with our working memory (consciousness), we have a chance to store this information in our long term memory system facilitated by the hippocampus. These memories can be recalled and reviewed, but they are quite malleable. Justification to reduce dissonance can change our ideas of how events actually occurred. Also explicit memory is SLOW compared to implicit memory. Experiences pass from perception to working memory and then to long term memory. As you’ll soon see, implicit memory completely bypasses consciousness which makes it pretty quick!
When you hear”implicit memory” think of “muscle memory”- When we detect stimuli with our senses, the experiences are automatically stored without using the working memory. Since this memory system never access the working memory, it is never stored as explicit memory. Thus, we cannot recall and review memory stored in this way. However, these memories affect our thoughts, feelings, behaviors, and reactions. When learning a new sport, for example, you spend all of your cognitive effort telling yourself where to place your feet, how to use your hips, where to aim, how to position your shoulders… it’s overwhelming to juggle all that information! After enough repetition, your implicit memory system is able assist you, and you focus your efforts on aiming instead of all those other things. Also, behaviorism (reinforcement and punishment) relies on implicit memory. Reinforced behaviors become unconscious reaction after enough time.
How does implicit memory change or influence our behavior? When I go to the store to purchase something simple like paper towels, which one will I chose? I’m not an expert on how to pick out “the best” paper towels, so I go down the aisle and find one that I’m the most comfortable purchasing. Wait. How could I be more comfortable buying one product over another? Simply put, exposure through advertising and dissonance reduction (see my post on dissonance for more information).
Even though I avoid advertisements at all costs, the more exposure I’ve had with the product through advertising, the more comfortable I’ll feel when making the purchase. This isn’t something that passes through my working memory, it by-passes consciousness and creates a “gut feeling” for a product. So if I’ve just seen the ad in the newspaper for a few years, but never read it, that’s still enough exposure. I’m unaware of how many exposures I’ve had, I just know when making that decision to purchase paper towels… one options felt MORE correct than the others.
Companies just LOVE it when you go with your gut. That means you’ll follow your brand loyalty or whichever product had the most advertising dollars behind it. This is because experiencing a stimulus multiple times changes our attitudes as we become more comfortable with the product. Perhaps due to explicit experience, or implicit ones. It can be hard to tease them apart!
What sort of research designs are appropriate for assessing explicit or implicit information?
- Explicit methods
- Focus groups
- Implicit methods
- Behavioral tracking (number of times visiting the gym for example, but TRACKED not ASKED!)
- Implicit attitude testing (reaction latency – see the glass slipper effect for an interesting example)
- Hybrid – using structured opinion to determine preference
- Maxdiff (see below)
- Conjoint (see below)
When you see explicit measures, you are directly asking about someone’s experience. Explicit basically means “opinion” to me and implicit means “data” to me. If our database shows that people in the program visited the gym 20% more frequently than non-participants, that’s great implicit data. Asking someone if they think they visited more often is still an opinion, and not very interesting explicit data. It gets even trickier when we ask people to say why they engaged in a behavior -“Did some silly website make me go to the gym more often, or did I chose to make healthy decisions on my own?” – my self serving biases will push me to believe I did it on my own, and I might be wrong. A few methods require a bit of opinion, but you can tease out some indication of implicit preference with a bit of creativity.
Imagine you want to know which feature of your gym is most important. In an initial attempt, you make a select-multiple question (multiple response/check all that apply style) and see which items were selected the most. The top three items were top-of-the-line equipment, spinning classes, and enough space in the locker room. The other 30 gym features were largely ignored, but how should the gym budget its renovations? Should you spend all your money on those three items alone? What about other needs or activities? The client knows the swimming pool and climbing wall always have a line of people around it, so clearly some people must like it. We can use Maxdiff analysis to help.
Allow your respondents to become the judge! Just ask, “which of the following items do you want at your gym the most, and which one do you want the least” then show some of the features (towel service, clean bathrooms, yoga classes, basketball courts, spinning classes). Since the gym has about 20-30 different features, you never show them all in one giant list. People won’t engage that way! You ask the question a few dozen times (showing a different assortment of the features each time) and always ask for two responses: most and least preferred or thumbs up/thumbs down. On the analytics side, you are able to determine which items received the best ratio (most frequent positive and least frequent negative), and you can rank your features in order. This is more informative than a simple check box since it forces people to compare features they might have not thought too much about. Maxxdiff gives us a more robust view of the actual preference of people instead of a simplistic opinion.
The other hybrid method is conjoint analysis, but don’t let the name scare you — It’s a pretty simple idea! With conjoint analysis, you have a product with multiple features, and each of these features has multiple levels. Unlike Maxdiff where we just compare the presence of a feature, in conjoint we’re measuring the level of the feature. The most common example we find is with cell phones. Cell phones have features that can vary wildly from brand to brand and version to version. We have screen size, camera resolution, case color, and battery life for this example. In addition to that, we need price. All we ask if whether or not someone would purchase a theoretical product with various levels of the variables. The following examples will make it a bit clearer:
- Large screen, high resolution camera, red color, short battery life – $400
- Medium screen, high resolution camera, blue color, medium battery life – $250
- Small screen, low resolution camera, black color, long battery life – $300
- Large screen, medium resolution camera, red color, medium battery life – $350
Respondents simply state yes or no, “Yes, I would buy this product at this price,” or “No, I would not buy it at this price.” Using these data, you can determine which feature level is most highly related to a yes vote (using logit regression). For example, we might observe that people are more willing to spend $50 more for any type of phone with a red case with a long battery life. Interestingly, if we asked people if color mattered to them in their decisions, they’d likely say “no.”
Additionally, a designer can see which price points will require which features in the market place. Granted, this is still technically in opinion territory, but the customer is not just saying, “I’m willing to spend $400 on a phone with a black case and long battery life.” Frankly, many users don’t understand what they actually want. Given the proper tools, you can learn things from respondents which they didn’t even know! Teasing out implicit memory can lead to great conclusions.
With all this in mind, can a survey accurately measure motivators of behavior? Without using more technical methods like conjoint or maxdiff style surveys, probably not. Much of the decision making process takes place in the implicit memory and questionnaires are just not sensitive enough to detect it. That doesn’t means surveys are useless. Surveys still provide valuable data to us, and sometimes the behavioral data is impossible to obtain. Still, it’s never acceptable to use invalid designs just because other methods are more expensive!
All images are royalty free and accessed from morguefile.com