As you would probably expect, the thing everyone wants to know with every research study is, what are the results and what do they mean? But there’s a saying in the data business that should be considered in the development of any questionnaire:

“Garbage in, garbage out.”

In short, the quality of the output is determined by the quality of the input. And that starts with questionnaire development. A great deal of time at Coleman Insights is spent behind the scenes thinking about things such as how respondents will see or hear questions, which order they should go in, and so on. Whether we’re writing a complex survey for a media brand or you’re creating a simple survey in SurveyMonkey to send to your database, there are a fundamental set of procedures to consider. Here are three we keep in mind:

#3:     Ask the most important questions in the beginning.

The beginning of a survey is when the respondent is most engaged. Focus on the most important issues facing your brand as close to the beginning of the survey as possible. If you’re a podcaster, you may want to ask listeners how familiar with and interested they are in the guests you plan to talk to in upcoming episodes. While that’s informative, it’s not as important as learning, and tracking, how many podcast listeners are familiar with your brand, and of those, how many listen regularly. The appeal of your guests is irrelevant if few people are even aware that your podcast exists.

#2:     Don’t ask qualitative questions in a quantitative project.

Over the past few years, we’ve become more intentional about discouraging verbatims in perceptual studies. While there are a few exceptions, it’s like trying to paint a house with a hammer; you’re using the wrong tool for the job. In a qualitative study, like a Campfire Online Discussion Group or 20/20 Focus Group, an experienced moderator can probe participants for more details. A question like “Why are you listening to (radio station) less than you did a year or so ago” can be insightful in this type of research.  However, an answer like “Because I listen to streaming more” may be interesting, but it isn’t actionable. With qualitative research, we have the opportunity to dig deeper to find the reasons why the consumer is listening to streaming more. Did your station do something to cause them to listen less? Was there something about a particular streaming service that attracted them? Or, did they get a new smartphone and haven’t downloaded your app yet? Or perhaps they didn’t know they can ask their new smart speaker to play your radio station?

That same listening momentum question can still be effective in a quantitative study, like a Plan Developer. Instead of an open-ended verbatim answer to the question, it is more effective to specifically test theories you have as to why they are listening less and give the respondent some choices. For example:

• The quality of the music has declined
• The station plays the same songs over and over
• The personalities on WAAA talk too much
• Big Bob left the WAAA Morning Show
• WAAA is playing too many commercials
• You are listening to (X) more
• Other (SPECIFY)

The result is likely to be much more actionable, which brings us to:

#1:     Ask questions that will provide actionable information.

Other than satisfying intellectual curiosity, there is no value in learning information that you can’t do anything with. For example, you may be curious what percentage of your listeners listen to your station on a radio vs. your app. However, what are you going to do with this information when you have it? If your goal is to get more people to use your app, you should consider a Campfire study to find out what people like and don’t like about it. Once you have the results, market what users like about the app and be sure to fix the issues that bother them.

In every Plan Developer℠ perceptual study we conduct at Coleman Insights, there are a multitude of ways to get at some of the answers every audio brand wants to know. Why do listeners choose my brand? Why don’t they choose it? How can I use that information to grow? While there is no doubt that one Plan Developer can be immensely valuable to craft and implement a strategic plan, the benefits of trended data, collected annually for our clients can be priceless.

For starters, listeners don’t always recognize changes in their behavior or perceptions. Sometimes these shifts are subtle and gradual, unconscious, or simply forgotten. Thus, asking what brand a consumer is using more or less doesn’t always give us obvious clarity. Trends allow us to uncover these patterns. We can see, over time, which brands have grown, and which have shrunk.

With trended data we can track which aspects of a brand have improved or declined. If a brand’s perception has slipped overall, we can investigate what’s driving that by examining the changes in images and brand attribute evaluation over the past several studies. For example, a radio station’s music image may remain strong, but its morning show image has eroded. In the following example, Station B and Station D were leading the variety image in the market in 2019 and 2020. But trending data show how a new competitor, Station A, gradually grew its variety image and took the lead in 2021.

Trends help us identify changes in content interest, which is applicable to both music and spoken word stations. Music tastes are not static and understanding their evolution in the market and among each station’s listeners can lead to one of the most actionable findings in a study. Tracking changing music appetites can help explain a decline in a brand’s performance, and a Plan Developer helps correct this negative trajectory by optimizing the station’s music blend. Over the past couple of years, monitoring these trends have allowed our clients to stay on top of pandemic-influenced behavior, such as heightened interest in news.

Consistently conducted research helps track the appeal of personalities as they become more familiar to the audience. Trends demonstrate if listeners are associating a station with a regular benchmark contest or feature. They tell us how fast consumption habits are changing and which platforms present the biggest threat.

A Plan Developer for your audio brand delivers “The Plan,” your strategic road map, but it’s the regular check-ups, the trendable studies, that keep your brand healthy and ahead of the competition.

Hi, my name is David, and I obsess over research questionnaires.

“Hi, David. What exactly are you obsessing about?”

I think about the coding that goes into the data file. What if there’s a typo? What if the answer key is incorrectly ordered? What if there’s a mindless copy and paste and the code we meant to assign an answer doesn’t match the code that was captured in the data file? What then??

“Are these the only things keeping you up at night?”

Besides my dogs? No, I also worry about what devices respondents use to take the surveys. We’d like to think they’re all using a shiny 48-inch OLED monitor, but let’s face it–lots of them are taking it on a small phone with a cracked screen! What if the survey isn’t optimized properly for them??

“That seems like a lot of pressure.”

Oh, it is. It’s been more than 16 years since my last on-air shift, and I still have that off-the-air alarm dream. But more often than not, I think, did I check to make sure that the 5-point on the scale says “love it” and the 1-point on the scale says “dislike it a lot?” Did the introduction for that section say “click the button on the left” when the button is actually at the bottom of the screen when you look at the survey on a smartphone?

In this example from an entertainment study, respondents are asked to read the statement on the left and right, but in this mobile view the statements are on the top and bottom.

Are the questions being asked in such a way that will truly get the unbiased answers we’re seeking???

“Well, are they David?”

(takes a deep breath) Yes. Of course they are.

“And why is that? Doesn’t everyone do that?”

(icy cold stare followed by sarcastic eye roll)

“I see.”

It’s so important to get the right people. I’ll dream about how important the sample is.

So, what should I do about this obsession?

Drawing by Charles Barsotti

“Do your clients worry about this kind of thing? Are they concerned these kind of mistakes will be made and things will be overlooked?”

No.

“And why is that?”

We do it so they don’t have to worry.

(silence, followed by pondering)

So, you uh…think I should…

“Keep obsessing?”

Yes.

“Yes.”

Have a good day.

“Bye, David.”

Last week, Harley-Davidson announced it would start shifting production of some of its motorcycles overseas. The company cited new tariffs from the European Union as its rationale. A prepared statement reads, “Harley-Davidson maintains a strong commitment to U.S.-based manufacturing which is valued by riders globally. Increasing international production to alleviate the EU tariff burden is not the company’s preference, but represents the only sustainable option to make its motorcycles accessible to customers in the EU and maintain a viable business in Europe. Europe is a critical market for Harley-Davidson.”

Unfortunately for Harley-Davidson, tariffs are just another bump in the road for a company that needs to focus, like all brands do, on what’s most important – its base position.

What is Harley-Davidson’s base position? What’s the core foundation of its business?

Just taking a shot here, but how about something like: “American-made, big, loud motorcycles”?

So, announcing you’re moving production – even some production – out of the United States is a major brand violation.

But it wasn’t the first violation.

A few months back, Harley-Davidson announced it would debut its first electric motorcycle in 2019.

Is that a brand violation? Take a look at some of the comments at the bottom of the article that appeared in The Verge announcing the news:

That would be like Apple saying you should buy their newest iPhone because you can install Android on it. Might work for some other motorcycle brands. This is Harley-Davidson.

We’re talking about Harley-Davidson, a company who tried to trademark the sound of their idling motorcycle engines. It’d be a hell of a 180 for them to advertise how quiet the bike is.

It’s just not really their kind of advertising, being in harmony with nature, listening to the wind as you silently cruise across the countryside.

For consumers to love the brand, they have to clearly understand the brand.

If you want and can afford a luxury car yet want to go all-electric, chances are you’re not going to buy a Chevy Volt. Or a Nissan Leaf or even a BMW i3.

You’ll probably want to buy the one that has consumers happily plunking down $1,000 to get on a minimum 12-18 month waiting list.

And why is Tesla so hot?

Because sporty electric cars is what they do. It’s their base position.

Tesla can add different kinds of models. They have a high-end model, an SUV, and the waiting list is for the more affordable Model 3. But none of the models deviate from the base position. They are all sporty electric cars.

Will consumers go for the Tesla pickup truck supposedly coming in a few years? Tough to say, but it certainly won’t look like the pickup trucks we’ve always been used to seeing. If it stays true to the Tesla brand, it could work.

Harley-Davidson’s challenge is real. Sales of Harleys are declining. The Harley consumer is aging out, while the younger demographic wants smaller, more affordable bikes. Indian, Ducati and Triumph are experiencing growth and success. Meanwhile, the legacy Harley-Davidson brand stands for large, loud, expensive bikes.

How do you adapt the brand to appeal to a younger market without compromising the base position?

One step would be to launch the electric motorcycle under another brand name.

Harley-Davidson is trying something worth noting – the Jumpstart Motorcycle Experience.  This simulator, which utilizes an actual Harley, is taken by many HD dealers to shows, festivals and expos with the goal of bringing new riders into the fold. Their goal is two million riders in the next 10 years.

You can see how this issue can translate to other brands and businesses, and it certainly applies to media.

Radio stations, morning shows, podcasts and television shows for example, all have to evolve to stay relevant.

Research can play an important role in helping to keep track of its audience’s tastes and how they evolve. Focus groups is a great way to help brands learn how their audiences perceive them and how their tastes correlate to the brand in their own words.

One thing is certain. While evolution is inevitable, the evolution must stay true to the brand’s base position. Changing brand images takes an incredible amount of time, especially for one so strongly cemented.

Harley-Davidson is the American-made, big, loud motorcycle brand.

Last I checked, people still like big, loud American things.