Author: David Baird

Three Best Practices of Questionnaire Development

As you would probably expect, the thing everyone wants to know with every research study is, what are the results and what do they mean? But there’s a saying in the data business that should be considered in the development of any questionnaire:

“Garbage in, garbage out.”

In short, the quality of the output is determined by the quality of the input. And that starts with questionnaire development. A great deal of time at Coleman Insights is spent behind the scenes thinking about things such as how respondents will see or hear questions, which order they should go in, and so on. Whether we’re writing a complex survey for a media brand or you’re creating a simple survey in SurveyMonkey to send to your database, there are a fundamental set of procedures to consider. Here are three we keep in mind:

#3:     Ask the most important questions in the beginning.

The beginning of a survey is when the respondent is most engaged. Focus on the most important issues facing your brand as close to the beginning of the survey as possible. If you’re a podcaster, you may want to ask listeners how familiar with and interested they are in the guests you plan to talk to in upcoming episodes. While that’s informative, it’s not as important as learning, and tracking, how many podcast listeners are familiar with your brand, and of those, how many listen regularly. The appeal of your guests is irrelevant if few people are even aware that your podcast exists.

#2:     Don’t ask qualitative questions in a quantitative project.

Over the past few years, we’ve become more intentional about discouraging verbatims in perceptual studies. While there are a few exceptions, it’s like trying to paint a house with a hammer; you’re using the wrong tool for the job. In a qualitative study, like a Campfire Online Discussion Group or 20/20 Focus Group, an experienced moderator can probe participants for more details. A question like “Why are you listening to (radio station) less than you did a year or so ago” can be insightful in this type of research.  However, an answer like “Because I listen to streaming more” may be interesting, but it isn’t actionable. With qualitative research, we have the opportunity to dig deeper to find the reasons why the consumer is listening to streaming more. Did your station do something to cause them to listen less? Was there something about a particular streaming service that attracted them? Or, did they get a new smartphone and haven’t downloaded your app yet? Or perhaps they didn’t know they can ask their new smart speaker to play your radio station?

That same listening momentum question can still be effective in a quantitative study, like a Plan Developer. Instead of an open-ended verbatim answer to the question, it is more effective to specifically test theories you have as to why they are listening less and give the respondent some choices. For example:

  • The quality of the music has declined
  • The station plays the same songs over and over
  • The personalities on WAAA talk too much
  • Big Bob left the WAAA Morning Show
  • WAAA is playing too many commercials
  • You are listening to (X) more
  • Other (SPECIFY)

The result is likely to be much more actionable, which brings us to:

#1:     Ask questions that will provide actionable information.

Other than satisfying intellectual curiosity, there is no value in learning information that you can’t do anything with. For example, you may be curious what percentage of your listeners listen to your station on a radio vs. your app. However, what are you going to do with this information when you have it? If your goal is to get more people to use your app, you should consider a Campfire study to find out what people like and don’t like about it. Once you have the results, market what users like about the app and be sure to fix the issues that bother them.

The Benefits of a Trendable Study

In every Plan Developer℠ perceptual study we conduct at Coleman Insights, there are a multitude of ways to get at some of the answers every audio brand wants to know. Why do listeners choose my brand? Why don’t they choose it? How can I use that information to grow? While there is no doubt that one Plan Developer can be immensely valuable to craft and implement a strategic plan, the benefits of trended data, collected annually for our clients can be priceless.

For starters, listeners don’t always recognize changes in their behavior or perceptions. Sometimes these shifts are subtle and gradual, unconscious, or simply forgotten. Thus, asking what brand a consumer is using more or less doesn’t always give us obvious clarity. Trends allow us to uncover these patterns. We can see, over time, which brands have grown, and which have shrunk.

With trended data we can track which aspects of a brand have improved or declined. If a brand’s perception has slipped overall, we can investigate what’s driving that by examining the changes in images and brand attribute evaluation over the past several studies. For example, a radio station’s music image may remain strong, but its morning show image has eroded. In the following example, Station B and Station D were leading the variety image in the market in 2019 and 2020. But trending data show how a new competitor, Station A, gradually grew its variety image and took the lead in 2021.

Trends help us identify changes in content interest, which is applicable to both music and spoken word stations. Music tastes are not static and understanding their evolution in the market and among each station’s listeners can lead to one of the most actionable findings in a study. Tracking changing music appetites can help explain a decline in a brand’s performance, and a Plan Developer helps correct this negative trajectory by optimizing the station’s music blend. Over the past couple of years, monitoring these trends have allowed our clients to stay on top of pandemic-influenced behavior, such as heightened interest in news.

Consistently conducted research helps track the appeal of personalities as they become more familiar to the audience. Trends demonstrate if listeners are associating a station with a regular benchmark contest or feature. They tell us how fast consumption habits are changing and which platforms present the biggest threat.

A Plan Developer for your audio brand delivers “The Plan,” your strategic road map, but it’s the regular check-ups, the trendable studies, that keep your brand healthy and ahead of the competition.

Protecting Your Brand Identity

One could argue that nothing is more important to a company or a product than its brand identity. From logo to tag line, from what colors to use, to what “voice” to portray in messaging, brand identity is the gateway to how consumers perceive your brand­—the primary driver that separates successful companies from failures.

When a federal judge in Virginia ruled a few weeks ago that Gruyere cheese sold in America doesn’t have to come from the Gruyere region of Europe, it was not only a blow to the Swiss and French cheesemakers that filed the lawsuit; it was a lesson about protecting your brand identity at all costs. It’s a lesson the Gruyere producers should have learned from their counterparts in the Champagne province in the northeast of France.

While you personally may refer to a bottle of sparkling wine produced outside of France as “Champagne,” vineyards may not (although there’s an odd California loophole). As far back as 1843, a group of Champagne producers in France’s Champagne wine region banded together to successfully sue producers of sparkling wines outside the region for referring to their products as Champagne. But the effort to protect Champagne’s brand identity hardly stopped there. The Comité Interprofessionnel du Vin de Champagne (CIVC) was founded in 1941 with the specific goal of “preventing misuse of the name Champagne to describe sparkling wines, whether fakes being passed off as Champagne, or sparkling wines from other regions that use the name Champagne as a generic term (eg, American Champagne).” The CIVC has a Champagne Bureau in Washington, DC to protect the word “Champagne” from being misused in America. In 1990, new legislation extended Champagne brand protection beyond wines and spirits, resulting in successful lawsuits that prevented the selling of “Champagne cigarettes,” and a perfume called “Champagne” created by Yves Saint Laurent. Not sparkling wine from Champagne, France? No soup for you.

Which brings us back to our cheesy friends from Gruyere. They wanted similar trademark protection to what Champagne gets, arguing that Gruyere cheese can only come from the Gruyere region in Europe. The U.S. Dairy Export Council said Americans understand the gruyere name to be generic, applying only to the style regardless of origin. The honorable T.S. Ellis III agreed, and the words in his decision are important:

“It is clear from the record that the term GRUYERE may have in the past referred exclusively to cheese from Switzerland and France,” Ellis wrote. “However, decades of importation, production, and sale of cheese labeled GRUYERE produced outside the Gruyere region of Switzerland and France have eroded the meaning of that term and rendered it generic.”

How could this happen? Champagne is hardly the only example of brand protection. There’s even a cheese example from a similar part of the world. Roquefort can only come from the village of Roquefort-sur-Soulzon in southern France. Why not Gruyere?

It all comes down to brand identity protection.

The originators of Champagne have spent nearly 200 years protecting their brand identity.

Regulations for Roquefort have been in place for nearly 100 years.

But because decades were allowed to pass without the producers of Gruyere protecting their brand identity, consumers (and an influential judge) perceived it to be a generic word.

In our world of audio entertainment, it’s a reminder to review your station, show, podcast, and streaming channel to ensure that your brand identity is protected at all times, and that no other entity is attempting to infringe upon it. While it requires an investment of time and money, the consequences of inaction may be a far worse fate.

Brand Subtraction: Less May Be More

Let’s say you’re responsible for overseeing a brand. If something is not working, you add something to make it more appealing. Right?

If something is working, you add more things to make it even better. Right?

We’ve addressed this instinct of addition a number of times in our Tuesdays With Coleman blogs. In “Too Many Messages,” Warren Kurtzman illustrated how adding messages to advertisements lowers the likelihood of remembering any single message from the ad. Jay Nachlis alluded to the explosion in entertainment options while quoting Jerry Seinfeld in “Lack of Focus=Lack of Greatness.” HBO’s ascent to juggernaut status happened by focusing on one great show at a time on Sunday nights, which Jon Coleman points out in “Can HBO and Radio Have it All?”

Now, there’s new science to back up addition by subtraction. Inc.’s Jeff Haden refers to a new University of Virginia study that revealed when people attempt to improve something, they default to “additive transformations,” while ignoring “subtractive transformations.”

It’s why a bar owner may think adding Taco Tuesday to his already loaded list of promotions will be just the thing to boost profit margins.

It’s why software developers think adding more features will make their applications easier to use.

And it’s why a radio program director may think adding more music or special features for the sake of quantity will result in more listening and higher ratings.

So, if we know that we’re inclined to add to solve problems, what happens when we’re prompted to subtract to solve the same problems?

When reminded they could remove items or elements, participants in the University of Virginia study were twice as likely to make subtractive changes than additive changes. And the changes were more effective.

Instead of considering what you can add to solve a problem, consider what you can subtract.

How would that focus your radio station’s music message? Or your podcast’s topic? Or one of your streaming service’s channels?

The takeaway is the take away.



A Researcher Goes to Therapy

Hi, my name is David, and I obsess over research questionnaires.

“Hi, David. What exactly are you obsessing about?”

I think about the coding that goes into the data file. What if there’s a typo? What if the answer key is incorrectly ordered? What if there’s a mindless copy and paste and the code we meant to assign an answer doesn’t match the code that was captured in the data file? What then??

“Are these the only things keeping you up at night?”

Besides my dogs? No, I also worry about what devices respondents use to take the surveys. We’d like to think they’re all using a shiny 48-inch OLED monitor, but let’s face it–lots of them are taking it on a small phone with a cracked screen! What if the survey isn’t optimized properly for them??

“That seems like a lot of pressure.”

Oh, it is. It’s been more than 16 years since my last on-air shift, and I still have that off-the-air alarm dream. But more often than not, I think, did I check to make sure that the 5-point on the scale says “love it” and the 1-point on the scale says “dislike it a lot?” Did the introduction for that section say “click the button on the left” when the button is actually at the bottom of the screen when you look at the survey on a smartphone?

In this example from an entertainment study, respondents are asked to read the statement on the left and right, but in this mobile view the statements are on the top and bottom.

Are the questions being asked in such a way that will truly get the unbiased answers we’re seeking???

“Well, are they David?”

(takes a deep breath) Yes. Of course they are.

“And why is that? Doesn’t everyone do that?”

(icy cold stare followed by sarcastic eye roll)

“I see.”

It’s so important to get the right people. I’ll dream about how important the sample is.

So, what should I do about this obsession?

Drawing by Charles Barsotti


“Do your clients worry about this kind of thing? Are they concerned these kind of mistakes will be made and things will be overlooked?”


“And why is that?”

We do it so they don’t have to worry.

(silence, followed by pondering)

So, you uh…think I should…

“Keep obsessing?”



Have a good day.

“Bye, David.”

Harley-Davidson Has More Problems Than Tariffs

Tuesdays With Coleman

Last week, Harley-Davidson announced it would start shifting production of some of its motorcycles overseas. The company cited new tariffs from the European Union as its rationale. A prepared statement reads, “Harley-Davidson maintains a strong commitment to U.S.-based manufacturing which is valued by riders globally. Increasing international production to alleviate the EU tariff burden is not the company’s preference, but represents the only sustainable option to make its motorcycles accessible to customers in the EU and maintain a viable business in Europe. Europe is a critical market for Harley-Davidson.”

Unfortunately for Harley-Davidson, tariffs are just another bump in the road for a company that needs to focus, like all brands do, on what’s most important – its base position.

What is Harley-Davidson’s base position? What’s the core foundation of its business?

Just taking a shot here, but how about something like: “American-made, big, loud motorcycles”?

So, announcing you’re moving production – even some production – out of the United States is a major brand violation.

But it wasn’t the first violation.

A few months back, Harley-Davidson announced it would debut its first electric motorcycle in 2019.

Is that a brand violation? Take a look at some of the comments at the bottom of the article that appeared in The Verge announcing the news:

That would be like Apple saying you should buy their newest iPhone because you can install Android on it. Might work for some other motorcycle brands. This is Harley-Davidson.

We’re talking about Harley-Davidson, a company who tried to trademark the sound of their idling motorcycle engines. It’d be a hell of a 180 for them to advertise how quiet the bike is.

It’s just not really their kind of advertising, being in harmony with nature, listening to the wind as you silently cruise across the countryside.

For consumers to love the brand, they have to clearly understand the brand.

If you want and can afford a luxury car yet want to go all-electric, chances are you’re not going to buy a Chevy Volt. Or a Nissan Leaf or even a BMW i3.

You’ll probably want to buy the one that has consumers happily plunking down $1,000 to get on a minimum 12-18 month waiting list.

And why is Tesla so hot?

Because sporty electric cars is what they do. It’s their base position.

Tesla can add different kinds of models. They have a high-end model, an SUV, and the waiting list is for the more affordable Model 3. But none of the models deviate from the base position. They are all sporty electric cars.

Will consumers go for the Tesla pickup truck supposedly coming in a few years? Tough to say, but it certainly won’t look like the pickup trucks we’ve always been used to seeing. If it stays true to the Tesla brand, it could work.

Harley-Davidson’s challenge is real. Sales of Harleys are declining. The Harley consumer is aging out, while the younger demographic wants smaller, more affordable bikes. Indian, Ducati and Triumph are experiencing growth and success. Meanwhile, the legacy Harley-Davidson brand stands for large, loud, expensive bikes.

How do you adapt the brand to appeal to a younger market without compromising the base position?

One step would be to launch the electric motorcycle under another brand name.

Harley-Davidson is trying something worth noting – the Jumpstart Motorcycle Experience.  This simulator, which utilizes an actual Harley, is taken by many HD dealers to shows, festivals and expos with the goal of bringing new riders into the fold. Their goal is two million riders in the next 10 years.

You can see how this issue can translate to other brands and businesses, and it certainly applies to media.

Radio stations, morning shows, podcasts and television shows for example, all have to evolve to stay relevant.

Research can play an important role in helping to keep track of its audience’s tastes and how they evolve. Focus groups is a great way to help brands learn how their audiences perceive them and how their tastes correlate to the brand in their own words.

One thing is certain. While evolution is inevitable, the evolution must stay true to the brand’s base position. Changing brand images takes an incredible amount of time, especially for one so strongly cemented.

Harley-Davidson is the American-made, big, loud motorcycle brand.

Last I checked, people still like big, loud American things.