Inside progressives’ message testing machine
Inside progressives’ message testing machine
Open Labs’ Ali Mortell shares why + how Democrats can use data to inform communications strategy
Welcome to Campaigner, a weekly newsletter exploring the tactics that drive winning political campaigns and highlighting the players pushing the buttons. Produced by Arena & FWIW Media.
Understanding whether a political ad or message is resonating with the intended audience is an incredibly difficult thing to measure. Over the years, various methods and measurement tools have been used by Democratic campaigners to help inform their framing of issues and messages to the public. Progressive research and testing hub Open Labs has helped lead the way on the Left, measuring the persuasive impact of political ads and advising clients on how to approach certain topics.
For this week’s Campaigner, FWIW’s Kyle Tharp spoke with Ali Mortell, Research Lead at Open Labs, on why she thinks this type of data-comms is important.
Q&A with Ali Mortell, Open Labs
Campaigner: How’d you get your start in politics?
Ali Mortell: I attended school in DC during the Obama administration and I was politically active on campus. I applied to intern in the White House mail room because that was the only office that offered part-time internships - but lo and behold, I was offered a full-time internship in the Office of Digital Strategy. My political path really took off from there.
Campaigner: …And now you’re at Open Labs - how’d you end up there and what made you interested in that work?
Ali Mortell: I've been with the team at Open Labs for about a year now. I had been working in the Democratic data space specifically since the 2016 election cycle, and had really run the gamut of the different ways that you can utilize data and analytics on campaigns and in political environments. Early on, I knew that my greatest passion was what I refer to as data-comms, where you're using data to inform communications strategy. So, I eventually chatted with the OpenLabs team, it was a good fit, and it’s been a really cool year working with the team so far.
Campaigner: Open Labs is an organization that operates kind of behind the scenes in Democratic politics. Can you explain what your team does for readers who are unfamiliar?
Ali Mortell: Sure, so we primarily focus on quantifying public opinion, but also studying how to change public opinion. Polling and message testing are really our bread and butter. I often refer to what I do as “data-comms,” because a lot of the strategic advice that we are giving to our clients is based on what they should be talking about, what they should be emphasizing, analysis of effective surrogates and effective language to use.
Our projects are generally pretty quick and have a fast turnaround. On any given day, we may have four to five different experiments that we're working on, launching, analyzing, or presenting.
Campaigner: My experience with your work has been around testing TV and digital ads. Is that the majority of your day-to-day, or are there other types of message testing that your team is doing?
Ali Mortell: I would say that written message testing has been a really important core of the work that we've been doing. We absolutely do a lot of ad testing, but a lot of times, really interesting, insightful research comes from a comparative corpus of messages that we've developed, that are directly comparable to each other, that are similar in length, that use the same intro. We’re able to provide insight on how a particular message fares against a thousand, two thousand other messages that we've tested over the course of the past year. That allows us to detect this really interesting and meaningful pattern of what are the most effective things for us to be talking about if we’re trying to persuade more people to vote Democratic.
We can learn, for instance, what types of subjects are effective for us, but not quite as effective as other core messaging… what messages are actually causing backlash for us, and pushing voters away.
Because we have this large body of work, when someone comes to us asking for example, about what have we seen with regards to people's reactions to abortion, we can very quickly and flexibly look at everything that we've tested on abortion. We now have this detailed database of messages on the subject, and we're able to kind of tell what's rising to the top, what kind of falls in the middle, what is effective, but not quite the best thing that we could be saying, and what's hurting us on that particular subject.
That can be useful because there are some issues that are really challenging for Democrats to talk about, but we still have to talk about them. So the recommendations that we provide are tailored to the campaign or the organization that we're working with and what their primary needs are on the testing front.
Campaigner: When you’re doing this type of testing, do you more often see shades of grey? Or do some messages just spike off the charts in terms of their general effectiveness?
Ali Mortell: When you are isolating our work to a particular issue, then you can definitely see certain things really spike. For instance, on a really challenging subject for us like immigration, you might find that the vast majority of messages either cause backlash or kind of hover around zero, or maybe are a little bit effective, but then you might have like two or three angles that really jump out as the most effective.
We're always politically sensitive to the fact that what we see in the testing might not be perfectly aligned with our political reality, and there are good reasons sometimes to push an issue or push a message that quantifiably is not as strong for us.
That’s why it’s so important to be doing all of this work in unison and in partnership with other subject matter experts and to make sure that we have buy-in from our clients that we're not just marching in and telling them, “here's what you have to say.” Instead, we’re telling them “here's what the data is showing, this is helpful context for you as you are making your decisions about your communications approach.”
Campaigner: That’s one thing I was going to ask you about. If politics are about both art and science, how much should campaigners rely specifically on the “science” of campaigning?
Ali Mortell: I have seen campaigns that rely *too much* on the science and I have seen campaigns that don’t rely on it enough. You do need both, but more importantly, you need collaborative relationships between your data teams, your organizing teams, your comms teams and other decision-makers. That’s because the right technical approach might not be the correct political or advocacy approach.
In terms of leaning on data versus not leaning on data, I think it is important to have the data to be able to make informed decisions. But it’s also important to know that data is only one piece of the pie when it comes to decision-making. Sometimes data might show that you shouldn’t talk about a certain subject, but you’re in a political situation where you have to talk about that subject.
The best thing that you can do is to figure out that venn diagram of what are the things to say about this subject that are effective and what are my values on this subject and find the messages that meet both criteria.
Campaigner: Do these kinds of message testing considerations only apply to larger campaigns?
Ali Mortell: I would say that a fundamental reason why our team at Open Labs was developed in the first place was to try to make this work much more accessible to people who have historically had a cost barrier to message testing and ad testing. Any form of data science tooling has historically been extremely expensive for campaigns, and now we're doing this for as low as $375 a message tested nationally. We’re trying to get this type of information in the hands of people who not only have historically been boxed out of this type of analysis, but that are increasingly the ones who are driving the media narrative around what the Democratic party stands for. The media isn’t just looking to institutional Democrats anymore to figure out what to write a story on.
They're also looking at activists and influencers and advocacy groups to craft the narrative on what to focus on. Our theory is that as our politics are becoming increasingly nationalized, the media has more and more sway over how people are developing their opinions on what the Democratic party stands for.
Therefore, it's increasingly important for data scientists in the Democratic party to be focusing on what should we be saying in general about what the Democratic party stands for, what our values are, and what we are collectively fighting for in a way that is effective across demographics. We're trying to hone in on what is effective across the board.
Campaigner: I’m curious about the specifics of your testing processes. What tools do you use for measurement? Is it traditional polling? Digital surveys?
Ali Mortell: Right now, we’re primarily using online surveys that people take on their computers and their phones. We have a very robust set of controls that we use. We’re asking both demographic and socio-cultural questions at the start of our surveys, just to get a really rich understanding who that person is, and trying to account for past biases that we've seen throw off polling in past elections.
Once we've asked those core demographic questions, we then use what we refer to as “augmented randomized control trials.” That’s where we utilize an RCT or a randomized control trial in our survey - where at a basic level, you have a random control group and a random treatment group.
The treatment group receives a piece of content, and the control group receives no content. Then, you ask both of them the same outcome metric question, like, “who are you voting for?” and you look at the difference in the responses between the control and the treatment. That difference is how we understand the impact of a piece of content or a message and whether or not it's working in our favor. Then, after that step, one of the really important things that we do is we collect more data from our respondents on content that they didn't see in the RCT component. This allows us to get a lot more rich information on the effects of this content and how it differs by demographic without having to recruit more people into the survey.
Campaigner: Do you have any examples of results of message testing or ad testing over the past year that have surprised you, or really stuck out to you?
Ali Mortell: Last year in 2021, we really started to see the economic effects of COVID and a real rise in inflation. When this was first emerging the public political narrative, people were still talking in terms of “supply chain issues.” That was a huge talking point over the course of COVID, and everyone knew that there were supply chain issues. Everyone believed that that was happening, and it became an emerging talking point in terms of understanding why prices were going up.
Naturally, Republicans did what they're very good at doing: they rallied and everyone pointed fingers at Biden and said it was his fault. We worked with a group called More Perfect Union, which has a growing share of voice in the progressive space. They identified early on that there’s something going on here that we're not really talking about or hearing out about, which is corporate price gouging. That wasn’t something that we were hearing too many people saying, or journalists reporting on. We met with them and started to do some testing on that to figure out, if talking about price increases and inflation through the lens of corporate price gouging, would be an effective message.
In the public’s eye, people generally understand that corporations do shady things for profit. That's not something that's hard to sell people on, for sure. And so when we tested these angles, we found that they were quite effective.
People understood the increase in our cost of living through the lens of corporations raising prices…while we're struggling to pay for gas and bread and milk, they're raking in record profits. More Perfect Union was one of the early institutions to start emphasizing that in the public, and soon enough, you really saw the broader sort of progressive left start talking about that, and then it kind of escalated into a more mainstream talking point.
The Left flank of the Democratic party was really the one that led the charge on describing that to the public, and what we saw correspondingly in our internal polling is that by early 2022, there was a dramatic increase in the percentage of voters who were placing any sort of blame on corporations for inflation.
That’s one of my favorite success stories because progressives led the charge on this important narrative that a lot of Democrats for whatever reason were really hesitant to embrace.
That’s why I love doing what we do. Smart people come to us with a great idea on an angle that we should be taking and they can work with us to test that assumption.
One last thing from Arena:
Arena just announced their plans to fund and place Arena-trained staffers on critical state leg races in AZ, MI, and PA. Read about their partnership with The States Projects and their plans in Politico>>
These interviews are meant to highlight different voices from across the campaign ecosystem. The views expressed therein are not necessarily reflective of the views of Arena or FWIW Media 🇺🇸