Background

Rare uses the science of human behavior to drive social change for people and nature. With a diverse team of world-class behavioral and social scientists, designers, and trainers, we connect research and insights from behavioral and social sciences and design thinking to help drive the public’s uptake of environmentally friendly behaviors.

What humans believe about the world is a crucial driver¹ of how we choose to behave in it. Changing these beliefs is, therefore, a complementary strategy to changing incentives and capabilities when trying to shift people’s behaviors, including climate-friendly actions such as choosing solar energy.

This begs the question: Which beliefs must be shifted to accelerate a given behavior’s uptake? Past research² has found that descriptive norms, perceived self-efficacy, and outcome efficacy are highly relevant drivers of climate behavior. Of course, a clear understanding of the journey to behavior adoption can help isolate the specific beliefs necessary to shift a given behavior.

If the correct behavior-driving beliefs are identified, measuring changes in those beliefs can clearly indicate an intervention’s impact along its psycho-social theory of change.

As part of a partnership with Meta, Rare wanted to test whether repeat exposure to social ads on Instagram and Facebook shifts key beliefs important for diving behavior change in two climate-friendly behaviors: adopting rooftop solar and community solar.

What question were we trying to answer?

 Could the Meta Brand Lift Study tool detect small but cost-effective changes in peoples’ beliefs about solar adoption through exposure to our ads?

Study details

From April to June 2023, Rare conducted two campaigns utilizing Meta’s Brand Lift Study (BLS) tools. These campaigns aimed to assess the effectiveness of social benefit advertising in shifting highly relevant beliefs and attitudes surrounding community solar and rooftop solar adoption.

Rare hypothesized that we could positively alter these attitudes and beliefs toward solar adoption through multiple exposures to well-crafted ads.

Research findings

Community Solar Campaign

The community solar campaign addressed renters’ lack of awareness about their access to community solar. Our preliminary research showed that renters often disregard solar offers due to their familiarity with rooftop solar only. The campaign sought to rectify this misperception by increasing renters’ confidence in obtaining community solar.

Rare’s Approach:

• Created six distinctive and creative video ads and six still images to showcase community solar as a relevant option for renters.
• The campaign ran for four weeks from April to May 2023, targeting three areas with community solar availability: greater Chicago, greater Baltimore, and Central New York state.
• The campaign reached 3,594,770 individuals across Meta’s Instagram and Facebook platforms.

• Conducted surveys to gauge respondents’ confidence levels in renters accessing community solar. We asked, “How confident are you that renters can have community solar for their homes or apartments?”

Results:

• Meta’s analysis revealed a 1.8% increase in respondents being “Confident” or “Very Confident” about renters accessing community solar.
• This analysis estimated a 90% chance that the actual impact was between –0.3% and 3.9%.
• A 1.8% increase across all those who viewed the content would represent an increase of 63,898 people being confident that renters could access community solar.
• Meta’s analysis found no evidence that the ad meaningfully increased having heard of community solar (0.05% lift, with a 90% chance of the impact being between -1.5% and 1.6%).

Survey responses were adjusted to minimize non-response bias and ensure that survey participants resembled the campaign audience.

Rooftop Solar Campaign

The solar attitudes campaign aimed to determine the impact of different message framings for encouraging people to adopt rooftop solar. Rare tested three distinct framings centered around climate motivations, personal financial motivations, and a combined approach highlighting both climate and personal benefits.

Rare’s Approach:

• Developed four unique video ads and four still images for each of the three message framings (climate motivation, personal financial motivation, and framing that combined climate and personal financial motivations).
• The campaign ran for four weeks in June 2023, targeting North and South Carolina, where intention to adopt solar is low despite being highly suitable for rooftop solar.
• Similar population distribution among the three message framing groups, averaging 855,782 people per framing.

• Conducted surveys to gauge respondents’ attitudes toward the importance of powering homes with rooftop solar energy. We asked two questions: 1) “How important is it that you power your home with solar energy? and 2) How important is it that people in your state power their homes with solar energy?”

Results:

• No meaningful differences were observed among the three framings regarding the personal importance of solar or its importance to others.
• In our climate motivation framing, Meta’s analysis estimated that the number of people who responded that it was “important” or “very important” to power their homes with solar increased by 1.3%, with a 90% chance that the actual impact fell between -1.0% and 3.6%. The belief that it was “important” or “very important” for other people in their state to power their home with solar increased by 1.1% higher, with a 90% chance that the actual impact was between -1.3% and 3.5%..

• In our personal financial motivation framing, Meta’s analysis estimated that the number of people who responded that it was “important” or “very important” to power their homes with solar increased by 1.2%, with a 90% chance that the actual impact fell between –1.0% and 3.5%. The belief that it was “important” or “very important” for other people in their state to power their home with solar increased by 1.2%, with a 90% chance that the actual impact was between -1% and 3.3%.

• Finally, in our framing exploring personal financial gain and climate gain, Meta’s analysis estimated that the number of people who responded that it was “important” or “very important” to power their homes with solar increased by 0.5%, with a 90% chance that the actual impact fell between -1.9% and 2.9%. The belief that it was “important” or “very important” for other people in their state to power their home with solar increased by 0.02% , with a 90% chance that the actual impact was between -2.4% and 2.5%.

Survey responses were adjusted to minimize non-response bias and ensure that survey participants resembled the campaign audience.

Statistical Limitations of the Brand Lift Study (BLS) Methodology:

The BLS methodology provides a unique opportunity for the strong causal inference of a randomized experiment in the natural setting of actual ad units delivered in a social media environment. However, this methodology is limited in its statistical power. A meta-analysis of 800 social benefit Brand Lift Studies found that the average campaign resulted in a lift of only 0.55%.³ This degree of lift may be incredibly cost-effective for social benefit campaigns given the low cost of intervention and broad immediate reach. However, detecting such small influences requires tools to match.

That same meta-analysis reveals that the average BLS study in their dataset is only powered to detect effects as small as a 6% lift, almost 10% larger than the average lift of 0.55%. Similarly, their analysis also shows that if we accepted all those results that meet a 10% confidence threshold, they estimate that 83% of those results would be a false positive.

This lack of statistical power complicates interpreting the findings from Brand Lift Studies. While our studies show little evidence for impact on some measures, it is unclear if that is due to the ad’s lack of meaningful impact or BLS not providing sufficient statistical power to detect such effects. Such findings highlight the need for tools to detect small but meaningful impacts on social media to inform better the decision of whether to scale a social benefit campaign.

Conclusion

Rare’s solar campaigns, powered by Meta’s BLS tools, yielded valuable insights into transforming public perceptions about community solar and rooftop solar adoption. Acknowledging BLS limitations, these campaigns demonstrated digital platforms’ potential to prompt belief changes and encourage sustainable behavior adoption. Ongoing efforts to refine measurement techniques will enhance the effectiveness of future social impact campaigns.

The Path Forward

Rare remains committed to enhancing testing and measurement methodologies for impactful social benefit campaigns and seeks further collaboration with partners sharing similar challenges.

References

1. Athey, S., Grabarz, K., Luca, M., Wernerfelt, N. Digital public health interventions at scale: The impact of social media advertising on beliefs and outcomes related to COVID vaccines. Proceedings of the National Academy of Sciences (PNAS), (2023). doi:10.1073/pnas.2208110120
2. Valkengoed, A.M. van, Steg, L. Meta-analyses of factors motivating climate change adaptation behavior. Nature Climate Change, 9(2), 158-163. (2019). doi:10.1038/s41558-018-0371-y
3. Athey, S., Grabarz, K., Luca, M., Wernerfelt, N. Digital public health interventions at scale: The impact of social media advertising on beliefs and outcomes related to COVID vaccines. Proceedings of the National Academy of Sciences (PNAS), (2023). doi:10.1073/pnas.2208110120