by K.T. Weaver, SkyVision Solutions

Key Article Point

Based upon the positive biases that are inherent in energy consumption feedback studies, there is considerable doubt on the validity of estimates that smart meters can deliver the promised energy savings.

Introduction

From a public policy perspective, smart meters are primarily promoted on the basis that they will result in consumers reducing energy consumption by providing them with feedback.  As stated in one study [1]:

“Smart meters offer possibilities for providing immediate and frequent feedback on household energy use via different means such as websites, mobile phones, and home displays [, and] it is important to study which type of feedback (e.g., financial, environmental, or social comparison feedback) is most effective to encourage sustainable energy behavior, and under which conditions these changes are most likely.”

Published studies going back some 20 years or so report that direct and indirect feedback to consumers on their energy consumption can result in savings anywhere from 0 to 15 %.  The mechanisms for feedback that were studied include smart meters or similar devices or other mechanisms as simple as “enhanced billing” for the customer.  As explained in a 2006 review conducted by Sarah Darby from the United Kingdom [2]:

“The norm is for savings from direct feedback (immediate, from the meter or an associated display monitor) to range from 5-15 %.  The role of the meter is to provide a clearly-understood point of reference for improved billing and for display.  If there is no separate, free-standing display then the meter must also be clearly visible, within the building.”

“Indirect feedback (feedback that has been processed in some way before reaching the energy user, normally via billing) is usually more suitable than direct feedback for demonstrating any effect on consumption of changes in space heating, household composition and the impact of investments in efficiency measures or high-consuming appliances.  Savings have ranged from 0-10 %, but they vary according to context and the quality of information given.”

In the United States, an often cited chart pertaining to feedback on electricity use is the one presented below which was prepared in 2010 by the American Council for an Energy-Efficient Economy (ACEEE).  The chart is represented as summarizing the results of a review of 36 feedback programs “throughout the United States (and other developed countries)” during the period of 1995 through 2010.  As stated in the ACEEE report [3]:

“There are small, back-of-mind behavioral shifts that can save large amounts of energy, because consumption varies dramatically between comparable buildings, depending on how their occupants behave.  Based on the findings of 36 residential pilot programs, the American Council for an Energy Efficient Economy (ACEEE) has reported that 12 % savings are possible for programs with real-time information and feedback.”

“As summarized in the chart below, these initiatives are opening the door to potential savings that, on average, have reduced individual household electricity consumption 4 to 12 % across our multi-continent sample.”

The ACEEE chart implies the following in terms of the average expected household electricity savings from smart meter enabled feedback to the consumer:

• 8.4 % savings from daily or weekly feedback to the consumer based upon access to granular data that may be somewhat delayed in nature by accessing a web portal maintained by the utility company or a third-party;
• 9.2 % savings from “real-time” consumption feedback of the type that might be viewed through an in-home display (IHD) that communicates with a smart meter;
• 12 % savings when real-time feedback on overall consumption is combined with information down to the appliance level of what is being used in the home.

As stated at the beginning of this article, from a public policy perspective, smart meters are promoted on the basis that feedback to consumers (enabled by smart meters) will help them conserve electricity.  Current policy decisions are in fact rooted in the types of numbers presented in the reviews conducted by Darby [2] and the ACEEE [3].  Are these numbers correct?  What do they really mean?  Are the published reviews misrepresented and are they biased?

My conclusion is that the results of studies showing energy savings from direct and indirect feedback mechanisms are extremely biased, so much so that the entire premise for how smart meters will help consumers save money and energy is completely undermined.  Simply stated, smart meter programs are based upon biased and false assumptions on how much energy can be conserved by providing consumers with so-called “eco-feedback.”

ACEEE 2010 Review Makes No Reference of “Bias” as a Possibility

To start in our discussion of possible bias in energy feedback pilot programs, we need only to search the 140-page ACEEE document for mention of the word “bias.”  The word does not exist in the document.  The ACEEE review is purported by its authors to be a “meta-review for household electricity-savings opportunities” and a “systematic assessment” of residential sector feedback studies.  The document states that data was “sliced and diced in different ways.”

Yet any serious and thorough systematic assessment would have discussed possible bias in the selection criteria for the studies included in the review or at least have explained how bias might have affected the results.  As stated in the published paper [4] quoted below, here is how “bias” should be treated during a systematic review of studies:

“A well-conducted systematic review attempts to reduce the possibility of bias in the method of identifying and selecting studies for review, by using a comprehensive search strategy and specifying inclusion criteria that ideally have not been influenced by a priori knowledge of the primary studies.”

“When examining the eligible criteria for study inclusion, the reader should feel confident that a potential bias in the selection of studies was avoided.” [4]

Since the ACEEE failed to even discuss possible bias in the studies reviewed in its 2010 document on residential feedback programs, one can reasonably conclude that the review itself is biased and of poor quality.

Studies of “Opt-In” Volunteers as “Enthusiasts”

All pilot studies that I have reviewed consist of “opt-in” volunteers but yet the ACEEE chart is labeled as average household electricity savings of 4 to 12 %.  These opt-in volunteers do not represent “average” consumers and therefore the chart is extremely misleading.  The consumers participating in feedback studies are effectively “enthusiasts” who desire to see if they can reduce their energy consumption during the course of the study.

Of course, a smart meter would not be necessary for someone to reduce their energy consumption, but the point here is that the study participants are not “average” consumers.  Therefore, the estimated savings includes considerable positive bias.  One study from 2014 [5] estimates that 20 % of household consumers are “enthusiasts” who would happily use energy consumption feedback in an attempt to reduce electricity usage if given the opportunity.  I think this number is high but for the sake of argument, let us assume it is correct.  To adjust ACEEE review results to account for this factor, you would multiply the 4 to 12 % range by the 0.20 number to arrive at a better estimate for the potential savings expected for the general population.  Thus, in this example, the potential average household savings would be reduced from 4 to 12 % to 0.8 to 2.4 %.  This is the type of calculation that is necessary so as not to overestimate the potential benefits of providing smart meters to all consumers (given the substantial financial costs for deployments).

Hawthorne Effect – Study Participants Affected by Knowledge of Being in a Study

Another primary source of positive bias in feedback studies is the Hawthorne effect, where participants who know they are in a study may change their behavior solely based on the additional attention they are receiving from the researchers.  The study creates an artificial environment that may not be indicative of real-world situations.

In 2013 a study was published on the Hawthorne effect and energy awareness [6]. According to the study:

“We find evidence for a ‘pure’ (study participation) Hawthorne effect in electricity use.  Residential consumers who received weekly postcards informing them that they were in a study reduced their monthly use by 2.7 % … — even though they received no information, instruction, or incentives to change.”

Yes, that is right, a group of consumers were sent postcards indicating they were part of a study on how much electricity they used in the home.  They received no feedback information on their usage and no incentives to change, but yet the consumer awareness that they were being studied resulted in an average 2.7 % reduction.

The authors of the study in [6] comment that:

“… if awareness alone can improve performance in contexts where people require no additional information, we might retire the ‘Hawthorne effect’ in favor of a ‘Hawthorne strategy’ of reminding people about things that matter to them but can get neglected in the turmoil of everyday life.”

Based upon accounting for the Hawthorne effect, what additional percentage of energy savings estimated by the ACEEE review would be eliminated as having nothing to do with feedback mechanisms enabled by smart meters?  Could we save billions and billions of dollars by periodically sending consumers postcards reminding them to save energy, rather than installing smart meters that are accompanied with various feedback mechanisms, such as IHDs and/or web portals?

Persistence of Savings Potential over Time

Another source of bias in the review of energy feedback studies is citing results that may not be valid over time.  Most research on energy feedback consists of short-term studies, and therefore there is little evidence on the long-term effects of eco-feedback.  Furthermore, there is some evidence that after an initial period of exposure to eco-feedback the tendency is towards a reduction in the attention provided to the feedback leading to behavioral relapse [7].

Kathryn Buchanan, et.al., in a 2015 published paper [8] found that smart meter and IHD feedback provided a “novelty factor” but that interest “wore off after a time.”

“We found that initially households enjoyed the novelty factor of the meters, comparing how much electricity different appliances used,” explained Kathryn, “but the interest in the meters wore off after a time.” [9]

One study published in 2013 attempted to study energy feedback over an extended period of time (52 weeks).  After one year of deployment, “it was not possible to see any significant increase or decrease in the household consumption” [7]:

“Our findings show that after 52 weeks there was no significant reduction in energy consumption but also no increase.  Our results contradict the literature that suggests a positive impact of eco-feedback on energy consumption.  We argue that such conclusions could be based on typical short-term (2 or 3 week) studies, which are not long enough to capture the relapse behavior pattern after the novelty effect of the eco-feedback.

We recognize that further research is needed to isolate the relationship between consumption and eco-feedback but when huge investments in smart grids and eco-feedback technologies are under way it would be important to deploy more long-term studies that investigate these results further.”

Darby in her 2006 review [2] asserts that new behaviors formed over a three-month period or longer are likely to persist, but she acknowledges that “changes are likely to fade away” if continued feedback is not maintained to encourage sustained changes in behavior.  In general, Darby concludes:

“Persistent feedback promotes persistent conservation behaviour.”

The results I have just provided above are not necessarily contradictory.  I believe what happens is that consumers are less likely to monitor the feedback mechanisms enabled by smart meters as the novelty factor wears off over time.  That is, “persistent feedback” just becomes less effective over time.

Publication Bias

Another source of possible bias that must be acknowledged is “publication bias.”  Darby [2] and the ACEEE [3] reviewed studies that have been published with positive results.  How many studies were conducted that did not provide positive energy savings and therefore were not published?   We don’t know what we do not know.  Unfortunately, it is a possibility that proponents of smart meters have not published results that are not favorable to their cause.

I discovered a form of publication bias in a widely publicized result in 2015 that “PG&E Pilot Yields 7.7 % Energy Savings.” [10] [11] [12]  Here are some quotes:

“Bidgely is an analytics company helping utilities drive greater efficiency and demand side savings.  This week the company announced study results showing that select participants of a Pacific Gas & Electric customer pilot group achieved up to 7.7 % energy savings, with about 80 % of participants saying they would recommend the program.” [11]

“The latest effort by California utility Pacific Gas & Electric (PG&E) and partner Bidgely yielded up to 7.7% energy savings among some 850 participants in a pilot program.  The results were announced recently and highlight one of several methods aimed at energy load disaggregation.” [12]

Do the results quoted above sound impressive?  Would it surprise you to know that the full results of the PG&E study were not promoted or advertised?   The overall results of the pilot study were actually quite dismal.

As reported by Nexant that prepared the report on the pilot study, no statistically significant reduction in consumption was found for most of the participants in the study [13].  One subgroup of users on a Time-of-Use (TOU) tariff saved 7.7 %.   Here is a table in the executive summary of the report indicating how the reported energy savings varied among the various groups:

More telling are the following excerpts from the Nexant report [13]:

“Caution is indicated in interpreting the results of this study because the customers that were recruited to participate in the pilot are among the most highly engaged of the PG&E residential customer base.”

“Additionally, all customers in this pilot are volunteers.”

“A final and also important limitation on the conclusions drawn from this study is that it was conducted for a very short period of time.  Changes in energy savings due to HAN devices may occur when the device is in the home for an entire year or more, and these changes could be materially different depending on the technology.”

“This study finds that the HAN devices yield weak, if any, additional demand response load impacts for SmartRate customers or on-peak load reductions for TOU customers.”

“The only group that shows statistically significant changes in energy usage is the Schedule E-6 TOU customer group.  On average, E-6 TOU HAN pilot participants show a 7.7% reduction in monthly electricity consumption. … The two other customer groups, SmartRate and EV TOU, showed energy savings of 0.8% and 1 %, respectively, but neither of these energy savings estimates are significantly different than zero at the 90% level of confidence.”

“Finally, the fact that no statistically significant incremental DR load impacts could be detected among more than 1,000 SmartRate HAN pilot participants should prompt a re-examination of whether DR event alert should continue to be supported by the HAN platform.”

Do the actual Nexant report results look anything like the public reporting for the pilot results at websites at [10] [11] [12]?  They do not, and this PG&E pilot study therefore represents an excellent example of publication bias.

Nexant, to its credit in the full report, highlighted many limitations of the report such as “highly engaged” customers, “volunteers,” “very short period of time” for the study, and “weak, if any, additional demand response load impacts.”

Conclusion

This article has revealed many positive biases that exist for energy feedback studies that supposedly support the deployment of smart meters.  There are published studies that were conducted years ago that unfortunately formed the basis for policy makers to support widespread deployment of smart meters with the expectation that consumers would significantly reduce consumption.

Smart meter programs are therefore based upon biased and false assumptions on how much energy can be conserved by providing consumers with so-called “eco-feedback.”  In this article we have elaborated on several of the inherent positive biases that were part of prior studies that indicated energy savings as high as 12 to 15 % based upon smart meter enabled feedback mechanisms.  These biases include:

• Lack of discussion of potential bias in systematic reviews of energy feedback studies
• “Opt-in” nature of pilot studies where participants were “among the most highly engaged” utility customers
• The Hawthorne effect
• Little recognition that potential savings fade over time due to a “novelty factor”
• Publication bias

Based upon the above positive biases that are inherent in past energy feedback studies, there is considerable doubt on the validity of estimates that smart meters can enable energy savings of up to 12 to 15 %.  In fact, one might argue that for the general population that the energy savings will be negligible, resulting in the unnecessary and wasteful expenditure of billions and billions of dollars for smart meter deployments.

In a subsequent article (Part 2), I will highlight the results of more recent systematic reviews of energy feedback studies that provide additional evidence that smart meter deployments are of little value to the consumer.  Objective evidence supports my prior article that the Vast Majority of Consumers Suffer a Financial “Net Loss” with Smart Meters [14].

References for this Article

[1] “Understanding the Human Dimensions of a Sustainable Energy Transition,” by Linda Steg, et.al., Frontiers in Psychology, June 2015, available at http://journal.frontiersin.org/article/10.3389/fpsyg.2015.00805/full

[2] “The Effectiveness of Feedback on Energy Consumption,” by Sarah Darby, University of Oxford, April 2006; available at https://skyvisionsolutions.files.wordpress.com/2016/05/effectiveness-of-feedback-on-energy-consumption-darby-2006.pdf

[3] “Advanced Metering Initiatives and Residential Feedback Programs: A Meta-Review for Household Electricity-Saving Opportunities,” American Council for an Energy-Efficient Economy (ACEEE), Report Number E105, June 2010; available at https://skyvisionsolutions.files.wordpress.com/2016/06/aceee-ami-feedback-programs-2010.pdf

[4] “Systematic review and meta-analysis: when one study is just not enough,” by A.X. Garg et.al., Clinical Journal of the American Society of Nephrology, vol. 3, no. 1, pp. 253–260, January 2008; available at http://cjasn.asnjournals.org/content/3/1/253

[5] “20∶60∶20 – Differences in Energy Behaviour and Conservation between and within Households with Electricity Monitors,” by Niamh Murtagh, et.al., published March 18, 2014; available at http://dx.doi.org/10.1371/journal.pone.0092019

[6] “The Hawthorne effect and energy awareness,” by D. Schwartz, et.al., Proceedings of the National Academy of Sciences, vol. 110, no. 38, 2013; available at http://www.pnas.org/content/110/38/15242

[7] “Understanding the Limitations of Eco-feedback: A One-Year Long-Term Study,” by Lucas Pereira, et.al., Human-Computer Interaction and Knowledge Discovery in Complex, Unstructured, Big Data; Volume 7947 of the series Lecture Notes in Computer Science pp 237-255, 2013, at http://link.springer.com/chapter/10.1007%2F978-3-642-39146-0_21

[8] “The Question of Energy Reduction: The Problem(s) with Feedback,” by Kathryn Buchanan, et.al., Energy Policy, Volume 77, February 2015, Pages 89–96; available at https://skyvisionsolutions.files.wordpress.com/2014/12/the-question-of-energy-reduction.pdf

[9] “Smart Meter Rollout a Waste of Money, According to Study,” SkyVision Solutions Blog Article, December 2014, at https://smartgridawareness.org/2014/12/21/smart-meter-rollout-a-waste-of-money/

[10] “PG&E Pilot Yields 7.7% Energy Savings,” April 2015, at http://www.bidgely.com/blog/pge-pilot-yields-7-7-energy-savings/

[11] “How customer empowerment is helping utilities cut ‘energy fat’,” March 2015, at http://www.utilitydive.com/news/how-customer-empowerment-is-helping-utilities-cut-energy-fat/379561/

[12] “PG&E-Bidgely Pilot Yields Energy Savings, Now It Needs to Scale,” April 2015, at http://www.navigantresearch.com/blog/pge-bidgely-pilot-yields-energy-savings-now-it-needs-to-scale

[13] “HAN Phase 3 Impact and Process Evaluation Report,” prepared by Nexant regarding Pacific Gas and Electric Company (PG&E) Home Area Network (HAN) Phase 3 Pilot, December 2014; available at https://skyvisionsolutions.files.wordpress.com/2016/06/pge-han-impacts-and-savings-report_final.pdf

[14] “Vast Majority of Consumers Suffer Financial ‘Net Loss’ with Smart Meters,” SkyVision Solutions Blog Article, February 2016, at https://smartgridawareness.org/2016/02/16/consumers-suffer-financial-loss-with-smart-meters/

Copyright Notice © SkyVision Solutions and Smart Grid Awareness, 2013 – 2016.  Unauthorized use and/or duplication of original material from this site without express and written permission from this site’s author and/or owner is strictly prohibited.  Excerpts and links may be used, provided that full and clear credit is given to SkyVision Solutions and Smart Grid Awareness with appropriate and specific direction to the original content.