«Prepared by BRE on behalf of the Department of Energy and Climate Change December 2013 BRE report number 288851 The EFUS has been undertaken by BRE ...»
Report 11: Methodology
Prepared by BRE on behalf of the
Department of Energy and Climate Change
BRE report number 288851
The EFUS has been undertaken by BRE on behalf of the Department of Energy and Climate Change (DECC).
Report editors and lead authors: Jack Hulme, Adele Beaumont and Claire Summers.
Project directed by: John Riley and Jack Hulme.
Data manager: Mike Kay.
Supporting authors and analysts: Mike Kay, Busola Siyanbola, Tad Nowak, Peter Iles, Andrew Gemmell, John Hart, John Henderson, Afi Adjei, Lorna Hamilton, Caroline Buchanan, Helen Garrett, Charlotte Turner, Sharon Monahan, Janet Utley, Sara Coward, Vicky Yan & Matt Custard.
Additional thanks to the wider team of reviewers and contributors at BRE, DECC and elsewhere, including GfK NOP Social Research, Gemini Data Loggers, Consumer Futures, G4S, Eon, British Gas, and for the input of the Project Steering Group and Peer Reviewers.
Table of Contents 1 Introduction
2 The English Housing Survey and the Energy Follow-Up Survey
3 The 2010 EFUS pilot study
4 Interview survey methodology
4.1 Questionnaire design
4.2 The Briefings
4.3 Sampling and response rates
4.4.1 Interview data weighting
4.4.2 Temperature monitoring weighting
4.4.3 Meter reading and combined meter reading / temperature monitoring weighting.......7 4.5 EFUS data quality
4.5.1 Survey errors
4.5.2 Uncorrected biases in the weighted data
4.5.3 EHS dwelling and household variables
5 Energy and temperature monitoring.
5.1 Temperature loggers.
5.2 Electricity monitors
5.3 Meter readings
5.4 Gas and electricity supplier name and tariff data
6 Scope of the reporting
6.1 Objectives and scope of the headline EFUS reporting.
EFUS 2010-11 Interview questionnaire
EFUS 2010-11 Interview Show cards
1 Introduction The last large-scale national survey to consider the detailed use of heating systems and other sources of energy use in homes was the 1998 Energy Follow-Up Survey (EFUS 1998). In that survey, householders were asked questions about the type and usage patterns of the main and secondary heating systems in their homes, the water heating system and usage, dwelling insulation, lighting, indoor temperatures and use of appliances. Although now more than ten years old, the information collected from that survey is still among the most up-to-date data on dwelling and household energy use available.
The main aim of the 2011 Energy Follow-Up Survey (EFUS) was to collect new data in these areas, in order to update the current modelling assumptions about how energy is used in the home. The 2011 EFUS consisted of a follow-up interview survey of a sub-set of households first visited as part of the 2010/2011 English Housing Survey (EHS). Additionally, a sub-sample of these households was selected to have temperature loggers and electricity monitors installed. A further stage of the EFUS involved the collection of gas and electricity consumption data from meter readings.
This report outlines the survey methodology used in the EFUS including sampling, data collection weighting and data quality.
2 The English Housing Survey and the Energy Follow Up Survey The EFUS is a follow-up to the English Housing Survey. The EHS is a continuous annual survey commissioned by the Department of Communities and Local Government (DCLG). It was created in 2008 through the merger of the Survey of English Housing (SEH) and the English House Condition Survey (EHCS).
The EHS has two primary components:
• A physical survey of a subsample of occupied and vacant dwellings.
In 2010 (the EHS year of which the EFUS sample is a subset) approximately 17,000 households were sampled for the main interview survey, and an ~8,000 dwelling sub-sample of these were selected for the physical assessment1. The interview covered topics such as demography, employment, incomes and housing tenure. The physical survey collects information on dwelling conditions, energy efficiency, disrepair & other physical metrics.
For more information regarding the EHS please see:
https://www.gov.uk/government/organisations/department-for-communities-and-localgovernment/series/english-housing-survey Note that the sample size of the EHS was reduced in 2011 The EFUS 2011 uses a subsample of households who had received both the physical and interview survey. During the EHS interview, households were asked if they would be willing to participate in a further DCLG or DECC study. Those who agreed were selected as potential households for the EFUS.
3 The 2010 EFUS pilot study The requirement for the 2011 EFUS was first highlighted by BRE in 2007 and 2008. BRE proposed to members of the Department of Environment and Rural Affairs (DEFRA) that a survey, similar in nature to the 1998 EFUS, was required to reflect changing patterns in home energy use.
Developments in technology would also allow for the collection of physical data on home temperatures and electricity consumption more easily than in 1998.
Following a process of assessing the feasibility of such a survey, an EFUS pilot was conducted, using a clustered sample of EHS cases, from January to September 2010. The pilot survey was carried out for three main reasons. Firstly, it allowed the testing of the full ‘follow-up’ procedure. This required the rapid acquisition of the EHS sample, re-contact with households, and subsequent matching of the returned data with the EHS. Secondly, it enabled the trialling of the temperature loggers and electricity monitors, and an examination of the performance of this equipment in the field. Thirdly, it allowed the testing and development of the initial draft of the EFUS questionnaire.
a) All feedback on the pilot survey was collated and assessed by the survey development team through series of project development meetings and discussions. These included BRE, GfK
NOP and DECC. Feedback was obtained through four main mechanisms:
b) Direct experience of the survey teams at BRE and GfK NOP. The EFUS survey teams at BRE and GfK NOP were able to draw on their experiences in delivering the pilot survey to improve the main stage of the survey. Interviewer observations in the field. At the interviewer briefing, it was stressed that as this was a pilot survey, interviewers should actively record notes for enhancing the survey in the ‘notes’ field which was part of the CAPI software (the computer assisted personal interviewing ).
c) Specific analysis of the pilot survey data. On receipt of the pilot survey data, BRE examined it to determine the quality of data returned, as well as that the required data was in fact being obtained and to examine the quantity of the data.
d) Interviewer debriefing. Immediately following the fieldwork period a full debrief was held with all pilot survey interviewers. This involved assessing the pilot questionnaire question-byquestion and gathering comments from the interviewers directly. Comments were made on individual questions, with suggestions from both respondents and interviewers for ways to improve the questionnaire for the main stage of the survey. The session was recorded to ensure that the interviewer’s observations were fully captured.
e) Questionnaire review. The pilot survey was then reviewed by the project steering group.
This included members of DECC’s customer insight team, BRE’s social research team and DECC’s science team.
4 Interview survey methodology
4.1 Questionnaire design The final interview survey questionnaire used in the 2010/11 EFUS is shown in Appendix A, together with the scorecards as Appendix B.
The questionnaire for the 2010/11 EFUS has been designed using a mixture of existing questions from relevant published large scale surveys and new questions (integrated into the pilot). Prior to the pilot, an early draft was sent for consultation to DECC and GfK NOP and questions were reviewed by the steering group and social research teams (although direct cognitive testing of the questionnaire was not done). Testing of the questionnaire took place using the 2010 pilot study, with feedback on new questions integrated through the process described above.
For the pilot a total of 106 households were interviewed from a sample of 166 chosen from the first three quarters of the EHS 2009/2010 (April to December 2009). The Interviewers contacted householders first by letter and then by a visit to the home, achieving a response rate of 64%. For the main survey telephone numbers were supplied to the interviewers to boost this response rate.
After the pilot, BRE social researchers conducted a detailed analysis of the pilot interview data. This gave an indication of questions that were not working in their current form. This analysis included an expert review of question form and structure, analysis of the ranges used for ordinal scales, and the investigation of verbatim responses to questions.
In order to maintain the quality of responses, the interview was intended to take an average of 45 minutes. In the field, however, the average length of the pilot survey interview was approximately 60 minutes. The lighting and appliances sections were considered particular time consuming, and respondents and interviewers described them as being quite repetitive. Following the workshops with DECC it was agreed that reductions should be sought in the areas of the questionnaire relating to appliances, lighting and cooking. The appliances section was adjusted to remove questions on use of home computers. The cooking section was simplified to only ask about hobs, ovens, grills and microwaves and the lighting section was greatly reduced in length by using a simpler question structure.
Questions were also reviewed by DECC social researchers (post pilot) who suggested alternatives. At DECC’s request, an attitudes section of the main stage questionnaire was added, which is aligned with pre-existing energy efficiency attitude questionnaires. This was done to enable comparative analysis between surveys, should this be required in the future. A monitoring study of hot water use had recently been completed by the Energy Saving Trust at the time of the EFUS survey, therefore this did not form part of the pilot survey questionnaire. A small section on hot water use, however, was added subsequent to the pilot to aid interpretation of energy consumption and electricity monitoring sample selection process (see section 5.2 below).
The EFUS questionnaire was conducted using a CAPI system. This system was tested extensively by BRE and GfK NOP staff prior to the survey starting. Using CAPI allows the interviewer to run through the questionnaire most efficiently and this is especially important for some of the more complex sections such as heating and lighting. The interviewer also has the option to go back and correct entries after discussion with the interviewee. All the surveys were downloaded and sent to GfK daily allowing monitoring of survey progress.
Several parts of the interview involved topics that interviewers may be less familiar with (e.g. types of heating system). There were a number of questions where interviewers were able to access extra information through the CAPI system such as definitions, or the key differences between different answer options.
As the EFUS took place several months after the EHS, there was a possibility that the previous occupant had moved and a new household occupied the property. The interviewers were instructed that the new households could still be interviewed but that additional data would need to be recorded. Towards the end of the questionnaire, extra questions are asked of the new households to gain information about the age, sex, employment status etc.
4.2 The Briefings All interviewers attended a full day briefing prior to starting field work. Ten briefings were conducted in total between 24th November 2010 and 7th January 2011. The briefings were led by a GfK NOP researcher, with active participation throughout from BRE staff providing expert input into interview topics as required.
The briefing began with an outline of the purpose of the survey, and a brief explanation of the sample. Instructions were given on handling the sample, including strategies for dealing with such a dispersed sample and techniques to maximise response. Interviewers were given a broad overview of the coverage of the questionnaire and then BRE staff explained some of the technical terms that it would be helpful for interviewers to understand, and which they may need to explain to respondents and how to access the additional support information Interviewers then went through a dummy interview using the CAPI system; with each interviewer taking it in turn to ask a few questions, and all had the chance to raise queries at any time. At the end of the dummy interview BRE staff explained the purpose of the temperature loggers and the process of installing them.
4.3 Sampling and response rates The EFUS interviews were carried out by GfK NOP between December 2010 and April 2011 with a total of 2,616 interviews being completed.
BRE supplied 3,288 addresses drawn from the first two quarters of the EHS (April to November 2010) to GfK. This represented all cases from these quarters where the respondent had previously agreed to a further interview (84% of cases). The EHS 2010 is a simple random sample of addresses from across England, with each quarter being a random quarter of the full sample (i.e. there is no clustering by survey period). The quarters passed to the EFUS survey team were chosen as they were available in time to conduct the survey, and were as close as possible to the start of the EFUS interview to minimise the chance of changes to the dwelling or household.
When this sample was considered to be exhausted, BRE supplied an additional 400 cases from the third quarter of the EHS. As the number of interviews had been capped at approximately 2,600, this second sample was not fully utilised before fieldwork was complete. In all 135 interviewers were used on the project, and they were allocated cases based on proximity to where they lived.
Interviewers were supplied with names, addresses and telephone numbers for the households in their assigned sample.