4 Improving Data

Although concerns about adolescent social media use and its influence on life satisfaction are mounting, our empirical understanding of the phenomenon is still predominantly informed by cross-sectional research. Reliable inferences about longer-term effects therefore remain elusive. This is the case, in part, because estimating effects observed within the same person over time is inherently different than probing differences in effects between people at one timepoint. Much of the current literature erroneously infers the former using the latter. In this chapter I use large-scale representative panel data to disentangle such between-person and within-person effects. To this end I find negative cross-sectional associations between social media use and life satisfaction and less consistent smaller reciprocal longitudinal relations. Gender plays a decisive role in this dynamic, as there are only few significant cross-sectional or longitudinal relations in adolescent boys. In contrast, in adolescent girls there are small but statistically significant reciprocal within-person effects 1) Increases in life satisfaction predict lower social media use one year later and 2) Increases in social media use predict lower life satisfaction one year later. This suggests that social media is not, in of itself, a robust predictor of life satisfaction across the adolescent population. Instead, its effects are nuanced, reciprocal, and contingent on analytic judgements made by researchers. Research therefore would benefit from an increased focus on the motives and modes adolescent girls adopt when using social media to fundamentally advance the understanding of social media’s wider influence on the adolescent population.

4.1 Introduction

Does the increasing amount of time adolescents devote to social media negatively affect their satisfaction with life? Set against the rapid pace of technological innovation, this simple question has grown into a pressing concern for scientists, caregivers, and policymakers. Research, however, has not kept pace (Bell, Bishop, and Przybylski 2015; Valkenburg and Piotrowski 2017). Focussed on cross-sectional relations, scientists have few means of parsing longitudinal effects from artefacts introduced by common statistical modelling methodologies (Hamaker, Kuiper, and Grasman 2015). Furthermore, as seen in Chapter 2, the volume of data under analysis, paired with unchecked analytical flexibility, enables selective research reporting, biasing the literature towards finding statistically significant effects (Gelman and Loken 2014). It is important to note that, when compared to other factors affecting youth well-being, the purported links between digital technology use and well-being are still extremely small (Orben and Przybylski 2019a, 2019b). Nevertheless, these modest trends are routinely overinterpreted by those under increasing pressure to rapidly craft evidence-based policies. This is especially the case for social media effects, as social media use is increasingly being highlighted as the kind of digital technology use which causes most concern in the broader population (House of Commons Science and Technology Select Committee 2019). This chapter will therefore focus exclusively on social media use.

Throughout this thesis, and in the scientific literature more generally, our understanding of digital technology effects has been predominately shaped by analyses of cross-sectional associations between technology or social media use measures and self-reported adolescent outcomes. Highly-popular studies highlight modest negative correlations (Twenge et al. 2017), but many of their conclusions are problematic (Ophir, Lipshits-Braziler, and Rosenberg 2019). It is not possible to assume that any reported between-person associations – comparing different people at the same timepoint – translate into within-person effects – tracking an individual, and what affects them, over time (Hamaker, Kuiper, and Grasman 2015). Drawing this flawed inference risks misinforming the public or changing policy on the basis of unsuitable evidence. This chapter therefore uses improved longitudinal data and modelling techniques to disentangle within- and between-person relations linking adolescent social media use and life satisfaction.

4.2 Methods

4.2.1 Datasets and Participants

To examine the diverse links between adolescent social media use and life satisfaction I analysed an eight wave, large-scale, and nationally representative panel dataset (UK Understanding Society, the UK Household Panel Survey, University of Essex, Institute for Social and Economic Research 2018). It entails 12,672 UK adolescents between ten and fifteen years of age and was collected as part of an annual longitudinal panel study sampling 40,000 households in the United Kingdom (England, Scotland, Wales and Northern Ireland, University of Essex, Institute for Social and Economic Research 2018). It is important to note that the number of participants for any of the analyses run varied by age and whether full or imputed data were used (range, n = 539 to 5,492; median, n = 1,699). The sample of households consists of a General Population Sample, an Ethnic Minority Boost Sample, a General Population Comparison sample, and samples previously included in the predecessor of Understanding Society, the British Household Panel Survey. Understanding Society data collection is typically done via face-to-face interview, with trained interviewers visiting households personally.

The youth questionnaire is administered on paper and each adolescent member of a sampled household is interviewed and re-interviewed every year until they graduate into the adult survey at age 16. This study uses seven waves of collected data from 2009 until 2017. The data collection takes place over a 24-month period for each wave, so the annual waves overlap and I determine each wave from when it was first collected. In total, the dataset includes 29,155 individual responses, with each year having between 3,466 and 5,020 responses. Participants had a mean age of 12.54 (s.d. = 1.69), and included 5,881 girls and 5,983 boys.

4.2.2 Ethical Review

The unique and extensive data collection is funded by the UK Economic and Social Research Council and other governmental departments with ethical approval from the University of Essex Ethics Committee (provided in 2007, 2010, 2013, 2014, 2015 and 2016, Knies 2017). In particular the University of Essex Ethics Committee approved all data collection on the Understanding Society main study and innovation panel waves, including asking consent for all data linkages except to health records. Requesting consent for health record linkage was approved at Wave 1 by the National Research Ethics Service (NRES) Oxfordshire REC A (08/H0604/124), at BHPS Wave 18 by the NRES Royal Free Hospital & Medical School (08/H0720/60) and at Wave 4 by NRES Southampton REC A (11/SC/0274). Approval for the collection of biosocial data by trained nurses in Waves 2 and 3 of the main survey was obtained from the National Research Ethics Service (Understanding Society – UK Household Longitudinal Study: A Biosocial Component, Oxfordshire A REC, Reference: 10/H0604/2).

4.2.3 Measures

Adolescents and their caretakers filled out a variety of questionnaires in each wave, some questionnaires were only completed on every second or third wave. For the current chapter, I focus on those questionnaires completed by participants in each wave, which included measures of life satisfaction and social media use. The code for data recoding can be found in the detailed online documentation (http://dx.doi.org/10.17605/OSF.IO/4XP3V).

4.2.3.1 Life Satisfaction

Life satisfaction was measured using six items. The response format was graphical: Individual options were presented as either frowning or smiling faces; the lower the number, the happier the face. The exact questions were as follows: “The next few questions are about how you feel about different aspects of your life. The faces express various types of feelings. Below each face is a number where 1 is completely happy and 7 is not at all happy. Please tick the box which comes closest to expressing how you feel about each of the following things: (a) Your school work?; (b) Your appearance?; (c) Your family?; (d) Your friends?; (e) The school you go to?; (f) Which best describes how you feel about your life as a whole?”. I take the mean of the six items to create another mean satisfaction measure and reverse the scale so that a higher score denoted higher life satisfaction.

4.2.3.2 Social Media Use

The social media use variable was more complicated to code as there was a mistake on the side of Understanding Society, who I found were inconsistent in how they directed children through the study in different years. In total there were two questions about social media use. One was dichotomous: “Do you have a social media profile or account on any sites or apps?”, 1 = Yes, 2 = No. The other was a 5-point scale: “How many hours do you spend chatting or interacting with friends through a social web-site like [bebo, facebook, myspace] on a normal school day?”, 1 = none, 2 = less than an hour, 3 = 1-3 hours, 4 = 4-6 hours, and 5 = 7 or more hours.

In some waves of the study, children who recorded that they do not use social media on the dichotomous measure were automatically coded as none on the 5-item scale and sometimes this was not the case. As there was no perfect solution for remedying this error on side of the Understanding Society team, I implemented a recoding approach. If an adolescent said they do not own a social media account on the dichotomous measure or they do not use social media to interact with friends on the 5-item measure I coded them as scoring the lowest score of ‘1’, for the rest of the participants I took their 5-item measure score that quantifies how much time they spend interacting socially online.

4.2.3.3 Control Variables

As control variables, I included age, maternal employment (0 = unemployed, 1 = employed), number of children of mother, maternal ethnic background (0 = white, 1 = non-white), socializing of mother with adolescent (“How often do you and your [child/children] spend time together on leisure activities or outings outside the home such as going to the park or zoo, going to the movies, sports or to have a picnic?”; 1 = never or rarely to 6 almost every day), support given by the family (“Do you feel supported by your family, that is the people who live with you?”; 1 = I feel supported by my family in most or all of the things I do, 2 = I feel supported by my family in some of the things I do, 3 = I do not feel supported by my family in the things I do), and maternal depression (“How often have you felt downhearted and depressed?”; 1 = none of the time to 5 all of the time).

4.2.4 Statistical Approach

This chapter combines Random Intercept Cross-Lagged Panel Models (Hamaker, Kuiper, and Grasman 2015) with the Specification Curve Analysis framework I adopted in the previous two chapters (Simonsohn, Simmons, and Nelson 2015).

4.2.4.1 Random Intercept Cross-Lagged Panel Models

Using Random Intercept Cross-Lagged Panel Models (RI-CLPM) on longitudinal data I can distinguish between-person from within-person variance when quantifying the relations between social media use and life satisfaction (Hamaker, Kuiper, and Grasman 2015). Differentiating these two sources of variance is important when wanting to investigate true directional effects; it isn’t uncommon to find that between-person and within-person relations are in opposite directions (Dietvorst et al. 2018). To illustrate this, one can imagine investigating people’s typing speed and how this relates to the number of typing errors they make. When comparing participants with one another (i.e. at the between-person level), the relationship between typing speed and typing errors is negative: Those participants who are more skilled and can type faster also commit fewer mistakes. When examining each person individually (i.e. at the within-person level), however, this relationship is positive: Typing faster leads to more typing mistakes. This is also known as Simpson’s paradox (Kievit et al. 2013; Simpson 1951). Random intercept models provide a valuable statistical tool to differentiate between such within- and between-person effects (Dienlin, Masur, and Trepte 2018). Their increasing use in the literature has highlighted how traditional cross-lagged panel approaches have often delivered false results (Dietvorst et al. 2018).

Figure 4.1: Random-Intercept Cross Lagged Panel Model

In this chapter, I specified a RI-CLPM as proposed by Hamaker and colleagues (see Figure 4.1, 2015). I constrained longitudinal effects to be equal across all waves. Doing so provides a measure of the effect of social media use on life satisfaction (and vice versa) that is: (a) More robust (the information of several years are aggregated) and, (b) More comprehensive (one single measure is produced). This procedure builds on the assumption that the effect of social media use on life satisfaction is time invariant, that it does not change from say year 2010 to 2012.

Furthermore, I examined how much variance in the outcome is due to between-person differences versus within-person changes, as with too little variance a RI-CLPM would be unsuitable to the data. Precisely, if there is no within-person variance, there are no causal effects; likewise, if there is only within-person variance or no between-person variance, one could simply run a regular Cross Lagged Panel Model (CLPM). I therefore explicitly calculated the Intra Class Correlation coefficients, which estimate how much variance is due to aggregation on the second level (i.e. in this case on the person level). I found that 30% of the variance in social media use is due to between-person differences. As a result, it is important to run CLPMs with random intercepts, to be able to partial out that variance. At the same time, 70% of the variance in social media use results from within-person changes, which leaves ample room for within-person effects. Similarly, depending on the exact life satisfaction domain analysed, I find that 27-43% of variance is due to between-person differences.

4.2.4.2 Specification Curve Analysis

The SCA approach in this chapter is composed of three analytical steps that provide an analytic and conceptual frame for me to implement all theoretically defensible analysis options, with the goal of exhaustively examining the research question of interest. It is important to note that in contrast to the previous two chapters, the RI-CLPMs chosen in this chapter do not allow for statistical testing of SCA results (e.g. via permutation testing or bootstrapping), I therefore refrain from including such statistical conjectures.

When determining what specifications to include in my SCA, I needed to consider two major dimensions. The first dimension refers to content: For example, one can control for different sociodemographic variables in statistical analyses. The second dimension refers to methodology: For example, one can analyse the same research question using either three waves of data, which would maximise the number of participants, or five waves of data, which would increase the robustness of the estimate. As one can deliberately confound SCAs by including defensible but still unlikely or sub-par analyses, thereby diluting the effect, I have specifically aimed to design equally defensible specifications, which all reach the gold-standard of data analysis aspired to.

I implemented the following specifications in my SCA:

• Life satisfaction measure: I analysed either the mean of all life satisfaction subdimensions (“mean satisfaction”) or the subdimensions individually (7 specifications)
• Number of waves: I built my model either on data of all participants who had successfully completed three, four or five subsequent waves (3 specifications)
• Control variables: I either included no control variables, each of the aforementioned control variables individually, or all control variables (9 specifications)
• Modelling: I either included MLR modelling treating the variables as continuous or WLSMV modelling that does not assume normal distribution and is therefore suited for modelling ordinal data (2 specifications)
• Missing data: I either used original data or imputed missing data using predictive mean matching (2 specifications)
• Gender: I either examined males and females separately or all the data together (3 specifications)

Overall, I ran 2,268 different specifications. Nonetheless, later in the chapter I highlight a specific specification in detail (the Preferred Model) to interpret results in more detail. The Preferred Model is the combination of analytical decisions that I believe is the best approach to model the data because of the data’s characteristics, and the methodological and theoretical literature. This allows me to compare effects (e.g. examine gender differences) and highlight practical and statistical significance in a simpler framework than when taking a whole SCA approach. While I still present the range of results found in the complete SCA to provide information about the effects’ variability in the light of analytical changes, I focus on a single result in complicated parts of the paper. I do not use a Preferred Model in Chapter 2 and 3 as the analysis and interpretation of these pieces of research was simpler and I did not compare effects across conditions like gender.

My Preferred Model uses the following analytical decisions I believe represent the most robust ways of modelling the data:

1. I use data from participants who completed four waves: Examining only three waves would be the absolute minimum necessary for our models (Hamaker, Kuiper, and Grasman 2015) and examining more would decrease participant numbers, thereby reducing power
2. I include all relevant control variables
3. I use WLSMV to account for ordinal scales
4. I implement predictive mean matching to impute missing data (Little 1988)
5. I consider female and male participants separately

4.2.5 Code Availability Statement

Intermediate analysis files and a live version of the analysis code can be found on the Open Science Framework (http://dx.doi.org/10.17605/OSF.IO/4XP3V).

4.2.6 Data Availability Statement

The data that support the findings of this study are available from the UK Data Service (Knies 2017).

4.3 Results

Figure 4.2: Results of a Random-Intercept Cross Lagged Panel Model Specification Curve Analysis relating social media use and life satisfaction. Left Panel: between-person correlations. Middle Panel: within-person effects of social media use on life satisfaction. Right Panel: within-person effects of life satisfaction on social media use. Each point on the x-axis represents a different combination of analytical decisions (i.e. life satisfaction domain, gender, number of waves, estimator, data imputation and control variables). The dashboard depicts what gender and life satisfaction domain the specific combination of analytical decisions analysed; the resulting \(\beta\) is shown in the plot above (red = p > 0.05, black = p < 0.05). For graphics including all analytical decisions see Figure C.1-C.3

I first examined between-person associations (Figure 4.2, Panel 1), addressing the question: Do adolescents using more social media show different levels of life satisfaction compared to adolescents using less? Across all operationalisations, the median cross-sectional correlation was negative (\(\psi\) = -0.13), an effect judged as small by behavioural scientists (Cohen 1992). Next, I examined the within-person effects of social media use on life satisfaction (Figure 4.2, Panel 2) and of life satisfaction on social media use (Figure 4.2, Panel 3), asking the questions: Does an adolescent using social media more than they do on average drive subsequent changes in life satisfaction? and To what extent is the relation reciprocal? Both median longitudinal effects were trivial in size (social media predicting life satisfaction: \(\beta\) = -0.05, life satisfaction predicting social media use: \(\beta\) = -0.02). The number of participants in each analysis can be found in Figures C.4-C.6.

When examining the range of possible specifications, the importance of gender became apparent: Only 16% of significant models in all three panels arose from male data. Across most models (Figure 4.2) the median between-person relation and within-person effects appeared more negative for females, hinting that gender plays an under-explored role in the influence of social media. I therefore conducted a focused analysis of my Preferred Model informed by the extant social media effects literature, statistical best practices, and the data characteristics.

Figure 4.3: Relation between life satisfaction and social media use in female (top) and male adolescents (bottom). Left Panel: between-person correlations. Middle Panel: within-person effects of social media use on life satisfaction. Right Panel: within-person effects of life satisfaction on social media use. Small dots: results of each possible combination of theoretically defensible analytical decisions. Large dots: results of the Preferred Model (white = p > 0.05, black = p < 0.05). The life satisfaction categories examined are satisfaction with school work (school work), satisfaction with school (school), mean of the six life satisfaction items (mean), satisfaction with life (life), satisfaction with friends (friends), satisfaction with family (family) and satisfaction with appearance (appearance).

Although some significant between-person correlations for both genders were in evidence (Figure 4.3, Panel 1), males showed only two significant longitudinal within-person effects: Social media predicted tenuous decreases in satisfaction with life and mean satisfaction (unstandardised bs = -0.08 to -0.04; standardised \(\beta\)s = -0.07 to -0.05; Figure 4.3, Panel 2). For females, however, social media was a predictor of slightly decreased life satisfaction across all domains excepting satisfaction with appearance (unstandardised bs = -0.13 to -0.05; standardised \(\beta\)s = -0.09 to -0.04; Figure 4.3, Panel 2). Furthermore, all domains of life satisfaction, excepting satisfaction with friends, predicted slightly reduced social media use (unstandardised bs = -0.17 to -0.05; standardised \(\beta\)s = -0.11 to -0.07; Figure 4.3, Panel 3).

4.4 Limitations

However, some caution is warranted: When comparing both genders the effects’ confidence intervals overlap, and the reduction in significant effects in males alone is not evidence that the effects are substantial for females (Gelman and Stern 2006) – especially as they are very small in size. Importantly, the yearly interval between measurements in my dataset might not be optimal for testing reciprocal social media effects, underlining how no single study can capture the full causal picture. I also highlight that, as detailed in Chapter 3, self-report measures only partially reflect the objective time adolescents spend engaging with social media (Scharkow 2016), yet they form the foundation of technological assessments included in the best quality datasets informing vital research in this area today.

4.5 Conclusion

The relations linking social media use and life satisfaction are therefore more nuanced than previously assumed: They are inconsistent, possibly contingent on gender, and vary substantively depending on how the data are analysed. Most effects are tiny – arguably trivial; where best statistical practices are followed, they are not significant in more than half of models. Yet some effects are worthy of further exploration and replication: There might be small reciprocal within-person effects in females, with increases in life satisfaction predicting slightly lower social media use, and increases in social media use predicting tenuous decreases in life satisfaction.

With the unknowns of social media effects still substantially outnumbering the knowns, it is critical that independent scientists, policymakers, and industry researchers cooperate more closely. Scientists must embrace circumspection, transparency and robust ways of working that safeguard against bias and analytical flexibility. Doing so will afford parents and policymakers with the reliable insights they need on a topic most often characterized by unfounded media hype. Finally, and most importantly, social media companies must support independent research by sharing granular user engagement data and participating in large-scale team-based open science. Only then will we truly unravel the complex constellations of effects shaping young people in the digital age.

4.6 Acknowledgements

This chapter is based on the published work Orben, A., Dienlin, T. & Przybylski, A. K. (2019). Social Medias Enduring Effect on Adolescent Life Satisfaction. Proceedings of the National Academy of Sciences of the United States of America.

Understanding Society is an initiative funded by the Economic and Social Research Council and various Government Departments, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by NatCen Social Research and Kantar Public. The research data are distributed by the UK Data Service.