Reliability and Validity of a Self-administered Online Assessment of Intrinsic Capacity: A Singapore Cohort Study
Article information
Abstract
Background
To ascertain the construct validity and reliability of a self-administered web-based assessment of intrinsic capacity (IC). The study design was a cross-sectional analysis using data from a prospective cohort.
Methods
We included data from 6,434 respondents (mean age 65.33±5.81 years; 52.4% women) of the Singapore Life Panel population study who participated in the online surveys in March 2022 and May 2022. Incremental nested factor structures of IC were modelled with confirmatory factor analysis (CFA) and their goodness-of-fit were assessed mainly with root mean square error of approximation (RMSEA), comparative fit index (CFI), Tucker-Lewis index (TLI), and standardized root mean square residual (SRMR). With the most parsimonious model as our eventual factor structure, we further evaluated IC and its domains with reliability indices.
Results
CFA demonstrated construct validity for the second-order factor structure with acceptable overall model fit: χ2(147)=7,696.276, p<0.001; CLI=0.947; TLI=0.938; RMSEA=0.089; SRMR=0.051. Amongst the domains, vitality had highest factor loading (0.889), whereas locomotion and cognition (0.534 and 0.601, respectively) had lowest loadings with the second-order IC factor. All five IC domains and the general IC factor fulfilled reliability thresholds (construct validity [CR] or hierarchical omega ≥0.7; average variance extracted ≥0.5); psychological and locomotion domains have high CR (>0.9), whereas vitality and sensory domains have lower values of CR.
Conclusion
Our study provides proof-of-concept evidence regarding the construct validity and reliability of a self-administered web-based assessment of IC index that can potentially be scalable in other population settings.
INTRODUCTION
The World Health Organization (WHO) 2015 report for healthy ageing1) underscores the cardinal importance of optimizing functional ability in older persons beyond a disease-centric focus. The report further clarified how functional ability is influenced by the interaction between the intrinsic capacity of older persons and their environment. Intrinsic capacity (IC) refers to the composite of all the physical and mental capacities that determine what an older person can be and do.1) Conceptually, IC is composed of five domains, namely locomotion, sensory, vitality, cognition, and psychology. These five domains reflect the underlying biology of IC2,3) and underpin the ongoing development of assessment tools to operationalize IC in clinical settings.2) IC can predict negative health outcomes in older persons, such as disability in activities of daily living, institutionalization, and mortality.4) As such, regular assessments of IC can help to identify at-risk groups such as pre-frailty or early-stage frailty, whereby IC has declined or is at risk of decline, and serve as the prelude to a more in-depth comprehensive geriatric assessment of their health-related and social needs.5)
However, gaps remain that impede the potential for scalability and widespread implementation of IC assessment in practice. One is the reliance on performance-based measures that necessitate the presence of a trained assessor. In particular, the sensory, locomotion and cognition domains are commonly assessed using performance-based measures. Although advances in technology have enabled the integration of self-administered assessment using digital platforms,6,7) it can remain a challenge for older adults who are less digitally savvy or have physical limitations. Thus, nurses and doctors applied the Korean Frailty Index for Primary Care (KPL_PC) on mobile devices to screen and monitor for IC decline and frailty of older adult patients.8) Furthermore, self-administered performance assessment in the cognition and locomotion domains may not be equivalent to assessment of performance tasks performed by a trained assessor. A recent study involving real-life users of the French health system using the Integrated Care for Older People (ICOPE) Step 1 screening tool compared self-assessment using the ICOPE Monitor app to screening performed by a health professional and reported lower agreement in the cognition and sensory domains, especially in those aged above 60 years.9)
As such, a questionnaire-based assessment could provide an option to address this gap with the potential for more widespread implementation using an online or digital platform. Moreover, an online interface would facilitate the automated scoring and storing of IC results, which would allow for immediate access by older adults and healthcare professionals and better tracking of IC changes over time. Although the five IC domains were individually operationalised using self-reported assessments in earlier studies, none has ever exclusively used self-administered online questionnaires to measure all five IC domains. It is also important to establish the validity and reliability of such a modality before it can be considered for widespread application in research or clinical practice.
Therefore, the aim of our study is to examine the psychometric properties of a self-administered web-based questionnaire on IC for community-dwelling older adults in Singapore. Specifically, we ascertain the factor structure using confirmatory factor analysis (construct validity) and evaluate the reliability of IC scores. If our results indicate good psychometric properties, this will provide proof-of-concept evidence for an online self-reported measurement index of IC that can be scalable in other population settings.
MATERIALS AND METHODS
Study Population
Since 2015, the Singapore Life Panel (SLP) study has been collecting self-administered response data to research on successful ageing. The SLP was constructed using a population-representative sampling frame from the Singapore Department of Statistics. Respondents were surveyed in monthly intervals primarily by an internet-based platform (only 5.5% responded by phone).10) Ethical approval for the study was obtained by the Centre for Research on Successful Ageing in the Singapore Management University (Reference: IRB-20-080-A052(720)). IC items were fielded twice in March and May 2022. Out of the initial sample of 15,200 respondents since 2015, 6,692 respondents participated in both surveys in March and May 2022. Inclusion criteria included: (1) respondents with complete, non-missing sets of data items, and (2) ≥55 years of age. Decline in intrinsic capacity often precedes the onset of frailty,5) hence it is important to study the younger age group (55–64 years) where IC, as an integrated concept of functional status, is most salient as an entity for frailty prevention. Hence, we excluded 139 respondents due to missing data items and 119 who were <55 years of age, leaving a final data sample of 6,434 respondents (Fig. 1).
Inclusion and exclusion criteria flowchart for the Singapore Life Panel respondents.
Item Measures of IC
Our survey items comprise existing items from the SLP surveys as well as new items that were added to create the IC domains of locomotion, sensory, and vitality (appetite and food taste). Vitality (exhaustion), cognition, and psychological items were selected from existing SLP survey items. Altogether, there were 19 items: locomotion (ambulation and gait); sensory (hearing and eyesight); vitality (appetite, food taste, and exhaustion); cognition (eight items reflecting cognitive shortcomings); and psychological (four items reflecting psychological well-being over the past one month). Items’ responses were rated on scales ranging from 4 to 6 points and reversed coded as necessary to ensure that higher response values represented positive levels of IC.
The locomotion domain was rated from 1 to 4 based on two questions: (1) do you have any difficulty walking 400 m, and (2) how much difficulty do you have in lifting and carrying 5 kg. Sensory domain was rated from 1 to 5 based on the acuity of hearing and eyesight, with 1 being "poor," to 5 being "excellent." For vitality, we used two items from the Simplified Nutritional Appetite Questionnaire (SNAQ),11) namely respondents’ perceptions on their appetite and food taste. In addition, we included an exhaustion item, “I could not get going”, which had previously been employed to measure exhaustion in frailty scales.12) For cognition, eight items assessed the frequency of experiencing subjective memory complaints13) relating to ordinary tasks (for instance, forgetting people’s names or forgetting where something was last placed). Finally, for the psychological domain, 4 items assessed the frequency of various psychological states in the past 1 month, namely stress, sadness, loneliness, and premonition. These items were reverse-coded and were scored from 1 to 6.
Statistical Analysis
Guided by prior theory, we assessed construct validity by performing confirmatory factor analysis (CFA) to examine one factor, second-order, and correlated factor structures in an incremental nested order.14,15) We proposed the CFA models to be reflective, whereby items are represented as manifestations of the factor construct.16) The choice of estimator was weighted least squares mean and variance adjusted (WLSMV), which makes no distributional assumptions about the observed items, but assumes a normal latent distribution underlying each observed ordinal item instead.17) We assessed goodness-of-fit with the following indices: root mean square error of approximation (RMSEA), comparative fit index (CFI), Tucker-Lewis index (TLI), and standardized root mean square residual (SRMR). We used standard criteria for good model fit—RMSEA ≤0.050 (acceptable fit if 0.050<RMSEA≤0.010), CFI ≥0.950 (acceptable if 0.900≤CFI<0.950), TLI ≥0.950 (acceptable if 0.900≤TLI<0.950), and SRMR ≤0.08018,19)—to choose the most parsimonious model as our direct model of interest.
To assess the reliability of the self-administered web-based questionnaire for IC, we adapted the framework proposed by Cheung et al.20) We evaluated construct reliability (CR) and average variance extracted (AVE) for the first-order IC domains. CR is used to assess the stability of a construct, in particular, with the related items measuring the first-order IC domains.21) AVE values estimate the amount of item variance captured by the construct, which reflects how well the items target the intended construct and no other constructs that may correlate with the items. In addition, for the second-order IC factor, a more appropriate index, omega hierarchical (ωh) was derived to assess its construct reliability.22) The criteria for good reliability are: CR or ωh ≥0.7 and AVE ≥0.5. All statistical analyses were performed using Stata/SE version 17.0 (StataCorp., College Station, TX, USA) and Mplus version 8.8 (https://www.statmodel.com/).
RESULTS
Study Population
The mean age of 6,434 respondents was 65.3±5.8 years, and almost half of the respondents were women (52.44%, n=3,374) (Table 1). The respondents were predominantly of Chinese ethnicity (88.16%, n=5,672), followed by Malay (5.02%, n=323) and Indian (5.14%, n=331). The average years of education of the respondents was 10.5±4.3 years and most of the respondents attained either secondary or tertiary educational level.
Summary of baseline sample characteristics (n=6,434)
Descriptive Statistics of Item Measures
Most respondents rated having no difficulty in the locomotive domain (Table 2). The sensory domain was mostly rated as good for their hearing (45.15%) and vision (48.18%). For vitality items, appetite was mostly rated as average (39.85%) and good (48.83%), and food taste was mostly average (30.45%) and good (56.82%). Additionally, most respondents had little (32.13%) or no exhaustion (44.96%). Across the cognition items, respondents occasionally (19.15%–43.63%), rarely (33.28%–43.50%) or never (13.79%–40.32%) endorsed the cognitive symptoms. Lastly, across the psychological items, the frequency of negative feelings ranged from sometimes (20.25%–25.38%), little (35.25%–40.50%) or none (22.52%–35.08%) over the past month.
Descriptive statistics of item measures of intrinsic capacity
Factor Structure of IC Construct
In Table 3, the one-factor CFA presented poor model fit, which indicated the IC construct was unlikely to reflect a single dimension. Conversely, CFA models that conceptualized IC construct as having five correlated domains (locomotion, sensory, vitality, cognition, and psychological) improved model fit substantially. Of the two five-factor CFAs, the second-order model was more parsimonious than the correlated model. Our eventual second-order model (Fig. 2) achieved an acceptable fit with the data—χ2=7,696.276 (df=147), RMSEA=0.089 (90% confidence interval, 0.088–0.091), CFI=0.947, TLI=0.938, and SRMR=0.052. The five domains correlated well with the second-order IC (factor loadings ranged from 0.534 to 0.889). Amongst the five domains, vitality had the highest second-order loading, whereas cognition and locomotion had the lowest. Not surprisingly, vitality has higher first-order correlations with the other domains (0.481–0.692), whereas the lowest correlation was between cognition and locomotion (0.311) (Table 4).
Model fit indices for various intrinsic capacity CFA models
Second-order five-factor confirmatory factor analysis model of intrinsic capacity.
Average variance extracted (AVE), construct reliabilities, first-order correlations and omega value (second-order)
Reliability of IC Construct and its Domains
For the first-order reliability indices, CR values ranged from 0.768 to 0.924 (≥0.7 recommended minimum), whereas AVE statistics ranged from 0.603 to 0.781 (≥0.5 recommended minimum) (Table 4). For the second-order IC factor, the ωh value was 0.722, which is above the recommended minimum of 0.700. Psychological and locomotion domains have high CR (>0.9), whereas vitality and sensory domains have lower values of CR.
DISCUSSION
The WHO Clinical Consortium on Healthy Ageing 2020 report23) highlighted the importance of monitoring IC as a predictor of adverse outcomes and as an outcome of longitudinal trajectories. By demonstrating the construct validity of a self-administered web-based assessment of the five domains of IC amongst community-dwelling older persons, this study contributes to the body of evidence by providing proof-of-concept evidence about the potential scalability for online assessment of IC in other population settings. To our best knowledge, our study was the first to evaluate the factor structure of IC from a self-administered web-based questionnaire assessment. The eventual second-order five-factor model had acceptable overall model fit, corroborating the construct validity of a second-order IC theoretical construct with multiple first-order domains.3) Furthermore, our results showed all five first-order domains and the second-order IC factor fulfilled the reliability criterion at various satisfactory levels. The strength of our study is the large sample size in a fairly representative population of community-dwelling older persons in Singapore and a comprehensive array of variables, which permits assessment of all five IC domains.
Our study contributes to the burgeoning evidence on the multidimensional factor structure of IC14,15,24) by affirming an analogous multidimensional structure in a self-administered web-based assessment. Specifically, the five domains encompassing locomotion, cognition, sensory function, psychological well-being, and vitality are mapped onto the overarching concept of IC. Similarly, earlier CFA studies examining the IC construct have reported worst fit with the first-order factor structure, and superior fit with second-order and bifactor models.14,15) Taken together, this provides proof-of-concept evidence about the construct validity of IC as assessed by self-reported items, which constitute the different IC domains.
Amongst the five domains, vitality had the highest factor loading onto the second-order IC factor, which was corroborated by higher correlations with the other first-order domains. These results align with the pre-eminence of vitality capacity underpinning the IC construct. The WHO Vitality Capacity Workgroup recently defined vitality capacity as a multitude of interacting physiological systems that reflect three broad subdomains: (1) energy and metabolism, (2) neuromuscular function, and (3) immune and stress response functions.25) As the vitality domain with its broad subdomains of attributes can be challenging to measure fully,26) it is reassuring that vitality as measured by our selected three items demonstrated the highest factor loading. Our first vitality item (I could not get going) corresponded to self-reported fatigue,27) which is a feature associated with physical frailty and characterizes the depletion of physiological reserve capacity (energy and metabolism subdomain). The SNAQ items (loss of appetite and taste) were used to screen for anorexia of ageing,28) which was a precursor of malnutrition in healthy older persons (energy and metabolism subdomain). Furthermore, both fatigue and anorexia represent symptoms that are associated with low-grade proinflammatory cytokines (immune subdomain) involved in the multi-systemic pathophysiological pathways of nutritional status, cognitive decline, vascular ageing, and bone/muscle metabolism.27,29)
In terms of reliability, our self-administered web-based assessment of IC demonstrated acceptable construct reliability (ωh=0.722) as a general second-order construct from the five IC domains. These results were consistent with the reliability findings of the overall IC construct from previous studies, which ranged from 0.67 to 0.78 in earlier studies.14,15) In our study, all five domains fulfilled the minimum recommended threshold of CR >0.730) with a range of values (0.768–0.924) higher than the Chinese study (0.55–0.72)15) and comparable to the English study (0.79–0.83).14) Specifically, psychological and locomotion have high CR (>0.9), whereas vitality and sensory domains have lower values of CR.
Vitality and sensory domains are conventionally difficult to measure, and as corroborated by our study findings, challenging to achieve high reliability using a self-reported questionnaire. For the sensory domain, our results corroborate earlier findings that subjective measurement remains less reliable compared with objective assessments of visual and hearing, such as Snellen’s chart and audio-scope. Furthermore, the broadness of a domain may require a range of measurements to fully encompass the multitude of attributes or subdomains. Of note, measurement of the vitality domain commonly involves performance measures and blood tests, and our self-reported items (exhaustion, loss of appetite, and taste) did not measure the neuromuscular subdomain. Thus, while our vitality items demonstrate good factor loadings, reliability is comparatively lower relative to the other domains. Developing additional self-report items is required to ascertain whether these limitations in the reliability of the sensory and vitality domains can be circumvented.
In contrast, cognition and psychological domains are conventionally assessed with self-reported items in research or practice.31) Not surprisingly, for our self-administered web-based IC assessment, the cognition and psychological domains using items from validated scales demonstrated the highest reliability. Items in the psychological domain were adopted from the CES-D32) and Beck Anxiety Inventory33) and reliably measured the subjective negative affect (feeling stressed, sad, and lonely and fearing the worst) that subsumed a variety of aversive mood states.34) Notwithstanding the high CR of items in the cognition domain, the lower AVE (indicator of the average item variance captured by the construct) alludes to the importance of choice of items to assess subjective memory symptoms. Future studies should evaluate whether the use of other validated scales, such as the AD8, would enhance the item reliability and AVE of cognitive capacity in IC measurement.35) In addition, the locomotion domain is conventionally measured using a combination of performance measures such as the Short Physical Performance Battery36) and handgrip strength, or self-reported items looking at prevalence of falls, mobility or functional impairments assessed with an activities of daily living scale.26) In this regard, the high reliability (both AVE and CR) of the locomotion domain is reassuring. The chosen items (difficulty walking 400 m and lifting and carrying 5 kg) reflect the perception of one’s physical performance associated with the diagnosis of probable sarcopenia,37) and together, both provided a reliable indicator of locomotion.
Some limitations are to be acknowledged. Firstly, as our IC questionnaire comprises new and existing items from the SLP study, the anchoring scale within each item, as well as the number of items within each domain, varied. Nonetheless, it is reassuring that domains which utilize a relatively smaller number of items, such as vitality, exhibit strong correlations with the general IC factor. Future studies of self-report IC scales should aim to standardize the anchoring scale and the number of items within each domain. Secondly, as to be expected in a healthy population, our self-reported items were skewed in varying degrees with a ceiling effect observed in domains such as locomotion. Despite this, the factor loadings of the locomotion items were high. Nonetheless, the use of pre-existing items from the survey panel, which were designed for a separate purpose, can limit good RMSEA, CFI, and TLI fit indices (with the exception of SRMR). Future studies should build upon our study to enhance the panel of self-reported items to further improve model fit in order to more comprehensively assess the construct of IC. Thirdly, due to the cross-sectional design, we did not assess other aspects of validity, such as predictive validity for outcomes. We are also unable to ascertain the change in IC trajectory over time. Lastly, our study population comprises mainly healthy and community-dwelling older adults with more than 10 years of education and reasonable digital literacy skills. Hence, the results of our online IC assessment cannot be generalized to older adults with a different demographic profile, such as those who are older, have fewer years of education, and are less comfortable with digital technology. In addition, the use of self-reported intrinsic capacity via an internet-based platform may not be appropriate for older persons who are frailer, institutionalized, and cognitively impaired. It will be important for future studies to explore how self-reported IC assessments using an online platform can be adapted in terms of delivery or content to facilitate the identification of frail or cognitively impaired seniors who may benefit from a comprehensive geriatric assessment.
In conclusion, our study provides evidence for the construct validity of a web-based IC assessment comprised entirely of self-reported items. We also demonstrated the reliability of the overall IC factor and its five individual domains, albeit with lower construct reliability in vitality and sensory domains. Taken together, our results provide proof-of-concept evidence that underscores the potential for self-reported IC to provide a scalable and convenient avenue for regular online assessment in large population surveys to monitor longitudinal progression for early identification of at-risk older adults with declining IC. However, future prospective studies are still needed to confirm the psychometric properties of self-administered IC questionnaires in different study settings and to refine the reliability of self-reported items, particularly in the vitality and sensory domains.
Notes
CONFLICT OF INTEREST
The researchers claim no conflicts of interest.
FUNDING
This research is supported by The Ngee Ann Kongsi, and by the Ministry of Education, Singapore, under its Academic Research Fund Tier 3 programme (Award reference number MOE2019-T3-1-006).
AUTHOR CONTRIBUTIONS
Conceptualization, YYD, SP; Data curation, LKL, GLLC, CSMT; Supervision, LKL, WSL, WT; Formal analysis, WMRC, LKL, WSL, YYD; Writing-original draft, WMRC; Writing-review & editing, WMRC, LKL, LLGC, CSMT, WT, PS, YYD, WSL.