The Content Validity and Interrater Reliability of Nursing Outcome Classification Self-Care for Toileting among Children with Physical Disabilities

Corresponding Author: Sri Hartini Department of Pediatric and Maternity Nursing, Faculty of Medicine, Public Health, and Nursing, Universitas Gadjah Mada, Indonesia Email: srihartini.psik@ugm.ac.id Background: Nursing Outcome Classification (NOC) Self-care for Toileting is an instrument for measuring the abilities of toileting self-care among children with physical disabilities. However, there has been no cultural adaptation in the Indonesian context, especially among children with physical disabilities, so that the validity and inter-rater reliability of the instrument has not been reported. Testing the content validity and interrater reliability of this instrument is necessary so that it could be used to measure the ability of toileting self-care. Purpose: This study aimed to examine the content validity and inter-rater reliability of NOC Self-care for Toileting among children with physical disabilities. Methods: This was a descriptive quantitative non-experimental research with a cross-sectional design. In the content validity test, this study employed seven experts teaching in nursing academies with a minimum degree of S2/specialist and had the experiences in basic nursing to assess the validity of NOC Self-care for Toileting. As for the inter-rater reliability test, this study involved two raters of nursing students who have passed block 2.4 lecture on “Growth and Development” and 36 children with physical disabilities at a Special School in Bantul, Yogyakarta. The validity test was analyzed using CVR, CVI, and Aiken’s V indexes, while the inter-rater reliability test was analyzed using Kappa and percent agreement. Results: The validity test showed the CVR value range of 0.71-1.00, the CVI value of 0.91, and the V coefficient range of 0.86-1.00. In the inter-rater reliability test, the Kappa value obtained was 0.958, and the percent agreement value was 97.22% for all indicators of NOC Self-care for Toileting. Conclusion: NOC Self-care for Toileting is a valid and reliable instrument for measuring the abilities of toileting self-care among children aged 6-18 with mild to moderate degrees of physical disabilities. Further research can be conducted by involving different respondents and cultures with more than two raters if possible.


Introduction
Based on the 2010 population and prevalence estimates from the World Health Survey and Global Burden of Disease, there were around over a billion children (or about 15% of the world's population) that were estimated to be living with disability; Global Burden of Disease estimates childhood disability prevalence to be 95 million (5.1%) children, of whom 13 million (0.7%) have a severe disability (World Health Organization and World Bank, 2011). The proportion of children with disabilities in Indonesia is 2.5% for children of 5-9, 3.5% for children of 10-14, and 4.2% for children of 15-17 years old (Ministry of Health Republic of Indonesia, 2018). The 2017/2018 report from the Department of Education, Youth, and Sports (Disdikpora) of the Special Region of Yogyakarta (DIY) revealed that the number of physical disability students in Special Schools incontinence, cleaning themselves after toileting, and closing zippers. According to the American Academy of Pediatrics (2012), the number of physical disabilities and illnesses can hinder a child's ability to become fully toilet-trained or easily adjust to bathroom use; children with visual disabilities and those with sight deficiencies experience a disadvantage at several stages of toilet training, and children who are deaf or have difficulty hearing may or may not find toilet training challenging, depending on their ability to communicate. Children with cerebral palsy not only tend to be slow in developing bladder control but may not have enough bladder awareness to begin toilet training at the age of two or three; limited physical activity, undeveloped muscle tone, or medications tend to cause constipation for children with cerebral palsy. Children and teens with disabilities may have urinary and/or fecal incontinence for a wide variety of reasons, including weak muscles and/or interruptions or delays in the signals to the brain that a person needs to void (Macias et al., 2006). For students with disabilities, this can include a lack of awareness that they need to use the restroom until it is too late (National Association for Continence, 2013).
Toileting self-care abilities can be measured using Nursing Outcome Classification (NOC) Self-care for Toileting. NOC is an instrument to measure client status before and after the nursing intervention. The outcome helps evaluate the effectiveness of an intervention given independently or in collaboration with nurses (Moorhead et al., 2013). Each outcome has a definition, a measurement scale, a list of indicators related to the concept and supporting references. NOC Self-care for Toileting is defined as a person's actions to go to the toilet independently with or without the aid of a device measured using 13 indicators with a 5-point Likert scale from severely compromised to not compromised (Moorhead et al., 2013).
It is important to consider the validity and reliability of data collection tools (instruments) when either conducting or critiquing research (Heale & Twycros, 2015). The higher the value of the validity and the reliability of an instrument, the more accurate the data obtained from a study would be (Hayati & Lailatussaadah, 2016). The NOC Self-care for Toileting instrument has been proven to be valid and reliable in the United States and Korea (Keenan et al., 2003;Lee, 2007). In Indonesia, this instrument has been tested for its inter-rater reliability in the population of stroke patients in the hospitals with a kappa value of 0.9347 and a percentage agreement value of 96.3% (Khasanah et al., 2018). If an instrument is developed worldwide in many languages, the translation and adaptation of the culture to new languages and countries are very relevant (Ljungberg et al., 2015). Furthermore, da Silva et al. (2015) stated that NOC was created in America and written in English. Therefore, cultural adaptation is needed before the indicators and results of nursing are used in overseas health services and education. In this regard, the researchers claim that there has been no cultural adaptation of NOC Self-care for Toileting in the Indonesian context. Until now, a tool to measure toileting abilities, namely NOC Self-care for Toileting, has not been proven to be used and has not been reported for the validity and interrater reliability, especially among children with physical disabilities. Accordingly, this study aimed to examine the validity and inter-rater reliability of NOC Self-care for Toileting among children with physical disabilities.

Research design
This was a descriptive quantitative non-experimental research with a cross-sectional design.

Setting and samples
This study was conducted in the Special School in Bantul, the Special Region of Yogyakarta, Indonesia, from October to November 2018. The population was students with physical disabilities. The samples were 36 children with physical disabilities who met the inclusion criteria, including: (1) in the age of 6-18, (2) not menstruating, (3) not having diarrhea or constipation, and (4) not taking laxative medications. The exclusion criteria were children who were not willing to be the respondents and those with multiple disabilities (having more than one disability, for example, deaf-speech impairment, deaf-blind, and physical disabilities-blind).
The content validity of the NOC Self-care for Toileting was validated by seven experts. The researchers recruited the experts who taught in nursing academies with a minimum degree of S2/Specialist and had experiences in basic nursing. The researchers gave the content validity assessment guide sheet, the NOC Toileting Self Care rubric, and the content validity assessment sheet to the experts simultaneously to fill out the content validity assessment of the NOC Self-care for Toileting. The inter-rater reliability test was conducted by two raters (assessors) of nursing students who had passed block 2.4 lecture on "Growth and Development." Two raters directly observed the respondents' self-care ability according to the rubric on the indicators of NOC Selfcare for Toileting.

Measurement and data collection
The NOC Self-care for Toileting was used to measure toileting self-care. This instrument consisted of 13 indicators with 5 levels of assessment criteria, namely: (1) severely compromised, (2) substantially compromised, (3) moderately compromised, (4) mildly compromised, and (5) not compromised. In each assessment criterion (see Appendix 1), there was an operational definition written in the rubric of the instrument adapted from Khasanah et al. (2018).
The cultural adaptation of NOC Self-Care for Toileting was done based on the translation model for cross-cultural adaptation of instruments (Jones et al., 2001;Lee et al., 2009;Sousa & Rojjanasrirat, 2011). First, NOC Self-care for Toileting instrument was translated from the original language (English) into Indonesian by two different translating institutions. NOC Selfcare for Toileting translated by the first translating institution was called translation 1 (T1), and the other one by the second translating institution was called translation 2 (T2). After the instrument was translated, the researcher asked for inputs from the Indonesian literary experts and the advisor concerning the results of the translation of the instrument to obtain the final version of the translation (T12). After the final version of the translation (T12) was agreed upon, a back-translation was done by a professional translating institution.
The next step was to test the content validity of the NOC Self-care for Toileting to seven experts. Meanwhile, the inter-rater reliability test was conducted by two raters (assessors). Before the data were collected, the researcher conducted training and formal meetings with the raters to increase the agreement. The data collection began with explaining and asking about the willingness to be research respondents represented by the respondents' parents. If the children, represented by the parents, were willing to serve as respondents, the parents would be asked to sign the informed consent and fill in the respondents' data. During the data collection, two raters directly observed respondents' self-care ability according to the rubric on the indicators of NOC Self-care for Toileting. Then the two raters conducted the assessment independently at the same time on the sheet provided by the researcher according to each rater's perception.

Data analysis
The content validity was analyzed using the CVR (Content Validity Ratio), CVI (Content Validity Index), and Aiken's V indexes. The CVR is an item statistic useful for rejection or retention of individual items and is internationally recognized as the method for establishing content validity (Wilson et al., 2012). CVI can be used to rate each instrument item in terms of its relevance to the construct. The CVI is CVR mean for all retained items (Hadzaman et al., 2018). When all panelists say that the tested knowledge or skill is ''essential,'' or when none say that it is ''essential,'' we are confident to include or delete the item. It is when there is no consensus that item issues arise. Two assumptions are made, each of which is consistent with the established psychophysical principles. The CVR and CVI values obtained were analyzed based on Lawshe (1975). The CVR values are in the range of -1 to 0 (not good), 0 (good), and 0 to 1 (very good) (Lawshe, 1975). The values of V coefficient were interpreted based on Guilford and Fruchter (1978) as follows: 0.80-1.00 (very high), 0.60-0.80 (high), 0.40-0.60 (medium), 0.20-0.40 (low), 0.00-0.20 (very low), and <0.00 (invalid). The interrater reliability was analyzed using Kappa and percent agreement for two raters (Cohen, 1960;Shweta et al., 2015). The degree of agreement of the Kappa value was analyzed based on the Altman benchmark scale as the following: <0.20 (poor), 0.21-0.40 (fair), 0.41-0.60 (moderate), 0.61-0.80 (good), and 0.81-1.00 (very good) (Gwet, 2014). The Kappa value is accepted if >0.41 (McHugh, 2012). The values of percent agreement value were also interpreted based on McHugh (2012) that the percent agreement can be accepted if the value is more than 80%.

Ethical considerations
The In the data collection process, the researcher explained the aims and objectives of the research to the parents and asked the children's willingness to be the respondents. Parents who agreed to participate in the study completed the informed consent as a sign of consent to participate in the study.

Characteristics of respondents
As presented in Table 1, the total number of respondents in this study was 36 with ages ranging from 6 to 18 years. Most children were aged in the group of 10-13 (44.4%) and 14-18 (44.4%) and experienced a moderate degree of disability (86.1%). Based on the gender, the number of boys and girls was similar (50% each).

Translation and cultural adaptation process of NOC Self-care for Toileting
The translation process of the NOC Self-care for Toileting adopted the theory of cross-cultural adaptation of Brislin (Jones, 2001;Lee et al., 2009;Sousa & Rojjanasrirat, 2011). The translation from the original language (English) to Indonesian was carried out by two professional translation agencies. The results of the Indonesian translation from the two professional translation institutions were then discussed by the Indonesian language and literature experts and research supervisors as input providers. The input providers compared the translation results from the two language institutions and made necessary adjustments to the Indonesian language and cultures.
After receiving input from the Indonesian language and literature experts and research supervisors, a final version of the instrument translation (translated synthesis) was obtained. One final version of the Indonesian language instrument translation that had been agreed upon was then further processed for back-translation in a different professional translation agency. After the back translation was carried out, the researchers with the English language and literature experts compared the back-translated instrument with the original English instrument.

The results of the validity test of NOC Self-care for Toileting
In the validity test of each NOC Self-care for Toileting indicator, the CVR values were in the range of 0.71-1.00. According to Lawshe (1975), the values fall within the very good category. The CVI value was 0.91, so it showed a very good validity value (Lawshe, 1975). Based on the Aiken's V approach, the values of V coefficient were in the range of 0.86 to 0.96 which fall within the very high category (Guilford & Fruchter, 1978). The assessment results of the content validity of the CVR and Aiken's V can be seen in Table 2.

The results of the interrater reliability test of NOC Self-care for Toileting
In the inter-rater reliability test of all NOC Self-care for Toileting indicators, the Kappa value obtained was 0.958. This value showed a very good degree of agreement (Gwet, 2014). The Kappa value was accepted since it exceeds the minimum limit (0.41) (McHugh, 2012). The NOC Self-care for Toileting produced the percent agreement value of 97.22%, so NOC Self-care for Toileting could be accepted as it exceeded the minimum limit of percent agreement value of 80% (McHugh, 2012). In the inter-rater reliability test, each indicator of NOC Self-care for Toileting produced the Kappa value in the range 0.86-1.00. The Kappa values of the 13 indicators showed a very good Nurse Media Journal of Nursing, 11(2), 2021, 259 Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811 degree of agreement (Gwet, 2014) and were accepted since the values exceeded the Kappa value limit of 0.41 (McHugh, 2012). The percent agreement value for each NOC Self-care for Toileting indicator was in the range of 91.67-100%. The value was accepted, for it had exceeded the minimum limit of percentage agreement value of 80% (McHugh, 2012). The results of the interrater reliability test of NOC Self-care for Toileting can be seen in Table 3.

Discussion
This study aimed to examine the validity and inter-rater reliability of the NOC Self-care for Toileting among children with physical disabilities. The validity test showed the CVR value range of 0.71-1.00, the CVI value of 0.91, and the V coefficient range of 0.86-1.00. In the inter-rater reliability test, the Kappa value was 0.958, and the percent agreement was 97.22% for all indicators of the instrument. On the basis of the data from this study and other similar studies (Khasanah et al., 2018), this translation and cultural adaptation process results in a clearer and  (Zamanzadeh et al., 2015). Sireci and Faulkner-Bond (2014) defines content validity as the degree to which the content of a test is congruent with testing purposes. The results of this study indicated that the values of the content validity of NOC Self-care for Toileting among children with physical disabilities were very good and very high with the CVR values in the range of 0.71-1.00, the CVI value of 0.91, and the values of the coefficient V in the range of 0.86-0.96. Based on the explanation, it can be concluded that NOC Self-care for Toileting has been proven to be valid for measuring the abilities of toileting self-care among children with physical disabilities in Indonesia (Meisyarah et al., 2019). This confirms a research finding by Mulyani and Nurjannah (2019) about the content validity test of the NOC on Toileting Self-care among stroke patients, which produced an S-CVI value of 0.96-0.98, indicating that NOC Selfcare for Toileting is valid to measure the abilities of NOC Self-care for Toileting among stroke patients in Indonesia.
A study in Korea showed that the mean Fehring ratio of the NOC Self-care for Toileting was 0.82, indicating that the tool had acceptable nursing sensitivity in Korea (Lee, 2007). A study conducted by Head et al. (2003) also revealed a similar finding regarding the outcome content validity of NOC Self-care for Activities of Daily Living (ADL) by the community health nurses, resulting in an outcome content validity score of 0.8429 and outcome sensitivity validity score of 0.7082. Thus, the results supported the content validity and nursing sensitivity of the study outcomes. Almeida et al. (2010) reported that NOC Self-care for Activities of Daily Living (ADL) was validated by experts. The study regarding toileting as indicator of NOC self-care for activities of daily living (ADL) showed a weighted average (0.76).
Based on Asaad (2004), several factors that can affect the validity include the internal factors of items (unclear direction of items, unstructured vocabulary and sentences, item length, and ambiguity of items), administrative factors and item scores, and nature and group criteria. In this regard, the researchers have considered these factors in this study. Therefore, all NOC Self-care for Toileting indicators produce very good validity values with the CVR values in the range of 0.71-1.00, the CVI value of 0.91, and the coefficient V values in the range of 0.86-1.00.
The inter-rater reliability test showed that the Kappa value and percent agreement fall within the very good category with the Kappa value of 0.958 and the percent agreement of 97.22%. Based on this explanation, the NOC Self-care for Toileting has proven to be reliable to measure the abilities of Toileting Self-care among children with physical disabilities in Indonesia (Meisyarah et al., 2019). A study by Khasanah et al. (2018) concerning the inter-rater reliability test for NOC Self-care for Toileting among stroke patients also showed a Kappa value of 0.934 and a percentage agreement of 96.3%.
A research carried out by Keenan et al. (2003) on the inter-rater reliability test for NOC Selfcare for Toileting in a home care setting in the United States also revealed similar results with percent close to the agreement value of 95%. A study by McHugh (2012) revealed that perfect Kappa value is rarely achieved; however, in this study, several indicators could produce perfect Kappa and percent agreement values (k=1.00 and percent agreement=100%). Of the 13 Toileting Self-care indicators, seven indicators have perfect Kappa and percent agreement values (k=1.00 and percent agreement=100%). The perfect Kappa value (k=1) and the percent agreement value (100%) were shown in the indicators of (2) responds to urge to have a bowel, (4) removes clothing, (7) gets to toilet between urge and evacuation of stool, (8) empties bladder, (9) empties bowel, (11) wipes self after bowel movement, and (13) adjust clothing after toileting. The high value of kappa and percent agreement in this study shows the suitability. Several factors can influence the high Kappa and percent agreement values to obtain good inter-rater reliability values.
Several factors must be considered by instrument designers to increase the potential of an agreement (Burns, 2014;Graham et al., 2012;Kane et al, 2015;Mueller, 2018;Nurjannah et al., 2017;Ragupathi & Lee, 2020). The first factor is rater training. According to Graham et al. (2012), the duration of rater training is also important, and effective rater training requires more than one or two hours. In this study, the researcher conducted training for two raters for four hours in two sessions. In the first session, the researchers explained the data collection procedures and NOC Self-care for Toileting observation sheet to be used and directed perceptions between the Nurse Media Journal of Nursing, 11(2), 2021, 261 Copyright © 2021, NMJN, e-ISSN 2406-8799, p-ISSN 2087-7811 raters regarding the NOC Self-care for Toileting observation sheet for two hours. In the second session, the researcher with two raters conducted a simulation of data collection for two hours.
The second factor is rater selection. According to Graham et al. (2012), the expertise and experience of raters that suit the instruments can increase agreement. Based on a study carried out by Kane et al. (2015), raters' professional background can influence inter-rater reliability. This study employed two raters who have the same educational background, experience, and expertise. Both of them have passed block 2.4 lecture on "Growth and Development" so that it is easy to increase the inter-rater agreement.
The third factor is the accountability of raters' assessment. Based on Graham et al. (2012), rater training does not guarantee the consistency of the assessment conducted by raters, so that re-training is needed. In this study, the researcher always monitored the performance of the two raters and asked them if there were things still not understood. The researcher also reminded them to assess independently so that the results obtained are truly reliable and not biased.
The fourth factor is the instrument rubric design. Graham et al. (2012) stated that the consistency of raters' assessment can be influenced by the instrument rubric design. According to Ragupathi and Lee (2020), rubrics offer the possibility of objective, consistent evaluation minimizing the difference in grades even when multiple raters are involved in the evaluation. In this study, the rubric used was an adaptation of the NOC Self-care: Toileting rubric by Khasanah et al (2018) with modifications and adjustments regarding the samples. This instrument is accompanied by an operational definition of each indicator and has been given a detailed assessment rubric for each indicator scale from severely compromised to not compromised. In addition, this rubric has been validated by seven experts in their field of expertise and produced very good CVR and CVI values and very high V coefficient values.
The fifth factor is the type of scale in the instrument. The best rubrics have three to five descriptive levels to allow for discrimination in the evaluation (Mueller, 2018). This research used NOC Self-care for Toileting observation sheet with 5 scales, namely severely compromised, substantially compromised, moderately compromised, mildly compromised, and not compromised, so that it has more varied scales.
The sixth factor is to consider fatigue when scheduling inter-rater reliability test time. Raters' fatigue may increase errors (Burns, 2014). According to Aslett (2006), rater fatigue can affect the results of instrument reliability. Nurjannah et al. (2017) revealed that a rater who conducts an assessment outside of working hours until exhaustion might affect the stability of the assessment. Researchers scheduled the interrater reliability test time in the morning when they were still fit, and data collection was carried out one to two times a week with a duration of 10-15 minutes for each respondent so that the fatigue factor did not really have an impact on this study.

Implication and limitation
The NOC Self-care for Toileting instrument, according to the results of the present study, is a valid and reliable instrument for measuring the abilities of toileting self-care among children aged 6-18 with mild-moderate degrees of physical disabilities; therefore, the evaluated NOC indicators ensure good reproducibility in clinical practice. Nurses can use this instrument to measure the ability of toileting self-care among children with physical disabilities in clinical practice so that nurses can provide good nursing care. The NOC Self-care for Toileting instrument can also be used in Indonesia because the cultural adaptation in Indonesia has been carried out in this study.
The limitations of this study include data collection techniques for four indicators of defecation in NOC Self-care and the back-translation process that only used one professional translation agency. Despite these limitations, this study offers seven experts who validated the NOC Self-Care for Toileting instrument and conducted direct observations to assess interrater reliability using two raters. The results of this study can help nurses or caregivers in assessing toileting abilities in the community and clinically using the NOC Self-Care Toileting instrument appropriately.

Conclusion
The overall content validity values of NOC Self-care for Toileting among children with physical disabilities are included in the very good and valid category, while the overall inter-rater reliability values are included in the very good and reliable category. The use of NOC Self-care for

Code
Indicator Indicator Operational Definition 031014 Gets to toilet between urge and passage of urine 1: Unable to reach the toilet between the urge to urinate and before the release of urine with the assistance of others and a wheelchair 2: Able to reach the toilet between the urge to urinate and the release of urine with a wheelchair 3: Able to reach the toilet between the urge to urinate and the release of urine independently with the assistance of others 4: Able to reach the toilet between the urge to urinate and the release of urine independently with assistive devices (for example, walking aids, such as crutches or walker) 5: Able to reach the toilet between the urge to urinate and the release of urine independently  Code Indicator Indicator Operational Definition 3: Able to perform 2 of 4 indicators: a) taking water with a water dipper, b) washing the anal area, c) washing the genital area from the front to back, d) putting the water dipper back 4: Able to perform 3 of 4 indicators: a) taking water with a water dipper, b) washing the anal area, c) washing the genital area from the front to back, d) putting the water dipper back 5: Able to perform all indicators: a) taking water with a water dipper, b) washing the anal area, c) washing the genital area from the front to back, d) putting the water dipper back 031008 Gets up from toilet or commode 1: Unable to move from the toilet or elimination aids 2: Able to move from the toilet or elimination aids with the assistance of others 3: Able to move from the toilet or elimination aids with assistive devices (for example: hand rail) 4: Able to move from the toilet or elimination aids independently without any assistive devices and the assistance of others but need direction or supervision 5: Able to move from the toilet or elimination aids independently without any directions or supervisions, assistive devices, and the assistance of others