Validity and reliability of neonatal infant pain scale (NIPS) in neonatal intensive care unit in Viet Nam

 MedPharmRes, 2019, 3 1 
*Address correspondence to Kien Gia To at the Faculty of Public Health, 
University of Medicine and Pharmacy at Ho Chi Minh City; 217 Hong Bang, 
Ward 11, District 5, Ho Chi Minh City, Vietnam; Tel/Fax: (+84) 91 95 11 121; 
E-mails: kiengiato@gmail.com or togiakien@ump.edu.vn 
DOI: 10.32895/UMP.MPR.3.2.1 
 © 2019 MedPharmRes 
MedPharmRes 
journal of University of Medicine and Pharmacy at Ho Chi Minh City 
homepage:  and  
Original article 
Validity and Reliability of Neonatal Infant Pain Scale (NIPS) in Neonatal 
Intensive Care Unit in Vietnam 
Hai Thanh Ngoa, Kathleen Fitzsimmonsb, Kien Gia Toc* 
aVINMEC General Hospital; 208 Nguyen Huu Canh, Ward 22, Binh Thanh District, Ho Chi Minh City, Vietnam; 
bGreenville County Schools, Greenville, South Carolina, USA; 
cFaculty of Public Health, University of Medicine and Pharmacy at Ho Chi Minh City; 217 Hong Bang, Ward 11, District 
5, Ho Chi Minh City, Vietnam. 
Received July 14, 2018: Revised June 11, 2019: Accepted July 04, 2019 
Abstract: Background: The study aimed to culturally adapt and validate Neonatal Infant Pain Scale (NIPS) 
for use in Vietnamese settings. Methods: The original NIPS was translated into Vietnamese using a standard 
protocol. Registered nurses of Neonatal Intensive Care Unit (NICU), Tien Giang General Hospital, Vietnam 
used the Vietnamese NIPS for assessing neonatal pain and then provided feedback on acceptability of the scale. 
Five registered nurses of NICU were randomly selected and used NIPS for assessing neonatal pain while 
watching thirty videos at two times, two weeks apart from each other. Pulse rates per minute and oxygen 
saturation (SpO2) were also recorded for validity evaluation. Intraclass correlation coefficients (ICC) with two-
way random effects were applied to assess intra-rater and inter-rater reliability. Multilevel linear regression was 
applied to assess the association between NIPS score with pulse rates and SpO2 adjusting for raters, three 
periods and two assessments. Results: The Vietnamese NIPS was accepted and valued by nurses at the NICU. 
ICCs between the first and second assessments were from 0.53 to 1.00 for five raters before, during and after 
clinical procedures showing moderate to excellent intra-rater reliability. ICCs among five raters were moderate 
to good before and after, but poor (ICC<0.4) during clinical procedures. NIPS score was not associated with 
SpO2, but with pulse rates per minute. Conclusions: The preliminary results showed that the Vietnamese 
version of NIPS is reliable and should be used. However, it is recommended that further research should be 
conducted to confirm its reliability and validity. 
Keywords: NIPS, reliability, validity, NICU, Vietnam. 
1. INTRODUCTION 
Pain is defined as “an unpleasant sensory and emotional 
experience associated with actual or potential tissue damage 
or described in terms of such damage” [1]. Pain in neonates 
has been extensively studied in recent years and evidence 
suggests that neonates experience pain much more severe than 
adult or older children do as neonatal pain seriously affects 
development of nervous system and growing of babies [2]. 
Literatures showed that neonates experience many painful and 
stressful procedures in Neonatal Intensive Care Unit (NICU) 
[3-5]. 
Pain should be assessed for optimal care of neonates in 
NICU [5, 6]; however, assessing pain is very difficult and 
complicated, particularly in neonates as they are unable to 
verbally communicate [5, 7]. Although some validated and 
reliable pain scales are available for assessing neonatal pain, 
2 MedPharmRes, 2019, Vol. 3, No. 2 To et al. 
they are rarely used in clinical practice and most of the scales 
are in English which require translation and cross-cultural 
adaptation for use in local contexts [4, 5, 8-11]. 
Neonatal Infant Pain Scale (NIPS) was developed by 
Lawrence et al. at Children’s Hospital of Eastern Ontario to 
assess pain in neonates based on behaviors. Their study 
included videotaping 90 medical procedures of 38 neonates 
before, during and after procedures [12]. The Cronbach’s 
Alpha coefficients before, during and after procedures were 
0.95, 0.87 and 0.88, respectively. The Pearson correlation was 
0.92 to 0.97 showed good inter-rater reliability. NIPS includes 
six items assessing six states of neonates including facial 
expression, cry, breathing patterns, movement of arms, 
movement of legs and state of arousal. Each item is scored 0 
to 1, except state of cry is scored 0, 1 and 2, for a possible total 
score of 7. Neonates have no pain if NIPS score is 0-2, 
moderate pain if NIPS score is 3-4 and severe pain if NIPS 
score is more than 4. 
NIPS has the advantage that it does not require users have 
additional skills or equipment to assess pain and would be a 
good tool for assessing and improving pain management of 
neonates [2, 4]. Moreover, ... armRes, 2019, Vol. 3, No. 2 5 
Table 3: The group-average and individual intra-rater reliability of NIPS before, during and after clinical procedures 
Before During After 
ICC 
(95%CI) 
p 
ICC 
(95%CI) 
p 
ICC 
(95%CI) 
p 
Overall intra-rater 
0.92 
(0.88; 0.94) 
<0.01 
0.81 
(0.73; 0.86) 
<0.01 
0.99 
(0.98; 0.99) 
<0.01 
Rater 1 
0.72 
(0.50; 0.86) 
<0.01 
0.85 
(0.72; 0.93) 
<0.01 
0.94 
(0.88; 0.97) 
<0.01 
Rater 2 
0.84 
(0.70; 0.92) 
<0.01 
0.53 
(0.22; 0.75) 
<0.01 
1 
(1; 1) 
<0.01 
Rater 3 
0.84 
(0.70; 0.92) 
<0.01 
0.67 
(0.42; 0.83) 
<0.01 
1 
(1; 1) 
<0.01 
Rater 4 
0.84 
(0.70; 0.92) 
<0.01 
0.88 
(0.77; 0.94) 
<0.01 
1 
(1; 1) 
<0.01 
Rater 5 
0.93 
(0.85; 0.96) 
<0.01 
0.66 
(0.40; 0.82) 
<0.01 
0.97 
(0.93; 0.98) 
<0.01 
ICC: intraclass correlation coefficient; 95%CI: 95% confidence interval 
Five nurses assessed 30 videos in first and second assessment for group-average intra-rater reliability using two-way mixed-
effects absolute agreement model. 
For each rater, individual intra-rater reliability was reported using two-way mixed-effects absolute agreement model. 
Table 4: The inter-rater reliability of NIPS between five nurses using NIPS to assess pain score before, during and after clinical 
procedures at the first and second assessment 
Before (n=30) During (n=30) After (n=30) 
ICC 
(95%CI) 
p 
ICC 
(95%CI) 
p 
ICC 
(95%CI) 
p 
First assessment 
0.77 
(0.66; 0.87) 
<0.01 
0.28 
(0.14; 0.48) 
<0.01 
0.60 
(0.45; 0.75) 
<0.01 
Second assessment 
0.77 
(0.66; 0.87) 
<0.01 
0.33 
(0.18; 0.53) 
<0.01 
0.63 
(0.48; 0.77) 
<0.01 
ICC: intraclass correlation coefficient; 95%CI: 95% confidence interval 
Five nurses assessed 30 videos, two-way random-effects absolute agreement model was used to assess inter-rater reliability 
Table 5: Multilevel linear regression assessing the association of NIPS score with SpO2 and pulse rates per minute adjusting for 
raters, period (before, during and after clinical intervention) and time (first vs. second assessment) (n=900) 
 SpO2 (%) Pulse rates per minute 
Coef. p-value 95%CI Coef. p-value 95%CI 
NIPS score -0.15 0.40 -0.51; 0.20 3.25 <0.01 1.19; 5.31 
Rater 1 1 1 
Rater 2 -0.02 0.95 -0.68; 0.63 0.45 0.82 -3.34; 4.24 
Rater 3 -0.01 0.98 -0.66; 0.65 0.14 0.94 -3.64; 3.92 
Rater 4 -0.00 0.99 -0.66; 0.65 0.05 0.98 -3.72; 3.83 
Rater 5 0.01 0.97 -0.64; 0.67 -0.27 0.89 -4.05; 3.51 
Before clinical intervention 1 1 
During clinical intervention -15.09 <0.01 -17.50; -12.69 25.54 <0.01 11.62; 39.47 
After clinical intervention -0.14 0.58 -0.65; 0.36 0.05 0.98 -2.90; 2.99 
Assessments (second vs. 
first) 
0.01 0.97 -0.41; 0.42 -0.15 0.90 -2.54; 2.24 
Multilevel linear regression model; Coef.: Beta coefficient; 95%CI: 95% Confidence Interval. 
The translation of NIPS into Vietnamese appeared to be 
successful as shown in Table 1. The English back translation 
version kept the original meaning despite some minor 
differences in wording. 
Table 2 shows the characteristics of the Vietnamese 
version of NIPS assessed by 14 nurses at the NICU, Tien 
Giang General Hospital. On average, assessment points for 
the characteristics ranged from 3.4 points for “helps nurses 
give decision on medical care” to 4.4 points for “clear and 
understandable”. The mean duration NIPS took nurses was 
1.6 minutes ranging from one to two minutes. 
Group-average ICCs of five nurses showed excellent intra-
rater reliability before (ICC=0.92, 95%CI: 0.88-0.94) and 
6 MedPharmRes, 2019, Vol. 3, No. 2 To et al. 
after (ICC=0.99, 95%CI: 0.98-0.99) clinical procedures, but 
good (ICC=0.81, 95%CI: 0.73-0.86) during clinical procedure 
(p<0.01) (table 3). All individual ICCs showed good to 
excellent agreements (ICC ≥ 0.84) with the exception of rater 
1 before clinical procedures, and rater 2, 3, 5 during clinical 
procedure just showing moderate agreement (ICC=0.53 to 
0.72, p<0.01). 
At first and second assessments, almost 100% videos were 
rated no pain before clinical procedure at the first and second 
assessments. Five nurses showed good agreements 
(ICC=0.77, 95%CI: 0.66-0.87) before clinical procedures, 
poor agreements (ICC=0.28, 95%CI: 0.14-0.48 and 0.33, 
95%CI: 0.18-0.53) during clinical procedures, and moderate 
agreements (ICC=0.60, 95%CI: 0.45-0.75 and 0.63, 95%CI: 
0.48-0.77) after clinical procedures (p<0.01) at the first and 
second assessments (table 4). 
After adjusting for raters, periods and assessments, the 
multilevel linear regression showed that NIPS score was not 
significantly associated with SpO2 (p>0.05). However, for 
every point increase in NIPS score, pulse rate per minute 
increased three beats (p<0.01) (Table 5). 
4. DISCUSSION 
This study translated and adapted the original NIPS into 
Vietnamese using the process guided by World Health 
Organization [26]. English-Vietnamese and Vietnamese-English 
translations of NIPS is highly consistent. The Vietnamese version 
of NIPS were also reviewed and accredited by experts and 
experienced NICU nurses. In order to apply the NIPS, it is 
important that hospitals’ nurses accept and are willing to use it. 
Our study indicated that all nurses agreed that the Vietnamese 
version of NIPS is applicable in the local hospital context. 
The NIPS mean score indicated that neonates have almost no 
pain before and after procedures but severe pain during them. 
Neonatal pain is highly frequent in NICU because neonates 
suffered seven invasive procedures per day on average and a half 
of them had at least pain once during their hospitalization [27]. 
Compared to other neonatal pain assessment tools, NIPS is easy 
to use and not time-consuming; and does not require extra 
equipment [5]. Therefore, NIPS is applicable in NICU. 
The Vietnamese version of NIPS had good test-retest 
reliability as its average ICCs of five raters were above 0.75 
before, during and after clinical procedures, and individual ICCs 
were from moderate to excellent (0.53 to 1). Five raters showed 
good agreements (ICC=0.77) before clinical procedures and 
moderate agreements (ICC≥0.60) after clinical procedures at first 
and second assessment. However, poor agreement (ICC<0.4) 
were detected between the five raters during clinical procedures 
at the two assessments. As rater 2 had lower ICCs during clinical 
procedures, this rater was removed from the calculation. ICCs 
were re-calculated among the other four raters which improved 
to 0.51 at the first assessment and 0.49 at the second assessment. 
A possible explanation was that four raters had different level of 
clinical experiences from the rater 2 as an experienced pediatric 
nurse tends to rate lower pain score [28]. However, the 
explanation could not be confirmed as personal data of raters 
were not collected in this study. More training on how to identify 
different aspects of pain may be provided to nurses to help 
increase inter-rater reliability. 
This study used SpO2 and pulse rates per minute which were 
objective measures to assess validity of the NIPS. Although the 
NIPS score was not significantly associated with SpO2, it was 
positively associated with pulse rates. This finding may reflect 
the inconsistent results in previous studies. While some showed 
that heart rate increased and SpO2 decreased during painful 
procedures [15, 18], the other found that heart rate and SpO2 were 
not sensitive and not associated with pain scores [29]. 
Our study has some limitations. Firstly, due to very low 
variability, it was unable to calculate internal consistency for the 
entire scale. Secondly, although there are many clinical 
procedures conducted in the NICU, only two clinical procedures 
were observed in this study, including intramuscular injection 
and intravenous catheter insertion. Thirdly, the sample size is 
small and the study was only conducted in one hospital in the 
South of Vietnam. Therefore, the results may not be 
generalizable to other areas of the country. 
In conclusion, the preliminary results showed that the 
Vietnamese version of NIPS showed acceptable reliability for 
use in clinical settings. As there was currently no validated tool 
for assessing neonatal pain available in Vietnamese, this tool 
should be used although it is recommended that further research 
should be conducted to confirm its reliability and validity. 
LIST OF ABBREVIATIONS 
ICC: intra-class correlation coefficient; NICU: Neonatal 
Intensive Care Unit; SpO2: Oxygen saturation. 
CONFLICT OF INTEREST 
The authors declare that they have no competing interests. 
ACKNOWLEDGEMENTS 
We would like to thank Dr. Quyen Gia To, Queensland 
University of Technology, for his helpful advice on statistics. 
Our special thanks should go to nurses of Neonatal Intensive 
Care Unit and the Executive Board of Tien Giang Hospital for 
accepting the study. We also thank Dr. Deborah Kupecz for 
her helpful comments on translation; the reviewers for their 
useful comments on the manuscript; and all participants for 
participating in this study. All other contributions to the study 
should be acknowledged. 
REFERENCES 
1. Loeser JD, Treede RD. The Kyoto protocol of IASP Basic Pain 
Terminology. Pain. 2008;137(3):473-7. 
2. Gallo AM. The fifth vital sign: implementation of the Neonatal Infant 
Pain Scale. J Obstet Gynecol Neonatal Nurs. 2003;32(2):199-206. 
3. Carbajal R, Rousset A, Danan C, Coquery S, Nolent P, Ducrocq S, et al. 
Epidemiology and treatment of painful procedures in neonates in 
intensive care units. JAMA. 2008;300(1):60-70. 
4. Polkki T, Korhonen A, Axelin A, Saarela T, Laukkala H. Development 
and preliminary validation of the Neonatal Infant Acute Pain Assessment 
Scale (NIAPAS). Int J Nurs Stud. 2014;51(12):1585-94. 
5. American Academy of Pediatrics. Prevention and management of pain in 
the neonate: an update. Committee on Fetus and Newborn and Section 
on Surgery, Section on Anesthesiology and Pain Medicine, Canadian 
Paediatric Society, and Fetus and Newborn Committee; 2006 Nov. 
Report No.: 1098-4275 (Electronic), 0031-4005 (Linking) Contract No.: 
5. 
6. Aukes DI, Roofthooft DW, Simons SH, Tibboe D, van Dijk M. Pain 
Management in Neonatal Intensive Care: Evaluation of the Compliance 
with Guidelines. Clin J Pain. 2014. 
7. Bhalla T, Shepherd E, Tobias JD. Neonatal pain management. Saudi J 
Anaesth. 2014;8(Suppl 1):S89-97. 
Psychometric properties of Vietnamese NIPS MedPharmRes, 2019, Vol. 3, No. 2 7 
8. Guedj R, Danan C, Daoud P, Zupan V, Renolleau S, Zana E, et al. Does 
neonatal pain management in intensive care units differ between night 
and day? An observational study. BMJ Open. 2014;4(2):e004086. 
9. Medicine COFNSOAP. Prevention and Management of Procedural Pain 
in the Neonate: An Update. Pediatrics. 2016;137(2):e20154271. 
10. Motta Gde C, Schardosim JM, Cunha ML. Neonatal Infant Pain Scale: 
Cross-Cultural Adaptation and Validation in Brazil. J Pain Symptom 
Manage. 2015;50(3):394-401. 
11. Stapelkamp C, Carter B, Gordon J, Watts C. Assessment of acute pain in 
children: development of evidence-based guidelines. International 
journal of evidence-based healthcare. 2011;9(1):39-50. 
12. Lawrence J, Alcock D, McGrath P, Kay J, MacMurray SB, Dulberg C. 
The development of a tool to assess neonatal pain. Neonatal Netw. 
1993;12(6):59-66. 
13. Hudson-Barr D, Capper-Michel B, Lambert S, Palermo TM, Morbeto K, 
Lombardo S. Validation of the Pain Assessment in Neonates (PAIN) 
scale with the Neonatal Infant Pain Scale (NIPS). Neonatal Netw. 
2002;21(6):15-21. 
14. Malarvizhi G, Manju V, Roseline M, Nithin S, Sarah P. Inter-rater 
reliability of neonatal infant pain scale (NIPS) as a multidimensional 
behavioural pain tool. Nitte University Journal of Health Science. 
2012;2(1):26-30. 
15. Sarhangi F, Mollahadi M, Ebadi A, Matinzadeh ZK, Tadrisi SD. Validity 
and reliability of Neonatal Infant Pain Scale in Neonatal Intensive Care 
Units in Iran (2010). Pak J Med Sci. 2011;27(5):1087-91. 
16. Suraseranivongse S, Kaosaard R, Intakong P, Pornsiriprasert S, 
Karnchana Y, Kaopinpruck J, et al. A comparison of postoperative pain 
scales in neonates. Br J Anaesth. 2006;97(4):540-4. 
17. Taddio A, Hogan ME, Moyer P, Girgis A, Gerges S, Wang L, et al. 
Evaluation of the reliability, validity and practicality of 3 measures of 
acute pain in infants undergoing immunization injections. Vaccine. 
2011;29(7):1390-4. 
18. Taksande AM, Vilhekar KY, Jain M, Chitre D. Pain response of neonates 
to venipuncture. Indian J Pediatr. 2005;72(9):751-3. 
19. Gokulu G, Bilgen H, Ozdemir H, Sarioz A, Memisoglu A, Gucuyener K, 
et al. Comparative heel stick study showed that newborn infants who had 
undergone repeated painful procedures showed increased short-term pain 
responses. Acta paediatrica (Oslo, Norway : 1992). 2016;105(11):e520-
e5. 
20. Porter FL, Wolf CM, Miller JP. Procedural pain in newborn infants: the 
influence of intensity and development. Pediatrics. 1999;104(1):e13. 
21. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass 
Correlation Coefficients for Reliability Research. J Chiropr Med. 
2016;15(2):155-63. 
22. StataCorp LLC. ICC - Intraclass correlation coefficients [Available 
from: 
https://www.stata.com/manuals/ricc.pdf#riccRemarksandexamples. 
23. Yoon PJ, Starr JR, Perkins JA, Bloom D, Sie KCY. Interrater and 
Intrarater Reliability in the Evaluation of Velopharyngeal Insufficiency 
Within a Single Institution. Archives of Otolaryngology–Head & Neck 
Surgery. 2006;132(9):947-51. 
24. Eliasziw M, Young SL, Woodbury MG, Fryday-Field K. Statistical 
methodology for the concurrent assessment of interrater and intrarater 
reliability: using goniometric measurements as an example. Physical 
therapy. 1994;74(8):777-88. 
25. McGraw KO, Wong SP. Forming Inferences About Some Intraclass 
Correlation Coefficients. Psychological Methods. 1996;1(1):30-46. 
26. World Health Organization. Process of translation and adaptation of 
instruments 2017 [Available from: 
27. Sposito NPB, Rossato LM, Bueno M, Kimura AF, Costa T, Guedes 
DMB. Assessment and management of pain in newborns hospitalized in 
a Neonatal Intensive Care Unit: a cross-sectional study. Revista Latino-
Americana de Enfermagem. 2017;25:e2931. 
28. Gomez RJ, Barrowman N, Elia S, Manias E, Royle J, Harrison D. 
Establishing intra- and inter-rater agreement of the Face, Legs, Activity, 
Cry, Consolability scale for evaluating pain in toddlers during 
immunization. Pain research & management. 2013;18(6):e124-e8. 
29. Backus LA. Validation of the Neonatal Infant Pain Scale. Michigan, 
United States: Grand Valley State University; 1996.