Neonatologists’ Opinions of the American Board of Pediatrics Maintenance of Certification Process are not Favorable
Phillip V. Gordon MD, PhD
Department of Pediatrics, Tulane University, New Orleans, LA
Phillip V. Gordon, MD
Department of Pediatrics
Tulane School of Medicine
1430 Tulane Avenue
New Orleans, LA 70112
Author Disclosure: The author has nothing to disclose.
Background: We sought to poll neonatologists using an online survey mechanism regarding their opinions about neonatal-perinatal Maintenance of Certification (MOC) and the American Board of Pediatrics (ABP).
Methods: A waiver was granted to the e-Journal of Neonatology Research by Tulane School of Medicine’s Internal Review Board for professional opinion surveys using a standardized format. A nine question, two minute, on line survey was created. Three questions addressing the neonatologists’ practice, board certification, and American Academy of Pediatrics (AAP) membership status were included to provide internal assurance of appropriate survey strategy. Six additional questions targeting the respondents’ opinions were utilized for the subject area of interest.
Results: More than a 165 respondents answered each survey question. The overwhelming majority identified themselves as practicing neonatologists and as members of the AAP (i.e. at least 3% of the U.S. neonatology work force). Approximately 2/5ths indicated that they had life time certification and three in five of those with life time certificates indicated they did not intend to participate in the MOC process. The vast majority of respondents did not think MOC would make the practice of neonatology safer; did not think part IV assessed performance (as it claims); did not think the ABP was responsive to the needs of neonatologists; and were not convinced that the ABP was a good steward of its finances. When asked if they would support the formation of a national neonatal board independent of the ABP, the majority response was mixed between yes and maybe, whereas only one in eight of the respondents said “no”.
Conclusions: The vast majority of respondents to our online survey gave unfavorable responses to questions regarding satisfaction with the MOC process, as well as the ABP itself. This degree of unfavorable opinion, as well as the size of our cohort, is consistent with a national mood of discontent regarding neonatal/perinatal MOC.
Keywords: online survey, American Board of Pediatrics (ABP), maintenance of certification (MOC), neonatology
In 2010, the American Board of Pediatrics (ABP) instituted sweeping new policy with regard to board certification. Individuals with life time certification were now encouraged to participate in Maintenance of Certification (MOC) or be listed on a national website as not actively participating in maintenance of certification.(1) Individuals with time-limited certificates were expected to participate in MOC or effectively lose board certification status.(2) This newly created process of MOC required 4 separate parts of certification and cost $1300-1500, depending on how each part was completed. It was initially required to be completed every 7 years.
This change was made with two underlying premises as rationales. The first was that the public wanted more accountability from physicians with regard to current knowledge. One basis for this argument was a survey created by the team that runs the C.S. Mott Children’s Hospital National Poll on Children’s Health (NPCH), administered by a group called Knowledge Panel of Knowledge Networks (Menlo Park, CA), and paid-for by the ABP Foundation.(3,4) It contained questions that they asked of 1687 parents which were demographically selected. For example, parents were asked to rate board certification or AAP membership in their decision making process when selecting a pediatrician for their child; options included “very important, important, unimportant, and unsure.” The arrangement of the answers had the potential to create survey bias because the number of positive options outweighed the options with negative connotations when referencing the topic of interest (“unsure” is a neutral answer).
This tactic can bias results towards a desired conclusion in other questions that are not similarly weighted in their answer structure. The one unweighted question that they printed in the publication was the only one that actually addressed whether or not failure to complete MOC might affect the parents’ decision to bring their child to their current care giver (answers were “very likely, likely, not likely, not at all likely”). The trends were not as robust in this answer format when compared to those questions with weighted answers, but there was a finding for respondents’ desire for MOC. There was no specific information on the survey describing what MOC consisted of (and the survey was done before MOC existed). Based largely on this, the ABP has claimed that the public demands pediatric MOC.
The second rationale was that the American Board of Medical Specialties (ABMS) was going to require all of its constituent organizations (in this case the ABP) to move towards MOC as means of cyclic life-long assessment.(5) It is worth noting that the ABP is a voting member within the ABMS. The nature of the relationships between the ABP and other regulatory bodies that govern education and licensure in medicine is best revealed in a manuscript entitled, A Continuum of Competency Assessment.(6) In this manuscript, a master plan of core competency education, certification, continuous education and cyclic recertification is described in detail. It is a collaborative, quixotic, and bureaucratically vast venture that the ABP, the ABMS and the Accreditation Council of Graduate Medical Education (ACGME) have set for themselves. Unfortunately, there is no credible evidence that these interventions will positively affect neonatologists’ performance or patient care. The existing evidence in favor of MOC as a performance enhancer that seems most often quoted by the ABP is circumstantial and has been done in the adult, outpatient setting on the topic of optimal hypertension medication dosing.(7) Pediatricians and pediatric subspecialists were not ignorant of these facts prior to the onset of MOC but early attempts to voice concerns were met with censorship by both the ABP and the AAP.(7,8)
Given that our medical education and board certification systems have been transformed without the burden of evidence that would typically be required for such broad change in clinical practice, it seems prudent to ask physicians to share their opinions, now that the transformation is upon them. With this survey, we focus specifically on MOC in neonatology, which is among the most specialized and largest of pediatric subspecialties. This makes it an ideal cohort to independently determine the true value of MOC to certificate holders, as well as to the American public.(9)
We received a three year waiver from the Tulane Institutional Review Board for online professional opinion surveys using PolldaddyTM technology to create anonymous survey questions in a standardized format linked to the e-Journal of Neonatology Research. The table of contents for the inaugural issue was sent to 3,807 recipients throughout the continental United States and Hawaii, divided into four batches over the course of four days, with each containing a direct link to the study survey entitled “What do Neos think: about recertification?” and can be found at: http://whatdoneosthink.wordpress.com/ The survey was limited to nine questions, with the intent that brevity might encourage voluntary participation. The first three questions were designed to verify that neonatologists were actually taking the survey and to provide basic demographic information about their status with regard to current practice, board certification and AAP membership. The last six questions were designed to elicit opinions about satisfaction with recertification, the ABP, and part IV of recertification specifically. The survey was set up such that participants could not take the survey twice from the same computer (via cookie technology), thereby discouraging repeated taking of the survey. The survey instructions also explicitly prohibited repeated taking of the survey.
Data was analyzed by extracting raw data within each question, then comparing the ratios of positive to negative answers (and where relevant, to do simultaneous comparison with the ambivalent answers). Also, when questions overlapped in content, answer ratios were evaluated for congruence by simple Boolean logic. The validity of the sample size was evaluated relative to the known size of the board certified neonatology work force as a percentage of the total. Additionally, because the emails were sent out in four equal batches, survey responses were analyzed over the subsequent four day period to determine if observed ratio trends were consistent and robust across that time period (i.e., we looked specifically for signs of large scale tampering).
Demographic survey questions:
We asked: 1) Are you a practicing Neonatologist?2) Are you currently a member of the American Academy of Pediatrics? 3) Which of the following answers best describes your board certification status? The responses are displayed in figure 1. The overwhelming majority of respondents were practicing neonatologists who were also members of the AAP. Approximately one-third of respondents held life time certification versus two-thirds who did not. Three-fifths of those with life time certification self identified themselves as non-participators in MOC versus one third of non-life certificate (also known as time-limited) holders. These included a mix of non-boarded individuals (9 respondents, not shown) and not yet having cycled into the MOC requirement (32 respondents, shown in Figure 1). There was not a specific response for time-limited certificate holders who were specifically choosing not to participate in MOC.
Figure 1: Characteristics of survey participants based on practicing/non-practicing status, AAP membership and MOC participation of those with life time certification.
Survey questions of opinion:
We asked six questions surveying neonatologists’ opinions, with congruence across three topics: the ABP (3 questions), recertification (3 questions), part IV (2 questions). The responses are displayed in Figure 2.
Opinion about the ABP:
The majority of respondents gave unfavorable responses to the ABP with regard to its responsiveness to neonatologists (121 versus 45). Respondents were less sure about the ABP as a good steward of their finances, with 94 responding that they didn’t know, 63 responding no and only 12 responding yes. Similar results were seen when respondents were asked if they would support the formation of an a national neonatal board independent of the ABP, with 88 responding maybe, 65 responding yes, and 25 with no. Responses on these three questions were congruent, with the vast majority of respondents failing to endorse or support the ABP on three separate issues.
Figure 2: Survey participant’s opinions regarding the underlying validity of MOC recertification, the appropriate function of the ABP and their willingness to support an independent neonatal board.
Opinion about recertification and part IV:
Unfavorable opinion about recertification was even more robust (Figure 2). When asked if recertification would make the practice of pediatrics or neonatology safer, 134 respondents said no, versus 12 who said yes and 24 who said maybe. When asked if part IV of recertification assesses satisfactory performance in practice, 88 respondents said no, 81 said they didn’t know (with 44 of those having participated in MOC and 37 not), versus only 1 respondent who said yes. Finally in a multiple choice question (Figure 3), respondents were asked “Which of the following should not be required for MOC?” Institutional quality collaboratives were the least popular (102 respondents), followed by parental satisfaction assessments (74), with assessment of current knowledge and recently read literature receiving weak tallies (28 each). Documentation of a valid medical license was chosen by only 11 respondents (indicating that the vast majority agreed with it as a requirement). In contrast, 32 respondents felt that all of the listed items should be included in MOC. Of note, assessment of parental satisfaction is the only item not currently in the ABP’s version of MOC, thus respondents who chose “all” endorsed a level of assessment that exceeds that currently required by the ABP.
Figure 3: Survey participants opinions regarding what should not be assessed within a maintenance of recertification program for neonatology.
Because the inaugural issue of the e-Journal of Neonatology Research was delivered in four distinct batches of equal sized emails, 24 hours apart, we were able to track the accumulation of respondent answers over time. We then compared the final response ratios (two weeks after publication) to the accumulating tallies over time in the first four days, thereby providing a built in control to assess for tampering (via malware, repetitive survey taking, or a practice known as survey spiking). In all cases, the final tallies were consistent with the uniform trends of accumulation in the first four days (examples shown in Figure 4).
A minimum of 165 respondents answered each question within our survey except for the one multiple choice question (for which the exact number of respondents could not be determined). Based on the ABP workforce reports and the fact that most respondents self-reported membership in the AAP, this survey captured more than three percent of the American neonatology work force. Based on this slice of demographic and the consistency in which survey responders answered questions over the first four days, our survey is a valid national assessment of neonatologists’ opinions regarding the ABP, MOC and part IV.
We found that neonatologists have unfavorable opinions of the ABP’s responsiveness and do not value MOC. They were mostly undecided as to whether or not the ABP was fiscally irresponsible, and whether or not an alternative neonatal board certification entity would be preferable (although the overall trends did not favor the ABP). The over-whelming majority did not feel that MOC would make patient practice safer or that part IV assessed satisfactory performance in practice. These sentiments suggest that recent self-publications by the ABP reporting high satisfaction and participation rates with MOC should be viewed with skepticism when it comes to neonatology. For example, our self-reported participation rates by individuals with life time certification was only 40%.(10) To our knowledge, The ABP has not published these specific rates, (but could if they chose to do so). This degree of non-participation is substantial, particularly when one considers that more than a third of all American neonatologists hold life time certification (following this extrapolation to its logical conclusion, this means roughly one in four board certified neonatologists currently do not intend to participate in MOC).
Like all surveys, ours has weaknesses. The first is the potential for responder selection bias. Our questions were mostly worded in neutral format, with a couple of noted exceptions designed to be polarizing so that the respondent would have to choose a side (for or against MOC or the ABP). This design has the weakness that respondents may have self selected for those who are less enamored with MOC and the ABP. However, the fact that we had two out of the four three-option opinion questions result in dominance of the neutral answer argues against a negative responder selection bias. The second concern is that our survey response rate was just over five percent (when emails with “return to sender” were subtracted and the average for single answer questions taken – data not shown). This was a low raw percentage, but a solid response rate for an unsolicited, single email driven survey and thus likely representative of the national neonatology workforce. The generalizability of our survey findings is bolstered by the open letter of consensus recently published by 69 academic neonatology section heads from across the United States, who specifically found part IV of MOC to be problematic and have requested the MOC to reconsider its implementation.(11)
Given the degree of discontent evident in our study, we believe it unlikely that neonatal MOC will be able to survive in its current form. If the ABP wants to continue to be the principle regulatory body for neonatology board certification, change will likely be necessary. Alternatively, at least 65 neonatologists anonymously indicated they would be willing to support a new board certification organization. Since most of them were at least once board certified by the ABP, perhaps that is the change that is really needed.
1. Miles PV. Pediatric recertification and quality of care: the role of the American Board of Pediatrics in improving children’s health care. J Pediatr. 2007;151(5 Suppl):S17-20.
2. Miles PV. Maintenance of Certification: the role of the American Board of Pediatrics in improving children’s health care. Pediatr Clin North Am. 2009;56(4):987-94.
3. Freed GL, Dunham KM, Clark SJ, Davis MM. Perspectives and preferences among the general public regarding physician selection and board certification. Research Advisory Committee of the American Board of Pediatrics. J Pediatr. 2010;156(5):841-5, 845.e1.
4. Knowledge Networks. “Knowledge Networks: Online research – privacy, rigor, collaboration.” 12 June 2011. http://www.knowledgenetworks.com/
5. Miller SH. American Board of Medical Specialties and repositioning for excellence in lifelong learning: maintenance of certification. J Contin Educ Health Prof. 2005; 25: 151-156.
6. Brown HJ, Miles PV, Perelman RH, Stockman JA III. A continuum of competency assessment: the potential for reciprocal use of the Accreditation Council for Graduate Medical Education toolbox and the components of the American Board of Pediatrics Maintenance-of-Certification Program. Pediatrics. 2009; 123(suppl 1):S56-S58.
7. Turchin A, Shubina M, Chodos AH, Einbinder JS, Pendergrass ML. Effect of board certification on antihypertensive treatment intensification in patients with diabetes mellitus. Circulation. 2008;117(5):623-8.
8. Personnal communication with V. Strasburger.
11. Gordon PV, Seidner SR, Davis JM, Schelonka R, Durand DJ, Segar J, et al. Regarding part IV of recertification in neonatology: A consensus perspective of 69 academic neonatology section heads. eJ Neonatol Res, 2011; 1:45-9.