Quantifying the Impact of the e-Journal of Neonatology Research
Jonathan R. Swanson MD1, and Phillip V. Gordon MD, PhD2
1Department of Pediatrics, University of Virginia School of Medicine, Charlottesville, VA
2Department of Pediatrics, Tulane School of Medicine, New Orleans, LA
Jonathan Swanson, MD
Department of Pediatrics
University of Virginia
Charlottesville, VA 22908
Author Disclosure: The authors have nothing to disclose.
Introduction: We sought to quantify readership response to our community supported, on-line, open access journal the e-Journal of Neonatology Research (eJNR).
Methods: Two separate means of quantification were employed. First, the front page of the web-based on line journal was enrolled with the free online site StatCounter.com™, with a start date of October 1, 2012. The subsequent quarter (including the months before and after publication) was reviewed for unique visitors, returning visitors and page downloads. A second method of quantification consisted of review of the response rates to all linked “What do Neos think?” online surveys, published quarterly by eJNR.
Results: Notification of publication of the 2012 Winter edition of eJNR occurred by emails sent to 4105 unique addresses of neonatologists and neonatal nurse practitioners. There were 3817 unique visits and 5831 page down loads in the subsequent two months following its publication based on monthly StatCounter.com™ reports. The response rate for the embedded quarterly survey was 1.8 % over the two year publication period with substantial change from survey to survey (with a high of 5.3% on the first survey and ending with 1.1% in Winter 2012).
Discussion: Our data indicate high readership engagement and an average response rate for embedded online surveys. These findings suggest that our online, open access, no cost publication model is competitive with traditional academic publishing for the rapid communication of research and opinions to neonatal clinicians. It remains to be seen if eJNR can become commercially viable.
Key Words: publication, open access, impact
The e-Journal of Neonatology Research (eJNR) is many things. It is a community supported, open access, online journal that publishes neonatology-oriented manuscripts at no cost. The community that supports it includes the three editors, the current 10 member editorial board, all of the authors who submit their papers, and you (the reader). The only current funding for eJNR comes from one editor (PVG) in the form of paying the annual server costs and licensing fees. eJNR is also an experiment. It was formed to create a new publication model at a time when the current peer-review and traditional publishing house models are failing neonatology by restricting access to articles (through access fees) and limiting the ability to publish (for the sake of improving journal impact factors). eJNR is trying to determine whether or not it is possible to find and sustain a better system of publication for neonatology.
To do this, eJNR alters the traditional peer review system. Rather than waiting months, a manuscript decision is made within days to weeks and manuscripts are published in the next edition. The editorial decision is made entirely by the editors (with rare exception when we call upon the editorial board for specific expertise). Specific revisions are requested by the editors when needed for either clarification or grammatical reasons. To maintain an element of independent peer review, eJNR then makes post-publication commenting available on all research manuscripts and editorials. Few readers have embraced the post-publication review concept, so eJNR has mostly become an online, open access repository of manuscripts in which the editors have ultimately performed all of the peer review. But there have been notable exceptions and one manuscript was withdrawn and resubmitted because of outside peer review. Several others have resulted in modification of the HTML version with editor’s notes in the comment section. So the journal does have a demonstrable independent peer review process, albeit one that is intentionally non-traditional.
This past November, eJNR passed the two year mark in publication and this edition marks the 8th actual publication. At this juncture, the senior editors of eJNR sought to quantify eJNR’s impact. Because eJNR is not (yet) a PubMed publication, it is not eligible for an impact factor that can be used to compare it with other neonatology content journals. However, because it is entirely online, website metrics can be used to assess readership engagement. Such an investigation is needed to determine if eJNR can be competitive with traditional publishing models and thus whether or not eJNR should continue to be published.
The e-Journal of Neonatology Research was enrolled with the free on line site StatCounter.com™, with a start date of October 1, 2012.1 The subsequent quarter (including the months before and after publication) was reviewed for unique visitors, returning visitors and page down loads and the data tabulated by month. The data was then used as the numerator against the number of notification emails sent for publication of the Winter 2012 edition as the denominator to determine what percentage of emails sent resulted in a person responding to our survey.
A second method of quantification of readership consisted of review of the response rates to all linked “What do Neos think?” on-line surveys, published quarterly by eJNR.2 The number of respondents to the first and last question of each survey was tabulated and used as the numerator against the number of notification emails sent for publication of the Winter 2012 edition as the denominator to determine what percentage of emails sent resulted in a person accessing eJNR.
Simple quantitation was used to describe our findings, including percent of emails sent that resulted in a person accessing eJNR and percent of emails sent that resulted in a person responding to our survey.
Using site StatCounter.com™, we were able to demonstrate nearly 3817 unique accessions to the front page of eJNR during the three month period of our study (Figure 1). This was 93% of the number of emails sent out for notification and of those visitors, 5831 chose to perform downloads (typically in the form of a PDF) suggesting that they read an average of one and half articles while visiting eJNR. This number may be an underestimate, since some readers may have chosen to read the HTML versions of the manuscripts.
Survey Response Rate
We reviewed the response rates from the first and last questions from all “What do Neos think?” online surveys (See Table 1). The average response rate was 1.8% for the first question and 1.6% for the last question. Our highest single survey response was 5.3% and 5.6% respectively and this survey was thought to be representative of neonatologists’ opinions nationally. Three other surveys ranged from 1.1% to 2.5% and were utilized as pilot surveys of opinion. Three had response rates that were too low and were considered invalid based on insufficient power.
Fledgling academic journals have a relatively Darwinian existence. In many cases they are started by a society or think tank as a regional or subspecialty journal. These entities persist for some period of time while being supported by their originators, but many eventually expire for lack of continued financing. Journals that survive the initial cut are generally picked up by national societies or sold to publishing houses, both of whom seek to utilize them for profit via traditional academic publishing models. The advantage of this is that they are instantly catapulted to PubMed and searchable online access. It also means that they sell articles online for fees, either bundled with other journals or individually, and they are always seeking to improve the journal’s impact factor (so they can charge more for advertisements). This translates into preferentially taking those manuscripts that are likely to be referenced in future publications (because citations are the fuel that generates impact factors). The means by which they do this are generally through editorial massage of the peer review system with a limit on the number of published manuscripts. The simplest way is to always take manuscripts with excellent reviewer scores but to allow editor discretion for manuscripts with good scores, thereby permitting avoidance of manuscripts that are generally unlikely to garner citations. The result is a bias of the medical literature that is increasingly obvious and harmful.3-8
Online, open access publishing has surged over the last decade. The predominant model is that of a non-profit which charges a fairly hefty publication fee per manuscript (generally $1500 or more). PLoS, BMC and others have quickly created or incorporated large portfolios of subject matter, slowly challenging traditional publishing houses for supremacy in academic publishing. With time, these venues have also acquired impact factors. However, a home for neonatology in this sphere has been notably absent. It should be noted that there are several open-access neonatology-specific journals currently available, all of which charge a publication fee to the majority of authors and none of which are found in PubMed.
The e-Journal of Neonatology Research pushes the online, open access model one step further via publishing at no cost. The rationale behind this model is that if the journal is sufficiently successful in establishing a readership, it will eventually be able to fund a small, part time editorial staff through advertising alone, thus creating a new and novel model of academic publishing (in essence it will create a journal with editorial chairs that can be passed from one generation to the next). However, this model can only become viable if eJNR is successful in establishing and maintaining a readership AND in establishing robust advertisement (something it has not yet attempted). Thus it remains to be seen if the eJNR can sustain itself as an independent entity or if it will need to partner with a larger corporate entity for long term success.
The purpose of this study was to quantify readership engagement using both web metrics and survey responsiveness. We found that in comparison to emails sent out for the Winter 2012 edition (which encompassed well over half of the U.S. neonatology work force), there was a 93% access rate and 1.1% survey response rate. Over the course of seven surveys, the average response rate was 1.8%. This is typical for a text-embedded, unprompted online survey and most of the time yielded sample sizes sufficient for interpretation.6,7 These findings support the supposition that eJNR is competitive with traditional publishing for communicating neonatology-oriented research. It can also be a useful tool for determining readership opinion when the survey topic is of intrinsic interest to the readership.
Future challenges for eJNR include transitioning to an advertisement-based revenue model to sustain editorial excellence through stipends, continued solicitation of submissions, sustaining ongoing consideration by PubMed Central and PubMed (and achieving their acceptance), as well as the active consideration of corporate offers for sponsorship or purchase. Navigation of these hurdles and opportunities in the next two years will be crucial to the success and survival of eJNR and may drive or diminish alternative publishing models in neonatology (depending on eJNR’s success).
The e-Journal of Neonatology Research had an excellent access rate and an average survey response rate within the United States neonatology community, based on web-based statistics and survey response data from the Winter 2012 edition. These finding suggest eJNR is currently well positioned for the upcoming decisions and planning that may be necessary for its evolution and growth.
- StatCounter. http://statcounter.com/
- Neonatal Care Givers Snap Survey Site. http://whatdoneosthink.wordpress.com/
- Clark RK. Peer review: a view based on recent experience as an author and reviewer. Br Dent J 2012; 213(4):153-4.
- Wager E, Parkin EC, Tamber PS. Are reviewers suggested by authors as good as those chosen by editors? Results of a rater-blinded, retrospective study. BMC Med 2006; 4:13.
- Suñé P, Suñé JM, Montoro JB. Positive outcomes influence the rate and time to publication, but not the impact factor of publications of clinical trial results. PLoS One 2013; 8(1):e54583.
- Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009 Jan 21;(1):MR000006.
- Rising K, Bacchetti P, Bero L. Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Med 2008 ; 5(11):e217; discussion e217.
- Jackson JL, Srinivasan M, Rea J, Fletcher KE, Kravitz RL. The validity of peer review in a general medicine journal. PLoS One 2011; 6(7):e22475.
- Nulty DD. The adequacy of response rates to online and paper surveys: what can be done? Assessment & Evaluation in Higher Education 2008; 33(3):301-14.
- Assessing the Representativeness of Public Opinion Surveys, Pew Research Center for the People and the Press. Retrieved from: http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-public-opinion-surveys/.