Receiver Operating Characteristic Curve Analysis of SEER Medulloblastoma and Primitive Neuroectodermal Tumor (PNET) Outcome Data: Identification and Optimization of Predictive Models

Modeling risk grouping for medulloblastoma (MB) and Primitive Neuroectodermal Tumor (PNET) is an ongoing process (Packer et al., 2012; Smee et al., 2012; von Hoff and Rutkowski, 2012). MB and PNET are the most common brain tumors in children (Packer et al., 2012; Smee et al., 2012; von Hoff and Rutkowski, 2012). The cause specific survival rates for both childhood and adult with MB or PNET are about 70% (Packer et al., 2012; Smee et al., 2012; Smoll, 2012; von Hoff and Rutkowski, 2012) and this study. Thus there is room for improvement in the treatment outcome. This study uses receiver operating characteristic curve to analyze Surveillance, Epidemiology and End Results (SEER) MB/ PNET outcome data. The MB and PNET have been noted to have similar clinical course and age distribution (Smoll, 2012). The aim of this study was to identify and optimize predictive MB/PNET models to aid treatment and patient selection. This study also examined why some predictive


Receiver Operating Characteristic Curve Analysis of SEER Medulloblastoma and Primitive Neuroectodermal Tumor (PNET) Outcome Data: Identification and Optimization of Predictive Models
Min Rex Cheung models may not work as expected.
Surveillance Epidemiology and End Results (SEER) is a public use cancer registry of United States of America (US). SEER is funded by National Cancer Institute and Center for Disease Control to cover 28% of all oncology cases in US. SEER started collecting data in 1973 for 7 states and cosmopolitan registries. Its main purpose is through collecting and distributing data on cancer, it strives to decrease the burden of cancer. SEER data are used widely as a bench-mark data source for studying MB/PNET cancer outcomes in US and in other countries (Barnholtz-Sloan et al., 2005;Bishop et al., 2012;Curran et al., 2009;Deorah et al., 2006;Gatta et al., 2002;Halperin et al., 2004;Lai, 2008;Smoll, 2012). The extensive ground coverage by the SEER data is ideal for identifying the disparity in oncology outcome and treatment in different geographical and cultural areas (Cheung, 2013a(Cheung, , 2013b(Cheung, , 2013cCheung, 2012;Cheung, 2013;Downing et al., 2010;Gross et al., 2008;Harlan et al., 1995;Lund et al., 2008;Martinez et al., 2010;Martinez Min Rex Cheung Schlichting et al., 2012;Shavers et al., 2003;Wampler et al., 2005;Yao et al., 2012). In addition to the biological staging factors and the treatment factors, this database also contains a large number of county level socio-economic factors data. This study aimed to identify barriers to good treatment outcome that may be discernable from a national database. SEER registry has massive amount of data available for analysis, however, manipulating this data pipeline could be challenging. SEER Clinical Outcome Prediction Expert (SCOPE) (Cheung, 2012) is designed and implemented to mine SEER data and construct accurate and efficient prediction models (Cheung et al., 2001a(Cheung et al., , 2001b.

Materials and Methods
The data were obtained from SEER 18 database. SEER is a public use database that can be used for analysis with no internal review board approval needed. SEER*Stat was used for listing the cases. The filter used was: 'Site and Morphology.AYA site recode' = '3.4. Medulloblastoma and other PNET'. This study explored a long list of socio-economic, staging and treatment factors that were available in the SEER database. We have designed and implemented SEER Clinical Outcome Prediction Expert (SCOPE) for this purpose. The codes of SCOPE have been posted on Matlab Central. SCOPE has a number of utility programs that are adapted to handle the large SEER data pipeline. All statistics and programming were performed in Matlab. Each risk factor was fitted by a Generalized Linear Model to predict the outcome (brain and other nervous system specific death). The areas under the receiver operating characteristic curve (ROC) were computed. Similar strata were fused to make more efficient models if the ROC performance did not degrade (Cheung et al., 2001a(Cheung et al., , 2001b. In addition, it also implemented binary fusion and optimization to streamline the risk stratification by combining risk strata when possible. SCOPE uses Monte Carlo sampling and replacement to estimate the modeling errors and allows t-testing of the areas under the ROC. SCOPE provides SEER-adapted programs for user friendly exploratory studies, univariate recoding and parsing.

Results
There were 3702 patients included in this study ( Table  1). The followup (S.D.) was 73.7 (86.2) months. 40% of the patients were female. The mean (S.D.) age was 16.5 (16.6) years. There were more adult MB patients listed from SEER data than the pediatric and young adult patients. Only 12% of patients were staged. The SEER staging has the highest ROC (S.D.) area of 0.55 (0.05) among the factors tested in Table 1. SEER Clinical Outcome Prediction Expert was used to perform ROC curve and area under the curve calculations. In this example, the ROC area of the 3-tiered SEER staging model as computed for 5 random samples ( Table  1). The results are shown in the upper panels. In the lower panels, SCOPE simplified the 3-layered risk levels (local, regional, distant) to a simpler non-metastatic (I and II) versus metastatic (III) model. The ROC area (S.D.) of the 2-tiered model was 0.57 (0.04) based on 5 random samples with replacement from the SEER data. Rural residence, county's family income level, county' education attainment and race were tested as socio-economic barriers to good outcome. None of these factors were predictive of brain cancer specific survival. They had a ROC area of around 0.5 that is expected for a random variable with no predictive power.
The staged patients fared better than the overall cohort (Table 2). Age older than 20 years old did not correlate with higher percentage mortality during this study period from 1977 to 2009. Neither surgery nor radiotherapy was associated with a lower risk of cause specific mortality in the overall cohort. The completely staged patients were found to do better than the overall cohort. Although most patients were treated with surgery, radiotherapy was about 15-20% under-utilized when compared with surgery ( Figure 1). Given the aggressive nature of this disease, these patients would uniformly require combined modality treatment.

Discussion
This study is interested in constructing models that will aid patient and treatment selection for MB/PNET cancer patients. To that end, this study examined the ROC models (Hanley and McNeil, 1982) of a long list of potential explanatory factors (Table 1). ROC models take into account both sensitivity and specificity of the prediction. Ideal model would have a ROC area of 1 and a random Min Rex Cheung model is expected to have an area of 0.5 (Hanley and McNeil, 1982). For example, a clinical ROC model can be used to predict if a patient receiving the recommended treatment will die from the disease. The SEER staging is most predictive of patient outcome (Table 1). After binary fusion, it reduces to non-metastatic versus metastatic classification of the MB/PNET patients (Table 1). Such efficient model may aid in reducing patients needed for clinical trials because it has fewer risk groups to balance.
When there are competing prediction or prognostic models, the most efficient (i.e. the simplest) model is thought to prevail (D'Amico et al., 1998). This has an information theoretic (D'Amico et al., 1998) underpinning. For practical purposes, simpler models require fewer patients for a randomized trials because fewer risk strata need to be balanced. In the clinic, simpler models are easier to use. SCOPE streamlined ROC models by binary fusion (Table 1). Two adjacent strata were tested iteratively to see if they could be combined without sacrificing the higher predictive power usually belong to the more complex models. This study has shown that SCOPE can built efficient and accurate prediction models.
For surgery and radiotherapy, the ROC areas were modest (0.5). Low ROC areas imply the information content (i.e. the staging accuracy) of the models may be limited. It is consistent with the fact that only 12% patients had complete SEER staging (Table 2). In addition, the outcome of the completely staged patients was much more superior when compared with the entire cohort (Table 2). It may be a consequence of having a better guidance model in treatment and patient selection.
In conclusion, this study has identified the staging models are the most prognostic of treatment outcomes of medulloblastoma and PNET patients. The high under-staging rates may have prevented patients from selecting definitive local therapy (Fig. 1). The poor rates of radiotherapy after surgery use may have contributed to the poor outcome in these patients with this aggressive disease.