Ultimate Meteorological Question from Observational Astronomers: How Good is the Cloud Cover Forecast?
To evaluate the capability of numerical cloud forecast as a meteorological reference for astronomical observation, we compare the cloud forecast from NCEP Global Forecast System (GFS) model for total, layer and convective cloud with normalized satellite observation from the International Satellite Cloud Climatology Project (ISCCP), for the period of July 2005 to June 2008. In general, the model forecast is consistent with the ISCCP observation. For total cloud cover, our result shows the goodness of the GFS model forecast with a mean error within in most areas. The global mean probability of forecast error (polar regions excluded) declines from 73% to 58% throughout the forecast period, and is more skilled than ISCCP-based climatology forecast up to . The comparison on layer clouds reveals a distinct negative regional tendency for low cloud forecast and a questionable positive global tendency for high cloud forecast. Fractional and binary comparisons are performed on convective cloud forecast and revealed the GFS model can only identify less than half of such cloud. In short, our result suggests that the GFS model can provide satisfactory worldwide total cloud forecast up to a week ahead for observation scheduling purpose, but layer and convective cloud forecast is less reliable than the total cloud forecast.
keywords:atmospheric effects, site testing.
The most important prerequisite of most successful astronomical observation, mainly optical observation, is no doubt a cloudless sky (see the story of Guillaume Le Gentil in the 1761/69 Transit of Venus, cf. Sawyer Hogg, 1951, for a rather unlucky example). However, cloud cover forecast had been difficult for a long time as limited by theoretical understanding of mesoscale circulation (i.e. modeling) and computation ability. Numerical simultaneous cloud cover forecast for astronomical observation had not been practically useful until the very end of 20th century.
In the meteorological context, cloud plays a key role in radiation balance of the Earth, and is widely accepted to be the main source of uncertainty of global weather predictions (cf. Stubenrauch et al., 1999; Stephens, 2005). It gathered much attention from atmospheric sciences community, both from modeling group and from observational group. The attempt to “parameterize” cloud activity started in the 1980s (cf. Sundqvist et al., 1989, and references therein). Several cloud schemes have been proposed since then, followed by ground- and space-based evaluation in aim for their refinement (e.g. Hinkelman et al., 1999; Luo et al., 2005; Yang et al., 2006).
Although many amateur and professional observatories have been taken advantage of open access to the outputs of major numerical weather models for a decade, the reliability of such forecast is poorly understood. The meteorology community is mostly interested in particular mesoscale events and/or particular regions, and pays relatively little attention to the model performance over broader environment; while there are only two studies from the astronomy community that have investigated this topic: Erasmus & Sarazin (2001), which suggested that only 15-25% of cloudy nights at the European Southern Observatory sites could be identified by the European Centre for Medium-Range Weather Forecasting (ECMWF) model; and our earlier study (Ye, 2011), which suggested a high detection rate accompanied with a moderate false alarm rate from the NCEP Global Forecast System (GFS) model, based on the cloud observations from several astronomical observatories. Even so, as the two studies are both limited at spatial densities and scales, investigations of numerical cloud forecasts over global scale are still lacking.
This study is therefore carried out in aim to assess the cloud forecast ability of a global numerical model to provide insight on its reliability as a reference for astronomical observation. In order to do this, we need: i) output from global numerical model as forecast data, and ii) observational data with appreciable temporal and spatial coverage. The selection and reduction of such data will be discussed in the following section.
2 Data Selection and Preparation
2.1 Modeling Data
The ECMWF and GFS models mentioned in §1 are among the major global numerical weather models used for daily weather forecast. In this study, we follow our earlier study and use the GFS model as well. Firstly, we retrieve the GFS data in grid 003 ( grid) for the period of July 2005 to June 2008 from the National Operational Model Archive & Distribution System (NOMADS; see Rutledge et al., 2006). The data is in 3-hourly interval for forecast lead time () up to 180h and is produced four times per day at 00Z, 06Z, 12Z and 18Z (we refer each of them as one “initialization” hereafter). The data is then weight-averaged into the spatial resolution of to match the definition of the observational data that to be described in §2.2.
The forecast cloud fraction in decimals, , is computed using the Xu-Randall cloud scheme (Xu & Randall, 1996):
where is the environmental relative humidity, is the liquid water mixing ratio, is the saturation specific humidity, and , , and are empirical coefficients. is calculated with respect to water phase or ice phase and environmental temperature.
We will deal with five types of cloud cover in this study: total cloud cover, which accounts for cloud cover over the entire atmospheric column and is most closely related to observational astronomy; layer cloud cover (low, mid, and high level), that divides according to the cloud-top pressure (680hPa, 440hPa, and hPa), which is particularly useful for observatories that mostly affected by a certain cloud type (e.g. high cloud for high altitude observatories); and convective cloud, which is mostly associated with convective weather that can be an unwarned threat for astronomical observation.
The GFS model divides the whole atmospheric column into 64 sub-layers for simulation, and cloud cover is derived under the assumption that clouds in all sub-layers for the corresponding layer are maximally randomly overlapped (Yang et al., 2006). The exception is convective cloud, which is derived based on the method proposed by Pan & Wu (1995).
2.2 Observational Data
Rather than using the “traditional” surface observation, we decide to use calibrated satellite observation in this study. The reason is that standard cloud observation is not practiced by a number of surface meteorological stations, as such observation needs to be carried out by expertise observers, therefore qualified observations are mostly limited to inhabited area with dense population, of which is avoid by most astronomical observatories. Satellite observation, on the other hand, has a better temporal and spatial coverage. Since it is carried out by robotic observers and is reduced following identical algorithms, it is also easier to determine the scale of uncertainty. A good source of such data is the International Satellite Cloud Climatology Project (ISCCP; cf. Schiffer & Rossow, 1983; Rossow & Schiffer, 1999), which we will use for our study.
We retrieve the 3-hourly ISCCP D1 data from the ISCCP database for the same time period as the GFS data. The D1 dataset is in a spatial resolution of 280km and includes cloud cover data for total, low, mid and high cloud as well as convective cloud. The data is determined from raw satellite observation of cloud top pressure and optical thickness, and is in the same definition of the GFS data that we used. We then transform the ISCCP data from equal-area grid into equal-angle grid following the method by Rossow et al. (1996), to match the projection setup of the GFS data.
The uncertainty of ISCCP data is estimated to be for individual cases and less than for 30-day means (Schiffer & Rossow, 1983; Rossow & Schiffer, 1999). However, additional studies did reveal some observational tendencies for each cloud type:
Rossow et al. (1993) (surface observations): ISCCP is 0.10 too low over land (less in summer and more in winter);
Rossow & Schiffer (1999) (Stratospheric Aerosol and Gas Experiments, High-Resolution Infrared Sounder, and surface observations): ISCCP high clouds are at least 0.05-0.10 too low;
Wielicki & Parker (1992) (satellite observations): overall tendencies of ISCCP low clouds are less than 0.1.
As noted by Curry & Ebert (1992); Rossow et al. (1993), the ISCCP observations over the polar regions suffered from strong seasonal tendencies due to low visual and thermal contrast between surface and clouds; therefore, we will focus at the region between S and N in our study, despite we will still include results from polar regions in figures.
3 Evaluation and Result
3.1 Evaluation Methodology
We generate over 1 million GFS-ISCCP data pairs of every initialization, forecast time point and cloud type for the entire time period of interest. To get a more objective evaluation of the model forecast skill for total cloud cover, we compose three additional models that are to be compared with the ISCCP data:
Climatological model, which created by averaging ISCCP cloud data from July 2004 to June 2005. This model will be used to assess whether the GFS model is more skilled than statistical climatology forecast;
Randomize model, which creates random series of pseudo global cloud fields under a uniform distribution. This model will be used to assess whether the GFS model is more skilled than unskilled guesses;
Persistence model, which fix at the observation at for a given initialization throughout the forecast period. This model will be used to compare the GFS model against a persistent “guess”.
Limiting by computational resource, we randomly choose July 2006 for such comparisons. We will show that monthly variation of forecast error is not significant in the period of study, so the July 2006 result is representative.
We use a different evaluation scheme for convective cloud. Although we are dealing with the term “convective cloud cover” or “fractional convective cloud”, there is virtually no scientific/observational meaning in this term. It is due to the small scales of most convective clouds comparing with the spatial resolution of global model or satellite camera (which are mostly at tens of kilometers), so such cloud can only be represented in fractional numbers in model outputs or observations. Therefore, in addition to fractional comparisons, we also binary degenerate the modeling and observational data, so binary statistical indicators can be used to assess the forecast skill (see §3.4.2).
3.2 Total Cloud
3.2.1 General Forecast Accuracy
Figure 1 shows the 3-year mean of forecast minus observation (abbreviated as below) of total cloud cover forecast at at 12Z initialization; we notice that the setup for other time points are more or less the same and do not include them in the paper. Figure 2 shows the 3-year global mean (excluding polar regions) probability with forecast error throughout the forecast period ( from 0h to 180h) at 12Z initialization. Although root-mean-square error (RMSE) is commonly used on prediction evaluation, we notice that the preliminary RMSE figure behaves almost the same to the tendency figure (Figure 1), which suggests that the main contribution of forecast RMSE to be persistent regional tendency rather than dispersion. Therefore, we argue that RMSE distribution and variation is representable with Figure 1 and Figure 2.
In Table 1 and Table 2, our analysis shows that the global mean (excluding polar regions) varies from -6.43% to -8.93% depending on initializations. Negative values are mostly contributed by low values over ocean, especially western coastal regions at mid latitude. The forecast over land matches the observation well (between and throughout the forecast period), but it may be positively biased according to the suggested underestimation of ISCCP data over land by Rossow et al. (1993). Meanwhile, the difference between each initialization is not significant comparing to the tendency (only or less). We also find the negative tendency over the southern hemisphere is stronger than that of northern hemisphere.
|Initialization||S to N||to N||S to|
Figure 2 shows the probability of forecast error gradually decays from at to at . One may argue that has covered 3/5 of the possible range () which may lift the probability; however, cloud cover is not uniformly distributed: in a significant number of occasions, the cloud cover is either close to or , so is a fairly reasonable constraint. We will also show that the GFS model is solidly better than the random-guess model in later section.
3.2.2 Daily and Seasonal Variation
Since most astronomical optical observations are conducted in night hours, we are interested in examining the daily variation of forecast accuracy. We divide the entire globe into 24 time zones with equally longitudinal spacing, and average the series with respect to local hour in each time zone. As illustrated in Figure 3, the daily value varies between to for the entire globe (excluding polar regions) and ocean, but for land it varies from in the morning to to in night hours. However, considering a underestimation of ISCCP data over land in day and underestimation in night (Rossow et al., 1993), the actual could be as low as in day but near in night.
Figure 4 shows a clear seasonal variation of mean that reaches maximum in winter and minimum in summer in respective hemisphere, but it might not reflect the actual situation, as the study of Rossow et al. (1993) has indicated that seasonal difference of ISCCP tendency can be as large as 9%, with a reverse maximum-minimum setup ( in summer and in winter) than Figure 4. In such a case, it is possible that the GFS model forecast is closer to actual situation in summer, rather than in winter as suggested in the figure. We also attempt to identify any annual variation (Figure 5), but no significant feature can be noted, possibly due to insufficient temporal coverage.
3.2.3 Comparison with Climatology, Randomize and Persistence Models
As described before, we generated three additional models and will use them to compare with the ISCCP data for July 2006. The result is shown in Figure 6 and, in our opinion, is comparable with Figure 2 despite different temporal coverage, as we have shown the seasonal variation to be insignificant comparing to overall tendency.
We can see the GFS model is superior than all three other models in most time points. Statistically, the persistence model is best of all for , but this is not meaningful as the GFS model data is not available after approximately 4-5h of the respective initialization time due to computational layover. We note that the ISCCP climatology model becomes better than the GFS model after , and that both models are more skilled than random guess at all times. From this result, we can conclude that the GFS model is skilled, and performs better than the ISCCP climatology model until .
3.3 Layer Cloud
The evaluation result (Table 3, Figure 7, Figure 8 and Figure 9) has revealed the global overestimation of high cloud from the GFS model, but this result is affected by underestimation of similar scale in ISCCP data (Rossow & Schiffer, 1999) and is questionable. We can also identify underestimation of low cloud off the west coast of major continents at mid latitude, which seems to be the major contribution to the underestimation of total cloud in these regions. At high latitude, the model tends to overestimate the low cloud.
|Cloud type||Region||3-year mean|
|Low cloud||S to N||+3.08%|
|Mid cloud||S to N||-1.21%|
|High cloud||S to N||+17.11%|
The probability of error forecast for low cloud is significantly lower than other two clouds (Figure 8), but does not appear to affect the total cloud at a small . This behavior, together with the unexpected negative for total cloud (while for layer clouds are mostly positive), could indicate that the assumption of maximally randomly overlaps (see §2.1) between each cloud layers may not apply at all times.
3.4 Convective Cloud
3.4.1 Fractional Evaluation
Direct fractional comparison (Table 4 and Figure 10) shows that model overestimation only occurs in tropical area (within latitude in both hemispheres), while moderate to strong underestimation (mostly ) occurs in mid-to-high latitude area. Considering the fact that convective weather is frequent in tropic area throughout the year but is relatively rare in mid-to-high latitude area, we may conclude that the current model misses and/or underestimates a significant fraction of convective cloud in mid-to-high latitude area, while it tends to overestimate the number and/or intensity of convective cloud in tropical area.
3.4.2 Binary Evaluation
An alternative way to evaluate the GFS convective cloud forecast is to degenerate the forecast and observation into binary value, so we can focus on the occurrence of such cloud rather than a mixture of occurrence and intensity. As discussed in earlier section, we set a cut-off limit at 1% for convective cloud fraction to put both the modeling and observational data into the categories of “convective cloud” and “no convective cloud”. We use the following statistical indicators for evaluation: Proportion of Perfect Forecasts (PPF), Probability of Detection (POD), False Alarm Rate (FAR) and Frequency Bias Index (FBI). In the following expressions, indicates “hits” (forecasted and observed), indicates “false alarms” (forecasted but not observed), indicates “missed” (not forecasted but observed), and indicates to not-forecasted and not-observed events. The result is shown in Figure 11.
As we have shown that tropical overestimation only composed a relatively small fraction in the sample, the setup of Figure 11 more or less represents the situation of mid-to-high latitude area. This is affirmed by the FBI value which lies way below 1, suggesting a global underestimation of convective cloud.
The PPF varies around 0.6 and creates a decent picture of the forecast ability, but the rare occurrence of convective cloud in most areas will lead to a significant fraction of PPF being contributed from “Z” events (not-forecasted and not-observed events), so we must take POD and FAR into account for a unbiased view. From POD we notice that only slightly less than half of the convective cloud can be detected by the model. Since convective cloud is most commonly seen in tropical area, the POD for mid-to-high latitude area must be lower. However, globally speaking, the model is still well skilled as the FAR is about 2 times lower than the POD.
4 Concluding Remarks
To study the reliability of cloud forecast from the GFS model as a reference of astronomical observations, we analyzed a 3-year sample composed with the GFS modeling data and satellite observation data from ISCCP. The results are summarized as follow:
For total cloud cover forecast, there is a slight global underestimation from the GFS model, but is more or less within the observational uncertainty of the ISCCP data. For local night hours, the model forecast roughly agrees with the observation if taking account the observational tendency of ISCCP data as suggested by earlier studies. The global mean probability (excluding polar regions) of forecast error gradually declines from 73% to 58% from to . Further investigation suggests that climatology model based on ISCCP observation overtakes the GFS model after , but both models are significantly more skilled than random guesses.
We found a strong underestimation of low clouds in subtropical regions off the west coast of both hemispheres. Overestimation of low clouds in high latitude areas can also be identified. We found some globally overestimation of high clouds, but is most likely compromised by a similar scale of observational tendency in the ISCCP data. We noted the inconsistency of mean global forecast errors between total and layer clouds, which might be due to the layer overlapping assumption built in the model.
For convective cloud forecast, the GFS model tends to overestimate the occurrence and/or intensity of convective cloud in tropical areas but tends to underestimate that in subtropical and high latitude areas. We found the GFS model can only identify less than half of the convective cloud globally. However, the convective cloud forecast is still skilled, as the detection rate is about two times higher than the false alarm rate, leading to a proportion of prefect forecast of .
In all, we observed a good overall consistency between the GFS model forecast and ISCCP observation throughout the time period of interest. For total cloud cover, our result suggested a satisfactory performance of the GFS model for the need of observation scheduling up to a week ahead. However, for layer and convective cloud, which can be considerably important for observatories located in certain environment (for example, high cloud for high altitude observatories), the success rate of the GFS model is relatively less satisfying than that of total cloud forecast.
We would like to thank several anonymous reviewers as well as Fangling Yang and William Rossow for their constructive comments and discussions that help us to make significant improvements of this work. We also would like to thank Chen Junwen from Atmospheric Exploration Laboratory (EESAEL) at Sun Yat-sen University, Cui Chenzhou (National Astronomical Observatory), Lin Qing and Tang Haiming (Shanghai Astronomical Observatory), for providing assistance on computational resources. We would like to especially thank the “meteo cat” who had visited the EESAEL a few times and brought some joys to us. The data analysis of this work was accomplished with the computer resource at EESAEL. The GFS data were obtained from NOMADS (National Operational Model Archive and Distribution System) at NOAA/NESDIS/NCDC. The ISCCP D1 data were obtained from the International Satellite Cloud Climatology Project data archives at NOAA/NESDIS/NCDC Satellite Services Group, firstname.lastname@example.org, on July to October, 2010.
- Curry & Ebert (1992) Curry J. A., Ebert E. E., 1992, Journal of Climate, 5, 1267
- Erasmus & Sarazin (2001) Erasmus D. A., Sarazin M. S., 2001, in Russell J. E., Schaefer K., Lado-Bordowsky O., eds, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series Vol. 4168 of Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Forecasting precipitable water vapor and cirrus cloud cover for astronomical observatories: satellite image processing guided by synoptic model dissemination data. pp 317–328
- Hahn et al. (1995) Hahn C. J., Warren S. G., London J., 1995, Journal of Climate, 8, 1429
- Hinkelman et al. (1999) Hinkelman L. M., Ackerman T. P., Marchand R. T., 1999, J. Geophys. Res., 104, 19535
- Luo et al. (2005) Luo Y., Krueger S. K., Moorthi S., 2005, Journal of the Atmospheric Sciences, 62, 1428
- Pan & Wu (1995) Pan H.-L., Wu W.-S., 1995, NMC Office Note, 409, 40
- Rossow & Schiffer (1999) Rossow W. B., Schiffer R. A., 1999, Bulletin of the American Meteorological Society, 80, 2261
- Rossow et al. (1996) Rossow W. B., Walker A. W., Beuschel D. E., Roiter M. D., 1996, WMO/TD, 737, 115
- Rossow et al. (1993) Rossow W. B., Walker A. W., Garder L. C., 1993, Journal of Climate, 6, 2394
- Rutledge et al. (2006) Rutledge G. K., Alpert J., Ebisuzaki W., 2006, Bulletin of the American Meteorological Society, 87, 327
- Sawyer Hogg (1951) Sawyer Hogg H., 1951, JRASC, 45, 173
- Schiffer & Rossow (1983) Schiffer R. A., Rossow W. B., 1983, Bulletin of the American Meteorological Society, 64, 779
- Stephens (2005) Stephens G. L., 2005, Journal of Climate, 18, 237
- Stubenrauch et al. (1999) Stubenrauch C. J., Rossow W. B., Cheruy F., Chedin A., Scott N. A., 1999, Journal of Climate, 12, 2214
- Sundqvist et al. (1989) Sundqvist H., Berge E., Kristjánsson J. E., 1989, Monthly Weather Review, 117, 1641
- Wielicki & Parker (1992) Wielicki B. A., Parker L., 1992, J. Geophys. Res., 97, 12799
- Xu & Randall (1996) Xu K. M., Randall D. A., 1996, Journal of the Atmospheric Sciences, 53, 3084
- Yang et al. (2006) Yang F., Pan H.-L., Krueger S. K., Moorthi S., Lord S. J., 2006, Monthly Weather Review, 134, 3668
- Ye (2011) Ye Q.-Z., 2011, PASP, 123, 113