I have always wondered exactly how those nice homogenised monthly temperature averages for some 7000 weather stations within CRUTEM and GHCN were actually produced. This is the first part of my investigation.
A vast amount of raw temperature data has now become available from the Australian Bureau of Meteorology. In this first post we looks at the hourly temperature measurements from 112 weather stations from 1950 to 2015. This data was provided by Ron Graf who had acquired it from BOM with the original intention of studying climate change effects on diurnal temperatures, contrasting inland sites with coastal sites. The distribution of available stations are shown below.
There 8 measurements per day at larger stations, while at many other stations only after 1986 when measurements were automated. Before 1986 these remote stations had just 2 manual readings per day taken at 9am and at 15pm. The volume of data is still huge with around 250,000 readings for large stations like Sydney. I have processed all the station data as follows.
- Daily averages were calculated over all hourly readings per day and the minimum and maximum temperatures recorded. In some early cases only two readings per day are available typically 9am and 3pm, thereby biasing up minimum temperatures. An example of this can be seen in Leonora. After around 1988 this effect disappears. This change in timing of the daily temperature measurements can also bias the mean temperature because it restricts the range of temperatures over which the average is calculated.
2. Monthly averages for each station were calculated by averaging together all available daily averages in that month. The monthly maximum and minimum temperatures were taken as the highest and lowest recorded daily extremes. Most stations have some problem of missing measurements and in the worst cases a few stations are missing several years of data. To handle this problem all missing daily, monthly and yearly values are assigned a NAN value, which means ‘not a value’. As such they are excluded from all calculations of averages and temperature extremes.
3. Yearly averages were then calculated from the daily averages in exactly the same way as for monthly averages. Likewise, the yearly maximum and minimum temperatures correspond to absolute daily extreme values for each year. The date and time of each extreme was also recorded. An example for Cairns is shown below.
In order to calculate temperature anomalies at each station I first had to calculate the 12 monthly averages over a 30 year period from 1961-1990. These are the so-called ‘normal’ climate expectation values for that station each month in the year. Temperature anomalies for any given month in the 75-year series are then just the +- deviations from this normal value. So for the case of Sydney Airport then we get the following result, which appears indeed to show a small warming of about 1C between 1955 and 2015.
However, most of the other stations show no effect at all or even a cooling trend – for example Boulia Airport.
Furthermore you would be hard pushed to detect any real difference between the 1950s and the 2010s in diurnal temperatures range even in Sydney. Anomalies pick out and accentuate tiny differences, but they are also very good at showing up any biases within the data.
Lets go ahead anyway and calculate the annual averaged anomaly over the whole of Australia. To do this I use a 2-d triangulation in Lat,Lon to form an area weighted average for each month and year. Here is the result
There is no warming ! At first sight this result disagrees with BOMs result based on the ACORN-SAT data. So what is wrong? Firstly it demonstrates just how sensitive anomalies are to any small underlying biases. To compare this trend with standard results I took the GHCN V3 data from which I extracted all the Australian stations and then performed exactly the same analysis on them as I did for the hourly data. This is how they compare.
The detailed yearly structure is identical but the trend is totally different. So what am I doing wrong. One obvious bias is the change in the number of hourly readings per day around 1986, so as a result I convinced myself that we couldn’t use the hourly data for climate change studies. However, while I was writing this post I had a brainwave. Since we know that the bias is caused by an increase in measurement times after 1986, why not just simply remove it by restricting the analysis to just the two original readings per day – 9am and 15pm. This means discarding about 70% of all the hourly measurement data, but it solves the problem. So here is the new result.
The hourly derived annual anomalies now almost exactly agree with those from the Australian GHCN V3 values. So the mystery is solved. The cooling trend after 1986 was simply due to extra sampling times being added following the automation of data recording. Basically, it was the minimum daily temperatures that fell after measurements at 3am were introduced, and as a consequence that biased downwards the monthly anomalies after 1986.
So what exactly does homogenisation do? What it is supposed to do is to detect shifts in temperature data caused either by either physical movements of the weather station to a different location, or by a change in instrumentation. Once detected the relevant temperatures are supposed to be corrected by shifting them up or down by a fixed amount to compensate. The other ‘black magic’ correction is intended to detect any systematic ‘errors’ in trends relative to those of nearby stations and then ‘correct’ for them. This second process is far from clear, especially in a place like Australia where the next nearest station can be located up to 1000km away. Let’s look at a couple of examples.
Winton clearly has a kink in temperatures around 1976 and perhaps another around 2000. So how is it possible that GHCN converts these into a linear rise in temperatures of about 1.5C over the same period? There are at least three levels to this puzzle. A) the raw temperature measurements. B) the adjusted (first level homogenisation) by the local Met Office which passes the monthly averages to GHCN & CRUTEM4. C) A second level regional homogenisation by GHCN or by CRU to derive their station normals and temperature anomalies.
This is an interesting case which was first described by Dr Marohasy in 2014 , and resulted in a twitter exchange with Gavin Schmidt. This is how NASA turn a fairly flat temperature distribution with at least one small systematic shift into a warming pattern.
There trend is flat until 1990 with perhaps a small net rise < 1C at most. The correction algorithm appears to have annulled at least one shift . The homogenisation algorithm is supposedly based on comparing trends with those at neighbouring stations but in Australia those distances can be huge.
There are stations which show a warming trend with the 8 readings per day resolution data, but these tend to be mainly large cities on the coast. For example Sydney, Melbourne, Perth and Adelaide.
Perhaps these large cities responsible for homogenising warming trends in other stations or does the algorithm itself has a built in warming assumption? Let’s look at the most isolated stations in Australia – Alice Springs. Here is the result from the raw BOM temperature data (9am & 3pm only).
There is a systematic kink between 1974 and 1983, but otherwise the trend shows an overall increase of < 1C. The nearest stations are about 1000km away, so it is difficult to believe that these could be capable of ‘increasing’ the trend. Yet this is exactly what GHCN seems to do.
Note how the kink has simply been blanked out. It is hard to believe that any objective homogenisation procedure should always end up increasing temperature trends . Why does the inverse never happen? Is there some built-in assumption that a warming trend is the norm.
The available hourly temperature measurements from the 112 stations have been processed into daily, monthly and yearly temperature averages, maxima and minima. Temperature normal for each station has been derived based on a baseline of 1960-1991. Many stations show a bias due to a step change in the number and timing of temperature measurements over 70 years. Before ~1986 measurements were carried out manually and remote stations had just 2 measurements, at 9am and 3pm. Thereafter there were 8 automatic measurements per day. This bias affects diurnal range trends and the daily Tav trends. The large stations were unaffected because they had always had ~8 measurements per day.
The average temperature anomaly for Australia calculated from the mean of all daily measurements shows no warming trend. However, if you simply use only the continuous 9am and 3pm readings for all stations then the result is very similar to that of GHCN and the official BOM trend on their web site. We will look at the ACORN-SAT data in the next post.