I remember some 10 years ago (maybe even 20!) there was a site that aggregated compared and combined forecast sources (I think at the time it only did the US, and the averaging weights didn’t vary by location). Does anyone know of a site like that nowadays? Or even just one that only compares models/sources by accuracy on different indicators? I’m curious bc NOAA and CMC often give me disparate forecasts for Australia, and so do these:
1 Like
Flowx has a compare feature (if you didn’t already know) where you can long-press the graph and select compare.
This doesn’t merge the data sources together but shows them side-by-side so you can get a visual consensus which I find really useful to give me greater confidence if they all agree.
I have a screenshot showing this.
2 Likes
I think you’re thinking of Model Output Statistics (MOS) and other machine learning algorithms.
This is what most weather providers provide - Wunderground, Accuweather, OpenWeatherMap, etc… They combine most of the models you see in Flowx and the difference between them is their secret sauce. It’s ironic that Flowx compares weather simulation results and the app you show above compares the different weather providers that merge these weather simulations.
It’s a bit pointless arguing who has the best secret sauce because they can be location dependent, season dependent, etc… Wunderground might work better for New York but Accuweather might be better for London, etc…
The most common real measured data used in MOS are from airports - pretty clear and open areas - if you live in a micro-climate, e.g., a valley, these MOS predictions will poor for your location.
Really you should generate your own MOS for your location - this is something I’d like to do one day.
You kind-of generate your own MOS when you watch and compare weather simulations, over time your brain starts to learn how to interpret the simulation results, merge them and make it’s own prediction.
3 Likes
I didn’t! Thank you, @jolley.
Interesting! Didn’t know that either. Doesn’t Wunderground also use private weather stations to make their own forecasts?
Right, so what I wanted to know is which one is best for where I am.
Yeah, well, I was looking for something a little more analytical than my own guesswork: some statistical analysis of which models, or combination thereof, are most accurate for a given location (and season, if necessary).
1 Like
I’ve found that model accuracy varies daily.
In my area UK, in general, Arome was my go-to model, then KNMI was the better model. But
recently Hirlam has proved to be the best.
This conclusion is based on observable wind direction, wind speed, temperature and precipitation.
But, in all honesty, on any particular day, any of the relevant models could be the better interpreter of data.
2 Likes
Yes and no. MOS needs real measurements to correct the bias in the simulated prediction. They use the measurements from private weather stations, along with other measurements (airports, satellite, etc…).
3 Likes
This is what I mean by your brain generating your own MOS over time. Interestingly, I find that my brain doesn’t give you corrected numbers like MOS does, but instead a rough image of what the day would be like. E.g., windy with passing showers through the day, or hot and humid.
3 Likes
I wonder if you get better predictions by creating your own MOS based on your own measurements compared to finding the best out of weather providers.
Here is a fun idea: did you know you can create a MOS for outside your house and one for inside the house (assuming you don’t have aircon) - even for individual rooms.
2 Likes
Way above my pay grade 
So, I take that as a no for the question of whether there are published accuracy comparisons across models, am I right?
2 Likes
The Models only show a prediction radar does not reach the ground so your radar can be showing alot of rain or high winds etc. but does not mean it will be the same on the ground. topography makes a difference as well. for example where I live I have mountains on the North West and South so the radar will show high winds alot of rain etc but because of the mountains it cold go around me or hit could be high winds or alot of rain or barely any because it evaporates/ reabsorbed back into the storm cell. that’s why the National Weather Service has a Skywarn Weather Spotter or Storm Chasers to get an eye on what’s going on. you won’t get an exact forecast from any model just a good idea of what might happen. another example one year we got a dusting of snow south end where I live at the north end of town we had 13 inches of snow.
2 Likes
Are you saying forecasting models aren’t meant to forecast, @BrianLY-38?
Perfectly accurate? Couse not. They’re models after all. (“exact”, as quant models, they all are)
Presumably, all models are calibrated against actual readings. I was just wondering about sites and/or publications that compare the models’ ability in matching those readings (by season, location, etc). I.e., their accuracy.
1 Like
Not Saying That at all Just saying Weather Forecasters / Meteorologists Use all the Models To Get a General Forecast. Weather Radars Can Tell You How much of something is in the atmosphere it can pick up dust a large flock of birds even a large swarm of insects so depending on the time of year they can determine what the radar is picking up and “Predict” from current model reports and past weather events to give a “Prediction” that’s why they say "we can get 1 to 3 inches etc. Radar can’t tell you how much of what is being seen is going to fall to the ground.
2 Likes
Right. And whoever builds those models backtests and calibrates them to give the most accurate predictions. How accurate are they for each location, is my question.
2 Likes
Or dubble tap on it, as a shortcut.
and then dubble tap while in compare view to go back, or to change to another model.
1 Like
In my experience, the “best” model depends on the type of weather, and the direktion of the wind.
Some models will more accurate tell if it is going to rain at all or not, while other models will more accurate predict the cloudiness.
I think that it’s worth pointing out that unfortunately “calibrating a weather forecast model” is not as simple as “turning just a few knobs”.
In a werry simple case, like the Malkus_waterwheel, one could make a model of it, and then calibrate the model to the outcome of a particular fysikal instance of such a experiment, given the inaccuracies in the experiment and various flow rates.
Trying to do the samme with a global weather simulation is a different game, since the number of different input data is so vastly larger.
Weather systems, just like the Malkus_waterwheel, are chaotic “systems sensitive to initial conditions”, so any calibration to a predicting model is only walid for a limited period of time, and hence needs to be done repeatedly.
1 Like
There is the key word “Prediction” = “Guess” They can provide the absolute best guess possible but again they can’t give EXACT. because you have Micro Climates and Topography.
I wish I could put what is in my head into words but it’s difficult for me to do my apologies
this might help
"the perception of the weather will never accurately match the reality of the weather. This is because forecast parameters include a good amount of probability and chance.
For instance, meteorologists use percentages to describe the chance of precipitation occurring at any point you select in the area. Mathematically, the percentage comes from the confidence that precipitation will occur somewhere in the forecast area multiplied by the percent of the area that will receive measurable precipitation.
Today, we often can be 100 percent confident that precipitation will occur somewhere in the forecast area. How much of that area will be covered by rain at a single time is the factor constantly changing and harder to determine. Probability is ultimately why the forecast is perceived to be accurate or inaccurate"
2 Likes
How long is a piece of string?
You need to be more precise. Accuracy for what - temperature, precipitation? What do you consider accurate - 1 degree or 0.1 degree? What part of the year?
What you are wanting is not readily available, if at all.
Weather simulations are designed to model weather physics (Navier-Stokes equations, turbulance, pressure systems, etc…) and they get improved based on how well they model those phenomena. They don’t care about location - they are location agnostic - other than the terrain.
If you really want to understand model accuracy, read the section on “Forecast Quality and Verification” here:
https://www.eoas.ubc.ca/books/Practical_Meteorology/prmet102/Ch20-nwp-v102.pdf
It’s worth skim reading the rest of the chapter too.
3 Likes
Seems like to me you’re confusing exactness, i.e. precision, with accuracy, @BrianLY-38. Any estimate (or forecast, a particular case), in any model, in any discipline, has a confidence interval around it. That doesn’t make the estimate any less accurate or precise (exact), just less significant, the wider the confidence interval.
Accuracy is not a matter of perception; it’s the measurable error between a model’s estimates/predictions and the observed quantity being estimated/predicted.
A model needn’t be chaotic for that to be the case. In fact, that’s true of pretty much all models, chaotic or not, simply bc models are, by definition, an approximation of the observed phenomenon projected on a subspace of its factors.
More and more, it seems like all we’re doing is…
Yes and yes. Of all models’ predictions, no less.
Right, but presumably models calibrated with, say, N American data are more accurate for N America than Oceania, and vice versa, no? Didn’t you say different models are better for different locations, @duane? How can they be “location agnostic” then?
All I’m trying to find out is which models, or combinations thereof, are best for my location (if not for all time and circumstance, for each of these).
1 Like