Introduction The 2023 Rugby World Cup is finally here! Hosted by France, this year current Champions South Africa will be looking to defend their title. The current book maker favourites to win the tournament are New Zealand, France, South Africa and Ireland.   This blog explores using probabilistic simulation to understand which teams are likely to win the pool stages and progress to the knock-out rounds. Given the sparsity of available information we do this by modelling the win-rate (i.e. fixture win percentage) of a given side and use random sampling to simulate fixture outcomes.   Interactive Dashboard available here Code available here   Why use Simulation? Simulation can be a powerful tool when working with sparse data. Unlike club level rugby, where leagues run each season with sides playing each other regularly, international rugby matches are played less frequently and with a notable divide between the ‘Tier 1’ and ‘Tier 2’ nations.   The ‘Tier 1’ nations tend to be large national sides which play every year in prestigious competitions such as The Championship and The Six Nations. The ‘Tier 2’ sides play infrequently, in some cases with semi-professional sides and rarely have an opportunity to play against ‘Tier 1’ sides.   The Rugby World Cup is played in two stages, firstly pools where teams are assigned by seeding and then a knock-out competition where the top two sides from each pool enter the Quarter-Finals. Outside of the Rugby World Cup, ‘Tier 2’ sides rarely get to play international test fixtures against ‘Tier 1’ sides.   Because of this, predicting likely outcomes in the tournament is difficult – since theoretically anyone can win it, but we have very few observations from past matches between sides.   Hierarchical Bayesian Models Bayesian statistics is a branch of statistics which combines prior beliefs, or information, with observed data to make inference over parameters of interest. It differs from traditional frequentist statistics by updating prior beliefs based on evidence. This makes it well suited to statistical tasks which seek to quantify beliefs or quantify uncertainty in data.   A key feature of Bayesian probability is that parameters of interest are treated as random variables. Thus, they have a distribution which we can draw samples from. The updated beliefs, known as the posterior distribution, can be characterised as:   p(θ|y) ∝ p(y|θ)p(θ)   Where θ represents our parameter of interest and y represents our observations. It is often not feasible to calculate this directly, hence we typically use an approach such as Markov Chain Monte Carlo (MCMC). This approach uses random sampling to iteratively adjust our belief on the shape of p(θ|y) over many samples. Samples taken from the converged distribution give us a direct sample from what we approximate as the posterior distribution.   A hierarchical Bayesian model is one which has multiple levels to it where the model has shared parameters instead of constants as priors. Each lower level of the hierarchy comes from a shared population parameter. Mathematically, we can characterise this as:       p(ρ,θ|y) ∝ p(ρ)p(θ|ρ)p(y|θ)   Where ρ represents the common population parameter and θ represents a vector of lower level parameters.   A benefit of using hierarchical models is that inter-dependencies and relative differences between lower levels in the hierarchy are learned directly. As well as this, estimates over one of the lower-level parameters simultaneously updates our beliefs on other members of the lower-level hierarchy. Its because of this that sports data is particularly well suited to hierarchical models. In the context of modelling the win-rate, our higher level of the hierarchy is an assumed population average win-rate and the lower-level is the individualised parameter distributions for team level win-rate.   One drawback of hierarchical models (within the context of sports analytics) is the problem of shrinkage. This is where teams who are at the extreme (i.e. top or bottom performance) tend to have their estimates shrunk towards the mean. This is a particular problem in this exercise since we have fewer observations for ‘Tier 2’ sides, and in many cases the model appears to overestimate the performance of these sides.   Ultimately, we are using Bayesian analysis here for uncertainty quantification. The more observations we have, the more certain we can be about the true underlying distribution of our random variables. Unfortunately, for sides with fewer observations we are more uncertain about performance. This is where simulation helps to stabilise our model – over many simulations on average the high performing sides consistently win a higher percentage of the fixtures leading to a more reasonable estimate of tournament performance.   That said, this is also the beauty of sport – in theory, anything can happen!   Project Workflow The workflow for this project can be described as below:     Here we combine two models. Firstly, we estimate the team level win-rate in order to generate a large sample from the posterior distributions. Secondly, we use these samples to iteratively simulate through the tournament. Since we do not have information available on England versus Chile, for example, we instead use the win-rates for each side to simulate the fixture winner. We do this by taking a sample from the distribution for England’s win-rate, and taking a sample from Chile’s win-rate. We then compare the sampled win rates and allocate the winner as the side with the higher win-rate. Put simply, it’s a bit like statistical Top Trumps!   Gathering and Cleansing Data Data was scraped from web pages and then parsed using the Python package BS4. For ‘Tier 1’ sides we get fixture results for the last 18 months, whereas for ‘Tier 2’ sides we extend this to fixtures going back to the last Rugby World Cup in 2019.   Once data is parsed and cleansed (using regular expressions) we use a combination of the SAS DataStep and FedSQL to pre-process the data. Finally, we wrangle the data such that we have the win-rate from fixture outcomes for both ‘Tier 1’ and ‘Tier 2’ sides where the win-rate is presented for the ‘home side’. Home advantage is not considered in the model as only the host nation has a true home advantage. Instead we flip the home and away teams to add more information to the model. For example, below, we have an equivalent row for France v Italy as Italy v France and a win-rate of 0.     Building our Probability Model The hierarchical Bayesian model is built using the Python library PYMC. The model prior for win-rate is a Beta random variable with hyperpriors for α and β. The prior is then used as the probability of a Bernoulli random variable with observations being the observed win-rate.     The posterior samples show that the Highest Density Interval (HDI) for the win-rate have wider intervals for teams where we have fewer, or lower quality, observations. This is where the model may overestimate the performance of sides like Portugal. Looking at their distribution interval there are situations where they have a higher win-rate than the book makers favourites to win.   Improving our Model As the tournament progresses, we’ll have more information with which we can update our beliefs. I hope to re-run this model as we exit the pool stages. The tournament is approximately 2 months long, re-running the simulations with more observations (and possibly trimming out some older observations) may lead to a more stable model.   Creating our Simulation Model The simulation model itself is a humble Python function. For each iteration we simulate the pool stages, rank and select the top two sides, then begin the knock-out stages. For each iteration a simulated tournament is created, returned and appended to a master dataset.     Preparing Simulation Data for Visualization The simulations are recorded in a long format. We firstly do some post-processing to create flags for winners at the pool and knock-out stages. This allows us to visualise the summary statistics on likely winners of each stage.   We also prepare the data for a Path Analysis. Visualising the pathways for teams via a Sankey Diagram is a really rich way of understanding the possible tournament outcomes for a given side. To do this we transpose the data then collapse it back into a long format with PROC TRANSPOSE and the DataStep.   Visualizing our Simulations Finally, we’re able to bring our simulations to life using SAS Visual Analytics. The report is simple, using only the Path Analysis and Bar Chart visualizations. The report is then exported as a Report Package where it is then available as an interactive offline report. I then deployed this onto cloud infrastructure to make it available for others to access.     Analysis So, can we use these simulations to learn anything about who is likely to win the tournament?   Likely Winners The simulation model has France as winners ~70% of the time. New Zealand come in second with ~14% of wins. Other strong candidates include Ireland (~9.4%) and Argentina (~6%).   Likely Runner-Up Whilst France are clear favorites in the model to win, second place is more tightly contested. New Zealand come out top with ~38%, followed by Ireland with ~26% of simulated outcomes.   Likely Third Placed Surprisingly, Ireland (recently ranked #1 in the world) do not feature in the top 5 for coming 3 rd . Argentina win bronze place in ~47% of simulations. Wales (~16%) and Georgia (~17%) are also highly ranked, however this may be where the model is falling victim to uncertainty. Wales have had a dip in performance recently (replacing Wayne Pivac with Warren Gatland less than a year out from the tournament start). With historic strong performance this leads to a reasonably wide interval of values for win-rate. Likewise, Georgia (who are a very strong ‘Tier 2’ side) have been very impressive in recent months but still do not have many opportunities to play ‘Tier 1’ sides. This may lead to over confidence in their win-rate as they do not often play the very top sides, but win many of the fixtures that they do play.   Likely Semi-Finalists The first Semi-Final is played by the winners of Quarter-Final 1 and Quarter-Final 2. These are made up of two of the sides from the winners of Pool C and B and the runner-ups of Pool D and A. The simulations have New Zealand (~37%), Ireland (~29%) and France (~18%) as winners of Semi-Final 1. This would suggest that the model expects the runner-up of Pool A to have a higher chance of winning Semi-Final 1 than the winner of Pool B, which contains South Africa and Ireland.   South Africa in particular have been surprisingly under represented in the model. One reason for this is that Romania has been potentially over-represented as likely winner of Pool B. The runner-up of Pool D does not feature in the top 5 teams most likely to win the first Semi-Final.   The second Semi-Final is made up of the winners of Pool D and A or the runner-ups of Pool C and B. France (~70%) are overwhelming favourites to win Semi-Final 2, which is the scenario where France wins Pool A and plays the runner-up from Pool B. Argentina (~14%) and New Zealand (~10%) are the other sides likely to win the second Semi-Final. This is the scenario where Argentina wins Pool D or New Zealand wins Pool A.   Likely Quarter-Finalists The further back into the tournament we go, the more we see an over-representation on the likely performance of ‘Tier 2’ sides. Quarter Final 1 the favourite side to win is Georgia. That said, this is not an impossible scenario. Australia have had a run of poor form and have taken a very inexperienced side to the world cup leaving stars like Quade Cooper and Michael Hooper behind. Wales have also had a run of poor form going into the world cup. Ignoring Chile as an anomaly (we created a synthetic variable for Chile as we had no data) the four teams most likely to win Quarter-Final 1 are Georgia, Wales, England and Argentina. It is more likely that England feature as likely runner-up of Pool D.   Quarter-Final 2 is played between the winner of Pool B and the runner-up of Pool A. The winner of this round is most likely to be New Zealand (~39%), Ireland (~31%) and France (~16%). This suggests that Ireland are most likely to win Pool B and if they win Pool B they stand a strong chance of winning in the first Quarter-Final.   Ireland have never gotten past the Quarter-Finals in a rugby world cup and the model suggests winning Pool B puts them in the best chance of getting to the Semi-Finals. In the scenario where they come as runner-up of Pool B they’re given  ~5% chance of reaching the Semi-Finals given that they’ll have to play the winner of Pool A which is likely to be France or New Zealand.   The most likely winner of Quarter-Final 3 is Argentina (~63%). This is where the winner of Pool D plays the runner-up of Pool C. Pool C is arguably a weaker pool based on recent form of Wales and Australia suggesting that the winner of Pool D has a very strong chance of making the Semi-Finals.   Finally, Quarter-Final 4 is dominated by the winner of Pool A with France (~72%) and New Zealand (~13%) strong favourites. This suggests that the runner-up of Pool B has a relatively low chance of making the Semi-Finals.   Pool Performance The model suggests strong favourites to win each pool with France (~81%), Ireland (~64%), Georgia (~48%) and Argentina (~87%) winning Pool A, B, C, and D respectively. The predictions for Pools A and D seem sensible, though I would expect New Zealand to have a higher chance than this, however Pools B and C both over-represent the performance of ‘Tier 2’ nations. I do think Georgia have a strong chance of doing well in the pools, and Georgia did beat Wales for the first time this year however there is a high level of uncertainty in this pool. Pool B, on the other hand, vastly overestimates Romania and puts them ahead of South Africa.   There is less certainty in the model on who is likely to be runner-up in the pools. Aside from New Zealand (~60%) in Pool A, all other pools pick the runner-up less than 50% of the time. Its clear that Ireland are favouirtes to win Pool B, according to the model, with both South Africa and Scotland likely runner-up. Pool C is between Wales and Australia as runner-up and Pool D (again, ignoring Chile) has England (~36%) followed by Samoa (~21%) as runner-up.   Possible Surprises I’ll be very interested to see how Georgia fares in the pools. Clearly, there is over-representation of ‘Tier 2’ sides however Georgia have shown a lot of improvement in recent seasons and have had some big victories this year including wins over Wales, USA and Italy recently.   Argentina will be strong favourites to win Pool D, and could be a surprise in the knock-out stages this year. The model has them as favourites to come third overall.   Ireland, despite being ranked world number one recently, could have a surprise exit at the Quarter-Finals if they don’t win Pool B given how stiff competition is in Pool A.   Final Predictions Using the model here are the predictions for First, Second and Third are: Winner – France Runner-up – New Zealand (though Ireland are a close second) Third Place – Argentina   Conclusion We’ve seen how a combination of probabilistic analysis and simulation can be used to set expectations for performance in the Rugby World Cup despite having only sparse observations readily available. The model has picked clear favourites in France, New Zealand and Ireland whilst unrealistically writing off South Africa. We’ve seen that Argentina can expect to do very well, but have less than a 5% chance of winning the tournament overall.   We’ve seen that the model, whilst imperfect, does appear to capture recent form of sides. The model suffers from a lack of information for ‘Tier 2’ sides, often leading to an overestimation of performance.   Once the pool stages are complete I plan to re-run the model to see how it updates the expected results. I expect that, with many ‘Tier 2’ sides likely to not win the pool we will see more realistic estimates for South Africa.   If you enjoyed this… Check out some of my other blogs which explore sports, simulation or Bayesian analysis: Selecting the starting line up of the British & Irish Lions using optimization techniques: https://communities.sas.com/t5/SAS-Communities-Library/Using-SAS-Viya-to-Select-the-British-amp-Irish-Lions-Rugby-Team/ta-p/755754 Finding a sunken vessel using Bayesian Search Theory: https://communities.sas.com/t5/SAS-Communities-Library/Finding-a-Needle-in-a-Haystack-Bayesian-Search-Theory-with-SAS/ta-p/886204 Understanding the likelihood of children sharing the same name at school with Bayes Theorem: https://blogs.sas.com/content/hiddeninsights/2022/09/06/what-are-the-odds-will-your-child-have-the-same-name-as-their-classmates/   Did you know? France rugby use SAS Viya to boost team performance with AI & Analytics? Read more here: https://www.sas.com/en_us/customers/french-rugby-federation.html   ... View more

Introduction Having recently finished reading “The Theory That Would Not Die”, which is a history on the evolution and applications of Bayesian statistics over the decades, I found myself particularly in awe of one of the applications during the Cold War for orchestrating search effectiveness. In 1966 a US bomber which had flown to Europe from Raleigh, North Carolina, unfortunately crashed into the waters around Palomares, Spain, after a collision during mid-air refuelling. The bomber was carrying four hydrogen bombs at the time. Three were found on land, but the fourth fell into the Mediterranean Sea.   A salvage operation was organised and after two and a half months the lost bomb was found. In order to mount the search effectively, US Naval scientist Dr. John P. Craven developed a novel mathematical approach to guide the coordination of search effort effectively.   Search effectiveness is an important factor in the success of any salvage operation. Salvage operations can be challenging, costly, and risky, depending on the depth, location, and condition of the wreckage. It is, therefore, essential to have a reliable and efficient method of locating and identifying the lost object before deploying a search.   In this blog, I will discuss how search effectiveness can be improved by using statistical methods, namely Bayesian Search Theory. By the end of this blog, you will have a better understanding of how Bayesian Search Theory works and how it can be used to estimate effective search windows when coordinating a salvage operation. In particular I use a combination of SAS code and SAS Visual Analytics to demonstrate a simulation of 250 ‘search days’ and evaluate how the search coordination evolves over time, and how the probability of success evolves. The code for implementing the search algorithm is on GitHub here.   I would also like to thank Katie Howgate, a mathematician and PhD candidate with the University of Lancaster, for her excellent blog introducing the computational steps in applying Bayesian Search Theory. With her blessing I have used her example search grid probabilities to apply Bayesian Search Theory with SAS IML. If you are interested in understanding how the probabilities are computed in more detail, I’d strongly recommend reading her original blog post found here.   What is Bayesian Search Theory? Bayesian Search Theory, also known as Search Effectiveness Probability, is an application of mathematics and statistics that deals with the problem of finding a target in a large and uncertain space. It is based on the idea of using prior knowledge and evidence to update the probability of the target's location, and then choosing the optimal search strategy to maximise the chance of success.   The evidence in this case is not finding the lost object in a given search. We use this to update the probability of finding the object such that, after several unsuccessful attempts, we shift our search effort from one area to another by looking at the relative probabilities of finding the lost object.   At a high level, the Bayesian Search Theory is applied by: Initially formulating beliefs on where the object might be Assign initial probabilities to each location it might be Assign a probability of successfully finding it, given it is actually there. For underwater salvage this might be a consideration on depth, visibility or known hazards Combine these probabilities to give an overall probability of finding the object in a given search effort. This represents an overall likelihood of finding the object on a salvage operation. The highest probability location should be seen as the initial search location If the search is unsuccessful this is used to update the combined probability in step 4. Continue steps 4 & 5 to direct search effort until either the object is located or the search is abandoned This methodology uses Bayes’ Theorem to update the probabilities. As our search continues, this new information on unsuccessful searches is used to update the probability of finding the object and consider refocusing on a different location. This approach is more efficient that just using random search, and both less costly and faster than running a full exhaustive search by optimizing the distribution of search effort.   Bayes' Theorem is a powerful tool for reasoning under uncertainty and learning from data. It can help us make better decisions and understand complex phenomena. A benefit of applying Bayes' Theorem in search effectiveness is the ability to continuously update our beliefs. Bayesian updating can be applied to any situation where we have uncertain beliefs and new information and it is a powerful tool for rational reasoning and decision making under uncertainty.   These methods have been applied to many salvage operations including the search for the USS Scorpion, Air France Flight 447 and the attempt to find Malaysia Airlines Flight 370.   Applying Search Effectiveness to an example of Maritime Salvage Operations Let's see how this can help us with search effectiveness and salvage operations. Suppose we are looking for a sunken ship in a large area of water. Having been raised in  Devon I’ve chosen to illustrate this as an example of a lost object in the waters around Plymouth, a port city known for its maritime history and ship building.   We have some prior information about where the ship might be, such as its last known location. Using this information we establish a search grid around this point. On any given day we coordinate one search operation in one of the grids of the search area. Our search zone is shown in Figure 1.   Figure 1 - Our best guess on where the object might be with search zone   We also have some domain understanding on the environment we’re searching in. We know which areas of the search will be harder given depth, visibility, etc.   Finally, we use Bayes' Theorem to update our probabilities with the new evidence (following a failed search). The posterior probability will tell us how likely it is that the ship is in each location given the evidence. We can then focus our search on the locations with high posterior probabilities and ignore the locations with low posterior probabilities. Over time, as the search progresses, the probabilities will begin to shift and we start to focus on the lower posterior regions.   Using this approach we can improve our search effectiveness by using Bayesian updating to narrow down our search space and increase our chances of finding what we are looking for.   Implementing an example of Bayesian Search Theory with SAS IML Using PROC IML I created some initial probabilities to demonstrate the search zones and beliefs shown in Figure 1. Figure 2 shows our probabilities as matrices. Firstly,  we have our initial belief on where the item was lost. Secondly, we have our belief on how likely we are to find the object in a search zone provided it is actually there. Finally, we combine these two probability tables to estimate a total probability of finding the object in a given search zone.   Figure 2 - Starting probabilities at the beginning of the search   Once the initial probabilities are created a macro is used to simulate a failed search in the area with the highest probability of success. Looking at Figure 2 we can see we would start our search at [x=3,y=3] as it has the highest probability (0.1462). We then update the tables using Bayes’ Theorem following an unsuccessful search. The updated probability of finding the object at [x=3,y=3] falls and our next highest probability zone is then searched.   Visualizing Search Coordination in SAS Visual Analytics By simulating 250 search days we can visualize how the probability of success changes over time and where the allocation of search effort shifts too. Intuitively, we start our search closest to the last known point searching the inner four squares the most. Looking at the dashboard shown in Figure 3, we can see that Zone 6 has a relatively low combined probability of success. This reflects that it is an area which has a more challenging terrain. It therefore makes sense to not search there initially if there is an equal chance it is in Zones 7, 10 and 11. However, over time, proportionally more search effort will be allocated to Zone 6 as the other zones become exhausted.   Figure 3 - Using SAS Visual Analytics to our posterior probabilities   We can see how the probabilities shift in the animation shown in Figure 4.   Figure 4 - Animated plot of how the posterior probabilities change as the search progresses   We can see that the search spirals out from the centre over time as we become less confident that the lost object is near the last known location. Figure 5 shows how the combined probability of success decays and after August 12 th there is a sharp drop in probability of success. This is where we begin to shift the search outside of the inner squares near the last known location.   Figure 5 also shows us that probability of success continues to drop with every failed search day. Despite simulating 250 days from August 1 st , it is clear that by September 24 th the combined probability is getting very close to zero.   Figure 5 - Plot of expected probability of success by day of search operation   This shows us another advantage of using Bayesian Search Theory in that not only can we estimate where we should be concentrating our effort, but we can also use this to estimate what our optimal search window is in order to plan budget and resources.   Finally, Figure 6 shows an animated bar chart of search coordination across time. As we approach September we significantly increase the number of searches focused on Zone 6.   Figure 6 - Animated bar chart of the total number of searches by zone by day of search    Summary We’ve seen how Bayesian Search Theory is a powerful tool for improving search effectiveness and salvage operations. Bayesian Search Theory can help us make better decisions in search and salvage operations by allowing us to incorporate new information as it becomes available, and adjust our search strategies accordingly.   We’ve also seen that Bayesian Search Theory can also help us optimise our budget and allocation of resources for search and salvage operations. By using Bayesian updating, we can estimate how likely we are to find the target in a given area, and how much it would cost to search that area. We can then compare these estimates with the expected value or benefit of finding the target, and decide whether it is worth searching that area or not. This way, we can avoid wasting time and money on areas that are unlikely to yield results, and allocate more resources to areas that are more promising.   And finally, we’ve demonstrated how easy it is to implement these principles using SAS IML and visualize our results for scenario analysis in SAS Visual Analytics. ... View more