Paper Summary Request

You are a student in a game theory course. Please summarize the provided paper in approximately 300 words, covering the following sections: Goal, Introduction & Motivation, Method, Results, Future Work, and Conclusion(bring the student's opinion and thoughts with some questions to try to challenge more about this paper). The summary should be concise, clear, and easy to understand, using easy words, and avoiding redundant and repetitive language.

Game Plan: What AI can do for Football, and What
Football can do for AI
Karl Tuyls, Shayegan Omidshaei, Paul Muller, Zhe Wang, Jerome Connor,
Daniel Hennes
karltuyls@ somidshafiei@ pmuller@ zhewang@ jeromeconnor@ hennes@ ...google.com
DeepMind
Ian Graham, William Spearman, Tim Waskett, Dafydd Steele
ian.graham@ william.spearman@ tim.waskett@ dafydd.steele@ ...liverpoolfc.com
Liverpool Football Club
Pauline Luc, Adria Recasens, Alexandre Galashov, Gregory Thornton, Ro-
muald Elie, Pablo Sprechmann, Pol Moreno, Kris Cao, Marta Garnelo, Pra-
neet Dutta, Michal Valko, Nicolas Heess, Alex Bridgland, Julien Perolat, Bart
De Vylder, S. M. Ali Eslami, Mark Rowland, Andrew Jaegle, Remi Munos,
Trevor Back, Razia Ahamed, Simon Bouton, Nathalie Beauguerlange, Jack-
son Broshear, Thore Graepel, Demis Hassabis
paulineluc@ arecasens@ agalashov@ gregthornton@ relie@ psprechmann@ polc@
kriscao@ garnelo@ praneetdutta@ valkom@ heess@ abridgland@ perolat@ bartdv@
aeslami@ markrowland@ drewjaegle@ munos@ back@ rahamed@ bouton@ nbeau-
guerlange@ broshear@ thore@ demishassabis@ ...google.com
DeepMind
Abstract
The rapid progress in articial intelligence (AI) and machine learning has opened unprecedented
analytics possibilities in various team and individual sports, including baseball, basketball,
and tennis. More recently, AI techniques have been applied to football, due to a
huge increase in data collection by professional teams, increased computational power, and
advances in machine learning, with the goal of better addressing new scientic challenges
involved in the analysis of both individual players' and coordinated teams' behaviors. The
research challenges associated with predictive and prescriptive football analytics require
new developments and progress at the intersection of statistical learning, game theory, and
computer vision. In this paper, we provide an overarching perspective highlighting how
the combination of these elds, in particular, forms a unique microcosm for AI research,
while oering mutual benets for professional teams, spectators, and broadcasters in the
years to come. We illustrate that this duality makes football analytics a game changer of
tremendous value, in terms of not only changing the game of football itself, but also in
terms of what this domain can mean for the eld of AI. We review the state-of-the-art
and exemplify the types of analysis enabled by combining the aforementioned elds, including
illustrative examples of counterfactual analysis using predictive models, and the
combination of game-theoretic analysis of penalty kicks with statistical learning of player
attributes. We conclude by highlighting envisioned downstream impacts, including possibilities
for extensions to other sports (real and virtual).

c 2021 AI Access Foundation. All rights reserved.
Tuyls, Omidshafiei, Muller et al.
1. Introduction
Recent years have seen tremendous growing interest in sports analytics, not only from an
economic and commercial perspective, but also from a purely scientic one, viz. the growing
number of publications (Baumer & Zimbalist, 2014; Beal, Norman, & Ramchurn, 2019;
Shih, 2017; Szymanski, 2020) and scientic events organized on the topic (e.g., CVSports
International Workshop on Computer Vision in Sports at CVPR, 2020; Machine Learning
and Data Mining for Sports Analytics, 2020; MIT Sloan Sports Analytics Conference, 2020).
As evident in many dierent downstream domains that have beneted from applications
of articial intelligence (AI) and machine learning (ML), this is due to important
technological advances in data collection and processing capabilities, progress in statistical
and in particular deep learning, increased compute resources, and ever-growing economic
activities associated with sports and culture (e.g., emergent consultancy ventures revolving
around sports data collection and statistics, Beal et al., 2019; ChyronHego, 2020; InStat,
2020; Kuper & Szymanski, 2018; Opta, 2020; StatsBomb, 2020).
Predictive analytics has been investigated and applied in the context of several sports
in the past decades, including basketball (Skinner, 2010), tennis (Anderson, Rosen, Rust, &
Wong, 2020; Gauriot, Page, & Wooders, 2016), and baseball (Albert, 2010; Albert, Bennett,
& Mead, 2002; Baumer & Zimbalist, 2014; Costa, Huber, & Saccoman, 2009; Michael,
2004; Puerzer, 2002; Song, Severini, & Allada, 2017), with data for the latter having been
systematically collected since the 19th century. Although statistical analysis of data has led
to impressive outcomes in various sports (e.g., Moneyball in baseball, Baumer & Zimbalist,
2014; Michael, 2004), football started participating rather late in this data collection and
number-crunching game, with the data science transformation that informs stakeholders
(e.g., decisions related to player transfers, scouting, pre- and post-match analysis, etc.)
still in its infancy (Kuper & Szymanski, 2018). Several factors in
uenced this late arrival.
Football takes place under far less controllable settings than other sports due to its outdoor
and highly dynamic nature, a larger pitch, a large number of players involved, a low number
of player changes and longer non-interrupted game sequences than sports such as basketball.
As a result, football analytics companies have only relatively recently started collecting socalled
big data (e.g., high-resolution videos, annotated event-streams, player tracking and
pose information). Concurrently, only recently have major breakthroughs been made in
deep learning, yielding techniques that can handle such new high-dimensional data sets
(Arel, Rose, & Karnowski, 2010; Bengio, 2009; Goodfellow, Bengio, & Courville, 2016;
LeCun, Bengio, & Hinton, 2015; Schmidhuber, 2015). Finally, for a long time, credibility in
decision-making primarily depended on human specialists such as managers, retired players,
and scouts, all of them with track records and experience in professional football, in part
due to cultural reasons (Decroos & Davis, 2019; Kuper & Szymanski, 2018). As a result of
these various factors, the potential in
uence and gains of predictive analytics on the football
game have also been less obvious, with sports analytics as a game-changing phenomena not
realized until recent years. In more philosophical terms, Kuper and Szymanski (2018)
highlight a cultural hesitation regarding the integration of data science into football and an
overdependence on gut instincts, noting that \until very recently, soccer had escaped the
Enlightenment".
42
Game Plan: What AI can do for Football, and What Football can do for AI
Computer Vision (CV)
Event detection
Pose detection
Activity recognition
Generative video models
Automatic highlight generation
Frontier 2
(SL&CV)
Game Theory (GT)
Normal-form game analysis of
penalty kicks and set pieces
Player behavior compared to
solution concepts (e.g.,
mixed-strategy Nash)
Empirical game theoretic
analysis
Frontier 1
(GT&SL)
Microcosm
Automated
Video
Assistant
Coach (AVAC)
Frontier 3
(GT&CV)
Statistical Learning (SL)
Player Vectors
Predictive modeling and Ghosting
Valuing Actions by Estimating Probabilities
Player Chemistry
Expected Possession Value
Pitch Control
Off-ball Scoring Opportunities
Figure 1: AI research frontiers associated with football analytics. We highlight three foundational
areas related to AI research{statistical learning, computer vision, and game theory{
which have independently been demonstrated to be eective in analyzing the game of football
(with example problems and works from literature shown above per eld). We argue
that the domain spanned by these research elds is most promising for establishing seminal
progress in football analytics in the coming years, along 3 frontiers: Frontier 1 (GT&SL),
Frontier 2 (SL&CV), and Frontier 3 (GT&CV). We claim that the above frontiers culminate
into a unique microcosm mutually beneting AI and football analytics.
Despite football's late adoption of sports analytics, there are a number of early-bird approaches
from dierent areas of AI such as statistical learning (SL), computer vision (CV),
and game theory (GT) that are making initial contributions to support decision-making of
managers, coaches and players. For example, basic statistical learning tools such as principal
component analysis (PCA) already enable automated means of identifying player types
43
Tuyls, Omidshafiei, Muller et al.
(Decroos & Davis, 2019), training of models predicting trajectories of individual teams or
imitating league-average behaviors (H. Le, Carr, Yue, & Lucey, 2017), and valuing individual
player decisions (such as passes or tackles) in a series of actions leading up to a goal
(Decroos, Bransen, Haaren, & Davis, 2019). The study of interactive decision-making as
formalized by game theory plays a critical role in AI for systems involving more than one
actor (human or articial). Game-theoretic tools shed light on players' strategic interactions
during scenarios such as penalty kicks, analysis of their decisions in comparison to
mathematically-principled baselines, and prediction of their goal-scoring probabilities when
playing according to a mixed-strategy Nash equilibrium (Chiappori, Levitt, & Groseclose,
2002; Palacios-Huerta, 2003, 2016). Enriched with empirical game theory (Omidshaei et
al., 2019; Tuyls et al., 2020; Wellman, 2006) the eects of various high-level strategies pitted
against each other can also be analyzed. Finally, recent developments in computer
vision have been employed for player tracking (Bialkowski et al., 2015; Gade & Moeslund,
2018; Liu, Carr, Collins, & Liu, 2013; Lu, Ting, Little, & Murphy, 2013), pose estimation
(Bridgeman, Volino, Guillemaut, & Hilton, 2019; Fastovets, Guillemaut, & Hilton,
2013; Sypetkowski, Kurzejamski, & Sarwas, 2019; Sypetkowski, Sarwas, & Trzcinski, 2019;
F. Zhang, Zhu, & Ye, 2019), and automated injury prediction (Kampakis, 2016) based on,
e.g., gait and fatigue analysis (Bartlett, 2006; Claudino et al., 2019; Kakavas, Malliaropoulos,
Pruna, & Maulli, 2019; Op De Beeck, Meert, Schutte, Vanwanseele, & Davis, 2018;
Ramos et al., 2020).
While these separate areas within AI research have independently been demonstrated to
be eective for football analytics, we believe that the most pertinent research problems lie in
the underexplored intersection of statistical learning, computer vision, and game theory (see
Figure 1). Specically, we pinpoint several frontiers at the intersections of these elds, and
identify the ultimate frontier to be the microcosm requiring integrated approaches across all
three elds. A large portion of the state-of-the-art research in football analytics, by contrast,
typically falls under the umbrella of one of these areas, with some initial activities taking
places at Frontier 2 (SL&CV) (Choi, Kwon, & Lee, 2019; Lu et al., 2013; Mehrasa, Zhong,
Tung, Bornn, & Mori, 2018; Mora & Knottenbelt, 2017; Quiroga, Carrillo, Maldonado,
Ruiz, & Zapata, 2020), and no notable research activities identied at Frontier 1 (GT&SL)
and Frontier 3 (GT&CV) for football or sports in general. At Frontier 1 (GT&SL), gametheoretic
analysis is blended with learned predictive models, combining interactive decisionmaking
with predictive modeling to provide more granular analysis tools. We present a
detailed case study illustrating this frontier, revisiting the work of Palacios-Huerta (2003) on
penalty kicks under this new perspective and illustrate how mixing with SL provides deeper
insights into penalty-kick taking. Frontier 2 (SL&CV) focuses on research that integrates
statistical learning with computer vision, directly learning from video as the primary input
and building predictive models (e.g., forecasting player and team behaviors directly). At
Frontier 3 (GT&CV), we classify football research integrating computer vision and game
theory, a largely uncharted territory focusing on generative models based on visual inputs,
which takes strategic interactions into account.
We claim that the above frontiers culminate into a unique microcosm mutually beneting
both AI and football analytics, to the point it becomes possible to develop, for example,
an Automated Video Assistant Coach (AVAC). The AVAC system is an example of what
we believe to be the future of human-centric AI research for football, with the aim of
44
Game Plan: What AI can do for Football, and What Football can do for AI
integrating all aspects of the frontiers into a cohesive system enabling both understanding
and improvement of human football play. Such an AVAC system is envisioned to improve
the overall experience of the game for players, coaches, and spectators alike. Finally, we
also point out that jointly these frontiers provide the necessary theoretical background for
analytics research to build upon in the microcosm, laying out a foundation to build accurate
models and predictions, as advocated for in Szymanski (2020).
In the following, we rst provide an overview of the literature associated with football
analytics, subsequently highlighting gaps and sketching a long-term vision for research in the
football microcosm. We lay out the long-term perspective of how AI can benet the domain
of football analytics, and vice versa, by combining the three identied axes in the literature
in manners that have not yet been fully explored. Next, we present illustrative examples
that examine penalty-kick taking from a game-theoretic perspective, building on the work
of Palacios-Huerta (2003, 2016), and bringing several new insights based on data from the
main European leagues. We subsequently demonstrate how this game-theoretic work can be
enriched via integration with statistical learning at Frontier 1 (GT&SL), providing several
new insights about penalty-kick taking followed by a high-level example of counterfactual
trajectory predictions (i.e., a what-if analysis), thus further justifying the football microcosm
as a useful AI research domain.
2. Literature Overview: AI for Football Analytics
The following sections outline a long-term research vision for AI applied to football analytics.
We rst consider the state-of-the-art with respect to each of the respective areas (GT, SL,
and CV) applied to the football domain, after which we look into the opportunities that each
of the frontiers brings forth to unlocking larger scientic challenges in football analytics.
The following sections summarize works that lie in these peripheral elds of Figure 1 and
highlight opportunities for further work and added value in each.
2.1 Statistical Learning
Football is arguably one of the most challenging to analyze of all the major team sports. It
involves a large number of players with varied roles, few salient events, and minimal scoring.
Statistical football analysis attempts to provide quantitative answers to questions that pertain
to dierent aspects of the game. Notably, these include the problem of characterizing
players' and teams' styles, evaluation of the impact that such teams have on the pitch,
and the temporal and counterfactual predictions of players' actions. When one compares
styles of players (and teams, by extension), one usually refers to high level and abstract
notions that summarize their unique characteristics. The goal of the statistical learning line
of research is to learn features capturing such information, normally to be used by other
down-stream tasks. For instance, one means of summarizing information about a football
player is through aggregation of their play statistics (e.g., oensive, defensive, or dribbling
abilities, shots on target, etc.), and that of their teams (Fernandez-Navarro, Fradua, Zubillaga,
Ford, & McRobert, 2016; Stats Perform, 2020). While such statistics are typically
either handcrafted or rely on simple measures of play outcomes, recent works have analyzed
them from a statistical learning perspective, using notions such as Player Vectors (Decroos
& Davis, 2019) (detailed in Appendix C) and analogous techniques in sports such
45
Tuyls, Omidshafiei, Muller et al.
as basketball (Franks, Miller, Bornn, Goldsberry, et al., 2015). Given the growing success
of unsupervised learning methods, there is potential for more advanced representations of
player traits to be learned directly from the data.
In football, it is particularly dicult to assess which players, or groups of players, deserve
credit for favorable outcomes. For high-scoring team sports (e.g., basketball) one can
provide a reasonable answer to this question by restricting attention to actions that have
immediate impact on scoring. By contrast, few goals are scored in football (e.g., 2.72 goals
were scored on average per game of the 2019/2020 Premier League season, Clarke, 2020).
Consequently, models considering only actions with immediate impact on goals (e.g. shots
or saves) capture a crucial yet narrow view of the game. Moreover, game states in football
are signicantly more complex than that estimated by current models. Features describing
them are mostly hand-crafted and only consider on-ball actions. A given pass might be extremely
valuable or a poor choice depending on the disposition of the players. While these
methods are able to value on-ball actions, they rely on sparse signals provided by goals.
Concurrently, o-ball actions signicantly impact each game, as exemplied by player actions
covering certain pitch areas to prevent attacks, running to open areas to create space
for teammates, and so on. Due to the temporally extended nature of football, the task of
inferring the value of actions is an instance of the temporal credit assignment problem in
reinforcement learning (RL) (Minsky, 1961). The combination of RL techniques with deep
learning has great potential to tackle the idiosyncrasies of football and to close the gap
between statistical and human analysis. It is exciting to see recent progress in this direction
in sports analytics, including ice hockey (Liu & Schulte, 2018) and football (Liu, Luo,
Schulte, & Kharrat, 2020; X. Sun, Davis, Schulte, & Liu, 2020) (with additional related
works detailed in Appendix A).
Overall, while the above pieces of work showcase the promise of modern statistical
methods for temporal predictions in football, this remains an open and challenging problem
that will likely require development of novel methods and means of leveraging the diversity
of newly-available football data (as later made more precise in terms of what football can
do for AI).
2.2 Game Theory
Game theory plays an important role in the study of sports, enabling theoretical grounding
of players' behavioral strategies. Numerous works have applied game-theoretic analysis
to sports over recent decades (Lennartsson, Lidstrom, & Lindberg, 2015; Sindik & Vidak,
2008), including football (Azar & Bar-Eli, 2011; Buzzacchi & Pedrini, 2014; Chiappori et
al., 2002; Coloma, 2012; Levitt, Chiappori, & Groseclose, 2002; Moschini, 2004; Palacios-
Huerta, 2003), tennis (Anderson et al., 2020; Gauriot et al., 2016), volleyball (K. Lin,
2014), basketball (Skinner, 2010), and American football (Emara, Owens, Smith, & Wilmer,
2014). High-level game theory can be applied to team selection and choice of formation.
Set pieces, such as corner kicks and penalties, are particularly amenable to game-theoretic
analysis, wherein identied high-level strategies can be pitted against one another and
ranked in terms of empirical performance (see Section 5 for details). Due to the real-world
nature of football, the types of game-theoretic analysis conducted in this domain have
typically been driven by the availability of data sources (in contrast to, e.g., simulation-based
46
Game Plan: What AI can do for Football, and What Football can do for AI
domains, where data of varying types and granularity can be synthetically generated). In
particular, the volume of available high-level statistics (e.g., match outcomes spanning across
dierent leagues, seasons, and competition levels) makes football a particularly attractive
topic from a behavioral and empirical game-theoretic perspective (Camerer, 2011; Camerer,
Loewenstein, & Prelec, 2005; Camerer, Loewenstein, & Rabin, 2011; Wellman, 2006). In
contrast to traditional game theory, which often assumes rational decision-making players
seeking to maximize utilities, behavioral game theory (and the closely-associated eld of
behavioral economics) studies the interactions of decision-making individuals that either
have imperfect knowledge of the underlying game, have bounded rationality, or are simply
not purely self-interested (Camerer et al., 2005; Nax, 2015). The complexity of real-world
football scenarios involving interacting humans, who may be in
uenced by a multitude of
factors other than purely maximizing their in-game utilities, implies that the use of such
techniques may prove invaluable for more accurate understanding of the game.
From a theoretical perspective, the majority of the existing works exploit the fact that
various football scenarios can be modeled as two-player zero-sum games. For example,
in football, the penalty kick situation may be straightforwardly modeled as a two-player
asymmetric game, where the kicker's strategies may be neatly categorized as left, center,
or right shots. The controlled nature of these scenarios compounds their appeal from a
quantitative analysis perspective; in the penalty example, the penalty taker, goalkeeper,
and ball's initial positions are generally static across all dataset trials (albeit, with minor
variations, e.g., scenarios where goalkeepers stand slightly o-center to entice the kicker
to shoot in one direction). In fact, the majority of the literature that analyzes football
under a game-theoretic lens focuses on penalty kicks (Azar & Bar-Eli, 2011; Buzzacchi &
Pedrini, 2014; Chiappori et al., 2002; Coloma, 2012; Levitt et al., 2002; Palacios-Huerta,
2003), which we contend is due to this amenability for analysis via classical game-theoretic
solution concepts (such as Nash equilibria).
However until the paradigm shifts away from the classical analysis of set pieces and
toward live-play settings such as those considered by Moschini (2004), the potential bene
ts of game-theoretic analysis for football are likely to remain largely untapped. Moving
beyond this classical paradigm involves resolution of signicant challenges: the number of
active players in football is quite large (22 including goalkeepers) and the exponential size
of the strategy space (with respect to the number of players) makes it more challenging to
analyze than games such as tennis or basketball; the mapping of low-level player actions to
strategies is no longer straightforward in durative live plays, due to the variability of player
trajectories; the duration of plays is also generally longer than sports played in more controlled
environments (e.g., tennis), implying that such scenarios may benet from analysis
in the so-called extensive-form, which explicitly consider each player's knowledge, opportunities,
and actions, unlike in the simpler (but more feasible to analyze) simultaneous-move,
normal-form approaches typically used for set piece analysis (later detailed in Section 5.1).
There exists potential for interesting team-to-team interactions to occur when use of
game-theoretic techniques for prescription of player behaviors becomes prominent. Once
such techniques are deployed en-mass to aid teams, one can safely assume that opposing
teams are also using such systems. This yields an interesting scenario where an AI system
used by one team may be used to, in the language of game theory, best-respond to the other
team's system. Even further, one can also consider this best-response behavior emerging
47
Tuyls, Omidshafiei, Muller et al.
from the human players themselves (e.g., to try to act adversarially to cause the opposing
team's analytics platforms to mispredict or otherwise provide non-useful advice to their
players). In recent years, a signicant amount of eort has been spent by researchers in
attempting to thwart or make less sensitive the predictions of AI and ML algorithms to
such adversarial attacks (e.g., Madry, Makelov, Schmidt, Tsipras, & Vladu, 2017), with
techniques for analyzing these challenges often inspired by game theory. It is similarly perceived
that incorporating such robustness mechanisms is likely to be benecial for deployed
football AI techniques (e.g., by rening training, or by explicitly considering sources of uncertainty
to quantify and convey risk levels when making recommendations to coaches and
decision-makers). One important distinction to bear in mind, however, is that the nature of
football implies that, ultimately, it will be the humans that will make decisions and directly
interact with one another on the pitch. Despite the advice received from a combination of
coaches and AI systems, these direct on-pitch interactions will remain human-to-human,
thus potentially making such adversarial countermeasures more dicult. This stands in
contrast, for example, to applications of AI to domains such as self-driving cars, where
ultimately the AI is in the driver's seat and directly interacts with humans on the road.
Overall, the availability of new data types increases the viability of conducting more
advanced game-theoretic analysis by bootstrapping to advances in statistical learning and
computer vision, as later detailed. Surmounting these challenges will benet football from
a game-theoretic perspective, enabling both a descriptive analysis (i.e., understanding the
interactions undertaken by players and teams in the presence of others), and a prescriptive
one (i.e., suggesting the actions such individuals should have executed).
2.3 Computer Vision
Computer vision has seen major breakthroughs in the past decade, thanks to the application
of deep learning approaches. Progress in tasks such as image classication (Deng
et al., 2009), video action recognition (Carreira & Zisserman, 2017), and pose estimation
(Alp Guler, Neverova, & Kokkinos, 2018) have unlocked the possibility of automatically
extracting complex information from videos. Similarly, in football analytics, computer vision
methods can enable enrichment of statistical and game-theoretic approaches, which
typically rely on hand-labeled, low-dimensional data. Video is a particularly appealing
signal for football analytics: it is a rich data source (likely to contain large amounts of
information of interest for decision-making) and cheap to acquire (using only widespread
camera sensors). While raw video frames are insuciently structured to be processed by
traditional rule-based processing techniques, computer vision methods enable the extraction
of high level, potentially spatially-detailed representations, ready for use by downstream applications.
Examples of such extraction processes include human pose estimation (Alp Guler
et al., 2018), object detection (Ren, He, Girshick, & Sun, 2015), segmentation (Long,
Shelhamer, & Darrell, 2015), tracking (Dong & Shen, 2018), depth estimation (Godard,
Mac Aodha, & Brostow, 2017), and event or action detection (Carreira & Zisserman, 2017).
In addition to explicit, human-interpretable representations, learned counterparts can
be used by downstream deep learning models. In football, several companies already commercialize
tracking information, relating to the players and the ball, automatically extracted
from videos recorded by dedicated static cameras, placed to have a view covering the entire
48
Game Plan: What AI can do for Football, and What Football can do for AI
terrain (Opta, 2020; StatsBomb, 2020). Moreover, immersive sports analytics has been gaining
popularity with the increased access to specialized hardware and portability (T. Lin,
Yang, Beyer, & Pster, 2020). Computer vision techniques enable reconstruction of real
game scenarios, which provide more feedback to players and coaches than 2D screens, and
have been extended to other sports media as well (StriVR Immersive Sports Analytics,
2020). Vision-based player tracking, pose estimation, and event detection can improve learning
of player skill vectors and predictive modeling, subsequently improving player ranking
and game-theoretic analysis of strategies.
While video has been traditionally used as the primary high-dimensional signal for
the above applications, other modalities such as audio and text streams can provide extremely
rich and complementary information in these settings. Audio commentary and
news threads readily provide a human interpretation of the events occurring in a scene,
which is highly complementary to the spatially ne-grained information available in videos.
Learning-based approaches for generation of text commentary have been previously investigated
(D. L. Chen, Kim, & Mooney, 2010), with temporally-aligned sound and text streams
seeing recent application in learning rich representations of video signals (Alayrac et al.,
2020; Arandjelovic & Zisserman, 2017; Miech et al., 2020), having been shown to be useful
in a variety of computer vision tasks. Although such information signicantly overlaps with
the structured annotations already used by downstream applications (event annotations,
etc.), its unstructured nature enables the presence of a greater information content (e.g., a
commentator's tone can provide some cues as to an action's value).
Unique challenges arise when seeking to improve the performance and to enlarge the
applications of computer vision methods for broadcast video of football games. In general,
the camera's eld of view is centered on the most interesting action in the eld, leaving many
players out of the frame. This poses an interesting geometric problem, as broadcast footage
shots typically do not overlap, and many state-of-the-art systems are unable to geometrically
relate these multiple shots. Other challenges arise due to the multiple occlusions of players
by one another, further complicating the detection, tracking, and identication tasks. These
problems can be addressed by geometrical approaches in settings where one can intervene
to correct camera positions or, ideally, access cameras' extrinsic and intrinsic parameters
(that is, the camera's position, orientation, as well as geometrical and optical properties
like its focal length). However, approaches that do not assume access to such information
have the potential to leverage more data. In contrast with other applications, football
broadcast videos present an interesting blend between real, large-scale, complex data and
a constrained setting (due to the rules of the game, Giancola, Amine, Dghaily, & Ghanem,
2018), hence providing an attractive setting for developing such approaches.
3. Game Plan: Long-Term Research Vision
In this section we outline a long-term research vision for football analytics in the microcosm
frontiers at the intersection of statistical learning, game theory, and computer vision (see
Figure 1).
49
Tuyls, Omidshafiei, Muller et al.
3.1 Frontier 1: Interactive Decision-Making (GT&SL)
Learning to make suitable decisions in the presence of other agents is where game theory
and statistical learning can converge. This interdisciplinary area of research has received
signicant attention in the multi-agent RL community over the past decades, with thorough
survey articles (Bloembergen, Tuyls, Hennes, & Kaisers, 2015; Busoniu, Babuska, & Schutter,
2008; Hernandez-Leal, Kartal, & Taylor, 2020; Panait & Luke, 2005; Shoham, Powers,
& Grenager, 2007; Tuyls & Weiss, 2012) available. When considering the football domain
in particular, it is evident that the potentials of game theory have yet to be fully exploited,
let alone its combination with machine learning techniques such as RL.
There are several promising routes in this area that not only are challenging from a
football analytics perspective, but also from an AI research one. Two of these routes are
studied further in this article in Section 5; one concerns the study of set pieces using the
combination of statistical learning with game theory, and a second focuses on predictive
modelling for counterfactual analysis. In the former, which has been mostly studied in a
game-theoretic setting, we show how augmenting the analysis with player-specic statistics
can provide deeper insights in how various types of players behave or take decisions about
their actions in a penalty kick scenario. In the latter case, we illustrate how machine
learning techniques can facilitate counterfactual analysis in football matches. The possibility
to predict, for example, trajectories of players can enable investigation of counterfactual
scenarios, (e.g., wherein one would like to know how a specic player or team would respond
in a specic match scenario). Doing this enables one to not only learn to generate behaviors,
but also leverage game-theoretic techniques for counterfactual analysis. We defer a more
detailed discussion of these research lines to Section 5.
Building on the counterfactual prediction of players' behaviors, one can also consider the
possibility of using this as a coaching tool. Specically, one can use counterfactual insights
to advise tactics to individual players, and even go further by optimizing the overall team
strategy depending on the specic opponent in an upcoming match. This would go beyond
the state-of-the-art, which focuses on simply predicting player behaviors; here, one would
seek to actively optimize suggested tactics based on the particular behaviors and play style
of the opposing team, and upcoming match-ups. Such tactical considerations can also be
conducted in an iterative manner (e.g., predictions can be made for the opposing team as
conditioned on the best-response behavior of the main team), and eective counter strategies
can be learned, for instance, using multi-agent RL. Such an approach opens the door to a
slew of interesting research challenges. For instance, use of multi-agent RL entails denition
of a reward function for the players; while rewarding goals is an obvious candidate, denser
reward signals (e.g., associated with successful passes, intercepts, etc.) may be useful for
accelerating learning of such policies. Such reward functions are also likely to depend on the
role of a player in the team and their respective skill set (i.e., may even be heterogeneous
across players in a given team), and could also be learned using techniques such as inverse
RL (Ng & Russell, 2000). Moreover, one may seek to rst dene an eective `action space'
for players (i.e., more granular or structured actions than `move left' or `move right'), before
solving the RL problem. Finally, the combination of the previous learnt models with pitch
control (Spearman, 2016), a technique to determine which player/team has control over a
50
Game Plan: What AI can do for Football, and What Football can do for AI
specic area of the pitch, will provide additional information on open space, passing and
scoring opportunities, yielding a powerful tool to enable in-match tailored coaching tools.
3.2 Frontier 2: Predictive Modeling from Videos (SL&CV)
Several challenges naturally lie in the frontier between statistical learning and computer
vision. Statistical learning depends on large quantities of labelled data. Many of the quantities
suitable for models of football are the product of hand-labelling data; on the other
hand, vision-based models could automatically identify events which could be fed into such
models. In addition to the quantity of events that vision-based systems could provide, the
quality could also be improved (e.g., with events being accurately registered to the corresponding
frame, with minimal temporal error compared to human-labeled annotations).
Furthermore, video is a much richer signal compared to what is traditionally used in
predictive tasks, such as forecasting future movement of players or predicting the value of
individual actions to the game outcome. The combination of advanced deep learning models
and a rich video signal enables learning over subtle clues otherwise not captured in eventstream
or tracking data. Capturing more of the partially-observable state of a game will
ultimately enable more accurate predictions. Richer information may additionally help to
shed light on the intention of players and thus better address the credit assignment problem
in action-value estimation.
On the other hand, models that better capture the game dynamics may be necessary to
resolve some of the limitations of vision-based tracking approaches, which arise as players
are occluded or move o camera. The resulting ambiguities can likely be resolved using
predictive model of player dynamics, which may be trained independently and used as a
source of additional inputs to the tracking system or trained as one component of a larger
pipeline. Explicit or implicit access to the game dynamics will very likely also improve
vision-based action labeling. Finally, presenting prediction outcomes by means of synthetic
video generation remains an ambitious challenge that combines the task of trajectory prediction
with video generation. Presenting predictions of counterfactual futures as video will
enable intuitive understanding both by coaching personnel and players (e.g., in the same
vein as recent work on video generation for tennis matches) (H. Zhang, Sciutto, Agrawala,
& Fatahalian, 2020).
3.3 Frontier 3: Generative Game-Theoretic Video Analysis Models (GT&CV)
Video modeling and game theory can mutually benet one another. In the simplest application,
computer vision can provide new features to drive game-theoretic models. In more
advanced applications, game theory can, in turn, guide video generation, as illustrated in
Figure 3. Throughout a football match, individual players carry out a series of encounters
with one another, which can protably be viewed through game theory. Each player may
have some idea of the likely success of strategies based on past performance and current
in-game observations. Penalty kicks are an almost idealized example, with a kicker and goalkeeper
observing each other for signs of intent, while simultaneously weighing preconceived
strategies. As described previously, computer vision models can be used to automatically
extract high-level and potentially spatially-detailed representations that can be complementary
to the low-dimensional, hand-collected inputs game-theoretic models typically rely on.
51

still got 30pages left

still have more pages

Figure 2: Example of a pose estimation model applied to a penalty kick. The results
showcased here follow a multi-phase approach (Papandreou et al., 2017) The rst stage
applies a Faster-RCNN candidate detector (Ren et al., 2015) which extracts bounding boxes
around the players. This is followed by a pose estimation model applied to these bounding
box proposals, which identify the player keypoints, and provide the pose estimate.
􀀨􀂵􀃒􀂏􀃈􀂴􀂏􀂋􀂡􀁹􀃒􀂏􀄯􀁏􀂏􀃅􀃈􀂏􀃌􀂏􀂵􀃒􀁹􀃒􀂡􀂺􀂵􀄯􀄘􀂅􀄙
􀁌􀂺􀃌􀂏􀄯􀄖􀄯􀀢􀂮􀂺􀃢􀄯􀄖􀄯􀁙􀃈􀁹􀂆􀂬􀂡􀂵􀂜
􀀨􀂵􀃒􀂏􀃈􀂴􀂏􀂋􀂡􀁹􀃒􀂏􀄯􀁏􀂏􀃅􀃈􀂏􀃌􀂏􀂵􀃒􀁹􀃒􀂡􀂺􀂵􀄯􀄘􀂋􀄙
􀁌􀂺􀃌􀂏􀄯􀄖􀄯􀀢􀂮􀂺􀃢􀄯􀄖􀄯􀁙􀃈􀁹􀂆􀂬􀂡􀂵􀂜
􀀁􀂵􀁹􀂮􀃨􀃌􀂡􀃌 􀀣􀂏􀂵􀂏􀃈􀁹􀃒􀂡􀂺􀂵
􀁏􀂏􀁹􀂮􀄯􀂛􀂺􀂺􀃒􀁹􀂜􀂏􀄯􀄘􀁹􀄙
􀀎􀂺􀂴􀃅􀃖􀃒􀂏􀃈􀄯􀁨􀂡􀃌􀂡􀂺􀂵
􀀣􀂏􀂵􀂏􀃈􀁹􀃒􀂏􀂋􀄯􀂛􀂺􀂺􀃒􀁹􀂜􀂏􀄯􀄘􀂏􀄙
􀀣􀁹􀂴􀂏􀄯􀁙􀂠􀂏􀂺􀃈􀃨
􀀗􀂴􀃅􀂡􀃈􀂡􀂆􀁹􀂮􀄯􀃅􀁹􀃨􀂺􀂛􀂛􀄯􀃒􀁹􀂅􀂮􀂏􀃌􀄯􀄘􀂆􀄙
Figure 3: Analysis and generation of video data. From footage video (a), computer vision
techniques can extract various representations (b), useful for empirical game-theoretic analysis.
In turn, empirical payo tables (c) can be used in a hierarchical generative process
of videos, possibly involving a step in intermediate representation spaces (d). Generated
videos (e) could be used for prescriptive analysis, or in the process of capturing the footage
videos themselves, closing the cycle between analysis and generation.
We illustrate this representation extraction pipeline in the left portion of Figure 3. In the
example of penalty kicks, vision models could identify visual signatures of intent that are
dicult to even perceive for humans, let alone annotate; such information includes, for
example, pose estimates extracted from broadcast footage (as visualized in Figure 2, with
technical details provided in Appendix B), which can enable inference of the intentions of
players, providing valuable insights to improve their respective strategies.
52
Game Plan: What AI can do for Football, and What Football can do for AI
In the reverse direction (right portion of Figure 3), game-theoretic outputs can be used
to improve the plausibility and usefulness of generated videos. Specically, generative video
models need to precisely capture the data distribution to be useful. The game-theoretic
context found within football oers an opportunity to constraint such models. The dynamics
of play are complex, with games taking place at the level of opposing individuals
to entire teams, though with specic constraints that can inform a generative process. For
example, a hierarchical generative process could condition on high-level latent variables
drawn from the distribution described by empirical payo tables, to represent player decisions
conditioning a particular sample. One could consider an additional hierarchical level
of generation targeting intermediate features, possibly building on approaches for predicting
future pose (Chan, Ginosar, Zhou, & Efros, 2019; Villegas et al., 2017), trajectory (Bhattacharyya,
Fritz, & Schiele, 2018, 2019; Kitani, Ziebart, Bagnell, & Hebert, 2012; Vu, Choi,
Schulter, & Chandraker, 2018), action (Abu Farha, Richard, & Gall, 2018; Vondrick, Pirsiavash,
& Torralba, 2016) and shape information (Jin et al., 2017; Luc, Couprie, Lecun, &
Verbeek, 2018; Luc, Neverova, Couprie, Verbeek, & LeCun, 2017; J. Sun et al., 2019; Xu,
Ni, Li, Cheng, & Yang, 2018).
Game-theoretic analysis could be used to assess the plausibility of game simulations.
Specically, these intermediate outputs could be used directly for prescriptive analysis (e.g.,
informing players of physical tactics to attempt in a play of interest, based on generated
poses) or serve as further conditioning for generation in the raw RGB space. An alternative
direction would be the formulation of new game-theory inspired metrics, for example
imposing that the payo tables extracted from generated samples match those obtained
from real data. This would be closely related to the Frechet Inception Distance (Heusel,
Ramsauer, Unterthiner, Nessler, & Hochreiter, 2017) for generative modeling of images, and
the derived Frechet Video Distance (Unterthiner et al., 2018) for video, which have successfully
driven progress in both elds (Brock, Donahue, & Simonyan, 2019; Clark, Donahue, &
Simonyan, 2019; Karras, Aila, Laine, & Lehtinen, 2017; Luc et al., 2020; Miyato, Kataoka,
Koyama, & Yoshida, 2018; Weissenborn, Tackstrom, & Uszkoreit, 2019; H. Zhang, Goodfellow,
Metaxas, & Odena, 2019). In turn, such metrics could serve as a basis to design
novel losses inspired by game theory. Besides their potential use for prescriptive analysis,
generative models of videos could lead to improvements in broadcast data itself, due to
automatic anticipation of the most plausible immediate future outcomes. Overall, gametheoretic
analysis can inform video generation, in turn enabling in
uence of the process by
which future data will be acquired, hence closing the cycle described in Figure 3.
3.4 Football as an AI Testbed: Microcosm
The development of performative AI algorithms relies on various recurring objectives: learning
and acting based on real-world data streams, interpreting the actions of other agents and
acting strategically in response, being able to understand and generate natural language for
ecient communication with humans, and so on. As discussed in earlier sections, football
analytics involves core problems associated with many of the above AI challenges, though
in a more well-controlled scope.
The value of the football domain for AI can be observed, at a low level, in the range of
useful data available for corresponding analytics (see Figure 4). Real-world data streams
53
Tuyls, Omidshafiei, Muller et al.
Highly-structured real-world data
Videos
Complete games, set pieces,
highlights, etc.
Audio
Commentators, crowd and
player reactions, referees, etc.
Text
Commentator transcripts,
online chats, news stories, etc.
Historical data
Player and, team statistics
(seasonal records, etc.)
2020
Annotated game data
Tracking data, event-stream
data (kicks, goals, tackles, etc.)
Crowd-sourced data
Pundit predictions, betting odds,
fantasy football data, etc.
Figure 4: The range of data types relevant to football. At a high-level, highly structured and
well-correlated data types are available for each match: video streams, audio streams, and
associated text streams. On a more aggregate level, historical play data (e.g., player and
team statistics) spanning many decades of play are broadly available for many professional
football leagues. Moreover, third-party companies (such as InStat, 2020; Opta, 2020; Stats-
Bomb, 2020; Wyscout, 2020) provide annotated player event and tracking data. Finally,
there exists a broad range of crowd-sourced data for football, including user preferences and
pundit predictions, betting odds, and so on.
such as vision, audio, and text are the mainstay of AI research, and are abundant in football.
Crucially, the various data types available in football are well correlated, in the sense that
they typically involve a large amount of shared context{a characteristic researchers can take
advantage of. For instance: football video feeds always involve two teams and a ball; an
enormous amount of text is available, yet it is centered on the current game; the sound of
the crowds and commentators can be relied upon to respond to temporally-adjacent events,
such as goals and penalties. There is a large amount of crowd-sourced data available such
as current betting odds and pundits' predictions, a novel form of data atypical in other AI
application domains. Football oers the opportunity for AI to evaluate multi-modal models
on synthesized vision, audio, and text data in a unied (though simpler) domain than
the broader real world. Naturally, the abundance of data raises important considerations
with respect to the computational eciency of both training and deploying such models in
54
Game Plan: What AI can do for Football, and What Football can do for AI
a scalable manner, which remains an open challenge. Moreover, despite the availability of
large-scale and increasingly-precise data, the issue of unobserved heterogeneity between data
and the real-world implies that AI and ML techniques that are robust to data uncertainty
(or lack of observations of certain key variables altogether) will play an important role in
advising football decision-makers with condence. Statistical Learning, however, already
provides useful techniques and models that can learn with missing and latent variables, and
as such provide a foundation for handling such challenges in the football domain (Goodfellow
et al., 2016; Murphy, 2013).
Football analytics also currently relies heavily upon hand-labeled data, such as ball
and player tracking and identication information. This reliance on hand-labeled data
imposes a signicant barrier for fast-paced analysis, due to the cost and time needed to
generate it. This provides a golden opportunity for AI to accelerate data collection and
the subsequent development of novel learning algorithms (by automating the labeling and
annotation process) and assisting coaches and decision-makers by instead allowing them
to focus their expertise on the tactical analysis of the game itself. As such, a worthy
long-term challenge for football analytics is to develop such an assistive agent, which uses
minimal hand-labeled data: an Automated Video Assistant Coach (AVAC). A successful
AVAC would help players, coaches, and spectators alike. Specically, it could help the
players by analyzing their play for weak points to further develop. A player's performance
throughout a game could be analyzed to suggest improvements in position play and assessing
the performance overall. Prior to a game, an AVAC could suggest strategies tuned to
the opponents of the day. Coaches also seek to get the best out of their players, but
have a limited amount of time and many players to observe and provide feedback to. An
AVAC would oer coaches many opportunities to help individual players and the team as a
whole, suggesting player rosters for a given game, as well as trading or scouting strategies
based on counterfactual evaluation of team performance with brand new players. Such an
AVAC system would have the ability to automatically sift and label huge quantities of video
streams, enabling broadcasters and spectators alike to retrieve key moments. An AVAC
could automatically keep a running tally of information the spectator may nd interesting
based on their reaction and the current state of play. To enhance the spectator experience,
the AVAC may automatically generate highlight reels that eectively re
ect the
ow of
the game or summarize the most exciting segments (Mahasseni, Lam, & Todorovic, 2017;
Merler et al., 2018; Xiong, Kalantidis, Ghadiyaram, & Grauman, 2019; Yang et al., 2015;
K. Zhang, Chao, Sha, & Grauman, 2016); moreover, the AVAC might suggest related games
predicted to engage or interest the spectator. For those interested in fantasy football, an
AVAC might search for players based on a set of qualities. Overall, the possibilities for such
an automated system are open-ended. To make the research objectives and intermediate
benets of developing an AVAC system concrete, we detail three associated research agendas
at increasing levels of abstraction in Section 4: representation learning, predictive modeling
and decision-making, and human factors.
4. Football for AI Research
We next consider the dual perspective of the potential unique challenges and opportunities
associated with the football domain that make it an interesting testbed for AI research. We
55
Tuyls, Omidshafiei, Muller et al.
Human Factors
Goal: Model intrinsic and extrinsic traits of players (i.e., on-pitch
behaviors), tie into strategic analysis (play style, risks of injury,
scouting, etc.).
Example outcomes: Predict player conditioning factors, injury risks,
for planning tactics and training curricula.
Predictive Modeling and Decision-Making
Goal: Strategic understanding of player and team behaviors under
various circumstances.
Example outcomes: Prescribe trajectories teams should have
executed in contrast to what they actually did, coupled with
counterfactual study.
Representation Learning
Goal: Establish a signal processing pipeline bypassing sparse
statistics available traditionally in football analytics.
Example outcomes: Detect events, player poses, learn useful
representations of players, teams, and contextual factors.
Frontier 2
(SL&CV)
Frontiers
1 (GT&SL)
and
3 (GT&CV)
Microcosm
(All Frontiers)
Automated Video Assistant Coach (AVAC)
Figure 5: Hierarchy of key research challenges, dened over three layers. The foundational
layer targets representation learning over the various input modalities available in football to
generate useful representations for more complex learning tasks targeted in the subsequent
layers: prescriptive and predictive analysis and modeling of human factors.
introduce here a hierarchy of key challenges associated with football research, illustrated
in Figure 5, as dened over three layers: the foundational layer concerns representation
learning, operating directly on the various input modalities available in football (e.g., timesynchronized
videos, event-streams, and tracking data) to generate useful representations
for the more complex learning tasks targeted in the subsequent layers of prescriptive and
predictive analysis, and modeling of human factors. We next detail each of these research
layers, drawing connections with the three aforementioned frontiers.
4.1 Representation Learning
The variety of hand-labeled football statistics make for natural fodder for machine learning
algorithms. These algorithms range from classication and regression tools (e.g., in expected
possession value models, Fernandez, Bornn, & Cervone, 2019), generative models (e.g., in
trajectory generation models, H. Le et al., 2017; H. M. Le, Yue, Carr, & Lucey, 2017; Li,
Jiang, Zhu, Che, & Liu, 2020; Yeh, Schwing, Huang, & Murphy, 2019), and variational autoencoding
models (player embeddings). The success of machine learning algorithms generally
depends on data representation, as dierent representations can entangle and obfuscate
various explanatory factors of variation behind low-level sensory data. In football analytics,
although expert knowledge is widely used to help design existing representations, learning
with generic priors bears promise for avoiding such hand-encoded knowledge. Under this
view, we identify three unique challenges related to representation learning, detailed next.
56
Game Plan: What AI can do for Football, and What Football can do for AI
The rst challenge concerns learning representations with multi-modal football data.
Particularly, in football analytics, it remains a fundamental challenge to eectively recognize
long-duration playing styles of individual players and teams given the variety of data types
available (as detailed earlier). While expert knowledge goes a long way towards analyzing
these multi-modal data sources, it remains insucient to process them eciently. The
increasing multitude of input modalities available to football analysts are likely to challenge
existing methods of representation learning, thus driving researchers to develop cohesive
models that take these many modalities into account simultaneously.
The second challenge concerns learning contextual representations of individual players.
Due to the dynamics and uncertainty of football outcomes, long-term static representations
for predictive modeling of in-game events are likely to be benecial when used in conjunction
with representations of individual players. For example, a player passing the ball may
take into account the context of the game to estimate the most appropriate receiver that
maximizes the probability of scoring. Another concrete example is using contextual representations
to identify the dynamic roles of players, which may change given the game
context and must be inferred and controlled to tactically counter the opposing team. Finally,
player behaviors depend not only on the game context, but also on their and the
opposing team's overall strategy (e.g., formations, tactical advice provided by the coaching
sta, etc.).
Finally, in addition to learning representations of individual players, identifying an effective
means of contextualizing or ranking entire teams is another unresolved challenge.
Teams are usually ranked with historical match results and the collective performance of
individual players, which can be coarse (i.e., may not reveal the long-term playing styles
of teams) and may fail to re
ect in-game dynamics. Overall, to tackle these challenges,
we aim to achieve two goals: i) learning representations that are able to characterize the
long-term playing styles of football teams, ii) learning contextual representations of football
teams that are able to depict in-game dynamics.
4.2 Predictive Modeling and Decision-Making
Learning useful representations (i.e., as opposed to hand-coded features) serves as an important
means of advancing subsequent predictive and prescriptive analysis of football matches.
Specically, dense embeddings that summarize not only the state of a particular game, but
also historical trends evident throughout many games (e.g., across seasons) will serve as enablers
of the more accurate, impactful, and longer-horizon predictions of match outcomes.
The interaction between predictive-prescriptive models is envisioned to be tightly-linked
with game-theoretic analysis, thus coupling this direction of research most closely with
Frontier 3 (GT&CV) and Frontier 1 (GT&SL) (see Figure 5).
The combination of these elds with game theory is likely to usher in new opportunities
for coaches and decision-makers. For example, predictive models of football players at the
trajectory-level (H. Le et al., 2017; H. M. Le et al., 2017; C. Sun, Karlsson, Wu, Tenenbaum,
& Murphy, 2019) currently treat the game as a black-box dynamical process (i.e., a
system of dynamic entities making decisions solely based on the joint on-pitch state of the
teams); such models do not yet account for the game-theoretically driven counterfactual
responses of players to one another (e.g., taking into account the current game score, time
57
Tuyls, Omidshafiei, Muller et al.
remaining, relative strength of the two teams, impact of current game decisions on upcoming
matches, etc.). Conducting such an analysis of these models involves identication of
high-level strategies typically used by empirical game-theoretic techniques (so-called metastrategies)
(Tuyls et al., 2020; Wellman, 2006). These meta-strategies, for example, could
be clusters of on-pitch behaviors correlated with play style, counterattack types, defense
schemes (such as zonal vs. man-to-man defense), and so on. While such meta-strategies
are typically manually dened, automatically learning them poses an interesting challenge.
Appropriate clustering and identication of such meta-strategies involves not only access to
a large, representative dataset of plays, but also the aforementioned learned representations
that summarize the most relevant context for game theory models.
Synthesis of empirical games over the possible meta-strategies of two opposing teams
can be used to forecast the performance of various team tactics when pitted against one another
(e.g., investigating the Nash equilibrium of football at a higher level, rather than the
typically considered low-level scenarios such as penalty kicks). Moreover, while characterization
and ranking of players has received considerable attention in the literature (Bransen
& Van Haaren, 2019; Decroos et al., 2019; Decroos & Davis, 2019), automated ranking
of tactics has received considerably less attention (Decroos, Van Haaren, & Davis, 2018;
Meerho, Goes, Knobbe, et al., 2019). Application of game-theoretic analysis techniques
here remains unexplored to the best of our knowledge. Analysis of empirical games using
meta-strategies conditioned on player identities would be benecial for counterfactually
evaluating player performance in new teams (i.e., for scouting). For training sta, a model
that enables accurate evaluation of players' contributions to the team's overall strategy
would be valuable, for example, for pinpointing which players to coach or to substitute. For
broadcasters, automatic identication of salient, exciting meta-strategies (e.g., those that
are typically low in probability yet high in payo, or games where there is a large dierence
in terms of the play styles or meta-strategies of the teams) can be used for automatic
generation of highlight reels.
Learning the appropriate meta-strategies and associated predictive models are, simultaneously,
challenging in football due to the number of players involved on-pitch (and the
exponential size of the strategy space with respect to this quantity). Despite this, the
development of richer models leveraging more complex input modalities (e.g., video-based
representations) is likely to unlock commensurate benets (in terms of improved predictions
and strategic decision-making) for football analysts.
4.3 Human Factors
The human-centric nature of football analytics stems from several factors: coaching and improvement
of individual play and coordinated team play through predictive and prescriptive
modelling, injury and fatigue prediction, and psychological analysis of players, which can
aect the decision-making of players and lead to non-rational behaviors, an aspect modeled
by behavioral game theory (Camerer, 2011; Camerer et al., 2005). This focus distinguishes
it sharply from, for example, challenges such as RoboCup (RoboCup project, 2020). In
contrast to the robot-centric focus of RoboCup (which revolves around developing robotic
footballing agents, Hanna & Stone, 2017; Hausknecht & Stone, 2016; Kalyanakrishnan &
Stone, 2010; Stone, Sutton, & Kuhlmann, 2005; Urieli, MacAlpine, Kalyanakrishnan, Ben-
58

4.3 Human Factors
The human-centric nature of football analytics stems from several factors: coaching and improvement
of individual play and coordinated team play through predictive and prescriptive
modelling, injury and fatigue prediction, and psychological analysis of players, which can
aect the decision-making of players and lead to non-rational behaviors, an aspect modeled
by behavioral game theory (Camerer, 2011; Camerer et al., 2005). This focus distinguishes
it sharply from, for example, challenges such as RoboCup (RoboCup project, 2020). In
contrast to the robot-centric focus of RoboCup (which revolves around developing robotic
footballing agents, Hanna & Stone, 2017; Hausknecht & Stone, 2016; Kalyanakrishnan &
Stone, 2010; Stone, Sutton, & Kuhlmann, 2005; Urieli, MacAlpine, Kalyanakrishnan, Ben-
58
Game Plan: What AI can do for Football, and What Football can do for AI
tor, & Stone, 2011; Visser & Burkhard, 2007), the focus in football analytics is entirely on
understanding and improving human gameplay and team coordination based on an integrated
approach from the three research areas involved. Another key dierence concerns
evaluation of the impact said analysis on human play, which is distinct from evaluation of
robotic agents in the RoboCup project. Namely, human play is signicantly more dicult
to realistically simulate and systematically evaluate (in contrast to evaluation on robotics
platforms). Moreover, the football analytics frontiers targeted here entail taking into account
human factors such as injury and fatigue, but also inter-player relationships and their
eects on play eciency and cooperation, psychological challenges such as pressure or mental
state, notably on recent transfers, and their impact on play performance, and overall
player discipline and tendency to follow the best plan for the team instead of the best plan
for themselves.
Injury prediction is another topic of interest. Injury prediction is the task of predicting
the probability that a player will sustain an injury given data on the past and current
seasons. Previous studies have investigated the acute-chronic workload ratio (ACWR) as
a predictor for sports-related muscular injuries (Gabbett, 2010). Acute workload measures
an athlete's workload over 1 week, while chronic workload is the average workload over the
past 4 weeks. Rossi et al. (2018) use richer workload measures extracted from Electronic
Performance and Tracking Systems (EPTS) to train a decision tree to predict the probability
of future injuries. Their approach uses manually designed features that aim to temporally
aggregate players' workload histories. Recent work uses Convolutional Neural Networks
(CNNs) applied to EPTS time-series data directly, thereby alleviating the need for handdesigned
time-aggregated features (Gabbett, 2010). Current injury prediction methods are
limited to a binary signal and do not explicitly capture uncertainty of the prediction, the
type of injury, the severity, nor projected time to recover. Progress in this direction is likely
limited by the availability of data; preventive measures are taken by sports clubs and as
a result injuries occur relatively infrequently, although accurate prediction of such injuries
(or determination of whether current performance sensors are sucient for doing so) is a
promising avenue for application of AI techniques.
5. Illustrative Examples: Frontier 1 (GT&SL)
In this section, we highlight some of the benets that the combination of frontiers can
yield for football analytics. We focus these examples on Frontier 1 (GT&SL), in particular,
given the track record of game theory and statistical learning work done in football
analytics in recent years; thus, this section provides a concrete sampling of the types of
multi-disciplinary contributions that can be made via the proposed microcosm-centric vision.
In the following, we rst provide an overview of the necessary game theory background.
Subsequently, we conduct an in-depth analysis of real-world football data under the lens
of Frontier 1 (GT&SL), providing new insights into penalty kick scenarios by combining
statistical learning with game theory.
5.1 Game Theory: Elementary Concepts
Empirical game theory has become an important tool for analysis of large-scale multi-agent
settings, wherein either a large amount of data involving agent interactions is readily avail-
59
Tuyls, Omidshafiei, Muller et al.

a11; b11
a21; b21
a12; b12
a22; b22

Figure 6: Bimatrix payo tables (A;B) for a two-player, two-action NFG. Here, the row
player chooses one of the two rows, the column player chooses on of the columns, and the
outcome of their joint action determines the payo to each player.
able, or is collected through simulations of the system under study for the construction
of the games (Tuyls et al., 2020; Wellman, 2006). Empirical game-theoretic modeling of
penalty kick taking and set pieces facilitates strategic understanding of player and team
behavior under various circumstances (e.g., play according to a Nash equilibrium), and
can assist both in predicting opponent behavior and prescribing how a player (or team)
should behave in the presence of other players (teams). These game-theoretic models can
be leveraged in pre- and post-match analysis, and can be combined with analysis of dynamic
trajectory behavior (e.g., generative trajectory prediction or `ghosting', as later described).
Additionally, the models can be enriched by carrying out Empirical Game Theoretic Analysis
(EGTA) on meta-game models of set pieces, automatically clustering and identifying
useful meta-strategies, and providing insights into higher-level team strategies.
A common representation of a game used for EGTA analysis is a Normal Form Game
(NFG), dened in the following.
Denition 1 (Normal Form Games (NFG)) A game G = hP; (Si); (ui)i consists of a
nite set of players, P, indexed by i; a nonempty set of strategies Si for each player; and a
utility function ui : j2P Sj ! R for each player.
In this work, we solely focus on bimatrix games, which are 2-player NFGs, G = hP; (S1; S2); (u1; u2)i
with jPj = 2. The utility functions (u1; u2) can be described in terms of two payo matrices
A and B, wherein one player acts as the row player and the other as the column player.
Both players execute their actions simultaneously. The payos for both players are represented
by bimatrix (A;B), which gives the payo for the row player in A, and the column
player in B (see Figure 6 for a two strategy example).
A player i may play a pure strategy, si 2 Si, or a mixed strategy, i 2
(Si), which is a
probability distribution over the pure strategies in Si. In a strategy prole  = (1; 2), each
player has a strategy i. We use notation 􀀀i to denote a strategy prole for all players
excluding i. Having dened NFGs, we can model empirical games as an NFG wherein
player payos are directly computed from data of real-world interactions or simulations.
For example, one can construct a win-loss table between two chess players when they both
have access to various strategies.
Given an NFG, the traditional solution concept used in game theory is the Nash equilibrium,
which selects strategy proles such that no player can benet from unilateral deviation:
Denition 2 (Nash Equilibrium) A strategy prole  is a Nash equilibrium if and only
if,
E[ui(
i ; 􀀀
i)]  E[ui(s0
i; 􀀀
i)] 8i 8s0
i 2 Si: (1)
60
Game Plan: What AI can do for Football, and What Football can do for AI
Figure 7: Visualization of shot distribution for the penalty kicks in the considered dataset.
Figure 8: Illustration of natural vs. non-natural sides.
In a so-called -Nash equilibrium, there exists at least one player who could gain by
deviating to another strategy, but that gain is bounded by . More formally:
Denition 3 (-Nash Equilibrium) A strategy prole  is an -Nash equilibrium if and
only if there exists  > 0 such that,
E[ui(
i ; 􀀀
i)]  E[ui(s0
i; 􀀀
i)] 􀀀  8i 8s0
i 2 Si: (2)
5.2 Game Theory for Penalty Kick Analysis
For our analysis we use a data set of 12399 penalty kicks based on Opta data (Opta, 2020).
In Figure 7 we show a heatmap of the shot distribution of the penalty kicks in our data set.
Table 1 shows the distribution of the penalty kicks over the various leagues we consider.
Palacios-Huerta (2003) examines penalty kick scenarios from a game-theoretic perspective,
using empirical payo tables to determine whether the associated kickers and goalkeepers
play a Nash equilibrium. Here we revisit the work of Palacios-Huerta (2003), by
rst reproducing several of its key results with a substantially larger and more recent data
set from the main professional football leagues in Europe, Americas, and Asia (for comparison,
the data set used in the work of Palacios-Huerta (2003) consists of 1417 penalty
kicks from the 1995-2000 period, whereas ours contains 12399 kicks from the 2011-2017
61
Tuyls, Omidshafiei, Muller et al.
Table 1: Penalty kick distribution over leagues considered (12399 kicks in total).
League # Kicks
Italian Serie A 607
US Major League Soccer 575
Engl. Npower Champ. 569
Spanish Segunda Division 568
Spanish La Liga 531
French Ligue 1 497
German DFB Pokal 441
Brazilian Serie A 440
Engl. Barclays Premier League 436
German Bundesliga 409
Dutch Eredivisie 398
German Bundesliga Zwei 389
Portuguese Primiera Liga 352
Saudi Arabian Profess. League 337
Russian Premier League 329
Chinese Super League 324
Copa Libertadores 322
Belgian Jupiler League 287
Turkish Super Lig 284
French Ligue 2 270
Argentina Primera (Anual) 261
English Capital One Cup 234
Mexican Primera (Clausura) 234
Colombia Primera Apertura 221
Norwegian Tippeligaen 219
AFC Champions League 193
International Champions Cup 188
Australian A-League 172
Copa Chile 172
English FA Cup 153
Copa do Brasil 153
League # Kicks
Chile Primera (Apertura) 151
Japanese J-League 149
English League 1 139
English League 2 130
Austrian Bundesliga 129
Danish Superligaen 115
European World Cup Qualiers 108
Internationals 93
African Cup of Nations 90
United Soccer League 80
Europ. Championship Qualiers 78
Swedish Allsvenskan 74
Coppa Italia 67
Copa America 51
FIFA Club World Cup 51
World Cup 48
Europ. Championship Finals 45
Champions League Qualifying 41
Confederations Cup 39
UEFA Europa League Qualifying 32
Coupe de France 29
Belgian UEFA Europa Play-os 24
German 3rd Liga 23
Russian Relegation Play-os 15
Dutch Relegation Play-os 13
Copa Sudamericana 9
Friendly 4
German Bundesliga Playo 3
German Bundesliga 2 Playo 3
Swedish Relegation Play-o 1
period). While several results of this earlier work are corroborated, we also nd surprising
new additional insights under our larger dataset. We then go further to extend this analysis
by considering larger empirical games (involving more action choices for both kick-takers
and goalkeepers). Finally, we develop a technique for illustrating substantive dierences in
various kickers' penalty styles, by combining empirical game-theoretic analysis with Player
Vectors (Decroos & Davis, 2019) illustrating the added value and novel insights research at
Frontier 1 (GT&SL) of the microcosm can bring to football analytics.
As in Palacios-Huerta (2003)'s work, we rst synthesize a 2-player 2-action empirical
game based on our penalty kick data set. Table 2a illustrates the 2  2 normal form
62
Game Plan: What AI can do for Football, and What Football can do for AI
Table 2: Natural (N) / Non-Natural (NN) payo tables for Shots (S) and Goalkeepers (G).
Here, (c) and (d) compare Nash and empirical (Empir.) probabilities.
(a) Palacios-Huerta (2003) payo table.
N-G NN-G
N-S 0.670 0.929
NN-S 0.950 0.583
(b) Reproduced table.
N-G NN-G
N-S 0.704 0.907
NN-S 0.894 0.640
(c) Palacios-Huerta (2003) Nash probabilities.
NN-S N-S NN-G N-G
Nash 0.393 0.607 0.432 0.568
Empir. 0.423 0.577 0.400 0.600
Jensen{Shannon divergence: 0.049%
(d) Reproduced table Nash probabilities.
NN-S N-S NN-G N-G
Nash 0.431 0.569 0.408 0.592
Empir. 0.475 0.525 0.385 0.615
Jensen{Shannon divergence: 0.087%
as presented by Palacios-Huerta (2003). The actions for the two players, the kicker and
goalkeeper, are respectively visualized in the rows and columns of the corresponding payo
tables, and are detailed below. The respective payos in each cell of the payo table indicate
the win-rate or probability of success for the kicker (i.e., a score); for ease of comparison
between various payo tables, cells are color-graded in proportion to their associated values
(the higher the scoring probability, the darker shade of green used).
The choice of player actions considered has an important bearing on the conclusions
drawn via empirical game-theoretic analysis. The actions used by Palacios-Huerta (2003)
in Table 2a correspond to taking a shot to the natural (N) or non-natural (NN) side for the
kicker, and analogously diving to the natural side or non-natural side for the goalkeeper.
Figure 8 provides a visual denition of natural versus non-natural sides. Specically, as
players tend to kick with the inside of their feet, it is easier, for example, for a left-footed
player to kick towards the right (from their perspective); thus, this is referred to as their
natural side. Analogously, the natural side for a right-footed kicker is to kick towards their
left. The natural side for a goalkeeper depends on the kicker in front of him. Specically,
when facing right-footed kickers, goalkeepers' natural side is designated to be their right;
vice versa, when they face a left-footed kicker, their natural side is to their left. Importantly,
shots to the center count as shots to the natural side of the kicker, because, as explained
in Palacios-Huerta (2003), kicking to the center is considered equally natural as kicking to
the natural side by professional football players (Palacios-Huerta, 2003).
Table 2b shows our reproduction of Table 2a of Palacios-Huerta (2003), computed using
12399 penalty kicks spanning the aforementioned leagues in our Opta-based dataset; importantly,
players (goalkeepers and kickers) appear at least 20 times each in this dataset,
to ensure consistency with Palacios-Huerta (2003). The trends in these two tables are in
agreement: when the goalkeeper and the kicker do not choose the same sides of the goal,
shot success rate is high; otherwise, when the keeper goes to the same side as the kicker,
success rate is higher for natural shots than for non-natural shots. We also include Nash
and empirical probabilities for Palacios-Huerta's dataset and ours, respectively in Tables 2c
63
Tuyls, Omidshafiei, Muller et al.
Table 3: Natural / Non-natural game restricted by footedness.
(a) Left-footed players payo table
N-G NN-G
N-S 0.721 0.939
NN-S 0.903 0.591
(b) Right-footed players payo table
N-G NN-G
N-S 0.700 0.898
NN-S 0.892 0.653
Table 4: Footedness equivalence p-value tables.
(a) Natural / Non-natural game p-values
N-G NN-G
N-S 0.924566 0.170504
NN-S 0.394900 0.407741
(b) Left / Center / Right game p-values
R-G C-G L-G
R-S 0.000011 0.947369 6.931197e-01
C-S 0.592054 0.868407 1.305657e-01
L-S 0.017564 0.764020 7.791136e-07
and 2d, enabling us to conclude that payos, Nash probabilities, and empirical probabilities
are all in agreement between Palacios-Huerta's results and our reproduction; more
quantitatively, the Jensen-Shannon divergence between Palacios-Huerta's results and ours
is 0.84% for the Nash distribution and 1.2% for the empirical frequencies. We also notice
that players' empirical action selection frequencies are quite close to the Nash-recommended
frequencies, as measured by their Jensen-Shannon Divergence, and are actually playing an
-Nash equilibrium with a very low  of 0:4%.
Having examined the similarity of payo tables and distributions, we verify whether
the Natural / Non-Natural game is statistically identical for left-footed and right-footed
players (Table 3), as assumed in Palacios-Huerta (2003). To do so, we use a t-test to verify
whether per-cell scoring rates are identical across footedness types. The t-tests' p-values
are reported in Table 4a, and reveal that the games cannot be proven to be dissimilar
across footedness and can, therefore, be assumed to be identical for left-footed and rightfooted
players. Figure 9 renes this result by representing the relationship between pvalues
of our t-test and minimal player appearance counts: when we modulate minimal
appearance count of players in our test, the Natural Shot / Natural Goalkeeper cell goes
from strongly dissimilar across footedness (low p-value) when including all players, to likely
non-dissimilar (high p-value) when only including the players appearing the most in our
dataset. This could be explained by low-appearance-counts-, which we take here as a proxy
for low experience, kickers being less able to control their kicks, resulting in dierent control
eectiveness for dierent footedness preferences, and in goalkeepers being less procient in
stopping shots going to their less frequently-kicked side (left) than to the other, a preference
that we infer has been trained away in professional goalkeepers. To remove potential sidee
ects of merging data from low- and high-experience players together, Figure 10 shows
the relationship between p-values of our t-test and experience category where we allow for
some overlap{between 1 and 7 shots, 5 and 12, etc.; the insight drawn from this gure is the
64
Game Plan: What AI can do for Football, and What Football can do for AI
Figure 9: P-value table as a function of minimal experience.
Table 5: Left (L) - Center (C) - Right (R) tables for Shots (S) and Goalkeepers (G), with
the three directions of kick/movement dened from the goalkeeper's perspective.
(a) Payo table.
R-G C-G L-G
R-S 0.684 0.939 0.969
C-S 0.964 0.160 0.953
L-S 0.964 0.960 0.633
(b) Nash probabilities vs. Empirical frequencies corresponding to (a).
R-S C-S L-S R-G C-G L-G
Nash 0.478 0.116 0.406 0.441 0.178 0.381
Empirical 0.454 0.061 0.485 0.475 0.089 0.436
Jensen{Shannon divergence: 0.75%
same as that of Figure 9, supporting the conclusion that experience removes the dierence
between left- and right-footed penalty kicks.
We also analyzed the game dened by kicking to the left, center, or right, and conrmed
Palacios-Huerta's intuition that it is fundamentally dierent across footedness preferences.
Specically, Table 5a synthesizes the empirical game corresponding to this new choice of
65
Tuyls, Omidshafiei, Muller et al.
Figure 10: P-value table as a function of player-experience.
actions, with aggregated scoring rates over both feet preferences. Note that in this case, left,
center, and right are measured from the goalkeeper's perspective, such that the natural kick
of a right-footed player would be considered a right kick. The per-cell t-tests' p-values for
this game are reported in Table 4b. Interestingly, the game is dierent when the goalkeeper
jumps to the same side as the ball, but is otherwise mostly similar across footedness preference.
The empirical play frequencies for kickers, as reported in Table 5b, are also further
away from Nash frequencies than observed in the Natural / Non-Natural game (Table 2d),
as can be seen from the Jensen-Shannon divergence between empirical frequencies and Nash
(0.75%, versus the the 0.087% of the Natural / Non-Natural game) These insights indeed
conrm the intuition that such a game is neither correct across footedness, nor the one the
players follow.
Overall, these results provide insights into the impacts that the choice of actions have
on conclusions drawn from empirical payo tables, and are illustrative of the practical
usefulness of theoretically well-founded principles in application to real-world analytics,
which is highlighted as a hallmark of useful theory in Szymanski (2020). However, behavior
and shooting styles also vary wildly per-player given footedness. If one is willing to consider
several payo tables (e.g., one per footedness), it seems natural to also take into account
kickers' playing styles, as considered in the next section.
5.3 Augmenting Game-Theoretic Analysis of Penalty Kicks with Embeddings
While the previous section undertook a descriptive view of the penalty kick scenario (i.e.,
providing a high-level understanding of kicker and goalkeeper play probabilities), here we
66
Game Plan: What AI can do for Football, and What Football can do for AI
Table 6: Cluster statistics.
# Players # Goals # Shots Success % Proportion left-foot goals (%)
Cluster 1 197 144 167 86.2 10.4
Cluster 2 216 494 612 80.7 21.9
Cluster 3 52 3 4 75.0 33.3
Cluster 4 82 58 73 79.4 51.7
Cluster 5 87 44 60 73.3 34.1.0
Cluster 6 1 0 0 - 0.0
Total 635 743 916 81.1 25.2
Table 7: Pair-wise comparison for the identied clusters. < indicates that data was missing
and minimum true p-value may be lower than the reported minimum p-value in the
table. The symbol * indicates we cannot conclude whether clusters are dierent at the 10%
condence level.
1 vs. 2 1 vs. 4 1 vs. 5 2 vs. 4 2 vs. 5 4 vs. 5
Min. cell p-value
of t-test over table
equality
4.49e-2 < 9.56e-2* < 1.09e-1* 4.49e-2 4.48e-2 < 3.39e-1*
Jensen-Shannon div.
between Nash distr.
(%)
0.03 0.57 0.09 0.35 0.02 0.21
Jensen-Shannon div.
between empirical
distr. (%)
0.06 0.01 0.06 0.08 0.24 0.04
Left footedness t-test
p-value
3.43e-4 1.37e-7 3.18e-3 4.92e-5 1.07e-1 7.52e-2
investigate whether we can nd the best strategy for a player given the knowledge of the
kicker's play style. In game-theoretic terms, we conduct a prescriptive analysis of penalty
kicks to enable informed decision-making for players and coaching sta in specic penalty
kick situations. Ideally, one would iterate the earlier empirical payo analysis for every
possible combination of goalkeeper and kicker in a given league, thus enabling decisionmaking
at the most granular level; however, the inherent sparsity of penalty kick data
makes such an approach infeasible. Instead, we introduce a meaningful compromise here by
combining statistical learning with game theory (i.e., Frontier 1 (GT&SL)), rst quantifying
individual playing styles, then using clustering techniques to aggregate players (i.e., both
strikers and goalkeepers) based on said styles, and nally synthesizing empirical games
for each identied cluster. We focus our analysis on penalties including all players who
participated in Premier League matches from 2016 to 2019.
67
Tuyls, Omidshafiei, Muller et al.
Table 8: p-values for t-test that empirical action distributions are equal among dierent
clusters. Minimum p-value (across kicker and goalkeeper roles) is indicated in bold for each
row.
Kicker clusters compared Kicker p-value Goalkeeper p-value
1 vs. 2 0.52 0.05
1 vs. 4 0.85 0.95
1 vs. 5 0.42 0.27
2 vs. 4 0.52 0.14
2 vs. 5 0.51 0.16
4 vs. 5 0.4 0.26
0.1
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.2
0.1
0.0
0.1
0.2
0.3
0.2
0.1
0.0
0.1
0.2
The striker clusters
The goalkeeper cluster
Outlier
Cluster#1
Cluster#2
Cluster#3
Cluster#4
Cluster#5
Cluster#6
(a)
0.2 0.1 0.0 0.1 0.2 0.3 0.4
0.2
0.1
0.0
0.1
0.2
0.3 Ashley Westwood (midfielder)
Callum Wilson (striker)
José Holebas (left back)
Matthew Lowton (right back)
Cluster#1
Cluster#2
Cluster#4
Cluster#5
(b)
Figure 11: Visualization of the identied player clusters. (a) visualizes the goalkeeper cluster,
the kicker clusters and an outlier automatically detected through K-means clustering.
To show the separation of the kicker clusters clearly, we visualize them in (b) after removing
the goalkeeper and outlier clusters, and we also label each cluster with a Premier League
player in it.
On a technical level, our approach consists of the three following steps. First, we characterize
the playing style of a player in a manner that can be interpreted both by human experts
and machine learning systems. In particular, we use Player Vectors (Decroos & Davis,
2019) to summarize the playing styles of kickers using an 18-dimensional real-valued vector.
These Player Vectors are extracted from historical playing trajectories in real matches, with
technical details provided in Appendix C. Each dimension of the Player Vector corresponds
to individual on-pitch player behaviors (e.g., styles of passes, take-ons, shots, etc.), and
the value of each dimension is standardized and quanties the weight of that particular
action style for the considered player. We also lter experienced players with at least 50
68
Game Plan: What AI can do for Football, and What Football can do for AI
Table 9: Nash probabilities and empirical (Empir.) frequencies tables for Shot (S) and
Goalkeepers (G) with Natural (N) and Non-Natural (NN) actions. Note that Cluster 3 is
omitted due to it consisting of very few shots (taken by goalkeepers).
(a) All players. 916 total shots.
NN-S N-S NN-G N-G
Nash 0.391 0.609 0.406 0.594
Empir. 0.503 0.497 0.413 0.587
-Nash equilibrium:  = 2:71%
(b) Kickers in Cluster 1. 167 total shots.
NN-S N-S NN-G N-G
Nash 0.423 0.577 0.379 0.621
Empir. 0.485 0.515 0.371 0.629
-Nash equilibrium:  = 0:08%
(c) Kickers in Cluster 2. 612 total shots.
NN-S N-S NN-G N-G
Nash 0.401 0.599 0.430 0.570
Empir. 0.520 0.480 0.418 0.582
-Nash equilibrium:  = 2:89%
(d) Kickers in Cluster 4. 73 total shots.
NN-S N-S NN-G N-G
Nash 0.320 0.680 0.375 0.625
Empir. 0.479 0.521 0.438 0.562
-Nash equilibrium:  = 5:17%
(e) Kickers in Cluster 5. 60 total shots.
NN-S N-S NN-G N-G
Nash 0.383 0.617 0.317 0.683
Empir. 0.450 0.550 0.400 0.600
-Nash equilibrium:  = 4:86%
Figure 12: Heatmaps of goals by all kickers and kickers in individual clusters with respect
to empirical probabilities. We exclude the goalkeeper cluster (Cluster 3) and the outlier
cluster (Cluster 6) because of insucient samples.
appearances in the Premier League matches from 2016 to 2019. In total, we obtain 635 such
vectors for the individual players in our dataset. Second, we cluster players in accordance
to their Player Vectors, using K-means with the number of clusters chosen as the value
causing the most signicant drop in inertia (a standard heuristic). This process yields 6
clusters in total, with statistics summarized in Table 6. In particular, K-means clustering
69
Tuyls, Omidshafiei, Muller et al.
detects an outlier cluster with only one player (Cluster 6), and we also observe that there are
very few shot samples in Cluster 3, as it consists of a cluster of goalkeepers (an interesting
artifact illustrating the ability of Player Vectors and K-means clustering to discern player
roles). Given the few samples associated with these two clusters, we henceforth exclude
them from the game-theoretic analysis. We observe that cluster pairs (1, 2), (1, 4), (2, 4),
and (2, 5) are signicantly dierent, with the minimum cell-wise p-values for these cluster
pairs smaller than 0.10 in Table 7. We therefore focus our game-theoretic analysis on these
cluster pairs. Moreover, we also qualitatively illustrate dierences between the clusters in
Figures 11a and 11b, which visualize the results of reducing the Player Vectors dimensionality
from 18 to, respectively, 3 and 2 via Principal Component Analysis. Here, we observe
that the goalkeeper cluster is well-separated from the kicker clusters in Figure 11a, and in
order to better visualize the kicker clusters, we project Figure 11a onto its x and y axis
after removing the goalkeeper and outlier clusters in Figure 11b. We also identify therein
the most representative kicker per-cluster (i.e., the player whose feature vector is closest to
the mean of the corresponding cluster)
Finally, we conduct the aforementioned game-theoretic analysis for each cluster. In our
earlier Table 6, we observe that the kickers in some clusters have dierent success rates
in penalty kicks. Moreover, a closer behavioral analysis yields deeper insights. We rst
examine the Nash strategies played by each cluster, and then visualize the actual play behavior
with respect to empirical probabilities in Figure 12. Table 9a summarizes the overall
Nash distributions for all players considered, with Tables 9b to 9e showing cluster-specic
distributions. These tables illustrate that the kickers have the same empirical behavior, an
assertion statistically conrmed in Table 8; yet their Nash-derived recommendations are
dierent: although kickers in all clusters are recommended by the Nash to shoot more to
their natural sides than to their non-natural sides, the recommended strategy for kickers in
Cluster 1 is actually quite balanced between natural and non-natural shots. This greater
imbalance is shown by comparing Jensen-Shannon divergence. As we see in Table 7, the
Jensen-Shannon divergence of the Nash probabilities between Cluster 1 and 4 (0.57%) is
6-7 times greater than that between Cluster 1 and 5 (0.09%) and 19 times greater than
that between Cluster 1 and 2 (0.03%). We also notice that the clusters' players are all playing
epsilon Nash equilibra with relatively low epsilon (Table 9). In other words, although
their empirical strategies seem to deviate from corresponding Nash strategies action-wise,
the expected payos of these two strategies are close, and they could still stand to gain
in "stability" by switching to corresponding Nash strategy. Nevertheless, most of these
Nash recommendations come from very low-sample empirical payo tables, which entails
potentially inaccurate Nash distributions. We nevertheless note that this low-data regime
is induced by the restriction of our analysis to players having played in matches of Premier
League only from 2016 to 2019. Obtaining Player Vector data for all players in our dataset
would allow us to study cluster behavior with greater statistical precision. Nevertheless,
the current study leaves no statistical doubt regarding the pertinence of clustering payo
tables using player embeddings{specically Player Vectors.
Qualitatively, in addition to analyzing the strategies with respect to Nash probabilities,
the patterns of positions of the ball of successful goals also vary from clusters to clusters,
as visualized in Figure 12. For instance, kickers in Cluster 2 tend to score mostly to the
70
Game Plan: What AI can do for Football, and What Football can do for AI
(a) (b)
Ball (truth) Attackers (truth) Defenders (truth) Defenders (predicted)
Figure 13: Predictive modeling using football tracking data. (a) visualizes predictions under
the original data. Here, ground truth information for all players and the ball is provided
to the underlying predictive model, with defender positions truncated and predicted by
the model after a cut-o time (as indicated by the yellow traces). (b) illustrates the same
scenario, after counterfactual perturbation of the ground truth ball direction to ascertain
the predicted reaction of the defending goalkeeper (far right).
bottom left corner of the goalmouth, while the scoring positions in other clusters are more
balanced, though these could also be partly due to lower sample sizes for some clusters.
5.4 Generative Trajectory Prediction Models for Counterfactual Analysis
Ghosting refers to the prescription of the trajectories the players in a sports team should
have executed, in contrast to what they actually did (Lowe, 2013). Solution of this and the
broader problem class of generative trajectory prediction implies benets spanning from
recommendation of trajectories or setups for constrained set pieces, then to short-term
plays involving a subset of players, and eventually to long-term strategies/plays for the
entire team. Team-level predictions would also strongly benet from game-theoretic and
multi-agent considerations, and is perceived to play a key role in an established AVAC
system. We here present an illustrative example to ground the earlier discussion regarding
the potential impacts of using learned predictive models to conduct counterfactual analysis
of football matches.
For example, one might train a trajectory prediction model on league data (e.g., as
done in H. M. Le et al. 2017), provide an input context to such a model (e.g., consisting
of the true state of the ball, defenders, and attackers up to some point in time), and
subsequently predict future trajectories of players. Figure 13a visualizes league-average
predicted behaviors conditioned on such an input context. This illustrative example was
trained using a baseline predictive model, similar to that of H. Le et al. (2017). Here we
trained a centralized long-short term memory model (of 2 layers, each with 256 units),
taking as input the raw trajectories of players and the ball, and predicting as output the
step-wise change in trajectory of the defensive players. The model was trained on 240 frames
71
Tuyls, Omidshafiei, Muller et al.
of 25 fps tracking data, downsampled to 12.5 fps, with half the frames in each play used for
providing a prediction context, and the other half occurring at the prediction cut-o. We
used the l2-loss on the tracking data for training, and randomized the order of attacking
and defending players to avoid the role-assignment problem mentioned in H. Le et al. (2017)
(similar to one of the baseline approaches of Yeh et al. 2019).
As pointed out in the literature (H. Le et al., 2017; H. M. Le et al., 2017; Li et al., 2020;
Yeh et al., 2019), a key advantage of generative predictive models is that they can be used
for counterfactual analysis of play outcomes. We illustrate such an example in Figure 13b,
where we perturb the trajectory of the ball, inferring the subsequent behaviors of defenders
in reaction (noting, e.g., the tendency of the goalkeeper to chase the ball in reaction to it
entering the penalty area). While simple, case-by-case counterfactual case studies such as
the above have been conducted to some extent in the literature, consideration of responses
to more complex perturbations (e.g., changes of one team's tactics or meta-strategy as
a whole, changes in player behavior due to injuries, or changes due to substitutions of
individual players) bear potential for signicantly more in-depth analysis.
6. Discussion
Football analytics poses a key opportunity for AI research that impacts the real world.
The balance of its reasonably well-controlled nature (versus other physical domains beyond
sports, e.g., search-and-rescue), considerations associated with human factors (e.g., heterogeneous
skill sets, physiological characteristics such as injury risks for players, etc.), and the
long-term cause-and-eect feedback loop due to the relative infrequency of scoring even in
professional play make it a uniquely challenging domain. Nonetheless, the rapidly-emerging
availability of multi-modal sensory data make it an ideal platform for development and
evaluation of key AI algorithms, particularly at the intersection of the aforementioned elds
of statistical learning, computer vision, and game theory.
In this paper, we highlighted three frontiers at the intersection of the above elds,
targeting the simultaneous advancement of AI and football analytics. We highlighted the
overlying goal of developing an Automated Video Assistant Coach (AVAC), a system capable
of processing raw broadcast video footage and accordingly advising coaching sta in pre-,
in-, and post-match scenarios. We subsequently illustrated how the combination of game
theory and statistical learning could be used to advance classical results in football analytics,
with an in-depth case study using a dataset comprised of over 15000 penalty kicks, and
subsequently combined with the Player Vectors analysis of Decroos and Davis (2019) to
discern kicking styles.
A notable observation for future work focusing on prescriptive football analytics is that
the domain and some of the state-of-the-art research bear key similarities to RL. At a high
level, the process of winning football championships can be cast as a sequential decisionmaking
problem, with a concrete reward structure centered on three timescales of increasing
abstraction: scoring goals, winning matches, and subsequently winning championships. We
illustrate this view in Figure 14. Under this hierarchical view of football, each layer can be
considered an RL problem at the designated level of abstraction. For example, at the lowest
level, the sequential decisions made by teammates that lead to a goal can be considered a
policy mapping states to actions, using the lexicon of RL. Likewise, estimates of the value
72
Game Plan: What AI can do for Football, and What Football can do for AI
Championship level
Match level
Action level
Pass Cross Shot Corner
kick
SCORE
Predict/Optimize Values VC of Events EC to win championship
Predict/Optimize Values VM of Events EM to win match
Predict/Optimize Values VA of Events EA to score goals
Loss
Win
Conceded
Scored
C
ONT
E
XT
Draw
Figure 14: A multi-level view of football analytics cast as a reinforcement learning problem.
We discern three levels: the top level aims to learn how to win championships by winning
matches; the middle level optimizes for winning a match; nally, the bottom level seeks to
optimize goal-scoring. The context between these various level is shared in both a top-down
and bottom-up fashion.
of player actions based on the outcomes associated with actions taken in real games (as
in VAEP, Decroos et al., 2019) can be considered analogous to those that learn actionvalues
associated with RL policies. Further expanding this analogy, learning to quantify
the contribution of individual players to a team's estimated goal-scoring value can be cast
as a so-called credit assignment problem, a key area of research in RL. Finally, given the
presence of multiple on-pitch players with both cooperative and competitive incentives, the
value function learning problem situates itself in the area of multi-agent RL. Multi-agent
RL, critically, seeks to understand and learn optimal policies for agents in such interactive
environments, linking also to game theory in providing the appropriate mathematical
foundations to model this strategic process. As such, the multi-agent RL approach ts well
under Frontier 1 (GT&SL), which considers the game-theoretic interactions of strategic players
given specied payos, and use of learning techniques for identifying optimal policies.
Moreover, this connection also highlights a potential overlap of interest between real-world
football and RoboCup, in that the RL paradigm can be used to optimize player and robot
policies alike, despite the widely-dierent player embodiments considered in each of these
two elds. Overall, such parallels can be drawn at all levels of abstraction highlighted in the
aforementioned hierarchical process modeling football championships, implying the foreseeable
importance of the RL paradigm as football analytics shifts from understanding the
game to subsequently optimizing player and team decisions at increasingly broader levels.
Moreover, the toolkits developed within the context of football analytics are also likely to
have direct benets for closely-related elds, and could be foreseeably adapted to many other
73
Tuyls, Omidshafiei, Muller et al.
sports. Specically, while we focus on football in this paper, analysis of team coordination
behavior is also being conducted in a multitude of other sports (Albert, 2010; Albert et al.,
2002; Anderson et al., 2020; Baumer & Zimbalist, 2014; Costa et al., 2009; Gauriot et al.,
2016; Michael, 2004; Puerzer, 2002; Skinner, 2010; Song et al., 2017; Yee, Lisy, & Bowling,
2016). Techniques developed for other sports could, in turn, be reciprocally applied to
football. In particular, the element of team play is a key characteristic of football that is
shared with sports such as basketball and hockey; in fact, the similarities between these
sports are strong enough that for certain problem regimes, the same foundational techniques
can be seamlessly applied across datasets for each sport (e.g., the work of Yeh et al. (2019) on
player trajectory predictions for basketball and football). A related mapping concerns the
application of football analytics techniques to the emerging eld of eSports, wherein there
is a large amount of data collected (in both raw video form, and structured data formats),
e.g., such data streams are available for games such as Dota 2 or StarCraft. In Dota 2,
for example, a coaching functionality analogous to that in football is available, wherein an
experienced player is connected to the game and advises other players on various strategic
tactics. Moreover, several of the most popular eSports games are inherently multi-player, in
the sense that their outcomes are not determined by only an individual's skill, but a team's
skill, mixing cooperative and competitive behaviors (as in football). Automatic analysis
of games could provide insights into weak and strong points of teams, tactics used, and
directions for improvement. These related domains could, therefore, provide a low-hanging
fruit for football analytics techniques to generalize, in a seamless manner, beyond football.
Overall, the combination of data sources, downstream benets on related domains, and
potentials for impact that AI could have on the football domain are quite evident. Perhaps
more importantly, the promising commensurate impacts of football analytics on AI research
(through the feedback loop established between the football microcosm to the three foundational
elds highlighted in Figure 1) are foreseen to make football a highly appealing
domain for AI research in coming years.
Acknowledgments
The authors gratefully thank Thomas Anthony and Murray Shanahan for their helpful
feedback during the paper writing process. The authors also thank the editor and three
anonymous reviewers for their constructive feedback. Karl Tuyls and Shayegan Omidshaei
are equal contributors. Karl Tuyls is the corresponding author.
Appendix A. Additional Works Related to Statistical Learning in
Football
Evaluating the eect of individual actions throughout the game is challenging as they naturally
depend on the circumstances in which they were performed and have long-term
consequences that depend on how the sequence plays out. Most works have focused on
measuring the quality of specic action types in distinct concrete game situations (Barr,
Holdsworth, & Kantor, 2008; Bransen & Van Haaren, 2018; Spearman, 2018). More recent
work has focused on a unifying view in which actions are valued according to how
they increase or decrease the likelihood of the play leading to a goal (Decroos et al., 2019;
74
Game Plan: What AI can do for Football, and What Football can do for AI
Fernandez, 2019). The main idea is to estimate the value of a given `state' of the game.
Intuitively, the state of a particular game includes everything that happened in the match
until this point, including the score, identities of players and associated traits, time left on
the clock, all prior actions, position of the players and the ball, etc.; moreover, one may
wish to also consider the state of a tournament as a whole (e.g., previous and upcoming
matches, the number of yellow cards accrued by players, etc.). A recent method used for
assigning values to on-ball actions is known as Valuing Actions by Estimating Probabilities
(VAEP) (Decroos et al., 2019). Actions are valued by measuring their eect on the game
state and in turn the probabilities that a team will score. These scores can then be used
to assess contribution of players to a team or measuring the mutual chemistry for a pair of
players (Bransen & Van Haaren, 2019).
Finally, a promising application of statistical learning is the development of models that
can carry out temporal predictions. This area is closely related to trajectory prediction
(Alahi et al., 2016; Deo & Trivedi, 2018; Fernando, Denman, Sridharan, & Fookes, 2018;
Gupta, Johnson, Fei-Fei, Savarese, & Alahi, 2018; Wang, Fleet, & Hertzmann, 2007). In
the context of sports analytics, such trajectory prediction models can be useful for conducting
the form of analysis known as ghosting, which, given a particular play, predicts
the actions that a dierent team or player would have executed. Beyond just capturing
game dynamics, models that can accurately carry out predictions could constitute valuable
tools for counterfactual reasoning, which allows us to consider the outcomes of alternative
scenarios that never actually took place. So far, such predictive models have been primarily
used for predicting the trajectory of the ball (Maksai, Wang, & Fua, 2016) and of players
themselves (H. Le et al., 2017; H. M. Le et al., 2017; Li et al., 2020; Su, Hajimirsadeghi,
& Mori, 2019; Yeh et al., 2019). Also of importance are models which identify player roles
from predicted trajectories (Felsen, Lucey, & Ganguly, 2018).
Appendix B. Pose Estimation
As previously illustrated, multi-person human pose estimation (Cheng, Yang, Wang, Yan,
& Tan, 2019; Y. He, Yan, Fragkiadaki, & Yu, 2020; Iskakov, Burkov, Lempitsky, & Malkov,
2019; Lassner et al., 2017; Pavlakos et al., 2019; Pavlakos, Zhou, Derpanis, & Daniilidis,
2017; Pavllo, Feichtenhofer, Grangier, & Auli, 2019) is a central part of vision-based analysis
of football video. Methods for this task can be grouped into two types: one the one hand,
bottom-up approaches rst detect human joints, and group them into pose instances (Fang,
Xie, Tai, & Lu, 2017; K. He, Gkioxari, Dollar, & Girshick, 2017; Huang, Gong, & Tao,
2017; Iqbal & Gall, 2016; Papandreou et al., 2017; K. Sun, Xiao, Liu, & Wang, 2019); on
the other, top-down approaches rst detect body instances and run single-person pose estimation
models on each instance (Cao, Simon,Wei, & Sheikh, 2017; Insafutdinov, Pishchulin,
Andres, Andriluka, & Schiele, 2016; Kocabas, Karagoz, & Akbas, 2018; Newell, Huang, &
Deng, 2017; Papandreou et al., 2018; Pishchulin et al., 2016). The computation cost of
top-down methods increases linearly with the number of people in an image, while that
of bottom-up methods stays constant. However, in cases where there is signicant overlap
between instances, top-down approaches are often more accurate (Y. Chen, Tian, & He,
2020).
75
Tuyls, Omidshafiei, Muller et al.
We experimented with G-RMI (Papandreou et al., 2017), a well-established top-down
approach, and give examples of predictions in Figure 2. In the rst stage, Faster-RNN (Ren
et al., 2015) is used to detect person instances. Inspired by detection methods, the second
stage combines classication and regression to process each resulting crop: a fully convolutional
network rst densely classies whether each spatial position is in the vicinity of a
given keypoint class, and then renes each prediction by predicting an oset. A specialized
form of Hough voting (see (Duda & Hart, 1972) for background) is introduced to aggregate
these predictions and form highly localized activation maps. A key-point based condence
score and non-maximum suppression procedure further improve results. We plan to build
on this approach to develop methods for the previously mentioned challenges.
Appendix C. Player Vectors
In particular, we follow denition of playing style in Decroos and Davis (2019), which is
dened as a player's preferred area(s) on the eld to occupy and which actions they tend
to perform in each of these locations, and generate our player vectors with the method
proposed in Decroos and Davis (2019). The procedure of generating player vectors unfolds
into four steps. First, we collect the event stream data of all Premier League matches that
Liverpool Football Club participated in from 2017 to 2019, and lter the actions of types
passes, dribbles, shots and crosses. Secondly, for each pair of player p, who is observed in
the event stream dataset, and relevant action type t, we overlay a grid of size 6040 on the
football pitch and count how many times player p performed action t in each grid cell. This
procedure yields a matrix which summarizes spatial preference of player p performing action
type t. Thirdly, we compress that matrix into a small vector. To do this, we reshape each
matrix into a vector and group it together with all other vectors of the same action type,
and we then perform non-negative matrix (NMF) factorization to reduce the dimensionality
of these matrices. This procedure yields a smaller vector, and the value of each dimension
quanties the preference of player p performing the action type t in the area a. Finally, for
each player, we obtain 4 vectors corresponding to the 4 action types, and we generate one
nal vector of 18 dimensions by concatenating his compressed vectors for relevant action
types.
References
Abu Farha, Y., Richard, A., & Gall, J. (2018). When will you do what?-Anticipating temporal
occurrences of activities. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition.
Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., & Savarese, S. (2016).
Social LSTM: Human trajectory prediction in crowded spaces. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition.
Alayrac, J.-B., Recasens, A., Schneider, R., Arandjelovic, R., Ramapuram, J., De Fauw, J.,
. . . Zisserman, A. (2020). Self-supervised multimodal versatile networks. In Advances
in Neural Information Processing Systems (NeurIPS).
Albert, J. (2010). Sabermetrics: The past, the present, and the future. Mathematics and
76
Game Plan: What AI can do for Football, and What Football can do for AI
sports, 3{14.
Albert, J., Bennett, J., & Mead, C. (2002, 10). Curve ball: Baseball, statistics, and the
role of chance in the game. Physics Today, 55 , 56-57.
Alp Guler, R., Neverova, N., & Kokkinos, I. (2018). DensePose: Dense human pose
estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition.
Anderson, A., Rosen, J., Rust, J., & Wong, K.-P. (2020). Disequilibrium play in tennis.
Working paper.
Arandjelovic, R., & Zisserman, A. (2017). Look, listen and learn. In Proceedings of the
IEEE International Conference on Computer Vision.
Arel, I., Rose, D., & Karnowski, T. (2010, 01). Deep machine learning - a new frontier in
articial intelligence research [research frontier]. IEEE Comp. Int. Mag., 5 , 13-18.
Azar, O. H., & Bar-Eli, M. (2011). Do soccer players play the mixed-strategy Nash equilibrium?
Applied Economics, 43 (25), 3591-3601.
Barr, G., Holdsworth, C., & Kantor, B. (2008). Evaluating performances at the 2007 cricket
world cup. South African Statistical Journal, 42 (2), 125{142.
Bartlett, R. (2006, 12). Articial intelligence in sports biomechanics: New dawn or false
hope? Journal of Sports Science & Medicine, 5 , 474-479.
Baumer, B., & Zimbalist, A. (2014). The sabermetric revolution: Assessing the growth of
analytics in baseball. University of Pennsylvania Press.
Beal, R., Norman, T. J., & Ramchurn, S. D. (2019). Articial intelligence for team sports:
A survey. The Knowledge Engineering Review, 34 , e28.
Bengio, Y. (2009, January). Learning Deep Architectures for AI. Foundations and Trends R

in Machine Learning, 2 (1), 1{127.
Bhattacharyya, A., Fritz, M., & Schiele, B. (2018). Long-term on-board prediction of
people in trac scenes under uncertainty. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition.
Bhattacharyya, A., Fritz, M., & Schiele, B. (2019). Bayesian prediction of future street
scenes using synthetic likelihoods. In Proceedings of the International Conference on
Learning Representations.
Bialkowski, A., Lucey, P., Carr, P., Yue, Y., Sridharan, S., & Matthews, I. (2015, 01).
Identifying team style in soccer using formations learned from spatiotemporal tracking
data. IEEE International Conference on Data Mining Workshops.
Bloembergen, D., Tuyls, K., Hennes, D., & Kaisers, M. (2015). Evolutionary dynamics of
multi-agent learning: A survey. J. Artif. Intell. Res., 53 , 659{697.
Bransen, L., & Van Haaren, J. (2018). Measuring football players' on-the-ball contributions
from passes during games. In International Workshop on Machine Learning and Data
Mining for Sports Analytics.
Bransen, L., & Van Haaren, J. (2019). Player chemistry: Striving for a perfectly balanced
77
Tuyls, Omidshafiei, Muller et al.
soccer team. In MIT Sloan Conference.
Bridgeman, L., Volino, M., Guillemaut, J.-Y., & Hilton, A. (2019, June). Multi-person 3D
pose estimation and tracking in sports. In IEEE Conference on Computer Vision and
Pattern Recognition Workshops.
Brock, A., Donahue, J., & Simonyan, K. (2019). Large scale GAN training for high delity
natural image synthesis. In Proceedings of the International Conference on Learning
Representations.
Busoniu, L., Babuska, R., & Schutter, B. D. (2008). A comprehensive survey of multiagent
reinforcement learning. IEEE Trans. Syst. Man Cybern. Part C, 38 (2), 156{172.
Buzzacchi, L., & Pedrini, S. (2014). Does player specialization predict player actions?
Evidence from penalty kicks at FIFA World Cup and UEFA Euro Cup. Applied
Economics, 46 (10), 1067{1080.
Camerer, C. (2011). Behavioral game theory: Experiments in strategic interaction. Princeton
University Press.
Camerer, C., Loewenstein, G., & Prelec, D. (2005). Neuroeconomics: How neuroscience
can inform economics. Journal of Economic Literature, 43 (1), 9{64.
Camerer, C., Loewenstein, G., & Rabin, M. (2011). Advances in behavioral economics.
Princeton University Press.
Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017). Realtime multi-person 2D pose estimation
using part anity elds. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition.
Carreira, J., & Zisserman, A. (2017). Quo vadis, action recognition? A new model and
the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition.
Chan, C., Ginosar, S., Zhou, T., & Efros, A. A. (2019). Everybody dance now. In
Proceedings of the IEEE International Conference on Computer Vision.
Chen, D. L., Kim, J., & Mooney, R. J. (2010). Training a multilingual sportscaster: Using
perceptual context to learn language. Journal of Articial Intelligence Research, 37 ,
397{435.
Chen, Y., Tian, Y., & He, M. (2020). Monocular human pose estimation: A survey of deep
learning-based methods. Computer Vision and Image Understanding, 192 , 102897.
Cheng, Y., Yang, B., Wang, B., Yan, W., & Tan, R. T. (2019). Occlusion-aware networks
for 3d human pose estimation in video. In Proceedings of the IEEE/CVF International
Conference on Computer Vision.
Chiappori, P.-A., Levitt, S., & Groseclose, T. (2002). Testing mixed-strategy equilibria
when players are heterogeneous: The case of penalty kicks in soccer. American
Economic Review, 92 (4), 1138{1151.
Choi, J., Kwon, J., & Lee, K. M. (2019). Deep meta learning for real-time target-aware
visual tracking. In Proceedings of the IEEE/CVF International Conference on Com-
78
Game Plan: What AI can do for Football, and What Football can do for AI
puter Vision.
ChyronHego. (2020). ChyronHego. Retrieved 2020-09-09, from https://chyronhego.com/
Clark, A., Donahue, J., & Simonyan, K. (2019). Adversarial video generation on complex
datasets. arXiv, arXiv{1907.
Clarke, A. (2020). Season trends: Quality on rise as mideld takes focus. Retrieved
2020-09-17, from http://www.premierleague.com/news/1745761
Claudino, J., Capanema, D., Souza, T., Serrao, J., Pereira, A., & Nassis, G. (2019, 07).
Current approaches to the use of articial intelligence for injury risk assessment and
performance prediction in team sports: A systematic review. Sports Medicine - Open,
5 .
Coloma, G. (2012). The penalty-kick game under incomplete information. University of
CEMA Economics Serie Documentos de Trabajo(487).
Costa, G., Huber, M., & Saccoman, J. (2009). Practicing sabermetrics: Putting the science
of baseball statistics to work. McFarland, Incorporated, Publishers. Retrieved from
https://books.google.fr/books?id=Kkf7gowrH UC
CVSports International Workshop on Computer Vision in Sports at CVPR. (2020). Retrieved
2020-09-07, from https://vap.aau.dk/cvsports/
Decroos, T., Bransen, L., Haaren, J. V., & Davis, J. (2019). Actions speak louder than
goals: Valuing player actions in soccer. In Proceedings of the 25th ACM SIGKDD in-
ternational conference on knowledge discovery & data mining, KDD 2019, anchorage,
ak, usa, august 4-8, 2019 (pp. 1851{1861). ACM.
Decroos, T., & Davis, J. (2019). Player vectors: Characterizing soccer players' playing
style from match event streams. In European Conference on Machine Learning and
Knowledge Discovery in Databases.
Decroos, T., Van Haaren, J., & Davis, J. (2018). Automatic discovery of tactics in spatiotemporal
soccer match data. In Proceedings of the international conference on knowl-
edge discovery & data mining.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A largescale
hierarchical image database. In 2009 IEEE conference on computer vision and
pattern recognition (pp. 248{255).
Deo, N., & Trivedi, M. M. (2018). Convolutional social pooling for vehicle trajectory
prediction. In Proceedings of the IEEE conference on computer vision and pattern
recognition workshops (pp. 1468{1476).
Dong, X., & Shen, J. (2018). Triplet loss in siamese network for object tracking. In
Proceedings of the european conference on computer vision (ECCV) (pp. 459{474).
Duda, R. O., & Hart, P. E. (1972). Use of the hough transformation to detect lines and
curves in pictures. Communications of the ACM, 15 (1), 11{15.
Emara, N., Owens, D. M., Smith, J., & Wilmer, L. (2014). Minimax on the gridiron: Serial
correlation and its eects on outcomes in the national football league. Available at
79