“He’s a skilled player, but it’s a risky pick.”
You’re sure to hear that line on draft day when it finally arrives, but what does that actually mean? Skill, and risk. In some ways, the two words have become almost synonymous in some circles of prospect evaluation, or at least in the perception of prospect evaluation. Hockey players who demonstrate a high aptitude in certain characteristics such as puck skills have almost become automatically perceived as risky types of players, despite a looming ignorance for whatever their defensive aptitude may be. Hopefully, by now you’re catching my drift that this is obviously a misconception. While those who evaluate players for a living, or even on a semi-serious basis are forced to confront this head-on, as a casual fan or fantasy owner, it’s quite easy to misconstrue the linguistic association of the two terms. In fact, the two factors – skill and risk, MUST be evaluated exclusively from one another.
Many NHL teams, in addition to moving towards an analytics-supported evaluation system, have also continued to develop analytics in terms of draft strategy, which sadly, doesn’t get quite the same amount of press. While some teams are sure to have unique iterations of their supplemental strategies, the concepts are likely very similar across the league. Today, I hope to convey the concept and how it can be applied to the draft, by using the method of comparative risk assessment that I’ve developed at my day job for less fun things, like triage and cost justification.
Quantitative risk assessment (QRA) is the concept of systematically evaluating and quantifying multiple factors that, when combined, produce a quantitative value reflective of risk. The concept is commonly applied in the sectors of health, and environment, but also in economics. One quick glance at some of the economic values being estimated for the MLB draft can quickly show why this is an important tool for NHL owners when it comes to the draft.
The first, and possibly most important thing to acknowledge about risk is that it is relative. That is to say that a player who might be a relatively risky selection in the mid-first-round, may very well be an innocent or safe selection in the early third-round. It might seem obvious when stated in those terms, but it is important to drive home the point that the selection of any one player is not inherently risky, but rather, varies in terms of risk depending on when they are hypothetically selected.
Risk, and more specifically the quantitative assessment of risk, when applied to the concept of drafting is ultimately the difference between two risk factors; 1 – the expected outcome/benefit (player value), and 2 – the expected input/cost (pick value). In common QRA practice, both variables are calculated by finding the product of consequence, likelihood, and exposure, then subtracting the bad stuff from the good stuff. When being applied to drafting, we need to manipulate the formula a bit…
1 – Expected Outcome (Benefit)
As mentioned above, the typical factors used to calculate both the outcome and the input of a comparative risk assessment are consequence, likelihood, and exposure.
When evaluating the expected outcome, we can stick to this formula relatively closely. I’ve found that the most comprehensive way of assessing a player’s expected outcome is by finding the sum of all possible outcomes. In this method of QRA, this step is probably the most important. It is what allows the person performing the assessment to combine the fruits of their raw scouting efforts throughout the year.
The sum of all outcomes can be found using this formula:
[(Outcome #1 x Likelihood #1) + (Outcome #2 x Likelihood #2) + …)
Where each product is related to a specific role that the player projects to be capable of satisfying (i.e. first-line/powerplay forward, third-pairing defenseman). Outcome should be quantified using a projection of WAR (wins above replacement), or your choice of comprehensive statistic in a certain role. For today, we can use this excellent descriptive correlation from Sean Butler
Left: distribution of wins above replacement seasons (from hockey-graphs). Right: 2016-17 standings.
Like @samanthacp_ said, I don’t think Larkin changes much. pic.twitter.com/xsX52ksX9F
— Sean Butler (@Butlrr) May 13, 2020
https://platform.twitter.com/widgets.js
Likelihood can be quantified by applying a 1-10 score from correlating with the following somewhat vague descriptions:
1-2 – Highly unlikely for this role to be satisfied in this players future
3-4 – Remotely possible that this role could be satisfied in this players future
4-5 – Somewhat possible that this role could be satisfied in the players future
6-7 – Player expected to satisfy this role in his future
8-9 – All but certain that this role could be satisfied in this players future
*Notice that the way each description is worded is such that the player wouldn’t need to be cast in that role for 100% of his deployment, but rather that he would be capable of satisfying the typical demands of that type of a role, among others simultaneously.
This factor makes things interesting because while the projected WAR outcome and likelihood associated with a player such as Cole Perfetti filling the role of a second-line forward is quite high, he likely wouldn’t receive much extra value in the roles of a third or fourth-line forward because his skill set simply doesn’t lend itself to being valuable in that type of deployment.
2 – Expected Input (Cost)
A couple of weeks ago, Jokke Nevalainen took a deep dive into which NHL teams have been the best at finding NHL talent in the draft. If you look around hockey Twitter or do a quick Google search, you’re bound to find plenty of articles focused on similar concepts as well as one of the most prominent concepts in draft theory today, draft pick value. Draft pick value is a concept that has been documented as a method by which teams determine the fair market for trading draft picks in a given draft, but another great use for it is in calculating the risk of a given selection.
Because it’s difficult to find an application for exposure in the input side of the equation, and because the likelihood of the selection being made is a constant, both factors can be eliminated from the formula.
This leaves consequence as the ONLY remaining factor for calculating the input. To quantify draft pick value we can use 1/10th of the values from Michael Shucker’s chart from the link above.
Putting it to the test!
So of course I didn’t plan on sharing this concept without following it up with a few exciting applications to get you rolling on your own fantasy QRA!
For the purposes of the sample calculations, I’ll be taking the seat of New York Rangers Director of Player Personnel, Gordie Clark on the first day of a hypothetical 2020 NHL Draft…
Today, Tankathon has blessed the Rangers with the 13th and the 23rd overall selections, giving us a good cross-section of the players available in the first round…
Also according to Tankathon’s mock draft feature, the top players available for each of my picks are, respectively:
Top Players Available 13th overall:
Potential Outcomes evaluated: 1st line forward+PP, 2nd line forward+PP, 4th line forward+PP
[(3.0 x 6) + (2.6 x 6) + (0.5 x 10)] = 38.6
Potential Outcomes evaluated: 1st line forward+PP, 2nd line forward+PP, 3rd line forward+PP, 4th line forward+PP
[(2.9 x 6) + (2.6 x 6) + (0.7 x 8) + (0.4 x 10)] = 42.6
Potential Outcomes evaluated: 2nd pairing defenseman+PP+PK, 2nd pairing defenseman+PK, 3rd pairing defenseman+PK, 3rd pairing defenseman
[(2.8 x 6) + (1.9 x 8) + (1.1 x 10)] = 43
Potential Outcomes evaluated: 1st line forward+PP, 2nd line forward+PP
[(3.5 x 6) + (3.0 x 6)] = 39
Since the cost (13th overall – 481) is a constant among the four available players, we can make our selection exclusively based on benefit. Meaning that, based on QRA, The New York Rangers optimal selection at 13th overall would be Braden Schneider. Schneider wins out on the basis that he projects to be an effective NHL player in a number of different deployment roles. While his ceiling may not rival that of Noel Gunler, he has a greater chance of being effective in several more ways and therefore mitigates the risk of selecting him at this point in the draft.
Where the comparative aspect comes into play is when a team is considering trading back. They can evaluate the risk associated with the players who they expect to be available at a later stage of the draft on a case by case basis. By doing this they can determine if they should expect to have a greater or lesser amount of risk exposure by trading back or maintaining their position. We’ll save that for another edition though.
***
Thanks for reading! I’d love your feedback, even if you want to poke some holes in the system.