Today, I decided to look at a topic that’s often discussed: how good exactly are NHL teams at scouting? NHL teams spend a lot of money on scouting. Every team has a large scouting crew on their payroll, and those scouts travel around the world to see prospects in action.

Nowadays, there are a ton of public draft rankings available online. These are usually done with minimal budget. The question that I wanted answered was: if you just combined those public rankings, could you get similar or even better results than what the NHL teams are doing?

To get some answers to this, I decided to review the 2013 NHL Draft and compare today’s results to what was the consensus back then. Why 2013? Well, it was seven years ago, so we should have a good understanding of the players from that draft class and what they ended up becoming. They are in their prime years right now. But also, there were some interesting names at the top of that draft, so that intrigued me.

Putting together the rankings

To get things started, I needed a consensus ranking from the public draft rankings that were available online at the time. Luckily, MyNHLDraft.com had 12 different top 30 rankings for the 2013 NHL Draft, so I took all of them and calculated the consensus from those. That gave me a total of 46 names that got at least one top 30 nomination, so I included all 46. I’ll refer to this as the Online Consensus from now on.

For the NHL Scouts consensus ranking, I obviously just used the draft results. It would be nice if all 30 teams shared their top 30 rankings with me but that’s not going to happen. To have a comparable list, I only included the first 46 picks and then everyone else was considered to be outside the top 46. Even though we actually know where they ended up being drafted – whether it was 50^th overall of 200^th overall – it’s not fair to use that information because we don’t have a similar size sample on the Online Consensus side.

2013 Re-draft

To get a current day comparison point, I had to do a re-draft. I didn’t take team needs or situations into consideration, I just ranked the best 40 prospects to come out of that draft. I was planning on ranking the best 50 but I had a lot of trouble even getting to 40, so I finished there. There wasn’t a ton of depth in this draft class but I also believe most people overestimate the number of good NHL players that come from each draft.

Below are the results of my re-draft. You probably won’t agree with everything on it but I’m fairly confident the players are in the right range at least, and that is the most important thing for my review. If you think the 11^th ranked player should be ranked ninth instead, that doesn’t really matter for the review.

To calculate the Online Consensus ADP (Average Draft Position), a value of 40 was used in situations were a player was not ranked in the top 30 in one or more rankings.

The review process

Then, for the review part, I decided to use a scale from negative ten to positive ten. I rated each placement on the consensus rankings, meaning I didn’t look at who else might have been available at that point but instead I compared the placement on the ranking to the re-draft results to see how closely they match.

My logic for this was that if the player ends up being the fifth best prospect from his draft class, it doesn’t matter if you ranked him first overall or 20^th overall, you didn’t place him in the right range in either of those cases. And yes, I understand that an individual ranking (public ranking or one NHL team) may have had him ranked fifth overall but we’re not looking at those, we’re looking at the consensus rankings and comparing the two sides.

The results

First of all, there were some pretty big minuses given to both sides equally. Neither consensus ranking had Jake Guentzel, Tyler Bertuzzi, Brett Pesce or Dominik Kubalik in the top 46. Guentzel was ranked fifth in my re-draft, and the other three were inside the top 20 as well. Juuse Saros, Oliver Bjorkstrand, Andreas Johnsson and Will Butcher were the other names who were missing from both top 46 consensus rankings but found their way to the top 25 in my re-draft.

Additionally, both consensus rankings had Curtis Lazar, Frederik Gauthier, Kerby Rychel, Mirco Mueller and Morgan Klimchuk in the first round, although luckily outside the top 15 which was a positive thing for them. But then the big one; both consensus rankings had Jonathan Drouin third overall. He clearly hasn’t lived up to that placement at all.

So those were the common mistakes made on both sides equally. But because those mistakes were made evenly on both sides, it doesn’t do much to see which side is superior. Next, let’s look at the differences between NHL Scouts and Public Rankings.

Where NHL Scouts got it wrong:

• Samuel Morin was drafted 11^th. The Online Consensus had him 23^rd. Both sides were clearly too high on him but wasting an 11^th overall pick is much worse than wasting a 23^rd overall pick. You can blame injuries but he was never going to live up to that 11^th overall selection either way. Corey Pronman, Chris Peters, Ryan Kennedy and Future Considerations had Morin outside their top 30, so props for that.
• Emile Poirier, Michael McCarron and Marko Dano were drafted in the first round. The Online Consensus had Poirier 39^th, McCarron 43^rd, and Dano wasn’t in their top 46. Again, these ended up being mostly just wasted picks, and wasting a second round pick is much better than wasting a first round pick.
• Pavel Buchnevich was outside the top 46 (drafted 75^th overall). He was 17^th in the re-draft. The Online Consensus had him 36^th thanks to Corey Pronman and Chris Peters who both had him inside their top 20 when others didn’t even have him in their top 30.
• Artturi Lehkonen was outside the top 46 (drafted 55^th overall). He was 30^th in the re-draft. The Online Consensus had him 37^th thanks to Craig Button, Corey Pronman and Chris Peters who all ranked him inside the top 30 when others didn’t.
• Anthony Mantha was drafted 20^th overall, the Online Consensus had him 16^th. He was eighth in the re-draft. Not a big difference but worth mentioning.
• Rasmus Ristolainen (drafted eighth overall) and Jason Dickinson (29^th) were drafted a bit too high, and the Online Consensus was slightly lower on both: 10^th and 44^th respectively when they were 12^th and 38^th in the re-draft. Again, the difference was very small.

Where Public Rankings got it wrong:

• Nikita Zadorov was ranked ninth overall, he got drafted 16^th. Obviously not a wasted pick by any means but spending a top ten pick on a guy like him is just too rich. He was 22^nd in the re-draft.
• Robert Hägg was ranked 26^th overall, he got drafted 41^st. Again, he became an NHL player so that’s not so bad. But he wasn’t worth a first round pick. He was 39^th in the re-draft.
• Valeri Nichushkin was ranked fifth overall, he got drafted tenth. Nichushkin was arguably the biggest bust from this draft (32^nd in the re-draft), and every public ranking had him within their top six. NHL teams didn’t do much better but I think there’s a clear difference between burning a top five pick compared to burning a top ten pick. Expectations are much higher at fifth overall than what they are at tenth overall.
• Hunter Shinkaruk (ranked 14^th overall), Adam Erne (19^th), Valentin Zykov (20^th) and Zach Fucale (25^th) were drafted between 10 and 17 spots later than the Online Consensus suggested (24^th, 33^rd, 37^th and 36^th respectively). These four mostly became busts, and NHL teams avoided burning higher picks on them. It’s still not great burning a late first round pick on Shinkaruk and early second round picks on the other three. But it’s obviously better than burning a top 15 pick on Shinkaruk and first round picks on the other three.
• Tristan Jarry was ranked outside the top 46. He was drafted 44^th overall and ended up becoming a top 40 player from this draft class (34^th in the re-draft). A slight miss there.
• Madison Bowey was ranked 30^th. He was drafted outside the top 46 (53^rd overall) and mostly became a bust.
• Shea Theodore was ranked 38^th overall thanks to Corey Pronman, SportsNet and Future Considerations who had him in their top 30 when others didn’t. But he was drafted 26^th overall and became a top 12 player in this draft class. Obviously he should have been ranked higher by both sides but NHL Scouts were closer.
• Sasha Barkov was ranked fourth overall. He was drafted second overall and became the second best player in this draft class. Props to the Panthers. The Online Consensus had a clear top three of MacKinnon, Drouin and Jones. Those three had ADP between 1.7 and 2.5, so very closely together. Barkov’s ADP was 4.4 and he was considered to be interchangeable with Nichuchkin who had 4.9.
• Josh Morrissey was ranked 18^th. He was drafted 13^th overall and became a top 12 player in this draft class. Not a big difference but worth mentioning.

And the winner is…

NHL Scouts! Based on the points system I created for this review, NHL Scouts got 37 points whereas Public Rankings got 15 points. The theoretical maximum was 400 points, so both sides were way below that. But that’s the reality of the NHL Draft – no one has the perfect formula and we all make mistakes. It’s just a matter of learning from those mistakes and getting better year after year. It’s difficult to accurately evaluate and project 18-year-old kids.

But to answer the original question, even though NHL Scouts won this competition, it appears you can get somewhat similar results with a minimal budget if you just use the publicly available draft rankings instead of spending a lot of money on the scouting department – at least in the first round or so; NHL Scouts may have a bigger edge in later rounds. But you wouldn’t know if the player you’re drafting is going to be a good fit in your organization which could make the results worse.

One thing worth mentioning is that NHL teams have one advantage over the Public Rankings – the ability to affect player’s development and opportunities. High draft picks usually get more opportunities. NHL teams can do a good or bad job with the development of their prospects. These things can affect the end-result. It’s impossible to say with absolute certainty which ranking was the best because not all players are developed similarly and given the same amount of opportunities.

*

And that’s all for now, thanks for reading. Remember to follow me on Twitter @JokkeNevalainen.

Main picture courtesy of TheAthletic.com