Nice thread. I agree that being a high rated player does not mean your a helpful coach. Not at all. @DrHack is claiming some major numbers. Wonder how he does it?
"The power of instruction is seldom of much efficacy except in those happy dispositions where it is almost superfluous" - Gibson
@malbin
Yes...I concur a single metric for Anything will almost always tell a lie. Even in that article that you linked it tells about using a balanced score card...Hence counter balancing metrics.
On Time Delivery has to be balanced with Inventory Turns.
Top Line Growth has to be balanced with Margin erosion (probably a bad example I hate sales metrics)
ELO Growth would have to be balanced with student drop rate, student retention, student denial rate.
So no I still disagree with you. If we measured coaches on how well they increased ELO and they had students that struggled to gain ELO knowing they would be penalized for dropping the student...the metric would drive them to become more innovative and creative. I have over 15 years experience driving cultural change through true north and line of sight metrics and this holds true 97% of the time.
Yes you will ALWAYS have a minority that skids the system...but if you try to develop a system around the minority instead of an audit process then you've already accepted defeat.
Yes...I concur a single metric for Anything will almost always tell a lie. Even in that article that you linked it tells about using a balanced score card...Hence counter balancing metrics.
On Time Delivery has to be balanced with Inventory Turns.
Top Line Growth has to be balanced with Margin erosion (probably a bad example I hate sales metrics)
ELO Growth would have to be balanced with student drop rate, student retention, student denial rate.
So no I still disagree with you. If we measured coaches on how well they increased ELO and they had students that struggled to gain ELO knowing they would be penalized for dropping the student...the metric would drive them to become more innovative and creative. I have over 15 years experience driving cultural change through true north and line of sight metrics and this holds true 97% of the time.
Yes you will ALWAYS have a minority that skids the system...but if you try to develop a system around the minority instead of an audit process then you've already accepted defeat.
@HarmlessChessNub Thanks for elaborating and for providing those examples. I agree with you that combining metrics (and particularly counter-balancing metrics) is more useful and less prone to manipulation.
@malbin Going to attempt some stuff on that.. not sure how far I will get.
I edited out some questions here after further research.
Also.. how does one figure out what time frame is relevant to slice the data? I'm having bad flashbacks to p value testing... Tell me I'm wrong :(. It would be nice to get stats out faster, but the stats might break down if the measure is too short. (guessing)
I edited out some questions here after further research.
Also.. how does one figure out what time frame is relevant to slice the data? I'm having bad flashbacks to p value testing... Tell me I'm wrong :(. It would be nice to get stats out faster, but the stats might break down if the measure is too short. (guessing)
@DrHack
"what time frame is relevant to slice the data" That's a loaded question and it depends on the desired out come...but when it comes to performance metrics...further back you look and the more inclusive it is of the data the better you understand your actual performance.
You could even look at it in a form of SPC or "statical process control"...coaching is a process and how in control is your level of improvement...
Obviously a small sample size of can give you an insight but is not statically significant. This will lead to confirmation bias potentially. But if all you are looking to do is to create a baseline then a sample size of 30 would most likely be enough to create a normal distribution of data...(probably not gonna be a normal distribution and probably be a bi-normal..)(high performer and low performers). Due to the huge amount of lurking variables you have of learning something like chess...
Further more... what's the average life time of a student...what's the std deviation of the life time of a student...and not just for you but for all coaches...So you have to take your sample size and compare at least as an interpolation to the population. If you took 10 Coaches who looked at 30 students each you'd actually be able to develop a form of baseline to measure your personal metrics on.
@malbin Always love a healthy conversation, enjoyed the article you linked because I preach single metric departments are failing themselves constantly.
"what time frame is relevant to slice the data" That's a loaded question and it depends on the desired out come...but when it comes to performance metrics...further back you look and the more inclusive it is of the data the better you understand your actual performance.
You could even look at it in a form of SPC or "statical process control"...coaching is a process and how in control is your level of improvement...
Obviously a small sample size of can give you an insight but is not statically significant. This will lead to confirmation bias potentially. But if all you are looking to do is to create a baseline then a sample size of 30 would most likely be enough to create a normal distribution of data...(probably not gonna be a normal distribution and probably be a bi-normal..)(high performer and low performers). Due to the huge amount of lurking variables you have of learning something like chess...
Further more... what's the average life time of a student...what's the std deviation of the life time of a student...and not just for you but for all coaches...So you have to take your sample size and compare at least as an interpolation to the population. If you took 10 Coaches who looked at 30 students each you'd actually be able to develop a form of baseline to measure your personal metrics on.
@malbin Always love a healthy conversation, enjoyed the article you linked because I preach single metric departments are failing themselves constantly.
Oh right. @HarmlessChessNub I was taking the angle of: "What if a new coach walks into the arena?" How soon is the data going to be relevant to telling a story, and when is it still unfair because of potential outliers weighing down the data - as some early posters drew attention to this potential pitfall.
For my personal data.. I've got 4-5 years of recorded back data. I can see my own performance changing over time, but what if I started over today?
For my personal data.. I've got 4-5 years of recorded back data. I can see my own performance changing over time, but what if I started over today?
@DrHack
Do you have any way of evaluating student performance other than rating? An earlier post referenced year-end rating assessment. How frequently do you check ratings?
In my opinion, having more granular data is better. Yes, there will be noise, but noise can be explained.
How far back to show? As far back as you have. Are your current students improving faster than your first students? That in itself shows that you as a coach may be learning how to coach more effectively.
Would be interesting to compare student improvement with longevity. That might be a good set of indicators to pair, like @malbin talked about. I like Andy Grove's take on this in "High Output Management." He's got great examples.
Is there a correlation with continued improvement that goes with longevity? That would be more valuable than just knowing how long students remained with you. Do the students that improve more quickly stay longer or shorter?
Do you have any way of evaluating student performance other than rating? An earlier post referenced year-end rating assessment. How frequently do you check ratings?
In my opinion, having more granular data is better. Yes, there will be noise, but noise can be explained.
How far back to show? As far back as you have. Are your current students improving faster than your first students? That in itself shows that you as a coach may be learning how to coach more effectively.
Would be interesting to compare student improvement with longevity. That might be a good set of indicators to pair, like @malbin talked about. I like Andy Grove's take on this in "High Output Management." He's got great examples.
Is there a correlation with continued improvement that goes with longevity? That would be more valuable than just knowing how long students remained with you. Do the students that improve more quickly stay longer or shorter?
@ericmsd I do not have any other quantifiable measure. I haven't put any deep thought into it. Asking them all a question like, "do you think you improved?" will probably carry too much bias to be worthwhile. I have a marketing background so I'm on the dumber end of smart.
Can you think of anything that might be worth a look?
Can you think of anything that might be worth a look?
@DrHack
I wonder if you could get a good sense of progress by having a program rate the ELO of their play given PGN files.
Lucas Chess will do this based on how far off the moves are from the engine's top choice. It assigns an ELO for the game overall , for the middle, and for the ending. It also gives a *bunch* of other statistics. That said, my own games vary wildly, but I haven't tried to do any type of statistical analysis on them to see if I could plot anything that makes sense.
In another thread, someone posted about categorizing their mistakes: lichess.org/forum/game-analysis/analyzing-blunders. That's going to take a lot more manual effort to track, but presumably you're going through games, anyway.
Could you use puzzle rating in various areas? Ask that your students do at least X number of puzzles a week, and then track their progress in the categories that Lichess uses? lichess.org/training/dashboard/30/dashboard
"I have a marketing background so I'm on the dumber end of smart." ROFL. I think we're in disagreement on your self-assessment. ;)
I wonder if you could get a good sense of progress by having a program rate the ELO of their play given PGN files.
Lucas Chess will do this based on how far off the moves are from the engine's top choice. It assigns an ELO for the game overall , for the middle, and for the ending. It also gives a *bunch* of other statistics. That said, my own games vary wildly, but I haven't tried to do any type of statistical analysis on them to see if I could plot anything that makes sense.
In another thread, someone posted about categorizing their mistakes: lichess.org/forum/game-analysis/analyzing-blunders. That's going to take a lot more manual effort to track, but presumably you're going through games, anyway.
Could you use puzzle rating in various areas? Ask that your students do at least X number of puzzles a week, and then track their progress in the categories that Lichess uses? lichess.org/training/dashboard/30/dashboard
"I have a marketing background so I'm on the dumber end of smart." ROFL. I think we're in disagreement on your self-assessment. ;)
This topic has been archived and can no longer be replied to.