While generally pleased as punch with its 2017-18 Key Performance Indicator (KPI) quality survey results (see http://stclair-src.org/news/need-know-news/st-clair-stays-top-five-grudge-less-kpi-results), St. Clair’s Board of Governors (BofG) and administration remain concerned about the provincial government’s method of gauging “Employer Satisfaction With Graduates”.
The matter was discussed at length during the BofG’s December 4th meeting.
A review of the generally favourable KPIs was presented to the Board by Corey Pyne, Coordinator of Quality Assurance and Outcome Based Education of the college’s Centre for Academic Excellence.
When it came to the Employer Satisfaction KPI, Pyne had a host of qualifying statements and critical analyses tied to his remarks.
The 2017-18 Employer Satisfaction survey saw 86.7 percent of the bosses of 2016-17 grads say they were “satisfied/very satisfied” with the job-preparedness and skills of their new, St.-Clair-educated employees. Eight percent were “neither satisfied nor dissatisfied”, and 5.3 percent were “dissatisfied/very dissatisfied”.
The average “satisfied/very satisfied” figure for the college’s two dozen colleges was 92.5 percent.
St. Clair, thus, ranked 21st in the province in this KPI category.
That was a stark contrast to its Top Five ranking in almost all of the other KPI survey categories, including those in which currently enrolled students adjudicated the quality of their educations.
Traditionally, St. Clair has floated around the high-80s to low-90s in the Employer Satisfaction category of the provincially mandated KPI surveys – sometimes above the provincial average, sometimes below.
“Is 87 percent really a problem?” President Patti France rhetorically asked the BofG. After all, in the academic field, that would be the equivalent of an “A” grade.
Nevertheless, she said, it is irksome that the number did place St. Clair near the bottom of the provincial list … So, yet again this year, the administration is setting out to understand what the result signifies …
… If it actually signifies anything at all.
The accuracy – and the usefulness – of this particular KPI statistic has long been questioned due to the methodology associated with its calculation. Pyne outlined some of its inherent “skews” to the Governors:
• A small sample size is involved because of the collection method.
First, grads who left school six months previously are contacted to respond to the “Graduate Satisfaction KPI Survey”. Participating in that question-and-answer process is voluntary on their part.
At the end of that survey process, the individuals are asked for the names and contact-information of their current employers – and if they will grant their permission for the survey-takers to contact their bosses to conduct the Employer Satisfaction KPI.
If the grad says “No” – and many do, because they don’t want their new bosses to be bothered – the matter is dropped, and no Employer Satisfaction survey is conducted.
If the grad agrees to having his/her boss interviewed, the survey company attempts to contact the employer. Again, participating in the process is voluntary on the employer’s part. Even if the grad-employee has given his/her permission, the boss can still refuse to participate, and the process ends there.
Assuming that both the employee and employer have cooperated through the initial phases, the KPI process is not as simply as asking “Are you satisfied/dissatisfied with Joe Schmoe, the St. Clair College graduate who you hired six months ago?”
Instead, there are several dozen questions involved in the formula that creates the final satisfaction/dissatisfaction percentage. The participating employer must devote a lot of time to answering all of those questions – and all of them must be answered because an incomplete survey is discarded.
All of the foregoing leads to the small sample size.
Colleges Ontario, the organization representing the administrations of the two dozen schools, noted that the survey-year in question featured a total graduate population of 100,752 students. Of those, less than half (47,200) completed their “Graduate Satisfaction” surveys. And, of those, less than a tenth agreed to have their bosses contacted, so only 3,200 employers completed the Employer Satisfaction survey.
Translation: Only three percent of grads are having their “employment satisfaction traits” evaluated by this KPI.
Think of what that may mean in a microcosmic example … A tiny school awards diplomas to 100 graduates in a particular year. At the end of the surveying process, only three of their employers have commented on their job performance. One of them isn’t thrilled. That one-out-of-three number means that it can be portrayed – and will be in this KPI scenario – that only 66 percent of employers are satisfied with this entire graduating class.
• Pyne noted that only 88 employers of 1,000-plus St. Clair grads were KPI-surveyed in 2016, and 72 in 2017.
• A truly odd anomaly popped up in this year’s provincial numbers, and has in the past too: Two schools boasted 100 percent, thumb’s-up Employer Satisfaction results – Lambton in Sarnia and Sault in Sault Ste. Marie.
St. Clair officials haven’t seen the detailed break-downs from all schools, but Pyne suspects that a result of that nature could only result from a miniscule survey pool.
Again, go back to the microcosmic example above … If the three surveyed employers of the 100 grads were all thrilled by the trio of new workers, that would give the school a perfect rating – unrealistic as that may, in actuality, be.
• One more “glitch” with the survey – this one raising the question “What is it actually evaluating?”
Here’s a paraphrasing of Pyne’s scenario, presented to the BofG:
Let’s say a student graduates from a program in a very complicated discipline; but, for one reason or another, secures employment in a totally unrelated and comparative menial field. Let’s say he has graduated as a brain surgeon, but is now working as a ditch-digger (no offence to ditch-diggers).
The boss of the ditch-digging crew completes the KPI survey. Maybe he is totally satisfied with his new employee, maybe he is completely dissatisfied with him.
In either case, what does that evaluation really mean? The employer is, obviously, not evaluating whether the employee was educated to be a competent brain surgeon. He, for the most part, is only providing an opinion about the graduate’s innate work-ethic, not about the specifics of his academically-provided skill-set …
… So, from the college’s perspective (and the public’s too), what constructive, informative value – if any at all – does such an individual KPI have? It doesn’t address the efficiency or efficacy of curriculum content or the educational delivery method. And, because it is so off-topic, it doesn’t really reflect on the potential-for-employer-satisfaction that would have been more realistically gauged if the graduate had actually been employed in his academic discipline. A response of “I’m satisfied/dissatisfied with him as a ditch-digger” says nothing about the school’s brain-surgery-training program, nor about his/her fellow grads, nor about the overall quality of the school as a whole. Such data is of no constructive use to the school in terms of improving itself, and should hold no weight with prospective students who are thinking about enrolling there to become brain surgeons.
Both Pyne and France described this particular KPI’s methodology and results as “flawed” and “prone to being skewed”.
They aren’t quite certain if the Ministry of Training, Colleges and Universities is prepared to revamp it – or how it might be able to develop a more accurate method.
But while the college isn’t convinced that it can attach a great deal of veracity to this particular survey, it does remain interested in feedback from the employers of St. Clair graduates.
It does so, now, in anecdotal fashion, with administrators and faculty members questioning employers about their satisfaction levels whenever they encounter them …
… And it is considering implementing its own, more scientific (numerically thorough) surveying procedure in the future – “not to replace the provincial KPI, because we really can’t do that anyway,” Pyne said, “but purely for our own information and research purposes.”
